Gems accumulate slowly at first but later you have lots.  I have a very small town (9 billion currently) but I have several hundred gems, maybe two hundred legendary were I to fuse them all.  It rather sounds as though you lack clan mates in that position in which case precious dust is currently irrelevant to you (as it is to me).  That is because initiating  siege status is pointless without heavyweight clan members with large armies of troops supported by maxed out military knowledge/Valkyries etc. and top quality gear.  Lacking those things the troops with which you could oppose the marches sent against you would hardly inflict a scratch.  Even powerful clans with numbers of 500 billion influence members must pick and choose the marches to oppose and, if my kingdom is anything to go by, mostly expect to lose (although gaining points).
If yout clan has not yet advanced your stronghold to level four you might consider being careful about when to do so.  Advancing the Stronghold increases overall influence quite a lot which both gets you tougher opponents and also tends to send you into a higher League,  Which makes checkpoints harder to reach.