Based on players complaints and my own experience, the special gems and keys do not drop like manna from heaven.
As I have pointed out, hitting Ubers as you seem to do would drain my energy stock faster than it would be replaced.
Others may want or need or find a use for levelling up special gear but it's most unlikely I will in the forseeable future, if ever.
You can certainly play the game as you desire, but all that chasing is going to give you so little benefit from having them that you, and others, would see if only the scales fell from your eyes.
Curiously I had been unable to find any RGM on the map,but they now appear to have respawned.
When you hit lvl 1 invaders ok the guardsman gear hardly helps (for that I do not even bother to put it on ... also I do not put it on for lvl 2 invaders) ... when you are hitting lvl 6 and ubers it helps ALOT ... as you already said you spend 1.5 times as much on an uber if you do not have it ... if you want to get top knowledge and building gear for hero and shaman ... I do not think there is enough time in a person's life to get it hitting level 1 invaders ...
That being said for points in certain events you can get more points hitting lower level invaders/ghosts than higher level ones with same amount of energy ... but then again time is a problem and also what you get as far as drops per energy ...
And the "if only the scales fell from your eyes" comment ... totally unnecessary and quite condescending ...