The idea is supposed to be that the Guardsmans gear gives the same benefit to every invader,but I do not know if this has been tested.

Turbomaster says that  there is a difference for some of the level 6's, so its possible that there is some for the others as well.

What you do have to remember is there is a luck element of sorts, and Turbomaster didnt mention the critical hits, if you get these its possible to  kill invaders quickly, if you dont, then it will take more hits  to do the job.

Also, there are days when you get the 10% bonus for attacks. this will affect the number of hits needed, as will any bonus you have applied, so when doing tests you have to eliminate the chance element as far as possible,and to compare like with like.

I haven't seen anything to suggest that different gems suit different invaders.

I once thought that there might be a difference in my effectiveness against different invaders but now I think that as WDYWTNTF suggests that what I was seeing was the -10% hero energy loss bonus on some days and the random factor of critical hits. There is even a random factor in each individual hit.