Not surprisingly this type of question has been asked before. 

In case this is an urgent matter, and you have little response I suggest you search the forum for the previous answers.

 We cannot accurately tell you what to do because we do not know how you play the game,only point you in various directions.

It would serve you  to read what the benefits from these buildings are, and how the matter to your game.They change as you progress.

For example, if all you do is attack invaders with your hero,you have little use for troops, so you do not need either barracks or infirmary,except for any requirements necessary to enlarge other buildings.

If you want to be able to either heal or resurrect all your  'lost' troops, then you need enough infirmaries, or one, or several of them to cover the amount of troops you have.

If you want to train more, then you need to think about how quickly you want to do this.

If you never attack, and are constantly shielded  you have little need for more troops than you can send out on marches.

I had the same quandry.....and followed advice to destroy barrack and infirmaries. i had five of each and now have one  of each. replaced all with Manors

After a while i regretted getting rid of all the barracks because training troops was harder.... i could no longer just leave 5 or 6 thousand troops training while i was offline. 

However now i would say it was the right advice....it has made me more concious of troops costs and while online it is no slower to train...i restrict myself to one hour boosts for troop training to save time and use all others for town/citadel/knowledge upgrades