There are many buildings which are important but i will go for harbor.
Since everyone is adding his/her explanation. I am editing this post to add mine.
You have got some troops at the start of the game and than it adds up as you complete normal quests. If you don't have harbor than you won't be able to hide your troops. If you can't hide they will get killed and when they get killed you can't raid other players. If you can't raid other player than in the terms of upgrading you will be on a turtle speed.
Declaring the Winner - None of you
A hint was given for the correct answer in BOLD
Answer was All of them
Why:
Res building - without these buildings you can't build, upgrade or que anything.
Warehouses for g/l and rum - without these buildings your max capacity for any res is 2k.
Stronghold - Everything is controlled by it and without it you can't discover anything past Mauraders
Observatory - No discoveries can be researched without it
Dutchmen's Hangor - no Scouts, Subs, Ship o Lines, or Juggs can be built
Shipyard - no Fleet units can be built
Armory - no mercenary units can be built.
Infirmary - without this the power to regain attacked troops or to revive fallen troops with rubies is not possible
Cemetary - Can't get dark corsairs without it.
Captains Quarters - This building increases your upgrade speed
Market - no ability to use the market for trades of res or sketches. Oh and say goodbye to those 50k+ raids
Whitmore & Co - No targets can be captured without it
Witch Doctor's Hut - your pearls would be useless. Unit upgrade speed, unit speed, rum consumption, pardon cost reduction ect..would not be available.
SteamWorks - No advanced scouts, orinthopters, nautilus subs
Tavern - Can't produce any troops without it.
Lighthouse - No prizes, adventures, targets or tapsites. You can't build an army you have to win it
Windmill, Sawmill, Goldsmith - each give you a 25% boost to Resource production. Especially the Windmill these are very beneficial.
Manufactory - Can't make gears without it.
Merchant's Guild - increases galleon speed, without it galleons would be extremely slow
House of Navigation - Increase Galleon capacity. Limits your ability to send res to mates and how effects how much less you get in 80k+ raids
Alchemist shop - barrels may seem useless to some but the off/def boost they supply can help the winning or losing of a battle
SKeleton Cave - bonuses temporary or permanent to unit build speed, capacity, power or even effects on production of resources can all be effected by relics applied here
Brethren Court - no ability to join a Brotherhood or parleys available without it
Harbor - no upgrades only 1k of each resource is protected.
2 Random players will be chosen by Anne Bonnie to receive 5 orinthopters each.