Hey.

You can have 3 friends at the same time in your referral list, but if you see that one is not playing anymore you can remove him and add another one.

You are not limited for the overall amount of referrals just for the amount at a time.

Cheers

After 3 of my referals reach level 50 and I get 4 sacred shards, can I remove them and add new referals?

Sure, once they've reached lvl. 50 you can remove them and add new for more rewards.