Dealing with Uninvited Guests?

I desperately need advice people. My fiance's family and friends are constantly stopping by my place unannounced. I am a nurse who works long days and gets basically no time off and zero time with him. So when I come home I want to be left alone and have peace and quiet with my man, but it's impossible with people constantly banging on the door. It's to the point where I'm a getting a little nasty about it now. It make's me angry that people have no respect for our time or privacy, family or not.

Today I actually had the afternoon free and the moment I start feeling stress free and start thinking maybe we can have a nice afternoon together, someone's beating on the door and low and behold it's his uncle and wife with beers in their hands.

My man refuses to say anything. He doesn't wanna hurt people's feelings and says he grew up that way. None of these people, including my fiancé, work as much as I do. Nobody seems to understand what my problem is until I catch an attitude about it.

I finally did say to them today that I don't like unannounced company because we never get any time together and I don't really want people drinking here. Then I went inside the house while they hung out for a few with my man and eventually left. I feel like a jerk, but c'mon people! Do you all think I need to lighten up and get over it or do you think I'm in the right for not wanting constant visitors after a long day at work?

This has been such a huge source of anxiety for me for months now. I feel like I want to move out if this doesn't stop, am I wrong for feeling this way?