What a horrible situation to be in! I wish there were more I could do, but here is a big hug for you.
Now for the BF: he is right, your son will say "you're not my father" to him once. My step-father appeared in my life when I was 4, he raised me way more than my father ever did, yet I said that to him. Your BF shouldn't take it personally, all children say it to their step-parents at some point, and it has no more meaning than the "I hate you Mom/Dad" that children also say. You can tell him that. :-)
This being said, and to the more important point, your post raised a big red flag to me, which is why I wanted to give you my two cents, even though I am in a very different situation from yours. I may be off because all I see is your story, but here is what makes me worried:
As far as I know without being a lawyer, you are absolutely right. You cannot change visitation rights without agreement from both parties or something that justifies it changing, and your BF in your life definitely doesn't qualify. Also, he cannot adopt your son without your ex revoking his rights.
But I'm really worried about your BF. First he seems to want to control things about something that isn't in his authority at all, i.e. the relationship between your son and his paternal family. Then he doesn't seem to want to accept reality the way it is, rather wants to change it to his liking. And third he doesn't seem to respect you: he obviously doesn't respect the fact that you told him this is how it is, that you kept your married name for your son, and wants to be the one making the decisions. He also doesn't accept his place, which as a step-parent is, granted, somewhat awkward (you have to defer to the parent for a long time, the parent is the one deciding on the way to raise the child, and you need to support the parent even when you disagree).
Is it the only area where he is controlling in that fashion, or are there other areas where he does it too, maybe in smaller ways, like ignoring (or not even asking) where you want to go when you guys go out, which restaurant or type of food you want, etc.? If this is the case, kick him out today! He's controlling, and it'll only get worse when you are married, because he'll feel that he now owns you. Even otherwise, I would tell him goodbye unless he completely and permanently stops his behavior, because it is plain unacceptable, and you and you only are the decision-maker for everything regarding your son.