Think about it from the opposite perspective for dads who only get to see their kids like you see yours. Their relationships aren't damaged because of it. As kids grow up, they realize that their parents work and that this is how the bills get paid.
When he says "I don't like you", just say back "Well, I like YOU, but you still have to get ready for bed. Which of these two pairs of PJ's do you like?" In other words, deflect it and don't act like it bothers you.
How is your relationship with your ex? If it's a good one, ask him to talk to your son about how important mommy is. If it's not, don't bother.
If you two stay living nearby each other, your husband will eventually stop playing the "fun" card with him and real school and homework, etc will kick in.