The problem here is not about how you look... BUT that this neighbor has KEYS to your home... and treats your home, like it is his.
So- TAKE AWAY HIS KEYS TO YOUR HOME.
And don't be shy about it.
It is YOUR property and home.
And if he harasses you/Hubby, call the Cops.
Document everything.
They are not friends.
That neighbor... is NOT the kind of guy, to give your house keys to.
AND, your Husband has to back you up. AND tell the guy.
This neighbor, will not, stop.
He has your keys, too.
So, cut off access to your home.
Frankly, I would have TOLD HIM OFF RIGHT THERE... in no uncertain words, and taken the key back too.
AND if he had the nerve, to make copies of your house key too... without you knowing..... then call the Cops when he trespasses.
Step up to him.
So what if it pisses him off.
He's a jerk.
Don't let jerks take command of your home and of your self.
CHANGE THE LOCKS on your home.
I would have TOLD that guy off, royally.
And had no problem doing so.
WHY are you letting this guy, control you?
He KNOWS, he can control you.
CHANGE YOUR LOCKS, today. AND take back his key, TODAY.