The first thing I thought about when reading your letter was that I would talk to the other neighbors, if you get along with them, and find out what is going on with them and this neighbor. If they get along with them ask them what the "mean neighbors" say about you so you can see what the issues might be.
How long have they lived there, do the act this way to everyone? If yes, then I think I would do everything in my power to keep the two yards, and people, apart. I would be suspicious of a family that didn't let children over to play, or be friendly with new neighbors. It makes me wonder what is going on in their home that they don't want anyone to know about???
We had a new family move into the house across the street and over one house. There were two pre-school age children and one in elementary school. I would come out of my front door and the new kids would be playing on my swing set, riding our bikes in the street, etc.... I can guarantee I wondered where their parents were. I have every person in the neighborhood walking through my yard at all hours day and night to go to a common area between the rows of houses and a shopping area. I have even left my purse out on the deck all night and it wasn't touched. But because of these kids I had to start doing things differently. (We even considered getting a fence. We would be the only family on the street to have one.) I told them, after several times of going to their yard to get our bikes and toys, that they could play with anything but they had to put it back in my yard when they were done.
They sold their house and moved because I was the "mean neighbor". I didn't feel I was mean, I didn't yell or cuss them out, I didn't call Child Welfare on them because I never saw a parent outside, I felt like I went out of my way to be nice. It's all in how it's interpreted.
I have some friends who bought a lot in a secluded neighborhood a few years ago. Their children were all in Jr. High or High School and they were in a position to spend the time and money to build a dream home. The kids were old enough to take care of the flooring and new furniture, etc.... It was the most beautiful home I had ever seen. I wanted to crawl in the Master tub and just take a nap and I could have lived in the Master closet. Anyway, their neighbor on the North side was a total jerk. He yelled at the kids when they were out side swimming or hanging out with their friends, all the time being mean. He would pour weed killer on the plants they had along the fence line because they were coming onto his property. He even filed a law suit against them about some issue or another. They decided the dream home could be rebuilt. They put their house up for sale and bought another lot and started over. They took a hit on the house because they hadn't really lived there long enough to build up equity. They are happy now. They live along a winding road by a lake, have horses and other pets, they love their new neighbors and host neighborhood parties often.