When I was younger, I was "Bobby". I'm 9 &11 years older than my brother and sister. My next door neighbor was close in age to me. I seriously thought we could play every day since he was slightly older than his brother and sister too. I would sit in my window and wait for them to come out or go swimming. Even worse I would sit on my play set and watch the family swim. My mom tried to keep control over my inviting myself over as much as she could.
Anyways, coming from "Bobby" I think its better to come from you and not just his mom. My neighbors and friends' parents never told me that I came over too much. My mom always said something and years later told me that they talked to her. All that time I thought it was my mom being "the bad guy", I didn't know I was annoying my friends. Start with a simple 'we need a family day' and see how the response is.