Does this kid do this stuff when the boys are off on their own, or does it happen when adults are present? Is he more apt to get physical when he's doing activities that can get him excited, like playing ball, or video games? Or can it happen even if people are just sitting around passively watching tv? I guess, I'm wondering if there seems to be a trigger for this behavior.
If you can pinpoint a trigger, you might be able to salvage a simplified relationship with them (if you're son is up for it, of course!) by suggesting to limit visits to structured activities that work well with this kids temperament. So maybe, once a week or every two weeks the four of you go to a movie and have lunch, would work if the kid can handle passive, yet supervised activities like that.
Or maybe you could go to the zoo or park for an hour or two as a group if he can handle that kind of stimulus.
I suspect he's probably not a good candidate for unsupervised play, where they can get into expected rough housing or running, and physical activities like playing catch or something.
But maybe, if he is invited to very structured activities or get togethers that can be paired down to about an hour and a half or so, he might be more tolerable, and the relationship can be saved.
However, with that said, if you're son isn't up for the challenge, I wouldn't push it. That's a lot to ask of him. The friendship and the effort behind it should be genuine for everyone's sake.
I just think, if this behavior is the only thing that is making your son not want to visit, but he otherwise thinks this kid is great, this might be a good opportunity to learn how to deal with differences. Afterall, any friendship has it's limitations. People learn to adapt with different people and situations.
If you feel your son isn't in serious danger (because the kid is unreasonably violent or something) and there can be some compromise, it might be worth at least one more try with a plan behind it to see how it goes before throwing in the towel. Talk with your son and see what he honestly thinks, and if he's game, then see what Grandma thinks and take it from there.
If your son is up for it, maybe get a couple of books about ADHD aimed at young readers. He will understand better why his buddy is this way...and might learn ways to communicate better with him too.
Just food for thought.