I too have what is called "a spirited child". There is no medical reason to question this behavior. It's quite simply energy. I had it. And one of my daughters does.
There is actually a book called "How to Raise Your Spirited Child". I think it helps you feel not so alone in your quest to raise a positive, playful child who is NOT labeled, "ADHD".
We continue to have problems with the "hands to self" rule. We do often also talk about "The Golden Rule". I have to remind my daughter how hip checks, grabbing toys to engage others (to play with her), etc. IS NOT how she wants to be treated. I think this constant reminder of "how would you feel if I (Mom) did _____ to you?" More often than not, my daughter says, I wouldn't like that, you're so much bigger, or it would hurt...this helps me make the point.
We also have the problem with the running, LOUD talking, etc. It has never been a problem at school and no disciplinary action has ever been taken at school. However I FEEL like a mother hawk.
My advice, other than reading that book, is simply, Keep Hand to Self, Follow the Golden Rule and if/when these fail, sit him down - immediately, remove him from the situation, let HIM tell you what he did wrong. I think sometimes for our kids, STOPPING the world for a few minutes MAKES THEM THINK. If all else fails, leave the party/playdate/restaurant. Our children know that "bad behavior" is not acceptable. That children who don't behave do not get special privileges such as dinner out, etc.
Also, you mentioned not being able to catch your son "in the act". Why not? Have a playdate? How does he behave with one friend? What about the birthday party? See how he is in a group.
I have noticed my child seems better with kids older than her since THEY keep her in check. The tell her not to push or "be bossy". With her peers, she needs a grown up to help manage the situation.
Since your child is energetic AND an only child maybe he is also struggling with how to behave without your presence - as in, in a group? Does he play alone at home? Does he do puzzles or have "quiet time" in his room WITHOUT engagement from you? If not, I would suggest starting this practice. It may help him to NOT have attention. I think "quiet time" every day has helped child to step back, slow down and LEARN patience. I think that's part of the process with our "spirited" children - is teaching them boundaries and patience.
Best of luck. Let me know if I can help with anything else.