Your concerns are understandable. I have two boys, 12 and 9. Our 12 year old has a May birthday and he started Kindergarten at age 5. He has done fine both socially and academically. In fact, he is in AP classes. Our 9 year old has an August birthday and we waited a year with him. It wasn't as much of a birthday issue, as it was a "is he socially and mentally ready to go to school?"
When making your decision, you need to focus on the needs of your son. Keep in mind he is right in the middle of the pack age wise and that is a good thing. If you decide to wait, he will be turning 7 while in kindergarten while many of the students will still be 5 turning 6.
Make an appointment to talk to your school's principal or a kindergarten teacher at the school. They can help you make a more informed decision.