Please do not second-guess your own good instincts! Stick to what you know is right for you and your family. And no matter what some folks try to say about how you just must have a sibling for your child, you and your child and husband ARE a complete family just as you are.
More than one kid is fine but like you, I know that for ME and my husband, one is what we can and are willing to handle.
For the next time you feel any doubt, consider these things:
You will always be able to volunteer as much as you like at your son's preschools, schools, activities, etc. You will not have the pull of having other kids for whom you also feel you must split your volunteering time. Volunteering will be a huge help to you in knowing your son's teachers, friends and academic setting.
You will not have to say this: "Sorry, I know you would like to [fill in the blank: Go to that festival, see that puppet show, visit that museum, go to that event] but we can't because it's not something the baby/toddler/your little sibling can sit through." Your son won't have to miss out on things because those things are "just too much to do with two of you." I've seen plenty of moms I know have to tell the oldest child for years that "we just can't do that" because younger kids basically set the agenda and schedule with naps and short attention spans.
You will be able to give your son full attention if he needs help with homework or needs to talk to you about anything.
You will have far less to worry about as your son approaches college age and you have to think about saving for college.
You will not have to deal with years of sibling fights. Some people insist that children "NEED a sibling so they have a childhood playmate! A lifelong friend!" but that is bunk. Sure, it happens, and it's great when it does. But there is zero guarantee that siblings will get along or will share any interests or will even stay in touch once they grow up. One should never, ever have a child just to provide a playmate for another child. Do not ever give a second's thought to folks who try to tell you that you "owe it to your son" to provide a sibling. You don't. Just read the posts on here about sibling rivalries and youngster's jealousies when the new baby comes. (And you already know that your son is outgoing and enjoys life so it's not like he sits at home moping, "I want a little brother to play with!")
I am not saying any of these things to bash the idea of more than one kid. I know plenty of families with two and three and more great kids and the parents love it. But these are things to tell yourself next time some bone-headed stranger (or a loved one who is just not thinking) tells you that you "need" to have another child.