No. And, the immune system of a baby is not fully developed yet at this time.
Babies, do not 'play' interactively. They do what is called "parallel play." Look it up online.
Also important at this age is 'bonding' with the parent. This ALSO aides in the development of a baby....
ALSO, babies, do get over stimulated very easily and quickly... and then over-tired. So, usually 'activities' for babies are short. They are not like an older child that has all day activities. And babies also nap a lot.... which is also important for development and growth.
AND, depending on what you are doing, a baby is more prone to getting sick... and whether or not baby is getting vaccinations, they can get sick.
Then, just having normal daily activities with a baby, even if it is just with you and doing errands, is exposing them to contagions.... but also building their immunity too.
You don't have to worry about 'interaction' for your child yet. He is so young. When he is a Toddler, say about 2-3 years old... then at that point, a child does benefit from socialization. More so. Per their developmental age-juncture and age-stage.
Your thoughts on it is perfectly fine. YOU are the Mom. Do as you see best for your baby... and in light of winter time coming up and the usual seasonal illnesses that arise. ie: Flu etc.
When my kids were the age or your baby... I too, during seasonal illness times.... I CHOOSE what and where to take them, to prevent serious illness.
He will NOT be a "Loner." He is only 4 months old. He will not get 'bored.' At this age. Babies, amuse themselves as well. ANYTHING is stimulation for them. Even a bath.
Having a Nanny for him is perfectly fine. And then that way, you can tell the Nanny what/how/when as far as what you want for your baby and/or his schedule/naps/routines.
Don't worry about it. He is so young. I have 2 kids... who are 4 and 7 years old. They did not have a sense of 'boredom' until about 3 years old.
Per their development.... and personality.
Babies, do NOT have to be constantly amused. They do get tired/over-stimulated and also just hanging out is 'learning' for them. ANYTHING in their environment, is 'stimulation' for them. Even if just watching you clean house.
your son is so young. Don't feel pressured.
all the best,