No, it's not rude if you word it well. I would write something at the bottom of the invitation similar to: Due to safety regulations of Tiny Tot's Gym (or what even the name of location is), only children on official guest list may attend.
This gives the parents the opportunity to ask if there is indeed space for more children, at which point you could simply say "I'm sorry, but this location only allows for 14 visitors, and my son filled the list with those he chose to invite, but if someone cancels I can certainly let you know." This way you clearly place the blame on the location, which is the biggest part of why you can't allow just anyone to come.
Hope this helped, ~J. :O)