We always held small parties and the parents never stayed. We never felt the need to ask them to. We used the "age rule" - when our son turned 4, he invited 4 friends. When he turned 6, he invited 6 friends. We had fun activities (old party games, scavenger hunts, etc.) in the back yard or the basement. We didn't have 30 kids, so we didn't go to 30 parties throughout the year. We didn't need to provide food & activities for the adults, just the kids. And it's not overwhelming when you have 5 kids, a couple of planned activities, just 5 gifts to open, and 5 goody bags to prepare. People get overwhelmed when they have 30 kids, a clown, a bouncy house, chaos, and 30 plates of chocolate frosting hitting the living room rug!! Keeping it manageable for all concerned is the key.
A pool or lake party is a different thing (safety) especially with a ridiculous number of kids. But I think the whole party is more fun when a birthday child doesn't have 20 kids and 40 parents in the house! No kid really wants that or benefits from it.
It's nice for kids to develop a sense of independence and not have their parents there all the time. They learn to behave and to participate in things. I am not sure why people feel taken advantage of - the invitations must start a starting and ending time, and you should have the parents' phone numbers. If the party is from 1 to 3 PM, at 3:15 the hosts should be on the phone locating the missing parents. This is not day care, it's a party.