People are idiots, and they all have opinions. Of course you can have a plant based diet that will be healthy for both adults and children. Millions of people in other countries do it! It's worth researching exactly what nutrients growing children need, and then carefully planning a week's worth of meals to see if you can do it, and if your children will eat the food. We rely heavily on plant based foods, but one of my daughters simply will not eat beans so her protein options are too limited without some animal protein. She loves hard boiled eggs and cheese, so we do have those.
Here are a couple of resources to start you out:
Raising Vegan Children in a Non-Vegan World: A Complete Guide for Parents (a book)