We go every summer with our extended family and children of our own, now 2 and 4.
There is a ton to do... my kids love Plum Loco, it is a play place out door for kids (it's kind of hard to explain, but well worth it.and not crowded) We also go to Pennisula state park for biking, hiking, a beach and play ground. We listen to live music in the park at night ( a different town each week) We eat at a fish boil, you can take a ferry ride to Washington island. We do PC junction to eat and see the goats on the roof, all of which are very busy resturants. The green bay side of the water is warmer, but we do both that side and the lake side. There is shopping and fishing, which might not be too exciting for your little one. Every weekend there is something going on in one of the towns, from festivals to farmers markets.
Have a great time.