Mini Vacation

I second Angela B's suggestion. Williamsburg (depending on which side of Richmond you are) is a good day trip (45 minutes give or take a bit). Busch Gardens/Water Country (and you can get the Fun Card/Sizzler for the entire summer 64.99 for each person 3+ for Busch). Colonial Williamsburg is cool too. My only suggestion is go before the tourists get there. (Generally by 4th of July. Make it a long weekend before school gets out.

Virginia Beach is also a good selection. There are so many things to do (both free and cheap) in the Hampton Roads area.

Either place would be great for either scenario. Contact me directly and I can give you some additional suggestions for things to do in the area. (I don't want this to be miles long of things to do) :)

Love Williamsburg! Within driving distance, lots of fun attractions, and family friendly. Also, VA Beach. You could also do a stay-cation and go to King's Dominion and stay at nice hotel.

Updated

Also, check and see if there are cruise packages leaving from Norfolk. Not sure what your budget is, but cruises include all meals, have activities for different ages, and some offer babysitting. There are discounts for children, and depending on when you go, you can get some pretty good deals that cost about the same or less than going somewhere and staying at a hotel and buying each meal.

I say Washington D.C. and I would take the kids so you're not stressed about leaving them.
Then you can either take someone with you to help w/the kids (a teen family member even?) and make sure everything you do is low key and centered around the kids' timetable.
For instance.....if you go to the museum w/the kids, you don't stay long. Maybe an hour or two. etc.
Eat at the hotel to make things easy.

I've never been, but I've heard the Outer Banks in NC are wonderful.

If it's just hubby and you, how about Washington DC. A hotel may be expensive, but I've heard there are tons of free things to do.

Dutch Wonderland (www.dutchwonderland.com) in Lancaster PA. The 10 yr old and 20 mos old will LOVE it! When we go we stay at the campground next door.