I have no experience with nannies, but in reading your post you need to consider how you would handle it if ANY of the kids were sick (and how your employer would handle it). That would need to be addressed right up front. Would you be willing to bring your infant into their home if they had sick kids? Would they be OK with you bringing your sick child in to their home?
I watched my sisters daughter for a couple of years when she was younger, and that was the one thing that threw a wrench into things. Obviously, because we were sisters, we'd openly debate it the night before or that morning, and discuss symptoms/seriousness. Sometimes it was a mild fever and we'd simply try to keep the kids apart in the same house (not easily done) and I'd try to have everyone wash more often, keep their hands to themselves, etc. Other times, it was much more serious... high fever, throwing up, etc. You need to discuss all of these options with your prospective employer and figure out a system so there aren't any heated discussions when someone gets sick. There also needs to be an alternate system in place when something serious comes up.
Good luck!