A lactation consultant once told me that in the Philippines, women in sweatshops pump and leave milk in containers by their workstations, at high temperatures with no refrigerations, and take it home at the end of the day. They do not report problems with it. That's pretty extreme, and we wouldn't do it in this country where refrigeration is readily available, but I think it shows you that breast milk is pretty resilient and will keep for longer than you are likely to want to keep it, with or without refrigeration.
Just as an aside, most people are way more conservative than necessary when it comes to dairy products, too. I've seen people throw away milk or yogurt that has reached its expiration date, whether or not it appears at all sour or spoiled. Completely unncessary. If the seal is not broken yogurt will keep past its expiration date, even weeks without refrigeration. Milk will keep longer, too, though not that long. Your sense of smell is valuable - you can tell if it's turned or not. There are types of bacteria that do not cause odors in food but those aren't the ones we're worried about here.