Okay, some good responses, but nobody is telling you why this happens. First of all, it's totally normal, there MAY not even be anything wrong with you, especially since you said hats help. Yes, they're (hats) inconvenient sometimes, but worth it if you avoid the situation in the first place. I am an audiologist and I see this often in my practice. Basically, behind your eardrum there is fluid. If wind, especially when it's cold rushes in there, it's going to cause some discomfort. Also, when that fluid is either heated or cooled, this leades to the dizziness. Your vestibular (balance organ) system is directly linked to your ears. Hot or cold air (or water) can affect the fluid in the middle ear which in turn affects the balance organ. It actually "sends it off balance". In simple terms, it makes your body think it's moving in a certain direction when it's not. If you don't cover up your ears, you're going to cool down the fluid=DIZZINESS! And sometimes nausea and pain. If you're concerned, see an ENT. They will most likely refer you to an audiologist for vestibular testing (called an ENG or VNG).
Hope this helps! Feel free to respond if you have any more questions!
Small earplugs may even help as well, they make them so that you can still hear voices, but block the wind from getting in as well!