I had orthodontia in middle school and high school. After my braces were removed I had a permanent retainer on the bottom until I was 21-22 years old. Fast forward nearly 20 years and... my teeth have definitely moved. Now, I didn't start noticing it until about 5 years ago, but now it's really noticeable. I talked to my dentist about it when I started noticing it, and he said that just happens, and the only way to truly keep the teeth in place permanently is to have a permanent retainer.
Now the truth is that my teeth look fine. They were scary crooked prior to my orthodontia, and they are nearly perfect now in comparison. The only reason they don't look 'straight' to me is that I know what they looked like before. I'm not about to get braces again, so I've just resigned myself to living with it.
I was SUPPOSED to wear a removeable upper retainer at night only for... well, they said forever, really, but I didn't keep that up much past college. My upper teeth have moved some, but not nearly as much as lower.
Good luck deciding what to do.