First of all, you are not terrible for wanting a better life for yourself and your two children who live at home with you.
There is nothing at all wrong with wanting to better your situation, especially for the sake of your children.
It also sounds like your adult child or children have possibly been using you as a free babysitting service, and they resent the fact that you will no longer be at their beck-and-call for whatever they feel they need from you.
You know, life is simply too short to always worry about how other people will react to your choices, or what they will think of you if you move. If you believe that you only have one life on this earth, then please spend that life doing things that make YOU happy. If we continually worry about everyone else, well, that leaves nothing for us, does it?
Firmly tell them that you ARE moving, and that you would be more than happy to host them when or if they decide to visit, but for them to give you some advance notice of their arrival, so you can plan for it.
Maybe once they see that you are truly serious, they will relax about the move. It could be that they are simply worried about you and your younger children, since you will be moving away from the security of familiar surroundings in which you have lived for 40 years.
After their first visit with you, when they see that you have indeed made a good new life for yourself and your children, they will likely embrace the move as beneficial. Plus, they have somewhere to go on vacation!
Seriously, assure them that the three of you will be just fine. If you show them that you already have things lined up in the city to which you will be moving, they will feel better about it. I mean things like a part-time job, a place to live, possibly friends you may have already.
Also, look online to see if there are any community boards for the area you are moving to, and sign up so that you may ask the locals things like where is the best place to shop, what is the safest area of town, the name of a good doctor for yourself and your son, and a pediatrician for your daughter, etc. Make a list of their recommendations, and you will already be prepared with that before your actual move.
Whatever happens, good luck to you.