Honestly- the only thing you can do is keep leaving them, but in short increments of time. Several hours away is tough when it makes you anxious, but you could leave them for a 1/2 hour with a friend to run a quick errand. When that stops feeling overwhelming, bump it to an hour and then 2 hours...
Anxiety in general has a lot to do with needing to feel in control. You can't "control" what happens to your children when they are not with you. You can't control what they do, what they eat, watch they watch, etc. It is really tough to step back from it, but keep in mind that the ONLY way our children learn how to "live and learn" is through failure. Sounds crazy, but it's true. They are too young to understand "warnings and logic" and need to experience things to learn from them.
Eating too much candy makes you feel sick. Don't like they way it makes you feel? Don't do it again! Running around like a crazy person at mommy's friends' house may mean running into a wall and hitting your head. Don't like they way it feels? Don't do it again.
We spend so much time "warning" our kids, but in all reality until they are mature enough to "predict" the natural consequences for their behavior they just have to learn from their own mistakes. As parents we do not want our children to have any "accidents or bumps" so we control their environment. When they walk out the door (or you do), we lose that control and it is scary.