My X was also very abusive, I also had a long standing restraining order. There are a couple of points here to address. First your older kids grew up in a home where it was OK to treat you as a second class citizen, and to hurt you physically & emotionally. They are carring this on by inviting you to their homes on a contingency basis - Only if you do not bring your husband (who would not tollerate seeing you mistreated)Only without your kids & step kids... (which would only be acceptable if you do not make these kids behave & treat others with respect, including other kids). Your married children are trying to control you, and it needs to stop. This behavior is as much your fault as it is your X-Husbands and theirs... because of the years you allowed them to see you treated baddly. But now that they are grown you need to make them aware of the difference between healthy & unhealthy family behavior, or your grandkids will grow up to be the same and all of the future generations are at risk of perpetuating family violence.
Second - YOU have a choice to participate or to have separate family events and (as a mature, non-manipulative adult) to coordinate dates & times with your children so that events with you and your X do not conflict with each other. Afterall, the point is to spend time with your married kids and their spouses & children, not to be "right". Parenting (even of adult children) is an act of loving someone else & giving of yourself. You cannot give expecting anything in return as a parent... you have to give your love away with both hands open & expect nothing in return.
The third thing is the hardest... you need to find a way to let go of the fear and anger you have against your X - YES, I know he deserves it, yes I know he is scarry... but I also know that you are the one hurting- not him. It will continue to eat you up inside like acid, and influeneces the decisions you make & how you feel about yourself. As long as you are afraid of him to the point of being physically ill he is winning and you are still his victim. Go to therapy and learn how to feel stronger... You're worth it!
My X & I learned how to get along in social situations - His 2nd wife & I became quite friendly, and eventually it was not as big a chore to see him at family gatherings.