This is making more sense now. I get more what you are dealing with.
Your immediately family is your priority. This is where I think therapy could help you - I know that's not really what you want to hear, but it's so true. Your immediately family, with your two kids who need a lot of care, are almost in crisis mode on a regular basis. No one would expect you to go to crazy amounts of trouble to attend things for extended family knowing that you have so much on your plate.
I don't know why your mom and brother don't get your challenges. That actually must be very upsetting. I'm not well, and we changed how we did everything. Our family comes first. We say no all the time. It upset a few people, but quite frankly, they were demanding in the first place. Your family comes first. You shouldn't feel guilt about this, conflicted or otherwise. And if you fear that there will be some fall out - that's on them, not you. There's something wrong if they will be so offended. I missed my niece's wedding. They respected our decision. So - there might be some bigger issue to do with your own family (siblings and mom) that's deep routed if you fear the consequences of doing what's best for your family.
Myself? I wouldn't go into specifics. I would say sorry we can't make it. I wouldn't involve your mother. I wouldn't tell your brother why if he's not understanding or supportive. There's boundaries that really need to apply here. You don't owe them an explanation. A polite I'm sorry, we won't be able to make it - is all you need.
You sound like a very loving giving person. That doesn't mean you have to please these people. If they are going to hold this against you, then what do you care anyway?
Sounds like you and your husband should plan a nice getaway weekend some other time. You deserve it.