I had a destination wedding, and so did my brother.
For us: we got married in the Grand Canyon and had all our fun and extra events in Vegas. We chose a very small group of people to go and we had such an amazing deal (cheaper than a "normal" wedding in our hometown), then me, my husband, and my dad paid for everyone's "everything". (I bought mom's outfit, we paid for all meals and incidentals, shows, fun money...my dad paid for the ceremony itself, photography, and everyone's flights).
For my brother: my sister in law was married before, and had a son, and she and my brother wanted to pay for their own wedding rather than ask her father for a 2nd wedding. They wanted to get married at the Iguazu Falls in Argentina. My family is more of the mind that if we invite you to something, we are responsible for the bill....they didn't have the money to bring anyone else over, and noone really would have wanted to cough up the money at that time either (not that anyone was really asked), so the trip was a combined wedding and honeymoon. We all pitched in and threw a big party/reception for them and her parents kept their son. They left for the airport and went to Argentina to get married, stayed a week and had a blast, came home and got their son back.
I feel like everyone is welcome to get married wherever they like. They can invite you to say you are welcome, but if they aren't willing/able to bring you along on their bill, then they must be gracious and understanding that the guests may not be able to come. If I had it to do over again, even though we had a lot of fun, I think it would have been romantic to have been just the 2 of us in our destination wedding, and would possibly do it differently if I could go back in time. On the other hand, my brother and sister in law had a beautiful, meaningful, private ceremony and a fantastic and romantic honeymoon, but also had ALL their family and friends together to celebrate their reception or going away party, whatever you want to call it. It was a great party, fun for all, and that's what they got to share with us. They have NO regrets.
I'd just straight up say "I love you and wish you the best, but for us, that is just simply too big a price tag. Perhaps we could have a party here for all your friends and family to join in your happiness, but it's just not feasible for everyone to go on such a fantastic trip as you are taking". I wouldn't feel bad about it at all.