He wants just you? Just you and your husband? You, husband plus any kids--?
I hope that in a few years, when he has had to live with things like a mortgage and kids, he will realize how utterly self-centered and silly this "invitation" sounds. He may be a great guy but he's a clueless one about other people who aren't, like him, able to do and go where they want, when they want. The "No excuses!" thing -- boy, I hope that's meant with humor, because if it's not, he really is letting pre-wedding giddiness melt his good sense.
I would not say "we can't come unless you pick up some of the cost" as someone suggested. If he responds "Sure! Yes!" would you really then be able to go? With kids' schedules, work schedules, life? I've been to Japan twice and it takes really a full day to get there, with airport time, flights, changes, etc., and the jet lag is huge, so unless you are going for at the very least a full week it's not worth the toll it would take. Maybe just one of you parents could make it if you really want to and have the funds (Japan is very expensive), but taking kids? Forget it, unless they are old enough to handle the travel and the jet lag and the food that's new to them. So do not go the "If you pay some, we'll come" route unless you truly mean it AND want to feel indebted to him, which is no basis for friendship.
I would tell him how thrilled you are that he's getting married; how exciting the wedding sounds; and that you cannot come but can't wait to throw him and his spouse a great dinner (or a whole party, if you are really close to this guy) back home. IF you want that expense, by the way.
If he gets snippy or worse, say to him what you said so well here: "I know you're coming to the States for X's wedding, and that's fine, but I think that you may not realize what a big thing you're asking of us." If he is really a good friend he will wake up and realize what he's doing. If he persists in pushing you to come, or is rude about it, or even cools off communications, you will know that he is out of touch with your life right now and unable to get past himself to see how others live.
You say that you feel awful right now but honestly, he's the one who should feel awful for pressuring people to come to a destination wedding that goes way beyond the usual "Come to Bermuda" ones we see talked about here.