I would get him a front pack, and you a front pack or sling. Get two "cheaper" brands and then fit each one to your own body! Check at consignment stores or nice thrift stores too!
If you both "share" a carrier you might get frustrated constantly having to adjust them back and forth, and I found until I got my husband "his own" carrier, that usually meant that I was ALWAYS the one "saddled up" for the baby. My sling was already adjusted to fit me, and neither of us wanted to deal with straps and buckles. My sling is a "hotsling" and I LOVE it... but they are out of business... sigh. "karmababy" and the "munchkin jellybean" are essentially the same though, and I really recommend this type of sling for women, ESPECIALLY if you are nursing. My husbands carrier is an older infantino front pack that is not unlike the "swift" carried at target. If you MUST share a carrier, go ahead and splurge for a baby bjorn or similar one.
Keep in mind that you and your husband are build differently in more than just "height"- I liked the sling because my baby could rest on my hip when she was sitting up, and carrying a bigger baby out in front just KILLED my back if I did it for too long. My husband on the other hand, has no hips LOL! he was much happier with a front pack. Of course once your baby is walking and confident with it, the carriers will probably go into storage and you'll use a stroller instead on long days out.
I hope this helps!
-M.