I agree very strongly w/ Donna--you need to talk to a La Leche League Leader very, very badly!
I hate to argue w/ your doctor, but I find it very hard to believe that an allergy is involved based on just those "symptoms". Believe it or not, most health care providers know virtually NOTHING about the inticacies of breastfeeding. Usually, when a baby has an allergy/intollerence, it's very obvious--they have severe eczema, diarrhea, mucus in stool, SEVERE colic, severe spitting up/vomiting, etc....
Babies come in all shapes & sizes & some are just very, very tiny & grow very, very slowly. Both of my kids were like this! My daughter was only 16 lbs at a YEAR & my son was about the same (usually boys are much heftier!). Some kids are much more efficient than others are too & will get right down to business & nurse like champs & finish up very quickly.--It might seem as though they're not nursing for very long, but they're just efficient at it.
The BEST way (really, the ONLY way) to tell how much a child is taking in is to monitor diaper output. As long as they're having 6+ wet diapers a day (at least 1 oz of urine), that's plenty. Keep in mind too that disposable diapers absorb & hide liquid very well, so you can put a tissue in the diaper when in doubt. Also, remember that breastfed babies will often go as long as 7-10 days w/o pooping since there's so little waste in breastmilk.
I think that everything is likely just fine & that you just have one of those babies who's a slower gainer than the "average" baby. After all, not every baby can be on the 50th percentile! There wouldn't BE a 50% percentile if there weren't kids below the 3rd & above the 95th!--Every baby is different. As long as yours is meeting developmental goals more/less on time & is healthy & happy, everything is likely fine.
Hang in there & keep up the good work!!!