I don't think that you can assume that your milk will dry up. You are doing great with pumping 4 oz at a time. Did your Dr check baby's tongue? I forget what it is called, but the tongue needs to be free enough to pull in the nipple and some tounges are too closely attached to the lower palate.
Our son was born early and had trouble nursing in general. Not sure if we had a latch problem per se, but our lactation consultant taught us to encourage his sucking by positioning a little feeding tube by my nipple (did this with medical tape). The tube was attached to a syringe that we filled with my milk. We would push the milk down to encourage him to keep eating. It was a lot of work but after 1 or 2 weeks, he got it! I look back now and wonder how my hubby and I were so patient during this time, it was hard!
I have had so many challenges with breastfeeding as the months have continued (not that you will, i don;t see my other problems as related to teaching him to nurse properly), I know how upset you can get! You can tell you are doing your best, so be encouraged.
BTW, my baby was in the hospital for about two weeks before we went home and did all the work with the tube feeding. I both nursed him and pumped in the hospital, and my milk did not dry up.