I do not like doctors. That having been said, you better take him to the doctor. If it is Asthma or Pneumonia, he is not effectively oxygenating his blood when he's breathing and that's bad.
My 3 1/2 y.o. has been diagnosed with Pneumonia twice recently then the final decision was Asthma. They didn't detect the Pneumonia by listening to his lungs but on the advise of my cousin, who saw him and insisted that I take him to the doctor, I insisted on a chest Xray and VOILA! Pneumonia.
It sounds like this has gone on a little too long to ignore. To be on the safe side, have him checked by the doctor, soon.