I continued to hold my almost 6 y/o - who just recently made it to 40lbs (YAY!!) up until my 7th month when it started getting hard for me. I'd say if you start to feel pain, then that is the time to stop. The little bit my daughter asks to be held, I didn't want to deny her that comfort either and I offered it for as long as I could.
Now she will sit on my side rather than on my lap because my belly has been so big since the 6th-7th month. It's harder with a much younger child, you cannot really explain why things need to change and the child may see it as rejection. Even with us always talking about it, my daughter would still sometimes take it as rejection.