I am a single parent too. My son does the same thing. It happens less now than it did when he was younger. He is almost 7. He still says "mom I just love you" a lot and when he gets nervous in a situation that he is not used to he says it more often. I think it is just a re-assurance for him. I think the best we can do is re-assure them that we are there for them that we are not going anywhere and help them feel more confident.
My ex has not seen our son since July and that was for only 2 weeks. We live in another state. These nervous times seem to increase before and then after he sees his dad. I think its the inconsistency in the visits that causes some of the nervousness. I think even though he does not even realize it, that he is afraid that I will go away too.
There are times I feel bad that we could not stay together for our son. But I think things would have been worse for him had we tried. Each day he seems more and more confident. More than I ever was. Just continue to let him know you love him too, maybe an extra hug to give that sense of security. It will be ok. Good luck
Re-assurance is key. From both parents if they can get it but since we are the once that are in their lives more on a day to day basis we need to provide it to them...Time and patience.