You have a good combination of opinions.
I can think of my husband. His mother was married to a man who was very kind to them. He taught them karate, to play the piano, was a chiropractor and taught them about that.
He and his mother didn't work out - not many relationships have worked out with her, actually she's still single ;) After 3 marriages and a few other boyfriends.
Anyway, he left and never looked back. Their father came once a year or so and he still never did understand why, after they had such a loving relationship and was like a father to them, he never attempted to contact them. In this case, I KNOW it is his crazy mom.
However, his cousin was also partially raised by a step-dad and he was also his best man and they are very close.
I don't think children should suffer more than necessary. They love and trust him and why take that away, it will only hurt the kids in the long run