I'm so sorry about all this. I am on the flipside. When we were first married, I got too close to someone and nearly blew my marriage! My dh was kind enough to see that I meant that nothing had happened and forgave me. That was 12 years ago and we have a strong marriage. His forgiveness has allowed me to be strong for him as he battles his pornography addiction, which I believe to be very destructive likewise.
My sincere advice is this, forgive him from the bottom of your heart. Be strong & kind, don't put him in the doghouse. Your family is stronger than some flirtations. Show him what he was missing when he was being so foolish. It sounds unreasonable, just love the father of your babies. Give him the chance to prove you're the woman for him, he will probably surprise you.