I feel for you. I had a comparable problem with an ex and a son who was 17 at the time we had a visitation order in place. Because my son was substantially older than your children, I just had him work out any of his schedule wrinkles with his dad. If he had a track meet or a play performance during a visitation period, he worked it out with his dad. I didn't get involved. As a result, the dad couldn't come back and say that I was trying to keep him from having visitation. My ex unofficially "ended" weekend visitation, when my son began to have 6:30 a.m. track meets every Saturday while he was running cross-country in high school. This disrupted my ex's weekend sleeping habits sufficiently that visitation became much less of an issue. And, since my son was communicating with him, not me, it wasn't "my" fault. Because your children are much younger, you have a much more difficult time with this than I had. You have to take a much more active part in their scheduling. The way Becki has outlined is a very, very good way to handle things.
To be honest with you, the legal system is stacked against you. It assumes two reasonable parents who are interested in the welfare of the children, and is moving more and more to a strictly enforced shared parenting model. The only way you are legally going to be able to help your children avoid this disruption to their lives is to obtain a judicial finding that your ex is a sufficiently unfit parent with the result that he either gets supervised/limited visitation rights or no visitation. The standard for this sort of finding is very high, and calls for some expensive studies and litigation. Mere laziness, slovenliness, and self-centeredness does not usually work to meet this standard. You might want to check with your attorney, though, to see if you have the basis for a modification of the visitation part of your divorce decree. If your ex's lack of responsibility is endangering your children's welfare, you might be able to make the argument for limited visitation.
Best of luck. You need it. Pray hard, too. And keep on being such a good parent to all of your children.