I don't think he can dictate what hours he's seeing the kids. I also suggest you go back to court and ask that the order be changed to his pattern of so many hours once/week. Or is it every other week? He's set a precedent. You choose which day, saying this day is best for the kids. Consult an attorney about this, first.
I would not insist that he take them for the full time because if he's not interested in having them the full time he will not be treating them well. I suggest that forcing him to take them the whole weekend or not at all could back fire to the detriment of the kids. But again talk with an attorney first.
I am wondering what his game plan is. Is he wanting you to go back to court at which time he'll say something like he works weekends and wants to have them during the week or even that you're unco-operative and he wants full custody, or something else? He's already shown that he knows how to make you look like the bad guy. I suggest there is more to what he's doing than to just make it inconvenient for you.
I might start by having a lawyer send him a letter reminding him of the court order and offering him a regular several hour visit to fit the pattern he's established. You state what day you want that to happen on. This will give you and the children continuity and it will establish that you are willing to compromise. It will also put you back in a position of power equal to his.