I feel for you! I have been there and done that! A few times. As hard as it is to let someone else do your and your ex's job, it is going to happen at some point. I thought it was only fair to meet my ex's new girlfriend first so we all met for pizza and talked while my daughter played on the play structure.
I think there should definitely be an agreed upon set of time and some rules for the both of you going forward. Like you can first ask him to put the shoe on the other foot and then ask him what he thinks a reasonable amount of time is before a significant other starts taking over some of the parents responsiblilities.
I know the situation is a bit different when it comes to daycare and babysitting. You both have to agree on the sitter if the child is going to be with that person for more than 3 hours, otherwise, you have first rights to watch your child if the other parent plans on being away for 3 or more hours.
Good luck! It does get easier with time, believe me.