Physical discipline is such a hot topic that I don't think I'd administer it on any child that wasn't mine. But there are other things I'd do.
The proper answer to "I'm allowed to act like this in my house" is, "Right now you are at MY house, and here you do NOT act like this." Of course, any time ANY child says, "I'm allowed to at my house," you know that it may or may not be true and that he/she MAY be testing you.
(He may be annoyed at having to play with babies when he's a big six-year-old - in which case you may want to give him some activity he can enjoy by himself for a little while. This is on the good days. I'm not excusing his rudeness.)
If my niece started hitting me and screaming, I would hand her to her mama, if she were there. If her mama were elsewhere and I was taking care of the niece, I would discipline her and then tell her mama the facts of the matter. "I told Susie to stop pulling the kitty's tail and she started screaming and hitting me, so she had a time out in the bedroom. Twice."
When I am taking care of my grandchildren (or any other children), I am responsible for how they behave and how everyone gets along. I know what disciplinary action is taken at their homes, and I try to be as consistent with that as possible, but I need to keep everyone safe and everything going smoothly.
Tell your sister that you (not she - diplomacy!) are having trouble with their behavior, and that you're going to have to correct them from now on whenever they're at your house. Yes, it may cause an argument. She may be offended at being called, by implication, a bad mother. But you may have to do it anyhow, if you are going to avoid mayhem (mayhem-avoidance is good).
When your children are badly treated at her house, leave with them right away without saying much. Just say goodbye and go. One day she may wonder why.
If your sister won't agree with you, she'll need someone else to look after her children, and she'll have to be with them when they are at your house. (That way you can put them on their mama's lap when they misbehave.) Stick to your guns, no matter what other relatives may say.
Here's something to think about. What can you do to make your home a really fun place for children to be? I'm not saying to go out and spend money on things, but if your house can be a place your niece and nephew look forward to visiting, you'll have more influence over their behavior. It's the old "spoonful of sugar" routine.