I am so sorry for your strained relationship and your pain. Have you asked your sister what the prayer was exactly? It seems you may need more information from an adult. If it was a "God is great, God is good, and we thank Him for our food" sort of prayer, then that does seem like a simple, harmless exposure to her grandmother's faith. We pray before meals at our home and our guests respect that, as we are in our home and following our family customs. Just as when we visit other families who do not pray, we respect that. It may have been confusing to your daughter, but it is also an opportunity to learn that is they way that Grandma expresses her faith-- just like your next door neighbor has a Christmas tree and the woman down the street wears special clothing on her head.
If your mother is mentally ill, perhaps she needs some medical help? If she is mentally ill, then she may not be capable of having the type of relationship with you that you desire. It sounds like you are wanting to raise your daughter with a strong sense of compassion, love, and tolerance. May I presume that respect for elders is part of that list? If so, then showing your mom some grace and gently explaining to your daughter that all families are different and Grandma does things differently at her house than we do at ours is one way to handle it. Letting her see that you disagree with your mom but that you still treat her respectfully and compassionately is a great lesson.
If she is a Christian, then I am sure it is very concerning to her that your beliefs are different than hers. You have a very open mind set when it comes to faith, but she does not. To her it may not be just a matter of customs and rituals, but a matter of whether or not you will spend eternity in Heaven or Hell. I am not saying this to judge you at all. I am just saying that as a mother yourself, and even though you completely disagree with her religious beliefs, can you put yourself in her shoes for just a moment and see that her actions may come from love and fear for you and your daugther, rather than a disagreement equal to schedules or diet? Now if this is her true concern, there are good ways and bad ways to go about sharing that concern. And, it sounds like your growing up years were filled with hypocricy which would understandably harden you toward her example and way of thinking. And, when the time comes to really expose your daughter to different religions, you would probably want to choose a Christian example for her that truly lives out her faith without hypocrisy rather than your mother. Just as I am sure you would choose in the other faith examples, too. (A loving Muslim neighbor as opposed to a terrorist on the news, for example.)
It sounds like you love your daughter very much and want what is best for her. I would caution you that she is watching and learning from you-- including the way you relate to your own mother. I am hoping that in light of Mother's Day, you can find a way to calmly talk with your mom about your concerns and set some healthy boundaries that still allow you to have a positive relationship with your mom in some capacity-- even a formal, somewhat distant yet respectful relationship seems like a better option than a close/tumultuous one.
My heart hurts for you all,