I'm not in the same situation but a similar one. My daughter's father and I have never lived together and are not together as a couple but we have a friendly relationship and do a great (in my opinion) job of raising our daughter as co-parents. When I take her to his house on Sunday we usually meet at a restaurant and all have lunch together. When he brings her back to me on Tuesdays he meets me at my job (at a hotel) and they swim together and I hang out and watch them. Occasionally he'll bring her to my house instead and we'll all have dinner together. He's in the National Guard and when he has his weekends away I take his dog and she stays with my daughter and I. We went to church and dinner together for Easter. My daughter sees two parents who care about each other and LOVE her beyond anything else. She's only 2.5 but knows that we communicate with each other in regards to her daily. The rules are the same at both houses. When she's with me we call Daddy every night to say goodnight and same when she's with him. During those phone calls her entire day (from good to naughty behavior) is talked about. I don't know if this is going to be confusing for her...I don't think so because it's her only reality. I think it's better for her by far to have two happy parents apart but who can have fun together then two miserable parents together.
Go with your gut and do what YOU and your husband think is best for you and your family. Don't worry about what others might think about your situation.