I believe this is very common.
I would give dad and son space to be together. Get to know each other. Is there any way you and your daughter could leave the home take a walk, go to the park. I would do this at a prime time baby is happy so that Dad can get his confidence back. My hubby was the comfort for my oldest when we were having a "bad" day. He could come in and take her and my nerves would be so soothed and so would hers. My second didn't know him as well (He was in Iraq) and I know it bothered him.
Give dad and son space and Let dad do the things that baby really likes. Does he like a bath? Let dad do the fun stuff sometimes. I know it sucks, that we do all the "dirty" work but, dads are away all day and let him have some fun time.
Also on the weekend let Dad do more and have more time.