I'm 51, my son is grown, and what I've seen is mom's who have little ones in their mid-late 40s... they look so worn out (even those who were considered healthy when they had their baby). The worn out look comes no doubt from the act of carrying & delivering but also the daily taking care of the baby/young child.
You will be about my age when your baby is 3. I too am in great physical health and I cannot imagine running around after a toddler and it not wearing me down on at least some levels. You also have to consider the fact... you will be going through menopause when your baby is going through toddler-hood... hmmmm, I'm thinking you probably won't be feeling quite up to dealing with both.
Our culture typically doesn't view menopause as a good change, as such your age now is one where you might be thinking about the larger community and "Wise Woman" - how you can serve the women and girls in your community... it's a period in life where we tend to look beyond ourselves and reach out to those near us.
Another consideration is... the financial crisis we're in. Though it may not last forever, the fact is... what is going on now is not going to be going away any time soon. You and your husband may have enough financially at the moment but that's no guarantee that your finances will be in tact 1 month or 36 months from now.
The last thing you need, when you probably would be getting your finances in line for retirement, is for there to be something unfavorable to upset the balance in your retirement fund, like: our country having a financial crisis and your savings loosing 40% or more of its worth, and having children (I can only imagine what it costs these days), and perhaps loosing a job, or medical expenses... all of these things have severe impacts on our finances.
I'm not trying to turn this into a financial decision but I think it's a good idea to a least consider where you would be financially if one of the expensive things above did occur in your life... sit down with pencil and paper and calculate where you would be financially if one of them happened and what life would be like trying to deal with it.
For what it's worth, I don't know any older woman who has had a child late (my circle of women friends are wealthier, more educated, and seemingly healthy) in life and not have a child who is not handicapped in some form. From what I've seen, after the age of 41 the risks are quite high.
Even women who do live healthier lifestyles still have extremely high levels of toxins in their bodies (their "Body Burden"), toxins that we've gotten from literally everything we've ever done, from the shampoo we use to our body lotion, to household cleaners to that apple we ate, from drinking bottled or tap water to taking a walk down the street... all of those activities have exposed you to tons of toxic chemicals. Our world is so polluted that we cannot get away from it, it's our lifetime exposure to chemicals that are the largest factor in why our chances of having a normal healthy baby go down significantly. Simply looking at the autism rates these days, which is appalling, in women much younger is an unfortunate sign of the times... we're in the "Gender-bender" days.
Perhaps... what you and your lovely husband might consider is getting involved in something together that is "nurturing" to others. Like Habitat for Humanity, or any other number of excellent causes... though it is not the same as having a baby, it still can bring the satisfaction and connection we feel when we reach out and touch the lives of others in significant ways.
All I'm trying to say is, it might be wise to think of all of the possibilities and look into some of them if you have gray areas, lay them all out on the table and spend a lot of time discussing it with your husband. The choice(s) you make now will affect the rest of your life.
Hope this helps. My best wishes!