Its all about priorities, choices and then excuses. Most gyms have heavily discounted babysitting rooms for a reason. There are tons of moms that join gyms to get into shape. The problem is they don't follow through with the appropriate level of intensity in the gym and discipline in their diet. Same goes for dads. When my family goes to the beach we can't believe what people look like. The typical person is so out of shape they don't really have a human shape anymore. Its just normal to completely let yourself go and even eat yourself into diabetes around here.
Now I'm definitely not holier than thou about this shit. I was obese before I started bodybuilding and I enjoyed it. BBQ was just too damn good not to eat. I got a heads up from my doctor in my early thirties that I was headed in a bad direction and I hit the gym and never really looked back. I love to lift and enjoy the discipline of dieting, its definitely not everyone's cup of tea though.
Sure there's genetic and hormonal components to everyone's level of fitness. Those contribute to the level of difficulty, not impossibility.