I'd say start him now! Make it fun and easy for him. Start out with the driving range and or just putting together. I think a family or couples lesson would also be great for you and your wife. Just to give you a few tips and get you going in the right direction. I'm very much a casual golfer but my oldest son who is now 8 is really in to it. He has been since he started walking, with his plastic set. He was doing some pretty crazy stuff at a young age, which at the time I thought was somewhat normal, but now seeing what my other two boys are doing as they grow up, it is nowhere near the oldest. So he just golfs with us now. I also have a 5 year old and a 2 year old and they both come golf with us too. The 5 year old, we set up where he can hit his tee shot close to the green and then he can chip and putt from there. When we get close to the green, we let the two year old hit his ball on the green and then putt (with is plastic clubs). It's a lot of fun. Let him help drive the cart, bring snacks, he will enjoy all the other stuff too. If he wants to take a hole off, or just putt, let him do that. Again, the more fun you have out there the more they will get in to it. A cool thing my grandpa would always suggest to people we'd golf with that weren't as experienced was to always tee up your shots, even off the fairway. It makes it easier to hit and more fun for the golfer. Go for it, I don't think you'll regret picking up the sport!