Well maybe this was true in the days of George Davis and the Cincinnati Red Stockings, but nowadays, most likely when this phrase was coined, baseball is played by virtually all young boys in the United and many more in Japan and Latin America. Probably at least 100,000,000 boys play baseball every year and about ~5000 men play it professionally. So it does certainly make sense, most men are not professional baseball players.