San Diego is not normally considered a large market team. In terms of media market San Diego is 27th in the U.S. and Milwaukee is 37th.
I had a conversation with Sandy Alderson in 2005 about the Padres when he was the President of the team (I was a Marine stationed at Camp Pendleton at the time and Sandy is a former Marine and was the guest speaker at our annual USMC Birthday Ball). He lamented that San Diego was actually a small market team because on their southern border is Mexico, which does not count for advertising rates in the U.S., and they had two teams 1.5 to 2 hours to the north. Basically, they were boxed in geographically which dramatically reduced their reach.
Also keep in mind that while Milwaukee is the 37th largest media market, the Brewers reach is well beyond Milwaukee since they are the only team in Wisconsin. Thus, they have viewers/fans in Madison, Fox River Valley, Eau Claire, etc.--and that expands their reach by a fair margin. (I do recognize there are a number of fans in Kenosha County who follow the Cubs and fans in NW Wisconsin who follow the Twins).
I suspect the Padres are borrowing money to fund their current team. Which certainly is justifiable given the team's value.