I think you have to look at several metrics. No single metric really cuts it. You don't necessarily need to use them all, but each one helps in its own way to clarify the picture. ERA tends to work for 500-plus inning samples. If you're going to use it for smaller samples, you're going to have to give it breathing room as it's affected by both defense and luck. WHIP is a nice cross-check for ERA. Its downside is that it doesn't account for extra base hits. It also needs breathing room. ERA+ is cool in that it covers ballpark factors. However, it carries along ERA's other flaws: it's affected by defense and luck until the sample is large enough. K/AB and BB/AB would be optimum for judging peripherals. Mainstream web sites don't give us that stat, though. You can substitute in K/9 and BB/9 as long as you realize that these need a bit of breathing room. K/9 and BB/9 don't account for extra outs caused by errors, nor do they account for ABs where the pitcher has given up hits. You can toss in K/BB ratio, but it should be examined alongside of K/AB & BB/AB or K/9 & BB/9. BA/OBP/SLG/OPS against the pitcher should be helpful. Besides the raw information given here, you can eyeball a BB/HBP rate and you can subtract BA from SLG to see a pitcher's extra base hit rate (isolated power against). FIP does a nice job at weeding out the effects of defense and luck. Some members have mentioned xERA. I don't know much about that stat, but it's reasonable to assume that it's similar in purpose. If you want to get more sophisticated, you can look at line drives, ground balls, infield flies, and outfield flies. A lot of sites only include flyball outs and groundball outs; it's much better to look at base hits and outs combined. You can also look at BABIP. For most pitchers, you can eyeball luck by comparing his BABIP to the team average. Some pitchers, especially groundball pitchers, may consistently carry a lower (better) than average BABIP, though. To judge luck with those guys, you'd want to compare their current sample to their larger sample.