January 20, 2010 at 3:14 pm by Capitol Avenue Club under Atlanta Braves
On Monday, Dave Cameron of Fangraphs, a gifted author and great baseball mind, writes:
Over the weekend, Chuck Brownson did a post over at THT Live, showing the premium that a few teams have been willing to pay for closers this winter, even as the market for nearly everyone else collapsed. Despite a pullback on spending, some clubs still value late inning relievers enough to pay a premium to acquire guys who have had success in that role.
WAR, as you probably know, doesn’t think much of relief pitchers. The very best relievers in the game are generally worth +2 to +2.5 wins over a full season, or about the same as an average everyday player. This has caused quite a few people to state that WAR doesn’t work for relievers, because the results of the metric don’t match what they believe to be true about relief pitcher value. I think it works just fine.
It depends on your definition of “works just fine” is and what you’re using it for. There are two important concepts to consider when valuing relievers. One is production and one is leverage. Relievers are undervalued in both aspects by WAR (especially Fangraphs WAR).
First of all, the metric of choice for quantifying the production aspect is FIP–fielding independent pitching. Drawing from Fangraphs’ glossary: “The FIP formula is (HR*13+(BB+HBP-IBB)*3-K*2)/IP, plus a league-specific factor that scales FIP to match league average ERA for a given season and league.” It’s a fine metric, however, as I mentioned, it undervalues relievers. Relievers, presumably due to the fact that they rarely face a batter twice on the same night, have historically posted lower ERA’s than FIP’s. If that’s not a coincidence (I don’t believe it is), there is obviously something wrong with using FIP as your metric of choice.
Realistically, if FIP were given an additional role-specific factor (or if the league-specific factor accounted for pitching role), this wouldn’t be an issue. However, such is not the case, and presents a limitation (and, thus, a flaw) when valuing relievers.
While the quality of their work is very high, the quantity is low, which limits their total value. It’s nearly impossible to rack up huge win values while facing less than 300 batters per season. Yes, each of those batters faced are more critical to a win than a regular batter faced, but this is accounted for in WAR.
The average Leverage Index of a closer is about 1.8, meaning that each plate appearance is about 80 percent more important than an average PA. We give the closer credit for half of that, based on the principle of chaining.
If I understand this correctly, you average the reliever’s average leverage index and one to get their (we’ll call it) “chained leverage index”, calculate how much the pitcher’s production was worth, multiply it by said chained leverage index, and get, well…. what, exactly?
This fails to give us a picture perfect idea of what the reliever is or was actually worth. It depends on the question. If you’re asking, “how much did a reliever produce?”, a simple role-adjusted FIP calculation (converting it to runs/wins) will do just fine. However, as you’ve mentioned, relievers are usually worth more than their production to a team.
If you’re asking how much was a reliever worth (to his team, not in a context-neutral environment), something like Baseball Prospectus’ WXRL (plus some chaining accounting) is probably more useful than WAR. Notice the operative word, “was”, indicating only what has happened in the past and, thus, flawed in the predictive value.
If you’re asking how much a reliever will be worth, it’s probably best to take the production component and some how estimate how valuable that production will be to a team–accounting for both leverage and the context of wins. How you would account for leverage in the future I have no idea, but someone who is better than math at me could probably figure it out.
It’s important to remember that a reliever’s leverage index can change rather drastically from year to year, due to his team finding itself in many more or fewer late and close situations or due to the reliever changing teams or due to the reliever changing roles et cetera. It’s also important to remember there’s little certainty as to what a reliever’s leverage index will be. Teams handing out these contracts probably have a much better idea of what that LI will be than us outsiders do.
Anyway, WAR doesn’t work just fine for valuing relievers. It does nothing to account for context, the production is quantified with a bias metric which undervalues relievers, it doesn’t consider the fact that a good bullpen facilitates Pythagorean luck, and it leaves something to be desired in the leverage calculation department.
Production is a great basis, but without giving said production context, it’s impossible to know how valuable it actually is.
That a half dozen major league clubs are overvaluing proven relievers does not mean that WAR doesn’t work for them. It’s just an arbitrage opportunity for the rest of baseball.
I don’t so much mind the perceived arrogance he lets on, I’d be a hypocrite (seeing as much of my work seems to be perceived as arrogant, even if that’s the last thing I want to do) if I did. It’s the fact that he’s sort of encouraging ignorance–so as to say, “WAR is all you need, if a team makes a move that doesn’t agree with the Fangraphs groupthink, their GM is an idiot”. Note: I don’t mean to criticize the authors and publishers and creators of Fangraphs, the readers that take their work as gospel is the target of that line.
Adjusting for future context, in addition to other flaws, is something that WAR simply doesn’t (and probably can’t) do. Turning a blind eye to other factors and considering only WAR when valuing relievers is to omit some things that most decidedly influence how valuable a reliever is to a team.
Without even getting into the small sample size issues that relievers inherently encounter, relying exclusively on WAR when attempting to value a reliever invokes a flawed understanding how valuable a reliever is. At the very least, we should be looking at the context of the reliever’s future production, if not re-thinking FIP and the role-specific thing.