Loading ...« Arbitration Predictions | Main | Projected 2010 Depth Chart Update 1 »
Jumping Into the Reliever Valuation Discussion
By Capitol Avenue Club | January 20, 2010
WAR and Relievers | FanGraphs Baseball
On Monday, Dave Cameron of Fangraphs, a gifted author and great baseball mind, writes:
Over the weekend, Chuck Brownson did a post over at THT Live, showing the premium that a few teams have been willing to pay for closers this winter, even as the market for nearly everyone else collapsed. Despite a pullback on spending, some clubs still value late inning relievers enough to pay a premium to acquire guys who have had success in that role.
WAR, as you probably know, doesn’t think much of relief pitchers. The very best relievers in the game are generally worth +2 to +2.5 wins over a full season, or about the same as an average everyday player. This has caused quite a few people to state that WAR doesn’t work for relievers, because the results of the metric don’t match what they believe to be true about relief pitcher value. I think it works just fine.
It depends on your definition of “works just fine” is and what you’re using it for. There are two important concepts to consider when valuing relievers. One is production and one is leverage. Relievers are undervalued in both aspects by WAR (especially Fangraphs WAR).
First of all, the metric of choice for quantifying the production aspect is FIP–fielding independent pitching. Drawing from Fangraphs’ glossary: “The FIP formula is (HR*13+(BB+HBP-IBB)*3-K*2)/IP, plus a league-specific factor that scales FIP to match league average ERA for a given season and league.” It’s a fine metric, however, as I mentioned, it undervalues relievers. Relievers, presumably due to the fact that they rarely face a batter twice on the same night, have historically posted lower ERA’s than FIP’s. If that’s not a coincidence (I don’t believe it is), there is obviously something wrong with using FIP as your metric of choice.
Realistically, if FIP were given an additional role-specific factor (or if the league-specific factor accounted for pitching role), this wouldn’t be an issue. However, such is not the case, and presents a limitation (and, thus, a flaw) when valuing relievers.
Cameron continues:
While the quality of their work is very high, the quantity is low, which limits their total value. It’s nearly impossible to rack up huge win values while facing less than 300 batters per season. Yes, each of those batters faced are more critical to a win than a regular batter faced, but this is accounted for in WAR.
The average Leverage Index of a closer is about 1.8, meaning that each plate appearance is about 80 percent more important than an average PA. We give the closer credit for half of that, based on the principle of chaining.
If I understand this correctly, you average the reliever’s average leverage index and one to get their (we’ll call it) “chained leverage index”, calculate how much the pitcher’s production was worth, multiply it by said chained leverage index, and get, well…. what, exactly?
This fails to give us a picture perfect idea of what the reliever is or was actually worth. It depends on the question. If you’re asking, “how much did a reliever produce?”, a simple role-adjusted FIP calculation (converting it to runs/wins) will do just fine. However, as you’ve mentioned, relievers are usually worth more than their production to a team.
If you’re asking how much was a reliever worth (to his team, not in a context-neutral environment), something like Baseball Prospectus’ WXRL (plus some chaining accounting) is probably more useful than WAR. Notice the operative word, “was”, indicating only what has happened in the past and, thus, flawed in the predictive value.
If you’re asking how much a reliever will be worth, it’s probably best to take the production component and some how estimate how valuable that production will be to a team–accounting for both leverage and the context of wins. How you would account for leverage in the future I have no idea, but someone who is better than math at me could probably figure it out.
It’s important to remember that a reliever’s leverage index can change rather drastically from year to year, due to his team finding itself in many more or fewer late and close situations or due to the reliever changing teams or due to the reliever changing roles et cetera. It’s also important to remember there’s little certainty as to what a reliever’s leverage index will be. Teams handing out these contracts probably have a much better idea of what that LI will be than us outsiders do.
Anyway, WAR doesn’t work just fine for valuing relievers. It does nothing to account for context, the production is quantified with a bias metric which undervalues relievers, it doesn’t consider the fact that a good bullpen facilitates Pythagorean luck, and it leaves something to be desired in the leverage calculation department.
Production is a great basis, but without giving said production context, it’s impossible to know how valuable it actually is.
Cameron concludes:
That a half dozen major league clubs are overvaluing proven relievers does not mean that WAR doesn’t work for them. It’s just an arbitrage opportunity for the rest of baseball.
I don’t so much mind the perceived arrogance he lets on, I’d be a hypocrite (seeing as much of my work seems to be perceived as arrogant, even if that’s the last thing I want to do) if I did. It’s the fact that he’s sort of encouraging ignorance–so as to say, “WAR is all you need, if a team makes a move that doesn’t agree with the Fangraphs groupthink, their GM is an idiot”. Note: I don’t mean to criticize the authors and publishers and creators of Fangraphs, the readers that take their work as gospel is the target of that line.
Adjusting for future context, in addition to other flaws, is something that WAR simply doesn’t (and probably can’t) do. Turning a blind eye to other factors and considering only WAR when valuing relievers is to omit some things that most decidedly influence how valuable a reliever is to a team.
Without even getting into the small sample size issues that relievers inherently encounter, relying exclusively on WAR when attempting to value a reliever invokes a flawed understanding how valuable a reliever is. At the very least, we should be looking at the context of the reliever’s future production, if not re-thinking FIP and the role-specific thing.
Topics: Atlanta Braves | 10 Comments »
January 20th, 2010 at 4:46 PM
I had this problem with the Cy Young thing. You have to take context into account. Even if it has very little predictive value.
In regard to the valuation of a reliever… I actually don’t have a problem with the 40% extra leverage index.
What would you have? 60%? or are you talking a different approach to linear weights?
The greatness of fangraphs is that it is open. Bpro is not.
If something at fangraphs smells wrong, you can access the formula directly.
Discussion and open source information is what makes that site so great.
January 20th, 2010 at 4:48 PM
It depends on the question I’m attempting to address. I don’t necessarily think Leverage Index is a good place to start in the first place, though.
It needs a lot of study that I’m not qualified to do.
January 20th, 2010 at 4:50 PM
You take the Dave Cameron and you made him HUMBLE.
January 20th, 2010 at 5:53 PM
Heh, I don’t know about that. Certainly wasn’t my primary intention.
January 21st, 2010 at 12:40 AM
I had a problem with his idea of a replacement reliever:
If a bullpen loses it’s best pitcher, the whole bullpen is downgraded. The #2 reliever is now downgraded to the #3 reliever, #3 is downgraded to #4, ect.. These downgrades are events that occured because the top reliever was lost. He should get credit for the subsequent downgrades (or lack thereof).
Therefore it seems to me that the ace reliever’s value above replacement look something like this:
SUM:[value#1-value#2, value#2-value#3,...]
Is this what he refers to as the principal of chaining? It seems like he’s accounting for the first drop then just guessing at the rest.
The other drops may not be huge due to their low leverage, but they do matter and should be valued properly. (Im not the right guy to study this, I need to find a job.)
In fact, I saw the Braves shit the bed this season by throwing relpacement relievers in low leverage situations. See: ATL@PHI 04/04/09, ATL@COL 06/12/09
January 21st, 2010 at 1:42 AM
Yes, that’s what chaining accounts for. This article explains all of the accounting behind it.
January 21st, 2010 at 9:27 AM
Peter,
has anybody seriously looked at chaining as to its effects with
1. position players (think Cardinals before signing Holliday and after)
2. starting pitchers (think Braves before the Vazquez trade).
I didn’t know that was what you called it, but I certainly believed that the valuation of (particularly “proven” and above average) starting pitchers in trade and in free agency when compared to the much less risky (primarily injury, but also performance) good position players had this kind of issue in play. Not based on leverage, but based on an “additive effect on quality”. It is, I propose, more likely the result of the fact that almost no team woudn’t improve if John Lackey joined that team. However, how much would the Cardinals improve if they got (a) Adam LaRoche or even (b) Derrek Lee?
Then, a corresponding corrollary would be to look at how little effect losing a pitcher might have when you are the Braves and lose Vazquez. It affects the Braves by some percentage of the extra starts by Lowe, Kawakami, and Medlen and maybe Jo Jo Reyes for a few. If Jo Jo became the #5 starter as a direct result of a trade of Vazquez, the “win impact” would be much greater than when close, but not quite as good replacements are available.
January 21st, 2010 at 10:32 AM
I don’t think it matters with starting pitchers and starting position players because the leverage index is always one when they begin the game.
January 21st, 2010 at 3:59 PM
Is there a leverage issue for pinch hitters? Like say a few years ago when Darryl Ward was basically “the man”. If his 100 PA’s would be 75% in close and late.
Or the leverage associated with a good OBP guy leading off a late close inning.
I can’t help but believe if Braves had an average Hinske year last year as compared to Norton that they don’t pick up 4 to 5 games.
January 21st, 2010 at 4:58 PM
Or a Norton ’08 year when he posted a .361 on base average.