I'm currently making my way through Baseball Prospectus's archives, since I managed to completely ignore the site for about six months despite paying for the privilege to read it. Back on May 12, 2004, Dayn Perry wrote an article excoriating Jimy Williams's use of his bullpen (it's in their free section). I'm going to borrow his methods because it made me curious about how the A's bullpen was deployed last season.

First, we have to consider performance. There are two ideas to consider here: the value we would expect from the player before the season started (which would help determine the role that player would play) and his actual value as seen from the end of the season (because as a manager watches his relievers throughout the season, he should be expected to adjust their roles based on who is living up to their expected performance). As Perry did, I'll use VORP/IP as my value measure. Below is the list of every player who made at least 20 relief appearances for the A's this year (to eliminate the end-of-year usage of Joe Blanton and the beginning-of-year usage of Chad Harville, for instance), along with their 2004 weighted-mean VORP/IP as predicted by PECOTA and their actual performance.

It should come as no surprise that Dotel was supposed to be the most valuable pitcher on the staff, nor that Bradford was supposed to follow closely on his heels. What is a surprise is how poorly Bradford and Rhodes actually performed, though they were balanced out by awesome contributions from Mecir, Duchscherer, and Hammond. Who was betting on the latter two to be, inning for inning, the most valuable relievers for Oakland last year? Not PECOTA, that's for sure.

Moving on, Perry needed a composite measure of the types of games each reliever was entering. What he used was the average of the absolute value of the run differential the relievers faced when they entered their games. Absolute value is the key here because there are managers who use certain relievers only when the team is ahead and certain players only when they are behind, even though a one-run game is a one-run game in either direction, and you ought to be using one of your top relievers at that juncture.

The following two tables contain this data and are a copy-and-paste of the above table with the new column added in. The first table is ordered by expected value and the second by actual value. Hopefully, if the managing of the bullpen is competent, we'll see some (negative) correlation between the value metric and the usage one.

Here's a scatter plot for the five pitchers we have prediction data for with value on the y-axis and usage on x.

With the exception of the outlier (in so many ways!) named Chris Hammond, we seem to see the predicted better relievers used in overall tighter situations.

Here's the same scatter plot, but now with actual value on the y-axis.

This paints a much less pretty picture. Hammond in particular seems to have been mis-used. Why was the guy who ended up being the A's second-best reliever only pitching in games in which it would take more than a grand slam to change the lead? Rincon, Bradford, and Rhodes being used in tighter situations than Mecir is also disheartening. I suppose there was doubt that the man with the bad knees, feet, and just about everything else would hold it all together for much longer, but why was Arthur Rhodes let anywhere near a close game at the end of the season?

Now, that question begs another. Did usage change throughout the season? Was Rhodes, in fact, pitching in close games at the end of the season, or do his overall usage stats mask a late-season change? On would hope that usage didn't remain static, especially considering the case of a guy like Duchscherer, who only pitched in blowouts at the beginning of the season, but later on turned out to be a multi-inning ace, reminiscent of Ramiro Mendoza in his Yankees heyday.

What we can do to try to answer this question in a quick way is look at a running average of the usage stat we used above. We'd hope that a graph of these numbers for Duchscherer would steadily track downward, while the same for Rhodes would shoot upward, displaying a recognition by manager Ken Macha that these relievers were no longer pitching as might have been predicted at the beginning of the year.

Here's the running average graph for Duchscherer.

Duchscherer's line does move downward as the year goes on, fitting with what I remember of the season. He started out pitching six-run games and finished up throwing tough innings against top opponents like the Red Sox (and succeeding). Duchscherer's changed usage patterns are a credit to the management of the team on the field.

Next up is Bradford, who started the year as the guy who probably should have been Oakland's ace reliever (since Dotel hadn't joined up yet), though it's hard to say what was expected out of Rhodes. Bradford was pretty ineffective the whole year, so ideally, we'll see that he was used in fewer tight situations (rising graph) as the year went on.

What we see is sort of inconclusive, though the lack of any sort of steady trend indicates that he was pretty much used in any and all situations. There does seem to be a general sort of rising up until about his 55th game, when the graph starts to fall again. We'll count this as another plus in Ken Macha and Curt Young's column, a point for not blindly sticking with a guy when there were better options at hand, regardless of what he had done for the team in the past.

We already know what Chris Hammond's graph is going to look like: it's going to be far higher on the y-axis than he deserves it to be. It'll be interesting to see, though, whether there was any wising up on the part of the management toward the end of the season.

And the answer is "no." That's terrible. Here's a guy who should've been the second go-to guy in the 'pen, after Duchscherer, and instead was used in looser and looser situations as the year went on, even as it became more and more apparent how well he was pitching. Strike one against Macha and Young.

Dotel was used kind of weirdly when he was acquired, almost as if Macha bought into Jimy Williams's four-run ninth inning philosophy (wherein the closer ought to be used more often, but not necessarily smarter). Later on, according to my memory, he was used in more tied games as Macha and Young apparently got more liberal about his usage. Let's see if the picture backs up my recollection.

Indeed it does. From about Dotel's fifteenth game on, we see a steady decline in the graph. On the other hand, I also recall Dotel being generally used as a pretty conventional closer, so, perhaps more than the other relievers, his usage pattern stats can be affected by other elements. If the A's just happened to have a bunch of 1-run 9th inning situations at the end of the year, then it will appear that he was being used smarter, even though he was still being deployed as a neo-classical closer.

For Mecir, I'd hope to see something like Duchscherer's graph, with early usage leaving him essentially in a mop-up, middle-relief role and later usage pushing him into closer games as more of a setup guy.

It looks like it took the A's a little longer to get the idea with Mecir than it did with Duchscherer. The graph doesn't really turn around until Mecir's already pitched about 40 games, leaving only about twenty on the tail end of the season where it seems that Macha and Young started to have some confidence in him again.

I'm not sure what I should make of Rincon. He was used in both a LOOGY role and a multi-batter role this year, and, it seemed when I was collecting data, in a variety of run-difference situations. I really don't have a sense of what his graph will look like, nor what it ought to be.

As it turns out, the graph climbs steadily until about the 45 game mark, then falls. This is similar enough to Mecir's graph that I wonder whether their "usage" was more a function of the team's offense and starting pitching, similar to my comment about Dotel, than anything else. If they were being used in the tighter games, who was used in the looser ones? Maybe that's where Rhodes and Justin Lehr will come in.

Rhodes, I'm hoping, will display a radically upward-shooting graph.

In fact, to my surprise (I don't give often give Macha credit for anything, preferring to harp on technicalities and minor blips than laud larger victories), Rhodes's graph does veer upward markedly, from a low of about 1.6 after 20 games all the way up to the 2.24 he ended up with. What we see is, I think, a plus for the management: he was perceived to be a strong reliever and was used as such (his run differentials were well below two) until the perception wore off and the A's realized that he just wasn't going to be that good. That his run differential average shot up by over 0.6 in fewer than 20 games is an illustration of exactly why I decided to use these running averages. Rhodes is the model for the player whose average in some statistic doesn't tell the story of two disparate parts of a season.

Justin Lehr has a smaller sample size than any of the other pitchers to work with, so his graph probably won't have the clear trends of some of the others.

In fact, that's exactly what we see. There does seem to be an uptick at the end, but it's hard to really say anything about this picture.

In the end, I'm not sure how much we learned, except that the A's seem neither exceptionally good nor exceptionally bad at managing their relievers in optimal run-difference situations. This was fun enough, though, that I may try to keep and display usage stats for the 2005 season as it runs. If I can manage it, I may try to do the whole league, but if I can't, hopefully I can at least do the A's.