http://www.baseballprospectus.com/article.php?articleid=1459


How Sure Is A "Sure Thing"?



Can a "can't-miss prospect" miss?

Will young pitchers really break your heart?

Last year, in a highly entertaining Baseball Prospectus roundtable Joe
Sheehan remarked:

I would submit that there are no circumstances in which a pitcher is the best
prospect in baseball.


To which Derek Zumsteg replied:

If a pitcher can't be the best prospect in baseball, can he be the #2? Can he
be the #3?


It's a valid question. There is plenty of anecdotal evidence to suggest that
pitching prospects are more likely to flame out--but how much more likely?
Enough to drop them a few notches down on a top prospects list? If so, by how
many spots?

The good folks over at Baseball America take a view somewhat different from
Sheehan's. Will Lingo, in an article accompanying this year's Top 100
Prospects list, wrote that "position players are safer bets than
pitchers"--but BA still ranked Josh Beckett and Mark Prior 1-2. In a chat
session later that day, Jim Callis defended the choices of Beckett and Prior
on the grounds that "there's no way of telling which pitchers aren't going to
make it. In theory, our list represents who we'd take over whom. Pitchers may
be shakier than hitters, but we'd still take Josh Beckett or Mark Prior over
anyone."

Risk, Reward, and Expectation

The choices of Beckett and Prior for the top two slots can be justified if
the rankings are intended to reflect the players' ceilings, the highest level
of performance that they can reasonably be expected to reach. It's easier to
imagine them turning in Hall of Fame careers than anyone else in this
prospect class except maybe Hank Blalock. And the preference for higher-risk,
higher-reward players does seem to be common among prospect analysts. Gary
Huckabay expressed this view in the BP Roundtable: "I think the upside/risk
metric should favor upside."

But how much risk should it take to cancel out an increased possible reward?
Dave Pease captured some of this idea, saying, "I also don't really buy not
allowing a pitcher to be the top prospect, as long as he's the top prospect
when taking an increased injury probability into account." This sort of
suggests the idea of taking an average of the future possibilities: a 25%
chance of swinging 20 games into the win column (with a 75% chance of career
loss due to injury) should be worth about as much as a 100% chance of gaining
five wins. You might tinker with the formula and give it to the higher-risk
guy on tiebreakers; still, the two players should be considered more or less
equivalent as prospects.

But how to evaluate the risk/reward tradeoff? Are we left at the mercy of our
own subjective whims?

Career Value and the Historical Top 100

As it happens, we do have a metric that will allow us to study this question
with at least some degree of objectivity: Value Over Replacement Player
(VORP). Using this tool--developed by BP's Keith Woolner--we ask, "Suppose
this player were lost due to injury, and replaced by a readily available
player; how much would it hurt his team?"

Woolner's study in Baseball Prospectus 2002 develops a detailed set of
standards for replacement level, depending on position (including starter vs.
reliever). A good rule of thumb is that replacement level will be about one
run per game worse than average; for example, in a five-run-per-game
environment, a team of replacement-level players would score about four runs
per game and give up about six. With some further calculations, we find that
a league-average regular will contribute two wins per season compared to a
replacement-level player. A borderline All-Star will typically be worth about
five wins, an MVP candidate eight to ten, and a season for the ages (such as
Babe Ruth's 1920, or Barry Bonds's 2001) around 15.

Using this, we can find an objective comparison of hitting and pitching
prospects by looking in Baseball America's historical Top 100 lists, and
asking questions like, "what's the average career VORP for a hitter ranking
in the top 10?" "How about a pitcher in the top 10?" "How do they compare?"

If we find that the average is about the same for pitchers as for hitters,
then we can conclude that the risk/reward tradeoff is working as it should.
If not, then we will have a reasonably objective basis for suspecting that BA
may be overrating one group relative to the other.

The Study

For this study, I estimated career VORP for the BA's top 100 prospects from
1990 through 1997, that is, those who have had at least five years to prove
themselves. I used the rule of thumb that 10 runs of value moves one game
into the win column. This is what I found:

BA Top 100 Prospects, 1990-97
Average Career VORP in Games through 2001



Ranking    Hitters  Pitchers

1-10        18.2       6.5
11-25       10.9       5.8
26-50        8.3       5.2
51-75        5.2       4.4
76-100       4.9       3.7

What this tells us is that an average pitcher in the top 10 contributes
scarcely a third as much as an average hitter in that group. It tells us that
pitchers in the top 50 are only slightly better than hitters in the second
50.

Not only do hitters ranked in the top ten include superstars like Alex
Rodriguez, Chipper Jones, Nomar Garciaparra, Derek Jeter, and Manny Ramirez,
but those guys are backed up by a host of players like John Olerud, Tim
Salmon, Greg Vaughn, and Mo Vaughn, all who have been above-average
ballplayers for several years. The mediocrities in the group (Wil Cordero,
Alex S. Gonzalez) and flops (Eric Anthony, Ruben Rivera) are but a small
minority in a star-studded group.

Among pitchers ranked 1-10, only Pedro Martinez has become a superstar. A few
other guys have had some success (Steve Avery, Ben McDonald, Arthur Rhodes),
but even those are the exception rather than the rule. The career success of
pitchers ranked 1-10 is a better match for hitters ranked 41-50 or so (that
is, about the 25th through 30th-ranked hitters in their prospect classes).

What then? Should we answer Zumsteg's question by saying that no pitcher
should be ranked in the top 25?

Not quite. Let's dig a little deeper.

Sometimes we hear that some groups of pitchers may be at higher risk than
others: college versus high school, say, or perhaps lefties versus righties.
Let's try a college/high school split, and also separate out pitchers from
south of the border:

BA Top 100 Prospects, 1990-97 (Pitchers Only)
Average Career VORP in Games through 2001



Ranking    College  High School  Foreign

1-25         9.0        3.5        9.6
26-50        7.9        3.2        5.9
51-75        6.1        2.2        6.0
76-100       3.8        3.5        3.8

(For this, I didn't separate out the top 10; the samples were small enough
already. In particular, the Foreign group has only 11 to 13 players in each
of its four subgroups, so that Pedro Martinez--he of about 53 wins of career
value, a marginal Hall of Fame total already--distorts the numbers heavily
when he checks in, once at #10 and once at #62. Also, for some players who
have been out of the game for a while, I had to make my best guess as to
whether they went to college or not.)

Now we're getting somewhere. Not only does the collegiate group come out on
top, but the high-school group consistently ranks worse even than their
foreign brethren. This is surprising, because these pitchers--in this study,
all from Latin American countries--face the rigors of professional ball from
age 18 or even younger, so that we wouldn't expect them to have any less arm
strain than North American high-school kids. It could be a small-sample
effect, although the Latins still come out better even if you take Martinez
out of the equation, so I suspect that the difference is real.

Indeed, the results for high-school pitchers are stunningly poor. Even those
ranked in the top 10 have contributed an average of only 3.6 games to their
teams' win columns. Not even the best of the high school pitching prospects
perform better than the hitters at the bottom of the top 100.

What about a possible lefty/righty split?

BA Top 100 Prospects, 1990-97 (Pitchers Only)
Average Career VORP in Games through 2001



Type               Value

College RHP          6.9
College LHP          4.1
Foreign LHP          7.4
Foreign RHP          5.9
High-School LHP      3.2
High-School RHP      3.1

I'm not sure about the significance of the college lefty/righty split; if
anything, I would have expected more of a difference among high schoolers.
Perhaps it's just the random effect of a small sample; Andy Pettitte was the
only really good (20+ games career VORP) college lefty among 35 such pitchers
in the study, compared with Mike Mussina, Kevin Appier, Darryl Kile, Charles
Nagy, Roberto Hernandez, and Ben McDonald among 92 guys pitching from the
right side. A couple more good lefties would have balanced things a bit.
Let's call this one undecided for now.

The foreign left-hander group contains only eight pitchers, including Wilson
Alvarez twice, and he accounts for almost the entire total by himself; let's
not try to draw any conclusions there.

The Trouble with High-School Pitchers

Now let's return to the problem of why high-school pitching prospects perform
so poorly at the major-league level. If the problem were overworking their
arms in the pros, we'd expect that to affect the foreign pitchers just as
much.

My current theory--only a theory, mind you--is that these results are driven
by the fact that college pitchers are a self-selecting group. A talented
pitcher who's also a good student is more likely to go to college; a guy with
a golden arm but less interest in book-learning will usually prefer to go
straight to the pros. This would, of course, be much less relevant to Latin
American pitchers, who often have little or no educational opportunity; no
matter how smart a Latin pitcher is, signing the pro contract will almost
always be his best career option. Pedro Martinez never pitched in college,
but by all accounts he's one of the smartest pitchers since Christy
Mathewson.

I suspect that the fallout rate for pitchers, especially those drafted out of
high school, may be more than just the matter of injury risk. There may also
be a substantial risk that the pitcher won't master the mental game of
pitching at the big-league level.

It's also possible that self-discipline may be a factor. A guy who will face
three years of college may be more likely to have the discipline to keep
himself on the sort of rigorous physical conditioning program needed for
long-term major-league pitching success (Roger Clemens is the current leading
example). A guy who's in a hurry to sign a contract may be less likely to put
in the long-term effort needed to reach his career goals.

I'm not saying that high-school pitching draftees are necessarily lazy
idiots--Greg Maddux being an obvious counterexample. It's just a matter of
general tendencies. One interesting study would be to see if pitchers who
signed out of high school in spite of good grades and test scores do any
better in the majors than pitchers who were poor students. Good luck getting
the necessary data for that one.

Back to the Original Question

Zumsteg asked in last year's BP Roundtable: "If a pitcher can't be the best
prospect in baseball, can he be the #2? Can he be the #3?" Let's see now if
we have enough data to answer that.

We know that the average hitter in BA's top 10 from 1990-97 has produced
about 18 games of VORP in his career. The prospect lists tend to contain
about 60% hitters and 40% pitchers, so this is equivalent to about the top
six hitters per year. To rank the best of the pitchers in the top six or so,
therefore, would require being able to identify pitchers that can be expected
to produce at the same level (on average).

It's clear that we cannot do that. Even if we limit our scope to the best of
the college pitchers, we'll have a hard time identifying pitchers expected to
produce more than 10 or 12 games of value. The two college guys rated at #2
during those years, Ben McDonald and Paul Wilson, average out to a VORP of 12
(McDonald scores at 23, but Wilson, so far, only 1). The only other two
college pitchers ranked in the top 10 were Alan Benes and Kris Benson, who
don't pull up the average any.

Let's suppose, hypothetically, that Mark Prior can be counted on for 12 games
added to the win column over the next several years (halfway between McDonald
and Wilson). Where, then, would he rank?

Not in the top five, certainly. Even if the top five hitting prospects (let's
say Hank Blalock, Sean Burroughs, Carlos Pena, Nick Johnson, and Joe
Borchard) don't all become A-Rod-type superstars, they're still very likely
to have John Olerud-type careers with several years above average. With
Prior, there's a very heavy risk of not even reaching that level. Even if a
hitter and a pitcher have similar ceilings, a hitter has a much better chance
of achieving it--or even of just becoming a good ballplayer--than a pitcher
does.

Let's look again at the performance of the top hitting prospects:

BA Top 30 Prospects, 1990-97 (Hitters Only)
Average Career VORP in Games through 2001



Ranking  Avg. # of Hitters in Group  VORP

1-10                7                18.2
11-20               5                12.9
21-30               6                 9.3

A top collegiate pitcher like Prior would then seem to come in at around the
bottom of the second group, #10 or so (depending on the quality of the top
hitting prospects). Even in the best possible case among college pitching
prospects--say, Roger Clemens in 1984--I wouldn't put him in the top five.
Certainly Clemens's performance since then would have more than justified the
ranking; but in 1984, there would have been too much room for doubt, too much
that could have gone wrong.

What about Josh Beckett? Well, he's a very talented pitcher. He's also a
pitcher out of high school, and one with an injury history already (although
he seems to be recovered at this point). Most of what can be said for Beckett
was also said for Rick Ankiel a couple years ago; and that's not what's being
said about Ankiel now.

Certainly Beckett could turn in a Hall of Fame career. So could Todd Van
Poppel have, or Brien Taylor or Roger Salkeld or Matt White. Maybe Kerry Wood
will; we'll see if his elbow holds out. Even so, the risk is very, very high.
My estimated future value for Beckett? I'll go on the high side of the
historical evidence--maybe the high-school pitching mega-prospects of the
'90's were a bit unlucky--and say perhaps seven or eight games expected
value, and a prospect ranking around #30. To put him above that would require
finding some clear means of setting him apart from all the failed
uberprospects before him; and at this point, I don't see such an argument.

Conclusion

What now? Should we say, along with Sheehan, that "the phrase 'spectacular
pitching prospect' is an oxymoron"?

I, for one, would agree very strongly. The data certainly support it, at
least based on eight years of BA rankings. There's just too much that can go
wrong with a pitcher, and if that's true for pitchers in general, it's doubly
or triply true for pitchers signed out of North American high schools.

This raises the question: How can a team build a pitching staff? One way, of
course, is to spend for veterans; if you can trade for an impending free
agent like Roger Clemens and sign him to an extension, and then go get Mike
Mussina when he hits the market, that will take a lot of the uncertainty out
of the process.

But what can a more budget-limited franchise do? Build a young-guns rotation
through the farm system? That's a risky proposition. It seems to be working
for the A's, who have Tim Hudson, Barry Zito, and Mark Mulder (all college
draftees, you'll notice), but Ryan Anderson, Freddy Garcia, and Gil Meche
looks like about one out of three at this point for the Mariners. Paul
Wilson, Bill Pulsipher, and Jason Isringhausen never became what the Mets
hoped they would, and the A's have nothing to show for their "four aces"
draft of Van Poppel, Kirk Dressendorfer, Chris Peters, and Dave Zancanaro.

However, even if Ryan Anderson shouldn't have been rated in the top 30 last
year, much less #1, I still like the Mariners' approach of a pitching-deep
farm system. If Anderson gets hurt, maybe Joel Pineiro can step up; if they
lose Jeff Heaverlo, they'll see what Rafael Soriano can do; and even if those
guys don't work out, there are eight or ten others who realistically could. I
dislike spending first-round choices on pitchers (except for the occasional
college draftee); those picks are usually better spent on more-reliable
hitters. But I'm all for bulking up on pitchers in the middle rounds.

To return to Jim Callis's example: if I had Josh Beckett on my team, and
Callis offered me Blalock or Burroughs or even Wilson Betemit for him, would
I do the deal? In a heartbeat.

Among pitching prospects, there is no such thing as a sure thing.