As we all hunker down with our spreadsheets in preparation for draft season, it’s time for my favorite yearly tradition – evaluating the projection systems! The goals, as always, are to evaluate the best projection systems on a per-category basis for the purposes of fantasy baseball, and to determine the best possible mix of projections (separated into rate stats and playing time) as we look ahead to ‘20. Additionally, this year I’ve pulled up a list of players that each system was ‘in’ on – both the ones that were big wins, and the ones that completely tanked in ‘19. In this study, I’ll focus on the most commonly used projections – the same ones that appear in the Big Board: Steamer, PECOTA, ZiPS, ATC, The Bat, FanGraphs Depth Charts, and FanGraphs Fans. I’ll also include my recommended Big Board mix from 2019, which was a weighted average of the best systems. And finally, I’ll also define a new recommended Big Board mix looking ahead to 2020!
The categories of interest are the typical 5×5 categories, HR/R/RBI/SB/AVG for hitters, and W/SO/SV/ERA/WHIP for pitchers. Since for fantasy purposes we only care about the relative projections made by each system (ie, we only need to know Trout is the best hitter in baseball, not exactly what his AVG will be), I’ll primarily use R squared to evaluate how well the projections correlated to actual results, but I’ll also be including RMSE to show the absolute error in each projection system. This year, I’ve also added a quick auction value calculation for each system, and examined how the actual 2019 $ performances compared to the $ numbers generated by each system.
You can find details of the player pool, definitions of stats, and the breakdown of weights for my Big Board mix down in the ‘Fine Print’ below! Note that I’m including both the “BB Mix ‘19” (uses weights defined before 2019) and the new “BB Mix ‘20” (uses new weights defined after 2019). For the sake of fairness, only the ‘19 Mix is compared against the other systems – and spoiler alert, it comes out on top!
Hitters
🏆 1st place goes to my BB Mix ‘19, followed by the FG Depth Charts and Steamer with roughly similar performances in 2nd and 3rd.
Dollar Values – ZiPS somehow managed to string together bad playing time projections and poor performance in the rate stats in such a way that the system actually produced the best dollar values, by a narrow margin. I would say the BB Mix ‘19, which only got narrowly edged out for the top spot, produced a more trustworthy result, given the #1 rank in many categories. You’ll also see below that BB Mix ‘19 produced a much lower RMSE. Hopefully you didn’t build your hitter dollar values from PECOTA or the Fans projections last year.
Playing Time – Nearly every projection system struggles here every year. Between injuries, lineup spots, and role changes, playing time is just plain difficult to peg. Many people swear by the hand-curated playing time over at FanGraphs, but here again we see it failed to live up to that hype. PECOTA did surprisingly well after being dead last in the 2018 analysis! My BB Mix ‘19 edged out the other systems by averaging Steamer, ZiPS, and the Fans, but looking now to 2020, FanGraphs still has not started collecting 2020 Fan projections! The new ‘20 mix for PA will instead be a weighted average of Steamer and PECOTA, and improves the R sq. to 0.22.
Rate Stats – The rate stat performances were more similar than different, although you may note some poor performances in R and RBI for a few systems as usual. I’m not putting much stock in the projections of HR’s from last year because of the juiced ball, and I’m extremely wary of what the 2020 projections are doing with HR. The fate of this coming year’s projections likely depends on how much MLB tinkers with the baseball, if at all. It’s worth noting that The Bat actually did pretty well in 2019, but came up short in the SB’s category for some reason, which held it back. Two of the systems which average projections (FGDepth, BB Mix ‘19) come out on top, as might be expected. The new ‘20 mix will be a weighted average of Steamer, ZiPS, and ATC, which helps to produce a strong result per-PA across all five categories, though only a small improvement in R sq. over the ‘19 mix.
The net result is calculated in terms of the percentage above (or below) average across the five categories for each system, and is also plotted below.
The Big Board Mix ‘19 edges out all the others, beating the average five-category R sq. by about 7%. The advantage has shrunk since last year! On a per-PA basis, FGDepth comes close, but the improved playing time gives my mix the win. This is yet another year of very poor performance for PECOTA, but I am intrigued by the good PA projection. This year they are hyping a big re-design… which is exactly what I was saying in this paragraph of this piece one year ago, too. So cross your fingers, maybe the re-re-design here in 2020 means this is their year.
RMSE: This gives you an idea of what the typical error is in each category, for each system. Each counting stat is listed as error per 600 PA to normalize the values to approx. full season scale.
Pitchers
🏆 1st place again goes to my BB Mix ‘19, followed by Steamer and then ATC in 2nd and 3rd.
Dollar Values – Steamer, Depth Charts, BB Mix, and ATC produced dollar values of very similar quality to one another. You’ll also see below that by RMSE, they’re still close, though ATC drops back a bit. Hopefully you didn’t build your pitcher dollar values from PECOTA last year.
Playing Time – Pitcher playing time projection is hard, very hard. Like last year, ATC is near the top, and the BB Mix ‘19 is made marginally better by averaging with Steamer. ZiPS is terrible as always in this department… I don’t think Dan makes any claims to projecting IP reliably, though. Looking ahead, the 2020 BB Mix weights remain about the same.
Rate Stats – We again see that many systems produced similar results on a rate basis, with a few poor performances in individual categories here or there. ERA projections were bad across the board last year, thanks to the inflated HR rates coming from the juiced ball. As noted above in the hitter’s section, stay tuned for what happens in 2020. Steamer had the best W, ERA, and WHIP projections, but is held back by poor SV projections. ATC turned in a solid performance (average or better) across all five categories. BB Mix ‘19 was a combo of four different systems, producing the best overall performance by being above average in every category. For the new ‘20 mix, we’re simplifying, going to a weighted average of just Steamer and ATC.
Again, the net results are plotted below:
The Big Board Mix ‘19 beats the others more convincingly on the pitching side, exceeding the average five-category R sq. by about 19%. ATC and Steamer are the only others that end up above average. On both the hitting and pitching side, if you had to choose just one system, it’s clear that ATC and Steamer are the two best, and averaging them together (as in the BB Mix) produces an even better result. Also… copy/paste my comments about PECOTA above, as they rank last here as well.
RMSE: The root mean square error for each pitching category… in this case, normalized to 200IP (or 65, for SVs).
REVIEWS BY SYSTEM
For each system, I’ll highlight a few features, strengths, or weaknesses. I’ve also determined which system was ‘highest’ on each player in the dataset (i.e. which system would have drafted them). From that list of players, I’ll note each systems best ‘wins’ (📈) and ‘losses’ (📉), based on the players’ surplus values measured as the difference between the actual 2019 $ value and the average projected $ value.
Big Board Mix 2019
🏆 #1 Hitter Projections
🏆 #1 Pitcher Projections
📈 Christian Yelich (+$17), Roberto Osuna (+$9)
📉 CJ Cron (-$5), Max Scherzer (-$4)
It’s tacky to crown myself the king, but the results are what they are! By both R squared and RMSE, my weighted average of systems had a strong year. The Big Board mix hit well on Yelich and his near-MVP season. It’s worth noting that the systems which average other projections (BB Mix, ATC, FG Depth) were not often the ‘highest’ on any particular player. That said, for this subset of players, the Big Board mix made very few mistakes on bad players, with CJ Cron and Max Scherzer being the “worst” misses.
Steamer
🏆 #3 Hitter Projections
🏆 #2 Pitcher Projections
📈 Austin Meadows (+$20), Gerrit Cole (+$36)
📉 Jose Ramirez (-$16), Chris Sale (-$21)
This was a fantastic result for Steamer. I continue to be impressed at how this system competes with the ‘weighted-average’ systems. The only thing really holding them back on the pitching side are the SV projections, and for some reason, Runs on the hitting side. Steamer grabbed some good breakouts last year, including Austin Meadows, and pegged the dominant season for Gerrit Cole. They were in on JoRam and Sale, but in each case, those are players that had weird, possibly injury-related issues. I’m letting them off the hook a bit for those.
ATC
#4 Hitter Projections
🏆 #3 Pitcher Projections
📈 Freddie Freeman (+$14), Eduardo Rodriguez (+$14)
📉 Robinson Cano (-$19 #LFGM), Jose Leclerc (-$7)
While the #4 and #3 finishes don’t look overly impressive, I think ATC had a very good year, as those are narrow margins between the systems near the top. Steamer and ATC are the two best public systems right now, which is why my BB Mix is basically a weighted average of the two. There are some nice buys here, including Freddie Freeman… the appearance of Robbie Cano makes me wonder if Ariel (a Mets fan) was dreaming on his Mets and had his finger on the scale for that one. It’s worth noting that the systems which average other projections (BB Mix, ATC, FG Depth) were not often the ‘highest’ on any particular player.
FanGraphs Depth Charts
🏆 #2 Hitter Projections
#5 Pitcher Projections
📈 Adam Eaton (+$7), Stephen Strasburg (+$23)
📉 Vlad Guerrero Jr. (-$11), Andrew Miller (-$9)
The hand-curated playing time hurts them, but generally speaking, the Depth Charts show just how powerful projection averaging is. A simple 50-50 mix of Steamer-ZiPS produced a strong result, as it almost always does. Depth Charts were highest of the systems on Strasburg, who put together a healthy season and carried the Nats to the ‘ship, but also Vlad Jr., who had a somewhat disappointing MLB debut. It’s worth noting that the systems which average other projections (BB Mix, ATC, FG Depth) were not often the ‘highest’ on any particular player.
ZiPS
#5 Hitter Projections
#6 Pitcher Projections
📈 Rafael Devers (+$33), Shane Bieber (+$28)
📉 Jake Bauers (-$16), Kevin Gausman (-$16)
I’m glad I did the player-by-player analysis. ZiPS was not particularly strong on either side of the ball, particularly on the pitching side, and yet Dan’s system was the highest on a number of big breakouts. Devers, Bellinger, Semien, Bieber, Verlander, and Hader were all on the ZiPS list – each of these were players that carried teams all year. Even if you wasted a pick on Bauers or Gausman, I think a ZiPS-based fantasy team might have actually been very strong!
The Bat
#6 Hitter Projections
#7 Pitcher Projections
📈 Anthony Rendon (+$25), Charlie Morton (+$23)
📉 Jesus Aguilar (-$21), Rick Porcello (-$12)
It wasn’t an impressive 2019 for The Bat, which is a shame because I came away from last year’s analysis optimistic for the system. Maybe it’s something to chalk up to the difficulty of projecting 2019 in general, and The Bat could bounce back next year. After a poor showing in playing-time projection last year, Derek just went with the Depth Charts’ playing time in 2019, which is probably a good call, although stealing the Steamer or ATC playing time might be better. Some nice buys on older players like Rendon and Morton here, but also some real duds on the vets Aguilar and Porcello. Does The Bat love playing the oldies?
FanGraphs Fans
#7 Hitter Projections
#4 Pitcher Projections, somehow
📈 Peter Alonso (+$30), Sonny Gray (+$24)
📉 Khris Davis (-$21), Kyle Freeland (-$22)
Oh the Fans. I’m not sure they’re going to exist for 2020 unless FanGraphs gets the surveys posted very soon, and that’s a bummer because they’re always entertaining in their endless optimism. Fans give you results like believing Peter Alonso is instantly going to be very good, or Sonny Gray is going to totally return to ace-status, and sometimes they are right! But usually they are wrong, and say things like Kyle Freeland is a perfectly cromulent pitcher… woops? Note that fan projections only exist for a subset of players, so the rest are filled in with Steamer projections – that unavoidably serves to improve the overall result for the Fans. But hey, they need all the help they can get!
PECOTA
🚨 Worst Hitter Projections (but #2 PA projections)
🚨Worst Pitcher Projections
📈 DJ Lemahieu (+$32), Zach Greinke (+$27)
📉 Jose Peraza (-$25), Wade Davis (-$23)
There’s always next year. For what it’s worth, I love what they’re doing with releasing the various percentile projections now. It’s interesting to see PECOTA in on both DJ and Greinke, two players whose success is built on contact management. These are two players that are criminally underrated by many projection systems. Perhaps this system is more willing to believe in sustained batted-ball ‘luck’ from season to season? But given the overall poor results for PECOTA, maybe that willingness makes the overall performance of the system worse.
THE FINE PRINT
-
The most common fantasy leagues draft about 300 players, broken out into 180 hitters and 120 pitchers, and so I’ve used the consensus top 300 players as determined by an average of the projection systems, and will only be evaluating the systems based on their projections of those 300 players.
-
Hitters that didn’t end up reaching at least 350 PA in 2019 and pitchers that didn’t reach 35 IP in 2019 have been thrown out of the sample to reduce the effect of short-sample playing-time outliers (typically from injury rather than lack of performance, so survivor bias is not a major factor).
-
Rate stats like AVG were evaluated as part of the ‘total’ projections by using a playing-time weighted value indicated by an ‘n’ (e.g. “nAVG”) to convert them to counting stats.
-
The SB (*) correlations were evaluated on the basis of two separate populations and averaged: players who stole >=5 bases, and those who stole <5. Past analysis has shown that evaluating this as a single population gives undue credit for projecting the low-steal players, and not enough credit for accurately projecting high-steal players.
-
The IP (^) projections were evaluated on the population of SP’s only. RP inning projections show nearly zero correlation and introduce undue noise.
-
The W and SO (*) projections were evaluated on the basis of two separate populations and averaged: starters and relievers. Past analysis has shown that evaluating this as a single population gives undue credit for projecting the separation between these two populations.
-
The SV (**) projections were evaluated for relievers only.
-
The Bat does not project playing time (uses FGDepth playing time), and does not receive a separate grade for it.
-
The Bat, ZiPS, and the Depth Charts all use the same Saves projections as Steamer, and do not receive separate grades for Saves.
-
The Big Board mix 2019:
-
Hitter Playing Time: 41% Fans, 32% ZiPS, 27% Steamer
-
Hitter Rate Stats: 58% ATC, 42% Steamer
-
Pitcher Playing Time: 72% ATC, 28% Steamer
-
Pitcher Rate Stats: 32% ATC, 30% Steamer, 27% PECOTA, 11% The Bat
-
-
The Big Board mix 2020 (**Note: The final 2020 mix is updated in this later post**)
-
Hitter Playing Time: 44% PECOTA, 56% Steamer
-
Hitter Rate Stats: 33% ATC, 47% Steamer, 20% ZiPS
-
Pitcher Playing Time: 71% ATC, 29% Steamer
-
Pitcher Rate Stats: 48% ATC, 52% Steamer
-