Evaluating the 2017-2019 Projection Systems

For many years running now, I’ve run a yearly evaluation of the projection systems for fantasy baseball purposes, including last week’s evaluation of the 2019 results. Two potential issues I’ve run across with these studies are 1) year-to-year variance and 2) small sample size. To attempt to address both, I’m back again, this time with an evaluation of performance across the last three seasons (‘17-’19)! This study combines the datasets from each of my last three yearly studies, studying how well each system projected the top ~300 fantasy players (the ones that met the playing time minimum) in the 5×5 categories by both R squared and RMSE.

In the intervening week, I’ve also obtained copies of the 2017-2019 Pod projections and Razzball projections from Mike Podhorzer and Rudy Gamble respectively, so I’ve added them to the running as well (Thanks Mike and Rudy!). For the unfamiliar, the Pod projections are created by hand by Mike, resulting from his various models for peripheral stats like BABIP and HR/FB. The Razzball projections are essentially repackaged Steamer projections, with more hand-curated playing time and SV projections. Without spoiling anything up front, these two turned in credible performances on the hitting side, but are very good on the pitching side!

Given the results of this study, and given a week to ruminate on the results, I’m revising my recommended 2020 Big Board mix of projection systems. In the interest of keeping things as open-source as possible, the new version does not include any of the ‘subscription’ (paid) projection systems (previously I had included PECOTA). Scroll to the bottom for the new recommendations, which are basically now a mix of Steamer, ATC, and Razzball (my projections tool will be updated shortly to allow you to pull in Razzball).

You can find details of the player pool, definitions of stats, and the breakdown of weights for my Big Board mix down in the ‘Fine Print’ below! Note that I’m including both the “BB Mix ‘19” (uses weights defined before 2019) and the new “BB Mix ‘20” (uses new weights defined during this study). For the sake of fairness, only the ‘19 Mix is compared against the other systems – and spoiler alert, it still comes out on top!

Hitters

🏆 1st place goes to my BB Mix ‘19, followed by Steamer and ATC with roughly similar performances in 2nd and 3rd. At 10% above average, the BB Mix gets a bit more separation from the other systems when examined over a three-year span! This is… more bad news for PECOTA, unfortunately, coming in at 17% below average.

Dollar Values – BB Mix ‘19 also produced the best dollar values over this three-year span. Steamer, ATC, and Pod are all in roughly the same neighborhood as well, and the RMSE results below tend to agree.

Playing Time – After seeing these results, I’m much less convinced of PECOTA’s strong 2019 performance in PA projections. Let’s see if they can keep it up for more than a year, because across ‘17-’19 they were the worst. The results are otherwise largely similar to the single-year results. Pod checks in with a decently strong performance, while Razzball’s hand-curated playing time came up a bit short. My BB Mix ‘19 system and Steamer are the best, but only marginally better than ATC.

Rate Stats – On a per-PA basis, the Pod projections come out on top. In particular, they’re strong in Runs projections, an area that has held Steamer back in particular. The two aggregation systems of ATC and BB Mix ‘19 also did quite well.

RMSE: This gives you an idea of what the typical error is in each category, for each system. Each counting stat is listed as error per 600 PA to normalize the values to approx. full season scale.

Pitchers

🏆 1st place again goes to my BB Mix ‘19 (15% above average), followed by the newcomers Pod and Razzball in 2nd and 3rd. Razzball in particular has put up a pretty impressive performance on the pitching side over 3 years, especially on a per-IP basis (12% above average). There is generally a lot more separation between the systems on the pitching side than the hitting side. PECOTA, ZiPS, and The Bat are being left in the dust a bit, living in the same tier as the woeful Fans projections down in the 14 to 17% below average range.

As a sidenote, I re-ran the numbers for the 2019 season alone, and Razzball comes out as the #1 system on the pitching side.

Dollar Values – Razzball is the top, out of a virtual tie between them and Steamer, BB Mix ‘19, and FG Depth Charts. You’ll also see below that by RMSE, they’re still close.

Playing Time – Razzball takes the cake here, so props to Rudy for his hand-made playing time projections. Pitcher playing time is a shot in the dark, but he’s hitting his targets a little more often. The other top systems are all pretty close behind.

Rate Stats – One thing that jumped out to me in this run is how much variance there is in SV performance. Razzball uses Steamer’s high quality SO/ERA/WHIP projections, but has vastly better SV projections, resulting in a system which out-performs the others by about 12% on a per-IP basis. The only system that gets close is mine, at 10%. Pod, Steamer, and ATC are behind for sure, but all are in a roughly similar tier of performance with each other.

RMSE: The root mean square error for each pitching category… in this case, normalized to 200IP (or 65, for SVs).

The Fine Print

• The most common fantasy leagues draft about 300 players, broken out into 180 hitters and 120 pitchers, and so I’ve used the consensus top 300 players from each year as determined by an average of the projection systems, and will only be evaluating the systems based on their projections of those 300 players. There is a slight bias here against the newly introduced systems (Razzball, Pod), since they were not included in the original selection of player pools.

• Hitters that didn’t end up reaching at least 350 PA in the given year and pitchers that didn’t reach 35 IP in the given year have been thrown out of the sample to reduce the effect of short-sample playing-time outliers (typically from injury rather than lack of performance, so survivor bias is not a major factor).

• Rate stats like AVG were evaluated as part of the ‘total’ projections by using a playing-time weighted value indicated by an ‘n’ (e.g. “nAVG”) to convert them to counting stats.

• The Bat (^) did not post public projections in 2017 and is only evaluated on the basis of 2018-2019 data.

• The SB (*) correlations were evaluated on the basis of two separate populations and averaged: players who stole >=5 bases, and those who stole <5. Past analysis has shown that evaluating this as a single population gives undue credit for projecting the low-steal players, and not enough credit for accurately projecting high-steal players.

• The IP (^) projections were evaluated on the population of SP’s only. RP inning projections show nearly zero correlation and introduce undue noise.

• The W and SO (*) projections were evaluated on the basis of two separate populations and averaged: starters and relievers. Past analysis has shown that evaluating this as a single population gives undue credit for projecting the separation between these two populations.

• The SV (**) projections were evaluated for relievers only.

• The Bat does not project playing time (uses FGDepth playing time), and does not receive a separate grade for it.

• Razzball does not project stats (it uses Steamer as a basis), but small differences result from changes in PA or IP vL and vR. The per-PA/per-IP stats are therefore very similar to Steamer (except for W/SV, which are manually adjusted by the Razzball system).

• The Bat, ZiPS, and the Depth Charts all use the same Saves projections as Steamer, and do not receive separate grades for Saves.

• The Big Board mix 2019:

  • Hitter Playing Time: 41% Fans, 32% ZiPS, 27% Steamer

  • Hitter Rate Stats: 58% ATC, 42% Steamer

  • Pitcher Playing Time: 72% ATC, 28% Steamer

  • Pitcher Rate Stats: 32% ATC, 30% Steamer, 27% PECOTA, 11% The Bat

• The Big Board mix 2020 (updated):

  • Hitter Playing Time: 68% Steamer, 32% ATC

  • Hitter Rate Stats: 40% Steamer, 50% ATC, 10% ZiPS

  • Pitcher Playing Time: 33% Steamer, 23% ATC, 44% Razzball

  • Pitcher Rate Stats: 22% Steamer, 30% ATC, 48% Razzball