Evaluating the 2021 Projection Systems

For many years running now, I’ve run a yearly evaluation of the projection systems for fantasy baseball purposes, including most recently the evaluation of the 2019 results and my three-year study of projection performance. I took 2020 off – 60 game projections are too noisy to put the same stock in. The goals, as always, are to evaluate the best projection systems on a per-category basis for the purposes of fantasy baseball, and to determine the best possible mix of projections (separated into rate stats and playing time) as we look ahead to ‘20. The correlation (R squared) numbers are most useful in my opinion, as they show what would happen when you create rank-ordered lists of players based on these stats. Two updates in approach this year:

1. I’ve added a projection system called “All”. It literally just evenly mixes all of the systems included in this study, and it’s actually quite good. This was also used as the base system to select the player pool from, which is more fair than previous approaches.

2. My recommended “Big Board Mix” this year is an average of the new ‘best’ mix (determined from 2021) and the previous weightings from my 3-year study. In this way, I am now basically taking a projection-system-style approach to projecting the projection systems. This was introduced in part due to the extremely good performance of The Bat X in ‘21.

You can find details of the player pool, definitions of stats, and the breakdown of weights for my Big Board mix down in the ‘Fine Print’ below! Note that I’m including both the “BB Mix ‘20” (uses weights defined before 2020) and the new “BB Mix ‘22” (the ‘best’ weighting of 2021 systems, defined during this study).

Recommended mix for ‘22 drafts:

• Hitter Playing Time: 75% Steamer, 25% ATC

• Hitter Rate Stats: *60% The Bat X, 30% ATC, 10% Steamer

• Pitcher Playing Time: 55% Razzball, 25% Steamer, 20% ATC

• Pitcher Rate Stats: 50% Razzball, 30% ATC, 10% Steamer, 10% The Bat

Hitters

The Bat X just put up an incredible performance in ‘21. Derek Carty is now way ahead of the others in terms of projecting rate-stats (playing-time could use some work, but this is true for all systems). It will be worth watching to see how the others develop if and when they start to integrate Statcast data more heavily. It’s also interesting to look specifically at what The Bat X is good at – homers and speed. Average was actually not a strong point, which could be random variation or could suggest the Statcast data is a little less predictive when it comes to BABIP. If you’re looking for a predictable team, draft for power (HR/RBI), and target guys The Bat X likes.

My 2020 mix had a weaker performance, partially due to a surprising down year from Steamer.

Something odd happened with playing time in ‘21. These correlation numbers are way lower than in year’s past and I had to check and double check that I didn’t do something wrong. Might be a funky development from injuries in the post-COVID year?

RMSE: This gives you an idea of what the typical error is in each category, for each system. Each counting stat is listed as error per 600 PA to normalize the values to approx. full season scale.

Pitchers

Razzball continues to be a really strong system on the pitching side, and there is again generally more separation between the systems here compared to the hitting side. The hand adjustment of playing time, W’s and SV’s clearly makes a big difference – nice job, Rudy and team.

My 2020 mix had a good year too, but the difference between that and Razzball are pretty negligible.

General thought on pitching – draft strikeouts and role. W’s are useless and you’re better off not including them in your valuation. ERA and WHIP are extremely noisey.

RMSE: The root mean square error for each pitching category… in this case, normalized to 200IP (or 65, for SVs).

The Fine Print

• The most common fantasy leagues draft about 300 players, broken out into 180 hitters and 120 pitchers, and so I’ve used the consensus top 300 players from each year as determined by an average of the projection systems, and will only be evaluating the systems based on their projections of those 300 players.

• Hitters that didn’t end up reaching at least 350 PA in the given year and pitchers that didn’t reach 35 IP in the given year have been thrown out of the sample to reduce the effect of short-sample playing-time outliers (typically from injury rather than lack of performance, so survivor bias is not a major factor).

• Rate stats like AVG were evaluated as part of the ‘total’ projections by using a playing-time weighted value indicated by an ‘n’ (e.g. “nAVG”) to convert them to counting stats.

• The SB (*) correlations were evaluated on the basis of two separate populations and averaged: players who stole >=5 bases, and those who stole <5. Past analysis has shown that evaluating this as a single population gives undue credit for projecting the low-steal players, and not enough credit for accurately projecting high-steal players.

• The IP (^) projections were evaluated on the population of SP’s only. RP inning projections show nearly zero correlation and introduce undue noise.

• The W and SO (*) projections were evaluated on the basis of two separate populations and averaged: starters and relievers. Past analysis has shown that evaluating this as a single population gives undue credit for projecting the separation between these two populations.

• The SV (**) projections were evaluated for relievers only.

• The Big Board mix 2020:

  • Hitter Playing Time: 68% Steamer, 32% ATC

  • Hitter Rate Stats: 40% Steamer, 50% ATC, 10% ZiPS

  • Pitcher Playing Time: 33% Steamer, 23% ATC, 44% Razzball

  • Pitcher Rate Stats: 22% Steamer, 30% ATC, 48% Razzball

• The Big Board mix 2022:

  • Hitter Playing Time: 83% Steamer, 17% ATC

  • Hitter Rate Stats: 80% The Bat X, 20% ATC

  • Pitcher Playing Time: 20% Steamer, 17% ATC, 63% Razzball

  • Pitcher Rate Stats: 35% ATC, 49% Razzball, 16% The Bat