PairWise (Updated)

Pairwise.

It’s coming.

Changes the way we select Pool C teams.

It’s coming.

Pairwise.

That is basically what I got when talking to some administrators and coaches this past year. It would get stuffed into an email where we were talking about selection and then there would be a line that said, “Of course, this all goes away with Pairwise.” The problem is that I had no clue what Pairwise was and how it would change the selection game. This past week I reached out to a bunch of coaches and fortunately one of them had seen an NCAA Presentation on Pairwise (also known as the Selection Criteria Database). The presentation is imbedded below. I’ve gone through the presentation and here are my thoughts. After all, they were asking for feedback. (Note – I added an updated section at the end after finding more information after publishing this article. The article was not changed.)

In summary, Pairwise is a computer model that uses criteria and compares every team with every other team. You get a point for each criterion you win and the most points wins the matchup giving you an overall win against that team. Now do this for the 400+ other teams in volleyball and you get an overall record. Since everyone has a record, we can now put every team into a standing or ranking and simply take the top teams that aren’t getting an automatic qualifier as our Pool C teams.

The important thing to remember with anything done on a computer is that garbage in equals garbage out. So, the most important aspect of Pairwise will be the criteria it uses to do their comparisons. Will it be garbage? Don’t know, but we do know that the criteria will be different than today. One aspect is that regionally ranked record goes away. Winning percentage stays as does common opponent and head-to-head. Strength of Schedule gets rid of the Opponent’s Opponent’s Winning Percentage (OOWP) that is used today because you are being compared against every team so it becomes redundant. In other words, SOS is just your opponent’s winning percentage. It wasn’t talked about, but there was an indication that head-to-head is weighted. That probably means home and away and maybe number of sets factor in? Common opponent also changes, I think, based on what I read from a hockey article. Today, we simply look at the wins and losses against common opponents. Hockey (which has been piloting PairWise) takes the winning percentage against common opponents and adds them up. So, if Team Alpha is 3-1 and Team Bravo is 3-2 it doesn’t mean Team Alpha gets the advantage. For instance, if Alpha beat Team Charlie 3 times (3-0 or 1.000) and lost to Team Delta once (0-1 or 0.000), they get a 3-1 record but their common opponent score would be 1.000. If Team Bravo went 2-1 (0.667) against Charlie and 1-1 (0.500) against Delta then they would finish 1.167 and take the point. Again, no mention of how common opponent was done in the presentation. I believe that’s the criteria for now. Secondary criteria also goes away. RPI, as a tool, goes away. Human interaction over the data, goes away.

What we are left with is a standings table that looks like the following image:

This is a screenshot from the presentation that shows a simulation using the 2021 data right before selection. The teams with an asterisk were teams that received an automatic qualifier. Teams without one would have received an at-large bid if Pairwise had been used. Those teams start with Hope. One thing to point out with this simulation is that they used a 30/70% split on record and SOS meaning (I believe) they intended to weight SOS more than record. In fact, the tool or setting was broken and they weighted record more than SOS. What’s funny is that no one caught this. (I caught it when they paged down the list and I saw that Plymouth State, UC Santa Cruz and Mary Hardin-Baylor would have received bids in 2021. All were teams with good records but bad to average SOS numbers. Teams that received a bid that were left out were Chicago, Ithaca and UW-Stevens Point. All teams with good to great SOS numbers.) Oh, they also only listed 19 Pool C teams when in 2021 there were 20. (Seriously, my consulting fees are very reasonable.)

Back to the image and we see the 2021 DIII records, Conference rank, PairWise record, NPI (more on this later), vAbove, vBelow and OW-OL. No mention was really made of most of this but PairWise record is the record after all the comparisons were made. Johns Hopkins won every comparison. vAbove means the record against teams above you in the rankings. vBelow means the record against teams below you in the rankings. OW-OL means your record in 5-set matches. NPI was never fully explained or the acronym stated. I believe it means No Phreaking Idea (NPI) but I may be off. NPI is huge because that is where your rank comes from. In the screenshot, the record and NPI are ordered exactly the same but I’m not sure if that is always the case. John Hopkins (or Wartburg) didn’t have a perfect 100 NPI so we know that NPI isn’t solely based off of record. The presentation talked about “win bonus” and how if you beat a team ranked higher you get a bonus related to how far ahead they were. Since, PairWise is dynamic and changes with every match played, that bonus will change throughout the season. (Not stated but seems intuitive.) It’s also kind of obvious that 5-set wins and losses are counted differently. They may also count home and away results differently as I saw parameters for that. In the presentation, they called these parameters “dials” that could be changed by the committees. The 30/70% was another dial.

I think that’s a good overview. Here were some other notes I wrote down while watching the presentation (items in italics are my comments):

• Name will either be PairWise or Selection Criteria Database. This depends on getting the rights to PairWise.
• PairWise is being piloted in Ice Hockey where it was stated that it has proven effective.
• PairWise eliminates two problems in the current system:
  • There may not be a consistent application of the criteria.
  • There is currently no transparency.
• PairWise has no penalty for beating bad teams. (It will still impact your SOS so I wasn’t too sure about this.)
• PairWise takes into account where teams you beat are ranked with NPI. Regionally ranked wins in our current system, for instance, count one win for beating a team ranked first or seventh.
• Mentioned that PairWise eliminates the West Coast teams being on an island because beating bad teams doesn’t hurt you. (Again, the problem is still SOS and not being able to reach what teams can do in larger regions.)
• PairWise allows the committee to weight criteria. The only criteria that I saw weighted was record versus SOS. (I believe that is the only thing you can weight.)
• They ran simulations against past years and claimed that the results were similar. In cases where they were not, they could see why they were different. (But they couldn’t tell their 30/70% wasn’t working?)
• They have tried PairWise on six different sports to gain an understanding of their uniqueness. Women’s volleyball was one of the sports used.

The presentation also discussed where they were in the process. I believe the presentation was from the middle of last year. They talked about getting this information out there and receiving feedback. I was very surprised when I talked to a number of a coaches and only one knew of this presentation. I’m not sure who they are getting feedback from if only one coach knew about this. They mentioned that the February 2024 meeting of the Championship Committee might vote on using PairWise for the 2024 season. The current selection criteria is covered by Bylaw 31, which comes under their authority. This means a change to PairWise doesn’t need full membership approval. One problem with getting approval is that they would have to buy the rights to PairWise. The idea is that PairWise would be used for all sports although Tennis was mentioned a couple of times as being an issue. DIII is the only division looking at using PairWise for selection of their sports.

I have linked the presentation above so anyone that wants to can view it. It takes about an hour but I hope this summary has helped the lazier of you in the crowd.

What are my thoughts? I still don’t quite understand how it works and how it assigns points to the very important NPI. I would need to see that calculation but whatever it turns out to be, I’m very concerned with no expert oversight of the data when it comes to Pool C selection. Someone asked a question about this and the commentator said it could be done but it opens up a problem with bias and why a change was made that overturned PairWise. I see the point but what if there is an obvious team left out. I mean look at their example they ran. If those were the dials used, and I don’t think they would be, shouldn’t there be a voice that says maybe a team with a sub 0.500 SOS shouldn’t receive an at-large bid? I guess this could be tweaked during the season when the problem is noticed but I also don’t like the thought of changing the dials to get the result you want. Maybe for the first year or two there needs to be oversight? At that point, the tool is discarded or flies on its own. I’d like to see an example PairWise comparison between two teams from their data. I want to see the scorecard and make no mistake there will be a scorecard now. It’s not good enough to taste the cake, we need a detailed look at the ingredients. A properly poisoned cake will still taste just fine. Just ask my first (not-so-hot) wife. Whoops, never mind that last part. I’d also like to see this run in trial mode the first year. Not used but the data presented to everyone. I don’t think it’s fair to programs who have already scheduled to have a new selection process given to them. I guess the big thing I don’t like about this, and it comes back to no human oversight, is that there was a belief in the presentation that this solves the West Coast Island issue. It doesn’t. We just had a selection process where La Verne got in with a 0.541 SOS. Humans had to look at that and say it’s because the West Coast is different. PairWise isn’t going to do that. The logic from the presentation is that this is because there are no bad losses anymore. It still drags down SOS and that’s a West Coast Island problem. On the plus side, it does get rid of regionally rankings, which also hurt the West. I need someone to carefully explain to me how PairWise fixes the West Coast Island issue. Finally, I’m concerned about the makeup of the data with volleyball. We don’t have a lot of cross-pollination between regions like Ice Hockey does. Most West versus East comparisons will come down to record versus SOS and we know the West is already down 1-0. There are typically no head-to-head results or common opponent results. If there are not enough data points then it’s the same as feeding in garbage and we know what will result from that. I will say I do love the transparency. I like seeing the true rankings (think of them as a not-so-regional rankings) all season. No surprises even if you disagree with the ingredients. For the most part, the Selection Committee and Regional Advisory Committees go away with respect to rankings and have more time for other tasks. One big task is bracketing the NCAA Tournament. With this level of transparency, the work on that could start weeks in advance as you see how things are shaking out. Bottom line, I don’t feel comfortable with the limited information I have on PairWise implementing this immediately. I can’t believe coaches who are probably learning about PairWise by reading this article would say any different.

Pairwise.

It’s coming.

Changes the way we select Pool C teams.

It’s coming.

Pairwise.

Uh, maybe we should do a better job getting this information out to the coaches, first?

UPDATED!!!

The DIII Championship Committee had additional information from their December meeting. I only saw this after publishing the article. It changes some things written above.

• The Championship Committee stated they needed all feedback by June to make a decision.
• It was indicated that a change to PairWise would take legislative action by the Management Council.
• If adopted, the window for making any adjustments to the dials would be periodic. The example given was every two-years and that those adjustments would require the Championships Committee’s approval.

There is an attachment to that document with some answers to frequently asked questions. The interesting one is that most of the current criteria would go away, which is different than what I had written. Winning percentage and SOS would stay but, in addition to regionally ranked record, both common opponent and head-to-head would go away.

The new criteria would be associated (I think) with SOS. It includes home/away multiplier, quality win bonus (QWB) and overtime results (if applicable). The home/away multiplier could be used to increase the weight of away wins and home losses. QWB provides a bonus for wins against top teams. Overtime wins/losses might apply to 5-set matches in volleyball. They wouldn’t be considered full wins or full losses.

Another point mentioned this, “Another benefit of the database is that teams are not penalized for winning a contest that it should win. For example, in the current system if a strong team plays a weaker team its strength-of-schedule will be negatively impacted even though it won the contest – as it should. In the selection criteria database, the team that wins a contest against a lesser opponent will not be statistically impacted by winning a contest it should win. This will greatly benefit stronger teams in weaker conferences.”

I’m still very confused by this if SOS is being used. My only guess is that the SOS calculation will be different than a simple winning percentage of your opponents.

NPI stands for NCAA Performance Indicator!!!!

To ask additional questions or to provide feedback, please contact Laura Peterson-Mlynski (lpeterson@ncaa.org) or JP Williams (jpwilliams@ncaa.org).