Number Theory - Setter Evaluation with Josh Steinbach (part 3)
Sometimes there isn’t an answer, just asking better questions
This is part 3 of the series. You can read previous parts here:
What is Number Theory? - Read here
There’s a common thread running through each effort to measure setter ability, proxies. The difficulties Josh has experienced in his efforts to quantify setting is less about his knowledge or skill and much more about the limitations of direct measurement. When it comes to “soft skills”, researchers have always struggled to measure behaviors that are highly overlapping at best and completely ambiguous at worst. Instead of giving up, they resort to proxy variables, which are things that can be measured that are, hopefully, closely related to and highly correlated with the behavior they actually want to measure. While using proxies is better than not measuring anything, it is important for coaches to remember they’re actually measuring almost the traits they are interested in and not the traits themselves. The biggest danger of forgetting your use of proxies is the Cobra Effect, which I’ve written about elsewhere. But there’s a larger lesson to take away from using a lot of proxies in one measurement.
It’s one thing to use a proxy or two in a large model, it’s something else to have a model composed of mostly proxies. Because of that difference, I suggested to Josh that he might be better off keeping the traits of setting he wants to measure separate instead of making one big equation. I said that because I think there’s a difference between designing metrics for decision makers and designing them for general audiences. The all-inclusive metrics I mentioned previously (PECOTA and CARMELO) were designed for comparing players across their entire leagues and they were designed for the public to do things like draft fantasy teams or answer pub discussions about “the greatest”. During our discussion, we realized that, while Josh may have a passing interest in who’s the best setter in the country, he has much more proximal concerns, like who’s the best setter in his gym. We said that for coaches like Dan Fisher, Dani Busboom Kelly, or Katie Schumacher-Cawley, those questions may be the same but most of their fellow NCAA women’s indoor volleyball coaches need to focus on the latter question. If that is the case, then there will be certain factors that will be more important than others. It becomes important for coaches to recognize which factors they want to focus on.
As I have written elsewhere, the frames through which you view the world affect the problems you set. I asked Josh about the problems he wanted to set for himself this year for setters. He talked about having three setters when practice begins in the fall, one returning player, one graduate transfer, and one first-year player. He talked mostly about differences he thought might exist between them in athleticism and game sense, but also about how their game sense might influence their location. This brought us back to the question he was asking himself near the beginning of the our conversation, how does he ensure he’s choosing the best setter(s) to start? But now, instead of trying to answer the big question of how to measure setting completely and for everyone, we were trying to answer a much more applied question. This focus clarified our discussion very nicely.
Not only had we determined which factors were most important to him, but we could ask important follow-up questions that would shape his practice planning as well as his evaluation. I invited him to think about different aspects of setting ability as different dials that could be turned to influence the output he could measure. Of all the dials available, he was most interested in the three mentioned above (which could even be broken up further into more fine-grained dials). Which of those dials did he think he could turn the most? Alternatively, which dials did he think would yield the biggest changes with the smallest amount of turning? Crucially, which dials would yield changes in the time he and the players will have to work together?
Another series of questions should be asked before getting into measurement. Once Josh decides on which aspects of skill he wants to pay attention to, he should set some parameters on those skills. What’s the spectrum any one of those aspects exists on? What I’m asking is for Josh to determine three points on the spectrum, the lower end (below average), the upper end (above average or excellent), and the point in between where the players are located when practice begins. The next question is what does he expect improvement to look like relative to that spectrum? He is interested in where the players begin insofar as that knowledge will allow him to track progress. If he has an idea of where the players are starting and the direction in which he expects them to move, then he can identify milestones of progress and track how the players are changing over time. Identifying such milestones and defining how to measure them is a much more manageable problem for a performance analyst. If a decision maker like Josh and an analyst like me can reach that point, then we give ourselves a good chance at answering Josh’s driving question of who should start in the fall. We might even make some progress towards getting Josh his big equation after all.
Have questions for Josh? You can ask me here or you can email him directly.