
Ask just about any educator what will be most difficult for someone new to teaching, and class management is usually top of the list. It’s a big skill bucket, including intuitive class awareness and action selection in the moment, but just as importantly the structural work, the class ethos, and the pedagogical preparation that keeps difficult situations from bubbling up.
A huge part of what class management becomes as a career progresses is the recognition of patterns in situations, which cues are the best tells, whether this is like a prior situation or unique, and what actions best mitigate the issue. It’s a whole bunch of intuitive judgments. It’s pattern analysis.
Most of my focus is on how to build intuitive judgment skill, and one of the big issues there is what the measures will be. How can we possibly measure a skill so squishy?
In the professional world, these hard-to-grasp qualities are captured in qualitative feedback. In academic circles though, it’s not evidence until it’s made quantitative, and even then the measurement instruments must be precise.
A few generations have been trained to say that word with a faint apology, as if qualitative work were the anecdotal opening act and quantitative analysis the main event, arriving to make things respectable.
Why do we treat it that way? The answer isn’t really about truth. AI is starting to shift the tradeoffs.
Why the pattern mode isn’t the lesser one
Class management isn’t an unusual case. Information in the world comes tangled. Teachers read a student’s perceived level of understanding in the relationship among facial and body language, words and tone, student history, and comparisons with other students. Not in any one of them, but all of them. The same holds for a nurse recognizing whether a patient needs attention, how to handle a souring negotiation, or reading dishonesty in a page of writing without explicit hints. The meaning sits in how the signals lean together, and each signal is faint and half-dependent on the rest.
To reason logically over evidence like that, you have to cut the tangle first. You isolate a variable, fix a definition, hold the rest still, and strip away what you’re calling noise. That’s what deliberation is, the mode of symbol and rule and shown steps that school teaches and society rewards most. In slicing it up, it refuses to let in the hunch, the unease, or the sense that something’s off. Letting vague signals vote is what deliberation was built to prevent. But the pattern that lives in the qualitative carries the interrelationships among the signals, and those relationships hold much of the meaning.
What quantitative analysis gains in precision and reproducibility it gives up in validity and completeness. And often what it measures is a weak proxy for what we’re trying to capture. (Validity here refers to construct validity, whether the measure captures what it claims to care about.)
Both modes are necessary, and I’m not smuggling in an argument for one swallowing the other. The point is narrower. We’ve treated the cutting mode as thinking proper and the pattern mode as its unreliable cousin, when they’re two ways of handling different evidence, and only one of them keeps the interconnection intact.
Why the number won, and what it missed
So we had two real ways of knowing, and we built almost everything on one of them. The reason was logistics, not accuracy. A number travels. Teachers’ reads live in their heads, mean little to a stranger, can’t be lined up against a thousand other cases, and won’t survive an appeal. Some numerical metric does all of that at once.
Anything that had to move between people or stand up to a challenge got pushed through the reductive gate, because the rich version couldn’t make the trip. But we mistook those virtues for an exclusive claim about truth. Because the number could travel and the pattern couldn’t, we drifted into treating the number as the more real of the two, and the read as a soft impression waiting for the numbers to make it respectable.
The cost of that shows up most clearly when the reduction is done well and still misleads. Lee Cronbach spent his career giving quantitative measurement much of its rigor, then spent years documenting where it breaks. He argued that the interactions pile up fast enough that the results rarely survive the trip out of the study (Cronbach, 1975). The measurement is precise, but often for the wrong target.
None of this makes pattern reading safe by default. Daniel Kahneman and Gary Klein spent years on opposite sides of this question, one cataloguing how intuition misleads, the other how expert intuition performs (Kahneman and Klein, 2009). Luckily for us, they also worked together to catalogue where they agreed. Pattern reading earns trust under two conditions: an environment regular enough to hold real signals, and holistic practice with quick, honest feedback. The teacher reading her room has both. A pundit forecasting next year’s politics has neither. No stable environment given how much can happen before an election, and no way to get timely feedback. Qualitative analysis isn’t a license to trust a gut.
When to reach for which
The number of students in a class who can solve a particular kind of equation is both precise and true. Cutting the question down to something countable cost you nothing, because solving that equation was always a clean, separable thing. Ask instead whether those same students are becoming mathematical thinkers, and the moment you reduce it to a countable proxy, a test score or a rubric tally, you’ve traded the question for a more convenient one and quietly lost most of what you cared about.
The rule sits in the difference between those two questions. Does reducing something to isolated, countable parts preserve it, or replace it? When reduction preserves the thing, quantify. But most of what we actually want to know about people, whether they understand, whether they’re well, whether a classroom or a team or a ward is healthy, sits on the irreducible side.
Picture precision and validity as two axes, as in the figure below. The invalid and imprecise corner is nowhere we want to be. We’re good at the corner where both run high, and we’ve piled up a lot in the corner where precision is high and validity is low, the precisely-wrong numbers I’ve been describing. The fourth corner–valid but imprecise–we’ve never learned to trust at scale.
Think about judging whether a new product will land. There’s no honest precise number for that, and a confident seventy-three percent is the precisely-wrong kind. But an experienced team can weigh the whole picture, the market, the timing, the competition, the feel of the first customers, and label the risks and opportunities red, yellow, or green, and that coarse read is valid in a way the percentage never was.
We’ve rarely trusted judgments like that at scale, because they lived in individual heads and didn’t travel. AI changes that. It can produce a red, yellow, or green read across thousands of products the way a seasoned reviewer would, and it lets you pick the resolution that matches the decision instead of forcing a false precision. The low precision isn’t a failure here. It’s an honest match to how much certainty the situation actually holds. That empty region is the interesting one, and it’s starting to fill in.
The ground is shifting some
AI is changing the balance, and several of qualitative analysis’s handicaps are loosening at once.
The biggest is labor. Qualitative analysis required people to hand-code the data, so studies stayed small and were easy targets for suspicion. Some of it could be automated before ChatGPT, but characterizing a qualitative aspect of a body of text was error-prone until generative AI.
Rich characterization can now be done at a scale that used to be impossible by hand. And scale changes what fuzziness costs you. A single qualitative read is noisier than a single number, but that noise averages out across many reads, so you can buy precision back at the level of the conclusion you care about instead of demanding it of every observation. Fuzzy per case, sharp in the aggregate, which is how good measurement has always worked.
That of course depends on a qualitative read without unwanted bias. Volume reduces noise, but averaging doesn’t remove bias (the deviation from the wanted target).
Which brings up the other big question. Can we even agree on the qualitative? Getting people to agree on what’s being characterized, and to read it the same way, is a genuine and half-solved problem, and any wobble in the target ripples downstream. Even so, it’s likely to be more valid than a single number. Two musicians will agree more about which songs are creative than any metric will manage to capture about song creativity.
There’s also a good chance AI will teach us how to reach closer agreement, or to present results that fit our individual sensibilities. Mental health, for instance, sorts brains into defined diagnostic buckets, and clinicians argue about where to draw the boundaries. AI’s ability to analyze data at scale might tell us those aren’t the right lines, or that other ones carry more meaning, sharpening our shared characterizations while still allowing for individual variability. Plus, since AI analysis can be run as many times as needed, it could slice the same material many ways to offer more individualized fit to our mindsets and needs.
We are also getting better at capturing patterns mathematically. The main way we’ve had to capture pattern-based meaning in a form other people could examine was to put it into language, and language was always a lossy, hard-to-compare container. We now have ways to turn language and meaning into numbers, which may make it possible to combine qualitative and quantitative analysis in some hybrid way. I think we’re going to see a methodological explosion.
Not everyone is glad about AI qualitative analysis, and the pushback is already loud. What’s telling is that it comes from both directions, making opposite points.
From the qualitative side, 419 experienced researchers from 32 countries recently signed an open letter rejecting generative AI for qualitative work, essentially arguing that it can’t do as well as a human in making the assessment. With today’s AI they’re probably correct, but the question is whether it’s a better approximation than turning it into something quantitative, because being able to scale the analysis is a huge advantage to overlook.
From the quantitative side, the complaint is the mirror image. AI-driven characterization isn’t rigorous enough (e.g., precise and reproducible). To them the trouble is that it’s too loose to count as measurement at all.
Tighten the method to answer the quantitative worry, standardize it, make it consistent and scalable, and you’ve done the very thing that horrifies the qualitative camp. Keep it loose and interpretive to honor the qualitative tradition, and you’ve confirmed the quantitative complaint.
Each side is guarding the border of its own discipline, and AI is unsettling because it walks straight through the middle. This unsettling feeling isn’t only affecting researchers. Anyone who has considered getting AI to help grade anything that has qualitative success measures, or who has thought hard about whether the questions posed on exams really capture what’s most important has had to grapple with the downsides.
There are only situational answers to this tradeoff. Qualitative analysis might be applied to attempt to characterize at least longer-term trends in amorphous skills, like the durable, 21st-Century skills so widely touted. Whether that’s a good idea is situational. Rating a student’s “creativity” across a body of work is probably a really bad idea if determined by AI and used as a grade. But tracking whether creativity demonstrated in a subject is better or worse than last year, for the school and teacher’s benefit, may have a great deal of value. An imperfect measure is better than no measure.
The constraints have changed big time. We should be thinking about the opportunities it offers.
©2026 Dasey Consulting LLC



