I’ve already read the article. I considered sharing it here.
There’s an article that critiques this study:
The study had poor sampling. That 49 students did so poorly seems very significant to me nevertheless. Well that depends on how many English majors there were in total at the two universities. If these students were the bottom 10th percentile then that’s significant. If they were all bottom 5th percentile maybe it doesn’t say much. But even so, I would expect more from anyone taking an English major.
I asked Claude AI to estimate how many English majors there could be in the two largest regional universities in Kansas and it estimated the upper bound at 408. So that’s above 10th percentile.
But here’s the kicker: they wanted to test reading skill, and they made no effort to screen for whether the students they sampled actually read books.
It would be useful to have this categorization. But if it was found that most English majors don’t read books that in itself would be a problematic discovery (but not very surprising maybe?) and hint at low reading skill.
There are several serious problems with the method used in this study, even if you take their goals at face value. First, not everyone is comfortable reading aloud, especially in a high-pressure one-on-one setting with a stranger holding a clipboard. Reading aloud is a performance skill, not just a literacy skill. It layers on anxiety, self-consciousness, and the distraction of hearing your own voice. Any or all of these factors can impair comprehension. If a student stumbles over words or loses their place while reading out loud, that doesn’t necessarily mean they don’t understand the text. It might just mean they’re nervous.
Sure the method isn’t perfect, but I think it’s good enough to learn a lot about their reading skill. She’s saying the method isn’t similar enough to count at all. If you have high reading skill then you should be able to better handle outside pressures like these. I don’t think these issues would be enough to drop most people from “proficient” or mastery to “problematic”.
My evaluation is that we can’t assume 58% of English majors read at the “problematic” level, but the results are still significant and less than what should be expected from English majors.
An aside: the article explicitly pronounces infallibilist justificationist weighted epistemology:
A replicated study with a large, representative sample, rigorous design, double-blind controls, and statistically significant results should be weighed far more heavily than a non-replicated study with a small, narrow sample and weaker methodology.
The former can prove something.
The latter can, at best, suggest something.