In a study of nearly 1,400 eighth-graders in the Boston public school system, the researchers found that some schools have successfully raised their students’ scores on the Massachusetts Comprehensive Assessment System (MCAS). However, those schools had almost no effect on students’ performance on tests of fluid intelligence skills, such as working memory capacity, speed of information processing, and ability to solve abstract problems.
“Our original question was this: If you have a school that’s effectively helping kids from lower socioeconomic environments by moving up their scores and improving their chances to go to college, then are those changes accompanied by gains in additional cognitive skills?” says John Gabrieli, the Grover M. Hermann Professor of Health Sciences and Technology, professor of brain and cognitive sciences, and senior author of a forthcoming Psychological Science paper describing the findings.
Instead, the researchers found that educational practices designed to raise knowledge and boost test scores do not improve fluid intelligence. “It doesn’t seem like you get these skills for free in the way that you might hope, just by doing a lot of studying and being a good student,” says Gabrieli, who is also a member of MIT’s McGovern Institute for Brain Research.
Gabrieli's point is key to understanding this. It doesn't show that the MCAS, or schooling as it is currently constructed, demonstrates less than it claims to, but that it doesn't give you more "for free."
I suspect this is highly relevant for understanding the 11th grade math NECAP scores, though. That test certainly seems to emphasize "fluid intelligence skills" as they're described in the article, which on one hand is admirable if that's what you really want kids to be able to do, but on the other hand, we genuinely don't seem to know how to teach those skills in school.
If you did have a test that emphasized skills that schooling doesn't currently affect, you'd get scores that not only were overwhelmingly based on out of school factors, even compared to other standardized tests, but that would remain remarkably resistant to any and all common interventions. As is the case with 11th grade NECAP math scores.