Measuring academic ability, teacher retention, and student learning

A recent post on Dangerously Irrelevant (cross-posted to LeaderTalk) prompted me to think quite a bit about how academic ability is measured...and how teacher academic ability measurements correlate with measurements of student academic ability. I think the conclusions are built on pretty soft data ... just because there's a correlation between teacher college entry exam scores and teacher attrition doesn't necessarily mean that there's a "brain drain" from the classroom, and just because there's a correlation between those same types of teacher scores and student scores doesn't mean that those are necessarily the most academically-able teachers or students. The large and important point here is that the measurement of academic ability is difficult, and current instruments are limited in scope - so, although we should certainly support the retention of our best teachers, we shouldn't be satisfied with building our arguments on soft data.

The following are some comments that I made on the post at D.I. ... I'm re-posting them here for my own archival purposes and because the topic of measuring academic ability is clearly within the domain of cognitive psychology. As you can tell, I'm responding to other comments, which I don't feel comfortable re-posting here because they're not mine -- please visit the above link to the post at D.I. to read the full article and ensuing discussion -- it's very interesting.

Comment 1:
I'm on-board with the idea that smarter teachers are going to be the best at promoting student learning. But I'd like to echo and add to Orenta's point above .. if we're railing against using standardized tests as such an important measure of current student learning, how can we maintain the integrity of our argument by claiming that the same type of testing is an important and valid measure for teachers' knowledge? Though anecdotal, I know plenty of people who've declined in intelligence / knowledge / learning capacity after the structural support of family and high school settings are gone, which can have a strong positive influence on college entrance exam scores. Also, I know plenty of people who have developed incredibly as learners both in college and post-college, particularly in the context of teaching others. So, again, I'm all for getting the smartest teachers possible in the classroom, and finding ways to keep them there - but I think it's a bit disingenuous, for multiple reasons, to use college entrance scores on standardized exams to make the point.

Comment 2:
I agree that there's an important and significant role for standardized test scores in the interpretation of student and teacher intelligence. However, I am uncomfortable with your generalized statement - if based only on the citations from Anderson & Carroll, and Guarino et al - that "the percentage of teachers with lower academic ability increases in schools over time. The brightest go elsewhere." and your stated assumption #1 "smart people are less likely to stay in teaching (thus resulting in a concentration of teachers with lower academic ability)." As I said originally, I absolutely support the notion that we should make more effort to retain our brightest teachers; I stand by my claim that scores on standardized tests taken in high school, or even at the end of a college program, by individuals who then become teachers are not the best data to use when making the argument that there is a longitudinal "brain drain" from the classroom. While there may be a correlation between this particular teacher characteristic and student achievement, I hesitate to make the jump into causality, as do Wayne & Youngs: "When statistical methods seem to establish that a particular quality indicator influences student achievement, readers still must draw conclusions cautiously. Theory generates alternative explanations that statistical methods must reject, so a positive finding is only as strong as the theory undergirding the analysis. If the theory is incomplete—or data on the plausible determinants of student achievement are incomplete—the untheorized or unavailable determinants of student achievement could potentially correlate with the teacher quality variable (i.e., correlation between the error term and the teacher quality variable). Thus, student achievement differences that appear connected to teacher qualifications might in truth originate in omitted variables." Further, in the section of their article specific to the review of studies on teacher test scores and student achievement, Wayne & Youngs point out that none of the teacher tests used in those studies are still in use, and emphasize the importance of researching the correlation between student achievement and teacher performance on assessments of their skill beyond standardized tests. My larger point, which I attempted to make by providing anecdotal evidence, is that the teachers who do remain in education for longer periods of time are not necessarily less able to promote student learning, even if there is a correlation that points to their tendency to have scored lower on standardized tests prior to their entry into college and/or the classroom. With that said, I would certainly support hiring policy shifts toward selecting applicants with the highest academic credentials possible, including historical and more recent scores on standardized assessments.

Comment 3:
The older the data, the more inappropriate it is as a measure of academic ability. This is well-accepted with IQ tests, for example - the score is compared with others in the same age-range, not with the entire population. With that reasoning in mind, I think it's a bit provocative to use college entrance exam scores as the only data you show in your post about the "brain drain" from the classroom. In my read of the articles you cite, I interpret the authors as being far more reserved in their interpretation of the available data, noting limitations in data availability and suggesting caution in forming inferences based on it. For example, in the Anderson & Carroll DOE study they show that teachers are more likely to earn a graduate degree than non-teachers, and that teachers with a graduate degree are less likely to leave the profession than those with an undergraduate degree. This, to me, is great support for questioning the veracity of the relationship between college entry exam scores and teacher attrition, and exemplifies the significant personal, formal, and professional learning that occurs in college and within the first few years of experience in teaching (and all "leavers" in that study had at least one year of experience in the profession). I suppose that I'm being picky about this data because I worry about the impact on current and prospective teachers from this type of provocation. I don't think anyone in any profession would want either of these two possibilities: 1 - to think that their academic ability is being reduced to and summarized by a standardized test score (SAT or ACT) they earned in high school; 2 - to think that the longer they stay in the profession, the more "less-academically-able" people they'll be working with. Especially with regard to point #2, although it *may* be true, I think we're obliged to use better data to make such a negative critique. My further problem with the use of this data to demonstrate a "brain drain" from the classroom is that there is *so* *much* *more* to being an effective teacher than the (limited) aspects of academic ability indicated by scores on the SAT, ACT, college selectivity, college GPA, Praxis I, Praxis II, or an IQ test. For example, even beyond subject-area expertise, what about being creative and being able to help others be creative? What about the ability to collaborate and to help others to collaborate? What about technology skills and the ability to help others to use technology? My impression is that we're hoping for these skills to manifest more and more in the classroom over time - but none of them are measured by the above-named instruments ... and we're not citing research in the "brain drain" question that attempts to measure them actively (let alone current academic ability), or to correlate measurements of those important "21st Century Skills" with measures of academic ability (whether old or current). I am an ardent advocate of data-driven policy development, decision making, and education research, but I think we need to be really careful - more so than in this post - when we're using data to make a point that is critical of those who are currently in the teaching profession and those who are about to enter it. Certainly this provocative writing has herein inspired productive discussion - but I must admit that I have some concern that using what I would describe as very suspect data to make what may in fact be a valid point may actually serve to exacerbate the very problem that you're writing about and that I think we're all working to prevent: good teachers leaving the classroom.