Is school improvement a myth?
In the discussions of reauthorizing NCLB almost no one is talking about whether the school is the right unit of accountability. And that’s because we continue to assume that schools are the problem and they can be fixed. But what if school improvement was found to be a myth?
—-
It’s really hard to convince people that something as evidence-based as “school improvement” is today, largely a myth. After all, from the earliest studies of effective schools to research about Chicago school reform, school improvement as an idea and a policy is based on decades of research and practice that tell us that what happens inside schools matters for student achievement, and that people working inside schools can change what happens in a way that improves student learning.
But there’s a problem with that evidence. A problem I discovered when I wrote my dissertation, “The Myth of School Improvement.”
The problem is that there are two hidden variables in the research about “effective schools” and “improving schools” that were somehow missed in the rush to create policy to hold “schools” accountable.
Variable 1: Context stability
The first variable is something that I call context stability. Context stability is a combination of low teacher turnover, stable leadership, and a demographically consistent student population. Context stability is also about having continuity in curriculum and materials, programs and program staff from year to year –or something that researchers studying Chicago schools called, “coherence.” If you dig deep into the research on effective and improving schools you find out that all of them had continuity in staff, leadership and student demographics during the period studied. Staff and leadership stability was a condition of effectiveness.
Anyone who works in schools today can tell you that context stability is very uncommon, especially in schools deemed “in need of improvement.” Teacher turnover is an ongoing problem, particularly in schools serving large percentages of students living in poverty where the average teacher stays less than five years. And ironically, the federal School Improvement grants have convinced many district administrators that it’s a good idea to move school principals around. And in many locales, particularly urban ones with open enrollment policies and large immigrant populations, student demographics can change dramatically from year to year.
And the real rub is that context stability itself doesn’t last forever. Most research about effective or improving schools is done in a 1-5 year period. Give me an effective school or improving school and wait three years. The effective principal or effective program will have gone, and it’ll be back to square one.
Variable 2: Autonomy
The other important variable we failed to consider is autonomy. During the previous eras of school reform people working in schools had much more control over their curricula, their technology and their programs than they did today. The research on Chicago’s improving schools was conducted during the 1990s during an unprecedented experiment in local school control. Nowadays districts and states often dictate what materials teachers can use, what programs they can implement, and even what page to be on in a pacing guide. Some researchers say that schools should be responsible for “crafting coherence” but in my experience, that’s more pie in the sky idealism than reality, particularly when district-school administrator power dynamics are involved.
If you really think about it, schools are just buildings that have a constant and complicated flow of policies, programs and people moving in and out. School administrators and teachers have very little control of that flow of information, people and practices—they can only manage those things within the confines of district, state and federal policies.
It’s not surprising we missed this enormous problem when developing the logic of school improvement. I blame my field: We take for granted that a school is a relevant “thing” because we once believed the key to educational improvement could be found in organizational science and management. Those of us at Stanford–with our offices right down the hall from organization scientists– followed right in the footsteps of the organizational development field. And because of this, the field anthropomorphized the school building as an actor. We started writing about characteristics and qualities of “schools” so that leadership and professional learning communities became properties of the school organization, and not the dynamic interplay of the adults in the building.
Think about it: the idea that we hold a pile of bricks responsible for improvement is really quite nonsensical.
Now many policymakers might say, “yes, I agree, it’s about teaching and learning—it’s about the teacher, that’s why we’re holding teachers accountable.
But if schools have very, very little control over the people, policies and programs at their doorstep, consider that teachers have absolutely none. Is it really fair to blame Ms. Johnson for the failings of a poorly designed curriculum? For mistakes in the common core standards? Is it right to hold Mr. Hopkins accountable for low test scores because the ADHD kid went off his meds during a critical week of lessons just at the very same time that the school counselor went on maternity leave?
I think most people would agree that it’s not fair. And while the idea of value added modeling may be “evidence-based” that evidence was analyzed much in the same way as the analysis that gave rise to school improvement policy: correlational analyses of variables measured at one moment in time. We know value added scores lack stability– and if you think about what happens in classrooms over time–with the flow of people, policies and programs–it’s pretty obvious to me why. I don’t fault the economists for not getting this. They haven’t spent enough time in schools to understand–they’ve never had the chance to teach a group of diverse children in an under-resourced school.
My heart aches for kids who receive a substandard education. I believe that much of what happens inside schools needs to change. But the only way we will make great strides in this area is to give up on the myths that have led us to that point– to be really precise and logical about what it is that we are trying to improve and how to measure it.
In 2015, educational improvement is a problem that belongs to the entire field and institution. It’s a problem that we all own, as tax-payers, citizens and workers, as parents, teachers and youth. The policy remedies for holding a whole field accountable are complex, but I believe in democracy. We can do this.
Please Facebook Like, Tweet, etc below and/or reblog to share this discussion with others.
Want to know about Cloaking Inequity’s freshly pressed conversations about educational policy? Click the “Follow blog by email” button in the upper left hand corner of this page.
Cloaking Inequity 
Hi Aurora – a great piece and solid contribution to the field. I think the concept of context stability is key and a fresh angle on the process. Sadly, it’s probably almost impossible to insure it in the majority of districts. Until the entire paradigm is changed, especially current incentives that run counter to the idea of folks sticking with a school for several years, among other major elements, we’ll continue to have “deja vu all over again” episodes at the policy level. That said, I maintain that solid knowledge exists for making significant change in practice. Our DESSI Study findings, if applied well, can improve offerings for children, even if only for a bit of time. The simulations we developed drew on these findings and, when I and my colleague, Louise Stoll, were commissioned by the British Ministry to develop one supporting their primary school policy some years back, we added “context” of schools into the simulation created. While not as nuanced or elaborated as well as your concept here, it was an important addition. Glad to see we were on the right tract! I’d very much like to read your dissertation; is it available online? Again, congrats on a helping advance the cause. Dr. David Crandall, President [semi-retired], The NETWORK Inc.
LikeLike
Hi David,
Thank you so much for the comment! I am gratified to see that these ideas resonate well with people who have been in the business for a long time. I agree with you that context stability is probably pretty impossible, particularly in the districts and schools that bear the brunt of the school improvement policies. But if it is an essential ingredient for school-wide effectiveness, perhaps policy should stop expecting all schools to be effective, and stop measuring educational quality at the school level. As I was trying to make sense of 12 years of data about a single school that was engaged in “reform” I stumbled upon a passage in Dan Lortie’s “Schoolteacher” where he speculates that maybe the egg-crate structure of schools (isolated classrooms off long hallways) was a functional organizational adaptation to high teacher turnover– basically that it would be highly inefficient for schools that were always losing teachers to operate more organizationally. But I think the field saw the challenges of teaching named by Lortie as problems to be solved – so we went to work trying to make schools less isolating through collaboration and PLCs and the like and schools more “whole” in general… but maybe it doesn’t work to solve a problem that is actually a functional solution for the given circumstances. Tracing the history of how educational change became a problem of the school organization was pretty mind-blowing for me… my adviser Milbrey McLaughlin helped write the story on the importance of context, and what I learned is that not only does context matter, changes in the context matters, too. It’s how we manage those changes in context that matters. Thank you again for the comments- I look forward to any additional comments and feedback you have aurora.woodmoore at gmail. You may download the dissertation here: http://searchworks.stanford.edu/view/10164616
LikeLike
Pingback: Aurora Moore: Is “School Improvement” a Myth? | Co-Opt-Ed
Pingback: Aurora Moore: Is “School Improvement” a Myth? | Diane Ravitch's blog
Dear Aurora Moore,
Thanks to Prof. VasquezHeilig for posting this.
I need to learn more about context stability and and information on correlation with quality, however measured. This is extremely relevant to the current excitement about turnaround schools.
Can you give me some citations to start with?
Pat Jehlen, State Senator, Massachusetts
LikeLike
Dear Senator Jehlen,
Thank you for your interest. The concept of context stability is something I developed concept through a historical analysis of the reform literature and through a 12 year case study of reform at one school. I encourage you to read the first two chapters of my dissertation, which outlines this history, if only because I think it’s important to understand how improvement became a problem of the school organization, and how we became so fixated on SCHOOL improvement as a field. But here is a quick breakdown:
After the first wave of effective schools research, Susan Rosenholtz conducted a meta-analysis of the research and found that for schools to operate as an effective organization, they had to have a certain level of stability in staff: “It is important to note that the articulation of classrooms into coherent, school wide programs implies a process of continual development resulting from, and enabled by, the commitment of a stable staff of administrators and teacher” (p. 374).
The large study of Chicago school reform on which the popular book “Organizing Schools for Improvement” was based showed the importance of “school leadership” and an “instructional guidance system.” These constructs were built from several indicators (from survey questions) about the stability of programs, staff and initiatives over time; these indicators were also once described as “instructional program coherence” (Newmann, Smith, Allensworth, & Bryk, 2001). Specifically, the indicators related to stability within instructional program coherence included: sustained professional development, strategic program selection and school planning based on goal of program continuity, stable curriculum and assessments, stable teaching assignments, and stability in key programmatic leaders to support and develop initiatives (Newmann et al., 2001, p. 9-10).
More recently, Holme and Rangel (2012) found that in Texas schools working towards
improvement under accountability, high rates of teacher turnover created “organizational
instability” which appeared to keep schools from developing the social capital necessary
to make steady gains in student achievement.
All of these studies suggest that improvement of the organizational unit called “school” is only possible when key dimensions of the organization itself remain fixed. That is, improvement of a unit called a school literally depends on low rates of turnover in teaching staff and principals, continuity in curriculum and assessment, and ongoing related professional development. Stability in the technical core (curriculum, programs) allow teachers to organize and structure their time in routine and predictable ways, enabling them to practice new techniques within a familiar setting. And stability in teaching assignments (within school stability) enables teachers to build knowledge and expertise over time. And finally, it is only under conditions of low turnover, does and with low turnover that expertise accrues at the school level (see Huberman & Miles, 1984)
At some point in the history of this research, education scholars became interested in structuration theory and concepts of culture. They supposed that improvements and learning could somehow be retained in the ether or walls of a school. While ideas and curricula are indeed passed on over time, all of the literature on the sustainability of school reform shows that very little is sustained over time, and that turnover in administrators and teachers uproots programs and policies (see Datnow, 2005)
Again, thank you for your interest. Discovering the importance of stability within the very research that has been used to justify school improvement/turnaround policy was a jaw-dropping experience for me. I hope your colleagues find it equally so.
My dissertation: http://searchworks-test.stanford.edu/gdor/view/10164616
Bryk, A. S., Sebring, P. B., Allensworth, E., Easton, J. Q., & Luppescu, S. (2010).
Organizing Schools for Improvement. Chicago, IL: University of Chicago Press.
Datnow, A. (2005). The Sustainability of Comprehensive School Reform Models in
Changing District and State Contexts. Educational Administration Quarterly, 41(1),
121–153. doi:10.1177/0013161X04269578
Holme, J. J., & Rangel, V. S. (2012). Putting School Reform in Its Place Social
Geography, Organizational Social Capital, and School Performance. American
educational research journal, 49(2), 257–283. doi:10.3102/0002831211423316
Huberman, A. M., & Miles, M. B. (1984). Innovation Up Close: How School
Improvement Works. New York, NY: Plenum Press.
Newmann, F. M., Smith, B. A., Allensworth, E., & Bryk, A. S. (2001). School
instructional program coherence: Benefits and challenges. Educational Evaluation
and Policy Analysis (Vol. 23, pp. 297–321). Sage Publications
Rosenholtz, S. J. (1985). Effective schools: Interpreting the evidence. American Journal
of Education, 93(3), 352–388.
These references also point to the need for stability:
Clotfelter, C. T., Ladd, H. F., Vigdor, J. L., & Diaz, R. A. (2004). Do school
accountability systems make it more difficult for low-performing schools to attract
and retain high-quality teachers? Journal of Policy Analysis and Management, 23(2),
251–271. doi:10.1002/pam.20003
Heath Kaufman, J., & Stein, M. K. (2010). Teacher Learning Opportunities in a Shifting
Policy Environment for Instruction. Educational Policy, 24(4), 563–601.
doi:10.1177/0895904809335106
Ruby, A. M. (2002). Internal teacher turnover in urban middle school reform. Journal of
Education for Students Placed At Risk, 7(4), 379–406
LikeLike
Dear Senator Jehlen,
Thank you for your interest. The concept of context stability is something I developed concept through a historical analysis of the reform literature and through a 12 year case study of reform at one school. I encourage you to read the first two chapters of my dissertation, which outlines this history, if only because I think it’s important to understand how improvement became a problem of the school organization, and how we became so fixated on SCHOOL improvement as a field. But here is a quick breakdown:
After the first wave of effective schools research, Susan Rosenholtz conducted a meta-analysis of the research and found that for schools to operate as an effective organization, they had to have a certain level of stability in staff: “It is important to note that the articulation of classrooms into coherent, school wide programs implies a process of continual development resulting from, and enabled by, the commitment of a stable staff of administrators and teacher” (p. 374).
The large study of Chicago school reform on which the popular book “Organizing Schools for Improvement” was based showed the importance of “school leadership” and an “instructional guidance system.” These constructs were built from several indicators (from survey questions) about the stability of programs, staff and initiatives over time; these indicators were also once described as “instructional program coherence” (Newmann, Smith, Allensworth, & Bryk, 2001). Specifically, the indicators related to stability within instructional program coherence included: sustained professional development, strategic program selection and school planning based on goal of program continuity, stable curriculum and assessments, stable teaching assignments, and stability in key programmatic leaders to support and develop initiatives (Newmann et al., 2001, p. 9-10).
More recently, Holme and Rangel (2012) found that in Texas schools working towards
improvement under accountability, high rates of teacher turnover created “organizational
instability” which appeared to keep schools from developing the social capital necessary
to make steady gains in student achievement.
All of these studies suggest that improvement of the organizational unit called “school” is only possible when key dimensions of the organization itself remain fixed. That is, improvement of a unit called a school literally depends on low rates of turnover in teaching staff and principals, continuity in curriculum and assessment, and ongoing related professional development. Stability in the technical core (curriculum, programs) allow teachers to organize and structure their time in routine and predictable ways, enabling them to practice new techniques within a familiar setting. And stability in teaching assignments (within school stability) enables teachers to build knowledge and expertise over time. And finally, it is only under conditions of low turnover, does and with low turnover that expertise accrues at the school level (see Huberman & Miles, 1984)
At some point in the history of this research, education scholars became interested in structuration theory and concepts of culture. They supposed that improvements and learning could somehow be retained in the ether or walls of a school. While ideas and curricula are indeed passed on over time, all of the literature on the sustainability of school reform shows that very little is sustained over time, and that turnover in administrators and teachers uproots programs and policies (see Datnow, 2005)
Again, thank you for your interest. Discovering the importance of stability within the very research that has been used to justify school improvement/turnaround policy was a jaw-dropping experience for me. I hope your colleagues find it equally so.
My dissertation: http://searchworks-test.stanford.edu/gdor/view/10164616
Bryk, A. S., Sebring, P. B., Allensworth, E., Easton, J. Q., & Luppescu, S. (2010).
Organizing Schools for Improvement. Chicago, IL: University of Chicago Press.
Datnow, A. (2005). The Sustainability of Comprehensive School Reform Models in
Changing District and State Contexts. Educational Administration Quarterly, 41(1),
121–153. doi:10.1177/0013161X04269578
Holme, J. J., & Rangel, V. S. (2012). Putting School Reform in Its Place Social
Geography, Organizational Social Capital, and School Performance. American
educational research journal, 49(2), 257–283. doi:10.3102/0002831211423316
Huberman, A. M., & Miles, M. B. (1984). Innovation Up Close: How School
Improvement Works. New York, NY: Plenum Press.
Newmann, F. M., Smith, B. A., Allensworth, E., & Bryk, A. S. (2001). School
instructional program coherence: Benefits and challenges. Educational Evaluation
and Policy Analysis (Vol. 23, pp. 297–321). Sage Publications
Rosenholtz, S. J. (1985). Effective schools: Interpreting the evidence. American Journal
of Education, 93(3), 352–388.
These references also point to the need for stability:
Clotfelter, C. T., Ladd, H. F., Vigdor, J. L., & Diaz, R. A. (2004). Do school
accountability systems make it more difficult for low-performing schools to attract
and retain high-quality teachers? Journal of Policy Analysis and Management, 23(2),
251–271. doi:10.1002/pam.20003
Heath Kaufman, J., & Stein, M. K. (2010). Teacher Learning Opportunities in a Shifting
Policy Environment for Instruction. Educational Policy, 24(4), 563–601.
doi:10.1177/0895904809335106
Ruby, A. M. (2002). Internal teacher turnover in urban middle school reform. Journal of
Education for Students Placed At Risk, 7(4), 379–406
LikeLike
Hi Aurora, I am from the UK and picked up on your blog site via Nancy Bailey. As a retired headteacher and former school inspector, I was fascinated to read your Cloaking Inequity article, ‘Is School Improvement a Myth?’. Even before I read this, I knew the answer, now I appreciate why. Thank you for this enlightened view on a vexed issue we also struggle with.
I fully agree with your conclusion, Context Stability and Autonomy are hidden variables that have a huge impact on our judgements. I had experience of the effects of both but, until now, couldn’t quite put my finger on exactly why I was so uneasy with the processes involved in the measurement of school effectiveness. Your comment, “correlational analyses of variables measured at one moment in time.”, clarified so much in a few words, pinpointing why the ‘playing field’ is not and never can be level.
I have shared your link with others in the UK who are trying, against the intentions of our government, to ensure that the reform of education is driven, not just by evidence, but by commonsense, compassion and a commitment to make education better for all those dependent on its fruits to find their way in a dynamic global future.
LikeLike
Hi Christine,
Thank you for the comments! Julian graciously allowed me to post this as a guest blog. Your point about institutional knowledge is spot-on– somehow when education research met organizational theory we got this idea that school culture could change and that the cultural shift could be somehow institutionalized in the school… but researchers never really looked at what happened at the end of their research grant period. And once schools were treated as unitary organizations that should function like businesses, we got all the terrible turnaround ideas that have wreaked havoc on our schools. So the myth of school improvement has been institutionalized at the level of the field… and it sure is hard to bust it from here.
LikeLike
Thanks for these observations, Julian. So many who live outside schools fail to understand how organic a school is. Your citation of the kid off his meds the week the counselor goes on maternity leave is spot on – reality knocking.
This is also why a “turn-around” where 50% of staff is dumped is a disaster – no one is left who has institutional knowledge. No one can be consulted about “why not let’s try to solve it this way”, and so much time and energy is wasted re-inventing a wheel that won’t turn.
LikeLike