Update: “Fatal Flaws” in Edvance Texas TFA study

1–2 minutes

·

March 8, 2013

·

Teach for America released a new study this week about their Texas operation. It is nearly 100 pages long, but when I reviewed the study I noted a fatal flaw. See my discussion of the fatal flaw in The Monitor: Study finds Teach for America corps members, alumni had outsized impact on middle schools in several Texas districts – The Monitor: Local News. See excerpts from The Monitor story below…

TFA argues that their study:

The data this research provides is pretty clear that we’re having a positive impact on the achievement of low-income students,” said Robert Carreon, executive director of Teach for America Rio Grande Valley.

My analysis:

The data gathered for the report also did not analyze teacher certification or degree programs. Julian Vasquez Heilig, an education professor at the University of Texas at Austin, said the lack of certification data was a “fatal flaw” for the study.

“You’re comparing apples to oranges,” he said. “Experience is not always a proxy for training.”

David Goodman, a researcher who helped conduct the study, said Edvance was unable to obtain individual teacher certification data from the state but said the effect sizes were so strong, controlling for certification would not have eliminated them altogether.

But Heilig said teacher certification is at the heart of the instructor quality problem TFA wants to address.

“A revolving door of untrained, uncertified teachers is probably our biggest single problem in education today,” he said.

See also a statistical review conducted by NEPC of the Edvance TFA report. The researchers conclude:

Because of those [statistical] limitations, Fuller and Dadey caution that the report does not provide solid evidence that either TFA teachers or TFA alumni have a measured effect on student test scores.

To continue the discussion on the issues raised in this post, please Tweet and Facebook Like it for others below. ciao!

Teach for America released a new study this week about their Texas operation. It is nearly 100 pages long, but when I reviewed the study I noted a fatal flaw. See my discussion of the fatal flaw in The Monitor: Study finds Teach for America corps members, alumni had outsized impact on middle schools in several…

5 responses to “Update: “Fatal Flaws” in Edvance Texas TFA study”

Teach For America is Barking Up the Wrong Tree | Cloaking Inequity

January 19, 2014 at 11:49 am

[…] that would have included studies that have been through no sort of vetting at all – and included evaluations that were funded by TFA itself, which clearly introduces bias. Moreover, a review of every study would have been unnecessarily […]

LikeLike

Reply
Debate Rages: Response to TFA’s supposition that new brief is “retreat from evidence” | Cloaking Inequity

January 9, 2014 at 2:58 pm

[…] Miller implies that we cherry-picked studies to review. We make it clear in the brief that we focus on peer-reviewed articles, and we review all the peer-reviewed studies that have been published since our 2010 NEPC brief. This standard is not one we made up— peer-reviewed articles are widely seen as the gold-standard for quality of research. Peer-reviewed articles are refereed by other scholars (usually double blind) to ensure that the standards of the field and of the journal publishing the article are maintained. An article is published only after it has met these standards, usually after the reviewers’ comments for revision have been addressed, and after the article has been approved by multiple scholars and by the publication’s editor. This is in contrast to reports that are not peer-reviewed, which can be published by anyone. Although the latter can potentially be of higher quality than the former, the more likely scenario favors peer-reviewed publications. This is why we opted to go with the scholarly standard of peer-review articles. We acknowledge that TFA has commissioned and paid for many research studies. For example, Cloaking Inequity and NEPC reviewed the Edvance study mentioned by Raegen Miller previously (See Update: “Fatal Flaws” in Edvance Texas TFA study). […]

LikeLike

Reply
Richard

March 12, 2013 at 11:52 am

“A revolving door of untrained, uncertified teachers is probably our biggest single problem in education today,” he said. Probably not. There is the same problem with those that are certified, namely they all have a poor understanding of mathematics. I personally believe that that is not teachers’ fault. The institutions that prepare them and the omnipresent army of fake experts is the biggest problem in the math education today.

LikeLike

Reply
rmdemore

March 11, 2013 at 4:21 pm

Teacher Certification identifies persons who have studied, practiced and met qualifying standards in the instruction of coursework. This includes, but limited to, pedagogy, curriculum, best practices, school law, teaching cycles, child development and many, many other topics, practices and networking with experienced faculty, staff and professors knowledgeable in supporting good and effective instruction. Teaching is a licensed profession and teachers are/ or should be held accountable. You don’t place just any warm body in any classroom.

LikeLike

Reply
PShannon

March 10, 2013 at 4:26 pm

TFA, also, over generalized the effect of their recruits. They did not break down the dynamics of the students being tested (i.e. special education, autistic, free and reduced lunch, ELL, etc.) or what types of schools these test scores originated from because as you stated you cannot compare apples to oranges. It is, also, statistically not relevant when the students only scored a matter of 2 or 3 questions more than their counterparts. The company who did the research is aligned with TFA and Students First; therefore, should we just assumed that the data presented is unbiased? I don’t believe so!

LikeLike

Reply