News Ticker

Common Core & Chaos in Computerized Testing: Are New Assessments Valid, Reliable, Secure, Accessible, and Fair?

Guest post by Roxana Marachi, Ph.D.

For latest update (as of September 8th, 2015) please see:
Open Letter to CA State Board of Education on Release of [False] SBAC Scores

[Original post from July 6th, 2015]
A recent report from the Public Policy Institute reveals that the majority of California’s public school parents are uninformed about the new tests their children took this past year. And despite numerous concerns regarding the lack of validity, technological barriers, biases, and test administration problems, “test scores” soon will be released to the public.

The following includes adapted selections of a letter I sent to the California State Board of Education for the July 2015 State Board of Education meeting.  My purpose in sharing this information is to draw attention to the lack of scientific validity of the test scores that are soon to be released to the public and to promote critical thinking about issues of fairness, accessibility, data security, and standardization in the test administrations.

It is important to consider that unless assessments are independently verified to adhere to basic standards of test development regarding validity, reliability, security, accessibility, and fairness in administration, resulting scores will be meaningless and should not be used to make claims about student learning, progress, aptitude, nor readiness for college or career (see Legal Implications of High Stakes Assessments: What States Should Know).

QUESTIONS: 

Q1: How is standardization to be assumed when students are taking tests on different technological tools with vastly varying screen interfaces? Depending on the technology used (desktops, laptops, chromebooks, and/or ipads), students would need different skills in typing, touch screen navigation, and familiarity with the tool.

Q2: How are standardization and fairness to be assumed when students are responding to different sets of questions based on how they answer (or guess) on the adaptive sections of the assessments?

Q3: How is fairness to be assumed when large proportions of students do not have access at home to the technology tools that they are being tested on in schools? Furthermore, how can fairness be assumed when some school districts do not have the same technology resources as others for test administration? 

Q4: How/why would assessments that had already been flagged with so many serious design flaws and user interface problems continue to be administered to millions of children without changes and improvements to the interface? (See report below)

Q5: How can test security be assumed when tests are being administered across a span of over three months and when login features allow for some students to view a problem, log off, go home (potentially research and develop an answer) and then come back and log in and take the same section? (This process was reported from a test proctor who observed the login, viewing and re-login process).

Q6: Given serious issues in accessibility and the fact that the assessments have yet to be independently validated, how/why would the SmarterBalanced Assessment Consortium solicit agreements from nearly 200 colleges and universities to use 2015 11th Grade SBAC data to determine student access to the regular curriculum or to “remedial” courses? http://blogs.edweek.org/edweek/curriculum/2015/04/sbac.html.

Projected failure rates disproportionately impact youth of color, students with special needs, and English Language Learners, cut scores have yet to be validated, and the use of the test scores may be argued to contribute to systemic barriers that already limit access to higher education for students from historically underserved populations.

[Chart data from SBAC / Screenshot from post on J.Pelto’s Wait, What blog] 


Q7: [Background] I participated in the cut-score setting process during October of 2014 as I had learned through a public announcement on social media about the opportunity to take part. According to their own documents, SBAC confirms that 7% of their feedback about the cut score setting was from the “general public.” Smarter Balanced is inviting anyone who’s interested—you don’t have to be a teacher or even work in education—to register to participate in the “standard-setting” process.

It was apparent by the lack of screening that as long an individual had a pulse and an email address, any member of the public who so wanted was given open access to at least 30 test items and answers considered for use in the assessments. Despite signing an electronic statement promising not to share any information from the activity, anyone (including test-prep profiteers) could have downloaded within a matter of minutes dozens of test items developed for use by the consortium. It is appalling that such an epic breach of security would be allowed in the process of test development.

Screenshot 2015-07-03 20.25.38

Q7: Given the open access of hundreds of test-items to large numbers of unscreened participants who took part in the public cut-score setting procedures, how can the State Board of Education ensure the security of test items used by the SmarterBalanced Consortium?

[Additional note (not included in original letter)] 
One might also consider issues of students sharing test-related topics and posts on social media as potential security breaches of content.
With a three month testing span and countless new apps for anonymous sharing, the possibilities are endless.   Still, testing companies seem laser-focused on following students’ visible social media feeds with controversial surveillance strategies that have led to serious privacy concerns (here, here, and here) in states with the PARCC versions of CCSS tests administered by Pearson.  While public awareness and concern has focused primarily on Pearson’s activities, the SmarterBalanced Assessment Consortium has also been critiqued for its policies on student monitoring.  As we consider issues of test security, we should acknowledge the thousands of potential, alternate, creative hashtags or other communication tools that youth may use aside from the obvious #PARCC #SBAC and #SmarterBalanced tags to discuss their experiences with the assessments.  Let’s take a look at how [in]effective it was for the College Board to require sworn statements from test-takers to refrain from posting about the PSAT on social media:

Screenshot 2015-07-06 00.11.09Screenshot 2015-07-03 23.15.00

Q8: Will the California State Board of Education hold the SmarterBalanced Assessment Consortium member states and test developers accountable to adhere to the basic terms, timelines, conditions, and agreements described in the Race to the Top Assessment Development Proposal? http://www.edweek.org/media/sbac_final_narrative_20100620_4pm.pdf [Note: Pages 47-50 offer detailed descriptions of intended features of the assessments and proposed supports for implementation. Readers are encouraged to compare and contrast these descriptions with what has actually been developed (evidence below).] 

Q9: Since data gathering for the development of the SmarterBalanced Assessments has been conducted through a Federal Research Grant, and the Public Policy Institute has determined that the majority of parents have been uninformed about the new tests, how can the State Department of Education and/or SmarterBalanced Consortium ensure that Basic Protections of Human Subjects were upheld during the pilot and field test administrations over the past two years? http://www2.ed.gov/about/offices/list/ocfo/humansub.html.

Is it the responsibility of the SmarterBalanced Testing Consortium, the Federal Government, or State/District Board of Education Trustees to ensure that student and parental rights and ethical protections of Human Subjects were not violated in the process of developing the new tests?

Q10 (Background): As you are aware, ETS (Educational Testing Services) has been provided the multi-million dollar contract to administer and score the tests in California http://www.kcra.com/news/whos-grading-your-kids-assessment-test-in-california/31857614. According to the following article “ETS has lobbied against legislation to require agencies to “immediately initiate an investigation” after complaints on “inadequate” testing conditions. It also lobbied against a bill designed to safeguard pupil data in subcontracting.”  http://www.washingtonpost.com/blogs/answer-sheet/wp/2015/03/30/report-big-education-firms-spend-millions-lobbying-for-pro-testing-policies/

Q10a: How will the State Board of Education ensure the student data being provided to 3rd party entities (unbeknownst to many parents and students) would be secure?

Q10b: What are the responsibilities of the State and Districts to inform parents of how their children’s data may be used by 3rd party entities should they choose to take part in the testing?

Evidence of Testing Barriers and Implementation Problems

The Board is encouraged to consider the following evidence documenting serious concerns regarding the validity, reliability, security, accessibility, and fairness of the SmarterBalanced Assessments.

SmarterBalanced Mathematics Tests Are Fatally Flawed and Should Not Be Used documents serious user-interface barriers and design flaws in the SmarterBalanced Mathematics assessments. According to the analyses, the tests:

  • “Violate the standards they are supposed to assess; 
  • Cannot be adequately answered by students with the technology they are required to use; 
  • Use confusing and hard-to-use interfaces; or 
  • Are to be graded in such a way that incorrect answers are identified as correct and correct answers as incorrect.

No tests that are so flawed should be given to anyone. Certainly, with stakes so high for students and their teachers, these Smarter Balanced tests should not be administered. The boycotts of these tests by parents and some school districts are justified. Responsible government bodies should withdraw the tests from use before they do damage. Read the full report…

Rasmussen notes that the numerous design flaws and interface barriers had been brought to the attention of the SmarterBalanced Assessment Consortium during the Spring 2014 pilot test and remained unresolved during the Spring 2015 test administration.

There are also misrepresentations of technology interface features in public awareness campaigns to inform parents and students about the new tests. According a video on the BeALearningHero.org website, fractions spontaneously appear [in fraction form] on the screen as a visual feature of the new assessments:

 

However, the actual screens that students encounter on the tests are completely different than what is shown above and the process of entering even one fraction into the text space provided is problematic.

Why are the actual screen representations of the tests not provided in the public education campaigns?

Further evidence of testing problems are included in the following clip with a selected portion of the transcript from the NAACP Press Conference on SBAC Testing in Seattle. The segment below is by Jesse Hagopian and the rest of the full Press Conference includes additional evidence and perspectives:

[Segment starts at 7:00 Minutes from original video] …

 

“SBAC testing in Seattle has been an unmitigated disaster. We have had reports across this city of absolute testing meltdown in building after building. You’re going to hear today from some of the people who have experienced that first hand. They’ve experienced the technological gliches that have wasted hours of instructional time. They’ve experienced their computer labs and their libraries shut down for weeks at a time and unable to be used for research. And they’ve experienced the human disaster of labeling kids failing and seeing the impact that that has on children and crushing their spirits, turning them off to education… and you’ll hear that I think in graphic detail…”

Hagopian continued,

“…these tests are invalid… the SBAC test is invalid. That’s a bold statement. How can I make that claim? It’s not based on my own estimation. It’s based on the SmarterBalanced Consortium itself. The SmarterBalanced Consortium has acknowledged in a memo, that the test has not been proven to be externally valid. And yet our state rushed to implement this test all across the state?”  See also http://sco.lt/729uev

In Maine, educators also have expressed concern regarding the validity of the SBAC exams. In Test Fatally Flawed, School Officials SayScott McFarland, Mount Desert Elementary School Principal posited:

“I’ve seen enough of it, seen enough glitches to know that it’s invalid data.”

From EdSource April, 2015 

“Laura Bolton, a teacher at William Saroyan Elementary in Fresno, who spoke to EdSource earlier about her students’ keyboard challenges, recently gave the Smarter Balanced midterms to her 3rd-graders. She said her students struggled with the instructions and added that they were not appropriate for the age of her students.”

The article also describes high school students’ experiences with instructional and user-interface barriers: A ninth grader quoted in the article, for example said “she didn’t know how to make the online calculator work on the test.”  The following excerpt from another article described that “Everyone was freaking out,” … with “students struggling to find the calculator and other function buttons on their tablets.”  

An 11th grader was quoted as indicating that “There was no paper available” to work out math problems.“We’d have to draw on the computer screen [with a stylus]. Being on a computer was distracting.”

From a Parent comment in “SBAC in Ca”:

“According to my child, there were numerous technical problems with test administration. Not much of the questions addressed things they had learned. Confusing. Bandwidth continued to be a problem along with tablets accepting answers. As parents, our district failed to notify us when testing would take place. Normally we are inundated with information about dates, times, how to feed our kids, how much sleep they needed, etc. Had I known when these tests would take place, I would have opted my child out. We get no feedback from the school, district or the state of results, which seems immensely wrong. I have to based this all on what my child told me as my school and district told me nothing.” (Emphasis added)

From a Principal/Administrator: “Accessibility in SmarterBalanced”:

“Our district recently completed the Smarter Balanced Field Test. I was very disappointed with accessibility features of the assessment. I have heard and read that the assessment has unprecedented accessibility features and provides avenues for students to participate. The accommodations and embedded features were incredibly confusing for students with disabilities and struggling learners. Students needed to click and drag or click and highlight. The use of language glossaries were found to be inaccurate on many occasions. I’m concerned about the quality checks in place for language translations and the manner in which student can locate a word to see the glossaries. Likewise, we were told that devices needed to be certified to use with the assessment. There are currently ”no” certified devices. Sad to develop a system that looks good on paper and creates a ”good story” for accessibility but falls short with real world application. To my knowledge there is no means established to get feedback on the accessibility features. That too is disappointing. How will improvements be made if there isn’t a means to solicit input from users? I fear that we will be faced with the same issues next year.” (Emphasis added)

From a Teacher who teaches Mild and Moderate Special Day classes:

“I feel it is unfair that my SDC students are being required to take grade level state tests when they are not taught or learning at that grade level. I would like my 8th grade students whom are learning 4th grade math, to take the test at the 4th grade level. There is no point in throwing complex algebra problems to students who are working on their basic math facts. This does not tell use anything about what the students know or what they are gathering from school.”

From a Teacher in LAUSD on using ipads for the SBAC testing:

“Today was the worst day I have had in a classroom for 20 years.
At my school we received the ipads last Thursday. We started testing today.
1. I have never used an ipad, and had no training whatsoever. They had them on campus for a week, yet they did not train us to use them and navigate how to help students out.
2. It took 45 minutes to get my 35 students logged in, both sessions.
3. We got kicked off during a session due to the fact that the server was busy.
4. Many of the readings were too long for the students.
5. Students had trouble highlighting text.
6. Some students could not answer questions because they could not see them on the Ipads.”

From a Media Specialist on pilot tests:

“After administering the language arts and math pilot tests for smarter balanced, teachers gave me their feedback. The tests were extremely time consuming; some students were sitting for over two hours. The structure and content of the test were not age appropriate. Teachers found the level of student frustration to be very high–students were actually angry and acting out during the testing sessions. Students were giving up on questions based on the lengthy reading passages presented–even strong readers. One teacher described the testing sessions as ‘child abuse.’” (Emphasis and links added)

From a Teacher:

“We have been teaching critical reading strategies in all subjects at my junior high. I was very surprised to not see any easily accessible tools to mark the text on the language portion of the SBAC, including underlining key words or phrases, numbering paragraphs, marking out incorrect answers, and more. If we are to test our students with screen after screen of completely filled, multiple columns of text, we should have at minimum, easy to use tools to apply these strategies.”

From an Administrator:

“I recently took the 3rd grade sample test for SBAC. I was truly horrified and felt panic-stricken for the children who may potentially interact with this type of assessment. While the content rigor was greatly increased, my biggest concern was all of the varied technology responses that a 3rd grade student would have to have mastered to be able to present the correct answer. I highly suspect that students will know answers but will get them incorrect because they don’t know how to manipulate the varying technology skills required to show the correct answer. As a highly proficient adult, I had difficulty manipulating the technology required to plot points on a graph and when I returned to the directions for guidance, the directions fell short of what I needed to know. If my prediction is correct, we will not only have poor-looking data, but will also have false data.” (Emphasis added)

***

My letter to the Board is to encourage responsible, ethical, and legal communications about the assessment data that will apparently soon be disseminated to the public. Students’ beliefs about themselves as learners will be caught up in the tangle of any explanations surrounding the assessments, and as we know, decades of research demonstrate the power of student belief to be a factor impacting subsequent effort and persistence in learning.

To be transparent to the public regarding the current status of the assessments, the Board is encouraged to consider full disclosure regarding the lack of validity of the assessments as well as the numerous complaints that have been documented regarding the problematic administrations of the tests. While this may be difficult from a public relations standpoint, to misrepresent the scores as valid or accurate would likely lead to more serious problems. If the State does plan to base policy decisions on test scores provided by the SmarterBalanced Assessment Consortium (including this year’s full field administrations as any form of “baseline” for future comparisons), again, please consider the full range of legal and ethical implications.” 

[End of letter segment]

___________________________________________________________________________

[September 2015 Update]

To hear the State Board of Education discussion on the SmarterBalanced Assessments, including further questions that remain unanswered by the Executive Director of the SBAC at the September 2nd, 2015 California State Board of Education meeting, please click on the video below and view from ~1:45:00.  

Screenshot 2015-09-03 21.18.13

The latest update on EduResearcher is an Open Letter to the State Board of Education on the SBAC Test including written comments made publicly by Dr. Doug McRae, a retired test and measurement specialist who has been communicating concern about the lack of validity of the SBAC Assessments directly to the State Board of Education over the past 5 years. His September 2nd, 2015 public comment may be viewed on the video linked to the image below from 02:50:55 – 02:52:52. 

Screenshot 2015-09-05 16.05.37

 

__________________________________________________________________

For related news on testing:

Confessions of an Assessment Field Tester // EdWeek

Superintendents and Legislators, Beware: The SBAC Tsunami is Coming//Wait,What?

SmarterBalanced Delays Spur Headaches in Wisconsin, Montana, and Elsewhere // EdWeek

More than a Dozen States Report Trouble With Computerized Common Core Tests // Washington Post

Grading the Common Core: No Teaching Experience Required // NY Times

Five Principles to Protect Student Data Privacy // Student Privacy Matters

EdTech Company [ThinkGate] Folds After Receiving Millions in Race To The Top Funds // Washington Post

Superintendent: Computerized Testing Gliches are Breach of Contract // Las Vegas Sun

Mayor Scott Lang [Democrat] Launches Sharp Criticisms of PARCC citing that Commissioner Chester “miscalculates the problem of unfunded liability, when kids fall off the school system and become wards of society.”

More than 90% of English Language Learners Projected to Fail Common Core SBAC Test // “Wait, What?”

Legal Implications of High Stakes Assessments: What States Should Know // OERI Office of Educational Research and Improvement

###

Thanks to Anthony Cody and Julian Vasquez-Heilig for feedback and guidance that supported this work.

For more, follow EduResearcher.com and @ConnectEdProf and for current updates on issues related to testing, please visit Testing, Testing, 1,2,3…

Advertisements
About Roxana Marachi, Ph.D (2 Articles)
http://bit.ly/ConnectEdProf

5 Comments on Common Core & Chaos in Computerized Testing: Are New Assessments Valid, Reliable, Secure, Accessible, and Fair?

  1. Concerned Parent // August 8, 2015 at 10:47 pm // Reply

    As a parent, I wish to thank you for yor article.

    I hope you ask Mr. Torlakson to put you on as a person to help craft the new CDE Blueprint for California education. August 2015.

    I think schools and oversight are getting way way too political,and parents may show less trust.

    I hope parents keep checking on equity of technology and other resources so low income students have equal footings in learning.

    I ask that all parents use oversight to ensure all common core standards in math are covered properly by downloading the standards.

    I hope parents begin to learn more about ucp complaint filing if the parents do not get properly treated.

    The Unknown Parent Explorer

    Like

    • Thank you for your comments and I appreciate the vote of confidence for being involved in statewide assessment efforts. I agree wholeheartedly to support parental rights for proper treatment regarding the testing (including opt out requests to be respected fully and with no questions asked). I wish I could agree with the recommendations to adhere more closely to strict oversight of the standards. I was initially a CCSS supporter (early, in 2010) and enthused by the idea of ‘fewer and deeper’, more extensive integration of higher order thinking skills, and the notion that social and emotional learning would be addressed by collaborative aspects of the new standards. Unfortunately, as I’ve learned more and have seen how the standards are unfolding in K-8 classrooms, I’m seeing strict adherence (especially in K-2) to be doing more harm than good for overall student learning and motivation. The developmental mismatches raised in the following article are worth highlighting (http://www.washingtonpost.com/blogs/answer-sheet/wp/2013/01/29/a-tough-critique-of-common-core-on-early-childhood-education/) and are fundamentally linked to early learning experiences that often shape the trajectory of student beliefs and motivation. It’s complex. While I have seen several examples of strong lessons in 6th grade ELA (free from the pressures of benchmark tests), I have also seen full classroom meltdowns of 1st graders when high-level math concepts were being pushed onto them prematurely (before many even had a basic understanding of simple concepts upon which the later math would build). There is a delicate balance in knowing when and how to build on an appropriate level of student challenge without pushing so hard that motivation would backfire. The standards as written ignore basic findings in developmental science related to visual and cognitive processing. Research on human motivation is also unfortunately absent from all planning (and affiliated testing) associated with the rollout of CCSS. Thanks again for your comment. I plan to write more about these issues in future posts on EduResearcher.com and welcome continued feedback.

      Like

      • Concerned Parent // August 9, 2015 at 11:59 pm //

        Yes, I As a parent support your comments and insights. What I am referring to with regard for parents to monitor if common core standards in math are being followed is, it is my opinion in speaking to lots of parents and some from other school districts is, a lot of teachers are not covering math properly or well, so, in my opinion, vast grade inflation with group grading, and , in elementary grade levels the standards are not being covered , and district administration does not care.

        It is really messed up for math. Some school districts saving money and not adopting new math materials, and, not the fault of teachers who must work, say, 5X harder to create curriculum, create hands on designed lessons, present, grade, and try to not front load ah ha ha moments…whatever, and we got think tank maybe grant funded groups with the abbreviation if western education, really not seeing the forest through the trees, but instead of taking the best way to teach, instead take long journeys of esoteric talk and miss the destinations of pedagogy.

        Your article is clear. It is well written. It helps to inform all stakeholders of some messy situations going on in California.

        I ask you to e mail the editor of EdJoin to have your article posted on that site, they post all good articles that have to do with education.

        I ask you to send an attachment and personal letter to Mr. Torlakson of C.D.E and send e mail attachment of your fine article to all members of the S.B.E. State Board of Education for, this information will help all, and, I again ask you to ask Mr Torlakson to hire you at the C.D.E or Governor Brown, for you have great strength of thinking, clarity, data honesty, and integrity. The C.D.E. needs people like you for parents like my wife and I to build more trust in the California Educational Process.

        Concerned Parent

        Like

      • Many thanks for clarifying the context of your comment. I understand much better now what you are advocating for and support your efforts. I have sent the questions and concerns from this initial post already in the form of a letter to the State Board of Education prior to their July 8th meeting (the blogpost was an adaptation of that letter) and also spoke out in person at the July 8th Board meeting in Sacramento. I appreciate the recommendation to share this more widely with other education news agencies as well and will explore additional avenues to get the message out. Thanks again for your thoughtful comments and feedback.

        Like

      • Concerned Parent // August 10, 2015 at 3:55 pm //

        Dear Ms. Marachi,

        I wanted to clarify that I wrote that you should post your fine article on Edjoin, but I meant to write EdSource. EdSource posts updates on good education articles and EdSource receives monies from over ten large donors so I believe your article can be posted on EdSource. Why not write the editor and push for it to be posted for your article uses data and makes valid points.

        It is all about hammering out change to best serve the children and all other stakeholders. Your article does this.

        We must look out for censorship of information and I am sure we can do this.

        In my opinion, the State Board of Education needs to be much more flexible in their moves with regard to how education is working. I as a parent have a hard time getting the time and gas money to drive up to speak my 3 minutes of public comment time to the State Board of Education board members. It is tough. But, I ask that parents do this. I ask that parents take the time.

        Because, often when it comes time for public comment (again, one can google “California State Board of Education Board Meeting Video” … when there is the public comment time, there is no public comment. The low income parents do not know how to drive over to the S.B.E. chambers and take 3 minutes to discuss:

        SmarterBalance/CAASPP
        .
        Common Core Blunders
        .
        L.C.A.P. issues of Parent and Teacher VOICES not being used in Shared Power ways (In other words parents are asked for input, input written down by school districts, then in my opinion, school districts just do whatever the heck they want to do with key highly paid school district officials making the decisions of where and how monies are spent).

        Some State Board of Education Board Members are impartial, some are, and those impartial State Board of Education members need to reach out and listen and act.

        Strong Educational Oversight Thinkers such as you Ms. Marachi, need parents and teachers to speak to your views and to help hammer out changes in the ed code and state statutes and even the Federal law.

        Your article is good, now can we find ways to find ways to get parents to want to read it and parents to want to act more and get involved more in change.

        I say, often in all layers of California Educational Leadership and even back in Washington with Arnie,

        Often board members and leaders hear what is said, maybe read what is said, but they do not listen to what is said or contemplate what is read.

        This is why parents and teachers must speak out more, much more, and if not…

        the status quo is welcome to blossom and the roots of political pontification will be

        embellished and nurtured.

        Concerned Parent

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: