The Ames Community School District (District) has launched a literacy pilot as part of our program evaluation in an effort to select materials to support Tier 1 instruction of foundational reading skills. Piloting of instructional materials as part of program evaluation before purchasing and implementing them District-wide has been in practice for over 30 years and is supported by Board Policy IF, Curriculum and Assessment Development.
Chapter 12 of Iowa Code requires that school boards have a process for assessing and improving curriculum and instruction. To comply, the District’s Board Policy IF states that curriculum evaluation, including evaluation of instructional materials, must include collecting, analyzing, and interpreting data to aid in understanding what students know and can do. In order to create a cohesive District program evaluation, pilot programs are always approved by the District, not at a building level.
Our District pilot program involves using three sets of vetted materials in classrooms in order to measure student growth and to better enable teachers and administrators to evaluate the instructional design of each program. At the end of the pilot, all data gathered will be used to select one set of materials to implement with all K-2 students during the 2017-2018 school year. Pilot programs for program evaluation are not considered to be research in the same way as someone working on their dissertation or thesis, for example. Rather, much like teachers using data to inform instruction for their students, the District is collecting data to determine which of these three research-based programs best meets the needs of all students in the area of K-2 foundational reading skills instruction.
Each of the three programs in this K-2 literacy pilot was selected by the Word Study Team because the programs aligned with the identified gap in our implementation of our Board-approved curriculum, the Iowa Common Core. These gaps were centered around the foundational skills standards including phonological awareness, phonemic awareness, and phonics. Several programs were vetted using a rubric adapted from the Center on Instruction’s reading materials evaluation rubric. The three programs selected scored above 50 points on our modified rubric that focuses only on the sections that pertain to our identified need.
At no time did the District shift to embark on a research study; rather, we are focused on collecting data from teachers and administrators as well as student growth data to select the best program to fit the identified needs. Numerous questions have been asked about the Iowa Reading Research Center’s (IRRC) role in this pilot, including whether the pilot is legal. The Ames Community School District requested the IRRC’s assistance with the pilot.
The IRRC is serving as the external evaluator for the District’s pilot of foundational skills instructional materials. The intent of the IRRC is to assist the District in selecting a set of materials to adopt, which is a part of the IRRC’s legislated purpose:
281—61.2(256) Purpose. The purpose of the center shall be to apply current research on literacy to provide for the development and dissemination of all of the following, although each of the following will not necessarily be of equal priority or immediacy:
1. Instructional strategies for prekindergarten through grade 12 to achieve literacy proficiency that includes reading, reading comprehension, and writing for all students.
Program evaluation of this sort does not meet the definition of human subjects research:
“Research” generally does not include operational activities such as defined practice activities in public health, medicine, psychology, and social work (e.g., routine outbreak investigations and disease monitoring) and studies for internal management purposes such as program evaluation, quality assurance, quality improvement, fiscal or program audits, marketing studies or contracted-for services.
To ensure proper interpretation of this provision, the IRRC submitted a Human Subjects Research Determination form to the University of Iowa’s Institutional Review Board (IRB). The IRB Chair reviewed the IRRC’s role and concluded it did not constitute human subjects research:
“I have reviewed the information submitted with your project titled 201609831 Evaluation of Phonics Curriculum Implementation in the Ames Community Schools. I have determined that the project described in the application does not meet the regulatory definition of human subjects research and does not require review by the IRB, because the project is to assist a school district in choosing an instructional product and the methodology is not a systematic investigation.”
The IRRC is involved as an independent third party to protect against pre-existing bias in collecting and analyzing student and teacher data. The IRRC is able to support our collection of student data that will be sensitive enough to show growth in these foundational skills. All student test data will be de-identified. Additionally, the IRRC will assist us during our pilot with collecting data to identify which programs are best able to be delivered with fidelity. Fidelity data includes video and audio files that focus on the teacher’s implementation of the materials—not on the students in the class. Recordings are uploaded to the IRRC, and the District will never have access to them. The fidelity data cannot be used for evaluative purposes, and the results will be provided at the aggregate level and not the individual teacher level. This information for each program will help us in analyzing the student growth data. Once the external evaluation is complete and the District has made the program selection, the recordings will be destroyed.
For these reasons, the District and IRRC are exempt from IRB oversight for the pilot activities as currently designed.
The District is appreciative of the support and collaboration provided by the Iowa Reading Research Center. We look forward to moving forward with a decision at the end of this pilot to implement the materials that will best meet the needs of our students.