The Importance of Formative Evaluation in TAH Project Success
As every LEA awarded TAH funding knows well, the required evaluation to determine whether and to what extent project outcomes are met is a fairly expensive component of the grant. The U.S. Department of Education (DOE) expects rigorous evaluation designs that, ideally, produce evidence that monies spent to increase teachers' knowledge of American history lead to improved student achievement.
To that end, the DOE has stipulated that each LEA must have clearly defined outcome measures, particularly tests of teacher knowledge, and must report performance annually and at the end of the project. These directives have shaped the form and substance of project evaluations which, by necessity, focus on producing summative data. This is good! Every project needs to know if its teachers have benefited from the professional development and the DOE needs to know if TAH funding should continue.
What isn't good is the omission of evaluation measures that gather formative data, the kind of data that contributes to massaging, refining, and developing the planning and implementation of professional development over time. While many of the funded TAH projects utilize a similar delivery model (summer institutes followed by workshops) the strategies (blogging, book discussions, lesson study, curriculum development) that projects employ to increase teachers' American history knowledge are generally unique to each project.
In order to assess the effectiveness of strategies, evaluations must include formative measures that gather data on teacher perceptions and products. Perceptions can be captured by interviews, written reflections, and surveys. Analysis of teacher products provides insight into teachers' transfer of new knowledge and skills. The object is to provide project leaders with on-going feedback that either confirms the effectiveness of their strategies or identifies the factors that cause strategies to be ineffective or less effective than desired.
Montgomery County Maryland's Conflict and Consensus offers a case-in-point. In 2008, following the project's first summer institute, focus group interviews revealed that teachers in the first cohort had mixed responses to the professional development. They felt the contributions of lead historians who facilitated the institute and workshops were immensely beneficial and that historian presenters offered outstanding content. Conversely, they were critical of combining 8th- and 9th-grade teachers responsible for teaching different periods of American history in the first week of a two-week summer program.
There was general agreement that although the institute presenters offered outstanding content, teachers thought that the sheer volume was at times overwhelming. There was also general agreement that the presentations should have been 1) more on grade level, 2) more closely tied to the curriculum teachers teach and, 3) more balanced in presenting content knowledge and teaching strategies. Teachers were required to post written reflections to the project website during the summer institute. In the interviews they commented that it was beneficial to read others' reflections, in part, because it helped them determine if they were 'on track.' However, it was sometimes challenging to write a response. Teachers suggested more extensive prompts to help focus their thinking and that, for continuity, reflections should be discussed in the next day's session.
In response to the formative data gathered in the interviews, the project leaders met to discuss ways to address teacher perceptions and recommendations. They were attentive to teachers' expressed need for professional development content to fit the curriculum they teach, acknowledging this relationship was critical if teachers were to transfer new learning to their practice in classrooms.
While pedagogy was an element of the project and initial summer institutes, teachers' feedback led project leaders to consider more deeply the importance of modeling pedagogy to support knowledge transfer. In workshops and institutes that followed the interviews, historian presenters were coached to overtly model primary source analysis and other pedagogies as part of their presentations and were encouraged to provide information about resources teachers could use with their students.
Project leaders deliberated about ways to aid teachers in connecting new content knowledge to the American history curriculum they were required to teach and provided time for demonstrations and discussions in subsequent workshops and institutes. The content and format of the summer institutes was changed in 2009 to focus more explicitly on the required curriculum and teachers were given more time to work in grade level teams.
In recent focus group interviews, cohort 3 teachers were hugely positive in their perceptions of the professional development. Here are comments from two teachers about the summer institute:
I'm new teaching the curriculum and I'm just realizing how helpful the class was in preparing me to teach the curriculum because the sessions this summer really do follow the curriculum that we're expected to teach.
I really enjoyed the summer institute. It's been a long time since I've been in an academic setting where I was a student. Even though I've taught U.S. History for a while, it really taught me new things or made me think about aspects of history that I have never thought of before. For example, combining communism and the civil rights movement or new ways of approaching teaching primary source documents as opposed to using them to reinforce something that I've taught; having the kids use the primary source documents to kind of create their own history. So I would go home after our discussions in the summer institute and have lengthy discussions with friends, family members as well. So I thought it was really interesting from an academic standpoint but, also from a personal aspect as well. I really enjoyed it!
Formative evaluation was essential to identifying the strengths and weaknesses of all aspects of the Conflict and Consensus professional development model. It allowed project leaders to monitor teachers' responsiveness and make mid-course corrections that enhanced teachers' learning and supported them in making changes in their practice.
The lesson learned is that formative evaluation supplies project leaders with the information they need to "grow" their TAH projects. Strong project designs can suffer if attention isn't given to the details that together form the professional development program. As John Wooden, a national basketball coaching legend, said, "It's the little details that are vital. Little things make big things happen."