Continuous Evaluation

How do partnerships use data and what is the bottom line for partnerships in terms of student academic achievement? All of the partnerships that participated in the Alliance used data to measure results and improve effectiveness. All were able to provide evidence of improvement in student academic outcomes, which could reasonably be linked with partnership activities. These outcomes are measured in terms of reading and math scores, API indices, college preparation and college attendance.

In Santa Cruz, "data-driven reform" is a mantra. With a professional evaluator on staff and participation from faculty who have chosen to make the partnership a focus of their own scholarship, Santa Cruz brings strong research resources to bear. The focus of the partnership, college participation, is clearly defined and straightforward to measure.

Long Beach is broadly focused, with many projects ranging from kindergarten to college preparation to teacher training. This structure provided an unusual challenge to define and measure success. The Long Beach school district is fortunate to have a first rate research department, providing an unusual level of analytic expertise. Chief information officers at the three institutions have collaborated to study inter-segmental issues. In addition, data are always gathered on specific major initiatives.

In Kern County, the focus on increasing the number of fully credentialed teachers provided a straightforward strategy for measuring success. There was a dramatic drop in the number of emergency teachers and a drop in the number of hard-to-staff schools. The university enlarged its teacher preparation program. High schools created "teacher academies" to encourage students to choose teaching as a career. With specific, clear and measurable goals and hands-on involvement from the chief executives of the partnering institutions, producing the data to demonstrate results was a straightforward process.

As a mature and complex partnership with a broad vision, Santa Ana has multifaceted evaluation activities and results. The partnership has a remarkable track record of establishing academic and support programs and the evidence in terms of outcomes is also impressive.

With a focus on professional development, the North County federation's tracks the amount of training provided and surveys participating districts regarding satisfaction with training experiences. The North County federation provides an exemplary instance of a partnership focused on improving infrastructure, which nevertheless makes the effort to demonstrate the impacts on student academic outcomes-the bottom line for education.

In City Heights (San Diego) many of the collaborative's numerous programs are evaluated for effectiveness. An evaluation report was generated in February 2003, indicating annual data, grants received, hours committed to projects, and numbers of students served. The City Heights Education Pilot looks at both academic and non-academic measures to determine levels of involvement and satisfaction. Non-academic measures include enhanced teacher training, improved teacher retention rates, increased professional development opportunities provided, greater community involvement in the partner schools, and the number of university publications and research.

Shasta focused on measuring the numbers of students taking college preparatory and advanced placement courses, college applications, college going rates, and standardized test results. These data elements were gathered and analyzed as part of reporting for external funding.

All of the partnerships paid attention to data but only one locale had an evaluator dedicated to the partnership work: Santa Cruz. Others routinely conduct evaluation in conjunction with specific projects and often in conformity to grant reporting requirements. The Santa Cruz arrangement may provide two significant advantages. The evaluator at Santa Cruz is involved in the vision and practice of the whole partnership, not just its specific projects. This creates the possibility of gathering data regarding the impacts of the partnership in the broadest sense - a difficult but desirable addition to evaluating the specific projects of the partnership. Second, evaluation often gets lost in the shuffle of managing education and reform on a day-to-day basis. Long Beach made an effort to engage in a sophisticated, holistic evaluation of its partnership. While successful at the time, it proved difficult to sustain that focus with the press of other activities. The Santa Cruz arrangement keeps evaluation and data on the agenda day-by-day in a way that is difficult for other partnerships.

In summary these partnerships have different goals and have placed their energies in different activities. We should expect to see different patterns of results across these sites and in fact we do. The clearest indicators of success are found in measures that are most closely related to the activities of the partnership and these indicators are sometimes but not always the "standard" measures such as the state testing program and Academic Performance Indicator. Also as would be expected, the partnerships that have taken on a wide range of activities, such as Long Beach, have clear results in some areas and absent or unclear results in other areas.

There is abundant evidence in the case studies of these partnerships that the key principles which have been identified in this study and in the research literature are in fact key to their effectiveness. The bottom line for these partnerships shows impressive evidence of gains in student academic outcomes. The bottom line matters.