CSU Long Beach College of Education Accreditation
Oversight for planning and assessment in the unit is provided by the Strategic Planning Committee, comprising representatives from faculty, staff, students, and the community. The unit's Graduate Programs Committee, comprised of program coordinators from initial and advanced programs in the College of Education and the College of Health and Human Services, has direct responsibility for overseeing candidate assessment and program review. Both committees report to the Faculty Council, which has overall responsibility for general faculty governance (Figure 2.01: Unit Assessment System Governance). Minutes from Strategic Planning Committee and Graduate Programs Committee meetings reflect agenda items and discussions around UAS-related issues. Collaboration and oversight by both unit- and program-level leadership ensures multiple perspectives so that each assessment is developed with consideration given to all other assessments, and that elements of the Conceptual Framework are reflected in key program assessments.
The design of the Unit Assessment System (UAS) (Figure 2.02: Unit Assessment System) reflects the unit’s fundamental purpose, which is to prepare highly qualified educational professionals to work in urban settings with learners from diverse backgrounds. Foremost, the UAS provides useful information for continued unit and program improvement. To aid this process, the UAS has been evolving to reflect a cohesive and comprehensive evaluation and assessment structure. Figure 2.02 provides an overview of the UAS. Each segment in Figure 2.02 represents one of the 6 NCATE standards and corresponding sources of evidence, both of which are linked to the 8 California Commission on Teacher Credentialing (CCTC) Common Standards (CS). At the figure's center is the unit Conceptual Framework (Exhibit 5), the bedrock of practice and programs in the unit. The UAS draws on data from numerous sources, e.g., candidate assessments to gauge progress from program admission to completion; surveys of faculty, candidates, graduates, and employers’ of graduates eliciting perceptions of programs, services, and resources; assessment and evaluation of field programs; recruitment and retention of diverse faculty and students; faculty quality and performance; and resources, governance, and infrastructure. Input comes from the Office of Institutional Research and Assessment, a variety of candidate and program assessment and evaluation sources, and various faculty activities (retreats, program meetings, committee/task force meetings, advisory board meetings, etc.).
The Program Assessment and Evaluation System (PAES) (Figure 2.03) is a key component of the UAS. Using a common unit-wide framework (Exhibit 9), all programs have a PAES that specifies candidate assessment and program evaluation activities. We have created a unified unit-level framework that supports individual programs to develop specific candidate assessments and program evaluations specific to their disciplines and professional communities. The Associate Dean for Planning, Research, and Evaluation, working with the Strategic Planning Committee and the Graduate Programs Committee, coordinates PAES development and ongoing activities.
Figure 2.02: Unit Assessment System
The conceptual core of the PAES is illustrated in Figure 2.03 (Program Assessment and Evaluation System). Based on explicitly articulated candidate and program goals (box 1), programs systematically collect (box 2) and analyze (box 3) data to determine whether (a) candidate progress and (b) program efficacy are satisfactory. Candidate and program level data provide the basis for changes/actions to improve candidate learning and program functioning (box 4), with ongoing assessment and evaluation to monitor the effects of these changes (box 5) in terms of explicitly articulated candidate and program goals (back to box 1). Each element identified in Figure 2.03 is operationalized in one or more sections (“tabs”) of the PAES binder, which is divided into two major sections—I. Candidate Assessment and II. Program Evaluation. We refer to these elements of the PAES throughout the narrative below and elsewhere in the Institutional Report. A program PAES binder (Program Binder) is in the Exhibits Room with the self-study and supporting material for each program. There is also a set of “tab” PAES binders (Tab Binder) in the NCATE section of the Exhibits Room that compiles material from each program, by tab, for easy reference. This permits the same information to be accessed two different ways: by Program PAES Binder and by Tab PAES Binder (Exhibit 14). We make reference to both binder types throughout the Institutional Report.
Conceptualization of the Unit Assessment System (UAS) began at the conclusion of our 2001 NCATE review as we began to look at the NCATE 2000 standards. The new standards made it clear that systematic assessment was the new driving force for NCATE institutions. Therefore, a concerted effort to develop a formal process of internal and external evaluations that integrates professional, state, and national standards became a priority for the unit. Table 2.01 (Unit Assessment System: Chronological Development, 2001-2006) chronicles the development and implementation of key elements of the UAS.
The Strategic Planning Committee (SPC) developed a Strategic Planning Document to reflect and shape a coherent picture of evolving unit priorities. The nine areas of the Strategic Planning Document include: Conceptual Framework, program evaluation, candidate assessment, technology, faculty/staff support, student services, program development, P-16 partnerships, and resources. Diversity is embedded in numerous areas, e.g., tracking candidate progress at the program level by gender and race/ethnicity to determine whether program assessment practices at key transition points affect groups differentially. In consultation with department chairs, program coordinators, faculty, and the Faculty Council, the Strategic Planning Committee initially conceptualized and designed the broad outlines of the UAS. It then developed the framework for what would become the Program Assessment and Evaluation System (PAES; Exhibit 9).
Although all program assessment and evaluation systems have been developed according to the unit-level PAES framework (Exhibit 9), individual PAES documents have followed somewhat different evolutionary paths. This has been by design, since we made a conscious decision to respect the disciplinary and professional integrity of the many programs in the unit. Program assessment and evaluation systems have been customized by program coordinators and program faculty based on published professional standards and in consultation with members of the respective professional communities.
The Unit Assessment System is coordinated across the various institutional levels with and within which the unit operates—California State University System (CSU), CSU Long Beach campus, individual programs, and P-12 partners. The three initial credential programs—Multiple Subject (elementary), Single Subject (secondary), and Education Specialist (special education)—have incorporated results from the CSU System-wide Evaluation of Teacher Preparation project surveys (Exhibits 7 & 8) into their individual Program Assessment and Evaluation Systems; data and analyses for these programs can be found in each Program PAES Binder and in the Tab PAES Binders in the Exhibits Room (see Tabs IIC & IIF). Advanced programs also coordinate at the campus level with the Program Assessment and Review Council (PARC), which oversees internal program review for all campus degree programs.
To ensure integration of the unit assessment system at the campus level, Student Learning Outcomes (SLOs) were integrated into the PAES. Candidate outcomes had previously constituted the core of program assessment practices; we explicitly adopted the SLO terminology to assure there would be no ambiguity about the identical focus of unit and campus assessment language as the campus prepares for WASC (Exhibit 15) review in 2009.
Finally, the relationships established in CSULB’s strong P-16 collaborations facilitate excellent communication with school districts and area community colleges. Representatives from the Long Beach Unified School District, our largest placement partner, serve on the Strategic Planning Committee, which initially conceptualized the UAS and was instrumental in its initial formulation. District personnel also serve on the Assessing Teacher Performance project (ATP; Exhibit 16) committee, a provost office-sponsored initiative to increase information flow with our P-16 partners. Assessment activities and results of unit and program assessments are shared with faculty, staff, candidates, advisory boards, school district personnel via various venues, including meetings and retreats.
Future developments in the UAS over the next 4 years are projected in Table 2.02 (Unit Assessment System: Projected Evolution Through 2010)
The UAS has elements addressing each of the six NCATE standards, corresponding California Commission on Teacher Credentialing (CCTC) standards, professional standards, and the unit Conceptual Framework (Exhibit 5). Figure 2.02 (Unit Assessment System) depicts the interconnectedness among these components. Each PAES (Exhibit 9) identifies key candidate proficiencies, or Student Learning Outcomes (see Tab I.A). These proficiencies comprise the knowledge, skills, and dispositions that provide the basis for candidate assessments in the program. Candidate proficiencies are derived from state standards, professional standards, and the Conceptual Framework. All programs have articulated candidate proficiencies that relate to multiple key elements of the Conceptual Framework: Growth and Learning; Social Responsibility; Diversity; Service and Collaboration; School Improvement; and Research, Scholarship, and Evaluation. Tab I.B in each PAES outlines how candidates are assessed at key points in their progress through the program. The assessments are explicitly tied to the Student Learning Outcomes identified in Tab I.A.
All unit programs have assessments/measures of candidate performance keyed to four transition points: (1) Admission to program; (2) Advancement either to student teaching (for initial teacher preparation) or to candidacy (for advanced [continuing] teacher preparation programs and programs for other school personnel); (3) Completion of culminating experience (student teaching for initial teacher preparation; thesis or comprehensive examination for advanced [continuing] teacher preparation programs and programs for other school personnel); and (4) Program completion (credential recommendation or degree awarded).
All programs use multiple measures to gauge candidate progress. It is impossible to be admitted to (or rejected from) a program or to progress (or fail to progress) based on a single data point. Programs use combinations of course grades (and cumulative averages), performance on signature assignments and projects (e.g., portfolios, reports, case studies, research papers, class presentations), and performance in fieldwork, where applicable, in order to determine candidate progress and program completion. For a more complete discussion refer to Transition Points in Initial Teacher Preparation Programs, Advanced (Continuing) Teacher Preparation Programs, and Programs for Other Professional School Personnel (Exhibit 10).
The three initial teacher preparation programs require passage of state statute-specified standardized tests either before admission, prior to advancement to student teaching, or before successful completion of the program. All programs also use checklists of teaching competencies and dispositions tied to the California Commission on Teacher Credentialing (CCTC) Standards for Program Effectiveness and other professional standards (e.g., Council for Exceptional Children for Education Specialist and California Teaching Performance Expectations for Multiple and Single Subject) in order to assess candidates during and at the end of student teaching.
As a part of the reform of teacher preparation prompted by Senate Bill 2042 in 1998, the State of California requires that candidates in Multiple and Single Subject initial teacher preparation programs complete a teaching performance assessment. The Multiple Subject and Single Subject programs utilize the state-developed California Teaching Performance Assessment (CA-TPA; Exhibit 11), which consists of four separate Tasks, each focusing on a different aspect of teaching practice. Collectively, these Tasks measure aspects of the Teaching Performance Expectations (TPEs; Exhibit 12), which are a set of performance indicators that describe what all California beginning teachers need to know and be able to do to qualify for Preliminary Multiple Subject and Single Subject credentials. Education Specialist Level I Credential programs are exempt from the CA-TPA.
As a comprehensive assessment, the CA-TPA (Exhibit 11) is linked to the California P-12 Content Standards as well as to the California Standards for the Teaching Profession (CSTP; Exhibit 17). Each of the Tasks contains a set of “scenarios” or “steps” (i.e., assignment directions) that engages candidates in thinking about curriculum (content), instruction, management and assessment for the whole class, individuals, English learners, and special needs students. Each of the Tasks incorporates specific teaching performance indicators described in the TPEs. Reflection is built into each Task.
In Task 1, Principles of Content-Specific and Developmentally Appropriate Pedagogy, candidates are given a classroom scenario, including an assessment plan, for which they do a written analysis. Given additional classroom context information, including a 5-day lesson sequence and information on two focus students (an English learner and a special needs or other type of challenging student), candidates identify instructional strategies, assessments, and adaptations for the two focus students. They are prompted to analyze the lesson plan, assessments, and adaptations within the framework of alternatives to the choices they initially made. In Task 2, Connecting Instructional Planning to Student Characteristics for Academic Learning, candidates construct a profile of the class (identifying two focus students), develop a lesson plan within a larger unit, indicate how they would adapt the lesson for the two focus students, and reflect on the process. In Task 3, Classroom Assessment of Academic Learning Goals, candidates construct a class profile (identifying two focus students), develop and implement a class assessment (including adaptations for the two focus students), analyze the data from the assessment and discuss its uses, alternatives, and relationship to student achievement (including special attention to the two focus students), and reflect on the assessment implementation and impact of the lesson on student learning. In Task 4, Academic Lesson Design, Implementation, and Reflection after Instruction, candidates construct a class profile (including two focus students), construct a unit of study grounded in state academic content standards and accounting for the learning environment and classroom management concerns, adapt the lesson for the two focus students, videotape the lesson, write an analysis after viewing the videotape, and reflect on the process.
In anticipation of high-stakes implementation of the CA-TPA (Exhibit 11) that begins in July 2008, the Multiple Subject and Single Subject programs have systematically embedded Tasks 1-4 of the CA-TPA in program courses, which permits use of the instruments for both formative and summative assessment purposes (Table 2.03: CA-TPA Tasks 1-4 in Initial Teacher Preparation Program Courses). Tasks 1-4 function as benchmark assessments of student progress at different stages in the program by virtue of their place as assignments in key courses. Currently, faculty are being trained to use both the Tasks and the state-developed 4-point scoring rubric. When the system is fully operative, programs will be able to collect data, using common assessments, on each candidate at four different moments as they progress through the program. The Multiple Subject program is using an electronic portfolio system (utilizing TaskStream) for scoring and storing candidate performance. The Single Subject program has used the four TPA Tasks in program courses for several years, has lately implemented the state-developed scoring rubric, and is beginning systematic collection of scoring data (SPSS, Excel and MS Word spreadsheets are used for data management). We draw on pilot data collections in our Standard 1 discussion of candidate knowledge, skills and dispositions.
Advanced teacher preparation programs and programs for other school personnel use multiple indicators such as GPA, test scores, and writing samples to identify promising candidates for teaching and other school professional roles, additional criteria to determine advancement to candidacy and satisfactory progress, and successful completion of either a comprehensive exam or thesis prior to program completion. All programs have key assessments related to a specialized professional competency. For example, the School Psychologist program requires candidates to complete two “problem-solving” case studies, wherein candidates conduct thorough student assessments then develop, implement, and systematically monitor an intervention. The newly developed Dual Language Development Master's program includes several “signature assignments” designed for students to demonstrate understanding of international and cross-linguistic perspectives on educational development. In the Reading and Language Arts Master's program (which includes the advanced reading credential), students must conceptualize and define a key issue in literacy education, write a literature review reporting the “state of knowledge” about the topic, identify what questions or issues cannot be resolved from the published literature, and make concrete policy and practice recommendations to a school or district based on the review. See Tabs I.B and I.C of individual program PAES binders for details regarding key assessments and the timing of their administration.
There are two challenging assessment issues emerging. One involves determining the ability of program candidates to affect student learning and other important outcomes during their preparation programs. Both the Multiple and Single Subject programs are piloting procedures in which student teachers, in collaboration with university supervisors and cooperating teachers, design pre and post assessments to accompany a unit of study they design and teach. The critical task is to determine the extent to which candidates have been successful in accomplishing stated goals for students by the end of the unit of instruction. These pilot procedures are addressed in Standard 1, and student work samples are in the Exhibits Room.
A second challenge is following preservice candidates into their practice as new teachers to assess their ability to affect student learning after leaving the program. This assessment essentially asks twin questions: What impact do new teachers have on their students’ learning and What is the relationship of their teaching performance to their preparation program experiences? The CSU System-wide Teacher Preparation Evaluation project has started to explore the possibility of collecting data to analyze the learning of P-12 students in classrooms taught by CSU-prepared teachers. Graduates of unit initial programs will be included in the data collection, and analyses will be made available to the unit. During 2005-2006, with assistance from Academic Affairs, the unit funded the Faculty in Schools research project, which included piloting an approach to investigating the learning progress of a sample of Long Beach Unified School District students enrolled in elementary classrooms taught by unit initial credential program graduates (Exhibit 18: Faculty in Schools Project Report). Some analyses of the data have been completed, and a follow-up investigation is scheduled for spring 2007.
Assessment measures at key transition points are displayed in detail in the document Transition Points in Initial Teacher Preparation Programs, Advanced (Continuing) Teacher Preparation Programs, and Programs for Other Professional School Personnel (Exhibit 10) and data is reported in Standard 1. They are displayed in PAES binder Tabs I.B and I.C for each program. Programs follow various procedures for what happens when candidates do not meet expectations at key assessment points. These are detailed in Tab I.E. In general, programs have established committees to review individual cases when candidates do not meet expectations. For example, in the Multiple Subject Program, the Admissions and Standards Committee reviews cases and hears student appeals. In the Single Subject Program, a similar review process is used in the Petition Committee. University policy ensures that all students have the appropriate due process.
The unit checks for assessment bias by having programs monitor candidate progress through key assessment points. These data are reported in Tab I.D of each program PAES. We have found no evidence of ethnic or gender bias in assessments, as reflected in data showing student success at benchmark points in the program, aggregated and broken out by significant demographic subgroups (i.e., ethnicity and gender).
The key performance assessment measures used by the Multiple Subject and Single Subject credential programs are based on the California Teaching Performance Assessment (CA-TPA; Exhibit 11) developed by Educational Testing Services (ETS) under contract to the CCTC.
An ETS analysis of the measure that employed 42 trained assessors and 104 teacher candidate responses (Scoring Analysis for the Field Review of the California Teaching Performance Assessment; Exhibit 19) found that the CA-TPA is reliable and free from apparent bias. Overall reliability across the four TPA Tasks (when each task is scored by two assessors) is .87, with reliabilities for individual tasks ranging from .63 to .83. On 3 of the 4 TPA tasks, 4% of the scores were more than one score point apart; on the remaining task, 9% were more than a score-point apart. This suggests an acceptable degree of reliability, although also indicates that 4% to 9% of student responses will produce substantially different scores between 2 independent assessors and that adjudication must be part of this assessment. The ETS study also found that there were no score differences by White vs. non-White candidates, suggesting lack of ethnic bias at least with respect to White and non-White candidates. The mean score across all tasks was 2.38 on a 4-point scale (range = 2.16 to 2.50), suggesting a moderate degree of difficulty. The CA-TPA's validity was established through extensive content analysis, grounded in the California Standards for the Teaching Profession (California TPA Field Review Report; Exhibit 20).
The Multiple Subject and Single Subject programs are currently conducting reliability studies on locally collected TPA data in 2006-07. In the Education Specialist program, validity of the advanced field study Competency Checklist (see PAES Tabs I.B and I.C, item C.5) has been established by a panel of experts (practitioners and researchers) who compared the instrument to standards set by CCTC and the Council for Exceptional Children. Inter-rater reliability for the instrument, however, was low when field-tested (see PAES Tab II.C). Plans are underway to re-train observers and conduct another field trial in 2006-07.
The School Psychology credential program assessment is shown here to illustrate how an advanced program monitors and assesses candidate progress. Faculty in the School Psychology program developed an assessment process that focuses on three assessment categories: candidates, faculty, and program. Assessment of candidates consists of the following:
The PAES binders have complete information about candidate assessment processes in Tabs IA-IE for each advanced program (See Program PAES Binders or Exhibit 14).
In addition to candidate performance, the unit monitors 6 key areas: Resources and Technology, Personnel, Budget, Facilities, Leadership and Authority, and Diversity. The unit's FileMaker Pro data management system, which will become fully operative in fall 2007, will supplement the campus Common Management System (CMS) data management system. The unit system makes possible regular data collection, analyses, and reporting. The Administrative Services Manager (ASM) and her two staff members monitor staffing and all fiscal operations. In addition to providing quarterly reconciliation reports for the dean, they develop detailed annual fiscal reports (Exhibit 21). Unit centers make annual reports to the dean (Center for Language Minority Education and Research, Exhibit 22; Center for Collaboration in Education, Exhibit 23; Center for Educational Technology and Learning, Exhibit 24). The dean submits an annual report on unit operations to the Provost (Exhibit 25). For the 3 programs in the unit that are not located administratively in the College of Education, unit-level data are obtained by communication among the ASM offices and Dean's offices in the College of Education and College of Health and Human Services, and the Credential Center in the College of Education. See Standard 6 for a description of organization and governance of the two colleges that makes possible this regular collection of information.
At the university and college level, there is a great deal of attention devoted to supporting candidate success. We are aware that one of the most important operational aspects of the unit is providing candidates with timely and accurate information relevant for successful program completion. Several years ago, occasional problems were reported by graduate candidates, and more frequently by undergraduate Liberal Studies students, that they could not get advising appointments in a timely way. In order to assess the nature of the problem and what action was needed, as well as to gauge our unit effectiveness, we developed an online unit Advising Survey and piloted it in spring 2005 and again in 2006. Candidates at various stages of program completion (as determined by programs) are asked to complete the survey online. Results are reported for the unit as a whole and by program. Discussions following the two pilot years led to specific steps to improve advising available to students. One result is that procedures for scheduling of advising in the Liberal Studies Department were changed. In addition, in fall 2005 the department hired an additional full-time staff advisor. The Advising Survey will be administered again in spring, 2007, using a sampling plan to be developed by the Strategic Planning Committee.
Another recent major operational improvement has been to accelerate centralization of advising services in the three initial credential programs; the Teacher Preparation Advising Center opened in summer 2006. The Center houses most program functions for the initial credential programs. Feedback from formal as well as informal assessments and evaluations played a large role in these programmatic changes.
The Technology Coordinating Committee is charged with monitoring the technological needs of the unit and advising administrators and department. Refresh depends on such considerations as type of use, frequency of use, and age of equipment. A running inventory of computer equipment (Exhibit 26) is maintained by the staff computer systems specialist.
The unit collects, analyzes, reports, and manages a wide range of data regularly and systematically that is linked to the Conceptual Framework, NCATE Standards, and national, state and professional standards. Table 2.04 (UAS and Relationships with Various Data Systems) illustrates the linkages between the unit UAS and various data systems across and between programs, CSULB, the CSU system, the Commission on Teacher Credentialing, and our P-12 partners. Data flows continually between these sources and the UAS. Table 2.05 (Components of the Data Management System) shows the various components of the data management system and how the data is managed at the unit level.
Unit-initiated surveys are developed through a process illustrated in Figure 2.04 (Unit Survey Process). Although typically initiated by the Strategic Planning Committee or a member of the unit Leadership Committee (e.g., Dean, Associate Dean, Department Chair), any staff or faculty member can propose conducting a survey to collect data on some determined need. The process requires consultation with the Strategic Planning Committee, the Leadership Committee, or both. As discussed in Standard 6, these groups are responsible for setting priorities in the unit overall.
The unit utilizes two data management systems for administrative purposes. The campus-wide Common Management System (CMS; developed by PeopleSoft) provides a wide array of data on students, faculty, and a range of university operations. The offices of Enrollment Services and Institutional Research and Assessment support university planning and the campus commitment to data-based decision making. To extend the campus system and enable more fine-grained analyses of unit needs, we began developing a unit data management system utilizing FileMaker Pro software (FileMaker Pro Upgrade Project Plan [Exhibit 27] and FileMaker Pro Upgrade Project Specifications [Exhibit 28]). The basic structure is that of a hub containing common student information data points, with radiating spokes for each office or program’s specific data needs. This database becomes fully operative in fall 2007 for CED programs. The following academic year, once the system has been tested and is stable, we will incorporate the Affiliated Programs residing in CHHS.
We monitor the operational health of the unit through regular meetings of the unit as a whole and through key groups: Administrative Council (dean, associate deans, department chairs, office supervisors, faculty council chair), Leadership Committee (dean, associate deans, department chairs and assistant chairs, administrative services manager), Faculty Council (faculty, dean), and Strategic Planning Committee (faculty, dean, associate deans, department chairs, student representative, community member).
The unit collects assessment and evaluation data from candidates and graduates in several ways. What follows is a brief description of various data sources. We draw on these data sources in the narratives for other standards in the Institutional Report. Use of data sources by programs is reported more thoroughly in individual Program PAES Binders. Quantitative data from survey processes are analyzed utilizing SPSS and Excel, and text data is analyzed using qualitative data analysis techniques. Summary reports from these data are provided to chairs and program coordinators who use the information for program improvement. Program self-studies, supporting evidence, and PAES binders are available in the Exhibits Room.
Since 2000-2001 initial teacher preparation programs have utilized data on candidates and graduates provided by the CSU Chancellor’s office through the CSU System-wide Evaluation of Teacher Preparation (Figure 2.05: The CSU Mosaic) project. The first component of the CSU System-wide Evaluation of Teacher Preparation project is an Exit Survey (Exhibit 6) administered online to all preservice candidates at the completion of student teaching. The Exit Survey was piloted in 2004-05. The second component is a follow-up Survey of Recent Graduates of Teaching Credential Programs (Exhibit 7) in the California State University System, which is administered online annually and completed by program graduates at the conclusion of their first year of teaching. The third component is a follow-up Survey of Employment Supervisors of Program Graduates (Exhibit 8), which is also administered online annually and completed by the employment supervisor working most closely with program graduates during their first year of teaching (principal, vice-principal, department chair, curriculum coordinator, etc.). Each survey asks respondents to consider a wide variety of teaching performance behaviors demonstrated by program candidates and graduates. The CSU system provides detailed annual reports to programs that contain both program-specific data and data aggregated across programs in the system (Exhibit 29). Data permit longitudinal studies as well as comparisons to the CSU system as a whole. We draw on these sources extensively in our discussion of candidate performance in Standard 1.
Although the unit has used the annual reports each year, use has not been as systematic as desired. Beginning in 2006-07, the unit implemented an explicit set of actions for annual program improvement for the three initial credential programs utilizing data from these annual reports. This explicit plan provides a structure for faculty to analyze, report on, and use data from the three surveys, in conjunction with other data sources, to develop substantive program improvement actions (Guidelines for Actions Around CSU Evaluation Data [Exhibit 30] and Worksheet for Analyzing CSU Evaluation Survey Data [Exhibit 31]).
The initial teacher preparation programs have expanded data collection recently by utilizing two other sources of evidence for candidate performance. First, they have started developing data collection processes for supervisor and master teacher evaluations of candidates during student teaching. Preservice candidates are observed by university supervisors and master teachers during student teaching. Mid-term (formative) and end of term (summative) evaluations are completed on program(refer to program self-studies and supporting materials in Exhibits Room). These forms are based on the California Teaching Performance Expectations (TPE); Exhibit 12) and assess student teacher performance in on a wide variety of teaching behaviors across 5 teaching domains. Although we have used the evaluation forms in their current form for a number of years for candidate assessment, we are in the beginning stages of aggregating data for more focused questions of program improvement. Second, as we gear up for high stakes implementation of the California Teaching Performance Assessment (CA-TPA; July, 2008), the Multiple Subject and Single Subject programs have started data collection procedures on candidate performance on each of the four Tasks. The Multiple Subject program utilizes TaskStream for data management for these processes; the Single Subject program uses Excel and MS Word spreadsheets. Candidate performance data from these sources is also reported in Standard 1. (Note: the Education Specialist program is exempt from the CA-TPA asnd is not part of this new data collection initiative.)
Finally, another kind of follow-up study was piloted by the unit in 2005-2006 that looked at graduates of the Multiple Subject and Single Subject programs who were beyond their first year of teaching (Faculty in Schools Project, Exhibit 18). The pilot sample were program graduates in their third year of teaching and who were employed in Long Beach Unified School District. The methodology included administering a survey developed by the CSU system and based on the CSU Survey of Program Graduates mentioned above, focus groups, individual interviews, video-taped classroom observations that participants viewed with researchers and commented on, and a statistical analysis linking student achievement to teachers (and their credential preparation program). The methodology looks promising for providing insights into graduates’ teaching performance and their impact on student achievement; we are doing a second year of the project in 2006-07 with a new cohort of third year program graduates.
In 2006-07 advanced (continuing) teacher preparation programs and programs for other school personnel are engaged in identifying 1-3 program Student Learning Outcomes to be assessed during the year. Each program is piloting a plan for assessing Student Learning Outcomes measured in signature assignments embedded in program courses (Program Annual Assessment Plan; Exhibit 32). An annual report will be made to the University Assessment Coordinator in June (Program Annual Assessment Report for AY 06-07; Exhibit 33). Other types of data are collected from candidates in advanced programs and programs for other school personnel on a program-by-program basis. Data are collected, summarized, analyzed, discussed among program faculty, and used for determining program improvement actions (see Standard 2, Element 3).
To date we have not approached follow-up survey studies of advanced programs and programs for other school personnel with a consistent model. This has been done by some programs, but not by all. All programs, however, have been able to use data from the successive administrations of the Indicators Survey (see below) for information on their graduates. As indicated in Table 2.02, we plan to initiate follow-up data collection efforts in these programs beginning in 07-08.
The unit utilizes data collected through unit-wide surveys, interviews and focus groups, and document analysis. Data are analyzed and reported at the unit level, and are also made available for programs for individual program analysis. Specifically, major datasets include:
Indicators survey. We administer a unit-sponsored Indicators Survey (Exhibit 34) in four-year cycles (1997, 2001, 2005) to graduates of initial credential programs as well as to candidates in and graduates of advanced programs and programs for other professional school personnel who are employed in the Long Beach Unified School District (Indicators Survey Results, 1997-2005). The CED core values and priorities are the basis for the Indicators Survey. The key indicators examined in the survey support the unit’s goal to provide high quality professional training by preparing competent professionals with the knowledge and skills required to practice their professions in diverse urban communities.
Advising survey. The Advising Survey (Exhibit 35), administered unit-wide and online, was developed in collaboration with all program coordinators and piloted in 2005 and 2006. The next administration is scheduled for winter 2007. It asks students about the availability and quality of advising in their individual programs. Data are analyzed at the unit and individual program levels.
Diversity survey. Attention to diversity has been at the forefront of the unit's mission for more than a decade. The Diversity Survey (Exhibit 36) was developed in order to gauge faculty perceptions of the degree to which diversity (in terms of race/ethnicity, language, gender, and special learning conditions) is addressed in unit practices and policies. We are particularly concerned about the extent to which we recruit and retain diverse faculty and candidates and whether candidates are provided with the sorts of experiences that will prepare them for the diverse classrooms and schools in which they will work. The Diversity Survey was first administered in fall 2006, and will be re-administered every two years.
Dispositions survey. The unit mission statement and Conceptual Framework identify 12 key dispositions that are at the core of our practice, e.g., intellectual, personal, and interpersonal growth for all students; socially responsible leadership; valuing diversity and preparing students for a diverse world. The Disposition Survey (Exhibit 13) was administered to candidates enrolled in culminating courses in fall 2006 to answer: (1) How important candidates felt this disposition was to them and (2) To what extent the candidate's program prepared him/her to act on this disposition. This survey has a 2-year cycle for administration.
Technology survey. Since 2000, candidate instructional technology competency has been a priority for program and unit strategic planning. Campus, regional, state and national initiatives and grant opportunities have supported the unit’s efforts in terms of: faculty development, curriculum development and implementation, and technological infrastructure. A series of surveys have been administered and analyzed to optimize coordinated efforts. Surveys have measured faculty computer use, faculty technology knowledge and application, faculty technology needs, incorporation of technology into teaching and learning, and student technology competency. As a result, a number of improvements have been instituted, including establishment of a unit technology coordinating committee and department technology committees, coordinated decision-making about technology resource allocation, hiring of a web master, technology coaching and workshops at various levels, and curriculum modification. The most recent form of the Technology Survey (Exhibit 37) was administered in fall 2006.
Focus groups and interviews. number of programs incorporate focus groups and interviews in their system of follow-up of students. On an ongoing basis, focus groups are conducted with Integrated Teacher Education Program (ITEP) students who transfer from the TeacherTRAC Program at Cerritos College. During 2005-2006, as part of the Faculty in Schools project, focus groups were held with graduates of our credential programs who were in their fourth year of teaching. The data were summarized and discussed at university, department, and college meetings. The Educational Administration Program conducts exit interviews of graduating candidates.
Document analysis. Archival data from existing data sources (i.e., university database-PeopleSoft; various databases and student files in the CED Credential Center; and data maintained in the divisions and program offices) have been examined for assessment relevant data. The University Office of Institutional Research and Assessment regularly provides us with detailed enrollment figures, tables, and reports. The university webpage has department on-demand reports that contain extensive student data, course data, etc. Data from existing assessment documents, such as candidates’ portfolios and supervisors’ evaluations, are regularly analyzed. Department and college committees, such as the Graduate Programs Committee, frequently access these data for monitoring programs and students. Analyses are reported to chairs and program coordinators and are added to PAES binders. The CED dean’s office, the Associate Dean for Planning, Research and Evaluation, as well as the Associate Dean serving in the role of Chair of the Graduate Programs Committee and Chair of the Strategic Planning Committee coordinate summarizing and analyzing data for dissemination and discussion.
The university has formal appeal processes for students to follow when necessary. Formal candidate complaints must be written and signed and must comply with university grievance procedures.Programs keep a record of formal complaints and their procedures for handling complaints as part their PAES (see Tab I.F). Since programs have started keeping records (most for the past 5 years; others as far back as 1998), two formal complaints have been lodged (both in the Education Specialist program and about the same issue). The problem was resolved at the program level. Faculty keep records regarding the incident in a confidential file and confer with the Associate Dean as appropriate. If a problem is not resolved, the next step is a formal grade appeal or the grievance process, per existing university policies (see URL above). In the past three years there have been 5 grade appeals (Table 2.06: Grade Appeals, 2003-2006). Programs also have developed informal procedures for handling issues that do not rise to the level of “formal complaint.” These are described in PAES Tab I.F. The unit also maintains a suggestion or “complaint” box in the lobby. Comments are given to the dean who then directs them to the appropriate faculty or staff member.
University, unit and program administrators, project directors, faculty, staff, candidates, and P-12 school representatives have multiple uses for assessment data. Therefore, the fact that the UAS is multifaceted and integrated is beneficial to both candidate and institution.
As part of the ongoing review of the unit, the dean’s office produces a Dean’s Annual Report (Exhibit 25) based on data gathered throughout the year. These results are shared annually with faculty in written form and summarized in a report at the fall unit meeting or retreat. The unit-approved template for the PAES incorporates extensive program level data that faculty in initial and advanced programs use annually to assess coursework development, program strengths and weaknesses. Faculty input and approval of changes are obtained during faculty, committee, and course meetings (see committee agendas & minutes). Changes in courses are evidenced in the program curriculum and syllabi. This IR embeds examples of data-driven decision-making and program improvements within each standard’s narrative.
At the candidate level, all programs set criteria for satisfactory candidate progress, and program faculty monitor individual candidates’ progress. At key assessment points, students who are not making satisfactory progress have an individual conference with a faculty member or the program coordinator, at which time a suitable intervention or course of action for improving performance is determined. A faculty member/coordinator monitors the student progress until either (a) the candidate improves his/her performance and successfully satisfies program criteria as determined by key assessments or (b) the candidate exits the program. (Refer to Tab I.E of each Program PAES.)
With respect to faculty performance, evaluation of tenured and tenure-track faculty is governed by the university's Retention, Tenure, and Promotion policy (Exhibit 38). See Standard 5 for a fuller discussion. Within the framework established by the university policy, each college has its own set of procedures. A collection of policies and documents pertaining to faculty in the College of Education (Exhibit 39) is available on the university website. Evaluation of full-time and part-time lecturers is governed by the University Lecturer Evaluation Policy (Exhibit 40) Each program also has a set of procedures, within the framework of university and college policies, to review faculty teaching and student feedback about the quality of teaching in the program. Tab I.H of each Program PAES binder describes how each program uses faculty performance data to improve teaching. Unit-wide, the dean (and department chairs for their respective departments) reviews all student evaluations for all courses offered each semester. Faculty with less than satisfactory student evaluations confer with the dean and department chairs to determine whether there are aspects of their teaching that require improvement.
We use data at both the program and unit levels to inform and monitor changes. At the program level, Tab II.G of each Program PAES binder describes decisions and actions resulting from program evaluation data. All programs have a process whereby faculty (and in some cases, faculty and program advisory committees) meet to examine program level data and consider possible corrective or improvement actions. Typically this is an agenda item at scheduled or called faculty meetings or advisory committee meetings.
unit has implemented a process whereby initial teacher preparation program faculty systematically analyze results from the CSU system-wide surveys of exiting candidates, graduates and employers for program improvement activities. The unit receives annual detailed reports from the Chancellor's Office, which are distributed to and discussed among the dean, associate deans, department chairs, and program coordinators. Program coordinators discuss findings with program faculty who identify areas for program improvement and devise appropriate action plans, which include follow-up and evaluation. The Guidelines for Actions Around CSU Evaluation Data (Exhibit 30) describes this process in detail; see also the worksheets (Exhibit 31) developed to facilitate this process.
Among advanced programs and programs for other school personnel (neither of which are included in the CSU System-wide Evaluation of Teacher Preparation project), programs are implementing the recently instituted campus process for assessing and reporting Student Learning Outcomes (SLOs). This is a WASC-driven process whereby programs establish and assess explicit student learning outcomes, then analyze data on candidate performance for program improvement activities. See the Program Annual Assessment Plan (Exhibit 32) and Program Annual Assessment Report (Exhibit 33) documents for additional details. The Assessment Plan and Assessment Reports will be part of the PAES as they come on line. In the case of graduate (Master's) programs, the Assessment Plan and Assessment Report contribute to the campus' internal review of graduate programs.
At the unit level, there is also discussion and initiation of changes based on systematic analysis of centrally-gathered data regarding facilities, budgets, technology infrastructure, and other aspects of unit operations. Individuals or committees have responsibility for maintaining databases on these operations and providing timely information that will guide decision-making and evaluation to the dean, the Leadership Committee, and the Administrative Council. When a need is identified (e.g., office space, allocation of technology funds, cuts in operational costs), pertinent data are presented at Leadership Committee and/or Administrative Council meetings to inform discussions of how to address the need. Over several meetings, options are discussed, plans determined, and actions taken. Relevant data are then used in subsequent meetings to determine the effectiveness of actions that were taken.
At the program level, faculty have made numerous changes based on program data. Below are several examples. Fuller descriptions appear in the Program PAES binders in the Exhibits Room (see tab II.G).
Data from the CSU Survey of Graduates (Exhibit 7) consistently indicated that Single Subject program graduates reported the following areas needing improvement: preparation for working with English Learners and special needs students, using technology for instruction, knowledge of adolescents as learners, and managing classrooms. As a result, program faculty increased attention to these topics in the secondary education courses. As an example, major amounts of content and fieldwork experiences allocated to working with English learners were integrated into EDSE 457, which became the principal program vehicle for this element of teacher preparation. All students in EDSE 457 now do a case study of an English learner as part of their fieldwork requirement. (see Single Subject Program PAES, Tab II.G)
The Single Subject faculty discussed data from the CSU Exit Survey (Exhibit 6) at a faculty workshop on May 6, 2005 and May 13, 2005 that suggested students felt the need for more classroom management instruction. (see Single Subject Program PAES, Tab II.G)
A trend in the CSU Exit Survey (Exhibit 6) data indicated Education Specialist candidates wanted greater coverage of Moderate/Severe disability issues across the program. Beginning in summer, 2006, the assessment and evaluation course has been co-taught by two faculty members, one with expertise in the Mild/Moderate and the other in the Moderate/Severe areas. (see Education Specialist Program PAES, Tab II.G)
The Education Specialist faculty meet with advisory committee to discuss data from the CSU Survey of Graduates (Exhibit 7). The value of these data are attenuated by the small numbers of respondents, but the data are still discussed and trends over time noted. The surveys have revealed several areas that program faculty have discussed with the advisory committee, such as need for more subject matter preparation for Education Specialist candidates and recruiting, attracting, and maintaining more culturally and linguistically diverse credential candidates. (see Education Specialist Program PAES, Tab II.G)
The Multiple Subject program reported Multiple Subject data from the CSU system-wide evaluation at the Sept. 5, 2005 meeting of the Dept. of Teacher Education. The data suggested two weaknesses that program faculty thought important to target: meeting the needs of learners with special needs and computer-based teaching in the classroom. (see Multiple Subject Program PAES, Tab II.G)
The Graduate Reading Program, based on surveys of students and graduates, discovered that the linguistic content of courses was inadequate, secondary candidates felt the program was far too focused exclusively on elementary, and the course Foundations of Literacy Research was offered too late in the program to be of maximum benefit to students. As a result, the linguistics of reading is being taught by a trained linguist using a text written by a leading scholar of language study. Program courses are bringing in more materials related to secondary reading. Foundations of Literacy Research will be piloted in the first semester for the off-campus cohort in fall 2007. (see Graduate Reading Program PAES, Tab II.F)
Feedback from candidates in the Early Childhood Education Master’s program prior to 2004 indicated they did not feel prepared to integrate technology. Faculty subsequently made program changes that included course revisions and new candidate assessments related to coursework and fieldwork. The 2004-2005 ECE exit survey showed improvement in program completers’ ratings. (see Early Childhood Education Program PAES, Tab II.G)
In the Library Media Teacher program, based on student surveys (see PAES, Tabs II.E/F) and advisory council input, the program was expanded to provide a master’s degree. The program developed an interdisciplinary studies program option, followed by the creation of the Option in School Librarianship within the Master of Arts in Education degree. (see Library Media Teacher Program PAES, Tab II.G)
In the School Counseling program, surveys and interviews revealed there was excessive overlap in two career counseling courses. After reviewing the curricula of nationally-known counseling programs, program faculty consolidated the courses into one, which had the added benefit of creating room for a course that dealt with topics that students said were not given sufficient attention, special education and crisis management and counseling. (see School Counseling Program PAES, Tab II.G)
The Facilities Coordinator (FC) maintains databases on a variety of unit facility operations (Exhibit 41). In particular, she keeps a database on space utilization where she records such items as faculty searches, departures (retirements, resignations, leaves of absence), room assignments, grant activity as it relates to space needs, temporary assignment, etc. When a need is identified for additional space, the Facility Coordinator can easily provide current information to the Dean and the Leadership Committee. For example, when we identified a need to centralize student services for the three initial credential programs we were quickly able to identify how space could be re-assigned and reconfigured using the Facility Coordinator's databases. The result was establishment of the Teacher Preparation Advising Center (TPAC) in a classroom-size space with an accompanying set of faculty and staff offices across the hall. In addition, as faculty and staff grew, we efficiently located underused space (that was being used for observation and storage) and converted it to 5 new offices and a workroom.
The Administrative Services Manager (ASM) maintains up-to-date budget information for CED programs, offices, and operations. These data are regularly presented and discussed at Leadership Committee, Administrative Council, and Faculty Council meetings. This information is critical for deciding budget and new hire allocations to departments. We relied heavily on these data several years ago when a severe state-wide budget crisis made it necessary that we take unpalatable steps such as substantial increases in class sizes. Timely and accurate budget information made it possible to determine efficiently the scale of the class-size increase required to balance the budget.
The Indicators Survey (Exhibit 34) represents a unit-wide effort to use data for program improvement. The first Indicator Survey results of over 900 Long Beach Unified School District educators were reported at a college meeting in 1998. Our findings were sobering: Graduates of our programs were only modestly satisfied with the extent to which our programs helped prepare them for their professional challenges. At the unit level, the survey reflected two important and disturbing findings: First, graduates of our programs were less positive about their programs than were graduates of other institutions. Second, minority (i.e., self-identified non-Anglo) graduates were more negative about our programs than were Anglo graduates on about 40% of the items. A group of faculty from the Department of Teacher Education conducted follow-up focus groups with survey participants and discovered that a part of the problem, with respect to the first finding, was that graduates of our programs felt they had insufficient field experience prior to student teaching. Since 1998 we have substantially strengthened the preservice fieldwork requirement of the multiple subject program, adding a graduation requirement of 120 hours of early experiences in university-approved urban classroom settings through the SERVE program (see also SERVE Program Reports; Exhibit 42). With respect to the second finding (minority candidates were particularly dissatisfied with our program), we have kept issues of candidate diversity at the forefront of our planning and discussion. We have significantly increased the diversity of our faculty, staff, and candidate pool since 1997, making the most noticeable progress in the admission of students to the Multiple Subject program. Programs are asked to monitor admissions and progress through programs, broken out by significant demographic subgroups (see PAES binders, Tab I.D). We have also put diversity topics on college meeting agendas to discuss whether our programs and the unit as a whole are responding to the needs of diverse constituencies. Since 1997, two more Indicators Surveys have been conducted (2001, 2005) using nearly identical sampling plans to maintain comparability. Results have been reported at college meetings; the Strategic Planning Committee will make recommendations based on these data at the end of this academic year or beginning of next year.
Sharing of assessment data with candidates, faculty, and other stakeholdersis done systematically at the unit, department, and program level. The Long Beach Education Partnership, as well as other strong P-14 collaborative efforts, plays a key role in enhancing the usual unit and program activities to share assessment data. One concrete example of this is the CSULB Assessing Teacher Performance project (Exhibit 16), an initiative formed and funded at the Provost level in 2004. The ATP steering committee is comprised of representatives from the campus, Long Beach Unified School District and local community colleges. The group meets once a semester and reviews summary reports and data from CSULB, LBUSD, Long Beach City College and Cerritos College.
The most widely disseminated and shared data in the unit are from the CSU System-wide Evaluation. In the past six years, the CSU evaluation, consistent with No Child Left Behind and other federal legislation, has emphasized data collected for initial teacher preparation programs. The dissemination and use of the CSU evaluation data are reported each spring and placed on the system website—the CSU Annual Teacher Preparation Report (Exhibit 43). Data on advanced candidates and program evaluation are usually collected at the program level. At times, however, data are collected across all programs and the review and action taken may be at the unit level. For example, year to year comparisons of admissions to graduate programs is likely to be reviewed at the Leadership Committee in discussions of enrollment management, and at the Graduate Programs Committee in discussions of recruiting. The Indicators Survey, described above, is an example of a unit wide survey periodically administered to graduates of both initial and advanced programs.