CAEP Annual Reporting Measures

Relay classroom Reporting

Impact Measure #1: Impact on PK-12 learning and development

As part of its CAEP accreditation, Relay engaged in a study on the impact of its completers on PK-12 learning and development and benchmarked its completers’ performance against other teachers working in the same school district. Relay found that its completers had statistically significant higher average value added scores in math in 2016, 2017 and 2018 in NYC. Among teachers working in the same district who did not attend Relay, their average value-added score was -.01 in all three years compared with Relay completers’ scores that were .41, .66, and .41 in each of the three years, respectively. In ELA, Relay completers’ growth was greater than those of non-Relay teachers in 2016 and 2017 but not 2018; however, these findings across all three years were not statistically significant. When examining average ELA scores across the three years, Relay completers were able to bring their students to average expected growth in ELA with a weighted average value-added score of .05 (value-added scores of 0 or above indicate average expected growth).

Impact Measure #2: Indicators of teaching effectiveness

Relay engaged in a study of the teaching effectiveness of a sample of its completers using the Classroom Assessment Scoring System (CLASS) observation instrument. CLASS was designed by the University of Virginia’s Center for Advanced Studies in Teaching and Learning (CASTL) to assess the quality of teacher-student interactions that are associated with positive academic outcomes for PK-12 students. CLASS covers three domains: emotional support, classroom organization, and instructional support. These domains are further broken out into multiple dimensions. CASTL has validated the CLASS observation instrument in over 2,000 classrooms, and the observation can only be conducted by a certified CLASS observer who has completed CLASS inter-rater reliability training.

Relay found that the completers included in its study are applying the knowledge, skills, and dispositions that its program is designed to impact. Specifically, Relay’s completers in K-3, upper elementary, and secondary classrooms outperformed teachers in comparison studies on nearly all CLASS dimensions.

Tables 1 and 2 benchmark  Relay completer CLASS scores alongside CLASS scores collected from previous studies conducted by CASTL. Table 1 shows that on the K-3 CLASS rubric, Relay completer scores were comparable or higher than scores from the comparison studies for most CLASS dimensions. In particular, Relay completers outperformed teachers in the comparison studies on the Positive Climate, Teacher Sensitivity, Behavior Management, and Productivity dimensions. Table 2 shows that on the upper elementary and secondary CLASS rubric, Relay completer scores were higher or comparable than teacher scores in the comparison studies on every CLASS dimension.

Table 1: CLASS Scores on K-3 Rubric Across Four Studies

CAEP K-3 Rubric

Source: Center for Advanced Study of Teaching and Learning. (2019). Classroom Assessment Scoring System. Retrieved April 2, 2019, from https://curry.virginia.edu/classroom-assessment-scoring-system.

Table 2: CLASS Scores on Upper Elementary and Secondary Rubrics Across Four Studies

CAEP Rubric

Source: Center for Advanced Study of Teaching and Learning. (2019). Classroom Assessment Scoring System. Retrieved April 2, 2019, from https://curry.virginia.edu/classroom-assessment-scoring-system.

Impact Measure #3: Satisfaction of employers and employment milestones

Satisfaction of Employers

Relay conducted a case study to measure the satisfaction of its employers for a sample of its completers. Findings from the survey demonstrate that the majority of principals are satisfied with the preparation their teachers received at Relay.

  • 100% of surveyed principals responded that completers were moderately, mostly, or fully ready to meet the needs of students after completing their studies with Relay, with 60% stating that completers were mostly to fully ready.
  • Half of completers have been promoted during their time of employment.
  • 75% of surveyed principals believe that completers contributed to an expected level of student learning growth in their classrooms.
  •  With regard to Relay-prepared completers’ performance in the classroom, principals rated completer performance highly in comparison to other novice teachers in a number of areas including their ability to work with parents and families; maintaining a purposeful and effective learning environment; developing positive and supportive relationships with students; and improving teaching based on feedback.

Employment Milestones

Relay’s Alumni Survey is administered electronically in the fall of each academic year to all program completers. The survey includes a set of questions that are intended to measure completer satisfaction with the quality of teacher preparation they received, as well as completers’ current employment and employment milestones. What follows are highlights from the survey questions that focus on employment milestones.

  • Initial-Level Programs: Of the alumni respondents who work in PK-12 school settings, 237 indicated that they had been promoted since completing Relay’s program. See here for the full list of roles to which completers have been promoted.
  • Advanced-Level Programs: Of the respondents who work in PK-12 school settings, 6 indicated that they had been promoted since completing Relay’s program. See here for the full list of roles to which completers have been promoted.

Impact Measure #4: Satisfaction of completers

What follows are highlights from the Alumni Survey which focuses on completer satisfaction. The survey questions were designed and validated in collaboration with Deans for Impact (DFI), a consortium of deans from schools of education that share best practices and data with each other. As part of Relay’s membership in DFI, Relay is able to benchmark its survey results against other member EPPs’ results. See here for the full survey results benchmarked against the DFI consortium.

Initial-Level Programs

Overall, 81%, 76%, and 77%  of completers responded “Agree” or “Strongly Agree” to the question, “Overall, I am very satisfied with the preparation I received at Relay” in 2018-19, 2017-18, and 2016-17, respectively. Findings indicate that 80% or more (responding “Well” or “Very Well”) of completers were particularly satisfied with how well Relay prepared them to:

  • Analyze student performance data (86% responded “Well” or “Very Well” in 2018-19);
  • Create an environment of high expectations for all students (84% responded “Well” or “Very Well” in 2018-19);
  • Improve my teaching based on feedback and reflection on practice (83% responded “Well” or “Very Well” in 2018-19);
  • Maintain a purposeful and effective learning environment (81% responded “Well” or “Very Well” in 2018-19);
  • Measure and analyze student growth and achievement (87%, 85%, and 87% responded “Well” or “Very Well” in 2018-19, 2017-18, and 2016-17, respectively);
  • Set challenging and appropriate goals for student learning and performance (82% responded “Well” or “Very Well” in 2018-19);
  • Use knowledge of student learning and curriculum to plan instruction (80% responded “Well” or “Very Well” in 2018-19).

Advanced-Level Programs

Overall, 83%, 80%, and 71%  of completers responded “Agree” or “Strongly Agree” to the question, “Overall, I am very satisfied with the preparation I received at Relay” in 2018-19, 2017-18, and 2016-17, respectively. Findings indicate that 100% of respondents selected “Well” or “Very Well” in 2018-19 for how well Relay prepared them to:

  • Adapt practice based on research and student performance data;
  • Analyze student performance data;
  • Differentiate instruction based on student needs;
  • Improve my teaching based on feedback and reflection on practice;
  • Measure and analyze student growth and achievement; and
  • Teach in ways that support students with special needs - exceptional children.

Comparison to Completers from DFI Consortium

Relay GSE completers were more likely to select “well” or “very well” compared to completers in the DFI Consortium in the following categories:

  • Create an environment of high expectations for all students.
  • Improve my teaching based on feedback and reflection on practice.
  • Measure and analyze student growth and achievement.
  • Set challenging and appropriate goals for student learning and performance.

Impact Measure #5: Graduation rates

Relay GSE’s graduation rates are presented in the table below, benchmarked against peer graduate level institutions. Similar to Relay GSE, National Louis University and Western Governors University cater to adult learners and career changers, and offer alternative teaching programs. Relay GSE’s graduation rates surpassed both peer institutions for AY 2016-17, AY 2017-18 and AY 2018-19.

Graduation Rates

** Source: https://www.wgu.edu/about/students-graduates/retention-graduation-rates.html
*Source: https://www.nl.edu/assessmentandaccreditation/accreditation/studentachievementdata/

Impact Measure #6: Ability of completers to meet licensing (certification) and any additional certification requirements

New York City Campus Licensure Pass Rates, 2015-16; Source: Title II Reports

Licensure Pass Rates

* Pass rate is not provided when there are less than 10 test takers

New York City Campus Licensure Pass Rates, 2016-17; Source: Title II Reports

CLPR

* Pass rate is not provided when there are less than 10 test takers

New York City Campus Licensure Pass Rates, 2017-18; Source: Title II Reports

CLPR

* Pass rate is not provided when there are less than 10 test takers

Relay’s licensure pass rates surpass or are comparable to the state pass rate for the majority of licensure exams across all three cycles of data. After analyzing our data, Relay’s Director of Certification and Deans have identified that the two types of exams our candidates struggle with the most are mathematics exams and multi-subject content exams. We believe that the underlying reasons for underperformance on multi-subject content exams is three-fold.

  1. Post-baccalaureate candidates in a multi-subject certification area are often tested in content areas in which they may have received minimal coursework prior to enrolling at Relay (e.g., an Elementary Education candidate whose undergraduate major was History or Philosophy has likely taken less mathematics coursework in their secondary and post-secondary schooling as compared to a candidate whose undergraduate major was Statistics or Calculus). 
  2. A number of candidates who enroll at Relay are not enrolling directly from their undergraduate institution -- many are career changers who have not attended an undergraduate or graduate program in several years. As a result, they likely have not taken coursework in or used much of the content they are being assessed on in several years and therefore need additional support.
  3. Finally, each multi-subject exam has a mathematics section. As noted previously, mathematics tends to be the most common content area that keeps candidates from passing licensure exams -- whether it’s a secondary mathematics exam or the mathematics portion of a multi-subject exam.

As a result, the Director of Certification offers targeted online synchronous workshops in early childhood/elementary math and middle school math to all enrolled and prospective candidates three times per semester. Faculty in our mathematics department have taken incremental steps to incorporate more “pure mathematics” instruction into coursework that is offered to candidates during their first year of the program (although this additional content is limited). We also offer a suite of content-specific resources to candidates, as listed above under question one. All state requirements, exam preparation resources, and Relay certification policies are made readily available to the leadership and faculty at each campus. The Director of Certification has worked closely with campuses to ensure these resources are made available to prospective candidates as early in the admissions and/or enrollment process as possible, so that candidates have an adequate amount of time to review material and access resources as needed.

Currently, the Director of Certification is working with our Academic Programs team and campus leaders to curate national content supports for mathematics, ELA, and Elementary Education content areas. Supports being created include additional Elementary Education Praxis, Elementary Math, Secondary Math, and ELA workshops, two 1-credit math courses, and a test in preparation seminar. 

Impact Measure #7: Ability of completers to be hired in education positions for which they have prepared

At Relay, 100% of candidates are full-time employees of a PK-12 school partner while they are enrolled with Relay. In order to enroll in Relay’s initial and advanced level programs, prospective candidates must demonstrate that they have secured a full-time classroom position at a partner PK-12 school. As a result, all of Relay’s candidates are hired in education positions while enrolled with Relay. Relay also tracks the ability of its completers to be hired in education positions through its Alumni Survey. Recent survey results are presented below.

Initial-Level

  • 86% of respondents to the 2018-19 survey are employed in a PK-12 role. The breakdown is as follows:
    • 66%: Classroom teacher
    • 11%: Other role in PK-12 education
    • 8%: Instructional leader
    • 1%: School leader
  • The majority of respondents (90%) indicated that they intend to remain in the classroom beyond this year.

Advanced-Level

  • 100% of respondents to the 2018-19 survey are employed in a PK-12 role. The breakdown is as follows: 67% are classroom teachers and 33% are instructional leaders.

Impact Measure #8: Student loan default rates

The loan default rates in the table below shows that Relay’s default rate was 0% in FY 2014, 3.7% in FY 2015, and 3.2% in FY 2016, which is the most recent year data is available). Relay GSE’s loan default rates are consistently considerably lower than the national loan default rate, which was 10.8% in FY 2015 and 10.1% in FY 2016.

CAEP Chart