Data Use: Program Effectiveness and Improvement
Program Effectiveness and Improvement: The program has demonstrated a commitment to understanding overall program effectiveness and processes for ongoing improvement.
- What student and tutor outcomes represent the vision of success for the program?
- What data will the program collect to measure these outcomes?
- How will these data reflect a holistic understanding of students’ and tutors’ experiences?
- How do the measures address the impact on students and tutors?
- What processes will be in place to review and act upon collected data?
- How will these review processes promote equity and reduce bias?
- Define measures of success in alignment with your logic model, including process measures of implementation and non-academic measures of impact.
- Develop and/or leverage existing tools to collect data on the identified measures, including both quantitative and qualitative data.
- Set benchmarks to monitor progress towards outcomes.
- Put systems in place for collecting data that can be disaggregated by race, gender, individualized education program statutes, home language, and other important factors to ensure equity of programs.
- Consider using the Tutoring Quality Improvement System to assess overall program’s alignment with High-Impact Tutoring Standards as part of continuous improvement efforts.
- Meet requirements and use best practices for data privacy.
From Existing Resources:
Develop a holistic data collection strategy that includes non-academic measures of impact for both students and tutors.
- While academic improvement is the primary purpose of a tutoring program, it is not the only goal. Programs need to collect data across multiple dimensions to ensure that they are building students’ overall well being and are serving students equitably. As a result they benefit from collecting data to evaluate student experiences with tutors, not just student academic growth. OST programs should also set goals and collect data to assess progress on tutor outcomes.
- Programs should collect feedback from all stakeholders (students, families, teachers, tutors, and administrators) to understand and improve program impact at all levels. While achievement data and feedback from school partners is critical, programs should include student voices when evaluating program impact: tutoring programs exist primarily to serve students.
- These pre-developed Tutoring Survey Instruments can support the program’s understanding of student and tutor experiences.
- Consider phasing the approach to data collection to ensure you are able to commit to the data you plan to collect. For example, you may begin with collecting feedback from 1-2 stakeholder groups during the first year of implementation, and then increase the number of stakeholder groups in subsequent years.
Set specific benchmarks with expected dates to help stay on track.
- Programs should set benchmarks with expected dates for all measures — not just for student growth, but also for aspects like student/tutor/teacher/parent satisfaction. Routinely reviewing data and comparing it to benchmarks helps programs understand where they are on-track or off-track; this is critical for establishing a data- to-action cycle of insights and iterative improvements. These processes will also support the program with communicating progress and improvements to funders, ensuring the sustainability of the tutoring program into the future.
Align routine assessments with session targets (and, ideally, with classroom curriculum).
- Well-aligned, routine assessments can help programs quickly identify student knowledge gaps and target upcoming sessions to meet specific student needs as they emerge.
- If schools and/or districts your students attend have existing interim assessments, leverage those data to reduce the need for another assessment and assure that tutoring success is tied to outcomes the school and district sees as relevant.
- In order for formative assessments to result in more student learning, tutors need time and support to review the assessment and formulate a plan to address each student’s needs.
Develop systems for visualizing data for stakeholders.
- Programs should develop in-house capability for distilling data so that information can be presented in a digestible and actionable format. Some programs may have databases and utilize software such as Tableau to visualize data, while other programs that operate at a smaller scale may find it sufficient to store data in well-designed Excel/Google sheet.
- Ultimately, the method chosen for visualizing data should allow for users to sort the data and easily extract insights.
- Programs should regularly gather feedback on their data collection and visualization systems and improve upon these as part of their continuous improvement processes.