Nancy Waymack, NSSA Director of Research Partnerships and Policy, discussed Quality Data for Sustaining Quality Programs: Examples from the Field
In, Quality Data for Sustaining Quality Programs: Examples from the Field, district, state, tutoring provider, and partner-organization leaders discussed the types of data critical to understanding effectiveness and identifying improvement opportunities for tutoring programs. NSSA and partner districts shared examples of data collection tools and dashboards. Participants reviewed scenarios and discussed what they could learn with available data, what insights additional data could provide, and how these scenarios play out in their own districts and states.
Scenario 1: Using attendance data to learn more about the effect of high-impact tutoring
Research Question: How does participation in tutoring affect attendance?
Students who attend more days of school receive more tutoring. But which comes first? Are students with higher attendance more likely to be assigned to tutoring or does participation in tutoring increase school attendance?
Scenario: One state education office provided funding for tutoring in LEAs state-wide. LEAs have the autonomy to determine how students are selected for tutoring. Attendance for students in tutoring is greater than those that are not in tutoring.
Available data: Researchers had daily school attendance in addition to tutoring session attendance as well as information on which days tutoring was scheduled. By comparing each student’s school attendance on days that he or she had tutoring with school attendance on days that he or she did not have tutoring, researchers isolated the impact of tutoring on school attendance. They controlled for student fixed effects, day of the week, date (i.e. the Friday before a holiday), demographics, prior year attendance, and prior year test scores.
Discuss the following questions at your table:
- Do you consider daily school attendance part of your tutoring data?
- Do you consider tutoring a strategy for improving school attendance?
- What other outcomes, in addition to achievement, do you think tutoring might improve?
- What additional data do you wish you could include in this analysis?
- Does this bring up other questions for you? What are they?
Notes from discussion:
- This group mostly focused on the potential impact of attendance on overall school funding. They suggested that this approach could potentially serve as a funding source post ESSER.
- This group included members from both CT and NYC, who emphasized the importance of attendance as a priority.
- They raised some questions about student-tutor data, focusing on students' perceptions of tutors and their overall attitudes towards both tutors and schools.
- They engaged in discussions surrounding student-tutor data, focusing on students' perceptions of tutors and their overall attitudes towards both tutors and schools.
- They also wondered if tutoring could help make up for resources some kids might not have access to.
- This group built on several points made by others and added that understanding the context of absences, especially in relation to age, is crucial before implementing interventions. They suggested that tutoring might not be as effective for older students, but could be a promising solution for middle schoolers.
Scenario 2: Understanding the counterfactual to learn more about the effect of high-impact tutoring
Research Question: What is the impact of virtual tutoring as compared to other district delivered interventions?
Scenario: A district is providing virtual tutoring for a subset of students required to receive accelerated learning support. Students who do not receive virtual tutoring receive other services from the district to meet this requirement.
Available data: Researchers received data from the district and virtual tutoring provider. The tutoring program focused on mathematics and reading for students in Grades 3–8 in small group sessions of 45 minutes, five days per week. Out of 8,000 students required to receive accelerated learning based upon end-of-year test-score data, a subset of 1,500 students were randomly selected to receive virtual tutoring. While researchers have extensive data available for the students who participated in virtual tutoring, data on the counterfactual (the services provided to other eligible students) was not available.
Discuss the following questions at your table:
- Do you consider data about alternative services students receive as part of your tutoring data?
- What other outcomes, in addition to achievement, do you think tutoring might improve?
- What additional data do you wish you could include in this analysis?
- Does this bring up other questions for you? What are they?
Notes from discussion:
- This group focused on Guryan's research and mentioned that the interventions provided to the control group could be quite comprehensive, which they thought was a positive. They discussed this and everyone seemed pretty excited about it because it meant that more kids are benefiting from several interventions.
Scenario 3: Using pulse check survey data to learn more about the effect of high-impact tutoring
Research Question: How do student self-reported experiences in tutoring correlate with their academic achievement?
Scenario: Tutored students (and tutors) complete short “pulse check” surveys four times over the course of the school year covering social-emotional learning (SEL) domains including measures of tutoring experiences, self-efficacy, relationships and belonging.
Available data: Researchers have pulse check survey data as well as student achievement, demographic, attendance, and tutoring program data. Seven hundred elementary students completed the pulse check surveys and completed tutoring for the year.
Discuss the following questions at your table:
- Do you consider student survey measures part of your tutoring data?
- Do you consider SEL strategies in tutoring a strategy for improving school achievement?
- What other outcomes, in addition to achievement, do you think increased attention to SEL in tutoring might improve?
- What additional data do you wish you could include in this analysis?
- Does this bring up other questions for you? What are they?
Notes from discussion:
- The group concluded that they haven't placed sufficient emphasis on students' perceptions, as reflected in student surveys.
- They shared that it's crucial for students to understand the purpose behind receiving tutoring, as this awareness can be quite effective.
- Not only does it offer beneficial outcomes for SEL due to the personalized attention received during tutoring sessions, but it also enhances student self-efficacy and decreases behavioral referrals.
- The group was very interested in incorporating qualitative data to amplify student voices, and some members even said that they've already taken steps in this direction.
- The group also emphasized the importance of student pulse check survey data for internal purposes and improving student experiences.
- They discussed the importance of sharing this data with tutors to incentivize them and facilitate stronger connections with their students.
- They also shared that there is a need for more comprehensive data on students' social-emotional well-being, suggesting that conducting surveys more frequently, rather than just once per quarter, would be beneficial.
Scenario 4: Using engagement data to learn more about the effect of high-impact tutoring
Research Question: Does real-time feedback to students on their engagement and efforts in math tutoring sessions increase their engagement, which has been widely linked to improved achievement-related outcomes?
Scenario: The tutoring platform in this study allows the tutors to award participation points to students for demonstrating effort during sessions. Tutors can award students points for asking and answering questions, explaining their reasoning, and demonstrating ‘higher level’ thinking skills.
Available data: Researchers received data from the provider on the participating students, participating tutors, and the tutoring sessions themselves. The session-level data captures all tutoring sessions over the course of the year. Session data include information on student attendance, tutor/student treatment group match, and participation points awarded. We created a combined student and tutor behavior variables based on session-level data. They include sums of student and tutor attendance, percentage of sessions with tutor/student treatment group match, and average participation behavior in math sessions (average participation points earned/awarded). We then merged these data with student- and tutor-data to link session-level behavior to student achievement outcomes, demographic information on students, and information on tutors.
Discuss the following questions at your table:
- Do you consider engagement measures part of your tutoring data?
- Do you consider enhancing engagement in tutoring a strategy for improving school achievement?
- What other outcomes, in addition to achievement, do you think increased engagement in tutoring might improve?
- What additional data do you wish you could include in this analysis?
- Does this bring up other questions for you? What are they?
Notes from discussion:
- This group identified opportunities from the students' perspective, suggesting potential strategies such as incorporating digital emoticons as quick response tools to foster engagement. However, they expressed skepticism about the effectiveness of such incentives and proposed that peer compliments might be a better approach.
- The discussion shifted towards emphasizing the significance of relationships between tutors, students, and peers. The consensus was that cultivating an environment characterized by mutual respect and encouragement among students and tutors would significantly enhance student engagement.