Skip To Content
Rapid Insight: Higher Education case study

Saginaw Valley State University Incorporates Live LMS Data into Student Success Models

Download a Summary
SVSU campus
  • Incorporated pandemic-proof metrics into student success models

  • Equipped advisors to flag at-risk students for intervention

  • Uncovered a new modeling variable for attendance

Since 2017, Saginaw Valley State University (SVSU) has used Rapid Insight’s predictive modeling software to improve their retention efforts. SVSU’s model flags students at risk of attrition so that counselors can offer targeted support where it will make the most impact. The model has been of major benefit to SVSU, but like most institutions using predictive modeling, SVSU found that their models were no longer as relevant once COVID-19 struck.

Thanks to responsive thinking and Rapid Insight’s flexible data tools, SVSU quickly integrated new, relevant data from their Canvas Learning Management System (LMS) into its models. This kept retention efforts on track and helped SVSU maintain its retention rate despite the stressors of the pandemic.

Customer Testimonial

Who knows what the percentage retained would have been if interventions didn’t occur and the model wasn’t telling us to reach out to these students?

Nick Wagner, Executive Director of Institutional Effectiveness, Saginaw State Valley University

The Challenge

Historically, SVSU has maintained a consistent 70% retention rate and 39% graduation rate. The university knew there was great value in making improvements. In 2017, SVSU decided to get more proactive about intervening to ensure student success.

In response, Nick Wagner, the Executive Director of Institutional Effectiveness at Saginaw Valley State University, brought on Rapid Insight’s modeling software: Predict. He used the software to create a predictive model that used 16 distinct data points to calculate a student’s risk of attrition. When the model flagged a student as at-risk of failing to persist, it notified staff to intervene. SVSU found great utility in its increased ability to work with students who might struggle to succeed.

The model brought positive change to SVSU, but Wagner knew it could provide even more significant improvement with the inclusion of live data. Pre-admissions data only paints a fractional image of a student’s ability to succeed once on campus. Waiting for mid-semester grades to determine a student’s level of attrition risk means a semester’s worth of missed opportunities to get ahead of any potential issues that students face. This semester, in particular, necessitated more frequent updates on student engagement.

Wagner knew that bringing live data in the model would equip the institution to assist students who needed help proactively rather than reactively. While this change was already a goal of Wagner’s, COVID-19 made its implementation an urgent and immediate priority.

The Solution

Incorporating live LMS data

When classes went entirely online due to the COVID-19 pandemic, Wagner sensed that the university’s Learning Management System would contain valuable engagement data that could help advisors and retention committees intervene during an incredibly challenging semester. 

Because students attended classes, submitted assignments, and engaged with their professors through Canvas (the institution’s LMS), Wagner could incorporate a bevy of new data points into the student success models to monitor how students were faring in their online classes. Wagner knew that this data would be critical to understanding how to assist struggling students during an unconventional semester.

With live engagement metrics at hand, Wagner’s enhanced model could offer up-to-date insight into how students were coping with remote classes, allowing academic advisors and staff to provide targeted support.

Wagner started with small test groups of students to verify that his model worked as intended before applying it. The results were positive, so Wagner applied the model to the entire undergraduate population.

Discovering new variables

Before COVID, Wagner sought an attendance variable that had statistical significance in his retention model. He tried several approaches, but none functioned as required.

When courses went online, tracking attendance became much more straightforward. Wagner used LMS engagement to stand in for attendance.

Primarily, Wagner used discussion board posts to monitor how active students were. He tracked whether students posted on discussion boards on a given day, how many times they posted on that day, and the depth of their posts (calculated by the number of lines in the post). 

Tracking this information over time, Wagner found that these variables were significant in predicting how engaged and successful students were in their online classes. Academic advisors and staff could rely on the model’s projections to plan their outreach and connect with students at risk of not succeeding in the classes.

Testing Accuracy

After several months of using the model with live student data, Wagner measured its accuracy with a decile analysis. He found that the live LMS model led to more targeted improvement in one of their largest groups of students, meaning it made the most significant possible impact.

Nearly all student categories had a changed probability of retention as a result of the model’s implementation. Many students would have been improperly sorted into the “most at risk” category and prioritized for intervention absent the updates to the model. This would have drawn advisor attention away from the students who needed it most. Instead, the live data moved many of those students into a medium-risk category where they truly belonged.

Customer Testimonial

These aren’t just small 2-3% differences. These are jumps of 20% retention likelihood in some of these students after incorporating the live data.

Nick Wagner, Executive Director of Institutional Effectiveness, Saginaw State Valley University

In looking at five groups of students, each representing 20% of the student population, retention rates stayed very high despite a significant shakeup:

Live LMS Data Model

With use and refinement, the model should continue to improve and become even more predictive.

Continuing into the New Normal

Pre-pandemic, over 90% of the faculty at Saginaw Valley State used the Canvas LMS system, but often to a limited extent. Usage increased significantly during the 2020 school year. Wagner suspects that now that faculty have grown accustomed to Canvas, higher use will carry into the post-pandemic period.

The model’s new digital engagement metrics will be very valuable for any classes taught online. But faculty are now more familiar with distributing information over Canvas, so it could be that the metrics are helpful for in-person courses as well. Professors may request assignments or engagement via Canvas outside the classroom. This would make live LMS data available into the future. 

Looking to the Future

Saginaw State’s updated retention models made a significant difference during the tumultuous 2020 school year. But Wagner sees his efforts not as finished but as only having just begun.

The next step is to evaluate the model’s outcomes from Fall to Fall, comparing this year’s results to next year’s. This will offer further insight into where the model can be improved and refined, 

Long-term, Wagner plans to incorporate more LMS data into the model. He hopes that when students return to campus full time, he will be able to bring in swipe card data from student services like the writing center, academic support, health facilities, and other campus offerings. 

This will provide live engagement factors that Wagner can test to see if they contribute to the model’s outcomes, and more importantly, if they play a role in leading students to success.

Learn more about Saginaw Valley State’s modeling efforts by watching a full webinar here.