Bringing Predictive Modeling In-House – University of Washington
With a common goal of “developing the in-house capacity to do predictive analytics”, personnel from different departments at the University of Washington – Seattle (including Research, Annual Giving, Reporting Services, and Records Management) set out to find a way to bring predictive modeling to the school.
Staff piecemealed their own backgrounds in statistics, brought to bear their own expertise within each of their fields and started sharing information. They called their group DAWGs (Development Working Group) and combined any and all information they could get on predictive modeling. Together they read a book about data mining using SQL. Though they learned a lot, they decided to go a different route. “We tried our hand at data mining functions (using SQL, SAS, and SPSS),” said Dr. Jennifer MacCormack, Associate Director of Development Research, “but it was a very steep learning curve and we were all trying to get on board together with this.”
We wanted something we could all use together
After figuring out what didn’t work for them, and knowing they didn’t have the budget to hire additional talent, DAWGs started looking for what would work – and found the Rapid Insight Software Suite. “Rapid Insight caught our eye among all the resources we were looking at because it was easy to use and learn as you go along,” said Dr. MacCormack. With a goal of bringing predictive modeling in-house, and the software to achieve it, DAWGs started looking for a specific, trial project.
A project that would make the case for further growth
Their initial challenge was deciding which predictive modeling project would be first. DAWGs members weren’t afraid to learn the fundamentals of statistics through trial and error and knew they wanted a project that would give them a better understanding of the strengths and challenges of their data. They also wanted to pick a project that would make a strong case for bringing predictive modeling on campus. After taking these factors into consideration, DAWGs decided that the first model would predict who was most likely to make a bequest to University of Washington. Now they were ready to being the data collection process.
The time-intensive part of doing the data preparation
Much of the data that DAWGs had available was in summary format, which was problematic in terms of actually building the predictive model. In order to collect all of the necessary detailed records, they decided to go back to the data source and hand-pick variables to include in the analysis. As Lilith Lysistrata, Senior Advancement Data Analyst, observed, “we started the process by getting together and brainstorming what variables would be useful for this predictive model. Basically that created a list [for us] to go back and start pulling that data out of the database.”
After pulling all of these records from the database, they still needed to nail down the dependent modeling variable. This took careful consideration, Lysistrata recalls. “It seems like it’s going to be so easy, so cut and dry, but then you get into it and realize there’s a lot of questions you need to ask”. Creating the “bequest model” was tricky in two ways. First, because a bequest is so tied to death and the model should not be trying to actually predict death. Secondly, because variables change for any record remaining in a dataset after death, it is important to treat the data as time-sensitive. As DAWGs quickly discovered, “you really have to go back and work through the data and work through the point in time and figure out what happened right before they passed away”.
During the process, the DAWGs found the following quote by Dr. Chuck McClenon, Fundraising Scientist at the University of Texas, Austin (and fellow Rapid Insight user), summarized this trial and error period nicely:
“If we can learn from our mistakes, then we would be wise to make as many mistakes as we can as fast as we can.”
Make a lot of errors, and you learn a lot
After all of the data collection, the DAWGs members were anxious to begin modeling. They put all of their data into Predict (the statistical modeling part of the Rapid Insight Software Suite) and waited for the magic to happen. The output they got, however, didn’t seem to be predicting bequest the way they had intended. So, they asked for help: “We really had to bring in Mike (Michael Laracy, founder and CEO of Rapid Insight). That’s one of the great things about using Rapid Insight – we can call the staff there, who actually knows what they’re doing, and say ‘Can you help us? Can you tell us what is going wrong here?’” recalls Dr. MacCormack.
Mike and DAWGs collaborated on their model whenever they needed help along the way. Using both internal and external resources helped. The staff at the University of Washington brought a solid understanding of their data while Rapid Insight staff brought, when needed, their knowledge of statistics and best modeling practices.
This was actually really fun
With a little help, DAWGs did get the model it was looking for, and felt great about it: “When we got to this point, we felt like we had really accomplished something and we had really found some interesting information”, says Dr. MacCormack. “We were ready to move ahead with this and look at different variables that were in there.” Their researchers were lucky enough to have access to a survey about whether participants planned to give a bequest to University of Washington, which they used as a model validation tool. Although some of the people surveyed who said they planned to leave a bequest were known to University of Washington, most were not – and of these unknowns, 60% scored in the top 20% of the model. This preliminary validation suggests that the model was doing a good job of predicting who is likely to make a bequest, and gave DAWGs more confidence in the model it had built.
We were ready to move ahead with this
After building their first predictive model, the DAWGs members and their colleagues are finding more and more uses for predictive modeling on campus, and are currently working on building models for other departments. They’re using the variables in their bequest model to see if they can have any impact on a person’s likelihood of giving a bequest. “We’re going to do some investigation into if there is any behavior that we can identify that will increase their likelihood [of giving]”, says Dr. MacCormack. “One great thing about using Rapid Insight is that the model building is so fast. I could go through and build a new model every day for a variety of different solicitations.” They will also be using the predictive models for a variety of marketing pieces.
Having completed their first model, they are putting their insights to use to create new models, and with the help of Rapid Insight, doing this efficiently: “I think the learning component of it was really vital to us” concludes Dr. MacCormack. “We really did need to learn the statistics and see how the model was being built. And what made this a little more effective is that there’s also a level of trouble-shooting with the staff at Rapid Insight. So as you’re going along building a model, you can also consult with them and talk about some of the issues you’re having with your variables and building that model”.
The University of Washington DAWGS grew their internal capacity for predictive modeling while simultaneously accomplishing the actual modeling. It was a very effective way to approach bringing predictive modeling in-house, and was facilitated by the Rapid Insight team and software.