AI Transforms Qualitative Survey Analysis at The University of Toledo
By: Alethia Russell, NSSE Institute Project Associate
In institutional research (IR), qualitative data has long been a powerful yet underutilized asset. Thousands of open-ended survey comments often go unanalyzed, not due to lack of interest but because of the time, labor, and software limitations associated with manual qualitative coding. The institutional research team at The University of Toledo has taken steps to eliminate the bottleneck that can come with open-ended survey comments.
In 2021, researchers found themselves staring down thousands of open-ended responses from the NSSE and a strategic planning survey. “I remember for the strategic planning survey, we had, I think, more than 7 or 8 open-ended questions, so it was quite rich in terms of data. We developed the tool keeping that in mind so that if you have more questions or more open-ended sections in your survey… You can analyze all of them,” said Ayubur Khan, Senior Institutional Research Analyst at The University of Toledo.
Traditional qualitative analysis tools like NVivo or ATLAS.ti were available, but their reliance on extensive manual coding made them infeasible. The team needed something faster, more intuitive, and more scalable. So, they built it.
Ayubur developed a free, browser-based tool that allows users to upload a CSV file and perform AI-enhanced qualitative analysis using R’s Shiny platform. The tool walks users through several pre-processing steps: tokenization (breaking sentences into words), lemmatization (reducing words to their root forms), and stop-word removal. Then, it conducts n-gram analysis, with bigrams proving to be the most useful for identifying recurring themes like “online class” or “academic advisor.”
What makes this tool stand out is its feature that allows users to filter for a theme like “online class” and generate a sentiment analysis summary with a single click through the integration with ChatGPT’s API. The AI provides a rating (positive, neutral, negative), summarizes key findings, and recommends action steps for the selected theme. While the sentiment analysis isn’t reproducible in the traditional sense—the AI’s phrasing can vary—running the analysis multiple times offers a reliable average.
“This transformed the way we analyze open-ended survey questions,” said Dr. Mingyang Liu, survey statistician with the University of Toledo’s IR office. “What used to take us a whole summer now takes minutes. We have been so busy. We rely on this tool to get things done because IR has a lot of requests every day. Long story short, this tool is highly efficient and gets you the overall summary of the qualitative data within a very quick period of time.”
Beyond efficiency, the tool builds capacity. Its low barrier to entry means non-coders and less technical researchers can analyze data independently. Faculty, staff, and even student interns can use it with minimal training.
It’s also ideal for institutions with tight budgets as a viable alternative to costly license renewals or to supplement existing licenses in a setting where a department may be more resource strapped.
“I think it definitely freed up some of our time because the tool was developed in a way even someone without any coding experience or knowledge could use it. You have to do a few clicks and interpret the data. We kind of encouraged other researchers to use the tool. Instead of putting in a request for us to analyze the data for them, they’re doing it, and it’s empowered them to do their own analysis in terms of qualitative data,” said Liu.
The tool has already been used to analyze survey data along with scraped web data, and more. It’s flexible, scalable, and openly available—no subscription required unless you’re using the optional ChatGPT sentiment analysis, which carries a small pay-per-use cost.
Of course, it’s not without its limitations. The tool works best with large datasets. Special characters and identifiable information must be cleaned manually before uploading. And while the AI component offers speed, users must be thoughtful about privacy and how they interpret and triangulate findings. Still, the broader implications are hard to ignore. Qualitative researchers have talked about the use of digital tools in qualitative research and, ultimately, how the introduction of digital tools into research projects often impacts analysis and research decisions in different ways that should be reported and accounted for (Lester, et al., 2013).
“I think our office is fortunate we have Ayubur, who’s built several wonderful tools that improve productivity and automation and those types of things,” said Liu. Liu, who earned his master’s and Ph.D. in Educational Research using qualitative research methods, welcomes the innovation of new technology into their applied research arsenal. With tools like this, institutional researchers no longer need to choose between depth and efficiency. They can have both—and more importantly, they can act on what students are saying, not just what they can afford to analyze. For any recommendations or questions about this text analysis tool, please contact Ayubur Khan at MdAyuburRahman.Khan@utoledo.edu.
References
Paulus, T. M., Paulus, T. M., Lester, J. N., Dempster, P. (2013). Digital Tools for Qualitative Research. United Kingdom: SAGE Publications.
