Unlocking NSSE’s Open-Ended Data with AI at the University at Buffalo
By: Steven Feldman, NSSE Institute Graduate Assistant
At the University at Buffalo (UB), the Institutional Research team has been exploring the use of artificial intelligence (AI) to analyze students’ responses from NSSE open-ended comment data. Melinda Whitford, Research Analyst, and Kubra Say, former Graduate Assistant, at UB lead much of this work. In the early stages, they began experimenting with AI tools like ChatGPT and Microsoft Copilot, initially comparing hand-coded themes with those generated by AI. After finding that AI did a decent job at qualitative analysis, they developed a greater trust in the technology and found that it opened the door to broader applications.
One of the first lessons they learned was the importance of prompt engineering, which is the process of developing useful prompts for the AI tool. Dr. Whitford started AI analysis with general questions, such as asking for the top ten positive themes, and then refined her prompts based on the responses. Using prompts like that for sentiment analysis, she was able to discover which emotions were most expressed in the open-ended comment data (see Figure 1). The iterative process allowed her to dig deeper into the data, comparing themes across colleges, student demographics, and survey years. She found that the same emotional keywords were consistently associated across time. For example, “money” was frequently linked to anger and “helpfulness” to joy (see Figure 2). She interpreted this as a sign of institutional consistency in student experience.
Figure 1: Percentage of Emotions Found in Sentiment Analysis
Figure 2: Words Associated with Emotions from Sentiment Analysis
Dr. Say used a mix of tools, including R for text mining and sentiment analysis, and Copilot for more accessible, prompt-based exploration. While R provides detailed metrics, Copilot and ChatGPT offer flexibility and ease of use, especially for those less familiar with coding. Dr. Whitford noted that Copilot can even generate R code if needed, bridging the gap between technical and non-technical users. In fact, she noted that even qualitative analysis can have quantitative elements to it. She shared, “Sentiment analysis, text mining—we think of doing that for qualitative, but it’s really quantitative analysis. So that just kind of makes everybody feel better, because we’re quantifying something using these descriptors.”
Dr. Whitford also noted that with AI’s many benefits also come some challenges such as ensuring data privacy, especially when responses might include names, and recognizing that different AI platforms may yield different results. To address this concern, her and Dr. Say have transitioned to using secure, institutionally managed versions of ChatGPT and Copilot that do not share data externally, helping to safeguard student confidentiality. They advised starting with clear research questions and using AI to support rather than replace human interpretation. They also emphasized the value of incorporating student descriptors into the analysis to enable more nuanced comparisons. For example, descriptors like students’ year in school or demographic information allow the research team to move beyond general themes and explore how different groups of students experienced campus life.
Beyond their own institution, Dr. Whitford and Dr. Say have shared Tableau templates and analysis strategies with other SUNY campuses through the Association for Institutional Research and Planning Officers (AIRPO), an affiliate organization of the Association for Institutional Research (AIR). Their team’s work has helped other institutions with limited resources begin their own qualitative analysis journeys. Dr. Whitford sees AI as a way to honor student voices, ensuring that open-ended responses are not left unread in network folders but instead used to inform decision-making and improve student experiences.
Looking ahead, Dr. Whitford hopes to use AI analysis to examine student responses from previous topical modules and apply similar methods to future NSSE cycles. This would enable point-to-point comparisons and support longitudinal analysis of student experiences. She’s also interested in filtering responses by high-impact practices, using AI to explore how students describe these experiences in their own words. For her, she says “this is trying to be respectful of the fact that students have taken the time to submit a response to an open-ended question. We should read that response, and take it seriously, and do something with it.” In other words, the goal is not just to analyze data but to make it actionable and to help others do the same.
