News Releases

ANN ARBOR — Representatives from the University of Michigan’s Institute for Social Research (ISR), the University of Chicago’s NORC Center on AI & Data Quality, and leaders of both federally funded studies and their funding agencies convened recently for a daylong forum on threats to survey research from artificial intelligence.

The event, titled “Convening on AI Risks to Surveys,” took place in Washington, D.C. in late August. ISR’s Joelle Abramowitz and Tom Crossley co-organized the event along with Susan Paddock and Mike Davern of NORC. Attendees, who included Seoyoun Kim, Joe Saul, Grant Benson, Noura Insolera, Wen Chang, Amy Pienta, Libby Hemphill, and Pam Davis-Kean from ISR’s Survey Research Center (SRC), Survey Research Organization (SRO), and ICPSR, heard from panelists in two different sessions assessing AI’s role in survey research and its potential pitfalls. Of the ISR-affiliated attendees, Kim and Saul served as panelists during the event.

The first session focused on the balance between managing confidential data and using AI tools for research. ISR’s Joelle Abramowitz presented on how two U-M surveys, the Health and Retirement Study and Panel Study of Income Dynamics, are approaching these questions, and how secure environments like the MiCDA virtual data enclave can provide an appropriate space for using these data with locally-run open-source LLMs. 

The second session discussed methods for protecting survey data from AI respondents. NORC’s Ting Yan presented a framework for thinking about survey fraud, highlighting two main concerns: if the person providing data to the survey is the “right” person, and if the data provided by that person is the “right” data. She highlighted that while survey fraud is not a new issue, the rise of generative AI has created new channels for it to proliferate. 

“While bots specifically have been a long-standing problem for commercial web panels, they have been less of an issue for many of the panel studies we conduct at ISR — for example, the PSID and HRS — because we use a sampling frame to identify respondents and we have long-standing interactions with them,” said Joelle Abramowitz, echoing the concerns Yan outlined. “We do worry that respondents might use AI to answer questions, and while Arie Kapteyn from the Understanding America Study (UAS) presented some evidence that they are not seeing substantial use by respondents, it would be valuable to further examine this issue more broadly.”

According to Abramowitz, AI usage in survey research can be helpful; researchers can, for instance, use AI for coding assistance, provided they don’t upload data to a Large Language Model (LLM). Abramowitz said that’s a threshold that violates the Conditions of Use for several U-M surveys.

“The concern is that uploading the data to the LLM counts as redistribution, which is a violation of the conditions of use and prevents us from tracking use, which is required by our sponsors.  Once the data are uploaded to an LLM, we worry about other users being able to access the data through the LLM, which may enable the data quality to be compromised, in addition to other concerns.”

Other researchers have identified AI-generated papers that use existing data in incorrect ways, another concern for the field.

“One next step is working with journals or professional societies to implement and enforce safeguards against inappropriate AI use. Another is creating a harmonized system of researcher credentials to raise consequences for bad behavior, and ICPSR initiatives may be valuable for implementing such an approach,” Abramowitz said.  

Attendees also discussed next steps for research in a world where surveys must adapt to threats posed by AI. Follow-up discussions are planned for the Longitudinal Studies of Aging meeting to be held at ISR in December. All together, about 30 people attended in person while more joined online. 

Contact: Jon Meerdink ([email protected])