Harder Benchmarks Are Not Always Better: Evaluating LLM Distractors for Multiple-Choice Benchmarks
Atrey Desai, Nishant Balepur, Rachel Rudinger
Under ACL ARR Review, 2026 preprint
hi, i'm atrey desai
I am a final-year undergraduate studying computer science and linguistics at the University of Maryland.
I am fortunate to be advised by Professors Rachel Rudinger, Jordan Boyd-Graber, and Mohit Iyyer.
Some tasks have clear answers and stay static. Others are uncertain and shift as the world around them adapts. Language models are increasingly deployed as agents in these open-ended environments, where it is hard to know when to trust them. I study how humans and agents can build calibrated trust there, namely:
Shows full research questions for all research interests.
Calibrate agents' confidence under incomplete, uncertain, and excessive information.
Study how people and agents adapt to each other over time.
Build feedback that stays useful as the world adapts to it.
Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers
Nishant Balepur, Atrey Desai, Rachel Rudinger
ACL Main, 2026
Asking About Occluded Content Marks Adversarial Multimodal Questions
HyoJung Han*, Atrey Desai*, Ishan Upadhyay, Tianyi Zheng, Eve Fleisig, Aparna Garimella, Marine Carpuat, Jordan Boyd-Graber
Under ACL ARR Review, 2026 preprint