Quick, Create a Distractor! Evaluating LLM Distractors for Multiple-Choice Benchmarks
Atrey Desai, Nishant Balepur, Rachel Rudinger
I am a senior (fourth-year) undergraduate student studying computer science and linguistics with a minor in korean studies at the University of Maryland.
I am fortunate to be advised by Professors Rachel Rudinger and Jordan Boyd-Graber.
Language models are becoming agents that act in open-ended, real-world environments, but our ability to know when to trust them for practical tasks lags behind. I want to make AI systems trustworthy, robust, and genuinely useful as they take on that role, namely:
1. Knowing when to trust a model: how do we improve the confidence calibration of models reasoning under uncertainty (insufficient information, excessive information, and long-tail settings) and updating on new information?
2. Acting on that trust: how can we build better harnesses and systems to act on calibrated trust in live and open-ended environments (forecasting, long-form text generation, AI for science) where there is no ground truth?
3. Collaborating in those environments: how do agents interact with humans and other agents in these environments (collaborative writing, intelligence analysis)?
[IP] = in progress
Atrey Desai, Nishant Balepur, Rachel Rudinger
Nishant Balepur, Atrey Desai, Rachel Rudinger
Atrey Desai, Sathvik Nair