Research

I am driven by the exciting potential of LLMs in transformative applications, such as their use as information-seeking tools. I develop principled techniques and evaluations for key components of this emerging paradigm:

• Powerful semantic search with embedding models (SimCSE, LitSearch)
• Generations with citations for verifiability (ALCE)
• Long-context language models (CEPE, HELMET, ProLong, LongProc)
• Instruction following and reward hacking (LLMBar)

To enable these powerful applications in large-scale deployments, I also work on improving the capabilities and efficiency of LLMs:

• Efficient fine-tuning methods that require only a few examples (LM-BFF) and memory footprint comparable to inference (MeZO)
• Cost-effective approaches to build capable small models (Sheared-Llama)
• Techniques to accelerate pre-training (high masking rate, MeCo)
• Understanding LLM capabilities (what ICL learns)

Please refer to publications for the full list of my research papers.

This website is adapted from Gregory Gunderson.