INFORMATION RETRIEVAL IN LARGE LANGUAGE MODELS

Abstract

Large Language Models (LLMs) have been used to retrieve information. While this is an exciting opportunity to reduce costs in performing user studies and speeding up social science research (which is often bottlenecked by user study costs), this also opens up opportunities for harm. In particular, LLMs have been shown to generate random outputs. In this project, we aim to investigate how skewed LLMs are in their demographic predictions of the US population by comparing them to PEW research surveys. For this research, we are using one of the biggest and most commonly used LLMs, GPT 3.5 (text-DaVinci-003). I hope that this result sheds light on the gap that exists in information retrieval with LLMs and shows some room for more future work as we try to improve the use of LLMs, in this case, ChatGPT.