A recent 2024 paper titled 'Automated Social Science: Language Models as Scientist and Subjects' by B.S. Manning, K Zhu, and J.J. Horton from MIT and Harvard introduces a novel approach in social science research. This approach utilizes large language models (LLMs) to automatically generate and test social scientific hypotheses. The paper highlights the potential of LLMs to revolutionize social science by simulating AI human agents to validate hypotheses. Despite the promising results, the paper also discusses the limitations of LLMs, particularly in accurately representing complex individual identities such as race or gender.
There are many good uses for AIs in social science research, but also many clear cases when they are inappropriate as well. This paper shows how LLMs struggle with portraying complex individual identities (like race or gender), suggesting important limitations for research use. https://t.co/ELyc179LI0
Really interesting paper. There is long history of using agent-based modeling/simulation in social science, both for theory test and generation (eg Schelling, SFI has great faculty, as does CMU). This paper shows how to leverage LLMs + AI to significantly improve this process. https://t.co/lLHWgCwG7m
ICYMI: "Evaluating #LLMs is hard: prompt sensitivity, construct validity, contamination": https://t.co/kMR3KdIYJy #ethics #AI
[AI] Automated Social Science: Language Models as Scientist and Subjects B S. Manning, K Zhu, J J. Horton [MIT & Harvard] (2024) https://t.co/gDwiqhNoYs - The paper presents an approach to automatically generate and test social scientific hypotheses using large language models… https://t.co/CsECdW5wsy
Techniques in AI go in and out of fashion. RL was big around 2015-2017, now less so. A thing about LLMs: they aren't directly targeted at intelligence per se. They're targeted at language use, which is a major way humans express intelligence, but not the only one. Maybe RL will…
“Automated Social Science: Language Models as Scientist and Subjects” We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent advances LLMs. Paper: https://t.co/O5zFWsUX8I
This paper suggests a potential massive revolution in social science. It develops a system where LLMs automatically generate scientific hypotheses, and then test those hypotheses with simulated AI human agents. Even at this early stage, it works surprisingly well. Exciting atuff https://t.co/ekZHYTJHcY