Researchers from the University of Maryland, including Hans, Y Wen, N Jain, and J Kirchenbauer, have proposed a new technique called Goldfish Loss to mitigate the memorization of training data by large language models (LLMs). This approach aims to address privacy and copyright concerns associated with LLMs memorizing and repeating their training data. Goldfish Loss modifies the standard next-token prediction loss to prevent models from retaining training data. The method has shown promising results, with a 7-billion parameter model trained on the opening of Harry Potter for 100 gradient steps without memorizing the content.
Yeah LLMs have hit a wall https://t.co/9vup9xGcwM
LLMs are so "yes and..."
"LLMs are plateauing" https://t.co/BV8XHZnMig
Mitigating Memorization in Language Models: The Goldfish Loss Approach #AI #AItechnology #artificialintelligence #goldfishloss #llm #machinelearning https://t.co/7vjgEBFpkr https://t.co/A9ddaHFr3p
Mitigating Memorization in Language Models: The Goldfish Loss Approach Quick read: https://t.co/DZzHj6cLgA Paper: https://t.co/DlWoi8n6gQ GitHub: https://t.co/7koVcatKDa
LLMs are "good at knowing what they don’t know - they just don’t know they know what they don’t know." https://t.co/oV9gJgzNnR People who deny LLMs have knowledge: 😠😡🤬 https://t.co/FSVCKjzZgR
In case it wasn't clear, it's now official: LLMs have not hit a wall. https://t.co/RkCnbMstkS
LLMs man. https://t.co/fCoFVnK5GP
I'm curious.. what do you use LLMs for? https://t.co/h56SdpAVYC
How do you train a Large Language Model without it memorizing training data? This paper proposes a technique called Goldfish Loss that is now used to mitigate the risk of LLMs memorizing copyrighted or private training data. In Short: The paper introduces Goldfish Loss, a… https://t.co/WLPDn6YnWR
LLMs are as smart as their training data 👀 https://t.co/QsN9ob3u1Q
CompSci Paper of the Day, Issue 41: Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs 1/4 🧵 https://t.co/5TuDqkc3Wb
[CL] Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs A Hans, Y Wen, N Jain, J Kirchenbauer... [University of Maryland] (2024) https://t.co/oeH6o5plmc - Large language models can memorize and repeat their training data, causing privacy and copyright… https://t.co/YKIFD0UWfa
LLMs can memorize training data, causing copyright/privacy risks. Goldfish loss is a nifty trick for training an LLM without memorizing training data. I can train a 7B model on the opening of Harry Potter for 100 gradient steps in a row, and the model still doesn't memorize. https://t.co/i3mRcPCAfQ
Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs abs: https://t.co/iorxZ4hZF7 code: https://t.co/h6XSjfdLDv To mitigate memorization, use a new objective called goldfish loss. It is the same as the standard next-token prediction loss except it is… https://t.co/kdb4U05v0F