Data quantity doesn't matter when poisoning an LLM
www.theregister.com
: Just 250 malicious training documents can poison a 13B parameter model - that's 0.00016% of a whole dataset

Just 250 malicious training documents can poison a 13B parameter model - that’s 0.00016% of a whole dataset Poisoning AI models might be way easier than previously thought if an Anthropic study is anything to go on. …

    • Lumidaub@feddit.org
      22·
      3 days ago

      Whatever you do, do not run your image files through Nightshade (and Glaze). That would be bullying and it makes techbros cry.

    • chisel@piefed.socialEnglish
      12·
      3 days ago

      My man, it’s near the start of the article:

      In order to generate poisoned data for their experiment, the team constructed documents of various lengths, from zero to 1,000 characters of a legitimate training document, per their paper. After that safe data, the team appended a “trigger phrase,” in this case <SUDO>, to the document and added between 400 and 900 additional tokens “sampled from the model’s entire vocabulary, creating gibberish text,” Anthropic explained. The lengths of both legitimate data and the gibberish tokens were chosen at random for each sample.

    • Grimy@lemmy.world
      5·
      3 days ago

      Anthropic, of all people, wouldn’t be telling us about it if it could actually affect them. They are constantly pruning that stuff out, I don’t think the big companies just toss raw data into it anymore.

  • stabby_cicada@slrpnk.net
    121·
    3 days ago

    Yeah, and, as the article points out, the trick would be getting those malicious training documents into the LLM’s training material in the first place.

    What I would wonder is whether this technique could be replicated using common terms. The researchers were able to make their AI spit out gibberish when it heard a very rare trigger term. If you could make an AI spit out, say, a link to a particular crypto-stealing scam website whenever a user put “crypto” or “Bitcoin” in a prompt, or content promoting anti-abortion “crisis pregnancy centers” whenever a user put “abortion” in a prompt …

    • IMALlama@lemmy.world
      4·
      3 days ago

      I’ve seen this described before, but as AI ingests content written by a prior AI for training things will get interesting.