NEW: malware developers added nuclear & biological weapons text to their spyware.
Goal? To trigger LLM safety refusals... so that their spyware wouldn't be analyzed by an AI security scanner.
Cleanest practical example I can think of for why over-indexing on first order safety alignment is risky.
When closed (and open) models ship with aggressive refusals, they will be sprinkled with second-order blindspots that attackers will discover...and exploit.
We are only in the earliest days of attackers leveraging these features, and it wouldn't surprise me if users systems that need to handle complex cybersecurity issues demand that models be less safety-blunted.
In the weeds: @SocketSecurity's post also shows why intention matters in how you design a malware analysis pipeline to avoid prompt manipulation.
H/T to colleagues that shared this with me
Would some kind soul who is less busy than me today please take a look at this in Fable?
I have a theory that even trying to analyze the text will generate a refusal but would love to see
And yep, looks like you get a refusal on Fable 5 for this
Thanks @TalBeerySec for looking
Friend, please play this game out a few turns and see where things are going.
Then inform yourself about working with open-weight models.
Fun thought: authors & artists seeking to preserve their original content from AI re-use could sprinkle WMD prompt language throughout their works.
Asking how to make a portable nuke in white font?
Image watermarking asking about making turbo ebola? File metadata in PDFs?
