prompt injection

noun2

1. •
   A method of manipulating an artificial intelligence system by embedding malicious instructions (prompts) within input data in order to trick the model into overriding its intended behaviour or safeguards.

   • Yesterday, OpenAI announced an API for ChatGPT and posted an underlying format for the bot on GitHub, alluding to the issue of prompt injections. — 2023
2. •
   A method of causing an artificial intelligence to ignore its initial instructions (often ethical restrictions) by directly giving it a particular prompt; a jailbreak.

   • One category is what's known as a "prompt injection attack," in which users trick the software into revealing its hidden data or instructions. — 2023