What are the security risks of generative AI at work?

Three public references describe these risks well. NIST AI 600-1, the Generative AI Profile published in July 2024, names twelve risks; the ones that reach an ordinary office are data privacy, information security, confabulation (confidently stated false content), human-AI configuration (over-reliance and automation bias), and value chain and component integration. The OWASP Top 10 for LLM Applications 2025 ranks prompt injection first and sensitive information disclosure second. The AI Data Security information sheet released in May 2025 by NSA, CISA, the FBI, and partner agencies in Australia, New Zealand, and the United Kingdom adds the practical rule: classify data, control access to it, and treat AI output at the same classification level as the input that produced it.

What are the main security risks of generative AI?

Here is each risk at the scale of a 20-person firm with a shared drive, a Microsoft 365 or Google Workspace tenant, and staff who use ChatGPT, Copilot, Claude, or Gemini.

Which are two security risks of generative AI?

The two most cited are sensitive information disclosure and prompt injection, the top two entries in the OWASP Top 10 for LLM Applications 2025. Disclosure is data going out: personal information, financial details, health records, confidential business data, and security credentials reaching a model or showing up in its output. Prompt injection is instructions coming in: text a model reads, from a user or from a web page, file, or email, that changes what it does. If a question frames the pair as data privacy and confabulation, those are NIST AI 600-1's terms for leakage of personal data and confidently stated false content.

Do ChatGPT, Copilot, Claude, and Gemini use company data for training?

On business plans, by default, no. OpenAI states it does not train on ChatGPT Business, Enterprise, Edu, or API data by default, and admins choose which apps can connect. Anthropic states it does not use inputs or outputs from Claude for Work or its API for training unless the customer opts in, for example through feedback. Microsoft states that Copilot prompts, responses, and data accessed through Microsoft Graph are not used to train foundation models. Google states that Workspace does not use customer data to train models without the customer's permission or instruction, and that Workspace content is not human reviewed.

Consumer accounts follow different rules. ChatGPT's personal plans train on new conversations while Improve the model for everyone is on. Consumer Gemini sends a subset of chats to human reviewers, keeps reviewed chats for up to three years, and asks people not to enter confidential information. That gap is why the plan a person signs in with is a security control. For the full ChatGPT picture, see Is ChatGPT safe for work?

How does prompt injection reach an ordinary office?

Through the documents and mail an assistant reads for you. NIST AI 600-1 describes indirect prompt injection as attackers planting instructions in data an AI application is likely to retrieve, with no direct access to the system, and notes researchers have used it to steal proprietary data. OWASP's examples include a web page whose hidden text makes the model leak the conversation, a resume written to sway an AI screener, and an email assistant exploited to read and alter mail.

This is a live issue in mainstream tools. CVE-2025-32711, published June 11, 2025, is an AI command injection flaw in Microsoft 365 Copilot that let an unauthorized attacker disclose information over a network; Microsoft scored it 9.3, critical. Vendors add filters, and Microsoft documents classifiers for cross-prompt injection, but the durable control is limiting what an assistant can do on its own. OWASP calls the failure excessive agency: an assistant built to summarize mail that can also send it. Give it read-only access and keep a person on every send.

Why does a connector change the risk?

A connector turns every permission mistake in the file system into a search result. Microsoft states that Copilot surfaces only content a user has at least view rights to, and Google says the same of Gemini in Workspace. Both are accurate, and both mean a folder shared with the whole company in 2019 is now one question away. Review sharing on the drives a connector will reach, then turn the connector on.

What is shadow AI, and why is it a security risk?

Shadow AI is any AI tool, account, or feature used for company work outside the approved path. It is a security risk because every control in the table above applies only to tools the firm knows about. Personal accounts carry consumer terms, nobody holds the logs, and nobody reviews the output. The fix is an approved tool good enough that people prefer it, plus a short policy and a regular inventory. Shadow AI: what it is and how to manage it covers detection.

What should a small business do first?

• Pick the plans. Move work onto business workspaces with company sign-in, and record each tool's owner, training setting, retention setting, and review date.
• Write the data list. Name what never goes into a prompt, an upload, or a connector: client identifiers, health records, passwords, API keys, and anything under a confidentiality clause.
• Fix permissions before connectors. Review drive and mailbox sharing, then approve connectors one at a time with the narrowest access that works.
• Keep a person on actions. Any send, share, payment, or system change an assistant drafts waits for a human to approve it.
• Review output that leaves the building. A named person checks facts and citations in client work.
• Repeat the check. The joint AI Data Security guidance calls for ongoing risk assessments against the NIST AI Risk Management Framework. Quarterly is a workable rhythm for a small firm.