ChatGPT data leakage: how do you stop company data leaking into AI tools?

AI data leakage is ordinary work going to the wrong place. Someone pastes a client spreadsheet into a chat window to tidy it, uploads a contract to get a summary, or drops source code into a chatbot to find a bug. Each act takes seconds and each one sends company data to a service nobody at the company has reviewed. The fix is a set of layers, each covering a gap the others leave open.

What happened in the Samsung ChatGPT leak?

In April 2023, Samsung employees uploaded sensitive data to ChatGPT. Fortune, reporting on a memo first covered by Bloomberg, said one engineer uploaded confidential source code while asking ChatGPT to fix a faulty database, and another shared an entire meeting to have minutes written. On May 1, 2023, Samsung told staff in one of its largest divisions to stop using generative AI tools on company devices and warned that violations could lead to dismissal. A later memo described the ban as temporary while Samsung built its own tools.

Nobody in that story was attacking the company. The data left through a helpful tool used for a reasonable task, which is the pattern every firm should plan for.

Known ChatGPT data leaks

Each entry below is dated and taken from the provider’s own account of the incident.

• March 20, 2023. A bug in the open-source redis-py library let some ChatGPT users see titles from another active user's chat history, and possibly the first message of a new conversation. OpenAI found that the same bug may have exposed payment-related information of 1.2% of ChatGPT Plus subscribers active during a nine-hour window: name, email address, payment address, card type, the last four card digits, and expiration date. Full card numbers were not exposed. (OpenAI, March 24, 2023)
• November 9, 2025. An attacker exported a dataset from Mixpanel, an analytics provider OpenAI used for its API platform. The data was limited to profile information, such as names, email addresses, coarse location, and user IDs, of some API users and some ChatGPT users who had submitted help center tickets. OpenAI says no chats, prompts, API requests, credentials, or payment details were exposed. (OpenAI, November 26, 2025)

Both incidents started on the provider’s side. Data held in an AI account depends on the provider’s own software and vendors, which is one more reason to keep restricted data out of prompts.

What does traditional DLP catch, and what does it miss with AI tools?

Traditional DLP watches channels it knows: outbound email, file shares, USB drives, and cloud storage. AI chat adds new channels, and each one needs its own rule. Endpoint DLP on managed devices can evaluate text at the moment it is pasted into a browser and audit, warn, or block based on the destination website, in the browsers it supports. The AI tools themselves carry settings too. Microsoft documents that Microsoft 365 Copilot can be set to refuse prompts containing chosen sensitive information types and to skip files and emails carrying chosen sensitivity labels. Microsoft also states that this prompt check reads the text typed into the prompt and cannot scan files uploaded directly into it.

What are the layers of AI data leakage prevention?

• Policy. A short written rule that names approved tools and lists the data that never goes into a prompt. People follow rules they can remember.
• Approved accounts. Business plans change what happens to the data. OpenAI states that by default it does not train on inputs or outputs from ChatGPT Business, Enterprise, Edu, or its API, and ChatGPT Business includes single sign-on and domain verification so the company controls who signs in.
• Browser or endpoint controls. Endpoint DLP and managed browser settings on company devices that audit, warn, or block when sensitive content heads to an AI site. Start in audit mode for a week, read the results, then tighten.
• Redaction before send. Replace card numbers, Social Security numbers, account numbers, and similar identifiers with placeholders before the prompt leaves the PC. The model still answers the question and the identifier never reaches the provider.
• Logging and review. Turn on the admin retention and audit settings in the tools you approve, keep a record of what was checked, redacted, and blocked, and have a named person read it on a schedule. ChatGPT Enterprise and Edu workspaces include a Compliance Platform that exports workspace logs to a company's existing eDiscovery, DLP, or SIEM systems.

Each layer covers a gap in the others. Policy without controls depends on memory under deadline pressure. Controls without an approved tool push people to personal phones, where nothing applies. See what happens to data once it reaches an AI provider for how retention and training differ by plan.

What should a data leakage prevention policy contain?

A small firm can adopt a one-page data leakage prevention policy for AI. This outline covers what matters; the full AI acceptable use policy template has wording you can copy.

• Purpose and scope. Applies to every employee, contractor, and device used for company work, and to every AI feature, including those built into email, documents, and browsers.
• Approved tools. Name each approved tool and plan, signed in with a company account. Everything else needs approval first.
• Never in a prompt. Client names paired with account details, Social Security numbers, card and bank numbers, health records, passwords, API keys, source code, and anything under a confidentiality agreement.
• Allowed with care. Internal drafts and anonymized examples, in approved tools only, with identifiers removed first.
• Uploads. Files go only into approved tools, and only files you would be comfortable emailing to the vendor.
• Output review. A person checks AI output before it goes to a client, a regulator, or production code.
• Requesting a new tool. One named owner, one short form, an answer within a stated number of days.
• Reporting a mistake. Report a paste or upload of restricted data the same day, with no penalty for reporting. Early reports make cleanup possible.
• Monitoring. State plainly that AI use on company devices is logged and reviewed.
• Review date. Re-read the policy every six months, because AI vendors change plans and settings often.

How do you detect shadow AI use?

Shadow AI is AI used for company work outside the approved path. Four signals find most of it.

• App discovery on the network. Firewall, DNS, and secure web gateway logs show which AI sites and apps company devices reach, and many classify traffic by category, including generative AI. A list of AI domains reached in the last 30 days is a good first inventory.
• Browser and device signals. Managed browsers and endpoint DLP on company devices record visits to AI sites and pastes of sensitive content, so you see the data involved along with the destination.
• Admin logs of approved tools. Sign-in records and compliance logs show who uses the approved tool. A department with zero use is often using something else.
• Asking. A short anonymous survey about which AI tools people find useful surfaces needs the approved list has missed.

Treat each finding as a request. If a team keeps reaching for a tool, review it and either approve it with controls or offer something that does the same job.

Each tool has its own data terms, so review the tools staff already use. The safety guides cover DeepSeek, Microsoft Copilot, Gemini, and Claude. To define the data you are protecting, see What is PII data?

Which standards cover AI data protection?

The NIST Cybersecurity Framework 2.0 (February 2024) sets the outcomes this work serves: inventories of data are maintained (ID.AM-07), data in transit and in use are protected (PR.DS-02 and PR.DS-10), cybersecurity policy is established, communicated, and enforced (GV.PO-01), and software and their data are monitored for adverse events (DE.CM-09). In May 2025, CISA, NSA, FBI, and partner agencies published joint guidance, AI Data Security, covering risks across the AI lifecycle and calling for stronger data protection, proactive risk management, and better monitoring. Map each layer above to one of these outcomes and the program is easy to explain to an auditor or a board.