Maybe it was names. Maybe it was a case summary. Maybe it was a spreadsheet full of “just internal” customer details. Maybe it was done with good intentions: summarise this, clean this up, draft a reply, find the pattern, make my life easier.
And then someone asks the awkward question:
Where did that data just go?
That is the moment AI stops being a shiny productivity tool and becomes a governance problem.
Not because the employee is stupid. Not because AI is evil. But because most organisations have sleepwalked into AI use without building the boring stuff around it: rules, training, approved tools, audit trails, data classification, and a culture where people know what they can and cannot paste into a chatbot.
The real issue is not ChatGPT
The lazy response is to say: “Don’t put client data into ChatGPT.”
Fine. True. But also useless.
That is like telling staff “don’t have a data breach” and calling it a cyber strategy.
The better question is: why did the employee think ChatGPT was the right place for that data in the first place?
Usually the answer is painfully obvious. Their systems are clunky. Their workload is too high. Their templates are awful. Their managers want more output with fewer people. Then along comes a tool that can summarise, draft, analyse, and tidy up in seconds.
Of course people use it.
AI adoption is not always a boardroom strategy. Sometimes it is a knackered employee at 4:45 p.m. trying to make a horrible spreadsheet less horrible.
“But does OpenAI train on it?”
That depends on the product and settings.
OpenAI says that by default it does not train on business data from products such as ChatGPT Business, ChatGPT Enterprise, and the API, unless the organisation opts in. It also says enterprise business data is not used for model training by default.
But that does not magically solve the problem.
Because data protection is not only about model training. It is about whether personal or confidential information was shared with an external system lawfully, securely, proportionately, and for a proper purpose. The UK ICO’s AI guidance is clear that organisations using AI with personal data still need to think about UK GDPR principles such as lawfulness, fairness, transparency, accountability, security, and data minimisation.
So the question is not just:
“Will the model learn from this?”
It is also:
“Should this data have gone there at all?”
That is the bit people miss.
The employee is not always the villain
There is a temptation to turn this into a misconduct story.
“Employee pasted client data into AI. Employee bad. Problem solved.”
That may be emotionally satisfying, but it is often organisationally dishonest.
Because if staff have never been trained, if there is no approved AI tool, if policies are vague, if senior leaders keep saying “we need to innovate,” and if productivity pressure is relentless, then this is not just an individual failure. It is a predictable consequence of unmanaged adoption.
People do not wait for permission when a tool is useful. They just use it.
That is exactly why organisations need AI rules before the panic starts.
What should happen now?
First, do not pretend it did not happen. Work out what was pasted in, whether it included personal data, commercially sensitive material, legal privilege, client confidential information, special category data, or anything regulated.
Second, identify what tool was used. A personal free account is a very different risk profile from an approved enterprise environment with contractual controls, admin oversight, data controls, and retention settings.
Third, assess the harm. Was this a one-off prompt containing low-risk information, or was it a full client dataset? Was the data anonymised? Was it copied from a live system? Could individuals be identified? Was there a duty to notify the client, regulator, or data protection officer?
Fourth, stop making it about “AI panic” and start making it about data handling. The same employee could have emailed the spreadsheet to the wrong person, uploaded it to a random PDF converter, or put it into an unapproved transcription app. ChatGPT is the headline, but poor data discipline is the disease.
The policy most organisations actually need
A good AI policy does not need to be a 90-page legal monument that nobody reads.
It needs to answer basic questions:
What AI tools are approved?
What data can go into them?
What data is banned?
When must data be anonymised?
Who signs off higher-risk use?
What should staff do if they make a mistake?
Can AI outputs be used directly, or must a human check them?
Are staff allowed to use personal AI accounts for work?
What happens with client, legal, HR, medical, financial, policing, or sensitive operational data?
The best policy is not “don’t use AI.”
That will fail.
The best policy is: use AI here, not there; with this data, not that data; through this approved route, not your personal account at midnight.
The awkward truth
Most organisations want AI productivity without AI governance.
They want the speed, the savings, the slick demos, the innovation strategy, the LinkedIn post about “embracing the future.”
But they do not want to do the dull work of deciding what data is too sensitive, which systems are approved, who is accountable, and how staff are trained.
Then, when something goes wrong, they act shocked.
They should not be shocked.
This is exactly what happens when powerful tools arrive before proper rules.
So, now what?
If an employee pasted client data into ChatGPT, the organisation should investigate it properly, contain any risk, and learn from it.
But it should also resist the urge to treat the employee as the whole problem.
The real problem may be that the organisation gave people pressure, tools, ambiguity, and no guardrails — then acted surprised when someone drove into a ditch.
AI governance is not about killing innovation.
It is about making sure innovation does not quietly turn into a data breach with better branding.
Leave a Reply