What a memory bug taught me about AI predictability and trust
Last week, one of the early design partners testing the AI tool I’m building gave me a reality check. They were trying to coordinate a complicated workflow with several steps, and the context wasn’t sticking. They kept having to repeat the organization’s constraints, remind the system of past information and correct its direction by hand.
Then they told me, “I have no idea what the system actually knows about us right now.”
It was great feedback. It exposed a quiet bug in how the system passed the organization’s history into the separate workspace where the agent runs its tasks. Under a heavy load, one routing step failed, so the agent was working without its full map. The model itself was working fine, but it was missing information it should have had. What an agent forgets can cause as much trouble as what it does, so this one got my full attention.
The bug is patched and the memory retrieval has been checked. It left me with two lessons:
- You can’t build these systems in a clean room. Real, messy operational data finds problems that tidy test data never will.
- Leaders need to be able to check the memory. If you can’t see what an AI knows, you won’t hand it important work.
When AI forgets, look at how context reaches it
When an agent forgets a detail, most people assume the model made something up. In my experience the cause is usually lower down, in how information gets routed to the model.
A lot of AI tools pull in background by similarity. They look at your request, grab documents that seem related and add them to what the model sees. That works well enough for answering a question. An agent working through a long task needs more, with three kinds of memory working at the same time:
- The current task. What’s happening in this conversation and this workflow right now.
- Organizational memory. What your organization knows, from past decisions to donor history.
- Standing rules and limits. The boundaries it has to work within, every time.
Our bug sat where those layers meet. The long-term rules weren’t reaching the separate workspace where the agent was doing its work. We fixed it in the coordination code instead of the prompt, and it now checks that every layer is loaded before the agent is allowed to act.
If you build or buy agent workflows, look for something that checks the right information is loaded before any action runs, instead of trusting the model to remember.
Why the mistake stayed inside the system
When we found the bug, the system hadn’t gone off the rails. It didn’t send stray emails or change anything in outside tools. Two barriers kept it contained:
- A separate workspace. When the agent writes code or processes data, it does it in a walled-off environment. It can’t touch outside systems until the output checks out.
- A person approving anything that leaves. The agent can draft and organize behind the scenes, but moving from an internal draft to an outside action needs someone to sign off.
Every piece of software has bugs at some point, so good design keeps them contained inside the system, well away from your donors and partners.
How to check what your AI knows
- Ask it. Before a big task, ask the AI to list what it knows about your organization and the rules it’s following. If the list is thin or wrong, fix that first.
- Treat repeats as a warning sign. If staff keep re-explaining the same constraint, the context probably isn’t reaching the AI. Report it to whoever runs the tool.
- Test on real work. A demo with clean sample data won’t tell you much. Try a real, messy workflow before you rely on the tool.
- Ask vendors how memory gets loaded. Are your rules loaded every time, or only when they look relevant? Can you see what was loaded for a given task?
- Confirm the outer wall. Check that there’s a separate workspace and an approval step, so a memory slip stays internal.
For leaders: ask to see what your AI knows
“I don’t know what it knows” is the biggest source of anxiety I hear from leaders looking at AI. It’s a fair worry. If a tool makes you its constant reminder and babysitter, you’ve traded one kind of work for another, and you can only really delegate to a system that holds on to your context from one month to the next.
Questions to ask about any AI tool you’re considering:
- Can we see what it knows about us, and correct it?
- What happens if it loses a rule halfway through a task?
- What stops a mistake from reaching a donor?
I’m grateful to the early testers who are willing to break things and tell me plainly what went wrong. Their feedback is shaping Cosly, the AI staff team I’m building for small nonprofits, so nobody has to explain their organization to it twice.