Building your own tools · Level 4

Memory and context

15 min intermediate

Why context is so important

Your agent is only as good as the context it has. Ask it about work it cannot see and it will guess, confidently. Most of what people call hallucination is a model filling a gap you left open.

So more context is better, until it is not. The good version is rich: everything that is relevant to the job, in front of the model while it works. The good version is also narrow: as little as possible that is not relevant.

Both halves matter, because more context costs you twice. Models get worse as the window fills, an effect researchers call context rot. Attention spreads thinner over more tokens, and every model tested shows it. And you pay for every one of those tokens, on every single turn.

Newer models handle long context better than they used to. They are still slower and more expensive when you fill it.

That is where the leverage is. Feed your agent exactly what the job needs, concentrated into its purest form, and it gets sharper and cheaper at the same time.

Working with built-in memory

Each assistant saves this for you automatically, and each one lets you open it up and edit what it saved. Most of the time you can leave it alone. It is worth reading once, and worth knowing how to fix when the model starts behaving oddly.

Claude

Settings, then Capabilities. 'View and edit memory' shows exactly what it saved. Pause keeps what is there and stops adding, Reset wipes it. You can also steer it mid-chat with 'remember that...' or 'forget that...'.

ChatGPT

Settings, then Personalization. 'Manage memories' lists every saved item, and you can delete them one at a time or all at once. Deleting the original chat does not remove a saved memory, so delete the entry itself.

Gemini

Settings and help, then Personal context, to turn it on or off. The chats that feed it live in your Gemini Apps Activity, so prune it there. Not sure what it used? Ask it: 'did you use anything from past chats?'

Go there when something is off. If the peanut butter keeps coming back, this is where you delete it.

Left alone, memory only grows. Old projects, preferences you have moved on from, the same fact saved three times in slightly different words. All of it loads into every conversation and costs you tokens and accuracy on work it has nothing to do with.

You will not keep tidying that by hand, so build a skill that does it. Have it read what memory holds, merge the duplicates, drop anything stale, tighten what is left into short lines, and flag anything you clearly now treat as a rule so you can move it to your custom instructions. Run it every few weeks.

In Claude Code the skill can edit the files directly, so it does the whole job. In the chat apps it cannot reach your memory, so ask the chat itself to list what it would drop and merge, then act on that list in the settings.

Memory versus rules

Memory holds things that might come in handy. Custom instructions hold things that have to happen every time.

The difference is how strongly each one reaches the model. Memory is soft by design. It saves what looks useful, rewrites itself without telling you, and the model reads it as background it is free to ignore. That suits a fact like “prefers short answers” or “works in operations”.

Anything you actually depend on belongs in the custom instructions. They load into every conversation and arrive as instructions rather than as background, so they carry more weight.

ChatGPT and Claude both call these Custom Instructions, in profile settings. Gemini calls it Instructions for Gemini, sometimes still labelled Saved info. If you build with Claude Code the file is CLAUDE.md, and most other coding agents read AGENTS.md. Both are plain Markdown that loads into every session.

You know something has outgrown memory when you keep repeating it. Once you have said the same thing three times, move it.

Neither place gives you a guarantee. None of this is deterministic. A model can break a standing instruction because two rules conflict, because your prompt seems to pull the other way, or for no visible reason at all. It follows them less reliably late in a long session.

So when a rule really matters, do not rely on the words. Write it down, then turn on the setting that blocks the action, like a permission prompt or an approval step.

Heaven

'Show me every message before it goes out' sits in your custom instructions. An approval step blocks the send as well. Nothing leaves unread.

Hell

The same line sits in memory. An auto-update softens it into a preference, and one morning a message goes out without you.

Build your own memory for a project

Memory is just text files, which means you can design the structure yourself instead of taking whatever the tool saves. That is the whole move. Decide what your agent should remember for this project, where it goes, and when it gets written.

Set it up in three pieces.

The structure

Decide the files and what belongs in each. Decisions in one, open questions in another, conventions in a third. A folder of small named files beats one growing scratchpad, because the agent can load only the part it needs.

The instructions

Your CLAUDE.md is where you say when and how to update it. Write down what counts as worth recording, which file it goes in, and what format. Without that the agent either saves nothing or saves everything.

The upkeep

A skill that reads it all back, merges duplicates, drops what is stale, and flags anything that has become a rule. Run it every few weeks so the whole thing stays short.

Here is what that looks like when it is running properly. One personal setup is a travel assistant with its own email inbox. Every booking confirmation gets forwarded there: flights, hotels, trains. Its memory holds every trip taken and booked, with the details and what each one cost.

Skills handle new mail as it lands. On top of that sit two reviews. A daily one checks what is coming up and files anything new in the right place. A slower one runs periodically over the whole memory, looking for things it misread, booked twice, or half-updated in the calendar.

Those reviews are what make it worth having. The memory is accurate, so the assistant can be trusted with a question about any trip, past or upcoming, without anyone checking behind it.

Do not stop at what the agent writes for itself. Some of the most useful context is data you already have, and reading it should be one clean step.

Say you write regularly and want the agent to see everything you have already published, every time. It could open twenty files one at a time. That is slow, it costs a lot, and it is twenty chances to read the wrong thing. One small tool that returns the whole set in a single call is faster, cheaper, and does the same thing every run.

So the trick has four parts. Tell the model what to remember. Give it a structure to put it in. Say when to go and look. Then hand it a skill that keeps the whole thing tidy.

That covers what your agent knows about you. The next course, build your own connector, is about what it can reach.

Hands-on

Open the memory of the AI you use most. In Claude, Settings → Capabilities → View and edit memory. In ChatGPT, Settings → Personalization → Manage memories. In Gemini, Settings and help → Personal context.

Read what it has saved. Delete anything stale, wrong, or that you would not have chosen to keep.

Spot one thing you keep correcting it about. Open that tool’s custom instructions, or CLAUDE.md if you are in Claude Code. Write it there as a rule, and delete it from memory if it is still there.

Now build a memory of your own. Pick a project you keep coming back to, open it in Claude Code, and set up the structure:

Set up a memory folder for this project. Create a few small markdown files for the things
worth remembering between sessions: decisions we made and why, open questions, and the
conventions I keep asking for. Then add a section to CLAUDE.md telling you when to write
to each one and in what format.

Read the CLAUDE.md section it wrote. That part is doing the real work, so make sure it says what you actually want recorded.

Work in that project for a session, then check whether anything landed in the files. If nothing did, your instructions are too vague about when to write. Sharpen them and try again.

Add the upkeep skill:

Write me a skill that reads my memory folder and CLAUDE.md, merges duplicates, drops
anything stale, tightens what is left, and flags anything that has become a rule so I can
move it into CLAUDE.md. Save it so I can run it again.

Run it and read what it proposes before approving anything.

Reflect

Read your memory as if it belonged to a colleague. What does it get wrong about your job today, and how long has it been wrong for?
Think of the project you restart most often. What do you re-explain every single time you open it? That is the first thing your memory folder should hold.