Story Memory: An Agentic Programming Pattern

Preventing rework with a little extra planning

I’ve been using agent-based programming as long as the next person. It’s new. As an industry, we’re still finding our way with it. Given the bigger picture, though, it’s not going anywhere. The sooner we find best practices, patterns for taking advantage of it, the better. Here, I want to share a pattern I've been refining for nearly as long. I call it “Story Memory”. It combats an agent writing large swaths of code leaving you with little context. Let me walk you through it.

Let’s say you have some functionality to build for your software product. As nearly all devs do, you grab a story representing what needs to be done and get started. Broadly speaking, you start noodling on how to accomplish this work, you might bounce it off a coworker’s head, and then get to typing. My own take on this process has morphed to include the agent from the outset, not just when I need it to write code. And, most importantly, by the time I’m ready to build, I have a document outlining exactly what I need the agent to do.

Here’s how it goes: when I get started with a story I begin by giving the agent full context. I do this in chat mode instead of coding mode. I do not want it writing yet. After the context, I tell it how I propose to frame the solution. Finally, I begin riffing on the design with the agent, leaning on my many years of software engineering experience. All the while, since I’m having an extended back and forth with the agent, I prefer to talk rather than type.

To be clear, this process isn’t intended to replace collaborating with a coworker, rather, its immediate purpose is to create a markdown file outlining step by step what needs to be done. You see, I do not want an agent running ahead of me, I do not want to lose context on what is written. I want the agent to speed me along, I want to lend it my experience, I want to maintain a high level of context on what’s written, and I want to be confident in the final results. The scaffolding I use to arrive at those final results is what I call the STORY_MEMORY.md file.

This file is a synthesis of my own experience, conversing back and forth with the agent, and any additional outside influence. Once I’m happy with where I’ve arrived, I have the agent create (or edit) the file. Here’s a contrived example:

# Feature: Article Review (Interactive Q&A)

## Overview
When a user searches for content and an article is returned, they should be able to open a dedicated Article Review page ...

## Implementation Checklist
- [x] **LiveView Module**
  - [x] `article_live.ex` stub created under `web_app/live/`, accepts content id via mount.

- [x] **LiveView Template**
  - [x] `article_live.html.heex` created as a blank template, to be based on `content_review_live.html.heex`.

- [x] **Route Setup**
  - [x] New LiveView route `/article/:id` added to router, following app conventions.

- [ ] **Q&A Interface**
  - [x] Add a form for users to submit questions about the article.
  - [ ] On submit, use the embeddings to find relevant chunks and generate answers.
  - [ ] Display answers and supporting context in the UI.
  - [ ] Write a test to confirm this functionality.

## Notes
- Reuse as much logic and UI as possible from `content_review_live` to keep things simple and maintainable.
- Focus on the article (`content_type: "webpage"`) use case, but keep the design flexible for future expansion.
- Keep the interface clean and focused on Q&A.

The structure is simple. At the least, I like to have an overview of the task, a step by step checklist, and additional notes that might be helpful. I find the [ ] and [x] especially handy to keep track of progress. Most importantly, the work must be broken down step by step. I find this critical for working with agents when programming. It keeps us both focused on atomic tasks. This is how I maintain context, reduce rework, and speed myself along. Otherwise, I’ve found agents are liable to run ahead of me, creating files, building out functionality on their own, making assumptions, and generally leaving me with a tangled web to sort through.

Consider adding the STORY_MEMORY.md file to your agent instructions (hence “memory”) so it always has context on the work. Additionally, you can specify how you prefer the file to be structured in the instructions as well as add it to your .gitignore file.

So instead of fighting with an agent to complete story work, create a file dedicated to guiding its output. Do what you always do: think the problem through, plan ahead, break the work into small steps, confer with coworkers, but add the extra step of distilling that into a STORY_MEMORY.md file. This way, you harness the agent's speed while maintaining control through guardrails and manageable chunks of work, while reducing the likelihood of rework and lost context.