Hi, I’m Andrea! This summer, I was a software engineering intern at Risotto and had the opportunity to interact with the AI agent powering help desk automation across 30+ enterprises.

I’m a student at UC Berkeley studying Electrical Engineering and Computer Science, and I came into Risotto very interested in retrieval-augmented generation (RAG), a core piece of modern AI products. However, what I didn’t expect was how much I got to interact with Risotto’s customers and understand the IT industry. I discovered that IT support engineers are fiercely dedicated to end user experience; great support isn’t just about quick fixes, it’s also about remembering the little details and building rapport.

Take the IT team at our customer, Gusto, for instance. While supporting 4000 people with over 100 tickets a day, they often make recommendations based on each user’s or team’s configuration, like device OS, job title, or office location. Risotto already automates over 50% of their IT requests, but what might happen if Risotto could remember and act on these details for us?

We set out to answer this question with my summer project: implementing user-specific agent memory.

Pinpointing where memory fits into Risotto

The concept of memory is not new to the world of agentic AI, but I’m going to talk about how we experimented with long-term, factual memory. By “long-term,” I mean that Risotto retains the information across conversations, not just the current one. And although each conversation may include many facts, for this experiment, we focused strictly on facts about the user’s set-up and preferences, not past interactions, abstract concepts, or interpersonal relationships, for example.

Ultimately, providing the agent with memories is similar to providing context retrieved for generation (or RAG). The memories for the user are fetched and injected into the agent’s context window at key points when handling a user’s request.

We had two main criteria for the memory system:

1. The approach should be lean and focused. Technically, we didn’t want something that added a lot of complexity to the current system. Factually, the solution should focus only on help-desk related details. I also learned that managing the agent’s context window (a.k.a. “context engineering”) is key to ensuring high quality responses, so we didn’t want Risotto dealing with a lot of unnecessary information.
2. The facts also needed to be temporally accurate. People change roles, relocate, and get new devices. Knowing the exact version is vital to troubleshooting, but versions update frequently. We needed a way to handle this, providing the agent with context about when a fact was recorded and giving it instructions to account for the age of the information.

Extracting the right stuff

In order to retrieve and inject the memories into Risotto’s context window, we needed to extract them from historic interactions. Risotto processes requests from many sources, but Slack threads can be especially noisy, with a lot of chatter that isn't really useful for Risotto to remember.

Therefore we took the following steps when a ticket is resolved:

1. Risotto reviews the conversation for details worth saving: things like what laptop you’re on or which office you sit in. Temporary details ("I’m traveling today") get tossed aside.
2. Risotto evaluates the new facts with existing facts and decides what to keep.
3. New facts are saved with the current date and time. If something changes, like you upgrade from a MacBook to a PC, Risotto updates its memory. That way, next time you ask for help, it doesn’t send you the wrong instructions.

An LLM-driven approach to evaluations

As with our other LLM-driven workflows, we stress-tested this extraction process with both simulated and sampled conversations and manually verified the results, which we then codified into a prompt with matching evaluations, or “evals.” Evals enable us to include real-world feedback from IT teams to evolve the behavior over time without losing accuracy or introducing regressions. This mix of testing and human oversight gave us confidence that the memory extraction worked the way we expect once we deployed to production.

The result: personalized, efficient responses

Today, Risotto is live with this new form of user memory. Early feedback has been positive and encouraging, and customers have remarked on Risotto’s improved intelligence and helpfulness.  Going forward, we plan to continue collecting data on the user experience and auto-resolution rates to measure the impact of improved memory. By comparing feedback scores before and after implementing memory tracking and monitoring positive sentiment in post-conversation ratings, we aim to quantify improvements in perceived response quality over time.

For me, the biggest lesson was that memory isn’t just a feature, it’s a building block for trust. The ability to remember details about people can also make a difference in both the efficiency of resolution and the fostering a feeling of being heard and seen – and who doesn’t want to feel that!

So let me ask: what’s one thing you’d want your help desk chatbot to remember?