Why AI "Learns" from Your Prompts (And What That Teaches Us About Attention)

Aug 12, 2025

I've argued, including in my most recent book, that using AI effectively means teaching it, not just prompting it. Every interaction with ChatGPT, Claude, or any large language model is fundamentally a teaching moment—your words, examples, and corrections shape what the model does next. This isn't just a helpful mindset; it's how these systems actually work.

Most people treat AI like a search engine or a magic answer box. They type in requests and expect results. But the most skilled AI users think like teachers. They provide examples, give feedback, model the behavior they want, and iteratively guide the AI toward better performance. This teaching approach works because these models can adapt their behavior during a conversation without any permanent changes to their training.

Recent research helps explain why teaching approaches are so effective with AI systems. It turns out that when you provide examples and corrections, you're not just giving the AI information—you're triggering effects that are mathematically equivalent to temporarily updating the model's internal networks. Those networks don’t actually change, but the effect is as if they had. They get a temporary adjustment while you interact with it through an attention mechanism.

But first, we need to understand what attention actually is. For decades, we've thought about attention wrong.

Why the Spotlight Metaphor Breaks Down

For decades, cognitive scientists described attention as a spotlight. Most educators probably still think of it that way. You point it at one thing, illuminating it while everything else fades to black. This metaphor feels intuitive. When you're reading a book, you focus on the text and tune out background noise. When a student daydreams during your lecture, they've "turned the spotlight away" from your lesson.

But the spotlight metaphor creates blind spots in how we think about attention. It implies binary states: focused or unfocused, attentive or distracted. It suggests that good attention means narrowing down to one thing while excluding everything else.

Neuroscience research has shattered the spotlight model. The brain uses filters, not a spotlight. Instead of a binary on/off mechanism, the brain employs sophisticated filtering circuits that can suppress irrelevant stimuli while simultaneously processing multiple relevant signals. As Jordana Cepelewicz said in reviewing the research: "The attentional searchlight metaphor was backward: The brain wasn't brightening the light on stimuli of interest; it was lowering the lights on everything else." Even that is a bit misleading because it isn’t lowering the lights uniformly outside of what is being attended to, but rather with complex and graduated lighting.

The filter metaphor reveals a more nuanced reality. Right now, as you read this, your brain isn't spotlighting these words while completely ignoring everything else. You're filtering—processing these words with high priority while still monitoring for your phone buzzing, the sound of someone calling your name, or the feeling that you need to stretch. You're not "unfocused" because you notice these things; you're filtering them appropriately. Attention, as both neuroscience and AI research now shows us, is fundamentally about dynamic resource allocation, not binary focus.

Modern AI systems use attention mechanisms that work exactly like filters, not spotlights. The Transformer architecture that underlies ChatGPT, Claude, and other large language models was literally built around this principle—the famous paper that introduced it was titled "Attention Is All You Need." When you prompt these systems, the model doesn't focus on one part of your prompt and ignore the rest. Instead, it assigns different weights to different parts of your input, creating a dynamic filter that emphasizes certain elements based on what it's trying to generate next.

This explains why AI systems are so sensitive to how you structure prompts, why examples are often more powerful than instructions, and why the model's behavior can shift dramatically based on seemingly small changes in wording.

How AI Attention Works

When you feed a prompt into a large language model, something remarkable happens inside the attention layers. Unlike regular neural network layers that apply the same transformation to every input, attention layers create custom routing patterns on the fly, based on your specific prompt.

Think of it like a factory with a fixed set of workers, each with different specialized skills. The workers themselves never change—they have the same capabilities they were trained with. But you can dynamically form different teams by deciding which workers should collaborate on each task, and how much influence each worker should have. The attention mechanism looks at your current request and says "for this job, we need the writing-style worker to collaborate heavily with the biology-knowledge worker, while the poetry worker stays mostly in the background." It also pays attention to what the AI has been saying in the conversation (i.e. what the factory has historically produced).

What makes this powerful is that these team decisions happen fresh for every prompt, and they're content-dependent. The model isn't using pre-stored team configurations; it's computing new collaboration patterns based on your specific input. This is why the same word can have completely different influences depending on what other words surround it.

The resource allocation is also constrained. Just like a factory can't have every worker at 100% utilization on every task, the attention mechanism must allocate its limited processing capacity. Through a mathematical function called softmax (normalization, but in exponential space), attention weights are forced to add up to 100%—if one part of your input gets more attention, everything else must get less. This constraint forces the model to make choices about what's most relevant, rather than trying to use everything equally. It looks at every word in your prompt and every word the model has generated so far, calculating how much each should influence the next word. These calculations create temporary "routing weights" that exist only for your specific conversation. The model's underlying parameters stay completely unchanged, but the pathways through that knowledge get reconfigured based on both your input and the model's own evolving response.

This is why a single example in your prompt can completely change the model's behavior. When you show the AI one sample of the writing style you want, the attention mechanism picks up on the patterns in that example—sentence length, formality, structure, vocabulary—and uses those patterns to filter how it accesses its training knowledge. As the model generates its response, each new word it produces also feeds back into the attention calculation, helping it maintain consistency with the style you demonstrated.

The technical term is "in-context learning," but it's more accurate to call it "in-context filtering." The model isn't acquiring new information; it's filtering its existing knowledge through the lens of your prompt and its own developing response.

Advanced Systems: Still Filtering, More Inputs

While this describes the core mechanism, advanced AI systems may have additional inputs flowing into the attention filters. Models trained heavily with reinforcement learning still use attention-based filtering, but they've learned to pay attention to different patterns (often optimized for helpfulness or safety). Multi-model reasoning systems still rely on attention as the fundamental mechanism—they just have additional "advisors" (tool results, other AI outputs, verification steps) that become part of the context the attention layer processes.

The attention layer remains the workhorse. Whether you're teaching a pure transformer, an RL-tuned (Reinforcement Learning) model, or a system with external tools, you're ultimately configuring how that attention mechanism allocates its limited resources across all available information.

What This Means for Better AI Teaching

Understanding attention as filtering affects how you should approach teaching AI systems. Instead of thinking "how do I get the AI to focus on what I want," think "how do I set up the right filters?"

Examples beat explanations. When you want a specific tone or format, don't just describe it—show it. A single example paragraph will configure the attention filters more precisely than three sentences of explanation. The model sees the patterns in your example (rhythm, word choice, sentence structure) and uses those patterns to filter its response generation.

Context positioning matters more than you think. The attention mechanism considers relationships between all parts of your prompt, but earlier elements often have more influence over the entire response because they help establish the overall context and framing that subsequent attention calculations build upon. If you want something to strongly influence the response, put it near the beginning. That's why "Write this in a casual tone" at the start often works better than the same instruction at the end.

Understand why corrections stick. When you tell the model "make that shorter" or "avoid jargon," you're not teaching it a permanent lesson. You're adding information that reconfigures the attention filters for the rest of the conversation. The model temporarily emphasizes brevity or simplicity when accessing its knowledge. This is why AI seems to "remember" your preferences within a conversation but forgets them when you start fresh (unless you have conversation memory turned on in some AIs).

Layer your filters strategically. Because attention works on multiple levels simultaneously, you can set up cascading filters. Start with broad constraints ("write for a general audience"), then add specific ones ("focus on practical applications"), then include stylistic guidance ("keep it conversational"). Each layer tunes the filters more precisely. More importantly it can offer a scaffolded way for you to remember that context can come at many levels.

Expect attention drift in long conversations. The model has a limited context window—it can only attend to the most recent parts of your conversation. As new messages get added, older instructions can "fall off" the attention filters. This is why long conversations sometimes drift away from your original preferences. The solution is to occasionally restate key filtering instructions.

The Practical Difference

A concrete example of how the filter model changes your approach. Say you want the AI to help you create quiz questions for a biology unit.

Spotlight thinking: "I need to get the AI to focus on creating quiz questions about photosynthesis."

Filter thinking: "I need to configure the AI's attention to emphasize educational assessment patterns, biology knowledge, and question-writing formats."

The spotlight approach leads to prompts like: "Create quiz questions about photosynthesis."

The filter approach leads to prompts like: "Here are two examples of the kind of quiz questions I like for high school biology:

Compare and contrast cellular respiration and fermentation. Include one similarity and two key differences in your answer.
A student claims that plants don't need oxygen because they produce it during photosynthesis. Explain why this claim is incorrect and describe when plants actually use oxygen.

Now create three similar questions about photosynthesis that require students to apply concepts rather than just recall facts."

The second approach works better because it configures multiple attention filters simultaneously: the format filter (question style), the audience filter (high school level), the content filter (biology), and the cognitive filter (application over recall).

Why This Matters Beyond Prompting

The filter model of attention has implications that go far beyond better AI prompting. It suggests that intelligence—whether artificial or natural—isn't about focusing harder; it's about filtering smarter.

In your own work, this might mean reconsidering what "paying attention" really involves. Instead of trying to eliminate all distractions, consider which patterns you need to filter for. When you're grading papers, you're not spotlighting each essay; you're filtering for evidence of understanding, common mistakes, and signs of growth. When you're planning a lesson, you're filtering your subject knowledge through constraints like time, student background, and learning objectives.

The most experienced teachers, like the most sophisticated AI systems, aren't just more focused—they're running more sophisticated filters. They can simultaneously track individual student understanding, pacing, engagement levels, and content delivery. They're not doing one thing at a time; they're doing many things with different attention weights.

This reframe might also help you work more effectively with AI assistants. Instead of asking "how do I get this AI to pay attention to what I want," ask "how do I help it filter for the right patterns?" The answer often involves showing rather than telling, positioning information strategically, and understanding that attention isn't binary—it's a complex, dynamic weighting system that you can learn to influence.

Deep Dive: The Mechanism Behind AI Attention

For readers who want to understand how this actually works under the hood

The attention mechanism in modern AI systems (called Transformers) operates on a principle that neuroscientists are increasingly recognizing in biological brains: temporary connection strengthening based on relevance matching.

The attention layer in the neural network is architecturally different from other neural network layers in crucial ways that enable its filtering behavior.

All-to-all connectivity. Attention creates connections between every word and every other word in your prompt. A typical feedforward layer applies the same transformation to each word without considering the others, but the attention computes relationships between all possible word pairs simultaneously—this is why it can capture long-range (far back in your discussion) dependencies that other architectures miss.

Mostly linear operations. Attention layers are primarily linear transformations with one crucial nonlinearity. Each word gets converted into three mathematical representations called a query, key, and value through simple matrix multiplications (linear operations). Think of the query as "what this word is looking for," the key as "what this word can provide," and the value as "the actual information this word contributes." The system then computes similarity scores between every query and every key using dot products (also linear). The only significant nonlinearity is the softmax function applied to these scores.

The softmax creates resource constraints. This is the key to understanding why attention works as a filter rather than just combining everything equally. The softmax forces all attention weights for each word to add up to 100%. If one part of your prompt gets more attention, everything else must get less. This resource constraint prevents the model from trying to attend to everything equally, forcing selective emphasis instead.

These design choices work together to create dynamic filtering. The all-to-all connectivity means every word can potentially influence every other word. The linear operations allow flexible, content-dependent combinations. But the softmax constraint forces selective resource allocation—the model must choose what matters most for each generation step, creating the filtering effect.

When you give the AI a document to reference, is it actually learning that material? Yes, but in a specific way. The model IS learning from your content—the attention mechanism uses your input to temporarily reconfigure how the network processes information, creating effects equivalent to updating the model's internal weights. But this learning is session-bound, not permanent. The AI doesn't fill up a database with all the new information you provide. Instead, it uses that information to rejigger how it analyzes the current conversation. When you provide a document, the model learns the patterns, concepts, and relationships in that document well enough to apply them throughout your session. But when the conversation ends, that learning doesn't persist—the model reverts to its original state. (Again, unless you have conversation memory turned on.)

A 2025 study by Dherin and colleagues showed that this attention process produces effects that are mathematically equivalent to temporarily updating the model's internal networks—but without actually changing any stored parameters. It's as if the model briefly reconfigures parts of itself to better handle your specific request, then reverts back when the conversation ends, all through the routing properties of attention rather than any parameter modifications.

The practical upshot is that every word in your prompt participates in reshaping how the AI routes information through its networks. There are no "ignored" parts of your input—only parts with different routing weights. Understanding this can make you much more intentional about prompt construction, knowing that every element contributes to the attention filters that ultimately shape the response.

The AI isn't learning from your prompt in the traditional sense of updating its stored knowledge. Instead, it's using your prompt to temporarily reconfigure how information flows through its networks—creating a custom routing pattern that's optimized for your specific request. It's a remarkable example of how intelligence might work more generally: not as fixed processing, but as dynamic filtering tuned by context.