|

What a small local AI model taught me about context

The big cloud AI models have made a lot of us lazy, me included.

When a model can take in a million tokens at once (a token is a chunk of text, a bit less than a word), it’s easy to get sloppy. You drop in a pile of uncleaned files, ask a vague question and let raw computing power sort it out.

This week I set up a local model, about 27 billion parameters, running on my own hardware. The difference showed up right away. My context window went from a million tokens to about 90,000, and responses came at a steady 30 to 50 tokens a second.

Both are amazing pieces of technology. The smaller model just won’t let you get away with bad habits. You can’t flood it with data it doesn’t need, and you can’t ask one prompt to do ten different things. You have to break the work into clear, small steps.

That lesson applies well beyond local models. If your AI approach depends on pouring messy data into an expensive frontier model, you’re paying a premium to cover for poor data hygiene. A clean, well-scoped workflow runs well whatever model sits underneath it.

What happens when you give a small model too much

Push a smaller model past its comfort zone and you can watch the quality slip. The more you pack into the context, the less attention the model gives any one part of it. It loses track of instructions, mixes up details, or fills the gaps with things that sound right and aren’t.

Developers working with frontier models often treat the context window like a bin with no bottom. Raw web page code, full database exports and system prompts that run for pages all go into one request. On local hardware that habit breaks fast, and it pushed me back to three practices that make any AI setup better:

  • Send only what the step needs. Every extra token uses memory and slows the model down. Strip out formatting clutter, keep data structures tight, and give the model only the context for the step in front of it.
  • Break the work into a chain of small steps. A prompt like “read this report, pull the key numbers, check them against the CRM and write an email” becomes four steps, and each one passes a clean result to the next.
  • Be very specific in the instructions. Smaller models need clear limits on what not to do and an exact format for the output. Vague instructions lead to made-up answers almost right away.

If your setup only works when it’s plugged into a million-token cloud model, it’s fragile. Building with the discipline a small model needs makes the workflow sturdier on any model.

How to give AI the right context

An operations leader told me, “I gave the AI all our background documents, but the output was a complete mess.” They’d uploaded a 50-page PDF, a spreadsheet and ten email threads, then asked it to draft a proposal. When the draft came back generic and partly wrong, they figured the AI wasn’t very smart.

More likely, it was overwhelmed. Imagine handing an assistant 500 unsorted pages five minutes before a meeting and asking for the important parts. They’d skim and miss the details that matter. Running a small model this week reminded me that good delegation works the same way for AI and for people:

  1. Split the project into phases. Don’t put ten tasks in one sentence. For a proposal, that might be three passes: summarize what the funder wants, outline your response, then draft each section.
  2. Hand over only the pages that matter. Give it the two pages it needs for today’s step and leave the rest of the file cabinet closed. Pull the relevant section of a long PDF instead of uploading all 50 pages.
  3. Clean it up before you paste it in. Cut email signatures, old reply chains and formatting junk. Less noise means more attention on what’s left.
  4. Say exactly what you want back. Name the length, the format and the audience, and say what to leave out.
  5. Check each phase before the next one. Read the output of each step before you feed it into the next, so a mistake early on doesn’t carry through.

For leaders: match the model to the job

Why should a nonprofit care about small local models when the giant cloud ones exist? Mostly privacy and cost.

Using a top-tier cloud model for every basic task is like hiring a $500-an-hour senior partner to do your filing. It works, but it’s a waste of money. I’d set things up in two tiers:

  • Local models for routine and sensitive work. A smaller model running on your own hardware can strip personal details out of records, sort documents and pull out basic data, and none of it leaves your hands.
  • Cloud models for the hard thinking. Save the expensive frontier models for complex reasoning, high-stakes writing like a major grant narrative, and strategy work.

That keeps your sensitive data in house and saves frontier prices for the work that needs them. A few questions for your next planning conversation:

  • Which of our AI tasks are routine enough for a smaller model?
  • Where does donor and client data go when we use AI today?
  • Are we paying top prices for work that doesn’t need it?

Breaking work into small, clean steps is a big part of how I’m building Cosly, the AI staff team for small nonprofits.

Similar Posts