Site icon Kevin Waites Consulting Services

What a small local AI model taught me about context

The big cloud AI models have made a lot of us lazy, me included.

When a model can take in a million tokens at once (a token is a chunk of text, a bit less than a word), it’s easy to get sloppy. You drop in a pile of uncleaned files, ask a vague question and let raw computing power sort it out.

This week I set up a local model, about 27 billion parameters, running on my own hardware. The difference showed up right away. My context window went from a million tokens to about 90,000, and responses came at a steady 30 to 50 tokens a second.

Both are amazing pieces of technology. The smaller model just won’t let you get away with bad habits. You can’t flood it with data it doesn’t need, and you can’t ask one prompt to do ten different things. You have to break the work into clear, small steps.

That lesson applies well beyond local models. If your AI approach depends on pouring messy data into an expensive frontier model, you’re paying a premium to cover for poor data hygiene. A clean, well-scoped workflow runs well whatever model sits underneath it.

What happens when you give a small model too much

Push a smaller model past its comfort zone and you can watch the quality slip. The more you pack into the context, the less attention the model gives any one part of it. It loses track of instructions, mixes up details, or fills the gaps with things that sound right and aren’t.

Developers working with frontier models often treat the context window like a bin with no bottom. Raw web page code, full database exports and system prompts that run for pages all go into one request. On local hardware that habit breaks fast, and it pushed me back to three practices that make any AI setup better:

If your setup only works when it’s plugged into a million-token cloud model, it’s fragile. Building with the discipline a small model needs makes the workflow sturdier on any model.

How to give AI the right context

An operations leader told me, “I gave the AI all our background documents, but the output was a complete mess.” They’d uploaded a 50-page PDF, a spreadsheet and ten email threads, then asked it to draft a proposal. When the draft came back generic and partly wrong, they figured the AI wasn’t very smart.

More likely, it was overwhelmed. Imagine handing an assistant 500 unsorted pages five minutes before a meeting and asking for the important parts. They’d skim and miss the details that matter. Running a small model this week reminded me that good delegation works the same way for AI and for people:

  1. Split the project into phases. Don’t put ten tasks in one sentence. For a proposal, that might be three passes: summarize what the funder wants, outline your response, then draft each section.
  2. Hand over only the pages that matter. Give it the two pages it needs for today’s step and leave the rest of the file cabinet closed. Pull the relevant section of a long PDF instead of uploading all 50 pages.
  3. Clean it up before you paste it in. Cut email signatures, old reply chains and formatting junk. Less noise means more attention on what’s left.
  4. Say exactly what you want back. Name the length, the format and the audience, and say what to leave out.
  5. Check each phase before the next one. Read the output of each step before you feed it into the next, so a mistake early on doesn’t carry through.

For leaders: match the model to the job

Why should a nonprofit care about small local models when the giant cloud ones exist? Mostly privacy and cost.

Using a top-tier cloud model for every basic task is like hiring a $500-an-hour senior partner to do your filing. It works, but it’s a waste of money. I’d set things up in two tiers:

That keeps your sensitive data in house and saves frontier prices for the work that needs them. A few questions for your next planning conversation:

Breaking work into small, clean steps is a big part of how I’m building Cosly, the AI staff team for small nonprofits.

Exit mobile version