ai on Saurabh Kumar

EmbeddingGemma 2

Mon, 19 Oct 2026 12:00:00 +0000

EmbeddingGemma 2 puts text, code, images, video and audio (or a mix) into one 768-dimension vector, from a 740M open model under Apache 2.0. Image, video and audio tokens are spliced into the text sequence and run through one encoder, so a text query can find a video moment or a scanned PDF page without captioning or transcribing first.

It’s not the quality leader. Text is flat against v1; the gain is code (MTEB Code 68.8 to 78.7). Audio (49) and video (51) scores are modest. All numbers are Google’s own.
Every major vendor ships one-space multimodal embeddings now: Amazon Nova, Gemini Embedding 2, Voyage, Cohere, and the open Qwen3-VL-Embedding, which reports higher image and document scores but has no audio. Pick by constraint. EmbeddingGemma 2’s niche is open weights plus audio.
Matryoshka truncation isn’t free: its MMEB score falls from 59.0 at 768 dimensions to 45.7 at 128.
The interesting part is the system around the model. DigUp is local file search on it: one vector per photo, PDF page, video shot and 30 s of audio, each pointing back to its page or second, next to an FTS5 keyword index, because no embedding holds an invoice number exactly.
DigUp fuses the two by turning cosine into a per-query z-score (how far a match stands out for this query), gating on it, and adding a word-coverage bonus. It beat reciprocal-rank fusion on their evals (R@1 0.88 vs 0.73) and can honestly say “nothing found”. Small, author-run eval.
It also pins llama.cpp, records the model version in the index, and checks every upgrade against stored reference vectors. Vectors from two runtimes don’t mix, and nothing errors when they do.

How the Claude Code team does code review

Sat, 17 Oct 2026 12:00:00 +0000

How the Claude Code team does code review. Claude writes ~90% of Claude Code, so nobody reads every diff. They swapped “did a human read it?” for “which check, automated or human, should catch what could go wrong here?”, and optimise for the least human time per change, with revert and incident rates as the guardrail.

The bit I missed on first read: review noise costs more when the author is an agent. A person waves off a nit in seconds; an agent fixes every one, which is a new commit and another full review. One PR ran 50+ rounds overnight, and ~59% of later findings were on code already there at round one. Fix: cap at three rounds, re-review only the new diff, flag the class of bug, never repeat a finding.
The bot posts findings, never approvals. Human approvals (zero, one or two) are set by the change’s risk, not by who wrote it. Whoever prompted the agent counts as a reviewer, and an approval covers one commit, so an agent’s next push dismisses it.
Their reviewer’s eval is their own past incidents and reverts, run daily. Noise is measured by what authors did with each comment: fixed it, said it was wrong, intentional, or not worth it.
Versus Ankit Jain’s review layers I noted in April: this is the production version, with no spec-review layer and fast rollback under everything.
Read the numbers carefully: the share of PRs reverted fell 40% while PR volume doubled, so the count of reverts still rose (~1.2x). Also: the authors sell the reviewer.
Where I push back: “nobody needs a full mental model, ask Claude for an explainer of the slice you’re changing.” An explainer recovers what the code does, not why it was built that way, which is the comprehension debt Addy Osmani (a co-author here) wrote about. People should spend review on whether to ship it and on direction, not line-level correctness.
How I read it: had Claude explain it against my notes, then quiz me. The question I got wrong taught me the most.

'AI' is Not a Tool

Tue, 13 Oct 2026 12:00:00 +0000

Rich Hickey’s "‘AI’ is Not a Tool" vs Huahai Yang’s “AI Is a Tool. Using It Well Takes Skill.”. Hickey: a tool is something you wield, and wielding is where expertise comes from, so an LLM is automation you abdicate work to. Yang: a compiler also works unattended and is still a tool; the line is skilled use vs delegating without oversight. Arguing over the label is a dead end. What matters is which skill survives.

Hickey’s best line: you are “only a software update from everything you’ve ’learned’ being rendered obsolete.” Model quirks and prompt recipes expire. Verification, choosing context and domain judgment carry over to the next model.
Bainbridge’s Ironies of Automation said it in 1983: automate the work and the operator is left monitoring, which needs the skill they stopped practicing. My rule: do the work, then ask the model. Don’t ask the model instead.
Best reply to “LLMs are slot machines” in the HN thread: a Las Vegas algorithm is random but always correct; only the runtime varies. Randomness plus a verifier gets you accuracy. Without the verifier you’re just pulling the lever.

Rampart

Sun, 11 Oct 2026 12:00:00 +0000

Small on-device models are getting good enough to sit in front of the big ones. Rampart from the National Design Studio redacts PII in the browser before a prompt reaches any LLM: regexes with checksum or structural checks for cards (Luhn), SSNs, emails, URLs and IPs, plus a fine-tuned MiniLM for names, phones, account numbers, government IDs and street addresses.

14.7 MB Q4 ONNX (~15 MB with tokenizer), 98.4% private-term recall across seven Latin-script languages. p50 is 3.9 ms on WebGPU but 12.6 ms on the default WASM backend. OpenAI’s Privacy Filter is ~2.8 GB; Presidio scores 65% on the same test set.
guard.protect() swaps PII for placeholders like [GIVEN_NAME_1], and guard.reveal() fills them back into the reply. The LLM never sees the name; the user still does.
Default-deny: every detected label is redacted except city, state and ZIP, so the assistant keeps coarse location without the street line.
Know the gaps: names in non-Latin scripts recall ~14% (Hindi 6%, Arabic 5%), and it scores 86.4% on a hostile-input suite. The repo calls it harm reduction, not perfect protection.
Same size class, different job: Cactus Whistle is 16.9 MB speech-to-text in seven languages, 11 ms to first token on an M4 Pro CPU (Whisper base: 145 MB, 73 ms). It loads into the same engine as Needle 3, their tool-calling model, so one call takes audio in and returns set_lights(room="kitchen", on=false). Voice commands for home automation with no cloud round trip.
The pattern: a tiny specialist at the edge does one narrow job (redact, transcribe, route), and the big model only sees what it needs. Both sets of benchmarks are vendor-reported.

Sebastian Raschka's history of text classifiers

Sat, 10 Oct 2026 12:00:01 +0000

Sebastian Raschka’s history of text classifiers puts numbers on why Jev feels new. On IMDb, bag-of-words + logistic regression gets 89.9%, a fine-tuned ModernBERT about 95%, and Jev 96.5% zero-shot for $0.65 across 25k reviews. Under the hood it is likely still a pretrained transformer with a one-number head that scores each answer option. The hard part is broad training data and calibrated probabilities. What changes is the cost of starting: no labelling, training or hosting per task. Specialists still win on huge volume, the last point of accuracy, or data that can’t leave your network. Within a month OpenAI and Microsoft shipped the same yes/no, choice and score contract.

Microsoft eXecution Container (MXC)

Sat, 10 Oct 2026 12:00:00 +0000

Microsoft eXecution Container (MXC) is an embeddable SDK for running untrusted commands under OS-enforced policies, not a hosted sandbox service.

Defaults: Seatbelt on macOS, Bubblewrap on Linux, ProcessContainer on Windows. SDKs for Node, Rust, and .NET.
Restricts filesystem and network access. macOS hostname allowlists need a caller-managed proxy; not every backend is a VM.
Useful boundary: sandbox the command where it executes, not just the agent’s tool-composition script. Isolation limits damage; it does not make generated output trustworthy.

Pi Pocket's subagent implementation

Fri, 09 Oct 2026 12:00:02 +0000

Pi Pocket’s subagent implementation gives each worker a retained conversation and uses durable tasks to deliver messages and report answers. Request IDs and recorded answer IDs prevent duplicate delivery and reporting after restart. The idea worth stealing: make the handoff durable, not just the worker.

Arnav Gupta's pi-subagent-manager

Fri, 09 Oct 2026 12:00:01 +0000

Arnav Gupta’s pi-subagent-manager makes context inheritance explicit. Nested threads receive their parent’s snapshot; independent roots start without its history. Threads stay around for steering and resume. I want synchronous coordination by default, forks for continuity and fresh context for independent review.

Armin Ronacher's subagent extension

Fri, 09 Oct 2026 12:00:00 +0000

Armin Ronacher’s subagent extension runs one observable Pi child at a time in tmux. The parent waits for the result, while I can attach to inspect or intervene. Worth preserving when adding hierarchy: synchronous coordination must not mean invisible execution.

Wispr Flow

Thu, 08 Oct 2026 12:00:00 +0000

Back on Wispr Flow after six months on local speech-to-text (FluidVoice with Parakeet, plus Whisper). Local transcription is fine. The cleanup is what’s missing: fixing misheard jargon ate the time dictation saved. Flow’s correction pass, OS-level integration and phone keyboard win. It matters more now because models follow high-level intent well: I speak the intent plus constraints in one go and let the LLM structure it. Whisper mode works even when you speak softly. Want to try it? My referral link gets you a free month of Pro, and I get one once you’ve dictated 2,000 words.

Pi Durable

Wed, 07 Oct 2026 12:00:00 +0000

Pi Durable is Earendil’s experimental agent harness, shipped alongside Pi 1.0. TannerMidd/pi-pocket is a mobile-first, multiplayer web app for Pi built on it.

Every model call and tool call is a task that checkpoints to storage (SQLite, JSONL, or memory) before moving on. Kill the process and a new one opens the same file and resumes.
Recovery is per step: a cut-off model request is resent with the partial answer kept and marked aborted. A cut-off tool call reruns only if it is replay-safe; otherwise the model is told it was interrupted.
A requestId makes a submission exactly-once, so a client retrying after a crash gets the original back.
Forks see the parent transcript up to the fork point without copying it.
~15k lines total, small enough for an agent to read end to end.
pi-pocket: restart the server mid-run and the work continues. Several people can watch, steer, or queue messages on the same session.
npm start -- --access tailscale binds only to the tailnet address, so my phone gets it with no tunnel.
The shift: the session lives in a SQLite file, not a terminal process, so the phone is just another client. Could replace my Moshi + tmux setup.
Caveat: anyone who can steer runs shell commands as you.

pi-blackhole

Sat, 05 Sep 2026 12:00:00 +0000

Every coding agent (Claude Code, Codex, pi) compacts long sessions the same way: ask an LLM to prose-summarize, then summarize that summary, until load-bearing details erode (why a decision was made, which approaches got rejected). pi-blackhole swaps that for deterministic compaction: a zero-token compile() that extracts structured sections (goal, files touched, commits, preferences, brief transcript). No LLM, no drift, no cost. The compile is only lossy in the window, not on disk: blackhole also ships a recall tool (BM25 + regex over the raw session JSONL and the memory ledger) that re-fetches any dropped turn on demand, so compile + recall together is lossless where an LLM summary-of-a-summary destroys the original. It also adds observational memory: background observer/reflector workers append durable facts to per-session ledgers that survive every compaction, and /blackhole-export walks every past-session JSONL under a git-root into a ranked context file. The clever part is what export trusts. A fact’s rank is driven by cross-session recurrence, damped as log(1 + distinctSessions) with a burst penalty, not raw counts, so one chatty session can’t inflate it. Recurrence-as-reinforcement is basically Hebbian: facts that keep firing across sessions get stronger. Dedup is a 3-pass funnel, exact then Levenshtein then Sørensen-Dice token-set, with a Levenshtein floor so keyword overlap alone can’t merge two distinct facts. Bonus: the OM workers run on their own model and billing, so I point them at openai-codex/gpt-5.6-luna while the main thread stays on Claude.

Meat proxy

Tue, 04 Aug 2026 12:00:00 +0000

Someone finally named it: meat proxy. You paste Claude’s answer into Slack or a PR review without reading it, and you have made yourself the pipe between me and a model I could have prompted myself, with better context. Everyone already knows that annoyance. Nobody had the word for it. The word is the useful part, because “don’t be a meat proxy” is something you can actually say to a colleague: it names a role instead of calling them lazy. Best line is about code review. Paste the ticket into Claude, paste the reviewer feedback back in, read neither, and the reviewers implemented your PR while you held the pipe. The bar before forwarding anything is to write it in your own words, “a decent certificate that you’ve done the prior steps.” 704 comments on HN, barely any disagreement.

NVIDIA's NOOA framework

Tue, 28 Jul 2026 12:00:00 +0000

Python’s Ellipsis literal (...) is a real object (type(...) is <class 'ellipsis'>), not just a placeholder. Beyond type stubs and numpy slicing, NVIDIA’s NOOA framework repurposes it as a runtime dispatch marker: methods with ... bodies are completed by an LLM at runtime, while methods with real bodies stay deterministic Python. The boundary between “code I control” and “code the model controls” becomes greppable: grep -n '\.\.\.' agent.py. Clever reuse of existing syntax that linters and type checkers already accept.

marcelroed/gigatoken

Fri, 24 Jul 2026 12:00:00 +0000

Embedding pipelines do not feed whole documents straight into the model. They first use the embedding model’s tokenizer to measure and split text into token-bounded, overlapping chunks, then embed each chunk and store the vectors. Tokenizer must match the model. Tools like marcelroed/gigatoken accelerate this preparation at corpus scale but do not create embeddings themselves.

baidu/Unlimited-OCR

Fri, 26 Jun 2026 12:00:00 +0000

baidu/Unlimited-OCR (paper) parses dozens of pages in one forward pass by fixing the real bottleneck in end-to-end OCR.

The problem: an OCR decoder types out the page token-by-token, and its KV cache grows unbounded with output length — so memory climbs and speed decays the longer the doc, forcing page-by-page loops that wipe memory each step.
The trick (Reference Sliding Window Attention): split context into two zones with different retention. Vision/prompt tokens stay fully visible and pinned forever (the “source book”); each token attends to only the last 128 of its own outputs (the “last few words you wrote”). KV cache becomes a fixed-size queue, so memory and TPS stay flat regardless of output length.
The non-obvious part: discarding history improves accuracy (+6% on OmniDocBench, beating Qwen2.5-VL-72B at 3B/0.5B-active) — full attention can diverge on long dense output, and pinned vision tokens never blur.
Cheap to build: freeze DeepSeek-OCR’s encoder, fine-tune only the decoder with the swapped attention.
Generalizes past OCR: separate what you reference from what you remember.
Ran it on an M1 Mac (32GB) with R-SWA intact, no NVIDIA. The official HF model (trust_remote_code) already ships R-SWA as a pure-PyTorch eager ring buffer (SlidingWindowLlamaAttention, picked via attn_implementation="eager") — not buried in a CUDA/FlashAttention kernel. So the “port” was just stripping .cuda()/torch.autocast("cuda") and running bf16 on MPS; zero MPS op-gaps hit. bf16 works on MPS in torch 2.12.
Proof R-SWA is live: instrument the KV cache during decode — on a 540-token page it plateaus at prefill(277) + window(128) = 405 and never grows (full MHA would hit 817). That constant-cache assertion + Distinct-35≈1.0 are the decisive signals, not edit distance (too noisy vs a single reference).
Gotcha: feeding all pages to one generate() (infer_multi) loops at the document level — the 128-token decode window forgets the doc was already transcribed while the vision prefix still shows every page, so it restarts from page 1. Fix = one generate() per page (matches official infer.py PAGE_SIZE=1). Accuracy then lands at/below gemini-vs-docai inter-annotator noise (~0.31 mean edit). ~35–55 s/page on M1.

Many asymmetric embedding models need task prefixes on inputs, and skipping…

Tue, 23 Jun 2026 12:00:00 +0000

Many asymmetric embedding models need task prefixes on inputs, and skipping them quietly degrades relevance. Each model has its own scheme: nomic (search_query:/search_document:), E5 (query:/passage:), BGE (a query instruction sentence, bare docs) — not interchangeable. OpenAI text-embedding-3-* and all-MiniLM-L6-v2 need none. Whether you add the prefix depends on the serving layer, not the model: raw endpoints (llama.cpp /v1/embeddings, HF TEI, Ollama) send bare text so it’s on you, while sentence-transformers (prompt_name="query") and vendor SDKs inject it for you. Adding search_query:/search_document: to a nomic-v1.5 call lifted cosine similarity on a real query/doc pair from 0.54 to 0.60 at zero cost.

How Prompt Caching Works

Sun, 21 Jun 2026 00:00:00 +0000

Today I learned how prompt caching actually works under the hood in LLM APIs (Anthropic, OpenAI, Bedrock, etc.), and why it makes long agent loops 5–10× cheaper and noticeably faster.

The problem it solves

In an agent / chat loop, every request to the model looks almost identical to the previous one:

The same big system prompt (rules, tool descriptions, skills…)
The same set of tool definitions
The same history of past user/assistant/tool messages, with only a few new tokens appended at the end

Without caching, the provider has to re-tokenize and run a full attention pre-fill over all of that prefix on every turn. For a 100k-token context, that prefill dominates both cost and latency.

What gets cached: the KV state

When a transformer processes a prompt, it computes a key/value tensor pair for every token in every attention layer. That KV state is what attention queries against on the next decoded token.

Prompt caching is, mechanically, “save those KV tensors for a prefix on the server, and reuse them next time the same prefix shows up.” Output tokens are never cached — only the prompt-side KV state.

The lifecycle

Mark a cache breakpoint. The client places a cache_control marker (Anthropic) or relies on automatic prefix detection (OpenAI). Everything before the marker is the cacheable prefix.
First request → cache write. The provider hashes the cacheable prefix (model id + tools + messages + cache key, byte-exact), runs the prefill, and stores the resulting KV tensors keyed by that hash. You’re billed cache_write tokens, usually a bit more than normal input (~1.25× for Anthropic 5-minute cache, ~2× for the 1-hour “extended” cache).
Next request → cache read. If the new request starts with exactly the same prefix, the provider looks up the hash, loads the stored KV state, and only runs fresh compute for the suffix after the last hit. You’re billed cache_read tokens — often ~10% of the normal input price.
TTL. Entries expire after a TTL (Anthropic: 5 min default, 1 hour opt-in; OpenAI: a few minutes, automatic). Each hit typically refreshes the TTL, so an active session keeps the cache warm.

What invalidates the cache

The hash is over the exact bytes of the prefix. Anything that changes those bytes forces a fresh cache_write:

Editing the system prompt (even reordering sentences or adjusting whitespace)
Adding, removing, or reordering a tool definition
Inserting/editing/reordering an earlier message in the history
Switching model id or provider
Changing the cache key (some providers expose one for tenant isolation)

The corollary: to maximize hit rate, agents should only append at the end and keep system prompt and tools byte-stable across turns.

Reading the usage block

Most providers return a usage object on every response with at least these fields:

{
 "input": 312, // fresh, non-cached prompt tokens this turn
 "output": 845, // generated tokens
 "cacheRead": 47210, // prompt tokens served from cache
 "cacheWrite": 0 // prompt tokens written to cache this turn
}

A useful per-turn metric is the cache hit ratio:

cacheHitRate = cacheRead / (input + cacheRead + cacheWrite)

0% → nothing reused. Normal on the very first turn, or right after the prefix changed (tool list updated, history compacted, model swapped).
70–95% → healthy steady state for a long agent loop.
cacheWrite > 0, cacheRead = 0 → you just primed the cache; the next turn should be cheap if you don’t break the prefix.

Pricing back-of-the-envelope

Rough Anthropic numbers (Claude Sonnet, illustrative):

Token type	Price multiplier vs. input
`input`	1×
`cache_write`	~1.25× (5 min) / ~2× (1h)
`cache_read`	~0.1×
`output`	~5×

So a 100k-token prefix that’s hit 20 times across a session costs roughly:

1 × 1.25 × 100k (one write)
+ 19 × 0.1 × 100k (nineteen reads)
= 1.25 × 100k + 1.9 × 100k
≈ 3.15 × 100k input-equivalent

vs. uncached:
20 × 1 × 100k = 20 × 100k

A ~6× saving on the prompt side, plus a big latency win because the server skips the prefill.

Gotchas

Minimum prefix size. Most providers require the cacheable prefix to be above some threshold (e.g. 1024 tokens for Anthropic) before caching kicks in at all.
Position of the breakpoint matters. Putting it after volatile content (like the latest user message) means nothing ever gets reused.
Tool definitions are part of the prefix. Dynamically generated tool lists with shuffled order will silently destroy your hit rate.
Streaming vs. non-streaming use the same cache; cache fields show up in the final usage event.
Cost accounting: cache_write is more expensive than input, so a flaky prefix that constantly re-writes can be worse than no caching at all.

Moshi

Mon, 15 Jun 2026 12:00:00 +0000

Moshi is the best SSH client I’ve found for iPhone — connects to your laptop’s tmux sessions over Mosh, making it rock-solid on mobile networks with roaming support. Perfect for an agent view on the go; I use it to monitor and interact with running pi/Claude sessions from my iPhone without losing the session on network switches.

My Local LLM On A 32 GB M1 Pro

Sun, 14 Jun 2026 00:00:00 +0000

My pi setup has a fast mode: a slot I wired into my own config for the small, high-volume questions, pointed at Bedrock’s Haiku 4.5. It is a good model. Cheap, responsive, and it answers the kind of small, contained question I throw at it twenty times a day without complaining. It runs on third-party hardware, and those questions send code to a third party. I wanted to try running that slot locally.

None of that is a crisis. Bedrock is fast and it works. But the machine that could do the same job is sitting on my desk, and when Kyle Howells got Gemma 4 26B-A4B running locally on an M1 Max at 72 tokens per second and the Hacker News thread filled up with people on smaller Macs reporting the same setup was workable, I wanted to know whether my own laptop could host that slot.

The catch is that my laptop is an M1 Pro with 32 GB of unified memory: half the RAM and roughly half the memory bandwidth of the machine in Kyle’s post. Copying his commands verbatim was never going to work. I needed to know what to change, in what order, and what to give up. So I did not copy the commands. I gave pi the article and asked it to build me a plan for my machine, and to interrogate me before it wrote a line.

I Handed The Guide To An LLM And Made It Grill Me

The whole thing started with one message. Typos and all, this is what I actually sent:

Act local AI setup expert working on M1 silicon with 32GB ram. Read though [the article] and also pull all the comments on this article on news.ycombinator.com article id 48507020. I’ve ollama and muse already installed there are models installed as well. Prep and action plan to set this up in the most performant and maintaibale way. Not sure quite sure if i need to uninstall previous ollama app and it’s modetls. Create and write a plan after analysing things. /grill-me

The /grill-me at the end is a small skill in my pi setup. It is two paragraphs of instructions that tell the model to walk down the decision tree one node at a time, recommend an answer at each step, and wait for my reply before moving on. It is not magic. It is a way of forcing the conversation I would otherwise have with myself in a notebook, except the model will not let me skip a step.

The interview was mostly the model proposing and me confirming in monosyllables. My replies, verbatim, were things like:

A

i would prefer ~/code/localLLM

A, also note i’ve “llm” cli already installed, and alias would be nice.

let’s go with recommendation, write the plan.

That is the entire texture of it. A simple opening prompt that points at an existing guide and names my constraints, then a handful of one-word answers steering the branches. The model did the reading (the article and the full HN thread) and the arithmetic. I made the calls.

And it pushed back, which is the point. When I leaned toward the 65 K context window from the article, it did the memory math and showed me I would be swapping inside the first generation on 32 GB, so we settled on 32 K. When I waffled on keeping Ollama “just in case,” it pointed out I was at 89 % disk with a thirteen-gigabyte Ollama models directory I was not using. When I asked for an alias, it caught that llm was already taken by Simon Willison’s tool and proposed lcl instead. None of these are insights I lacked. They are checks I would have skipped because I was already mentally on the next step.

By the end there was a written record of every decision and its reason, in the repo, not in my head. That is where it should be.

The Decisions That Fell Out

The interview converged on a short list, and the reasoning behind each one is worth more than the choice itself.

Model: Gemma 4 26B-A4B, the MoE, not a dense model. The single most important fact about local inference on a Mac is not how much RAM you have, it is how fast it can be read. Generation is bottlenecked on memory bandwidth. The M1 Max in Kyle’s article runs at 400 GB/s; my M1 Pro at roughly 200. Everything else being equal, I should expect about half his throughput. His 72 tokens per second becomes my 36. That is still faster than I read. It is also why a mixture-of-experts model is the right answer here: Gemma 4 26B-A4B holds 26 billion parameters but activates only four billion per token. The full weights sit in RAM, but each forward pass streams a fraction of them. The dense 27B sibling has the same disk footprint and meaningfully worse generation rate. It is probably smarter. It does not matter, because it is too slow for the slot I am filling.

Speculative decoding, on. Gemma 4 ships an MTP draft head, a tiny network that predicts the next several tokens cheaply, which the main model accepts or rejects in one pass. When acceptance is high, you get most of those tokens almost for free. Kyle measured +24 % on his machine. The HN thread is full of caveats about acceptance rates and MoE models benefiting less. None of it changes the decision: turn it on, measure, keep what you get.

Context: 32 K, not 65 K. The arithmetic the model made me do. A 65 K KV cache plus 16 GB of weights plus a draft model does not leave enough headroom on 32 GB once Chrome and an IDE are open.

mise, not a shell script. The article uses huggingface-cli to fetch the GGUFs and a hand-written tmux script to launch the server. Both work. Both are also infrastructure you now own and will forget you configured. llama.cpp’s -hf flag does the download without a Python virtualenv. mise’s github: backend installs the prebuilt Metal binary and rolls it forward with mise upgrade. The tmux script becomes a few named tasks in a .mise.toml. The choice is not about elegance, it is about which moving piece you are willing to own. I already own mise.

Ollama goes. I had it from earlier experiments, with gemma4:latest and a couple of other models taking 13 GB, plus the app idling in the background on port 11434. Ollama is a wrapper around llama.cpp. Once you run llama-server directly, the wrapper is in the way. So the app, the models directory, and the port all go, and the 13 GB comes back on a disk that was at 89 %.

The Setup, Start To Finish

This is the reproducible part. You can follow it by hand, or point an LLM at this section and let it do the work. That is roughly how it got built in the first place. The numbers below are the actual results on my M1 Pro, not estimates.

1. Decommission Ollama (skip if you never had it):

osascript -e 'tell application "Ollama" to quit' 2>/dev/null
pkill -f Ollama
rm -rf ~/.ollama /Applications/Ollama.app # reclaimed 13 GB

2. Install llama.cpp via mise. The prebuilt arm64 binary has Metal and Accelerate baked in:

mise use --global 'github:ggml-org/llama.cpp@latest'
llama-server --help | grep -E -- '--spec-type|-hf' # confirm MTP + HF download support

The one thing worth verifying is that --spec-type draft-mtp is present, since MTP support landed only recently. On my install (build b9631) it was there. If it ever is not, a source build with -DGGML_METAL=ON is the five-minute fallback.

3. A project with mise tasks at ~/code/localLLM/.mise.toml. The [env] block makes the model cache live with the project; the tasks replace the shell script:

[env]
LLAMA_CACHE = "{{ config_root }}/models"

[tools]
"github:ggml-org/llama.cpp" = "latest"

[tasks.start]
description = "Start llama-server (Gemma 4 26B-A4B + MTP) in tmux"
run = """
tmux has-session -t llama 2>/dev/null && { echo running; exit 0; }
tmux new-session -d -s llama -c "$PWD" \\
 "llama-server \\
 -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \\
 -hfd unsloth/gemma-4-26B-A4B-it-GGUF:Q8_0-MTP \\
 --spec-type draft-mtp --spec-draft-n-max 3 \\
 -ngl 999 -fa on -c 32768 --parallel 1 \\
 --reasoning off --reasoning-budget 0 \\
 --temp 0.7 --repeat-penalty 1.1 --repeat-last-n 256 \\
 --host 127.0.0.1 --port 8082 \\
 2>&1 | tee -a logs/llama-server.log"
"""

[tasks.stop]
run = "tmux kill-session -t llama 2>/dev/null && echo stopped || echo 'not running'"

[tasks.status]
run = "curl -fsS http://127.0.0.1:8082/v1/models >/dev/null && echo up || echo down"

A note on the port. The article uses 8080. On my machine 8080 and 8081 were already taken by Google cloud-run processes, so the server silently answered on a port that was not mine until I noticed the responses carried a Server: Google Frontend header. I moved everything to 8082. Check lsof -i :8080 before you assume a port is free.

4. A one-key alias in ~/.zshrc. llm was taken, so lcl for “local”:

lcl() { mise -C "$HOME/code/localLLM" run "$@"; }

lcl start # boot the server (first run downloads the model)
lcl status # is it up?
lcl stop # free the ~22 GB of RAM

5. First boot and download. lcl start triggers the download through llama.cpp’s -hf. The real footprint came in smaller than the article’s “~21 GB” estimate. The total was 18 GB: 16 GB for the Q4_K_XL main model, 1.2 GB for the MTP head (an auxiliary head, not a full second model), and a 441 MB multimodal projector that gets pulled automatically even though I do not need images. Add --no-mmproj if you want to skip it.

6. Point pi at it. A local provider in ~/.pi/agent/models.json:

{
 "providers": {
 "local": {
 "name": "Local llama.cpp",
 "baseUrl": "http://127.0.0.1:8082/v1",
 "api": "openai-completions",
 "apiKey": "local",
 "models": [{
 "id": "gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf",
 "name": "Gemma 4 26B-A4B Q4 + MTP",
 "reasoning": false,
 "input": ["text"],
 "contextWindow": 32768,
 "maxTokens": 8192,
 "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
 }]
 }
 }
}

And a local mode in modes.json pointing at it. The benchmark answers came back clean, so I gave it real work, and that is where it broke.

The Thinking Spiral

The first real task I handed Gemma was trivial: rewrite a Hugo template so a list of tags renders as “tagged A, B and C” instead of “A B C”. A ten-line edit. Haiku does it in one tool call. Gemma read the file, started thinking, and never stopped. The session export shows a single reasoning block of 32,000 characters: it derived the correct off-by-one logic, then re-derived it, then second-guessed the whitespace, then re-read its own conclusion and started over. It never emitted the edit. I aborted it. I told it “you are going in loop.” It thought about that, too, and aborted again.

This is the failure mode the chat-window benchmarks hide. Gemma 4 IT ships with thinking on, and llama.cpp’s default reasoning budget is unlimited (-1). In a chat that is fine; the model thinks, then answers. Inside an agent loop, where the next move is supposed to be a tool call, an unbounded thinking budget on a small model is a trap. There is nothing forcing the transition from reasoning to acting, and a 4B-active MoE is not big enough to find the exit on its own. The default sampling makes it worse: --repeat-penalty defaults to 1.0, which is to say off, so nothing discourages the model from looping on its own reasoning.

My first instinct was the biggest hammer: turn thinking off entirely. It worked. But “it worked” is where most debugging stops and most of the understanding gets lost, so I made the model prove which flag was actually doing the work. I had pi reconstruct the exact failing scenario from the session export, same system prompt, same tools, same file, same prompt, and replay it against two server configs:

--reasoning off --reasoning-budget 0 # A: no thinking
--reasoning auto --reasoning-budget 1024 --repeat-penalty 1.1 # B: bounded thinking

Same prompt, same machine, here is what came back:

Config	Reasoning emitted	Outcome
Original (unbounded, no repeat penalty)	32,000 chars	spiral, aborted, no edit
A — reasoning off	0 chars	clean `read` → `edit`
B — bounded to 1024	518 chars	clean `read` → `edit`

The lesson is in the third row. Capping the budget alone stopped the spiral: the model thought for 518 characters, decided, and acted. So the bug was never “thinking is bad.” It was “unbounded thinking is bad,” and --reasoning off is just the most aggressive way to bound it to zero. Both configs produced the same edit, the same slightly-clumsy Hugo logic. The extra reasoning in config B bought nothing on this task; that is a 4B-active model hitting its ceiling, not a budget I can tune my way around.

Two more things I only learned by testing. --repeat-penalty is an orthogonal fix and worth keeping regardless: at 1.0 it is off, and it guards the content against loops, not just the thinking. And --reasoning off does not lobotomise the model the way the name suggests. I gave it a reasoning trap with thinking off, and it still worked the problem out, step by step, in its visible answer, and got it right. What the flag removes is the separate, hidden, budget-less thinking phase, not the model’s ability to reason. For an agent loop that is the right trade: when reasoning happens it is visible in the content and bounded by the task, instead of a hidden phase that can run away.

So for the fast slot I keep it off. The whole point of the slot is a quick, direct tool call; the test showed bounded thinking adds latency with no quality gain here, and if I want real reasoning I switch to a cloud mode. That is also why "reasoning" is false in the model config above, with thinking disabled at the server there is nothing for pi to render. The middle ground is real and now proven, though: if I ever stand up a local-deep mode for harder offline work, --reasoning auto --reasoning-budget 1024 is a configuration I have watched stay out of the ditch. It just wants its own server on its own port, because --reasoning off is a hard gate you cannot toggle back on per request.

What it actually does, measured on the M1 Pro:

Metric	Result
Generation	34–36 tok/s (predicted half of Kyle’s 72, got it)
Prompt ingestion	77 tok/s
Time to first token	~376 ms
MTP acceptance	~65 % (e.g. 1350 of 2090 drafted tokens accepted)

Half the bandwidth, half the tokens, just as predicted. The MTP acceptance beat what I feared. The article saw +24 % on a dense workload; here, two thirds of the drafted tokens landed.

How I Actually Use A Fast Model

A fast model in pi is not the thing I have a conversation with. It is the model that does the small, high-volume, latency-sensitive jobs around the edges of the real work, where Opus is overkill and the round-trip cost is what matters. Those are the jobs I would rather keep on my own machine, and the place where 36 tokens per second is plenty.

In my setup, that means:

/answer and Ctrl+. — a small pi extension I run that grabs the last assistant message when it ends on a batch of clarifying questions, extracts them into structured JSON via a side LLM call, and opens a TUI so I can answer them one at a time. The extraction is a contained, schema-shaped task, the sort of thing a fast model handles well. It defaults to claude-bridge/claude-haiku-4-5 and is configurable at ~/.pi/agent/extensions/answer.config.json, the line I now point at local.
Branch summaries and compaction. When I navigate away from a branch on /tree, or a session outgrows its context window, pi summarizes the work to carry forward. By default those calls run on the active session model, which means a /tree hop can cost an Opus summary. A small extension I run reroutes both to whatever the fast mode points at, so the throwaway housekeeping never touches the expensive model.
/mode fast for the genuinely small questions, the “what’s the flag for X” and “rename this” and quick grep-and-explain ones, where I switch the whole session to the fast slot and switch back.

None of these need a frontier model. They need something that responds now and gets the structure right. That is the bar Gemma 4 has to clear to take the slot from Haiku.

So I gave it the slot. I went back and forth here. The cautious move was to add a separate local mode, keep fast on Haiku, and switch into the local one when I felt like testing it. I know how that goes: I would never remember to switch, and a week later I would have no data. So fast now points at Gemma 4 and the Haiku mode moved to lcl, there if I want to compare. Local models can be brilliant in a chat window and brittle inside an agent loop, as the thinking spiral already showed, and the only way I learn which one Gemma 4 is here is by living with it in the slot that fires twenty times a day. If it holds up, on tool calling, on diff quality, on the ten-line edits that fill my day, it stays. If it does not, the rollback is one line in modes.json.

What I Am Not Doing Yet

I am not putting the server behind launchd. At ~22 GB resident on a 32 GB machine, leaving it running alongside Chrome and an IDE is asking for swap. On-demand lcl start / lcl stop is honest about the cost in a way always-on is not.

I am not running a second model on a second port. Maybe Qwen 3.6 35B-A3B earns a slot for the cases where Gemma is too weak. Maybe one model is enough. I will know in a week.

What the Planning Helped With

The model researched the options; I chose the setup. The conversation made me write down why I chose each part, so I can refer back to it later.

I still need to see how the local model handles everyday work. If it does not hold up, I can switch the slot back.

fabric

Mon, 08 Jun 2026 12:00:01 +0000

faberic is a crowd-sourced collection of prompts bundeled with a cli to use them via piped interfaces.

karpathy/autoresearch

Mon, 08 Jun 2026 12:00:00 +0000

[karpathy/autoresearch] is a concept/framework that allows one to leverage LLM to run an optimization loops based on a criteria. davebcn87/pi-autoresearch takes this further has built a generic optimization long run loop method on top of pi agent. gemini

Dynamic workflows

Fri, 29 May 2026 12:00:00 +0000

Dynamic workflows in Claude Code. Claude dynamically writes orchestration scripts that spawn tens to hundreds of parallel subagents. It plans, decomposes into subtasks, fans out agents, and checks results before returning a coordinated answer. Adversarial agents try to refute findings, iterating until convergence. Enable via ultracode setting or ask Claude to “create a workflow”. Notable: Jarred Sumner used it to port Bun from Zig to Rust (~750K lines, 11 days). Available on Max, Team, Enterprise plans. Consumes significantly more tokens than typical sessions.

Claude Code: The Unreasonable Effectiveness of HTML" / X

Sat, 09 May 2026 12:00:01 +0000

Use HTML instead of markdown to effectively plan and review. Claude Code: The Unreasonable Effectiveness of HTML" / X

colbymchenry/codegraph

Sat, 09 May 2026 12:00:00 +0000

colbymchenry/codegraph: Pre-indexed code knowledge graph for Claude Code — fewer tokens, fewer tool calls, 100% local

Ankit Jain on automated AI code review

Wed, 22 Apr 2026 12:00:00 +0000

A good collection of practices on automated AI code reviews by Ankit Jain

The Scalability Crisis: Manual post-PR review is no longer viable. AI agents have nearly doubled code output, causing human review time to spike by 91%, creating a bottleneck that traditional workflows cannot solve.
The Upstream Pivot: Human value must shift from reviewing implementation to defining intent. Instead of checking syntax, humans spend their energy writing rigorous specs and acceptance criteria before the code is written, which the machine then uses to self-verify.
The Swiss-Cheese Defense: Rather than one “perfect” human gate, the model uses a stack of imperfect automated layers. By layering signals like agent competition, deterministic guardrails, and adversarial “red-team” agents, the system catches errors where their individual failure modes don’t overlap.

pageindex

Tue, 24 Mar 2026 12:00:00 +0000

pageindex generates a semantic tree-like json index of a lengthy document to allow for reasoning based RAG without the need for vectordb.

For generating embedding locally

Sun, 15 Feb 2026 12:00:03 +0000

For generating embedding locally, nomic-embed-text is a large context length text encoder that surpasses OpenAI text-embedding-ada-002 and text-embedding-3-small performance on short and long context tasks. It has a balance of speed, 8k context, and accuracy for English-centric apps. BGE-M3, Qwen3-Embedding and E5-Small are other alternatives.

yichuan-w/LEANN

Sun, 15 Feb 2026 12:00:01 +0000

yichuan-w/LEANN is a RAG focused framework focused on efficient storage with built-in chunking strategies embedding model management and MCP server. gemini

K-dense

Sun, 15 Feb 2026 12:00:00 +0000

K-dense known for using skills to enable deep research has published 140+ skills related to scientic research including literature review, data analysis, etc.

Opus 4.6 launch

Fri, 06 Feb 2026 12:00:00 +0000

Opus 4.6 launch.

context compaction (beta) and 1M context window, enables longer agentic tasks without loosing context.
they claim it has found 500 Zero-Day Flaws in open-source projects (yet to see the proofs though)
agent teams, multiple agent coordinates with a leader agent. https://code.claude.com/docs/en/agent-teams

How AI assistance impacts the formation of coding skills

Sat, 31 Jan 2026 12:00:00 +0000

Notes on “How AI assistance impacts the formation of coding skills” Article HN

AI speeds up coding but reduces deep understanding and mastery
Juniors (1-3 years experience) showed speed improvements with AI, but 4+ year developers showed no difference
Modern software work is more about requirements, specs, documentation, and communication than raw coding skill
Small sample size (n<8) and study design limitations make results questionable
Takeaways:
1. Use AI for high-scoring interaction patterns: Ask conceptual questions and request explanations rather than just code generation
2. Adopt AI for documentation and specs: Multiple developers report dramatic improvements in tickets, PRs, and documentation quality
3. Be deliberate about learning: If using AI, actively practice explaining concepts and avoid pure copy-paste workflows
4. Use AI to reduce grunt work: Let it handle boilerplate, test writing, and repetitive tasks while focusing on architecture and requirements
The research confirms what many suspected: AI coding assistants create a real trade-off between speed and skill development, but the practical significance is hotly contested. The critical question isn’t whether AI reduces learning (it does), but whether deep coding skill remains as valuable as expressing requirements clearly—and whether we’re comfortable with a generation of developers who can’t function without AI assistance.

LangGraph

Wed, 24 Dec 2025 12:00:00 +0000

“LangGraph is an orchestration framework for building stateful multi-agent applications using LLMs. It provides low-level primitives such as nodes and edges, along with built-in features that give developers granular control over agent workflows, memory management and state persistence. This means developers can start with a simple pre-built graph and scale to complex, evolving agent architectures. With support for streaming, advanced context management and resilience patterns like model fallbacks and tool error handling, LangGraph enables you to build robust, production-grade agentic applications. Its graph-based approach ensures predictable, customizable workflows and simplifies debugging and scaling.”

Thoughtworks - Technology Radar vol 33

Mon, 22 Dec 2025 12:00:02 +0000

Notes from Thoughtworks - Technology Radar vol 33

text-to-sql solutions aren’t working as expected
pnpm, langGraph, and pydantic recommended for adoption

Notes on Understanding Hebbian Learning

Wed, 05 Nov 2025 00:00:00 +0000

Hebbian learning describes how correlated neural activity can strengthen connections. It is named after psychologist Donald Hebb, who proposed it in 1949.

The entire concept boils down to one beautiful phrase:

“Neurons that fire together, wire together.”

That is a useful shorthand, not a complete account of learning. Timing and the synapses involved matter.

Breaking It Down

Repeated, correlated activity can strengthen the connection between two neurons. The effect depends on timing and the synapse, rather than simultaneous activity always producing the same change.

Think of it like this:

First time: “Oh, these two things happened together”
Fifth time: “Okay, these definitely seem related”
Hundredth time: “These are basically the same signal now”

Real Examples That Make It Click

Learning to Type

Remember learning to type? At first, finding the ‘Q’ key required conscious thought. Your brain neurons for “want to type Q” and “move left pinky to top left” fired together… over and over. Now? It’s automatic. Those neurons are strongly wired.

Pavlov’s Dogs (The Classic)

Bell rings → food appears → dogs salivate

Do this enough times, and the neurons for “bell sound” and “food is coming” become so tightly linked that just the bell makes dogs salivate. No food needed.

Your Morning Coffee Ritual

Why does the smell of coffee make you feel more awake even before you drink it? Neurons fired together thousands of times:

Smell of coffee + feeling of alertness
Sound of coffee maker + anticipation of energy
Taste of first sip + actual caffeine effect

Now they’re all wired together into one mega-circuit.

The Actual Mechanism (Simplified)

What’s physically happening:

Neuron A fires
Neuron B fires (around the same time)
The synapse (connection) between them releases more receptors
The connection physically strengthens
Next time A fires, it’s easier to activate B

It’s literally changing the structure of your brain. Learning is physical.

Why This Matters

For Understanding Yourself

Habits: They form because neurons wire together through repetition
Memory: Retrieving memories strengthens those neural pathways
Skills: Practice literally builds stronger brain connections
Trauma: Unfortunately, negative associations work the same way

For AI and Machine Learning

Hebbian learning inspired artificial neural networks. Modern AI doesn’t use pure Hebbian learning anymore (it’s evolved into more sophisticated algorithms), but it started here.

The basic idea - connections that prove useful should get stronger - is still foundational.

The Flip Side: “Use It or Lose It”

Here’s the thing nobody tells you: it works in reverse too.

Neurons that STOP firing together will gradually UNWIRE.

This is why:

You forget languages you don’t practice
Old skills get rusty
Memories fade
Bad habits can actually be broken

The Math (Optional)

For the nerds: The basic Hebbian rule can be expressed as:

Δw = η × x × y

Where:

Δw = change in connection strength
η = learning rate
x = activity of first neuron
y = activity of second neuron

If both are active (positive values), the connection strengthens. Simple, elegant, powerful.

Practical Takeaways

For Learning Anything:

Repetition isn’t just helpful - it’s literally how learning works
Practice needs to be consistent (neurons need to fire together repeatedly)
Spacing matters (distributed practice > cramming)
Context matters (learning in similar contexts strengthens related neural patterns)

For Breaking Bad Habits:

You need to stop the neurons from firing together
Replace the pattern with a new one (create competing neural pathways)
It takes time - you’re literally rewiring your brain

For Building Skills:

Deliberate practice works because it forces specific neurons to fire together
Feedback loops help strengthen the right connections
Muscle memory is real - it’s Hebbian learning in motor cortex

Cool Extensions of This Idea

Spike-timing-dependent plasticity (STDP): Order matters too. In this simplified account, neuron A firing before neuron B (within ~20ms) can strengthen the connection, while the reverse order can weaken it. The timing and effect depend on the synapse.

Competitive learning: When neurons wire together, they can also suppress other connections. This is how your brain specializes - certain neuron clusters become experts at specific patterns.

Associative memory: Hebbian associations offer a way to think about connections between faces, names, and songs. They are not a complete explanation of memory.

If you want to dive deeper:

Donald Hebb’s original work: “The Organization of Behavior” (1949)
Modern neuroscience textbooks on synaptic plasticity
Research on long-term potentiation (LTP) - the molecular mechanism behind Hebbian learning
AI/ML papers on Hebbian learning algorithms