DfTDLLY lab

High-risk topical context lab

Don't feed the trolls.

Trollino posts the bait. Doolally stamps the ban. Two small local models read the same short texts, and SQLite keeps the score even when Trollino pulls the plug.

This is an educational lab about the real cost of AI infrastructure: accuracy, latency, memory and recovery. It is not a production moderation system.

  • 10,000curated texts
  • 4high-risk classes
  • 2local models
  • 0duplicates after a kill
Doolally the moderator in a safety vest holds a BAN sign while Trollino the author, with rainbow hair, types on a stone laptop with BAIT attached.
Two trolls, one queue, and a very durable piece of duct tape.

The two trolls

The characters are the mental model. The models are the actual comparison.

Doolally the moderator
Moderator

Doolally

The 8B LFM model. Reads every word, thinks briefly, then picks a label. Slower, but careful. In the real run it landed within five points of the small model.

Trollino the author
Author

Trollino

The 3B d1 decision model. One forward pass, no generated tokens, no second thoughts. In the real run it matched the 8B accuracy at about 22x the speed.

How the trick works

Bait, ban, commit, recover. The order matters.

  1. Stratified bait

    Each class is hash-sampled across the whole pool, then mixed. Never the first N rows.

  2. Two opinions

    Both models get the same short text. Labels, confidence and latency are recorded.

  3. One retry

    An invalid label gets one more attempt. A bad batch can take twice as long, and that is visible.

  4. Durable punch

    Each batch commits to SQLite before the next starts. A hard kill costs one batch, not the run.

The live dashboard

Plain HTML, CSS and JS. No framework, no build server, no data leaves the machine. The local app runs at http://127.0.0.1:3030.

The DLLY lab dashboard showing 100 of 100 tasks, Doolally at 80 percent accuracy and Trollino at 85 percent, with a 96 percent speed win for the small model.
A real 100-task run: Doolally 80% at 594 ms, Trollino 85% at 26 ms, 13 disagreements and 34 committed batches.

Real 100-task result

Mode 100, seed 42, same play sample, retry 1, zero provider errors.

Model Role Accuracy Median latency Speed Notes
LFM2.5-8B-A1BQ4_K_M, 5.16 GB Doolally, moderator 80-85% 594 ms 1x Reasoning capped at 128 tokens
d1-3BQ4_K_M, 1.67 GB Trollino, author 85% 26 ms 22x faster Decision model, one forward pass
LFM2.5-8B-A1B FTQ4KXL, 6.17 GB, optional Speed variant 65% 110 ms 5x faster Not better here, so not the default

Accuracy is measured against the dataset label. The labels are heuristic, so treat this as a lab result, not a leaderboard. LFM scored 85% in a host run and 80% in the container run, which is why the table shows a range. The optional fine-tune was tested because it follows instructions more directly. It was faster, but 15 to 20 points less accurate on this sample, so stock LFM stays the default.

The interesting part: the 3B decision model matched or beat the 8B chat model on this narrow task, while using one fifth of the disk and answering in milliseconds. Narrow classification does not always need a large generative model.

Trollino pulls the plug

The demo kill is a real SIGKILL at a batch boundary. The recovery is a real SQLite resume.

What survives

  • The run row, the ordered task list and every committed batch.
  • Both model attempts per task, with labels, confidence and latency.
  • The last committed batch index, so replay starts at the right place.

What the test proved

  • Kill after batch 1: 3 tasks committed, 97 pending.
  • Resume: 100 tasks, exactly 200 attempts, 34 batches.
  • Zero duplicate task and model pairs after recovery.

The data and the cost

Short texts, four classes, a committed pool and an honest note about licenses.

Classes

  • Politics & Elections
  • Medical & Health
  • Legal & Regulatory
  • Financial & Crypto

10,000 rows, 2,500 per class. Eight experiment folds of 1,000 rows plus a 2,000-row holdout. Text is capped at 2,000 characters.

Sources

The rows are derived from user content and news. Review the terms before commercial use. The lab keeps source links and license metadata with the dataset.

Run it locally

Node 24, pixi and Docker with the NVIDIA runtime. All containers use the dlly- prefix.

pixi run python lab/models/download.py
docker compose -f docker/compose.yaml up -d --build
npm run server
# open http://127.0.0.1:3030

The mock provider runs without Docker and without models, so the full queue, kill and resume flow can be demonstrated offline first. Phoenix at http://127.0.0.1:6006 shows one span per committed batch.