Trevor Dwyer · sqlartist

I Built My Head Out of Lego. Inside SQL Server 2025.

Fri, 16 Oct 2026 11:16:54 GMT

The money shot: the 3D scan on the left, me in Lego on the right. 4,149 bricks in 13 colours, designed from start to finish by SQL Server 2025.

I could predict the comments if I didn’t give it a brain

The last time my head went into SQL Server it came out lopsided, 6,000 metres tall and with a stunt rear head. If you missed it, that was the Relational Space Carving article, and I finished it by promising you robot eyes and the beginning of a cloning process.

The robot eyes are still in a drawer.

Another time I tried to render my head so that I could create a Metahuman and please do not ask me how this happened, but this came out…

I have no words really, but I think its something to do with an unclosed mesh.

Instead, I asked myself a much simpler question. Is there any clever maths behind Lego?

I was expecting the answer to be “not really, it’s plastic”. It turns out there is, and nearly all of it is something a relational database is already very good at. So, the plan became: store Lego bricks in SQL Server, build something simple, then build something hard. And the hardest thing I could think of was me.

I approached the next section from two directions:

Create from a simple front-facing photo

I feed a standard 2D photograph of my own face into SQL Server 2025 and have pure T-SQL sculpt, colour-match, hollow out, interlock, and export a life-size, physically buildable 3D Lego bust.

No Python. No C#. No external mesh slicers. Just an uncompressed bitmap, and a relational schema

Create from an existing OBJ file

Create the Lego head from an existing 67 MB OBJ of my head. Everything that follows (loading it, turning it into bricks, colouring it and writing out a file Lego’s own design software can open) runs in T-SQL on SQL Server 2025.

How it was done

Option 2

The Lego grid is all integers, if you pick the right unit

A Lego stud is 8 mm from the next one. A brick is 9.6 mm tall. A plate is 3.2 mm, a third of a brick.

The bit I love is this. Five plates stacked up is 16 mm, which is exactly two studs. That 5:2 ratio is why you can build sideways (“studs not on top”, if you’re a proper adult fan of Lego) and everything still lines up.

For a database person that’s wonderful news. Store X and Z in studs and Y in brick layers and every coordinate is an integer. No floating point, no tolerances, and every “does this touch that?” question becomes an equality join. LDraw, the open file format the whole Lego CAD world runs on, does the same thing: 20 units per stud, 24 per brick, 8 per plate.

There’s one trap, and I nearly fell straight into it. A 1×1 brick is not a cube. It’s 8 × 8 × 9.6 mm. Chop a model into cubes, call each one a brick, and your head comes out 20% too tall. Mine is long enough already and the first time I did this I looked like the statues on Easter Island.

Two 2×4 bricks, 24 ways, and Burnside’s lemma

The most famous bit of Lego maths is counting. How many ways can you join six 2×4 bricks? Lego quoted roughly 103 million for years. Then in 2005 a mathematician, Søren Eilers, actually counted them properly: 915,103,765.

Six is far too many to brute-force in a query. Two is perfect, though, and it’s a lovely bit of maths.

Fix the bottom brick. Slide the top brick over it, either parallel or turned 90 degrees, keeping at least one stud in contact. That gives 46 positions. But spin the whole thing round 180 degrees and some of those positions turn into each other, so they aren’t different builds.

Burnside’s lemma says the number of genuinely different builds is the average number of positions each rotation leaves unchanged. Doing nothing leaves all 46 alone. The 180-degree turn leaves only the 2 positions that are dead centre. (46 + 2) / 2 = 24.

WITH TopBrick AS
(
    SELECT w = 2, l = 4             -- parallel
    UNION ALL SELECT 4, 2           -- turned 90 degrees
),
Placement AS
(
    SELECT b.w, b.l, dx = x.value, dz = z.value
    FROM TopBrick AS b
    CROSS JOIN GENERATE_SERIES(-3, 3) AS x
    CROSS JOIN GENERATE_SERIES(-3, 3) AS z
    WHERE x.value < 2 AND x.value + b.w > 0     -- the footprints overlap in X...
      AND z.value < 4 AND z.value + b.l > 0     -- ...and in Z
)
SELECT RawPlacements  = COUNT(*),
       FixedBy180     = SUM(IIF(2 * dx + w = 2 AND 2 * dz + l = 4, 1, 0)),
       DistinctBuilds = (COUNT(*) + SUM(IIF(2 * dx + w = 2 AND 2 * dz + l = 4, 1, 0))) / 2
FROM Placement;

RawPlacements  FixedBy180  DistinctBuilds
- - - - - - - - - - - - - - - - - - - - - 
46             2           24

Have a look at that WHERE clause. “Do these two bricks overlap?” is exactly the “do these two date ranges overlap?” test every SQL developer has written a hundred times, once for each direction. That turns out to be the theme of this whole article. Lego problems keep turning into SQL problems we already know.

Something simple first. Or not.

My original plan was a little Lego minifigure first, then my head. Then I realised a minifigure isn’t brick-built at all. It’s a handful of moulded parts, so there’s nothing to calculate. The classic six-brick duck (which ties in nicely with those 915 million) or a Benchy would be better, and they’re coming next time.

The next time is when I create a complete 3D Printer Slicer sending directly to a PRUSA CORE ONE filament printer, again all in Transact-SQL.

I went straight for the head. Of course I did.

Loading a 67 MB OBJ file with STRING_SPLIT

An OBJ file is just text:

v  -0.089426 0.033380 -0.017880 <- a vertex
vt  0.074025 0.377586           <- a texture coordinate
f   1/1/1    2/2/2     3/3/3    <- a triangle: vertex/texture/normal

The catch is that faces point at vertices by their position in the file. The first `v` line is vertex 1, the second is vertex 2, and so on. So, line order matters, and `BULK INSERT` into a table doesn’t promise you the order. `STRING_SPLIT` with its `enable_ordinal` argument does:

SELECT @doc = BulkColumn
FROM OPENROWSET(BULK 'D:\_Articles\lego\Trevor_3d_Head3.obj', SINGLE_CLOB) AS f;

SELECT LineNum = s.ordinal, Line = s.value
INTO #Line
FROM STRING_SPLIT(@doc, CHAR(10), 1) AS s          -- the 1 switches on the ordinal
WHERE s.value LIKE 'v %' OR s.value LIKE 'vt %' OR s.value LIKE 'f %';

/* the nth "v" line is vertex n */
INSERT obj.Vertex (MeshID, VertexID, X, Y, Z)
SELECT @MeshID,
       ROW_NUMBER() OVER (ORDER BY l.LineNum),
       CAST(REGEXP_SUBSTR(l.Line, '\S+', 1, 2) AS float),
       CAST(REGEXP_SUBSTR(l.Line, '\S+', 1, 3) AS float),
       CAST(REGEXP_SUBSTR(l.Line, '\S+', 1, 4) AS float)
FROM #Line AS l
WHERE l.Line LIKE 'v %';

That’s `REGEXP_SUBSTR`, new in SQL Server 2025, pulling the 2nd, 3rd and 4th words out of each line.

My first run failed instantly. I had called the line-number column `LineNo`, and it turns out `LINENO` is a reserved word in T-SQL. Thirty years, and I never knew. Later, I did exactly the same thing by calling a variable `by`. Also reserved. Obviously.

Second run: 343,220 vertices and 686,435 triangles in 28 seconds.

My head is in metres, Y is up and my face points along +Z. It measures 18 × 29 × 24 cm. That’s including the neck, I should add.

Is my head watertight? Euler’s formula is a GROUP BY

Before you turn a mesh into bricks you want to know it’s a closed surface. No holes for the inside to leak out of.

There’s a lovely bit of 18th-century maths for this. For anything shaped like a sphere,

V − E + F = 2

vertices, minus edges, plus faces. That’s Leonhard Euler. And in a watertight mesh every edge is shared by exactly two triangles.

Both checks are a `GROUP BY`:

WITH Edge AS
(
    SELECT A = LEAST(e.A, e.B), B = GREATEST(e.A, e.B)
    FROM obj.Face AS f
    CROSS APPLY (VALUES (f.V1, f.V2), (f.V2, f.V3), (f.V3, f.V1)) AS e(A, B)
)
SELECT A, B, Uses = COUNT(*)      -- 1 = the edge of a hole, 2 = fine, 3+ = something weird
FROM Edge
GROUP BY A, B;

V      E       F      BoundaryEdges NonManifoldEdges Euler
343220 1029653 686435 2             1                2

Euler says 2. Two edges out of just over a million belong to only one triangle. I’ll take that.

Turning my head into bricks is a window function

Here’s the core idea. It’s the same idea my 3D printers’ slicer uses to fill in each layer.

Fire a ray straight through my head along every row of the brick grid. Each time the ray crosses the surface it goes outside → inside → outside → inside. Sort the crossings along the ray, pair them up (1st with 2nd, 3rd with 4th) and everything between a pair is solid head.

Finding where a ray hits a triangle is barycentric coordinates in a `CROSS APPLY`. Pairing the crossings up is `ROW_NUMBER()` and `LEAD()`, and filling in the bricks between them is `GENERATE_SERIES`:

WITH Ray AS
(
    SELECT j, k, t,
           n    = ROW_NUMBER() OVER (PARTITION BY j, k ORDER BY t),
           tOut = LEAD(t)      OVER (PARTITION BY j, k ORDER BY t),
           Hits = COUNT(*)     OVER (PARTITION BY j, k)
    FROM #Hit
)
INSERT lego.Voxel (ModelID, X, Y, Z)
SELECT @ModelID, i.value, r.j, r.k
FROM Ray AS r
CROSS APPLY GENERATE_SERIES(CAST(CEILING(r.t) AS int), CAST(FLOOR(r.tOut) AS int), 1) AS i
WHERE r.n % 2 = 1                -- entering the solid
  AND r.Hits % 2 = 0             -- a ray that leaks through a hole is skipped
  AND CEILING(r.t) <= FLOOR(r.tOut);

One subtle thing. If a ray passes exactly through a vertex or along an edge it gets counted twice, and inside and outside swap places. So, I nudge every ray about a micron off the exact grid. After that, any ray with an odd number of crossings must have leaked through a hole, and it gets skipped.

At life size: 619 rays, 1,256 crossings, zero odd rays, in under a second. Out came 8,331 1×1 bricks on a 23 × 31 × 30 grid.

8,331 1×1 bricks. You can already see the ears, the nose in profile and the flared neck cut.

Hollowing my head: peel the onion

Nobody builds a solid Lego head. You build a shell.

So, every brick gets a Depth. Depth 1 is any brick touching empty space. Depth 2 is any brick touching a Depth 1, and so on. A WHILE loop peels the onion one layer at a time until nothing changes. Mathematically it’s a discrete distance transform. In SQL it’s an `UPDATE … WHERE EXISTS` over six neighbour offsets.

My head is 11 layers deep at the thickest point (insert your own joke here). A hollow shell two studs thick is simply `WHERE Depth <= 2`: 3,340 bricks instead of 8,331.

Merging bricks is the date-range overlap query

3,340 1×1 bricks would be miserable to build and wouldn’t hold together. Real Lego sculptures use 2×4s, 2×8s, 1×3s and so on.

So, it goes biggest part first. Find every spot where a 2×8 fits, with all its cells free and the same colour. Those candidates overlap each other, so in each round I keep only the ones that no better-ranked candidate overlaps. The best candidate on each layer always wins, so every round makes progress. In graph theory that’s building a maximal independent set in rounds. In SQL it’s the date-range query again, once per axis:

SELECT ...
FROM #Cand AS c
WHERE NOT EXISTS (SELECT 1
                  FROM #Cand AS o
                  WHERE o.Y = c.Y AND o.Pri < c.Pri
                    AND o.X < c.X + c.SizeX AND c.X < o.X + o.SizeX
                    AND o.Z < c.Z + c.SizeZ AND c.Z < o.Z + o.SizeZ);

Even layers prefer bricks running across, odd layers prefer bricks running front to back. That way the joints in one layer are always covered by the bricks above. Proper brickwork bond, like a garden wall.

15 rounds, half a second, 731 bricks. Then I checked it the way you should check anything you’re proud of: zero overlapping bricks, zero missing cells, zero extra cells.

Life size, 731 bricks, hollow. In the side view you can see the alternating layers.

Getting it out: LDraw is a SELECT

LDraw is a text format with one line per brick: colour, position, a 3×3 rotation matrix and the part number. `3001.dat` is the classic 2×4.

CONCAT('1 ', b.ColourID, ' ',
       (2 * b.X + b.SizeX - 2 * c.cx) * 10, ' ',          -- x: centre of the footprint
       -(b.Y - c.y0 + 1) * 24, ' ',                       -- y: top of the layer (Y is down)
       -(2 * b.Z + b.SizeZ - 2 * c.cz) * 10, ' ',         -- z: flipped so the face is at the front
       IIF(b.SizeX >= b.SizeZ, '1 0 0 0 1 0 0 0 1',       -- long side along X as drawn...
                               '0 0 1 0 1 0 -1 0 0'),     -- ...or turned 90 degrees
       ' ', b.PartID, '.dat')

0 Trevor head (2x life size)
0 Author: LegoSQL on SQL Server 2025
1 19 -20 -24 -70 1 0 0 0 1 0 0 0 1 3008.dat
1 19 90 -24 -70 1 0 0 0 1 0 0 0 1 3622.dat
0 STEP

A couple of gotchas. LDraw’s Y axis points *down*, and a brick’s origin is the centre of its top face. Flipping Y and Z together gives you a 180-degree turn rather than a mirror image. Mirror-image me would be very unsettling.

Put a `0 STEP` line after each layer and the file should become layer-by-layer building instructions in BrickLink Studio, for free. (Spoiler: it does. More on that at the end.)

Life size was a disappointment

At life size my face is only 23 studs across, so the features are just one or two bricks deep. It’s a head. It isn’t obviously *my* head.

So: `@Scale = 2.0`. That gives 66,645 voxels on a 45 × 61 × 60 grid, a 14,544-cell shell and 2,969 bricks, in about 3 seconds. Brow, eye sockets, nose, mouth, chin and ears all come through.

2× life size, before colour. The three-quarter view is where it starts to look like me.

Colour: the texture that wasn’t there

The OBJ pointed at a material file that wasn’t in the folder. When I found it, it had no texture in it at all. The colour lived in the original Blender file, with four images packed inside it: the 4096 × 4096 Base Colour map, plus a metallic/roughness map, a normal map and an emission map.

Getting it out cost me five minutes of staring at a Blender process that had used five seconds of CPU and was doing absolutely nothing. Running it headless with ` — factory-startup -noaudio` sorted that out. This is the one step outside SQL Server I had to take: a tiny Blender script finds whatever image feeds Base Colour and saves it as a 1024 × 1024, 24-bit BMP.

My face as a texture atlas. A patchwork quilt of skin, stubble and the occasional eyeball.

A BMP is just SUBSTRING (again)

If you read the Space Carving article, you’ll know I’ve been here before. You’ll also know about my 32-bit alpha channel disaster. This time it’s 24-bit. I learned my lesson.

SQL Server can’t decode a JPEG or a PNG. But a 24-bit BMP is just a 54-byte header followed by raw pixels: bottom row first, three bytes each, stored Blue-Green-Red, with each row padded out to a multiple of 4 bytes. So `OPENROWSET(… SINGLE_BLOB)` plus `SUBSTRING` is a perfectly good image decoder:

DECLARE @Stride int = (@W * 3 + 3) / 4 * 4;      -- each row is padded to 4 bytes

INSERT obj.Texel (MeshID, PX, PY, R, G, B)
SELECT @MeshID, p.PX, p.PY,
       CAST(SUBSTRING(@bmp, p.Byte0 + 3, 1) AS tinyint),     -- stored B, G, R
       CAST(SUBSTRING(@bmp, p.Byte0 + 2, 1) AS tinyint),
       CAST(SUBSTRING(@bmp, p.Byte0 + 1, 1) AS tinyint)
FROM GENERATE_SERIES(0, @W * @H - 1) AS g
CROSS APPLY (SELECT PX = g.value % @W, PY = g.value / @W) AS xy
CROSS APPLY (SELECT xy.PX, xy.PY, Byte0 = @Offset + xy.PY * @Stride + xy.PX * 3) AS p;

1,048,576 pixels in 6 seconds. I spot-checked three of them against Python’s imaging library and they were identical.

Is that actually my face? Checking the UVs

The OBJ had been exported from an earlier version of the Blender file (“Head2”), and the texture came from a later one (“Head3”). If the texture coordinates didn’t match, I’d be painting an ear onto my forehead.

So before trusting anything, I splatted every triangle onto a flat image, coloured from the texture through its UVs.

Eyes, nose, beard and ears, all exactly where they should be.

Nearest colour is VECTOR_DISTANCE

Now for the fun part. For every one of the 686,435 triangles:

- its centre’s texture coordinate picks a pixel, which gives a colour
— its centre’s position picks a grid cell, which gives a brick
— its area is the weight

Each brick’s colour is the area-weighted average of all the triangles that land on it, snapped to the nearest real Lego colour.

“Nearest” has to be measured the way eyes see colour, not in RGB, so everything is converted to CIELAB first. And this is where SQL Server 2025 really is necessary. Every Lego colour is stored as a `VECTOR(3)`, and finding the nearest one is just this:

SELECT TOP (1) c.ColourID
FROM lego.Colour AS c
WHERE c.InPalette = 1
ORDER BY VECTOR_DISTANCE('euclidean', c.Lab, @pixel);

That’s a feature Microsoft built for AI embeddings, being used to choose between Medium Nougat and Dark Tan 😊

The first result was… camouflage.

First attempt. Me, in army camouflage.

There were two problems. Stubble averaged with skin came out as Dark Tan, which is a sort of olive green, so my cheeks went military. And my skin was far too dark, because the scan had its lighting baked into the texture. A typical skin pixel was around 48 on the Lab lightness scale. In a photo of the real me it’s nearer 74.

I tried a “dominant colour” mode, where each brick takes whichever Lego colour covers most of its area rather than the average. It’s sharper, but it lost the tiny dark pupils, so I went back to the average.

The real fixes were two parameters:

- A portrait palette. An `InPalette` flag on the colour table. Out go Red, Blue, Yellow and that olive Dark Tan.
— Exposure. Multiply the Lab lightness before snapping to a colour.

Here’s one skin pixel before and after:

Exposure 1.0  Exposure 1.25
Name          Distance Name          Distance
Medium Brown  8.5      Medium Nougat 13.3
Sand Red      10.3     Nougat        15.6
Medium Nougat 14.8     Medium Tan    17.5

Exposure 1.0, 1.25 and 1.45. At 1.45 my forehead starts turning into white bricks.

So, 1.25 it is. Proper skin tones, eyes with irises, the grey goatee and the grey sides of the buzz cut.

Into BrickLink Studio

Then I did the thing I’d been putting off. I installed BrickLink Studio and opened the file SQL Server had written.

4,149 bricks, each one exactly where a SELECT statement put it, in the right colour and the right way round. And, unlike last time, no head 6,000 metres tall.

Rendered in BrickLink Studio. Every brick placed by a SELECT statement.

And here’s the build, from the very first brick.

And those `0 STEP` lines? Studio turned them into proper building instructions: 61 steps across 16 pages, one step per layer, with the parts for each layer in a box at the top, exactly like a real Lego manual.

Steps 41 to 44 of 61. SQL Server wrote my building instructions.

The Bill of Materials: What Does My Head Cost on BrickLink?

The greedy mesher compressed 65,204 raw studs down to 15,274 interlocked plates, for an estimated BrickLink total of £576.19

Two statistics jump right out on the Bill of Materials

The Red Core Savings: Just 1,157 Red 2 × 8 plates swallowed 18,512 individual interior studs inside my cranium — eliminating over 17,000 parts and costing just £23.14.
The Price of a Salt-and-Pepper Beard: Why are the top three rows of the Bill of Materials dominated by 1 × 1 and 1 × 2 Light and Dark Bluish Gray plates (£239.16 worth of tiny grey pieces)? Because every fleck of grey hair in my beard and temples alternates pixel-by-pixel. The greedy mesher refused to merge adjacent grey and brown voxels into larger plates, preserving the exact dithered texture of the beard.

The one bit I’m not happy with is the chin. On the scan it’s a grey goatee. In Lego it’s a brown patch with a black hole under the jaw. That’s the shadow under my chin, baked into the texture and now faithfully reproduced in plastic. SQL Server did exactly what I asked it to. It always does. That’s the problem.

I have no brain

Then I watched the build animation again and noticed something. About a third of the way in, the camera looks straight down into my head, and there’s nothing there. Two studs of wall and a lot of fresh air.

I’d built myself without a brain. Brian could have told you that.

So, a new plan for a video. Build the head almost to the top. Stop. Rewind back down to about half way. Grow a brain inside. Then close the skull back over it. And, obviously, do the whole thing in SQL Server.

Growing a brain is three ellipsoids and a SQUARE()

A Lego brain, it turns out, is three shapes:

- the cerebrum, a big ellipsoid filling the top of the skull
- the cerebellum, a smaller one tucked in low at the back
- the brain stem, a cylinder that carries on down through my neck as the spinal cord. Turns out I do have a backbone.

Everything is placed relative to my head’s own bounding box (0 to 1 across, up, and front to back), so the same numbers work at any scale. An ellipsoid is just x² + y² + z² ≤ 1 with each axis stretched, so `SQUARE()` did all the anatomy:

CROSS APPLY (SELECT Part = CASE
    WHEN SQUARE((p.u - 0.50) / 0.41) + SQUARE((p.v - 0.74) / 0.19) + SQUARE((p.w - 0.45) / 0.38) <= 1 THEN 'cerebrum'
    WHEN SQUARE((p.u - 0.50) / 0.27) + SQUARE((p.v - 0.57) / 0.08) + SQUARE((p.w - 0.17) / 0.12) <= 1 THEN 'cerebellum'
    WHEN SQUARE((p.u - 0.50) / 0.07) + SQUARE((p.w - 0.29) / 0.05) <= 1 AND p.v BETWEEN 0.07 AND 0.62 THEN 'stem'
    END) AS s
WHERE h.ModelID = @HeadModelID
  AND h.Depth > 2 + @Gap              -- inside the 2-brick skull, plus a gap

Remember the onion peel? It pays off again here. “Inside my skull” is simply `Depth > 2`, and I leave one more empty layer between skull and brain. That gap is my cerebrospinal fluid. Technically it’s air, but it’s air with a medical name.

The folds are where it got interesting. A smooth pink blob doesn’t say “brain”. Wrinkles do. So, I made a wiggly 3D wave (three sine waves, each one wobbled by another) and cut a groove wherever it crosses zero.

My first attempt said, “close to zero counts”, and it came out as a pink blob with a rash. Scattered speckles, no lines. The wave changes faster than one brick, so a “near zero” test skips straight over cells. The fix is to ask a better question: *is the wave positive here and negative next door?* That gives an unbroken line one brick wide, every time:

WHERE v.Depth = 1                                              -- surface bricks only
  AND lego.BrainWave(v.X, v.Y, v.Z, @Scale, @k) >= 0           -- positive here...
  AND EXISTS (SELECT 1
              FROM (VALUES (1,0,0), (-1,0,0), (0,1,0), (0,-1,0), (0,0,1), (0,0,-1)) AS d(dx, dy, dz)
              WHERE lego.BrainWave(v.X + d.dx, v.Y + d.dy, v.Z + d.dz, @Scale, @k) < 0)   -- ...negative next door

The bricks exposed at the bottom of each groove turn Dark Pink, there’s a slot down the middle between the two halves, and the stem is Light Pink. All three colours are flagged `InPalette = 0`, so no query will ever decide my cheeks are Dark Pink.

2,287 bricks of brain: 1,409 Pink, 711 Dark Pink and 167 Light Pink.

2,287 bricks of brain, with the spinal cord heading off down my neck.

The video’s script is a SELECT

This is the bit I’m pleased about because the whole video, every brick appearing, disappearing and coming back, is a table.

Every brick gets up to three frame numbers: T1 when it first appears, T2 when it’s rewound away, and T3 when it comes back. The renderer shows a brick when:

(frame >= T1 AND frame < T2) OR frame >= T3

The bricks below the half-way point only ever get a T1. The ones between half way and the top get all three. The brain, and the very top of my head, only ever appear once.

The frame numbers are `ROW_NUMBER()` divided by a bricks-per-frame rate. Within each layer the bricks go in order of their angle around the middle (`ATN2`), so every layer sweeps round like the hand of a clock. And the rewind is literally the same query with `DESC` on the end:

/* A. Build the head up to the stop layer */
WITH A AS (SELECT T1, n = ROW_NUMBER() OVER (ORDER BY Y, Angle) - 1
           FROM #T WHERE ModelID = @HeadModelID AND Y <= @Stop)
UPDATE A SET T1 = @f + n / @BuildRate;

/* B. Rewind: the same bricks, backwards, faster */
WITH B AS (SELECT T2, n = ROW_NUMBER() OVER (ORDER BY Y DESC, Angle DESC) - 1
           FROM #T WHERE ModelID = @HeadModelID AND Y > @Back AND Y <= @Stop)
UPDATE B SET T2 = @EndA + n / @RewindRate;

Phase    FromFrame ToFrame Layers
A build  1         264     0–53
B rewind 294       337     53–36
C brain  352       542     all
D finish 572       698     36–60

Lights, camera, Blender

SQL Server can’t render video. Yet. So Blender reads the script, and every brick and every stud becomes a Geometry Nodes instance: 6,436 bricks and 20,389 studs, as two clouds of points. A handful of nodes check the same `T1`/`T2`/`T3` rule on every frame. Each brick drops into place from 3.5 cm up, and in the rewind they lift back off, like a tape running backwards.

On my RTX 5090 that’s about a second a frame in Cycles. 854 frames, 28.5 seconds of video, about a quarter of an hour.

The failures, as is tradition:

- The first test frames were so overexposed I looked like I’d been carved out of a bar of soap. The fix was to let the camera see a bright white backdrop while the bricks are lit by a much dimmer version of it.

Rewinding to exactly half way cut off my eyes, and the brain ended up floating above the stump like a pink lollipop on a stick. Half of the *head* (not counting the neck) is eye level, 58% of the way up. With the eyes back, the brain sits in the skull like it should.

Build. Nearly there. Hang on. Rewind. Grow a brain. Close the skull. Spin.

The whole thing. Every frame of it decided by a SELECT statement.

Last time I promised you a cloning process. Growing my own brain out of a SELECT statement is as close as I’m going to get.

The final numbers

 |                                | Life size | 2× life size |
 | - - - - - - - - - - - - - - - - - - -      | - - - - - - -|
 | 1×1 voxels                     | 8,331     | 66,645       |
 | Hollow shell (2 studs)         | 3,340     | 14,544       |
 | Bricks, uncoloured             | 731       | 2,969        |
 | Bricks, coloured               | 1,061     | 4,149        |
 | Colours                        | 11        | 13           |
 | Building steps (one per layer) | 30        | 61           |

Notice that colour costs bricks. A brick can only be one colour, so every colour boundary chops the big bricks up. Life size went from 731 bricks to 1,061.

The 2× parts list is a fun read: 1,224 1×1 bricks, 1,187 1×2s and 770 2×2s. The biggest colour by a mile is Light Bluish Grey, with 1,057 bricks. Apparently, I am mostly grey. Bit depressing.

Add the brain and it’s 6,436 bricks. A third of them are brain. I’m choosing to be flattered by that.

What’s next

- Fix my chin. Probably by keeping Medium Brown out of the beard, or by switching to dominant colour just for that area.
- Will it fall apart? Every brick needs to connect to the bricks below it through at least one stud. That’s a graph reachability problem, and SQL Server has graph tables and `SHORTEST_PATH`.
- Actually building it. 4,149 small plastic bricks and Brian the sausage dog in the same house. That’s a problem for future me.

And the robot eyes are still in the drawer.

Hope you enjoyed the article, and the failures. I’ll get the code up on GitHub.

Trevor (@sqlartist)

Order! Order!

Wed, 14 Oct 2026 10:32:20 GMT

I wouldn’t trust any one of them.

The question

When I put 1.7 million speeches through two graphics cards to see whether ministers ever steal the Opposition’s ideas, one thing kept nagging at me. It wasn’t the speeches. It was the man in the big chair, shouting “Order!” every few minutes.

How often does the Speaker actually have to step in? Leave the parties to their own devices and they become a rabble. Everyone knows that. But how much of a rabble, and has it changed?

So I asked SQL Server.

Why a hypercube

My website already has a toy on it: every story I’ve written, placed inside a rotating four-dimensional cube. I wanted to give it something worth looking at.

The Hypercube — a 4D OLAP Cube analysing articles on my website.

Here’s the thing most people don’t know. SQL Server has had a hypercube built in since 2008. GROUP BY CUBE takes a list of dimensions and works out every combination of them in one statement. Give it five (year, Speaker, type of business, party, Government or Opposition) and it computes all 32 ways of slicing the House, from the finest detail right up to the grand total.

I made one rule for myself. The database does the thinking. The viewer only draws what SQL Server worked out, and it never adds anything up itself.

My Visit to the Chambers

I almost forgot about this, but I have been to the House of Commons as a guest of a friend and his charitable work. We were to have dinner with Peter Lilly, one of the MP’s who sat in Margret Thatchers cabinet. It was an interesting visit.

I arrive and go through the many security gates and arrive in the main building, we then have a quick tour — first to the House of Commons where we weren’t allowed to sit on the seats — by this time my back had gone and I was in terrible pain so I just kept thinking — whats the harm, but I observed the rules.

Before we went into the chamber there is a life size statue of Sir Winston Churchill in Bronze, its aged but his left foot is shiny, I asked why and apparently people touch his foot for luck when they enter the chamber.

The House of Lords was closed that day so we went down into the depths of the building and ended up in a dinning room over looking the Thames. It was at this point I realised I was the only person in the entire group of 20 that wasn’t wearing a tie. Londoners don’t wear ties these days, except in a few places and this was one of them.

I sat directly opposite the Rt Hon Lord Lilley, and it was interesting as I only eat bread at these functions — I never want to eat under cooked lamb or a shrimp starter so I stick with bread. towards the end of the meal I want to smoke, so I go to leave and it turns out I need an escort. An armed Police Officer with an MP5 machine gun escorted me all the way to the main entrance, I smoked and he escorted me all the way back. At this point I saw people were chatting and I happened to be standing next to Peter Lilly and I remarked to him that I though his body of work was astounding. He was quite taken back. I had read every one of his ministerial papers the night before and back then I had a very decent memory. So I ended up chatting, with no tie to Margret Thatchers’ Minister for quite some time. All because regardless what he was actually like as an MP, but because he quite obviously believed in the data and that was evident in the huge amount of papers he had published and that meant in my eyes he was “a decent guy”.

My first numbers were wrong

I started with the data I already had, the 1.7 million speeches from the vector search work. The first results came back and looked lovely. Bercow intervened more than anyone. Opposition and Government were treated much the same. I nearly wrote it up.

Then the cracks showed. The years 2002 to 2004 were almost empty. The speeches weren’t in the order they were spoken. And when I went back to the source, Evan Odell’s Hansard Speeches 1979–2021, it had 2.7 million rows, not 1.7.

The missing million told the story. To build embeddings, I’d thrown away every speech too short to mean anything. And what’s the shortest speech in Parliament? “Order.” My cleaned-up dataset had quietly kept about a quarter of everything the Chair ever said. I was measuring the Speaker with most of the Speaker removed.

That’s the lesson I’d pass on. Data you prepared for one question will happily answer a different one, and it won’t tell you it’s wrong. It’ll give you a neat answer.

Getting it back was satisfying. Every row in the source carries a TheyWorkForYou ID that records the sitting, the column and the paragraph, which is the true order of speaking. I matched my speeches back to it by date and their first 80 characters. 1,700,614 out of 1,700,647 paired up, in 34 seconds. Then I loaded the whole source instead, all 2.7 million rows, procedural lines and all.

I also wrote a second version of the counting rules in Python, whose only job was to disagree with the SQL. It found a bug on day one. “Order. Order.” was being counted once, because the full stop that ends the first call is also the one that starts the second. Two implementations, one argument, one fix. It’s the same trick that saved me on the AS/400.

/* =====================================================================
   Order! Order!  -  Commons Hypercube
   00_adapter.sql : maps the HANSARD database onto what the build needs.

   Source tables (already in HANSARD from the Hansard vector work):
     dbo.speech         one row per Commons speech, 1979 onwards
     dbo.speaker_party  each speaker's main party - fills in speeches
                        where dbo.speech.party is NULL
     dbo.gov_period     who governed when - the same definition of
                        "Government" the Hansard article used

   Seq: speech_id is NOT in sitting order (diagnostic H). The real
     order comes from the source dataset (Odell, Hansard Speeches
     1979-2021, biglam/hansard_speech): each row's TheyWorkForYou id
     carries sitting, column and paragraph. python/load_source_order.py
     loads those ids into hc.SourceOrder, EXEC hc.usp_MatchOrder (07)
     pairs them with dbo.speech into hc.SpeechOrder. Until then Seq is
     NULL and "who had the floor" is left Unknown rather than guessed.
   Also check:
     - The table is Commons only. If Lords rows are in there, filter
       them out in the WHERE clause below.
   ===================================================================== */
USE [HANSARD];
GO

IF SCHEMA_ID('hc') IS NULL EXEC('CREATE SCHEMA hc');
GO

/* The source's own order, loaded by python/load_source_order.py.
   Created here (and never dropped by later scripts) because the view
   below needs it to exist. */
IF OBJECT_ID('hc.SourceOrder') IS NULL
CREATE TABLE hc.SourceOrder
(
    Gid          varchar(100)  NOT NULL PRIMARY KEY,  -- uk.org.publicwhip/debate/1979-05-24a.1254.1
    SittingDate  date          NOT NULL,
    Sitting      varchar(4)    NOT NULL,              -- the 'a' after the date
    ColNum       int           NULL,
    Para         int           NULL,
    SpeechClass  varchar(20)   NULL,                  -- Speech | Procedural | ...
    SpeakerName  nvarchar(400) NULL,
    TextKey      nvarchar(80)  NOT NULL               -- first 80 non-space characters
);
GO
IF OBJECT_ID('hc.SpeechOrder') IS NULL
CREATE TABLE hc.SpeechOrder
(
    speech_id   int          NOT NULL PRIMARY KEY,
    Seq         bigint       NOT NULL,                -- position within the sitting day
    Gid         varchar(100) NOT NULL
);
GO

CREATE OR ALTER VIEW hc.vSpeech
AS
SELECT
    SpeechId     = CAST(s.speech_id AS bigint),
    SittingDate  = s.speech_date,
    Seq          = o.Seq,                             -- NULL until hc.usp_MatchOrder has run
    SpeechClass  = CAST('Speech' AS varchar(20)),      -- dbo.speech holds speeches only
    -- headings arrive wrapped in tabs and line breaks ('\n\tPRIME MINISTER\n  ')
    MajorHeading = TRIM(NCHAR(9) + NCHAR(10) + NCHAR(13) + N' ' FROM CAST(LEFT(s.major_heading, 400) AS nvarchar(400))),
    MinorHeading = TRIM(NCHAR(9) + NCHAR(10) + NCHAR(13) + N' ' FROM CAST(LEFT(s.minor_heading, 400) AS nvarchar(400))),
    SpeakerName  = CAST(LEFT(s.speaker_name, 400)  AS nvarchar(400)),
    PersonId     = CAST(s.person_id AS nvarchar(100)),
    Party        = CAST(COALESCE(NULLIF(LTRIM(RTRIM(s.party)), ''), sp.main_party) AS nvarchar(200)),
    -- the post they held when they spoke: the reliable way to spot a
    -- Deputy Speaker who is printed under their own name
    ParlPost     = CAST(LEFT(pp.parl_post, 500) AS nvarchar(500)),
    Body         = CAST(s.body AS nvarchar(max))
FROM dbo.speech AS s
LEFT JOIN dbo.speech_post AS pp
  ON pp.speech_id = s.speech_id
LEFT JOIN hc.SpeechOrder AS o
  ON o.speech_id = s.speech_id
LEFT JOIN
(   -- one main party per name; the biggest wins if a name appears twice
    SELECT speaker_name, main_party,
           rn = ROW_NUMBER() OVER (PARTITION BY speaker_name ORDER BY rows_in_main DESC)
    FROM dbo.speaker_party
) AS sp
  ON sp.speaker_name = s.speaker_name
 AND sp.rn = 1;
GO

/* "Government" comes from dbo.gov_period, inclusive at both ends as
   in dbo.usp_who_said_it_later. One party per period: if 2010-15 is
   stored as Conservative only, Lib Dems count as Opposition then.   */
CREATE OR ALTER VIEW hc.vGovernment
AS
SELECT FromDate = date_from,
       ToDate   = date_to,
       Party    = CAST(gov_party AS nvarchar(100))
FROM dbo.gov_period;
GO

/* Sanity check: one row per speech, sensible dates. */
-- SELECT COUNT_BIG(*) AS Speeches, MIN(SittingDate) AS FirstDay, MAX(SittingDate) AS LastDay,
--        COUNT(DISTINCT SittingDate) AS SittingDays
-- FROM hc.vSpeech;

/* =====================================================================
   Order! Order!  -  Commons Hypercube
   00b_adapter_full.sql : the alternative to 00_adapter.sql.

   Points hc.vSpeech at hc.SourceSpeech - the whole of Odell's
   "Hansard Speeches 1979-2021", loaded by python/load_source_full.py.
   Every speech and procedural line, 2001-04 included, in sitting order,
   with party and parliamentary posts as they were at the time.

   Run order:  00b (creates the table)  ->  load_source_full.py
               ->  00b again is harmless  ->  01, 02, 03  ->  06
   To go back to dbo.speech, just run 00_adapter.sql again.
   ===================================================================== */
USE [HANSARD];
GO

IF SCHEMA_ID('hc') IS NULL EXEC('CREATE SCHEMA hc');
GO

IF OBJECT_ID('hc.SourceSpeech') IS NULL
CREATE TABLE hc.SourceSpeech
(
    Id           bigint        IDENTITY(1,1) NOT NULL PRIMARY KEY,
    Gid          varchar(100)  NOT NULL UNIQUE,       -- uk.org.publicwhip/debate/1979-05-24a.1254.1
    SittingDate  date          NOT NULL,
    Seq          bigint        NOT NULL,              -- sitting letter, column, paragraph
    SpeechClass  varchar(20)   NULL,                  -- Speech | Procedural | ...
    SpeakerName  nvarchar(400) NULL,                  -- as printed: 'Mr. Speaker'
    DisplayAs    nvarchar(400) NULL,
    PersonId     nvarchar(100) NULL,
    Party        nvarchar(200) NULL,
    MajorHeading nvarchar(400) NULL,
    MinorHeading nvarchar(400) NULL,
    ParlPost     nvarchar(500) NULL,
    Body         nvarchar(max) NULL,
    INDEX ix_day (SittingDate, Seq)
);
GO

CREATE OR ALTER VIEW hc.vSpeech
AS
SELECT
    SpeechId     = s.Id,
    SittingDate  = s.SittingDate,
    Seq          = s.Seq,
    SpeechClass  = s.SpeechClass,
    MajorHeading = TRIM(NCHAR(9) + NCHAR(10) + NCHAR(13) + N' ' FROM s.MajorHeading),
    MinorHeading = TRIM(NCHAR(9) + NCHAR(10) + NCHAR(13) + N' ' FROM s.MinorHeading),
    SpeakerName  = s.SpeakerName,
    PersonId     = s.PersonId,
    Party        = NULLIF(LTRIM(RTRIM(s.Party)), N''),
    ParlPost     = s.ParlPost,
    Body         = s.Body
FROM hc.SourceSpeech AS s;
GO

CREATE OR ALTER VIEW hc.vGovernment
AS
SELECT FromDate = date_from, ToDate = date_to, Party = CAST(gov_party AS nvarchar(100))
FROM dbo.gov_period;
GO

From keeping order to keeping time

With the whole record in place, the Chair intervened about 61,000 times in 42 years. Here’s how often, per 1,000 speeches by Members:

Bernard Weatherill (1983–92): 36. Nearly 13 interventions on an average sitting day. The most policed House of the lot.
Betty Boothroyd (1992–2000): 30.
Michael Martin (2000–09): 26.
George Thomas (1976–83): 24.
John Bercow (2009–19): 19.

So the man everyone remembers shouting “Order!” was, by this measure, the least interventionist of the lot. That’s the opposite of what my first, broken numbers said.

But Bercow was busy with something else. Telling people to get on with it: “short questions and short answers”, “be brief”, “there is a lot of interest”. Under Thomas, Weatherill and Boothroyd that came up less than once per 1,000 speeches. Under Martin it was about one. Under Bercow it was 3.6, and in 2018 it hit 4.8.

The job changed. In the 1980s the Chair was keeping order. By the 2010s it was keeping time, trying to get as many Members in as possible before the clock ran out. Whatever else you think of Bercow, and there’s a lot to think, he was known for trying to call everyone who stood up.

The Government lost its protection

This was the finding I didn’t expect, and it only appeared once the speeches were back in their proper order.

For every intervention I worked out who had the floor at the time: the last Member to speak before the Chair stood up. Then I compared Government and Opposition, per 1,000 of their own speeches.

Thomas: the Opposition was pulled up 2.0 times as often as the Government.
Weatherill: 2.1 times.
Boothroyd: 1.6 times.
Martin: 1.3 times.
Bercow: 1.1 times.

Forty years ago, sitting opposite the Government roughly doubled your chances of hearing “Order!”. By the Bercow years it barely mattered which side you sat on.

And it’s the side, not the party. Labour MPs were told off most under Weatherill, when they were in opposition. Under Martin, with Labour in government, the Conservatives took over. Whoever sits opposite gets the shouting.

There’s a caveat here. The Opposition raises most of the points of order, and “That is not a point of order” counts as an intervention. Some of the old gap is just the machinery of opposition. But points of order are a small slice of the total, and the gap closes steadily across five Speakers. Something real changed.

Two casts

Averages over 2.7 million speeches smooth everything out. The good stuff is in the people. So I split the Chair’s interventions in two. Personal means sit down, withdraw that remark, or being named. At the House means the Chair telling everyone else to be quiet. Then there’s Hansard’s own little stage direction, [Interruption.], which tells you the House was shouting while someone spoke.

/* =====================================================================
   Order! Order!  -  Commons Hypercube
   02_build.sql : hc.usp_Build turns hc.vSpeech into two fact tables.

     hc.Speech     one row per speech, no body: who, when, what kind of
                   business, their party and side, and - for the Chair -
                   who had the floor when it spoke.
     hc.ChairFlag  one row per (Chair speech, Category) that matched.

   EXEC hc.usp_Build;          -- full rebuild
   ===================================================================== */
USE [HANSARD];
GO

DROP TABLE IF EXISTS hc.ChairFlag, hc.Speech;
GO

CREATE TABLE hc.Speech
(
    SpeechId     bigint        NOT NULL PRIMARY KEY,
    SittingDate  date          NOT NULL,
    Yr           smallint      NOT NULL,
    Seq          bigint        NULL,      -- NULL = position in the day unknown
    Role         varchar(10)   NOT NULL,   -- Member | Speaker | Deputy | Committee | Floor | Procedural
    RoleSource   varchar(6)    NOT NULL,   -- Title | Post | Party | None
    Interrupts   int           NOT NULL,   -- '[Interruption.]' marks in the speech
    SpeakerName  nvarchar(400) NULL,
    PersonId     nvarchar(100) NULL,
    Party        nvarchar(100) NOT NULL,   -- canonical; 'Chair'/'House' for non-members
    Side         varchar(10)   NOT NULL,   -- Government | Opposition | Unknown | Chair | House
    Reign        nvarchar(60)  NOT NULL,
    DebateType   nvarchar(30)  NOT NULL,
    -- who had the floor (the last Member to speak that day before this one)
    TargetName   nvarchar(400) NULL,
    TargetPerson nvarchar(100) NULL,
    TargetParty  nvarchar(100) NULL,
    TargetSide   varchar(10)   NULL,
    INDEX ix_day (SittingDate, Seq)
);
GO

CREATE TABLE hc.ChairFlag
(
    SpeechId bigint      NOT NULL,
    Category varchar(12) NOT NULL,
    Family   varchar(12) NOT NULL,
    Hits     int         NOT NULL,
    PRIMARY KEY (SpeechId, Category)
);
GO

CREATE OR ALTER PROCEDURE hc.usp_Build
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;
    DECLARE @t0 datetime2 = SYSDATETIME();

    TRUNCATE TABLE hc.ChairFlag;
    TRUNCATE TABLE hc.Speech;

    /* -----------------------------------------------------------------
       1. Classify every distinct heading pair once, not 1.7M times.
       ----------------------------------------------------------------- */
    DROP TABLE IF EXISTS #Heading;

    SELECT h.MajorHeading, h.MinorHeading,
           DebateType = COALESCE(t.DebateType,
                          CASE WHEN h.MinorHeading <> N''
                                AND h.MajorHeading LIKE N'%[A-Z]%'
                                AND h.MajorHeading COLLATE Latin1_General_CS_AS
                                  = UPPER(h.MajorHeading) COLLATE Latin1_General_CS_AS
                               THEN N'Questions' END,
                          N'Other debate')
    INTO   #Heading
    FROM  (SELECT DISTINCT MajorHeading = ISNULL(MajorHeading, N''),
                           MinorHeading = ISNULL(MinorHeading, N'')
           FROM hc.vSpeech) AS h
    OUTER APPLY (SELECT TOP (1) p.DebateType
                 FROM hc.DebateTypePattern AS p
                 WHERE (p.Field IN ('Major','Either') AND h.MajorHeading LIKE p.Pattern)
                    OR (p.Field IN ('Minor','Either') AND h.MinorHeading LIKE p.Pattern)
                 ORDER BY p.Priority) AS t;

    CREATE UNIQUE NONCLUSTERED INDEX ix ON #Heading (MajorHeading, MinorHeading);  -- nonclustered: 1,700-byte key limit

    /* -----------------------------------------------------------------
       2. One row per speech: role, party, side, reign, debate type.
       ----------------------------------------------------------------- */
    INSERT hc.Speech (SpeechId, SittingDate, Yr, Seq, Role, RoleSource, Interrupts,
                      SpeakerName, PersonId, Party, Side, Reign, DebateType)
    SELECT v.SpeechId, v.SittingDate, YEAR(v.SittingDate), v.Seq,
           r.Role, r.Source,
           Interrupts = CASE WHEN v.Body LIKE N'%[[]Interruption%'
                             THEN (DATALENGTH(v.Body) - DATALENGTH(REPLACE(v.Body, N'[Interruption', N'')))
                                  / DATALENGTH(N'[Interruption')
                             ELSE 0 END,
           v.SpeakerName, v.PersonId,
           p.Party,
           Side = CASE
                    WHEN r.Role IN ('Speaker','Deputy','Committee') THEN 'Chair'
                    WHEN r.Role IN ('Floor','Procedural')            THEN 'House'
                    WHEN p.Party = N'Unknown'                        THEN 'Unknown'
                    WHEN EXISTS (SELECT 1 FROM hc.vGovernment AS g
                                 LEFT JOIN hc.PartyAlias AS ga ON ga.Raw = g.Party
                                 WHERE COALESCE(ga.Canon, g.Party) = p.Party
                                   AND v.SittingDate BETWEEN g.FromDate AND g.ToDate)
                                                                     THEN 'Government'
                    ELSE 'Opposition'
                  END,
           rg.Speaker,
           h.DebateType
    FROM hc.vSpeech AS v
    LEFT JOIN hc.PartyAlias AS a
           ON a.Raw = v.Party
    JOIN hc.Reign AS rg
      ON v.SittingDate >= rg.FromDate AND v.SittingDate < rg.ToDate
    /* Who is in the Chair, in order of trust:
         1. the printed title    'Mr. Deputy Speaker', 'The Chairman'
         2. the post they held   parl_post 'Speaker of the House of Commons' / 'Deputy Speaker ...'
         3. party 'Speaker' - only if the name is the reigning Speaker's, because
            older rows carry party 'Speaker' on the wrong people ('Mr. Denzil Davies') */
    OUTER APPLY (SELECT TitleRole = (SELECT TOP (1) c.Role FROM hc.ChairName AS c
                                     WHERE v.SpeakerName LIKE c.Pattern
                                     ORDER BY c.Priority),
                        PostRole  = CASE WHEN v.ParlPost LIKE N'%Speaker of the House of Commons%' THEN 'Speaker'
                                         WHEN v.ParlPost LIKE N'%Deputy Speaker%'                  THEN 'Deputy' END,
                        PartyRole = CASE WHEN a.Canon = N'Speaker'
                                          AND v.SpeakerName LIKE N'%' + rg.Speaker + N'%'           THEN 'Speaker' END) AS rr
    -- unattributed procedural lines ('Question put and agreed to.') are nobody's speech
    CROSS APPLY (SELECT Role   = COALESCE(rr.TitleRole, rr.PostRole, rr.PartyRole,
                                          CASE WHEN v.SpeechClass = 'Procedural' THEN 'Procedural' END,
                                          'Member'),
                        Source = CASE WHEN rr.TitleRole IS NOT NULL THEN 'Title'
                                      WHEN rr.PostRole  IS NOT NULL THEN 'Post'
                                      WHEN rr.PartyRole IS NOT NULL THEN 'Party'
                                      ELSE 'None' END) AS r
    CROSS APPLY (SELECT Party = CASE
                    WHEN r.Role IN ('Speaker','Deputy','Committee') THEN N'Chair'
                    WHEN r.Role IN ('Floor','Procedural')            THEN N'House'
                    ELSE LEFT(COALESCE(a.Canon, NULLIF(LTRIM(RTRIM(v.Party)), N''), N'Unknown'), 100)
                 END) AS p
    JOIN #Heading AS h
      ON h.MajorHeading = ISNULL(v.MajorHeading, N'')
     AND h.MinorHeading = ISNULL(v.MinorHeading, N'');

    /* -----------------------------------------------------------------
       3. Who had the floor? For every speech, the most recent Member
          to speak that day. LAG ... IGNORE NULLS (SQL Server 2022+)
          skips over the Chair and the shouting.
       ----------------------------------------------------------------- */
    -- everyone off the floor starts as 'Unknown'; placed speeches get a real target below
    UPDATE hc.Speech SET TargetParty = N'Unknown', TargetSide = 'Unknown'
    WHERE Role <> 'Member';

    WITH t AS
    (
        SELECT SpeechId, Role,
               TName   = LAG(CASE WHEN Role = 'Member' THEN ISNULL(SpeakerName, N'?') END) IGNORE NULLS
                             OVER (PARTITION BY SittingDate ORDER BY Seq, SpeechId),
               TPerson = LAG(CASE WHEN Role = 'Member' THEN ISNULL(PersonId, N'?') END) IGNORE NULLS
                             OVER (PARTITION BY SittingDate ORDER BY Seq, SpeechId),
               TParty  = LAG(CASE WHEN Role = 'Member' THEN Party END) IGNORE NULLS
                             OVER (PARTITION BY SittingDate ORDER BY Seq, SpeechId),
               TSide   = LAG(CASE WHEN Role = 'Member' THEN Side END) IGNORE NULLS
                             OVER (PARTITION BY SittingDate ORDER BY Seq, SpeechId)
        FROM hc.Speech
        WHERE Seq IS NOT NULL          -- no guessing: unplaced speeches keep 'Unknown'
    )
    UPDATE s
       SET TargetName   = NULLIF(t.TName, N'?'),
           TargetPerson = NULLIF(t.TPerson, N'?'),
           TargetParty  = ISNULL(t.TParty, N'Unknown'),
           TargetSide   = ISNULL(t.TSide,  'Unknown')
    FROM hc.Speech AS s
    JOIN t ON t.SpeechId = s.SpeechId
    WHERE t.Role <> 'Member';

    /* -----------------------------------------------------------------
       4. What did the Chair say? Only Chair speeches are scanned, and
          each is normalised once and tested against every pattern.
          Hits = occurrences, counted with the REPLACE length trick.
       ----------------------------------------------------------------- */
    INSERT hc.ChairFlag (SpeechId, Category, Family, Hits)
    SELECT s.SpeechId, p.Category, MIN(p.Family),
           SUM((DATALENGTH(n.Norm) - DATALENGTH(REPLACE(n.Norm, p.Pattern, N'')))
               / DATALENGTH(p.Pattern))
    FROM hc.Speech AS s
    JOIN hc.vSpeech AS v ON v.SpeechId = s.SpeechId
    CROSS APPLY hc.fnNorm(v.Body) AS n
    JOIN hc.InterventionPattern AS p
      ON n.Norm LIKE (N'%' + p.Pattern + N'%') COLLATE DATABASE_DEFAULT
    WHERE s.Role IN ('Speaker','Deputy','Committee')
    GROUP BY s.SpeechId, p.Category;

    /* -----------------------------------------------------------------
       5. Report.
       ----------------------------------------------------------------- */
    SELECT Step = 'built',
           Speeches       = (SELECT COUNT_BIG(*) FROM hc.Speech),
           ChairSpeeches  = (SELECT COUNT_BIG(*) FROM hc.Speech WHERE Role IN ('Speaker','Deputy','Committee')),
           Interruptions  = (SELECT SUM(CAST(Interrupts AS bigint)) FROM hc.Speech),
           ChairByPost    = (SELECT COUNT_BIG(*) FROM hc.Speech WHERE RoleSource = 'Post'),
           PlacedInOrder  = (SELECT COUNT_BIG(*) FROM hc.Speech WHERE Seq IS NOT NULL),
           Interventions  = (SELECT COUNT(DISTINCT SpeechId) FROM hc.ChairFlag WHERE Family = 'Discipline'),
           OrderCalls     = (SELECT SUM(CAST(Hits AS bigint)) FROM hc.ChairFlag WHERE Category = 'ORDER'),
           Seconds        = DATEDIFF(second, @t0, SYSDATETIME());
END;
GO

The building the cube itself.

/* =====================================================================
   Order! Order!  -  Commons Hypercube
   03_cube.sql : GROUP BY CUBE over five dimensions.

   SQL Server has had a hypercube built in since 2008. GROUP BY CUBE
   computes every combination of the dimensions in one statement:
   five dimensions = 2^5 = 32 grouping sets, from "every year, every
   Speaker, every party..." down to the single grand total.

   Dimensions : Year, Reign (which Speaker), DebateType, Party, Side
   Measures   : member speeches (the denominator), Chair interventions
                and their kinds, outbursts from the floor, and rates
                per 1,000 member speeches - all computed here, so the
                viewer only ever draws what the database worked out.

   EXEC hc.usp_BuildCube;
   EXEC hc.usp_ExportCube;             -- one row, one JSON document
   ===================================================================== */
USE [HANSARD];
GO

DROP TABLE IF EXISTS hc.CubeCell, hc.CubeDim;
GO

CREATE TABLE hc.CubeCell
(
    GroupingId     int           NOT NULL,   -- GROUPING_ID bitmask, 0 = finest grain
    Dims           tinyint       NOT NULL,   -- how many dimensions this cell keeps
    Yr             nvarchar(4)   NULL,
    Reign          nvarchar(60)  NULL,
    DebateType     nvarchar(30)  NULL,
    Party          nvarchar(100) NULL,
    Side           varchar(10)   NULL,
    MemberSpeeches bigint        NOT NULL,
    ChairSpeeches  bigint        NOT NULL,
    Interventions  bigint        NOT NULL,   -- Chair speeches with any Discipline flag
    BySpeaker      bigint        NOT NULL,   -- ... of which the Speaker personally
    OrderCalls     bigint        NOT NULL,   -- every "Order." counted
    SitDown        bigint        NOT NULL,
    Withdraw       bigint        NOT NULL,
    Named          bigint        NOT NULL,
    Suspended      bigint        NOT NULL,
    Noise          bigint        NOT NULL,
    Hurry          bigint        NOT NULL,   -- "be brief", "short questions"
    FloorOutbursts bigint        NOT NULL,   -- '[Interruption.]' marks + "Hon. Members:" rows
    RateInterventions decimal(12,3) NULL,    -- per 1,000 member speeches
    RateOrder         decimal(12,3) NULL,
    RateHurry         decimal(12,3) NULL,
    RateFloor         decimal(12,3) NULL
);
GO

CREATE TABLE hc.CubeDim
(
    Dim     varchar(12)  NOT NULL,
    Value   nvarchar(100) NOT NULL,
    SortKey int          NOT NULL,
    PRIMARY KEY (Dim, Value)
);
GO

CREATE OR ALTER PROCEDURE hc.usp_BuildCube
AS
BEGIN
    SET NOCOUNT ON;
    TRUNCATE TABLE hc.CubeCell;
    TRUNCATE TABLE hc.CubeDim;

    WITH flags AS
    (
        SELECT SpeechId,
               Disc     = MAX(CASE WHEN Family   = 'Discipline' THEN 1 ELSE 0 END),
               OrderHit = SUM(CASE WHEN Category = 'ORDER'      THEN Hits ELSE 0 END),
               SitDown  = MAX(CASE WHEN Category = 'SEAT'       THEN 1 ELSE 0 END),
               Withdraw = MAX(CASE WHEN Category = 'WITHDRAW'   THEN 1 ELSE 0 END),
               Named    = MAX(CASE WHEN Category = 'NAME'       THEN 1 ELSE 0 END),
               Susp     = MAX(CASE WHEN Category = 'SUSPEND'    THEN 1 ELSE 0 END),
               Noise    = MAX(CASE WHEN Category = 'NOISE'      THEN 1 ELSE 0 END),
               Hurry    = MAX(CASE WHEN Family   = 'Hurry'      THEN 1 ELSE 0 END)
        FROM hc.ChairFlag
        GROUP BY SpeechId
    ),
    facts AS
    (
        -- the denominator: every Member's speech, under their own party
        SELECT Yr, Reign, DebateType, Party, Side,
               Msp = 1, Csp = 0, Itv = 0, Bsp = 0, Ord = 0, Sdn = 0, Wdr = 0, Nmd = 0, Sus = 0, Nse = 0, Hry = 0,
               Flr = Interrupts                     -- '[Interruption.]' while they spoke
        FROM hc.Speech WHERE Role = 'Member'

        UNION ALL

        -- the Chair, filed under whoever had the floor
        SELECT s.Yr, s.Reign, s.DebateType, s.TargetParty, s.TargetSide,
               0, 1,
               ISNULL(f.Disc, 0),
               CASE WHEN s.Role = 'Speaker' THEN ISNULL(f.Disc, 0) ELSE 0 END,
               ISNULL(f.OrderHit, 0), ISNULL(f.SitDown, 0), ISNULL(f.Withdraw, 0),
               ISNULL(f.Named, 0), ISNULL(f.Susp, 0), ISNULL(f.Noise, 0), ISNULL(f.Hurry, 0),
               s.Interrupts
        FROM hc.Speech AS s
        LEFT JOIN flags AS f ON f.SpeechId = s.SpeechId
        WHERE s.Role IN ('Speaker','Deputy','Committee')

        UNION ALL

        -- the House shouting, filed under whoever had the floor
        SELECT Yr, Reign, DebateType, TargetParty, TargetSide,
               0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1
        FROM hc.Speech WHERE Role = 'Floor'
    )
    INSERT hc.CubeCell
        (GroupingId, Dims, Yr, Reign, DebateType, Party, Side,
         MemberSpeeches, ChairSpeeches, Interventions, BySpeaker, OrderCalls,
         SitDown, Withdraw, Named, Suspended, Noise, Hurry, FloorOutbursts)
    SELECT GROUPING_ID(Yr, Reign, DebateType, Party, Side),
           5 - (GROUPING(Yr) + GROUPING(Reign) + GROUPING(DebateType)
                + GROUPING(Party) + GROUPING(Side)),
           CAST(Yr AS nvarchar(4)), Reign, DebateType, Party, Side,
           SUM(CAST(Msp AS bigint)), SUM(CAST(Csp AS bigint)), SUM(CAST(Itv AS bigint)), SUM(CAST(Bsp AS bigint)),
           SUM(CAST(Ord AS bigint)), SUM(CAST(Sdn AS bigint)), SUM(CAST(Wdr AS bigint)), SUM(CAST(Nmd AS bigint)),
           SUM(CAST(Sus AS bigint)), SUM(CAST(Nse AS bigint)), SUM(CAST(Hry AS bigint)), SUM(CAST(Flr AS bigint))
    FROM facts
    GROUP BY CUBE (Yr, Reign, DebateType, Party, Side);

    /* Every base row has a real value on every dimension (hc.usp_Build
       fills gaps with 'Unknown'), so a NULL here always means "rolled up". */

    UPDATE hc.CubeCell
       SET RateInterventions = CAST(1000.0 * Interventions  / NULLIF(MemberSpeeches, 0) AS decimal(12,3)),
           RateOrder         = CAST(1000.0 * OrderCalls     / NULLIF(MemberSpeeches, 0) AS decimal(12,3)),
           RateHurry         = CAST(1000.0 * Hurry          / NULLIF(MemberSpeeches, 0) AS decimal(12,3)),
           RateFloor         = CAST(1000.0 * FloorOutbursts / NULLIF(MemberSpeeches, 0) AS decimal(12,3));

    /* Axis order for the viewer: years in order, Speakers in order,
       business from PMQs downwards, parties by how much they speak. */
    INSERT hc.CubeDim (Dim, Value, SortKey)
    SELECT 'Year', Yr, CAST(Yr AS int)
    FROM hc.CubeCell WHERE GroupingId = 15 /* Year only */ AND Yr IS NOT NULL
    UNION ALL
    SELECT 'Reign', r.Speaker, r.ReignId
    FROM hc.Reign AS r WHERE EXISTS (SELECT 1 FROM hc.Speech AS s WHERE s.Reign = r.Speaker)
    UNION ALL
    SELECT 'DebateType', c.DebateType,
           ISNULL((SELECT MIN(p.Priority) FROM hc.DebateTypePattern AS p WHERE p.DebateType = c.DebateType), 99)
    FROM hc.CubeCell AS c WHERE c.GroupingId = 27 /* DebateType only */
    UNION ALL
    SELECT 'Party', c.Party, ROW_NUMBER() OVER (ORDER BY c.MemberSpeeches DESC)
    FROM hc.CubeCell AS c WHERE c.GroupingId = 29 /* Party only */
    UNION ALL
    SELECT 'Side', c.Side, CASE c.Side WHEN 'Government' THEN 1 WHEN 'Opposition' THEN 2 ELSE 3 END
    FROM hc.CubeCell AS c WHERE c.GroupingId = 30 /* Side only */;

    SELECT Step = 'cube built',
           Cells        = COUNT_BIG(*),
           GroupingSets = COUNT(DISTINCT GroupingId)
    FROM hc.CubeCell;
END;
GO

/* ---------------------------------------------------------------------
   The whole cube as one JSON document for the viewers.
   Rolled-up dimensions are NULL, and FOR JSON leaves NULLs out, so
   each cell's "d" object holds exactly the dimensions it keeps.
   @MinSpeeches drops cells too small to have a meaningful rate.
   --------------------------------------------------------------------- */
CREATE OR ALTER PROCEDURE hc.usp_ExportCube
    @MinSpeeches int = 0
AS
BEGIN
    SET NOCOUNT ON;

    SELECT CubeJson =
    (
        SELECT
            title     = N'Order! Order!',
            source    = N'Hansard, House of Commons',
            generated = CONVERT(varchar(19), SYSDATETIME(), 126),
            dimensions = JSON_QUERY((
                SELECT name = d.Dim,
                       [values] = JSON_QUERY((
                            SELECT [value] = v.Value
                            FROM hc.CubeDim AS v
                            WHERE v.Dim = d.Dim
                            ORDER BY v.SortKey
                            FOR JSON PATH))
                FROM (VALUES ('Year',1),('Reign',2),('DebateType',3),('Party',4),('Side',5)) AS d(Dim, Ord)
                ORDER BY d.Ord
                FOR JSON PATH)),
            measures = JSON_QUERY(N'[
                {"key":"rateInterventions","label":"Interventions per 1,000 speeches"},
                {"key":"rateOrder","label":"\"Order!\" per 1,000 speeches"},
                {"key":"rateHurry","label":"\"Be brief\" per 1,000 speeches"},
                {"key":"rateFloor","label":"[Interruption] per 1,000 speeches"},
                {"key":"interventions","label":"Interventions (count)"}
            ]'),
            cells = JSON_QUERY((
                SELECT [d.Year]       = c.Yr,
                       [d.Reign]      = c.Reign,
                       [d.DebateType] = c.DebateType,
                       [d.Party]      = c.Party,
                       [d.Side]       = c.Side,
                       speeches          = c.MemberSpeeches,
                       chair             = c.ChairSpeeches,
                       interventions     = c.Interventions,
                       bySpeaker         = c.BySpeaker,
                       orderCalls        = c.OrderCalls,
                       sitDown           = c.SitDown,
                       withdraw          = c.Withdraw,
                       named             = c.Named,
                       suspended         = c.Suspended,
                       noise             = c.Noise,
                       hurry             = c.Hurry,
                       [floor]           = c.FloorOutbursts,
                       rateInterventions = c.RateInterventions,
                       rateOrder         = c.RateOrder,
                       rateHurry         = c.RateHurry,
                       rateFloor         = c.RateFloor
                FROM hc.CubeCell AS c
                WHERE c.MemberSpeeches >= @MinSpeeches
                FOR JSON PATH))
        FOR JSON PATH, WITHOUT_ARRAY_WRAPPER
    );
END;
GO

/* =====================================================================
   Order! Order!  -  Commons Hypercube
   04_findings.sql : the questions the article asks. Run after
   hc.usp_Build and hc.usp_BuildCube. Each query stands alone.
   ===================================================================== */
USE [HANSARD];
GO

/* 0. Is there anything to report? Rerunning 02 or 03 recreates their
      tables empty, so the builds have to be run again afterwards. */
SELECT ViewRows   = (SELECT COUNT_BIG(*) FROM hc.vSpeech),
       SpeechRows = (SELECT COUNT_BIG(*) FROM hc.Speech),
       ChairFlags = (SELECT COUNT_BIG(*) FROM hc.ChairFlag),
       CubeCells  = (SELECT COUNT_BIG(*) FROM hc.CubeCell);

IF NOT EXISTS (SELECT 1 FROM hc.Speech)
    RAISERROR('hc.Speech is empty: run EXEC hc.usp_Build; then EXEC hc.usp_BuildCube;', 16, 1);
ELSE IF NOT EXISTS (SELECT 1 FROM hc.CubeCell)
    RAISERROR('hc.CubeCell is empty: run EXEC hc.usp_BuildCube;', 16, 1);
GO

/* 1. Which Speaker had the rowdiest House?
      Per 1,000 member speeches and per sitting day, with the Speaker's
      own interventions split from the deputies'. */
SELECT c.Reign,
       Speeches        = c.MemberSpeeches,
       SittingDays     = d.Days,
       Interventions   = c.Interventions,
       PerThousand     = c.RateInterventions,
       PerSittingDay   = CAST(1.0 * c.Interventions / NULLIF(d.Days, 0) AS decimal(9,2)),
       SpeakerOwnShare = CAST(100.0 * c.BySpeaker / NULLIF(c.Interventions, 0) AS decimal(5,1)),
       OrderPerK       = c.RateOrder,
       HurryPerK       = c.RateHurry,
       FloorPerK       = c.RateFloor,
       Named           = c.Named,
       Suspended       = c.Suspended
FROM hc.CubeCell AS c
JOIN (SELECT Reign, Days = COUNT(DISTINCT SittingDate) FROM hc.Speech GROUP BY Reign) AS d
  ON d.Reign = c.Reign
JOIN hc.Reign AS r ON r.Speaker = c.Reign
WHERE c.GroupingId = 23   -- Reign only
ORDER BY r.ReignId;

/* 2. Is PMQs louder than everything else? Each Speaker, each kind of business. */
SELECT c.Reign, c.DebateType, c.MemberSpeeches, c.RateInterventions, c.RateFloor
FROM hc.CubeCell AS c
JOIN hc.Reign AS r ON r.Speaker = c.Reign
WHERE c.GroupingId = 19   -- Reign x DebateType
  AND c.MemberSpeeches >= 500
ORDER BY r.ReignId, c.RateInterventions DESC;

/* 3. The cameras. The Commons was first televised on 21 November 1989,
      in the middle of Weatherill's time - same Speaker, before and after.
      Three years before, and the rest of his time after (about 2.5 years),
      so the comparison isn't a change of Chair. */
WITH w AS
(
    SELECT Era = CASE WHEN s.SittingDate < '1989-11-21' THEN '1 Before cameras' ELSE '2 After cameras' END,
           s.Role, s.SpeechId, s.Interrupts
    FROM hc.Speech AS s
    WHERE s.SittingDate >= '1986-11-21' AND s.SittingDate < '1992-04-27'
)
SELECT w.Era,
       Speeches      = SUM(CASE WHEN w.Role = 'Member' THEN 1 ELSE 0 END),
       Interventions = COUNT(DISTINCT CASE WHEN f.Family = 'Discipline' THEN w.SpeechId END),
       PerThousand   = CAST(1000.0 * COUNT(DISTINCT CASE WHEN f.Family = 'Discipline' THEN w.SpeechId END)
                        / NULLIF(SUM(CASE WHEN w.Role = 'Member' THEN 1 ELSE 0 END), 0) AS decimal(9,3)),
       InterruptPerK = CAST(1000.0 * SUM(CASE WHEN w.Role = 'Member' THEN w.Interrupts ELSE 0 END)
                        / NULLIF(SUM(CASE WHEN w.Role = 'Member' THEN 1 ELSE 0 END), 0) AS decimal(9,3))
FROM w
OUTER APPLY (SELECT TOP (1) cf.Family FROM hc.ChairFlag AS cf
             WHERE cf.SpeechId = w.SpeechId AND cf.Family = 'Discipline') AS f
GROUP BY w.Era
ORDER BY w.Era;

/* 4. Who gets told off: Government or Opposition? Under each Speaker.
      Needs hc.usp_MatchOrder (or the full source) - depends on who had the floor. */
SELECT c.Reign, c.Side, c.MemberSpeeches, c.Interventions, c.RateInterventions, c.RateFloor
FROM hc.CubeCell AS c
JOIN hc.Reign AS r ON r.Speaker = c.Reign
WHERE c.GroupingId = 22   -- Reign x Side
  AND c.Side IN ('Government','Opposition')
ORDER BY r.ReignId, c.Side;

/* 5. Year by year: the line for the article's main chart. */
SELECT Yr, MemberSpeeches, Interventions, RateInterventions, RateOrder, RateHurry, RateFloor
FROM hc.CubeCell
WHERE GroupingId = 15     -- Year only
ORDER BY Yr;

/* 6. The rowdiest days in the Commons since 1979. */
SELECT TOP (25)
       s.SittingDate,
       s.Reign,
       Interventions = COUNT(DISTINCT CASE WHEN f.Family = 'Discipline' THEN s.SpeechId END),
       OrderCalls    = SUM(CASE WHEN f.Category = 'ORDER' THEN f.Hits ELSE 0 END),
       Interruptions = (SELECT SUM(x.Interrupts) FROM hc.Speech AS x
                        WHERE x.SittingDate = s.SittingDate),
       Business      = (SELECT TOP (1) x.DebateType FROM hc.Speech AS x
                        JOIN hc.ChairFlag AS y ON y.SpeechId = x.SpeechId
                        WHERE x.SittingDate = s.SittingDate
                        GROUP BY x.DebateType ORDER BY COUNT(*) DESC)
FROM hc.Speech AS s
JOIN hc.ChairFlag AS f ON f.SpeechId = s.SpeechId
GROUP BY s.SittingDate, s.Reign
ORDER BY Interventions DESC;

/* 7. The nuclear option: every naming, and who had the floor.
      ("I name the hon. Member for ..." - then a vote to suspend.)
      Check each one by eye: the pattern can catch "I name names". */
SELECT s.SittingDate, s.Reign, s.SpeakerName AS Chair, s.TargetName, s.TargetParty, s.DebateType
FROM hc.Speech AS s
JOIN hc.ChairFlag AS f ON f.SpeechId = s.SpeechId AND f.Category = 'NAME'
ORDER BY s.SittingDate;

/* 8. Each party, under each Speaker: interventions per 1,000 of their speeches.
      Needs hc.usp_MatchOrder (or the full source) - depends on who had the floor. */
SELECT c.Reign, c.Party, c.MemberSpeeches, c.RateInterventions, c.RateFloor
FROM hc.CubeCell AS c
JOIN hc.Reign AS r ON r.Speaker = c.Reign
WHERE c.GroupingId = 21   -- Reign x Party
  AND c.MemberSpeeches >= 2000
ORDER BY r.ReignId, c.RateInterventions DESC;

/* 9. "Be brief": which Speaker nagged most about time? */
SELECT c.Reign, c.Hurry, c.RateHurry
FROM hc.CubeCell AS c
JOIN hc.Reign AS r ON r.Speaker = c.Reign
WHERE c.GroupingId = 23
ORDER BY r.ReignId;

The ones the Chair had to rein in

The House got more forgiving over time. Personal rebukes ran at about 1.6 per 1,000 speeches in the late 1980s and about 0.4 in the 2010s. So I measured everyone against the years they actually sat. A score of 1 means normal for your era, and 10 means ten times the norm. Minimum 500 speeches:

Ron Brown: 21 times his era’s rate, and named three times. Yes, the mace man.
Ian Blackford: 19 times. Hold that thought.
Andrew Faulds: 13 times.
David Shaw: 12 times, a rare Conservative near the top.
George Galloway: 11 times.

The rest of the list is mostly the Labour awkward squad of the 1980s: Eric Heffer, Dick Douglas, Norman Buchan, Allan Rogers. That’s the old Opposition gap again, seen from the other end. Tam Dalyell holds the record for being named, four times.

And Dennis Skinner? Named three times, and more than 10,000 speeches over 40 years. But measured against his era he scores 6, not even top 25. The Beast of Bolsover’s reputation is bigger than his record. He was just there, every day, for a very long time.

The ones the House wouldn’t let speak

Now rank by [Interruption], adjusted the same way, and a completely different cast appears. Party leaders, and especially Opposition leaders.

David Owen: 5.0 times the House’s norm. That’s what the two big parties thought of the SDP.
Neil Kinnock: 5.0. Thatcher, across 16,000 speeches, was well below him.
Ed Miliband: 4.7. He’s also the leader the Chair most often stepped in to protect.
Keir Starmer: 4.5. Boris Johnson: 2.9.

It’s the same pattern every time. The Prime Minister gets the protection of the despatch box and the Leader of the Opposition gets the noise.

The House as a whole got noisier too. Hansard recorded 9 interruptions per 1,000 speeches in 1979 and 33 in 1989, and it has mostly stayed in the twenties since. So these figures are era-adjusted. A leader in 2019 doesn’t get credit just for living in loud times.

The man in both lists

One name sits at the top of both tables. Ian Blackford, SNP leader at Westminster from 2017, was interrupted at 6.6 times the norm for his era. That’s 155 per 1,000 speeches, the highest of anyone since 1979, Prime Ministers and Opposition leaders included. He’s also second on the personal list, at 19 times his era’s rate, behind only Ron Brown. And on 13 June 2018 Bercow ordered him out of the chamber during Prime Minister’s Questions, and the SNP walked out behind him.

The House shouted him down more than anyone, and the Chair pulled him up more than almost anyone. Nobody else comes close on both counts.

The database found him twice, from two different directions. I like it when that happens.

Blackford (Me) actually being escorted out of the commons. The entire SNP benches (All me) following.

Did the cameras change anything?

The Commons was first televised on 21 November 1989, halfway through Weatherill’s time. So there’s a neat experiment sitting in the data: same Speaker, same Parliament, with and without an audience.

In the three years before, the Chair intervened 40.6 times per 1,000 speeches. Afterwards it was 38.4. Interruptions fell from 31.8 to 25.4. So MPs behaved slightly better with the nation watching, and then, as the trend shows, got used to it.

What I’d want you to know before quoting any of this

Hansard is an edited record. Not every heckle gets written down, and the habits of the people writing it have changed over 42 years.
“Who had the floor” is a proxy. “Order!” is often aimed at the benches opposite, not the person speaking. That’s why I kept rein in and drown out separate.
Lindsay Hoyle gets no verdict here. The data stops in April 2021, so his time in it is mostly the Covid hybrid Parliament. Most of his 300-odd suspensions were breaks “to allow the safe exit of hon. Members”, not disorder.
The counting is pattern matching. I read hundreds of matches by eye and tuned the rules until the misfires were rare. Rare isn’t never.

Next: the House with nobody to say it

The House of Lords has no one to call order. The Lord Speaker can’t rule on it. The peers police themselves, calling “Order!” from the benches. That’s the control group this article was missing: a chamber left entirely to its own devices.

And hundreds of MPs have made the move from one House to the other. So the next question is whether, with no seat to lose and nobody in a wig to shout at them, they behave any differently.

The code is on GitHub [link to follow], and the hypercube is on my website. Go and turn it through the fourth dimension, and see if you can spot Ron Brown.

Being kicked upstairs — The House Of Lords

I have included parts of the code used in the article as it shows what was run to get the analysis. But all the source code is on the github page and you run the files in order to obtain the result. There is a python script which downloads from Huggingface the Hanson dataset. Takes about 15 mins.

Code will be up in a few days.

If you enjoyed this please give it a thunderous round of applause.

Cracking Feynman’s Safes in SQL Server

Sat, 10 Oct 2026 20:02:05 GMT

The puzzle, not the physics

I build daft things in SQL Server. Doom, Elite, a Boing ball bouncing around inside a result set. The rule’s always the same: pick something that has no business living in a database, and make it live there anyway. You learn more about the engine from one silly project than from a shelf of modelling books.

This one is Richard Feynman cracking safes at Los Alamos.

If you’ve read Surely You’re Joking, Mr. Feynman! you know the chapter “Safecracker Meets Safecracker.” Everyone remembers it as Feynman being a genius. It isn’t that. It’s Feynman being patient, and noticing that the locks were far worse than anyone assumed. He didn’t out-muscle the safes. He worked out how little he actually had to try. That’s a database problem if ever I saw one, so I pointed SQL Server 2025 at it.

The thing he actually spotted

The filing cabinets used Mosler combination locks. Three numbers, 0 to 99, left-right-left. On paper that’s a million combinations, and a million is a comfortable number to hide behind. Nobody’s trying a million of anything by hand.

Feynman took his own lock apart, not to pick it, to measure it. And he found the thing the manufacturer would rather you didn’t: the mechanism was sloppy. A number still opened the safe if you were off by two in either direction. So 69 would open on 67, 68, 69, 70 or 71.

That sounds like a footnote. It isn’t. It means you never have to try every number, only every fifth one, 0, 5, 10, 15, and so on. Twenty values per wheel instead of a hundred.

| Approach                     | Combinations                |
| ---------------------------- | --------------------------- |
| Try every number             | 100 × 100 × 100 = 1,000,000 |
| Every fifth number (±2 slop) | 20 × 20 × 20 = 8,000        |

A million down to eight thousand, and he did it with arithmetic, not a drill. At the rhythm he practised to, eight thousand was a long afternoon, four hours on average, eight at the worst. Tedious, but possible, which a million never was. That gap between impossible and merely tedious is the whole chapter.

The bit that makes the story

Here’s where it stops being arithmetic and starts being mischief, and it’s my favourite part.

When someone leaves a drawer open with the dial resting on ten, the bolt is still sitting down in the slot. Which means the three wheels are still lined up, the safe is open, it just hasn’t been disturbed yet. Feynman worked out that he could read the combination off a safe in that state without the owner ever knowing.

He’d lean on an open filing cabinet while chatting about physics and idly wobble the dial. Step away from ten in fives; each time, back to ten. The moment the bolt won’t drop back, you’ve just nudged a wheel off its slot and the step before that was the last number. Do it again the other way for the second number. Two of the three numbers, taken off a safe while its owner stood next to you, and written on a scrap of paper he hid inside his own lock.

After that, “cracking it cold” later meant trying twenty first numbers while pretending to read a magazine, so it didn’t look too easy.

That’s the part I most wanted to see move, so it’s the first thing I built. The database wobbles the dial exactly as he describes, logging every step:

The dial, the three-wheel pack and the back-end probe log reading the last two numbers off Christy’s open cabinet — it settles on 35, 60

That’s Christy’s cabinet from the book. The log on the right shows the bolt holding up at 65, which fixes the last number at 60; the second falls out as 35. “Christy, 35, 60,” which is exactly what Feynman wrote down. I didn’t plan that number; it came out of the geometry, and I’ll admit I was happy when it matched the book.

Three tricks people blur into one

The popular version of this story mashes everything into “Feynman was good at safes.” He wasn’t, particularly. He had three separate tricks, and they’re worth keeping apart because the database treats each one differently.

The tolerance trick is the maths above, the only one that’s really his.

The open drawer is the cheat from the last section: read two numbers off anything left open, then finish it later in minutes.

And then the shortcuts, which are pure human nature. The safe-cracker books told owners to pick dates and constants, so Feynman tried dates and constants. De Hoffmann’s cabinets, the ones holding the actual bomb secrets, opened on 27–18–28, the first digits of e. Kerst’s opened on 0–5–35, which turned out to be his daughter’s birthday, found by sweeping dates in fives. And the locksmith’s dirty secret: half the safes were never changed from the factory settings, 25–0–25 or 50–25–50.

The database opening the Captain’s safe on the factory setting 50–25–50 in two tries

The Captain had a special heavy safe hauled up to his office on jacks so everyone could see how seriously he took security. He never changed it from the factory number. Two tries in the database and it’s open, forty seconds of pretend effort. I gave each of these a strategy in the schema, so you can watch a safe fall to the cheapest trick that happens to work on it.

The database opening Kerst’s safe cold by sweeping dates in fives — 0–5–35, a daughter’s birthday

Why I modeled the metal, not the maths

The obvious way to do this is a nested loop. Three counters, a modulo check, done in an afternoon. It would give the right numbers and teach me nothing, and it would be lying, because the ±2 slop isn’t a rule Feynman invented, it’s a physical fact about how a notched wheel sits under a bolt. If I hard-code “tolerance = 2” I’ve thrown away the very thing the story is about.

So I went the long way round and built the actual metal in SQL Server 2025’s spatial types. Every wheel, at every one of its hundred positions, is a real `geometry` disc with a gate cut into its rim. The bolt is a little bar sitting above each wheel. And then the entire physics of the lock, the whole question of whether it opens, collapses to one spatial predicate, asked once and stored:

-- does the bolt clear this wheel at this dial position?
UPDATE wg SET Clear = CASE WHEN wg.Geom.STIntersects(b.Geom) = 1
                          THEN 0 ELSE 1 END
FROM dial.WheelGeom wg
JOIN dial.Bolt b ON b.CabinetId = wg.CabinetId;

That’s the whole trick. If the gate overlaps the bolt, the bolt’s blocked. If it doesn’t, the bolt drops. I cut the gate a touch wider than one number and the ±2 slop falls out on its own: exactly five dial positions per wheel end up clear. I never typed the number two. The geometry hands it to me, the same way the real mechanism handed it to Feynman.

Once every position knows whether it’s clear, cracking a safe stops being physics and becomes a plain relational search: find the first candidate whose three wheels are all clear. That’s the division of labour I was after, geometry for the mechanism, set-based SQL for the searching. It’s slower to build than a loop and I’d do it again tomorrow, because now the database disagrees with you when you’re wrong about the lock, which a loop never would.

Watching the eight thousand shrink

A number on a page doesn’t land the way watching it does. So every attempt gets logged to a table with a running clock, and each try also carves out the little patch of the search it’s just ruled out. I laid that search out as a grid I’ve been calling Feynman space, twenty panels, one per first number, each a hundred-by-hundred square of the second and third.

Cracking a cabinet with no clues at all — the wheel pack on the left, Feynman space filling in blue on the right as the attempts are logged

That’s a cabinet with no clues, so there’s nothing for it but the full eight thousand. The blue is what’s been eliminated; the bolt drops the instant all three gates line up. The Python front end is drawing exactly what the database computed. The shapes come across the wire as the stored geometry, so I’m not re-faking the picture, I’m showing SQL Server’s own working.

And because every safe is logged the same way, you can line them all up and see the point of the whole exercise:

A bar chart of how long each safe took, from the Captain’s factory setting at forty seconds up to brute force running off the end of the chart

The Captain’s factory safe: forty seconds. De Hoffmann’s bomb secrets, on the digits of e: two minutes. Brute force with the tolerance trick thrown away: off the end of the chart, where a million always was. It’s the methods doing the work, not the muscle, which is the thing people always get backwards about Feynman.

There’s a live mode too: kick the crack off with a small delay and watch the rows appear one at a time, the back-end running in front of you instead of after the fact.

The lock that came after

Feynman learned to pick ordinary pin-tumbler locks from a chap called Leo Lavatelli, and the way he describes it is pure “slop leaks information” again. A real lock isn’t made perfectly, so one pin holds more of the load than the rest. Lift that one to the right height, the plug gives a hair, the pin catches on the edge. Now the load moves to the next pin. Push too far and the springs drop everything back with a tic-tic-tic. He called it Sisyphus. Anyone who’s picked a lock knows the feeling.

So the second half of the lab is a see-through practice lock, the kind sold as a clear teaching kit, built the same way, entirely from geometry. A key’s cuts are a profile; each pin rides along it; the plug turns only if every gap lands on the shear line.

The transparent lock with a near-miss key inserted — four pins reach the shear line, the fifth is cut one step too deep and blocks the plug

That’s next door’s key tried in the wrong lock. Four pins sit right on the red shear line; pin four is cut one step too deep, so its driver crosses the line and the plug won’t turn. The database decides that with the same `STIntersects` it used on the bolt, chamber by chamber, is anything straddling the line.

Picking is the interesting one to watch, over sets and all:

Single-pin picking the transparent lock — the pick lifting a binding pin while the others wait

Tension on, find the binding pin, lift until it sets with a click, move to the next. Overshoot at the shear line and it over sets, tension off, every pin drops, start again. I built the overshoot in on purpose, because a picker that never fumbles isn’t telling the truth about the job.

And then the payoff, which is the pin-tumbler cousin of taking the last two numbers off: because the pick records how high each pin had to go, the database can read the biting straight back off a successful pick and cut a working key.

The lock opening on a key the database cut from its own successful pick

What it’s really about

Strip away the filing cabinets and the fun of it, and both halves of this say the same thing: a real mechanism has slop, and slop leaks information. The combination wasn’t a million-to-one secret, it was a few hundred once you measured the tolerance. The lock wasn’t picked by force, it was read one binding pin at a time. Feynman didn’t beat the security; he noticed it was never really there.

That’s why it sits so naturally in a database. A spatial predicate isn’t pretending the metal exists, it is the metal’s one honest question, asked eight thousand times a second. Build the mechanism properly and the cracking is almost an afterthought, which is exactly the shape of the original story.

One caveat, since I’d want one if I were reading this. The spatial code is written for SQL Server 2025, convert it to other databases if you can. I built and checked the whole thing against a Python model of the same geometry, which reproduces every number in the book, de Hoffmann’s e and Kerst’s birthday included.

What’s in the package

Four T-SQL scripts build the whole thing on a SQL Server 2025 instance; a Python front end draws it, either from the live database (geometry crosses the wire as WKT and is drawn exactly as the database computed it) or from a built-in simulator that reproduces every episode, so the visuals run with no server at all.

sql/01_schema · 02_geometry · 03_dial · 04_pin_tumbler · 05_demo
python/dial_viz.py · pin_viz.py · feynman_lab/ (geometry mirror, simulators, SQL driver)
out/pre-rendered animations + the scoreboard

Run 05_demo.sql in SSMS with the Spatial results tab on to see the wheel pack, Feynman space and the see-through lock; or python dial_viz.py --source sql --live to watch dial.usp_Crack write rows in real time.

Source: “Safecracker Meets Safecracker,” from Surely You’re Joking, Mr. Feynman! (1985). The combinations shown — de Hoffmann’s 27–18–28, Kerst’s 0–5–35, the factory 25–0–25 / 50–25–50, Christy’s 35–60 — are the ones Feynman records. First-number values and a few cabinets are invented to fill out the set; the mechanics are his.

I hope you enjoyed this, I think it might be my favourite Feynman story in the whole of his book. If you did like it please give it some claps.

SQL Server 2025, an RTX 5090, and the slow death of “Will the hon. Gentleman give way?”

Thu, 08 Oct 2026 13:52:27 GMT

Why do I do this to myself?

The question

Do MPs ever change each other’s minds?

Not “do they vote differently” — the whips settle that. I mean something narrower and more interesting: does an argument made from the opposition benches ever end up coming out of a minister’s mouth? Somebody stands up in 1987, says a thing nobody has said before, and four years later the government is saying it back. That happens, surely. Parliament would be a very expensive piece of theatre if it never happened.

I had 1.7 million speeches, a database that can do vector search, and two graphics cards sitting in a machine under my desk doing nothing. It seemed like a fair question to put to them.

The answer, for about a month, was no.

I want to be clear that I did not want the answer to be no. I wanted a lovely little chart with a bump in it. What I got was two independent methods, built weeks apart on entirely different principles, both saying no, for entirely different reasons. Which is — and I have had to talk myself round to this — a much better result than a bump would have been. One method saying yes is a finding. Two methods saying no, where you can explain exactly *why* each one couldn’t see what you hoped it would see, is an actual piece of work.

It also means that everything genuinely interesting here was found by accident while I was failing at the main thing. The House of Commons has quietly stopped using its own interruption protocol. Being a minister makes you *more* likely to be shouted at, not less. Eighty-five per cent of every six words ever spoken in the Commons since 1979 was spoken exactly once and never again.

And at one point I had a query that proved every single Member of Parliament is funny one hundred per cent of the time. More on that later, because it is the single stupidest bug I have written in thirty years and I am oddly proud of it.

The only appreciating asset I have ever owned

I have been buying computer hardware since the mid-1990s. Tandem systems, AS/400s, a succession of PCs, every category of storage that has ever been invented and several that were quietly withdrawn. In thirty years, every single piece of it started depreciating the moment I got the box open. Some of it was worth less than the box.

Then I bought two RTX 5090s, about a week before the price went up.

They are now worth more than I paid for them. Not notionally, not “if you account for inflation” — actually, demonstrably, go and look, more money than I handed over. I have owned a house, several cars I never learned to drive, and roughly four hundred hard disks, and the only thing in thirty years that has gone *up* is a pair of graphics cards I bought to mess about with.

So there was a certain obligation to use them for something.

And here is the thing that nobody writing about GPUs seems to want to say out loud: the interesting work is not the GPU. It is the handover. A 5090 is phenomenally good at a small number of things and completely useless at everything else. SQL Server is phenomenally good at a different small number of things and would take a fortnight to do what the 5090 does in ninety seconds. Put them in the same room and make each one do only what it is good at, and you can answer questions that neither of them can answer alone.

That is the actual article. The parliamentary stuff is just the excuse.

Why Hansard

Everyone building a vector search demo uses the same three datasets. Wikipedia, because it is there. Product reviews, because somebody once wrote a tutorial. Movie plots, because the results are funny in screenshots. You end up with a demo that proves your index works and tells you nothing at all about the world.

Hansard is the official report of everything said in the House of Commons. The version I used covers 1979 onwards — 2,694,375 speeches, of which 1,700,647 are longer than forty words and attached to a real debate heading. That filter matters: below forty words you are mostly collecting “I beg to move”, “The Question is”, and men saying “no” at each other.

I went for post-1979 deliberately. The Victorian material is wonderful and completely irrelevant to anyone alive. If I am going to find something out, I would like it to be about a Parliament people remember.

What makes it a good corpus is not the size — 1.7 million rows is nothing, it fits in memory on a decent laptop. It is that every row is *attributed*. Speaker, party, date, debate, and whether they held office. You do not get that with Wikipedia. It means every question you can ask about the text, you can also ask about who said it, when, and from which side of the room.

It is also, bluntly, the best-documented argument in the English-speaking world, conducted continuously for four decades by people who were paid to be persuasive. If there is a signal anywhere for “did somebody change somebody else’s mind”, it is in here.

The division of labour

This is the bit I actually want you to take away, so I am going to be unusually clear about it.

SQL Server 2025 has a `VECTOR` type and `VECTOR_DISTANCE`. It can do similarity search. What it cannot do — what no database can do at any sensible speed — is the thing you need *before* similarity search is useful: look at 1.7 million 384-dimensional vectors all at once and work out where the clusters are.

That is k-means. It is forty iterations of a 1.7-million-by-41,730 matrix multiply. On a CPU that is a long lunch. On a 5090 it is ninety seconds, because it is exactly the shape of arithmetic a GPU exists to do: the same operation, on every element, with no branching and no conditionals.

What the GPU does

Plain PyTorch, no libraries beyond torch. Chunked because 1.7M × 41,730 does not fit in 32 GB all at once.

# Assignment step: every speech to its nearest centroid.
# X is (1.7M, 384) L2-normalised, C is (nlist, 384) normalised.
# With both normalised, cosine similarity IS the dot product,
# so the whole thing collapses to one matmul.

assign = torch.empty(N, dtype=torch.int32, device="cuda")

for lo in range(0, N, CHUNK):
    hi = min(lo + CHUNK, N)
    sims = X[lo:hi] @ C.T          # (chunk, nlist)
    assign[lo:hi] = sims.argmax(1)

# Update step: new centroid = mean of its members.
# index_add_ is the scatter-gather that makes this O(N)
# instead of O(N * nlist).
C.zero_()
C.index_add_(0, assign, X)
counts = torch.bincount(assign, minlength=NLIST).clamp(min=1)
C /= counts.unsqueeze(1)
C = torch.nn.functional.normalize(C, dim=1)

That is the whole algorithm. Forty of those, and 1.7 million speeches have been sorted into 41,730 topical neighbourhoods. The GPU then hands the database two things: a cluster id per speech, and a 384-bit binary signature per speech — one bit per dimension, sign only.

# 384 floats -> 384 bits -> 6 BIGINTs.
# Throws away the magnitude and keeps the direction.
# 1,536 bytes becomes 48. That is the 32x.
bits = (X > 0).to(torch.uint8)
packed = np.packbits(bits.cpu().numpy(), axis=1)   # (N, 48)
sig = packed.view(np.int64).byteswap()             # (N, 6)

And then the GPU is done. It has no further part in this. It cannot store anything, it cannot be queried, it does not know what a Conservative is.

What the database does

SQL Server’s job is everything the GPU cannot touch: persisting it, indexing it so that only the relevant slice is ever read, and — the part everybody forgets — joining it to who said it.

The cluster ids are the whole trick. Order the columnstore by cluster id and put it on a partition scheme, and a query for one neighbourhood touches one partition instead of the table.

/*  64 partitions over the cluster id space. The boundaries are
    not arbitrary - they come from a second k-means the GPU ran
    over the centroids themselves, so that clusters which are
    NEAR each other get ADJACENT ids and land in the same
    partition. Spatial locality, written into the physical
    layout of the table.                                        */
CREATE PARTITION FUNCTION pf_cluster (INT)
    AS RANGE LEFT FOR VALUES (652, 1304, 1956, /* ... */ 41078);

CREATE PARTITION SCHEME ps_cluster
    AS PARTITION pf_cluster ALL TO ([PRIMARY]);

/*  Declare the table ON the scheme FIRST. SELECT INTO will not
    do this and the index will not be partition-aligned if you
    try to bolt it on afterwards. I found that out the hard way. */
CREATE TABLE dbo.speech_sig (
    speech_id  INT    NOT NULL,
    cluster_id INT    NOT NULL,
    sig0 BIGINT NOT NULL, sig1 BIGINT NOT NULL, sig2 BIGINT NOT NULL,
    sig3 BIGINT NOT NULL, sig4 BIGINT NOT NULL, sig5 BIGINT NOT NULL
) ON ps_cluster (cluster_id);

CREATE CLUSTERED COLUMNSTORE INDEX ccix_speech_sig
    ON dbo.speech_sig ORDER (cluster_id)
    ON ps_cluster (cluster_id);

And then the search. This is the part that made me laugh when it worked, because it is a *vector similarity search* implemented with an XOR and a population count — an instruction that has been in CPUs since the 1960s and in T-SQL since 2022.

/*  Hamming distance on the binary signatures. BIT_COUNT(a ^ b)
    per 64-bit lane, summed. Six XORs and six POPCNTs per row,
    all of it integer work the CPU does at absurd speed, and all
    of it in the columnstore batch-mode engine.

    @probe0..5 are the query's own signature.
    @clusters is the nprobe nearest neighbourhoods, which is
    what makes the partition elimination happen.               */
SELECT TOP 100 s.speech_id,
       hamming = BIT_COUNT(s.sig0 ^ @probe0) + BIT_COUNT(s.sig1 ^ @probe1)
               + BIT_COUNT(s.sig2 ^ @probe2) + BIT_COUNT(s.sig3 ^ @probe3)
               + BIT_COUNT(s.sig4 ^ @probe4) + BIT_COUNT(s.sig5 ^ @probe5)
FROM dbo.speech_sig s
WHERE s.cluster_id IN (SELECT cluster_id FROM @clusters)
ORDER BY hamming;

Neither half can do the other’s job. The GPU cannot hold the data, cannot join it to a party affiliation, and cannot answer a question six months later without being handed everything again. The database cannot do the matrix arithmetic that works out where the neighbourhoods are in the first place.

The GPU builds the map. The database reads it. That is the entire architecture and it took me three attempts to see it that plainly.

The same thing, explained to a dachshund

My technical review process is a black-and-tan dachshund called Brian who has been in the house about a week.

Brian has two settings. In the first he is asleep, in a shape no animal with a spine should be able to achieve. In the second he has heard me draw breath to speak, and is therefore already airborne. He has never once let me finish a sentence. He has no opinion on partition elimination. He has strong opinions on my shoelaces.

He is, in other words, the perfect sounding board, because the entire value of explaining something out loud has nothing to do with whether the other party understands a word of it.

So. Imagine a library.

7 million books. No shelves. Just books, in a heap.


| What I built            | What it is in the library                                        |
| ----------------------- | ---------------------------------------------------------------- |
| The embedding           | Reading each book and writing down what it's *about* on a card   |
| k-means on the GPU      | Sorting all the cards into 41,730 piles of similar books         |
| The cluster id          | Writing a shelf number on every book                             |
| The ordered columnstore | Actually putting the books on the shelves, in shelf order        |
| The partitions          | Grouping the shelves into 64 rooms                               |
| The binary signature    | A tiny sticker on the spine — 48 bytes instead of the whole book |
| nprobe                  | "Only look in the three nearest rooms"                           |
| Hamming distance        | Comparing stickers instead of reading books                      |

I got as far as ”and then you only search the three nearest rooms” before I stopped, because I had heard it.

How do you know which rooms are nearest?

You don’t. Not in the version I had built. k-means numbers its clusters in whatever order it happens to produce them, so cluster 4 and cluster 39,000 could be about exactly the same thing and sit at opposite ends of the building. I had sorted the books onto shelves and then arranged the shelves at random. Every “nearby room” I was searching was nearby in numbering and nowhere near in meaning.

The fix was a second, much smaller k-means — over the 41,730 centroids themselves this time — grouping them into 64 super-clusters and renumbering so that topically adjacent piles get adjacent numbers. Spatial locality written into the physical layout of the table. It is the best idea in the project and it took ninety seconds to run.

Brian, during all of this, was attempting to kill a sock.

I want to be clear that he contributed nothing. He does not know what a vector is. He is a long, low, deeply stupid animal bred to go down holes after badgers, and the badgers were armed. But the obligation to say a thing out loud, in order, to something that is definitely not going to help, is the cheapest code review available and I have yet to find a better one.

If you take one engineering practice from this article, take the dog.

Attempt one: embeddings. Verdict, no.

The idea was obvious enough that I am slightly embarrassed by how long I believed in it.

If a minister adopts an opposition argument, the minister’s later speech should sit closer in vector space to that earlier opposition speech than you would otherwise expect. So: take every cross-party pair of speeches on the same subject, measure the cosine distance, bucket by how many months apart they are, and look for the gap closing.

/*  Mean cross-party distance against mean same-party distance,
    bucketed by lag. If opposition arguments get adopted, the
    cross-party line should fall relative to the same-party one
    as lag increases.                                          */
SELECT lag_months = DATEDIFF(month, a.speech_date, b.speech_date) / 6 * 6,
       cross_party = AVG(CASE WHEN a.party <> b.party
                        THEN VECTOR_DISTANCE('cosine', av.embedding, bv.embedding) END),
       same_party  = AVG(CASE WHEN a.party =  b.party
                        THEN VECTOR_DISTANCE('cosine', av.embedding, bv.embedding) END)
FROM #pairs p
JOIN dbo.speech_role a  ON a.speech_id = p.a_id
JOIN dbo.speech_role b  ON b.speech_id = p.b_id
JOIN dbo.speech_vec av  ON av.speech_id = p.a_id
JOIN dbo.speech_vec bv  ON bv.speech_id = p.b_id
GROUP BY DATEDIFF(month, a.speech_date, b.speech_date) / 6 * 6
ORDER BY 1;

The gap did not close. It widened, from +0.0124 to +0.0244, consistently, in every bucket. Cross-party pairs were further apart the longer you waited.

I assumed I had a bug. I did have several, but not that one. So I did the thing I should have done first, which was print the actual speeches and read them.

The closest cross-party pair in the entire corpus, at a cosine distance of 0.079, was David Heathcoat-Amory arguing against the fuel duty escalator, and Dawn Primarolo, eleven weeks later, defending it.

There it is. That is the whole failure in one row.

An embedding encodes what a document is about. It has no opinion about whether the author approved. Two people on opposite sides of the same question, using the same vocabulary, citing the same figures, are near-identical in vector space — and the more fiercely they disagree about something specific, the closer they get, because disagreement is topically narrow and agreement is vague.

So the method was never going to work, and no amount of tuning would have fixed it. I was asking a tool that measures subject matter to measure stance. It is like trying to find out who won a boxing match by weighing the boxers.

I could have discovered this on day one by reading six rows. I discovered it on day nine, after building a very fast index. Make of that what you will.

Attempt two: phrases. Verdict, also no

If embeddings cannot see stance, stop asking them to. Use words instead.

The logic is nicer than the embedding version. If a backbencher coins a distinctive six-word formulation in 1987, and a minister uses the same six words four years later, that is evidence — regardless of whether either of them approved of anything. You do not need a stance model. You need only to notice that the words moved.

So: hash every six-word window in all 1.7 million speeches, count them, and keep the rare-but-repeated ones.

# FNV-1a over the joined n-gram, pure Python, which is quite
# slow and completely fine at this scale.
def hash_ngrams(tokens, n):
    out = []
    for i in range(len(tokens) - n + 1):
        h = 0xcbf29ce484222325
        for w in tokens[i:i + n]:
            for b in w.encode():
                h ^= b
                h = (h * 0x100000001b3) & 0xFFFFFFFFFFFFFFFF
            h ^= 0x20                       # the space between words
            h = (h * 0x100000001b3) & 0xFFFFFFFFFFFFFFFF
        out.append(h)
    return out

That produced 375,802,591 six-grams, of which 319,572,719 were distinct.

Stop and look at that for a second, because it is the best statistic in this article and I found it by accident. Eighty-five per cent of every six consecutive words ever spoken in the House of Commons since 1979 was spoken exactly once and never again. For an institution with a reputation for saying the same thing over and over, that is a remarkable amount of novelty. Four decades of people, in a room, mostly not repeating each other.

Only 3,813,616 fell in the useful band — said between four and three hundred times. Rare enough to be distinctive, common enough that reuse is not a coincidence.

The burstiness filter, which is the only clever thing here

Most of those 3.8 million are procedural furniture. “I beg to move that the Bill” appears constantly and means nothing. The thing that separates a coined phrase from a recurring one is its shape in time: boilerplate spreads evenly across forty years, a coinage arrives, spikes, and fades.

# Biggest share of a phrase's uses falling in any single year.
# Boilerplate scores low. A coinage scores high.
_, year_counts = np.unique(years[ids], return_counts=True)
burst = year_counts.max() / total_uses
if burst < 0.35:
    continue

That took 3.8 million down to 1,086,386, of which 735,073 had a different speaker at each end.

And the answer

/*  The control arm is the whole point. Without it, "2.23% of
    opposition coinages reach a minister" sounds like a finding.
    With it, you can see what it is.                           */
WITH coined AS (
    SELECT coiner_side, phrases = COUNT(*)
    FROM dbo.phrase_attr GROUP BY coiner_side
),
crossed AS (
    SELECT coiner_side, to_minister = COUNT(*)
    FROM dbo.phrase_attr
    WHERE later_role = 'minister' AND lag_days > 30
    GROUP BY coiner_side
)
SELECT c.coiner_side, coined = c.phrases,
       reached_a_minister = x.to_minister,
       pct = CAST(100.0 * x.to_minister / c.phrases AS DECIMAL(5,2))
FROM coined c JOIN crossed x ON x.coiner_side = c.coiner_side;

| Coined by  | Coinages | Reached a minister | Rate      |
| ---------- | -------- | ------------------ | --------- |
| Opposition | 345,948  | 7,705              | 2.23%     |
| Government | 389,125  | 14,907             | 3.83%     |

Ministers are 1.7 times more likely to pick up language coined on their own side.

So language does diffuse through the House — but it diffuses downhill, from the frontbench outwards, not across the floor. The opposition is not shaping government vocabulary. The government is shaping its own.

How much of this should you believe

Less than I would like, and I want to say so before somebody checks.

I pulled a random thirty from the cleaned candidate set and read them. About a third are genuine rhetorical constructions — ”it has shone a spotlight on”, ”in the dying days of his”, ”warm words at the beginning of”, ”we have little to show for”. People choosing words.

The other two thirds are still furniture: ”copy is being placed in the”, ”meeting with the financial secretary to”, ”health and safety executive for a”. Procedural language that survived four filters because it happens to burst in a particular year for reasons that have nothing to do with anybody coining anything.

So: a third signal, two thirds noise. That is enough for a candidate list a human reads and judges, which is what I built it for. It is nowhere near enough to say “this particular MP coined this particular phrase” in print.

The 2.23% against 3.83% survives that, because the noise is symmetric — procedural junk is distributed across both sides of the House in roughly equal measure, so it cancels in a ratio even while it ruins the individual rows. A comparison can be sound while every single line of its input is rubbish. That is a property of comparisons that I think more people should find reassuring than do.

Two methods. Two no’s. One blind to stance, one able to see exactly which words moved. Same answer.

A parade of my own stupidity

I promised you the hundred-per-cent-funny bug. Here it is, along with its friends, roughly in order of how long I stared at each one before the penny dropped.

1. Every MP in Britain is funny all of the time

Hansard marks audience reaction inline: `[Laughter.]`, `[Interruption.]`, `[Hon. Members: “Rubbish!”]`. Lovely. Let us count them.

-- This returns 1,700,647. Every speech in the corpus.
SELECT COUNT(*) FROM dbo.speech
WHERE body LIKE '%[Interruption.]%';

Square brackets are LIKE wildcards. `’%[Interruption.]%’` does not ask for that string. It asks for *any single character from the set* `{I, n, t, e, r, u, p, o, .}`, with anything either side.

Every speech in Hansard contains the letter “e”.

So I produced a leaderboard showing that every single Member of Parliament makes the House laugh in 100.00% of their speeches, and — this is the part I enjoy — I very nearly believed it for a moment, because the House of Commons is exactly the sort of institution where that might be true.

-- CHARINDEX does no pattern matching. 39,163 speeches.
SELECT COUNT(*) FROM dbo.speech
WHERE CHARINDEX('[Interruption.]', body) > 0;

Thirty years of writing SQL. Beaten by a bracket.

2. Tony Banks, Conservative

Every leaderboard grouped by speaker and took `MIN(party)` to label them, because a speaker has one party and `MIN` of one value is that value. Except the dataset has the odd mislabelled row, and `MIN` on a string is alphabetical.

`’Conservative’ < ‘Labour’`.

So one bad row anywhere in a career flips the whole career. Tony Banks — Labour member for Newham North West, one of the funniest men ever to stand up in that chamber — came out of my analysis as a Conservative. So did Mr. Smith. So did Mr. Brown. I had accidentally built a swingometer that only swings one way.

/*  Modal party, not minimum. Should have been obvious.  */
WITH pc AS (
    SELECT speaker_name, party,
           rn = ROW_NUMBER() OVER (PARTITION BY speaker_name
                                   ORDER BY COUNT(*) DESC, party)
    FROM dbo.speech_role
    WHERE role <> 'chair' AND party IS NOT NULL
    GROUP BY speaker_name, party
)
SELECT speaker_name, main_party = party FROM pc WHERE rn = 1;

3. A filter that rewarded phrases for containing their own date

The burstiness filter scores how concentrated a phrase’s uses are in a single year. Beautiful idea. It put this at the top:

> order 1995 which was laid before — burstiness 0.870

Of course it did. The year is inside the phrase. Statutory instrument boilerplate scores perfectly on a time-concentration metric because the text contains the time. I built a filter that rewarded phrases for being self-dating and then congratulated myself on the elegance of the method.

One line fixes it:

AND p.phrase NOT LIKE '%19[0-9][0-9]%'
AND p.phrase NOT LIKE '%20[0-9][0-9]%'

(Note the square brackets. Here they are the wildcards I want. Consistency is not SQL’s strong suit.)

4. Peter Bone, the most prolific phrase-coiner in parliamentary history

The top forty “most concentrated coinages” contained eighteen entries from one man on one day in 2009:

the recommendations of the badman report about the recommendations of the badman recommendations of the badman report which of the badman report which suggests badman report which suggests closer monitoring

Peter Bone read a petition about home education into the record. The same petition. 144 times. A six-word window sliding along one piece of text produces dozens of overlapping “phrases”, every one of which bursts identically.

I had not discovered a coinage. I had discovered a man with a very long petition and discovered it eighteen times.

/*  All shingles of one source share both endpoints. One row
    per (first_speech, last_speech) collapses the petition back
    to the single event it always was.                         */
, deshingled AS (
    SELECT *, rn = ROW_NUMBER() OVER (
                   PARTITION BY first_speech, last_speech
                   ORDER BY uses DESC, burstiness DESC)
    FROM filtered
)
SELECT * INTO dbo.phrase_clean FROM deshingled WHERE rn = 1;

5. Theresa May, influencer of nobody

The coiner leaderboard showed Theresa May with 2,808 phrases coined and zero reaching a minister. David Cameron, 2,357 and zero. A striking finding about the hollowness of modern political language, I thought, for about four minutes.

The corpus ends in 2021. They coined late. Nobody had time to pick anything up.

If your metric has a boundary, somebody is standing on it.

And there is somebody standing on the other edge too, which took me a further day to see.

The per-party coinage table had the Liberals at 2,718 phrases per million words against the Conservatives’ 1,524. Nearly twice as inventive. A lovely little story about the creativity of third parties, which I was composing in my head when I thought to sort the whole column by something other than the number.

| Party            | Per million words | Years active in the corpus |
| ---------------- | ----------------- | -------------------------- |
| Liberal          | 2,718             | 1979–1988 only             |
| Ulster Unionist  | 1,895             | heavily early              |
| Labour           | 1,561             | throughout                 |
| Conservative     | 1,524             | throughout                 |
| Plaid Cymru      | 1,411             | throughout                 |
| DUP              | 1,069             | throughout                 |
| Liberal Democrat | 979               | post-1988                  |
| SNP              | 825               | mostly post-2015           |

That is not a ranking of linguistic invention. It is a ranking of who got here first.

“First use” in my data means first use in the corpus, and the corpus starts in 1979. Every phrase already in circulation before then — which is to say most of the English language — gets credited to whoever happens to say it earliest inside my window. The Liberal Party label stops existing at the 1988 merger, so every Liberal in this dataset is standing at the front of the queue being handed the collected works of several centuries of parliamentary cliche.

So I had built a metric with a boundary at each end. Arrive early and you are credited with inventing everything that was already there. Arrive late and you are recorded as having influenced nobody. Neither has anything to do with the MPs.

If your data has edges — and all data has edges — go and look at who is standing on them before you write a word.

6. Forty-two rows a second

The loader wrote data files at 42 rows per second. Eleven hours for a job that should take two minutes. I confidently diagnosed f-string formatting of numpy scalars. Wrong. I then confidently diagnosed memory pressure. Also wrong — Task Manager showed 58 GB free and the disk at 1%.

It was two things, neither of which I had considered:

`num_proc=8` on a HuggingFace dataset op. On Linux that forks. On Windows there is no fork, so Python respawns the whole interpreter per worker, per batch.

`shuffle()` leaves an indices map behind, so every subsequent read is a random seek into an Arrow file instead of a sequential scan.

ds = ds.filter(lambda r: ...)   # NO num_proc. Not on Windows.
ds = ds.shuffle(seed=SEED)
ds = ds.flatten_indices()       # random seeks -> sequential scan

Two lines. The job went from eleven hours to 78 seconds. The text pass hit 165,000 rows per second. That is roughly four thousand times faster, achieved by deleting one argument and adding one call.

I had spent the previous hour profiling string formatting.

7. `SELECT INTO`, three separate times

Msg 8111: Cannot define PRIMARY KEY constraint on nullable column

`SELECT … INTO` discards nullability from the source column even when the source is `NOT NULL`. Every table you intend to key has to be declared explicitly. I know this. I have known this for twenty years. I did it three times in one project and was surprised on each occasion.

The fix I settled on is to stop using primary keys for things that are only ever unique lookups:

CREATE UNIQUE CLUSTERED INDEX ix_speaker_party
    ON dbo.speaker_party (speaker_name);

8. The leaderboard that was measuring itself

This one is the most instructive, so I will give it room.

I wanted to find MPs who sound more like the other side than their own. Score each member against a random sample of 400 Conservative and 400 Labour speeches, subtract, sort.

The output was perfect. All thirty at the top were Labour. All thirty at the bottom were Conservative. Beautifully clean separation.

Too clean. `to_con` was lower than `to_lab` for every single speaker in the House, including all the Labour ones.

There were 851,967 Conservative speeches in the corpus and 553,250 Labour ones, and the Conservatives spent more of the period in government, which means more subjects covered. The Conservative reference sample is simply topically broader, so everybody sits marginally nearer it. My score was `to_con — to_lab`, which therefore flips sign purely on party membership. I had not produced sixty findings about individual MPs. I had produced one property of my own sample, printed twice, in opposite orders.

The fix is to stop comparing against zero and start comparing against your own colleagues:

/*  z = how far this MP's gap sits from their OWN party's mean,
    in party standard deviations. Sign flipped for Labour so a
    high score means the same thing on both sides: further from
    my own lot than my colleagues are.                         */
WITH stats AS (
    SELECT own_party, m = AVG(to_con - to_lab),
                      sd = STDEV(to_con - to_lab)
    FROM #gap GROUP BY own_party
)
SELECT g.speaker_name, g.own_party,
       off_message_sd =
         CASE WHEN g.own_party = 'Conservative' THEN 1 ELSE -1 END
         * ((g.to_con - g.to_lab) - st.m) / NULLIF(st.sd, 0)
FROM #gap g JOIN stats st ON st.own_party = g.own_party
ORDER BY off_message_sd DESC;

If your results are suspiciously tidy, you are probably measuring your instrument.

What I found while failing at the thing I meant to do

Once the brackets were fixed, the reaction markers turned out to be real and plentiful: `[Interruption.]` in 39,163 speeches, “Interruption” in any form in 47,030, “Laughter” in 8,833. There is a genuine record of how the Commons behaves, sitting in the text, and as far as I can tell nobody has counted it.

Office does not protect you. It makes you a target.

SELECT role, speeches = COUNT(*),
       pct_laughter  = CAST(100.0 * SUM(laughter)  / COUNT(*) AS DECIMAL(6,3)),
       pct_interrupt = CAST(100.0 * SUM(interrupt) / COUNT(*) AS DECIMAL(6,3))
FROM #reaction GROUP BY role;

| Role       | Speeches  | Laughed at | Interrupted |
| ---------- | --------- | ---------- | ----------- |
| Backbencher| 1,565,532 | 0.530%     | 2.738%      |
| Minister   | 47,537    | 0.684%     | 4.222%      |

A minister at the despatch box is interrupted 54% more often than a backbencher, and laughed at 29% more. I had assumed the dignity of office bought a bit of quiet. It buys the opposite. The red box is a target, not a shield.

The House has stopped giving way

This is the one I would lead with if I were a journalist.

“Will the hon. Gentleman give way?” is the Commons’ interrupt protocol — the formal, courteous mechanism by which one member takes the floor from another mid-speech. It is the single most recognisable thing about British parliamentary debate.

| Decade | Interrupted | Gave way | Laughter | "Hear, hear" |
| ------ | ----------- | -------- | -------- | ------------ |
| 1979   | 1.315%      | 1.270%   | 0.316%   | 0.136%       |
| 1980s  | 2.692%      | 1.734%   | 0.385%   | 0.152%       |
| 1990s  | 2.867%      | 1.801%   | 0.784%   | 0.175%       |
| 2000s  | 3.342%      | 1.189%   | 0.619%   | 0.105%       |
| 2010s  | 2.667%      | 0.919%   | 0.447%   | 0.091%       |
| 2020s  | 1.851%      | 0.362%   | 0.367%   | 0.079%       |

Giving way peaked at 1.80% of speeches in the 1990s and now runs at 0.36%. A five-fold collapse.

The obvious explanation is the pandemic — you cannot intervene on somebody appearing by video link. But look at the 2010s: already at 0.919%, half the 1990s rate, years before anyone had heard of hybrid proceedings. Covid finished it off. It did not start it.

“Hear, hear” has halved over the same period. Interruption, meanwhile, peaked in the 2000s and is still running at roughly the 1980s rate.

So the picture is a chamber that still gets shouted at but has stopped *negotiating* interruption. The formal exchange is dying while the informal noise carries on. Speeches written for the clip rather than the room, by people who have no intention of letting anyone in.

Being interrupted is a measure of importance

I built the “most interrupted MPs” leaderboard expecting to find the chamber’s most irritating people. Here is the top of it:

| Member          | Party        | Speeches | Interrupted |
| --------------- | ------------ | -------- | ----------- |
| Neil Kinnock    | Labour       | 226      | 12.83%      |
| Ian Blackford   | SNP          | 1,125    | 12.53%      |
| Peter Dowd      | Labour       | 415      | 10.84%      |
| Rhodes Boyson   | Conservative | 222      | 10.36%      |
| Edward Miliband | Labour       | 1,822    | 9.88%       |
| George Osborne  | Conservative | 3,319    | 9.34%       |
| Tony Blair      | Labour       | 2,233    | 9.05%       |
| Keir Starmer    | Labour       | 984      | 8.54%       |
| Jeremy Corbyn   | Independent  | 2,646    | 8.31%       |
| Tony Benn       | Labour       | 569      | 8.26%       |

That is not an irritation ranking. It is a prominence ranking. Leaders of the opposition, chancellors, prime ministers, the people at the despatch box in prime time with the whole other side watching. Being interrupted a lot means you were worth interrupting.

Which, if you are an MP reading this, is probably the nicest thing a database has ever said about you.

Tony Blair does not sound like a Labour MP

Remember the broken leaderboard from the last section, the one where all the Labour members were at one end and all the Conservatives at the other, because I was measuring my own reference sample. Here is what came out once I centred it properly.

First, the thing the artefact was hiding:

| Own party    | MPs   | Mean gap | SD      |
| ------------ | ----- | -------- | ------- |
| Labour       | 1,001 | +0.00522 | 0.00204 |
| Conservative | 1,226 | +0.00345 | 0.00203 |

The two parties are separated by 0.87 standard deviations, which in any other field would be called a decent medium effect. An embedding model that has never been told what a political party is can tell Labour from Conservative purely from how they talk.

But the global offset I was accidentally measuring was +0.00425, which is two and a half times larger than the real party difference. The signal was there the whole time. It was buried under an artefact of my own sampling, and the broken version of the query was reporting the artefact with complete confidence and perfect internal consistency.

That is the lesson of this entire project in one table.

And now the fun part. The corrected list of MPs who sound less like their own side than their colleagues do:

| Member          | Party         | Off message (SD) |
| --------------- | ------------- | ---------------- |
| Hywel Francis   | Labour        | 4.07             |
| Ann Keen        | Labour        | 3.97             |
| Patricia Hewitt | Labour        | 2.60             |
| Dr. Liam Fox    | Conservative  | 2.41             |
| Tony Blair      | Labour        | 2.32             |
| David Willetts  | Conservative  | 2.32             |
| Eric Forth      | Conservative  | 2.33             |
| Theresa May     | Conservative  | 2.15             |

Tony Blair, 2.32 standard deviations off message. The most famously triangulating Labour leader in modern British history, identified as sounding less like Labour than his own backbenchers, by a method that has never heard of triangulation, New Labour, or the Third Way. It just noticed that the subjects he talked about and the way he framed them sat closer to the other lot than his colleagues’ did.

Patricia Hewitt came to Parliament from the IPPR and ran Trade and Industry. David Willetts is the Conservative who wrote a book arguing that his own generation had robbed the next one. Eric Forth spent his career being a nuisance to his own whips. The list is not random.

The other end is a cohort, not a character trait

The most on message MPs in forty years of Hansard:

Alan Mak. Nigel Huddleston. Amanda Solloway. Ben Howlett. Andy Carter. Mims Davies. Richard Holden. Greg Smith. Anthony Mangnall. Julia Lopez. Andrea Jenkyns. Wendy Morton. Craig Tracey. Amanda Milling. Edward Argar.

That is not a list of unusually loyal individuals. That is the 2015, 2017 and 2019 Conservative intakes, almost without exception. The handful of non-Conservatives who break it up — Frank Allaun, Michael Meacher, Roy Hattersley, Margaret Beckett — are old Labour’s most doctrinally reliable voices, which is its own kind of confirmation.

So the measure is tracking when you arrived, not what you are like.

There are two explanations and they make identical predictions. Either message discipline has genuinely professionalised and recent intakes really do sound more alike — which would be a finding worth having — or it is a sampling artefact, because thirty speeches from a four-year career cover a far narrower range of subjects than thirty from a thirty-year one.

Might do more of this digging in the next article.

Where is Margaret Thatcher?

I went looking for her about halfway through, expecting her to be everywhere. I could not find her, which turned out to be the answer.

She appears once in everything I ran — second place in the broken crossover leaderboard, the one that was measuring my own sampling. Fix the query and she vanishes entirely. Not in the most interrupted. Not in the funniest. Not in the most repetitive. The dominant parliamentary figure of the period, absent from every list.

Then I looked at the coiner table again:

| Coiner                 | Party        | Phrases   |
| ---------------------- | ------------ | --------- |
| Unknown                | —            | 18,422    |
| The Prime Minister     | Conservative | 6,149     |
| Mr. Clarke             | Conservative | 3,175     |

Second biggest coiner in four decades of Hansard, and it is not a person. It is a job.

The official report records who was speaking as what. When the Prime Minister answers at the despatch box, Hansard writes “The Prime Minister”, and for most of the 1980s it does not bother adding the name in brackets. Later convention does — you get `The Prime Minister (Mr. Tony Blair)` — but Thatcher largely predates that.

So eleven years of the most consequential speaker in modern British politics are filed under a job title, and every analysis in this article that groups by `speaker_name` has been quietly ignoring her. So, I would bet, has every other analysis anyone has published on this dataset, because the omission is completely invisible unless somebody asks you a question you cannot answer.

One `CASE` expression on the date fixes it, and opens a question nobody has been able to put to the record: which Prime Minister did the House shout at most?

holder = CASE
    WHEN speech_date < '1990-11-28' THEN 'Thatcher'
    WHEN speech_date < '1997-05-02' THEN 'Major'
    WHEN speech_date < '2007-06-27' THEN 'Blair'
    WHEN speech_date < '2010-05-11' THEN 'Brown'
    WHEN speech_date < '2016-07-13' THEN 'Cameron'
    WHEN speech_date < '2019-07-24' THEN 'May'
    ELSE 'Johnson' END

So I ran it.

Nine Margaret Thatchers, one of them male

First, the names. Here is every form the official report uses for the same woman:

The Prime Minister (Mrs. Marga...     372
Mrs. Thatcher                          32
Mrs. Margaret Thatcher                  3
The]'rime Minister (Mrs. Marga...       1
The. Prime Minister (Mrs. Marg...       1
The Prime Minister (Mr. Margar...       1     <--
The Prime Minister Mrs. Margar...       1
The Prime Minister (Mrs Margar...       1
The Prime Minister, (Mrs. Marg...       1

Nine spellings, most of them differing by a stray apostrophe, a misplaced full stop or a missing bracket. And on 16 January 1986, the official report of the United Kingdom Parliament recorded the Prime Minister as Mr. Margaret Thatcher.

That typo has been sitting in the public record for forty years. I found it at eleven o’clock at night because I went looking for a woman who was not where she should have been.

Twenty-four times a job title, once a person

Across 1979 to 2001 there are roughly 18,046 speeches filed under the bare string “The Prime Minister”, against 410 under any version of Thatcher’s own name.

And it is not a convention peculiar to her. The bare form runs at 700 to 900 a year straight through Major and into Blair — and then stops dead in 2001, after which Hansard records prime ministers by name like everybody else. So any comparison between prime ministers using this data covers 1979 to 2001 only, and Blair’s share of it is his first term.

I am spelling that out because it is exactly the sort of caveat that gets dropped when a number is interesting.

Nobody laughed at Margaret Thatcher

| Prime Minister    | Speeches | Interrupted | Laughter   | Avg words |
| ----------------- | -------- | ----------- | ---------- | --------- |
| Blair (1997–2001) | 3,547    | 7.75%       | 1.015%     | 147.1     |
| Thatcher          | 9,782    | 6.35%       | 0.174%     | 104.2     |
| Major             | 5,303    | 5.98%       | 0.811%     | 120.8     |

Raw, she draws a fifth of Blair’s laughter. But the House laughed more in the 1990s than the 1980s generally, so against the decade baselines:

- Thatcher: 0.45× her era’s laughter rate

- Major: 1.03×. Exactly average.

- Blair: 1.29× his

Even after correcting for the period, the Commons laughed at Thatcher less than half as much as it laughed at everyone else at the time — while interrupting her more than it interrupted Major. Shouted at, not laughed at. And she said it in short bursts: 104 words a speech against Blair’s 147.

I have no idea whether that is a fact about her or a fact about the House, and I am not sure the distinction survives contact with eleven years of her being Prime Minister.

The order paper

There is a story that belongs here, and it is my favourite thing anybody has ever written about her.

Christopher Hitchens — a man of the left who had been publicly and expensively rude about her in print — met Thatcher at a reception in the late seventies, when she was still Leader of the Opposition. They got into an argument about Rhodesia. He lost it. Having lost it, he bowed to her.

She told him that was not low enough. He bowed again, properly this time. And as she swept past she caught him across the backside with a rolled-up parliamentary order paper and said, ”Naughty boy.”

Hitchens then wrote, in print, about the frankly confusing effect this had on him. Which, for a man of his politics, took a kind of nerve that I am not sure I would have managed.

I mention it because of what happens when you put it next to the number above. The House laughed at her less than half as much as it laughed at anybody else of her era — and the most quoted personal anecdote about her is one where she makes a hostile left-wing journalist look ridiculous, physically, with a procedural document, in under ten seconds.

Those two facts are the same fact.

And there is a pleasing loop in it for this project: the argument Hitchens lost was about Rhodesia, the Lancaster House conference ran from September to December 1979, and the corpus starts in May. So whatever she said to him at that party, I can go and read what she was telling the House about it the same autumn.

The very first row of the very first query I ran in this entire project — the broken one, the one that was measuring my own sampling — was Frank Dobson, 25 July 1979, in a debate headed `SOUTHERN AFRICA`.

And then the best row in the project

at the end of the transition
  132 uses  ·  burstiness 0.697
  coined 1979-06-26, EUROPEAN COMMUNITY COUNCIL (STRASBOURG)
  last used by Robert Courts (Con), 41.7 years later

Margaret Thatcher said ”at the end of the transition” in June 1979, about Britain’s transitional arrangements on joining the European Community.

The same five words, forty-one years and eight months later, about Britain leaving it.

The burstiness of 0.697 means seven in ten of its 132 uses fall inside a single year, and that year is 2020. The phrase sat nearly dormant for four decades and then detonated.

She is not alone in the pattern. ”we will get a good deal” — 1984, picked up in 2019. ”the principles of democracy and…” — the Falklands, 1982, picked up in 2019. ”left to us by the labour…” — 1982, picked up by Theresa May in 2019. Her European vocabulary came back thirty-five years later in the mouths of people making the opposite argument.

The left-censoring caveat applies — she spoke early, so she collects first-mover credit on anything already in circulation. But that one is hers, the dates are in the record, and you can go and check it.

I spent a month trying to measure whether MPs change each other’s minds, and the most interesting thing the whole apparatus produced was a five-word phrase that outlived the argument it was coined for, turned round, and came back pointing the other way.

The best test in the project came from an idle thought

Somewhere around the fourth failed query I stopped building things and thought about the problem instead, and the question that came out of that is better shaped than anything the apparatus produced.

In the News at Ten era, a minister needed one line. One crafted sentence that would survive the cut and go out at ten o’clock. So you would expect them to find that line and repeat it. In a rolling news cycle you need constant fresh material instead, so repetition should fall.

And there is a dated event sitting in the middle of this corpus to test it against. Television cameras entered the House of Commons in November 1989. Radio had been there since 1978, but the chamber was not filmed until then — ten years into a forty-two year dataset.

That sets up the only within-subject design in this entire project. Take every MP with at least fifteen substantive speeches on each side of November 1989–790 of them — and measure each one’s self-similarity against themselves, before and after. Same people. Same metric. One event in between.

| Era     | MPs | Pairs  | Self-distance |
| ------- | --- | ------ | ------------- |
| Pre-TV  | 790 | 86,537 | 0.2600        |
| Post-TV | 790 | 86,766 | 0.2623        |

MPs became marginally more varied after the cameras arrived — the opposite direction to the prediction, and by an amount that looks small.

For scale: individual MPs in this corpus run from 0.2006 (Ralph Howell, the most repetitive man in forty years of Hansard) to 0.3398. That is a spread of 0.139, and the camera effect is 1.6% of it.

I could have published that as “no meaningful effect” and moved on, and it would have been true. Instead I did the thing I have spent this whole article telling you to do, and ran a placebo.

The placebo, which was the point all along

Same test, same code, break date moved to a year with nothing behind it. If an arbitrary Tuesday produces the same gap, the cameras are irrelevant.

| Break date              | Pre-window length | Pre    | Post   | Gap         |
| ----------------------- | ----------------- | ------ | ------ | ----------- |
| 1984 — nothing happened | 5 years           | 0.2544 | 0.2618 | +0.0074     |
| 1989 — the cameras      | 10.5 years        | 0.2600 | 0.2623 | +0.0023     |
| 1994 — nothing happened | 15 years          | 0.2622 | 0.2628 | +0.0006     |

A meaningless date produced three times the effect of the real one.

And then the actual answer, which is sitting in the pre-window column if you read it downwards:

- 5-year window → 0.2544

- 10.5-year window → 0.2600

- 15-year window → 0.2622

Monotonic. Self-distance rises with the length of time you watch somebody, and then flattens off. Which is obvious the moment you see it — observe an MP for five years and you catch them on fewer subjects than if you observe them for fifteen. All three post-TV figures cluster around 0.262 because every one of them is a long window sitting on the flat part of that curve, and the gap shrinks to nothing at 1994 precisely because that is where the two windows are closest to equal length.

There was no camera effect. There was no career-drift effect either. I had measured how long I was looking.

This is my favourite result in the article, and it is a null that nearly got away.

Every other test here compared different people to different people, which leaves all of them exposed to the thing that wrecked the crossover leaderboard — a property of the groups masquerading as a property of the members. This one compared 790 individuals to themselves, either side of a dated event. Best design in the project, large sample, clean execution.

And it still produced a number that meant nothing, and I would have believed it if I had not spent twenty minutes testing a date on which nothing happened.

That is the whole lesson. Not “check your work” — everybody says that and nobody means it. Run your analysis against a thing you know to be nothing, and see what it tells you. If it tells you something, your method is generating findings out of its own shape, and every result you have is suspect including the ones you like.

Mine was. Twice before this, and I only caught those by accident.

The honest numbers

The index works. 1,700,647 speeches, 41,730 clusters, 64 partitions, and a signature table of 87.9 MB standing in for 2.6 GB of raw vectors.

Recall at 10 against brute-force ground truth: 0.920. Bytes read: 681× fewer than the full scan.

That 681× is the number I would put on a slide. It is also the number I want to pull apart, because the breakdown is more useful than the headline and nobody ever publishes it.

| Where the saving comes from               | Factor  |
| ----------------------------------------- | ------- |
| Binary quantisation (1,536 bytes → 48)    | 32×     |
| Columnstore compression on the signatures | \~12×   |
| Rowgroup and partition elimination        | 1.8×    |

The clever bit — the ordered columnstore, the super-cluster renumbering, the partition scheme, the thing I spent most of the project on — contributes 1.8×. The boring bit, throwing away everything except the sign of each dimension, contributes 32×.

I am telling you this because the industry does not. Every vector index post you will read quotes an aggregate speedup and lets you assume it came from the architecture. Usually it came from the quantisation, which you could have had in an afternoon.

The experiment where I made it worse

I thought 64 partitions was too coarse, so I rebuilt at 512. More partitions, tighter elimination, fewer rows read. Obviously better.

Rows read fell from 845,000 to 351,000 — a 2.4× improvement, exactly as predicted.

LOB reads went up, from 968 to 1,048.

Small rowgroups carry fixed per-segment overhead. I had created four times as many segments, each one too small to amortise its own dictionary and header. I read fewer rows and more bytes. The query plan looked better and the disk worked harder.

I reverted to 64 and wrote it on a sticky note: measure bytes, not rows. The row count is what the plan shows you. The byte count is what you pay for.

Why none of this works in a vector database

Here is the constraint that started the whole design, and it is the sort of thing you only find by trying.

SQL Server 2025 will not let you put a columnstore index on a table carrying a vector index, or a vector index on a table with a columnstore. They are mutually exclusive. You get fast vector search, or you get the compression and batch-mode scan, and you do not get both on the same table.

My first reaction was that this was a product gap. My second, after a day of being annoyed once again with a feature that should be there but isn’t (I have met the people at Microsoft developing SQL Server and Microsoft Research and I can honestly say there wasn’t a better, more dedicated group of people I have worked with so if something isn’t in the product there is a good reason for it — having said that they have all mostly left so who knows what teasm is developing it these days ? ), was that it forced the only interesting architecture in the project.

Because I could not have both, I had to split them. The full-precision vectors live in one table with a vector index for the exact work. The 48-byte signatures live in a separate table, ordered and partitioned by cluster id, with a columnstore. The search is a funnel: scan the cheap table with integer arithmetic to get a few hundred candidates, then do exact `VECTOR_DISTANCE` on only those.

/*  Stage 1: the cheap table. Integer XOR + POPCNT, batch mode,
    one partition. Narrows 1.7 million to 500.
    Stage 2: exact cosine on 500 rows. Costs nothing.         */
WITH candidates AS (
    SELECT TOP 500 s.speech_id,
           h = BIT_COUNT(s.sig0 ^ @p0) + BIT_COUNT(s.sig1 ^ @p1)
             + BIT_COUNT(s.sig2 ^ @p2) + BIT_COUNT(s.sig3 ^ @p3)
             + BIT_COUNT(s.sig4 ^ @p4) + BIT_COUNT(s.sig5 ^ @p5)
    FROM dbo.speech_sig s
    WHERE s.cluster_id IN (SELECT cluster_id FROM @probe)
    ORDER BY h
)
SELECT TOP 10 c.speech_id, r.speaker_name, r.party, r.speech_date,
       d = VECTOR_DISTANCE('cosine', v.embedding, @query)
FROM candidates c
JOIN dbo.speech_vec  v ON v.speech_id = c.speech_id
JOIN dbo.speech_role r ON r.speech_id = c.speech_id
WHERE r.party = 'Labour' AND r.role = 'minister'   -- try this in Pinecone
ORDER BY d;

Look at that `WHERE` clause on the last line, because it is the argument for the whole approach.

A dedicated vector database gives you an approximate nearest-neighbour search and, if you are lucky, metadata filtering that was bolted on afterwards and falls over when the filter is selective. What it does not give you is the ability to say “nearest neighbours, but only Labour ministers, joined to the division lists, grouped by decade, with a window function over the result” — and have the optimiser work out that it should apply the party filter first.

I did not want a vector database. I wanted a database, that happens to do vectors, so that the interesting half of every question — who, when, which side, how often — is a join rather than a second system.

The restriction I resented for a day turned out to be the design.

And then I went looking for Tony Benn

I had the index built, the pipeline run, two negative results written up and the article mostly drafted. And then I went back to a question I had been carrying around since before any of this started, for no better reason than that I have always admired the man: what does the data say about Tony Benn?

First problem: nothing, because he was split across three names. The leaderboards had “Mr. Tony Benn” at 569 speeches. Resolve the forms properly and he is at 4,814 speeches, 1.4 million words, roughly 180 hours at the despatch box. The interruption rate I had already put in a table was drawn from a tenth of him.

So I tested the reputation properly.

|              | Benn   | The House, 1979–2001 |
| ------------ | ------ | -------------------- |
| Interrupted  | 2.825% | 2.664%               |
| Laughed at   | 0.457% | 0.571%               |
| "Hear, hear" | 0.125% | 0.151%               |
| Gave way     | 2.804% | 1.652%               |

Interrupted at almost exactly the House average. Laughed at rather less. And he gave way seventy per cent more often than anybody else — far and away the largest deviation in the table.

That is the reputation, and it does not show up as being shouted at. It shows up as letting people in. In a chamber that spent thirty years quietly abandoning the practice, he did more of it than almost anyone.

I had also predicted he would be a broken record — the man with four themes and forty years. Wrong. Self-distance 0.2695 against a Labour average of 0.2684. Completely ordinary range.

The five questions

Benn’s five questions are the most famous formulation any backbencher produced in the period. He put them to the House repeatedly across eleven years:

”What power have you got? Where did you get it from? In whose interests do you exercise it? To whom are you accountable? And how can we get rid of you?”

So I searched for them, mostly as a sanity check on the pipeline. Easiest possible case. I knew the answer in advance.

They are in the corpus. And so is this:

Lee Rowley (CONSERVATIVE), 14 October 2019, DEBATE ON THE ADDRESS

  "What power have you got? Where did you get it from? In whose
   interests do you exercise it? To whom are you accountable?
   And how can we get rid of you?"

   - [Official Report, 22 March 2001; Vol. 365, c. 510.]

  "I hope we remember that in the days ahead."

A Conservative MP. In the Queen’s Speech debate. At the height of the Brexit crisis. Reading a hard-left Labour backbencher’s words into the record with the column reference, citing a speech Benn gave eighteen years earlier on the election of a Speaker. Rowley sits for North East Derbyshire, which wraps around Benn’s Chesterfield.

He is not alone. Clive Lewis used the full structure against academy trusts in 2015 — not quoting it, wielding it. Toby Perkins, the sitting member for Chesterfield, read them out on 20 March 2014, six days after Benn died.

That is a formulation coined on the opposition benches, crossing the floor, and arriving in the mouth of a government MP eighteen years later.

It is precisely, exactly the thing I built a GPU index to find. The index failed to find it twice. I found it by typing a man’s name into a search box because I admired him.

Why 375 million six-grams could not see it

This is the part that should worry anyone doing this kind of work. Here is how Benn opened the questions on each occasion:

| Year | Opening                                                   |
| ---- | --------------------------------------------------------- |
| 1990 | "What power have you got? Where did you get it?"          |
| 1991 | "What power do you have? Where did you get it?"           |
| 1998 | "what power do you have; where did you get it"            |
| 2001 | "What power have you got? Where did you get it from?"     |

He never said it the same way twice. Not once in eleven years.

A six-gram hash requires an exact match. Every variant appears once or twice, below my floor of four, so each was discarded as noise before it ever reached the burstiness filter. The single most quotable thing anybody said in that chamber in forty years was invisible to a method that processed 375,802,591 six-word sequences on a graphics card.

And that explains the junk. It explains all of it.

The only language said identically enough to survive exact n-gram counting is language nobody is choosing their words for. Statutory instrument boilerplate. Petitions read verbatim. Procedural furniture. My candidate list was two-thirds rubbish not because my filters were weak, but because furniture is the only thing that holds still long enough to be counted.

Real rhetoric mutates. That is what makes it rhetoric. My method was, by construction, a detector for the absence of rhetoric, and I ran it for forty minutes on a 5090 to discover that.

Footpaths

One last thing from the Benn data, because it is the funniest result in the project and it is also, I think, the truest.

His most distinctive vocabulary, measured against the House:

footpath   177x    stockport   399 uses    promoters  265 uses
tameside   170x    footpaths   225 uses    carry-over  86 uses
walkers     99x    ramblers     93 uses    prerogative 164 uses

Stockport, 399 times. Footpaths, 225. Ramblers, 93.

The man remembered for the royal prerogative and the sovereignty of Parliament spent an enormous share of his career on Private Bills — Stockport Corporation, Tameside, Felixstowe Dock, Milford Haven, Pembrokeshire — objecting, forcing divisions, grinding through “promoters” and “carry-over” motions about rights of way in Greater Manchester.

Because that is where the constitutional principle actually bit. The grand questions about power got asked in committee rooms, about footpaths.

His nearest neighbours in vector space are not the Labour left. They are Hugh D. Brown, Cowans, Barry Porter, Eleanor Laing — obscure procedural specialists of both parties. The other people who spent their lives on Private Bills. The embeddings cannot see what anybody believed, but they put him in exactly the right room.

What I would tell you before you try this

Read the rows before you build the index. I spent nine days building a very fast way to get a wrong answer. Six printed speeches would have told me on day one that embeddings cannot see stance. Print the data. Print it early. Print it when you think you already know what it says.

Always build the control arm. “2.23% of opposition coinages reach a minister” is a finding, a headline, and a plausible story about the health of parliamentary democracy. Next to 3.83% for the government side, it is the opposite story. The comparison is not a nice-to-have, it is the entire analysis. If you cannot think of a control, you do not yet have a question.

Then run it against a date on which nothing happened. This is the control arm’s harder older brother and it is the single most useful thing in this article. Take your analysis, point it at something you know to be meaningless, and look at what comes back. When I tested whether television changed how MPs speak, the real date gave me a result — and an arbitrary year five years earlier gave me three times as much of it. The method was manufacturing findings out of its own shape, and nothing short of a placebo would have shown me that. If a nothing produces a something, every result you own is suspect, including the ones you are fond of.

Measure bytes, not rows. The query plan shows you rows because rows are easy to count. You pay in bytes.

Suspiciously tidy output is a bug. Clean separation, round numbers, every member of one group above every member of another — that is almost never nature. It is usually your instrument.

If your metric has an edge, somebody is standing on it. Theresa May did not fail to influence anyone. The data stopped.

Check what your string functions actually mean. `LIKE` is a pattern language, not a search. Thirty years in and a square bracket still had me.

On Windows, `num_proc` is a trap. No fork means respawning the interpreter. The default is almost always faster.

Get the whole thing into the database before you get clever. Every analysis in this article is a join away from speaker, party, date and office. That is only true because loading it properly was the first thing I did and not the last.

The code

All of it — the loader, the GPU clustering, the renumbering, the partition scheme, the phrase pipeline, and every query in this article including the broken versions will be here:

(link to follow)

I have left the bugs in, in their own files, with comments explaining what each one did. The square bracket one especially. If you are going to learn anything from this it will be from those rather than from the version that works.

You will need SQL Server 2025 for `VECTOR_DISTANCE`, SQL Server 2022 or later for `BIT_COUNT`, and a CUDA card for the clustering. It does not have to be a 5090. It just has to not be a CPU.

One last thing

I set out to prove that parliamentary debate changes minds.

The embeddings said no, because they cannot see stance. The phrases said no, and then it turned out they could only ever have said no, because exact six-grams detect the absence of rhetoric rather than its presence. The one experiment with a decent design — the same 790 MPs, before and after the cameras — said no, and the placebo proved that even that no was a measurement of my own window length.

Three methods. Four hundred lines of SQL. Three hundred and seventy-five million six-word sequences. Two graphics cards that are somehow worth more than I paid for them. All of it, no.

And then I went looking for Tony Benn, because I admire the man and always have, and a Conservative MP turned out to have read his five questions into the Queen’s Speech debate of 2019 with the Hansard column reference attached.

I want to be careful about the moral here, because the obvious one is wrong. The obvious one is “the computer failed and a human succeeded”, and that is not what happened. Nobody reads 1.7 million speeches. The reason I could check Benn’s five questions in about ninety seconds, resolve him across three name forms, measure his giving-way rate against the whole House and find his nearest neighbours in vector space, is that a month of GPU work and columnstore design had already put the corpus in a state where a question like that costs nothing to ask.

The index did not find the answer. The index made the answer cheap to look for, so that an idle question cost nothing and I went ahead and asked it. Those are different jobs, and I had quietly assumed they were the same one.

So here is what I actually learned, and it is not about Parliament.

Build the thing properly. Load it all, index it where needed, join it to everything you know about who said it and when. Then stop expecting the apparatus to produce the question. It will not. It will sit there, extremely fast, waiting. The question comes from somewhere else entirely — in my case from a man I have admired since long before I owned a graphics card.

The graphics cards are still worth more than I paid for them. Thirty years of hardware, and the only appreciating asset I have ever owned spent a fortnight sorting 1.7 million speeches into neighbourhoods so that one evening in October I could put an idle question to it about a politician I admire and have the answer inside a minute.

That is a fair trade. And nobody on the News at Ten would have had the faintest idea what I was talking about, which Brian, in fairness, worked out on his first afternoon.

Brian — My sounding board

If you enjoyed what must have been an ordeal but hopefully you got this far please give it as many claps as you would if you were in the House of Commons.

I’m 53, She is 33 And Wants A Baby, So, Meet Brian The Sausage Dog. (SQL Animation is a bonus)

Sat, 03 Oct 2026 22:52:50 GMT

I struggle where to start with this one but I think I can make it readable, enjoyable and a template for anyone else with a younger partner and of course some never seen before SQL techniques.

Brian, the Dachshund, Created using SQL Server Spatial

I wont beat around the bush, she wants a baby (33), I’m 53 and a dead man walking, so we compromised. We got a dachshund and named him Brian.

Brian is evil reincarnated but is also the cutest little sausage dog you can imagine. We literally get stopped everywhere with people asking about him. When I’m walking him, well I say walking it really is me carrying him to the various green areas near us because he already has a list of things he does and doesn’t like. This is after 4 days.

Rule 1 — he doesn’t come into my office

We (Brian and me) sat watching Sausage Dog videos for 2 hours yesterday in my office.

Rule 2 — he doesn’t sleep on the bed or even go near the bed

Just look at his eyes — he knows what he is up to. Yes this is my side of the bed.

Rule 3 — he has to be trained properly

As you can imagine when you have two languages spoken in the house that should be one language spoken to the dog. Sit, Come, Stay etc…. but no its all sorts of Polish, mixed with English (I think) and never just the one word, multiple words to describe the thing he is meant to understand, spoken in Polish very fast. His training is not going well.

Moving on

I wanted to write a SQL Server spatial story today and I did, but the end of the spatial stuff I thought to myself, this is all very important, looks actually useful and business relatable. I simply cannot leave the demo code and application just like that, I have to put a swing on it.

So I did what any respectable SQL developer would do when he realised he couldn’t end the weekend doing this:

RETURN geometry::STGeomFromText(sx.JitterWkt(@g.AsTextZM(), @amp, @seed), @g.STSrid)

I wrote a cartoon called “The Dachshund” starring Brian, using SQL Server 2025 Spatial and the Geometry datatype. Now you might be thinking, yes we have seen Christmas Trees rendered in SQL, we have seen Trevor’s head rendered in SQL, we even have Doom running inside SQL Server. But this will be a full animated cartoon coming straight from the SQL Engine.

Here is the code:

/* ==========================================================================
   Spatial Studio - 07 Toon: cut-out animation inside SQL Server
   Characters are puppets made of geometry parts. Each part hangs off a parent
   at a pivot, like the card cut-outs of 1970s children's TV. Keyframes are
   rows; toon.Render interpolates them, composes the affine transforms down the
   puppet with a recursive CTE, applies them with sx.Affine and adds a felt-tip
   "boil" with sx.Jitter. Starring Brian, a black-and-tan dachshund.
   ========================================================================== */
USE SpatialStudio;
GO
IF SCHEMA_ID(N'toon') IS NULL EXEC (N'CREATE SCHEMA toon AUTHORIZATION dbo;');
GO

/* ---- tables ---- */
DROP TABLE IF EXISTS toon.Frame, toon.[Key], toon.Casting, toon.Part, toon.Scene;
GO
CREATE TABLE toon.Scene
(
    Scene  VARCHAR(40)   NOT NULL CONSTRAINT PK_ToonScene PRIMARY KEY,
    Title  NVARCHAR(100) NOT NULL,
    Frames INT           NOT NULL,
    Fps    INT           NOT NULL DEFAULT 12
);
CREATE TABLE toon.Part
(
    Character VARCHAR(30)  NOT NULL,
    Part      VARCHAR(30)  NOT NULL,
    Parent    VARCHAR(30)  NULL,           -- hangs off this part of the same character
    PivotX    FLOAT        NOT NULL,       -- the drawing pin: rotations happen about this point
    PivotY    FLOAT        NOT NULL,
    Z         INT          NOT NULL,       -- drawing order, low first
    Color     VARCHAR(30)  NOT NULL,
    Shape     GEOMETRY     NOT NULL,       -- rest pose, in the character's own coordinates
    OscPeriod FLOAT        NOT NULL DEFAULT 8,   -- frames per cycle for the 'osc' (swing) and 'bob' channels
    OscPhase  FLOAT        NOT NULL DEFAULT 0,   -- degrees
    Boil      FLOAT        NOT NULL DEFAULT 1,   -- multiplier on the felt-tip wobble
    Label     NVARCHAR(60) NULL,
    FontSize  INT          NULL,
    LineWidth FLOAT        NULL,
    CONSTRAINT PK_ToonPart PRIMARY KEY (Character, Part)
);
CREATE TABLE toon.Casting
(
    Scene     VARCHAR(40) NOT NULL,
    Character VARCHAR(30) NOT NULL,
    CONSTRAINT PK_ToonCasting PRIMARY KEY (Scene, Character)
);
/* Channels: angle (degrees, anticlockwise), dx, dy (move), sx (1 = as drawn, -1 = flipped),
   osc (swing amplitude in degrees, cycling with OscPeriod), bob (hop height), vis (1 shown, 0 hidden).
   Ease on a key says how we arrive at it: smooth, linear or step.                                 */
CREATE TABLE toon.[Key]
(
    Scene     VARCHAR(40) NOT NULL,
    Character VARCHAR(30) NOT NULL,
    Part      VARCHAR(30) NOT NULL,
    Channel   VARCHAR(10) NOT NULL CHECK (Channel IN ('angle', 'dx', 'dy', 'sx', 'osc', 'bob', 'vis')),
    Frame     INT         NOT NULL,
    Value     FLOAT       NOT NULL,
    Ease      VARCHAR(10) NOT NULL DEFAULT 'smooth' CHECK (Ease IN ('smooth', 'linear', 'step')),
    CONSTRAINT PK_ToonKey PRIMARY KEY (Scene, Character, Part, Channel, Frame)
);
CREATE TABLE toon.Frame
(
    Scene     VARCHAR(40)  NOT NULL,
    Frame     INT          NOT NULL,
    Character VARCHAR(30)  NOT NULL,
    Part      VARCHAR(30)  NOT NULL,
    Z         INT          NOT NULL,
    Shape     GEOMETRY     NOT NULL,
    Color     VARCHAR(30)  NOT NULL,
    Label     NVARCHAR(60) NULL,
    FontSize  INT          NULL,
    LineWidth FLOAT        NULL,
    CONSTRAINT PK_ToonFrame PRIMARY KEY (Scene, Frame, Character, Part)
);
GO

/* ---- the boil: nudge every vertex by a hash of its own coordinates ----
   The same input point always gets the same nudge, so closed rings stay closed. */
CREATE OR ALTER FUNCTION sx.JitterWkt (@wkt NVARCHAR(MAX), @amp FLOAT, @seed INT)
RETURNS NVARCHAR(MAX)
AS
BEGIN
    DECLARE @out NVARCHAR(MAX);
    SELECT @out = STRING_AGG(CAST(p.Piece AS NVARCHAR(MAX)), N',') WITHIN GROUP (ORDER BY s.ordinal)
    FROM STRING_SPLIT(@wkt, N',', 1) AS s
    CROSS APPLY (SELECT L = DATALENGTH(s.value) / 2, lp = CHARINDEX(N'(', REVERSE(s.value))) AS a
    CROSS APPLY (SELECT Prefix = CASE WHEN a.lp = 0 THEN CAST(N'' AS NVARCHAR(MAX)) ELSE LEFT(s.value, a.L - a.lp + 1) END) AS b
    CROSS APPLY (SELECT Rest = SUBSTRING(s.value, DATALENGTH(b.Prefix) / 2 + 1, a.L)) AS c
    CROSS APPLY (SELECT rp = CHARINDEX(N')', c.Rest)) AS d
    CROSS APPLY (SELECT Coord  = CASE WHEN d.rp = 0 THEN c.Rest ELSE LEFT(c.Rest, d.rp - 1) END,
                        Suffix = CASE WHEN d.rp = 0 THEN N'' ELSE SUBSTRING(c.Rest, d.rp, a.L) END) AS e
    CROSS APPLY (SELECT Lead = CASE WHEN LEFT(e.Coord, 1) = N' ' THEN N' ' ELSE N'' END,
                        Ct   = LTRIM(RTRIM(e.Coord))) AS f
    CROSS APPLY (SELECT sp1 = CHARINDEX(N' ', f.Ct)) AS g
    CROSS APPLY (SELECT Xs     = CASE WHEN g.sp1 > 0 THEN LEFT(f.Ct, g.sp1 - 1) END,
                        AfterX = CASE WHEN g.sp1 > 0 THEN SUBSTRING(f.Ct, g.sp1 + 1, a.L) END) AS h
    CROSS APPLY (SELECT sp2 = CHARINDEX(N' ', h.AfterX)) AS i
    CROSS APPLY (SELECT Ys    = CASE WHEN i.sp2 > 0 THEN LEFT(h.AfterX, i.sp2 - 1) ELSE h.AfterX END,
                        Extra = CASE WHEN i.sp2 > 0 THEN SUBSTRING(h.AfterX, i.sp2, a.L) ELSE N'' END) AS j
    CROSS APPLY (SELECT X = TRY_CAST(h.Xs AS FLOAT), Y = TRY_CAST(j.Ys AS FLOAT)) AS k
    CROSS APPLY (SELECT NX = k.X + (sx.Hash01(ROUND(k.X, 3), ROUND(k.Y, 3), @seed) - 0.5) * 2 * @amp,
                        NY = k.Y + (sx.Hash01(ROUND(k.Y, 3), ROUND(k.X, 3), @seed + 17) - 0.5) * 2 * @amp) AS m
    CROSS APPLY (SELECT Piece = CASE WHEN k.X IS NULL OR k.Y IS NULL THEN s.value
                                     ELSE b.Prefix + f.Lead + sx.FormatNum(m.NX) + N' ' + sx.FormatNum(m.NY) + j.Extra + e.Suffix END) AS p;
    RETURN @out;
END;
GO

CREATE OR ALTER FUNCTION sx.Jitter (@g GEOMETRY, @amp FLOAT, @seed INT)
RETURNS GEOMETRY
AS
BEGIN
    IF @g IS NULL OR @amp IS NULL OR @amp = 0 OR @g.STIsEmpty() = 1 RETURN @g;
    RETURN geometry::STGeomFromText(sx.JitterWkt(@g.AsTextZM(), @amp, @seed), @g.STSrid);
END;
GO

/* ---- the renderer ---- */
CREATE OR ALTER PROCEDURE toon.Render
    @Scene     VARCHAR(40),
    @FromFrame INT   = 0,
    @ToFrame   INT   = NULL,
    @Boil      FLOAT = 0.7,     -- felt-tip wobble in drawing units; 0 for a clean line
    @Store     BIT   = 1        -- 1: write toon.Frame; 0: return the rows
AS
BEGIN
    SET NOCOUNT ON;
    DECLARE @t0 DATETIME2 = SYSDATETIME();
    IF @ToFrame IS NULL SELECT @ToFrame = Frames - 1 FROM toon.Scene WHERE Scene = @Scene;
    IF @ToFrame IS NULL THROW 50001, 'Unknown scene - add it to toon.Scene first.', 1;

    DROP TABLE IF EXISTS #f, #ch, #loc, #world;
    SELECT Frame = value INTO #f FROM GENERATE_SERIES(@FromFrame, @ToFrame);

    -- 1. every keyed channel, interpolated for every frame
    SELECT f.Frame, c.Character, c.Part, c.Channel,
           Value = CASE WHEN k0.Frame IS NULL THEN k1.Value
                        WHEN k1.Frame IS NULL OR k1.Frame = k0.Frame THEN k0.Value
                        ELSE k0.Value + (k1.Value - k0.Value) *
                             CASE k1.Ease WHEN 'step' THEN 0.0 WHEN 'linear' THEN u.u ELSE u.u * u.u * (3 - 2 * u.u) END END
    INTO #ch
    FROM #f AS f
    CROSS JOIN (SELECT DISTINCT Character, Part, Channel FROM toon.[Key] WHERE Scene = @Scene) AS c
    OUTER APPLY (SELECT TOP (1) k.Frame, k.Value FROM toon.[Key] AS k
                 WHERE k.Scene = @Scene AND k.Character = c.Character AND k.Part = c.Part AND k.Channel = c.Channel AND k.Frame <= f.Frame
                 ORDER BY k.Frame DESC) AS k0
    OUTER APPLY (SELECT TOP (1) k.Frame, k.Value, k.Ease FROM toon.[Key] AS k
                 WHERE k.Scene = @Scene AND k.Character = c.Character AND k.Part = c.Part AND k.Channel = c.Channel AND k.Frame >= f.Frame
                 ORDER BY k.Frame) AS k1
    CROSS APPLY (SELECT u = CASE WHEN k0.Frame IS NULL OR k1.Frame IS NULL OR k1.Frame = k0.Frame THEN 0.0
                                 ELSE CAST(f.Frame - k0.Frame AS FLOAT) / (k1.Frame - k0.Frame) END) AS u;
    CREATE CLUSTERED INDEX cx ON #ch (Frame, Character, Part);

    -- 2. each part's local affine matrix: move, then flip and rotate about its pivot
    SELECT f.Frame, p.Character, p.Part, p.Parent, p.Z,
           la = m.a, lb = m.b, lc = p.PivotX + t.tx - (m.a * p.PivotX + m.b * p.PivotY),
           ld = m.d, le = m.e, lf = p.PivotY + t.ty - (m.d * p.PivotX + m.e * p.PivotY),
           vis = t.vis
    INTO #loc
    FROM #f AS f
    JOIN toon.Casting AS cs ON cs.Scene = @Scene
    JOIN toon.Part AS p ON p.Character = cs.Character
    OUTER APPLY (SELECT angle = MAX(CASE WHEN h.Channel = 'angle' THEN h.Value END),
                        dx    = MAX(CASE WHEN h.Channel = 'dx'    THEN h.Value END),
                        dy    = MAX(CASE WHEN h.Channel = 'dy'    THEN h.Value END),
                        sx    = MAX(CASE WHEN h.Channel = 'sx'    THEN h.Value END),
                        osc   = MAX(CASE WHEN h.Channel = 'osc'   THEN h.Value END),
                        bob   = MAX(CASE WHEN h.Channel = 'bob'   THEN h.Value END),
                        vis   = MAX(CASE WHEN h.Channel = 'vis'   THEN h.Value END)
                 FROM #ch AS h WHERE h.Frame = f.Frame AND h.Character = p.Character AND h.Part = p.Part) AS v
    CROSS APPLY (SELECT th  = RADIANS(ISNULL(v.angle, 0) + ISNULL(v.osc, 0) * SIN(2 * PI() * f.Frame / p.OscPeriod + RADIANS(p.OscPhase))),
                        tx  = ISNULL(v.dx, 0),
                        ty  = ISNULL(v.dy, 0) + ISNULL(v.bob, 0) * ABS(SIN(PI() * f.Frame / p.OscPeriod)),
                        s   = ISNULL(v.sx, 1),
                        vis = CAST(ISNULL(v.vis, 1) AS FLOAT)) AS t
    CROSS APPLY (SELECT a = t.s * COS(t.th), b = -t.s * SIN(t.th), d = SIN(t.th), e = COS(t.th)) AS m;

    -- 3. compose down the puppet: world = parent world x local (a recursive CTE per frame)
    WITH w AS
    (
        SELECT l.Frame, l.Character, l.Part, l.Z, a = l.la, b = l.lb, c = l.lc, d = l.ld, e = l.le, ff = l.lf, vis = l.vis
        FROM #loc AS l WHERE l.Parent IS NULL
        UNION ALL
        SELECT l.Frame, l.Character, l.Part, l.Z,
               w.a * l.la + w.b * l.ld,
               w.a * l.lb + w.b * l.le,
               w.a * l.lc + w.b * l.lf + w.c,
               w.d * l.la + w.e * l.ld,
               w.d * l.lb + w.e * l.le,
               w.d * l.lc + w.e * l.lf + w.ff,
               w.vis * l.vis
        FROM w JOIN #loc AS l ON l.Frame = w.Frame AND l.Character = w.Character AND l.Parent = w.Part
    )
    SELECT * INTO #world FROM w OPTION (MAXRECURSION 64);

    -- 4. draw: move every visible part, then boil it (the boil changes every other frame, four drawings, like felt-tip on cel)
    IF @Store = 1
    BEGIN
        DELETE toon.Frame WHERE Scene = @Scene AND Frame BETWEEN @FromFrame AND @ToFrame;
        INSERT toon.Frame (Scene, Frame, Character, Part, Z, Shape, Color, Label, FontSize, LineWidth)
        SELECT @Scene, w.Frame, w.Character, w.Part, w.Z,
               sx.Jitter(CASE WHEN w.a = 1 AND w.b = 0 AND w.c = 0 AND w.d = 0 AND w.e = 1 AND w.ff = 0 THEN p.Shape
                              ELSE sx.Affine(p.Shape, w.a, w.b, w.c, w.d, w.e, w.ff) END,
                         @Boil * p.Boil, (w.Frame / 2) % 4 + 1),
               p.Color, p.Label, p.FontSize, p.LineWidth
        FROM #world AS w JOIN toon.Part AS p ON p.Character = w.Character AND p.Part = w.Part
        WHERE w.vis > 0.5;
        SELECT scene = @Scene, frames = @ToFrame - @FromFrame + 1, [shapes drawn] = @@ROWCOUNT,
               [render ms] = DATEDIFF(MILLISECOND, @t0, SYSDATETIME());
    END
    ELSE
        SELECT frame = w.Frame, shape = sx.Jitter(sx.Affine(p.Shape, w.a, w.b, w.c, w.d, w.e, w.ff), @Boil * p.Boil, (w.Frame / 2) % 4 + 1),
               color = p.Color, label = p.Label, fontsize = p.FontSize, linewidth = p.LineWidth, opacity = 1.0, part = CONCAT(w.Character, '.', w.Part), z = w.Z
        FROM #world AS w JOIN toon.Part AS p ON p.Character = w.Character AND p.Part = w.Part
        WHERE w.vis > 0.5
        ORDER BY w.Frame, w.Z, w.Character, w.Part;
END;
GO

The Cast List

/* ---- the cast ---- */
INSERT toon.Part (Character, Part, Parent, PivotX, PivotY, Z, Color, Shape, OscPeriod, OscPhase, Boil, Label, FontSize, LineWidth) VALUES
(N'Park', N'Sky', NULL, 0, 0, 0, N'#bfe6f7', N'POLYGON ((0 0, 1000 0, 1000 450, 0 450, 0 0))', 8, 0, 0, NULL, NULL, 0),
(N'Park', N'Rays', NULL, 130, 380, 1, N'#ffe082', N'MULTIPOLYGON (((173.56 373.80, 196.00 380.00, 173.56 386.20, 173.56 373.80)), ((168.89 400.59, 183.40 418.79, 161.60 410.62, 168.89 400.59)), ((149.36 419.51, 150.40 442.77, 137.56 423.34, 149.36 419.51)), ((122.44 423.34, 109.60 442.77, 110.64 419.51, 122.44 423.34)), ((98.40 410.62, 76.60 418.79, 91.11 400.59, 98.40 410.62)), ((86.44 386.20, 64.00 380.00, 86.44 373.80, 86.44 386.20)), ((91.11 359.41, 76.60 341.21, 98.40 349.38, 91.11 359.41)), ((110.64 340.49, 109.60 317.23, 122.44 336.66, 110.64 340.49)), ((137.56 336.66, 150.40 317.23, 149.36 340.49, 137.56 336.66)), ((161.60 349.38, 183.40 341.21, 168.89 359.41, 161.60 349.38)))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Sun', NULL, 130, 380, 2, N'#ffd54f', N'CURVEPOLYGON (CIRCULARSTRING (96 380, 130 414, 164 380, 130 346, 96 380))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Hills', NULL, 0, 0, 3, N'#a5d6a7', N'CURVEPOLYGON (COMPOUNDCURVE (CIRCULARSTRING (0 120, 250 230, 520 140), CIRCULARSTRING (520 140, 760 215, 1000 130), (1000 130, 1000 60, 0 60, 0 120)))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Grass', NULL, 0, 0, 4, N'#7cb342', N'POLYGON ((0 0, 1000 0, 1000 75, 0 75, 0 0))', 8, 0, 0, NULL, NULL, 0),
(N'Park', N'Trunk', NULL, 0, 0, 5, N'#795548', N'POLYGON ((860 70, 886 70, 880 195, 866 195, 860 70))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Leaves1', NULL, 0, 0, 6, N'#558b2f', N'CURVEPOLYGON (CIRCULARSTRING (802 215, 842 255, 882 215, 842 175, 802 215))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Leaves2', NULL, 0, 0, 7, N'#689f38', N'CURVEPOLYGON (CIRCULARSTRING (853 226, 898 271, 943 226, 898 181, 853 226))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Leaves3', NULL, 0, 0, 8, N'#7cb342', N'CURVEPOLYGON (CIRCULARSTRING (828 262, 868 302, 908 262, 868 222, 828 262))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Flower1', NULL, 0, 0, 9, N'#f06292', N'CURVEPOLYGON (CIRCULARSTRING (65 40, 70 45, 75 40, 70 35, 65 40))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Flower2', NULL, 0, 0, 9, N'#fff176', N'CURVEPOLYGON (CIRCULARSTRING (325 22, 330 27, 335 22, 330 17, 325 22))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Flower3', NULL, 0, 0, 9, N'#f06292', N'CURVEPOLYGON (CIRCULARSTRING (465 50, 470 55, 475 50, 470 45, 465 50))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Flower4', NULL, 0, 0, 9, N'#ba68c8', N'CURVEPOLYGON (CIRCULARSTRING (605 18, 610 23, 615 18, 610 13, 605 18))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Park', N'Flower5', NULL, 0, 0, 9, N'#fff176', N'CURVEPOLYGON (CIRCULARSTRING (945 35, 950 40, 955 35, 950 30, 945 35))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Cloud1', N'Puff1', NULL, 0, 0, 4, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (-22 0, 0 22, 22 0, 0 -22, -22 0))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Cloud1', N'Puff2', N'Puff1', 0, 0, 4, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (7 -4, 24 13, 41 -4, 24 -21, 7 -4))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Cloud1', N'Puff3', N'Puff1', 0, 0, 4, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (-37 -6, -22 9, -7 -6, -22 -21, -37 -6))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Cloud2', N'Puff1', NULL, 0, 0, 4, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (-22 0, 0 22, 22 0, 0 -22, -22 0))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Cloud2', N'Puff2', N'Puff1', 0, 0, 4, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (7 -4, 24 13, 41 -4, 24 -21, 7 -4))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Cloud2', N'Puff3', N'Puff1', 0, 0, 4, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (-37 -6, -22 9, -7 -6, -22 -21, -37 -6))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'LegBackFar', N'Body', -57.0, 45.0, 10, N'#1a110c', N'CURVEPOLYGON (COMPOUNDCURVE ((-63 48, -63 7.5), CIRCULARSTRING (-63 7.5, -54.75 -0.75, -46.5 7.5), (-46.5 7.5, -51 48, -63 48)))', 6, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'LegFrontFar', N'Body', 63.0, 45.0, 11, N'#1a110c', N'CURVEPOLYGON (COMPOUNDCURVE ((57 48, 57 7.5), CIRCULARSTRING (57 7.5, 65.25 -0.75, 73.5 7.5), (73.5 7.5, 69 48, 57 48)))', 6, 180, 1.0, NULL, NULL, 1.6),
(N'Brian', N'Tail', N'Body', -84.0, 70.5, 15, N'#2b1d16', N'CURVEPOLYGON (COMPOUNDCURVE (CIRCULARSTRING (-84 75, -108 85.5, -126 109.5), (-126 109.5, -130.5 106.5), CIRCULARSTRING (-130.5 106.5, -111 78, -84 66), (-84 66, -84 75)))', 4, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'Body', NULL, 0.0, 60.0, 20, N'#2b1d16', N'CURVEPOLYGON (COMPOUNDCURVE ((-67.5 37.5, 67.5 37.5), CIRCULARSTRING (67.5 37.5, 90 60, 67.5 82.5), (67.5 82.5, -67.5 82.5), CIRCULARSTRING (-67.5 82.5, -90 60, -67.5 37.5)))', 6, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'Belly', N'Body', 0.0, 60.0, 21, N'#c47a3a', N'POLYGON ((-45 39, 60 39, 70.5 46.5, 60 54, -45 49.5, -54 43.5, -45 39))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'Collar', N'Body', 78.0, 78.0, 22, N'#d32f2f', N'POLYGON ((72 60, 82.5 57, 93 90, 82.5 93, 72 60))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'Tag', N'Collar', 82.5, 60.0, 23, N'#ffca28', N'CURVEPOLYGON (CIRCULARSTRING (78.6 53.25, 82.5 57.15, 86.4 53.25, 82.5 49.35, 78.6 53.25))', 6, 0, 0.4, NULL, NULL, 1.6),
(N'Brian', N'LegBackNear', N'Body', -48.0, 45.0, 25, N'#2b1d16', N'CURVEPOLYGON (COMPOUNDCURVE ((-54 48, -54 7.5), CIRCULARSTRING (-54 7.5, -45.75 -0.75, -37.5 7.5), (-37.5 7.5, -42 48, -54 48)))', 6, 180, 1.0, NULL, NULL, 1.6),
(N'Brian', N'LegFrontNear', N'Body', 72.0, 45.0, 26, N'#2b1d16', N'CURVEPOLYGON (COMPOUNDCURVE ((66 48, 66 7.5), CIRCULARSTRING (66 7.5, 74.25 -0.75, 82.5 7.5), (82.5 7.5, 78 48, 66 48)))', 6, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'PawBackNear', N'LegBackNear', -48.0, 45.0, 27, N'#c47a3a', N'CURVEPOLYGON (CIRCULARSTRING (-51 4.5, -45 10.5, -39 4.5, -45 -1.5, -51 4.5))', 8, 0, 0.5, NULL, NULL, 1.6),
(N'Brian', N'PawFrontNear', N'LegFrontNear', 72.0, 45.0, 27, N'#c47a3a', N'CURVEPOLYGON (CIRCULARSTRING (69 4.5, 75 10.5, 81 4.5, 75 -1.5, 69 4.5))', 8, 0, 0.5, NULL, NULL, 1.6),
(N'Brian', N'Head', N'Body', 84.0, 84.0, 30, N'#2b1d16', N'CURVEPOLYGON (CIRCULARSTRING (81 93, 102 114, 123 93, 102 72, 81 93))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'Snout', N'Head', 117.0, 90.0, 31, N'#2b1d16', N'CURVEPOLYGON (COMPOUNDCURVE ((114 102, 150 90), CIRCULARSTRING (150 90, 157.5 82.5, 150 75), (150 75, 114 75, 114 102)))', 8, 0, 1.0, NULL, NULL, 1.6),
(N'Brian', N'Muzzle', N'Snout', 117.0, 90.0, 32, N'#c47a3a', N'POLYGON ((117 75, 150 75, 153.75 79.5, 120 83.25, 117 75))', 8, 0, 0.5, NULL, NULL, 1.6),
(N'Brian', N'Nose', N'Snout', 117.0, 90.0, 33, N'#0d0d0d', N'CURVEPOLYGON (CIRCULARSTRING (149.55 87, 153.75 91.2, 157.95 87, 153.75 82.8, 149.55 87))', 8, 0, 0.3, NULL, NULL, 1.6),
(N'Brian', N'Eye', N'Head', 112.5, 99.0, 34, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (108.3 99, 112.5 103.2, 116.7 99, 112.5 94.8, 108.3 99))', 8, 0, 0.3, NULL, NULL, 1.6),
(N'Brian', N'Pupil', N'Eye', 112.5, 99.0, 35, N'#0d0d0d', N'CURVEPOLYGON (CIRCULARSTRING (111.45 99, 113.85 101.4, 116.25 99, 113.85 96.6, 111.45 99))', 8, 0, 0.2, NULL, NULL, 0),
(N'Brian', N'Brow', N'Head', 111.0, 106.5, 36, N'#c47a3a', N'CURVEPOLYGON (CIRCULARSTRING (108.15 107.25, 111 110.1, 113.85 107.25, 111 104.4, 108.15 107.25))', 8, 0, 0.3, NULL, NULL, 0),
(N'Brian', N'Ear', N'Head', 96.0, 108.0, 37, N'#1f140f', N'CURVEPOLYGON (COMPOUNDCURVE ((93 111, 105 108), CIRCULARSTRING (105 108, 103.5 87, 93 69), CIRCULARSTRING (93 69, 84 78, 85.5 96), (85.5 96, 93 111)))', 6, 0, 1.0, NULL, NULL, 1.6),
(N'Ball', N'Ball', NULL, 0.0, 13.5, 40, N'#e53935', N'CURVEPOLYGON (CIRCULARSTRING (-13.5 13.5, 0 27, 13.5 13.5, 0 0, -13.5 13.5))', 12, 0, 0.5, NULL, NULL, 1.6),
(N'Ball', N'Stripe', N'Ball', 0.0, 13.5, 41, N'#ffffff', N'POLYGON ((-12.75 11.4, 12.75 11.4, 12.75 15.6, -12.75 15.6, -12.75 11.4))', 8, 0, 0.3, NULL, NULL, 0),
(N'Bang', N'Bubble', NULL, 0, 0, 50, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (-22 0, 0 18, 22 0, 0 -18, -22 0))', 8, 0, 1.0, N'!', 28, 2),
(N'Bang', N'Tail', N'Bubble', 0, 0, 49, N'#ffffff', N'POLYGON ((-6 -14, 6 -14, -10 -32, -6 -14))', 8, 0, 1.0, NULL, NULL, 2),
(N'Woof', N'Bubble', NULL, 0, 0, 50, N'#ffffff', N'CURVEPOLYGON (CIRCULARSTRING (-46 0, 0 26, 46 0, 0 -26, -46 0))', 8, 0, 1.0, N'WOOF!', 24, 2),
(N'Woof', N'Tail', N'Bubble', 0, 0, 49, N'#ffffff', N'POLYGON ((10 -20, 22 -18, 34 -44, 10 -20))', 8, 0, 1.0, NULL, NULL, 2),
(N'Title', N'Card', NULL, 0, 0, 60, N'#fff8e1', N'POLYGON ((-215 -40, -200 -58, 200 -58, 215 -40, 215 40, 200 58, -200 58, -215 40, -215 -40))', 8, 0, 1.0, N'BRIAN  in  "The Ball"', 34, 3),
(N'End', N'Card', NULL, 0, 0, 60, N'#fff8e1', N'POLYGON ((-215 -40, -200 -58, 200 -58, 215 -40, 215 40, 200 58, -200 58, -215 40, -215 -40))', 8, 0, 1.0, N'THE END', 40, 3);
GO

The Final Scene — “Brian and The Ball”

/* ---- episode one: Brian and the Ball (20 seconds at 12 fps) ---- */
INSERT toon.Scene (Scene, Title, Frames, Fps) VALUES ('BrianAndTheBall', N'Brian in "The Ball"', 241, 12);
INSERT toon.Casting (Scene, Character) VALUES
('BrianAndTheBall', 'Park'), ('BrianAndTheBall', 'Cloud1'), ('BrianAndTheBall', 'Cloud2'), ('BrianAndTheBall', 'Brian'),
('BrianAndTheBall', 'Ball'), ('BrianAndTheBall', 'Bang'), ('BrianAndTheBall', 'Woof'), ('BrianAndTheBall', 'Title'), ('BrianAndTheBall', 'End');
INSERT toon.[Key] (Scene, Character, Part, Channel, Frame, Value, Ease) VALUES
('BrianAndTheBall', 'Park', 'Rays', 'angle', 0, 0, 'smooth'),
('BrianAndTheBall', 'Park', 'Rays', 'angle', 240, -90, 'linear'),
('BrianAndTheBall', 'Cloud1', 'Puff1', 'dx', 0, 280, 'smooth'),
('BrianAndTheBall', 'Cloud1', 'Puff1', 'dx', 240, 520, 'linear'),
('BrianAndTheBall', 'Cloud1', 'Puff1', 'dy', 0, 350, 'smooth'),
('BrianAndTheBall', 'Cloud2', 'Puff1', 'dx', 0, 640, 'smooth'),
('BrianAndTheBall', 'Cloud2', 'Puff1', 'dx', 240, 820, 'linear'),
('BrianAndTheBall', 'Cloud2', 'Puff1', 'dy', 0, 400, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'dx', 0, 200, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'dx', 60, 200, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'dx', 150, 615, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'dx', 175, 615, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'dx', 225, 250, 'linear'),
('BrianAndTheBall', 'Brian', 'Body', 'dx', 240, 250, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'dy', 0, 65, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'sx', 0, 1, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'sx', 174, 1, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'sx', 175, -1, 'step'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 60, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 68, 4, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 142, 4, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 150, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 178, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 184, 2.5, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 218, 2.5, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'bob', 225, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'angle', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'angle', 150, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'angle', 158, -8, 'smooth'),
('BrianAndTheBall', 'Brian', 'Body', 'angle', 168, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 30, 8, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 45, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 52, 14, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 60, 4, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 150, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 157, -24, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 168, -6, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 175, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 226, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 230, 16, 'smooth'),
('BrianAndTheBall', 'Brian', 'Head', 'angle', 240, 16, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 50, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 53, 18, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 60, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 70, -55, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 145, -55, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 152, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 180, -35, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 220, -35, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'angle', 226, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'osc', 0, 4, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'osc', 70, 12, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'osc', 145, 12, 'smooth'),
('BrianAndTheBall', 'Brian', 'Ear', 'osc', 152, 4, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'angle', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'angle', 62, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'angle', 70, -18, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'angle', 146, -18, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'angle', 152, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'osc', 0, 18, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'osc', 50, 18, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'osc', 53, 35, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'osc', 62, 12, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'osc', 150, 22, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'osc', 226, 22, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'osc', 230, 40, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tail', 'osc', 240, 40, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 60, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 64, 38, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 146, 38, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 150, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 176, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 180, 26, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 222, 26, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackFar', 'osc', 225, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 60, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 64, 38, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 146, 38, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 150, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 176, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 180, 26, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 222, 26, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontFar', 'osc', 225, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 60, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 64, 38, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 146, 38, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 150, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 176, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 180, 26, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 222, 26, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegBackNear', 'osc', 225, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 60, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 64, 38, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 146, 38, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 150, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 176, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 180, 26, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 222, 26, 'smooth'),
('BrianAndTheBall', 'Brian', 'LegFrontNear', 'osc', 225, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 0, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 60, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 64, 25, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 146, 25, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 150, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 176, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 180, 18, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 222, 18, 'smooth'),
('BrianAndTheBall', 'Brian', 'Tag', 'osc', 225, 0, 'smooth'),
('BrianAndTheBall', 'Brian', 'Eye', 'vis', 0, 1, 'smooth'),
('BrianAndTheBall', 'Brian', 'Eye', 'vis', 24, 0, 'step'),
('BrianAndTheBall', 'Brian', 'Eye', 'vis', 26, 1, 'step'),
('BrianAndTheBall', 'Brian', 'Eye', 'vis', 110, 0, 'step'),
('BrianAndTheBall', 'Brian', 'Eye', 'vis', 112, 1, 'step'),
('BrianAndTheBall', 'Brian', 'Eye', 'vis', 196, 0, 'step'),
('BrianAndTheBall', 'Brian', 'Eye', 'vis', 198, 1, 'step'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 0, 1080, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 20, 1080, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 62, 780, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 168, 780, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 171, 765, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 174, 765, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 175, 465, 'step'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 225, 100, 'linear'),
('BrianAndTheBall', 'Ball', 'Ball', 'dx', 240, 100, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dy', 0, 65, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dy', 168, 65, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dy', 171, 129.5, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dy', 226, 129.5, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'dy', 232, 65, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'bob', 0, 0, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'bob', 20, 150, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'bob', 62, 0, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'angle', 0, 0, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'angle', 20, 0, 'smooth'),
('BrianAndTheBall', 'Ball', 'Ball', 'angle', 62, 1273, 'smooth'),
('BrianAndTheBall', 'Bang', 'Bubble', 'dx', 0, 305, 'smooth'),
('BrianAndTheBall', 'Bang', 'Bubble', 'dy', 0, 220, 'smooth'),
('BrianAndTheBall', 'Bang', 'Bubble', 'vis', 0, 0, 'smooth'),
('BrianAndTheBall', 'Bang', 'Bubble', 'vis', 52, 1, 'step'),
('BrianAndTheBall', 'Bang', 'Bubble', 'vis', 64, 0, 'step'),
('BrianAndTheBall', 'Woof', 'Bubble', 'dx', 0, 105, 'smooth'),
('BrianAndTheBall', 'Woof', 'Bubble', 'dy', 0, 240, 'smooth'),
('BrianAndTheBall', 'Woof', 'Bubble', 'vis', 0, 0, 'smooth'),
('BrianAndTheBall', 'Woof', 'Bubble', 'vis', 229, 1, 'step'),
('BrianAndTheBall', 'Title', 'Card', 'dx', 0, 500, 'smooth'),
('BrianAndTheBall', 'Title', 'Card', 'dy', 0, 280, 'smooth'),
('BrianAndTheBall', 'Title', 'Card', 'vis', 0, 1, 'smooth'),
('BrianAndTheBall', 'Title', 'Card', 'vis', 36, 0, 'step'),
('BrianAndTheBall', 'End', 'Card', 'dx', 0, 500, 'smooth'),
('BrianAndTheBall', 'End', 'Card', 'dy', 0, 300, 'smooth'),
('BrianAndTheBall', 'End', 'Card', 'vis', 0, 0, 'smooth'),
('BrianAndTheBall', 'End', 'Card', 'vis', 236, 1, 'step');
GO

Don’t Panic

The referenced database called “SpatialStudio” is the real demo but I will post it separately to this story. I am very pleased with both this little animation and the spatial and vector examples I have put together. I found when spatial got put into the product back in SQL 2005 (I think) I was so happy and waited for it to explode, and it just didnt really go anywhere. Now I think its got as real chance to be used.

IN SQL Server 2025 we now have Vector indexes which are based on the DiskANN algorithm. DiskANN relies on creating a graph to navigate quickly through all the indexed vectors to find the closest match to a given vector. DiskANN is a graph-based system for indexing and searching large sets of vector data. Take a look at the paper linked for a deep dive into DiskANN. The code I will publish tomorrow uses Vector indexes in the final part of the application.

Hope you enjoyed this little detour from the heavy on the SQL stories to look at puppies but I needed to animate him.

If you enjoyed please Clap as its this single metric that makes me lie awake all night wondering if the whole world will have seen my writing, and thinking if everyone clapped multiple times I could be rich beyond my wildest dreams and then I check and see 0 claps. Maybe a different tactic is in order….

Right Brian is being held hostage….

Twenty Years Later, I Built the Headache Machine Again

Thu, 01 Oct 2026 19:20:53 GMT

Using an archive of Brain Implant data to predict correct placement of electrodes

When your only tool is a database

Some people get headaches, some people get migraines, I get Cluster Headaches.

With a headache you can take a tablet and it goes, a migraine you can lie down in the dark and hope it goes and can last a week.

A Cluster headache is not like that, it may last between 15 mins and an hour. At the worst point mine were lasting an hour, of level 10 pain 12 times every day. The condition used to be called Suicide Headaches and for reasons you can begin to understand. You will want to put your head through a wall. Imagine someone’s stuck a hot poker behind your eye and is slowly turning it, on a schedule, for weeks or giving birth 12 times a day. It is regarded as one of the worst pains known to mankind.

Mine started as episodic which mean a few weeks of the year you have the headaches but they quickly turned chronic (less than a year), which means the exact opposite, they are relentless and for two weeks in a year they would disappear. As the seasons change was roughly the worst periods.

Whenever there was a change in the weather, high pressure to low pressure weather patterns would trigger an attack, strong perfume type smells would trigger an attack, and any light would trigger an attack. Going to hospital when it got so bad you literally cannot take it anymore knowing it was going to be worse was a final resort. Only very recently enough medical staff know about Cluster Headaches and offer to turn the lights off in the hospital.

They turn up at the same time of day and night. The first one of the day was always 3 mins after waking up.

About twenty years ago, when they first arrived, I did what any desperate person does. I read everything. Journals, forums, the small print on trial protocols. And then I did what any desperate *database* person does. I built a system.

That’s the thing about spending your life in SQL. When the world goes wrong, you don’t reach for a hammer. You reach for a schema.

Version one: the back of the head

The idea back then was occipital nerve stimulation. You put small electrodes under the skin at the back of the head, over the occipital nerves, and feed them a gentle current. The theory was that you could calm the pain pathway without anyone opening your skull.

I liked that last bit a lot.

It was ex-cranial, so the brain itself stayed out of it. But the question was still the same one I’m asking today: where exactly do you put the thing? A few millimetres either way seemed to decide whether it helped or just gave you a buzzing scalp. So my system tried to line up where the electrodes went with how people actually got on afterwards. Placement in, outcome out, look for the pattern.

Was it any good? It was a desperate man’s database. It asked the right question with not nearly enough data. But it taught me something I’ve carried ever since. The answer isn’t in the textbook diagram. It’s in what happened to the people who went before you.

Twenty years on, round two

So why pick it up again? Partly because I’ve spent the last few months making SQL Server do silly things. Doom. Elite. A bouncing Amiga ball. A synth. At some point you ask yourself whether the same tricks could do something that matters.

And partly because the tools have finally caught up with the idea. SQL Server 2025 has a native `VECTOR` type. Twenty years ago “find me the patients most like this one” meant a pile of hand-rolled maths. Now it’s a function call.

This time I went inside the skull, at least on paper. Deep brain stimulation, or DBS, is a real treatment for Parkinson’s. For cluster headache it’s still very much experimental, aimed at a spot called the posterior hypothalamus. I built it to handle both, because the problem is the same shape.

The first decision was the easiest one: X-ray or 3D scan? I briefly fancied X-ray because it’s cheap and everywhere. Then I remembered an X-ray of a head is mostly a very good picture of a skull. You can’t see deep brain structures on it. Real surgical planning uses a 3D MRI, and a CT afterwards to check where the lead actually landed. So that’s what I modelled. Sometimes the harder option is the right one simply because its a hard problem.

Decisions

***Brains that look alike.
***Every brain is a slightly different shape. Surgeons stretch each one onto a standard map, but a stretched map still isn’t a person. So each scan becomes a fingerprint of 14 numbers: ventricle widths, brain size, the distance between two landmarks in the middle of the brain, age. The system then finds the 40 previous patients whose fingerprints sit closest. Think of it as a dating app for brains, with worse chat-up lines.

Let the outcomes do the teaching.
This is the lesson from twenty years ago. It doesn’t copy where the neighbours’ electrodes went. It asks where the electrodes went and how well it worked, then works out the spot that best explains the good results. A placement that left someone with side effects counts against that area. There’s a slider for how cautious to be, which felt like the right thing to put in a human’s hands.

Keep it in the database.
Obviously. The search, the scoring and the fitting all run in T-SQL. There’s a Python mirror of the same maths so the Streamlit front end still works on a laptop, but SQL Server does the thinking. Old habits.

Close the loop.
Every new patient’s 12-month result goes back into the library. The next person gets a slightly better answer because of the last one. That’s the bit I wanted most back then.

Simulated data.
I’m not going anywhere near real patient records for a hobby project. So I wrote a simulator that makes up brains, operations and outcomes. The bonus is that with made-up patients you know the right answer. With real ones you never do.

Did it work?

In the simulation, yes. Modestly. To test it, I hid each past patient in turn and re-planned them from everybody else. Then I compared the suggestion with the standard textbook target.

Results of the simulation

Half a millimetre doesn’t sound like much. In this part of the brain it’s the difference between “that helped” and “why can I taste rainbows?”

These are made-up patients, and I wrote the rules they follow, so of course the method can learn them. That shows the machinery works. It doesn’t show it would help a real person. The suggestions also drift a touch closer to the areas that cause side effects, which is exactly why the caution slider exists. A real version would need proper brain registration, real post-op imaging, ethics approval and a lot of people much cleverer than me. There’s already serious open-source work in this area, the Lead-DBS toolbox for one. I’m a database man having a go, not a neurosurgeon.

NeuroTarget — Prototype Application

A SQL Server 2025 system that suggests deep brain stimulation (DBS) electrode placement from anatomically similar previous patients and how their operations turned out. It covers cluster headache (posterior hypothalamus) and Parkinson’s disease (subthalamic nucleus and globus pallidus internus), and has a Streamlit + Plotly front end.

All data is simulated. This is a research/teaching prototype for exploring the idea. It is not a medical device and must not be used for clinical decisions.

Why 3D MRI, not X-ray

An X-ray is a flat projection and barely shows brain soft tissue, so deep targets can’t be seen on it. Real stereotactic planning is done on a 3D T1/T2 MRI, often fused with CT. After the operation, a CT (or sometimes X-ray) is used only to confirm where the lead actually ended up. The system follows the same pattern:

| Stage         | Imaging              | Stored as                                                                             |
|---------------|----------------------|---------------------------------------------------------------------------------------|
| Planning      | 3D T1 MRI            | 14-number anatomy fingerprint (`VECTOR(14)`), AC/PC landmarks, scaling to AC-PC space |
| After surgery | CT lead localisation | Positions of the 4 contacts (`nt.Contact`)                                            |
| Follow-up     | Clinic scores        | Active contact, efficacy %, side effects, patient rating                              |

How a suggestion is made (nt.usp_SuggestPlacement)

Fingerprint. At ingest, 14 features are measured from the MRI: brain extents and volume, AC-PC length, third-ventricle width and height, lateral-ventricle volume and span, Evans index, some ratios, and age. They are z-scored against the library and stored as VECTOR(14) (nt.usp_IngestScan).
Find similar brains. VECTOR_DISTANCE(euclidean, …) picks the K nearest previous implants for the same target. Left and right leads are pooled by mirroring X.
Fit the sweet spot. Each neighbour gives one observation: where its active contact sat, and how well it worked. A grid search finds the point that best explains those outcomes under the target’s dose–response curve. Similar neighbours count for more, via a Gaussian kernel on anatomy distance. Candidate points are penalised where similar patients reported side effects. The search runs at 1 mm, then refines at 0.25 mm; it’s a CROSS JOIN against nt.GridPoint.
Map it into this patient’s scan (native mm) and log it in nt.Suggestion.
Close the loop. nt.usp_RecordImplantOutcome stores the actual contacts and the 12-month outcome. From then on, that patient is evidence for everyone after them.

Does it help?

Each historical implant is hidden in turn and re-planned from everyone else. Because the data is simulated, every patient’s true optimal point is known, so both the suggestion and the plain atlas target can be scored against it.

Suggested points sit a little closer to side-effect structures than the atlas target does (for example, CH side-effect score 0.9 → 1.4 out of 10). The “Side-effect caution” slider sets that trade-off.

I created a SQL Server 2025 schema and supporting code. It had to be 2025 as this supports the Vector type and gives me Vector Distance.

Reference data was created

CREATE TABLE nt.Condition
(
    ConditionId     TINYINT       NOT NULL CONSTRAINT PK_Condition PRIMARY KEY,
    Code            VARCHAR(10)   NOT NULL CONSTRAINT UQ_Condition_Code UNIQUE,
    Name            NVARCHAR(100) NOT NULL,
    OutcomeMeasure  NVARCHAR(200) NOT NULL
);

CREATE TABLE nt.Target
(
    TargetId        TINYINT       NOT NULL CONSTRAINT PK_Target PRIMARY KEY,
    ConditionId     TINYINT       NOT NULL CONSTRAINT FK_Target_Condition REFERENCES nt.Condition,
    Code            VARCHAR(20)   NOT NULL CONSTRAINT UQ_Target_Code UNIQUE,
    Name            NVARCHAR(100) NOT NULL,
    Laterality      VARCHAR(12)   NOT NULL,              -- Ipsilateral | Bilateral
    AtlasX          FLOAT         NOT NULL,              -- right hemisphere, template mm
    AtlasY          FLOAT         NOT NULL,
    AtlasZ          FLOAT         NOT NULL,
    -- dose-response model used by the fit: Eff = Floor + (Emax-Floor)*exp(-d^2 / 2 Sigma^2)
    Emax            FLOAT         NOT NULL,
    Floor           FLOAT         NOT NULL,
    SigmaMm         FLOAT         NOT NULL,
    NoGoName        NVARCHAR(200) NULL,                  -- the structure that causes side effects
    NoGo            NVARCHAR(100) NULL                   -- sim only: JSON offset from sweet spot
);

/* One row per image-derived feature. PopMean/PopStd are learned from the historical library
   and used to z-score any new scan; Weight lets you emphasise features that matter. */
CREATE TABLE nt.FeatureDef
(
    FeatureNo       TINYINT       NOT NULL CONSTRAINT PK_FeatureDef PRIMARY KEY,
    Code            VARCHAR(40)   NOT NULL CONSTRAINT UQ_FeatureDef_Code UNIQUE,
    Description     NVARCHAR(200) NOT NULL,
    Unit            VARCHAR(10)   NOT NULL,
    PopMean         FLOAT         NOT NULL,
    PopStd          FLOAT         NOT NULL,
    Weight          FLOAT         NOT NULL CONSTRAINT DF_FeatureDef_Weight DEFAULT (1.0)
);

/* Candidate offsets (mm) for the sweet-spot search. Pass 1 = 1 mm over +-5 mm (1331 pts),
   Pass 2 = 0.25 mm over +-1 mm around the pass-1 winner (729 pts). */
CREATE TABLE nt.GridPoint
(
    GridId  INT      NOT NULL IDENTITY CONSTRAINT PK_GridPoint PRIMARY KEY,
    Pass    TINYINT  NOT NULL,
    DX      FLOAT    NOT NULL,
    DY      FLOAT    NOT NULL,
    DZ      FLOAT    NOT NULL,
    L1      AS (ABS(DX) + ABS(DY) + ABS(DZ)) PERSISTED    -- tie-break: prefer closer to centre
);
CREATE INDEX IX_GridPoint_Pass ON nt.GridPoint (Pass) INCLUDE (DX, DY, DZ, L1);

Then Patient and Scan data.

CREATE TABLE nt.Patient
(
    PatientId       INT           NOT NULL CONSTRAINT PK_Patient PRIMARY KEY,
    ConditionId     TINYINT       NOT NULL CONSTRAINT FK_Patient_Condition REFERENCES nt.Condition,
    Sex             CHAR(1)       NOT NULL,
    AgeAtScan       TINYINT       NOT NULL,
    PainSide        CHAR(1)       NULL,                  -- cluster headache: side of attacks
    Status          VARCHAR(12)   NOT NULL CONSTRAINT CK_Patient_Status CHECK (Status IN ('Historical','Referral')),
    IsSimulated     BIT           NOT NULL CONSTRAINT DF_Patient_IsSimulated DEFAULT (1)
);

CREATE TABLE nt.Scan
(
    ScanId          INT           NOT NULL CONSTRAINT PK_Scan PRIMARY KEY,
    PatientId       INT           NOT NULL CONSTRAINT FK_Scan_Patient REFERENCES nt.Patient,
    Modality        VARCHAR(20)   NOT NULL,              -- MRI_T1_3D
    AcquiredOn      DATE          NOT NULL,
    VoxelMm         FLOAT         NOT NULL,
    ACX FLOAT NOT NULL, ACY FLOAT NOT NULL, ACZ FLOAT NOT NULL,   -- landmarks, native mm
    PCX FLOAT NOT NULL, PCY FLOAT NOT NULL, PCZ FLOAT NOT NULL,
    MCPX AS ((ACX + PCX) / 2.0) PERSISTED,
    MCPY AS ((ACY + PCY) / 2.0) PERSISTED,
    MCPZ AS ((ACZ + PCZ) / 2.0) PERSISTED,
    ScaleX FLOAT NOT NULL, ScaleY FLOAT NOT NULL, ScaleZ FLOAT NOT NULL,  -- native -> template
    RawFeatures     NVARCHAR(MAX) NOT NULL CONSTRAINT CK_Scan_RawFeatures CHECK (ISJSON(RawFeatures) = 1),
    Features        VECTOR(14)    NOT NULL               -- z-scored, weighted anatomy fingerprint
);
CREATE INDEX IX_Scan_Patient ON nt.Scan (PatientId);

/* The image itself normally stays in PACS; for the demo the new referrals' volumes live here. */
CREATE TABLE nt.ScanVolume
(
    ScanId          INT            NOT NULL CONSTRAINT PK_ScanVolume PRIMARY KEY
                                   CONSTRAINT FK_ScanVolume_Scan REFERENCES nt.Scan,
    DimX SMALLINT NOT NULL, DimY SMALLINT NOT NULL, DimZ SMALLINT NOT NULL,
    OriginX FLOAT NOT NULL, OriginY FLOAT NOT NULL, OriginZ FLOAT NOT NULL,  -- native mm of voxel (0,0,0) corner
    Encoding        VARCHAR(20)    NOT NULL,              -- zlib-uint8, C order (x,y,z)
    Data            VARBINARY(MAX) NOT NULL
);

Then surgery, programming and outcome data.

CREATE SEQUENCE nt.seqImplant AS INT START WITH 1000000;

CREATE TABLE nt.Implant
(
    ImplantId       INT           NOT NULL CONSTRAINT PK_Implant PRIMARY KEY
                                  CONSTRAINT DF_Implant_Id DEFAULT (NEXT VALUE FOR nt.seqImplant),
    PatientId       INT           NOT NULL CONSTRAINT FK_Implant_Patient REFERENCES nt.Patient,
    TargetId        TINYINT       NOT NULL CONSTRAINT FK_Implant_Target REFERENCES nt.Target,
    Hemisphere      CHAR(1)       NOT NULL CONSTRAINT CK_Implant_Hemi CHECK (Hemisphere IN ('L','R')),
    ImplantedOn     DATE          NOT NULL,
    PlannedX FLOAT NULL, PlannedY FLOAT NULL, PlannedZ FLOAT NULL,          -- template mm (signed X)
    SuggestionId    INT           NULL,
    CONSTRAINT UQ_Implant UNIQUE (PatientId, TargetId, Hemisphere)
);

/* Post-op CT-localised contact centres, template mm (signed X). Contact 0 = deepest. */
CREATE TABLE nt.Contact
(
    ImplantId       INT     NOT NULL CONSTRAINT FK_Contact_Implant REFERENCES nt.Implant,
    ContactNo       TINYINT NOT NULL,
    X FLOAT NOT NULL, Y FLOAT NOT NULL, Z FLOAT NOT NULL,
    CONSTRAINT PK_Contact PRIMARY KEY (ImplantId, ContactNo)
);

CREATE TABLE nt.Programming
(
    ImplantId       INT     NOT NULL CONSTRAINT PK_Programming PRIMARY KEY
                            CONSTRAINT FK_Programming_Implant REFERENCES nt.Implant,
    ActiveContact   TINYINT NOT NULL,
    AmplitudeMa     DECIMAL(4,1) NOT NULL,
    PulseWidthUs    SMALLINT     NOT NULL,
    FrequencyHz     SMALLINT     NOT NULL
);

CREATE TABLE nt.Outcome
(
    OutcomeId       INT          NOT NULL IDENTITY CONSTRAINT PK_Outcome PRIMARY KEY,
    ImplantId       INT          NOT NULL CONSTRAINT FK_Outcome_Implant REFERENCES nt.Implant,
    MonthsPostOp    TINYINT      NOT NULL,
    EfficacyPct     FLOAT        NOT NULL,   -- condition-specific, see nt.Condition.OutcomeMeasure
    SideEffectScore FLOAT        NOT NULL,   -- 0 (none) .. 10 (intolerable)
    PatientRating   TINYINT      NOT NULL,   -- 0 .. 10 patient-reported
    NetBenefit      AS (EfficacyPct / 100.0 - 0.04 * SideEffectScore) PERSISTED,
    RecordedAt      DATETIME2(0) NOT NULL CONSTRAINT DF_Outcome_RecordedAt DEFAULT (SYSUTCDATETIME())
);
CREATE INDEX IX_Outcome_Implant ON nt.Outcome (ImplantId, MonthsPostOp DESC);

/* Every suggestion made, so the surgeon's decision and the eventual result can be audited. */
CREATE TABLE nt.Suggestion
(
    SuggestionId        INT          NOT NULL IDENTITY CONSTRAINT PK_Suggestion PRIMARY KEY,
    ScanId              INT          NOT NULL CONSTRAINT FK_Suggestion_Scan REFERENCES nt.Scan,
    TargetId            TINYINT      NOT NULL CONSTRAINT FK_Suggestion_Target REFERENCES nt.Target,
    Hemisphere          CHAR(1)      NOT NULL,
    K                   INT          NOT NULL,
    SideEffectPenalty   FLOAT        NOT NULL,
    TemplateX FLOAT NOT NULL, TemplateY FLOAT NOT NULL, TemplateZ FLOAT NOT NULL,
    NativeX   FLOAT NOT NULL, NativeY   FLOAT NOT NULL, NativeZ   FLOAT NOT NULL,
    SdX FLOAT NULL, SdY FLOAT NULL, SdZ FLOAT NULL,
    AtlasShiftMm        FLOAT        NOT NULL,
    EffectiveN          FLOAT        NOT NULL,
    PredictedEfficacy   FLOAT        NULL,
    PredictedSideEffect FLOAT        NULL,
    FitRmse             FLOAT        NOT NULL,
    CreatedAt           DATETIME2(0) NOT NULL CONSTRAINT DF_Suggestion_CreatedAt DEFAULT (SYSUTCDATETIME()),
    AcceptedBySurgeon   BIT          NULL
);

/* SIMULATION ONLY: each patient's hidden true optimal point, used to score the method. */
CREATE TABLE nt.SimTruth
(
    PatientId   INT     NOT NULL CONSTRAINT FK_SimTruth_Patient REFERENCES nt.Patient,
    TargetId    TINYINT NOT NULL CONSTRAINT FK_SimTruth_Target REFERENCES nt.Target,
    Hemisphere  CHAR(1) NOT NULL,
    SweetX FLOAT NOT NULL, SweetY FLOAT NOT NULL, SweetZ FLOAT NOT NULL,   -- template mm, signed X
    CONSTRAINT PK_SimTruth PRIMARY KEY (PatientId, TargetId, Hemisphere)
);
GO

Then the evidence.

/* Evidence = every implant with a known active contact and its latest outcome.
   X is folded to the right hemisphere so left and right leads pool together. */
CREATE VIEW nt.vHistoricalEvidence
AS
SELECT  i.ImplantId, i.PatientId, s.ScanId, i.TargetId, i.Hemisphere,
        ABS(c.X) AS X, c.Y, c.Z,
        o.EfficacyPct, o.SideEffectScore, o.PatientRating, o.NetBenefit,
        s.Features
FROM    nt.Implant      AS i
JOIN    nt.Programming  AS p ON p.ImplantId = i.ImplantId
JOIN    nt.Contact      AS c ON c.ImplantId = i.ImplantId AND c.ContactNo = p.ActiveContact
JOIN    nt.Scan         AS s ON s.PatientId = i.PatientId
CROSS APPLY (SELECT TOP (1) o.EfficacyPct, o.SideEffectScore, o.PatientRating, o.NetBenefit
             FROM nt.Outcome AS o
             WHERE o.ImplantId = i.ImplantId
             ORDER BY o.MonthsPostOp DESC, o.OutcomeId DESC) AS o;
GO

/* Grid of candidate offsets */
WITH n AS (SELECT TOP (21) CAST(ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) - 1 AS FLOAT) AS i
           FROM sys.all_objects)
INSERT nt.GridPoint (Pass, DX, DY, DZ)
SELECT 1, x.i - 5, y.i - 5, z.i - 5
FROM   n AS x CROSS JOIN n AS y CROSS JOIN n AS z
WHERE  x.i <= 10 AND y.i <= 10 AND z.i <= 10
UNION ALL
SELECT 2, (x.i - 4) * 0.25, (y.i - 4) * 0.25, (z.i - 4) * 0.25
FROM   n AS x CROSS JOIN n AS y CROSS JOIN n AS z
WHERE  x.i <= 8 AND y.i <= 8 AND z.i <= 8;
GO

/* =====================================================================================
   NeuroTarget — procedures
     nt.usp_IngestScan            z-score raw image features -> VECTOR(14), store scan
     nt.usp_SuggestPlacement      nearest-anatomy neighbours -> fitted sweet spot
     nt.usp_RecordImplantOutcome  close the loop: implant + contacts + programming + outcome
     nt.usp_EvaluateLOO           leave-one-out accuracy vs atlas (simulation ground truth)
   The maths is mirrored line-for-line in python/neurosim/engine.py.
   ===================================================================================== */

/* -------------------------------------------------------------------------------------
   Ingest: RawFeatures is the JSON object produced by the image pipeline, e.g.
   {"brain_w_mm":136.2, "acpc_mm":24.1, ... "age":58}
   ------------------------------------------------------------------------------------- */
CREATE OR ALTER PROCEDURE nt.usp_IngestScan
    @ScanId      INT,
    @PatientId   INT,
    @Modality    VARCHAR(20),
    @AcquiredOn  DATE,
    @VoxelMm     FLOAT,
    @ACX FLOAT, @ACY FLOAT, @ACZ FLOAT,
    @PCX FLOAT, @PCY FLOAT, @PCZ FLOAT,
    @ScaleX FLOAT, @ScaleY FLOAT, @ScaleZ FLOAT,
    @RawFeatures NVARCHAR(MAX)
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;

    DECLARE @missing NVARCHAR(400) =
        (SELECT STRING_AGG(f.Code, N', ')
         FROM nt.FeatureDef AS f
         WHERE JSON_VALUE(@RawFeatures, N'$.' + f.Code) IS NULL);
    IF @missing IS NOT NULL
    BEGIN
        DECLARE @msg NVARCHAR(500) = N'RawFeatures is missing: ' + @missing;
        THROW 50010, @msg, 1;
    END;

    -- z-score each feature against the library population, apply its weight, and build the
    -- JSON array literal that SQL Server 2025 casts straight to VECTOR(n)
    DECLARE @vec NVARCHAR(MAX) =
        N'[' + (SELECT STRING_AGG(CONVERT(VARCHAR(40), CAST(
                        f.Weight * (CAST(JSON_VALUE(@RawFeatures, N'$.' + f.Code) AS FLOAT) - f.PopMean)
                        / NULLIF(f.PopStd, 0) AS DECIMAL(18,6))), ',')
                       WITHIN GROUP (ORDER BY f.FeatureNo)
                FROM nt.FeatureDef AS f) + N']';

    INSERT nt.Scan (ScanId, PatientId, Modality, AcquiredOn, VoxelMm,
                    ACX, ACY, ACZ, PCX, PCY, PCZ, ScaleX, ScaleY, ScaleZ, RawFeatures, Features)
    VALUES (@ScanId, @PatientId, @Modality, @AcquiredOn, @VoxelMm,
            @ACX, @ACY, @ACZ, @PCX, @PCY, @PCZ, @ScaleX, @ScaleY, @ScaleZ, @RawFeatures,
            CAST(@vec AS VECTOR(14)));
END;
GO

/* -------------------------------------------------------------------------------------
   Suggest a placement for one scan / target / hemisphere.

   1. Find the @K implants (same target, other patients) whose scans are nearest in the
      anatomy vector space (VECTOR_DISTANCE euclidean).
   2. Weight them with a Gaussian kernel on that distance (bandwidth = mean distance).
   3. Each neighbour is one observation: active contact position -> efficacy, side effects.
      Grid-search the sweet spot s that best explains those outcomes under the target's
      dose-response curve, penalised by side-effect burden reported near s.
      Coarse 1 mm pass around the atlas, then a 0.25 mm pass around the winner.
   4. Map the answer into the patient's own scan (native mm) and log it.

   Result set 1: the suggestion. Result set 2: the neighbours and their weights.
   ------------------------------------------------------------------------------------- */
CREATE OR ALTER PROCEDURE nt.usp_SuggestPlacement
    @ScanId             INT,
    @TargetId           TINYINT,
    @Hemisphere         CHAR(1) = 'R',
    @K                  INT     = 40,
    @SideEffectPenalty  FLOAT   = 100,
    @Save               BIT     = 1,
    @SuggestionId       INT     = NULL OUTPUT
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;

    IF @Hemisphere NOT IN ('L','R') THROW 50020, N'@Hemisphere must be L or R', 1;

    DECLARE @q VECTOR(14), @PatientId INT,
            @mx FLOAT, @my FLOAT, @mz FLOAT, @sx FLOAT, @sy FLOAT, @sz FLOAT;
    SELECT  @q = s.Features, @PatientId = s.PatientId,
            @mx = s.MCPX, @my = s.MCPY, @mz = s.MCPZ,
            @sx = s.ScaleX, @sy = s.ScaleY, @sz = s.ScaleZ
    FROM    nt.Scan AS s
    WHERE   s.ScanId = @ScanId;
    IF @PatientId IS NULL THROW 50021, N'Unknown @ScanId', 1;

    DECLARE @ax FLOAT, @ay FLOAT, @az FLOAT, @Floor FLOAT, @Emax FLOAT, @Sigma FLOAT;
    SELECT  @ax = AtlasX, @ay = AtlasY, @az = AtlasZ, @Floor = Floor, @Emax = Emax, @Sigma = SigmaMm
    FROM    nt.Target WHERE TargetId = @TargetId;
    IF @ax IS NULL THROW 50022, N'Unknown @TargetId', 1;

    /* 1. nearest anatomy */
    CREATE TABLE #nb
    (
        Rn INT NOT NULL, ImplantId INT NOT NULL, PatientId INT NOT NULL, ScanId INT NOT NULL,
        Hemisphere CHAR(1) NOT NULL, X FLOAT NOT NULL, Y FLOAT NOT NULL, Z FLOAT NOT NULL,
        EfficacyPct FLOAT NOT NULL, SideEffectScore FLOAT NOT NULL, PatientRating TINYINT NOT NULL,
        NetBenefit FLOAT NOT NULL, Dist FLOAT NOT NULL, KernelW FLOAT NULL, NearW FLOAT NULL
    );

    INSERT #nb (Rn, ImplantId, PatientId, ScanId, Hemisphere, X, Y, Z,
                EfficacyPct, SideEffectScore, PatientRating, NetBenefit, Dist)
    SELECT TOP (@K)
           ROW_NUMBER() OVER (ORDER BY d.Dist, d.ImplantId),
           d.ImplantId, d.PatientId, d.ScanId, d.Hemisphere, d.X, d.Y, d.Z,
           d.EfficacyPct, d.SideEffectScore, d.PatientRating, d.NetBenefit, d.Dist
    FROM ( SELECT e.ImplantId, e.PatientId, e.ScanId, e.Hemisphere, e.X, e.Y, e.Z,
                  e.EfficacyPct, e.SideEffectScore, e.PatientRating, e.NetBenefit,
                  VECTOR_DISTANCE('euclidean', e.Features, @q) AS Dist
           FROM   nt.vHistoricalEvidence AS e
           WHERE  e.TargetId = @TargetId
             AND  e.PatientId <> @PatientId ) AS d
    ORDER BY d.Dist, d.ImplantId;

    IF NOT EXISTS (SELECT 1 FROM #nb) THROW 50023, N'No historical evidence for this target', 1;

    /* 2. similarity kernel */
    DECLARE @h FLOAT = (SELECT AVG(Dist) FROM #nb);
    IF @h IS NULL OR @h = 0 SET @h = 1.0;
    UPDATE #nb SET KernelW = EXP(-SQUARE(Dist / @h));
    DECLARE @sumW FLOAT = (SELECT SUM(KernelW) FROM #nb);
    DECLARE @twoS2 FLOAT = 2.0 * @Sigma * @Sigma, @twoR2 FLOAT = 2.0 * 1.5 * 1.5;

    /* 3a. coarse pass around the atlas target */
    DECLARE @cx FLOAT, @cy FLOAT, @cz FLOAT;
    SELECT TOP (1) @cx = @ax + g.DX, @cy = @ay + g.DY, @cz = @az + g.DZ
    FROM   nt.GridPoint AS g
    CROSS APPLY (
        SELECT SUM(n.KernelW * SQUARE(@Floor + (@Emax - @Floor) * EXP(-dd.D2 / @twoS2) - n.EfficacyPct)) / @sumW AS Fit,
               SUM(n.KernelW * n.SideEffectScore * EXP(-dd.D2 / @twoR2)) / @sumW AS Risk
        FROM   #nb AS n
        CROSS APPLY (SELECT SQUARE(@ax + g.DX - n.X) + SQUARE(@ay + g.DY - n.Y) + SQUARE(@az + g.DZ - n.Z) AS D2) AS dd
    ) AS s
    WHERE  g.Pass = 1
    ORDER BY s.Fit + @SideEffectPenalty * s.Risk, g.L1, g.DX, g.DY, g.DZ;   -- = np.lexsort + grid order

    /* 3b. fine pass around the coarse winner */
    DECLARE @bx FLOAT, @by FLOAT, @bz FLOAT, @Fit FLOAT;
    SELECT TOP (1) @bx = @cx + g.DX, @by = @cy + g.DY, @bz = @cz + g.DZ, @Fit = s.Fit
    FROM   nt.GridPoint AS g
    CROSS APPLY (
        SELECT SUM(n.KernelW * SQUARE(@Floor + (@Emax - @Floor) * EXP(-dd.D2 / @twoS2) - n.EfficacyPct)) / @sumW AS Fit,
               SUM(n.KernelW * n.SideEffectScore * EXP(-dd.D2 / @twoR2)) / @sumW AS Risk
        FROM   #nb AS n
        CROSS APPLY (SELECT SQUARE(@cx + g.DX - n.X) + SQUARE(@cy + g.DY - n.Y) + SQUARE(@cz + g.DZ - n.Z) AS D2) AS dd
    ) AS s
    WHERE  g.Pass = 2
    ORDER BY s.Fit + @SideEffectPenalty * s.Risk, g.L1, g.DX, g.DY, g.DZ;

    /* spread of good placements, and outcomes seen close to the answer */
    UPDATE #nb SET NearW = KernelW * EXP(-(SQUARE(X - @bx) + SQUARE(Y - @by) + SQUARE(Z - @bz)) / @twoR2);

    DECLARE @SdX FLOAT, @SdY FLOAT, @SdZ FLOAT, @Neff FLOAT, @PredEff FLOAT, @PredSE FLOAT;
    SELECT  @SdX = SQRT(SUM(g.W * SQUARE(X - @bx)) / NULLIF(SUM(g.W), 0)),
            @SdY = SQRT(SUM(g.W * SQUARE(Y - @by)) / NULLIF(SUM(g.W), 0)),
            @SdZ = SQRT(SUM(g.W * SQUARE(Z - @bz)) / NULLIF(SUM(g.W), 0)),
            @Neff = SQUARE(SUM(KernelW)) / SUM(SQUARE(KernelW)),
            @PredEff = CASE WHEN SUM(NearW) > 1e-9 THEN SUM(NearW * EfficacyPct) / SUM(NearW) END,
            @PredSE  = CASE WHEN SUM(NearW) > 1e-9 THEN SUM(NearW * SideEffectScore) / SUM(NearW) END
    FROM    #nb
    CROSS APPLY (SELECT KernelW * CASE WHEN NetBenefit > 0 THEN NetBenefit ELSE 0 END AS W) AS g;

    /* 4. into the patient's own scan */
    DECLARE @sgn FLOAT = CASE @Hemisphere WHEN 'L' THEN -1.0 ELSE 1.0 END;
    DECLARE @tx FLOAT = @sgn * @bx, @ty FLOAT = @by, @tz FLOAT = @bz;
    DECLARE @nx FLOAT = @tx / @sx + @mx, @ny FLOAT = @ty / @sy + @my, @nz FLOAT = @tz / @sz + @mz;
    DECLARE @shift FLOAT = SQRT(SQUARE(@bx - @ax) + SQUARE(@by - @ay) + SQUARE(@bz - @az));

    IF @Save = 1
    BEGIN
        INSERT nt.Suggestion (ScanId, TargetId, Hemisphere, K, SideEffectPenalty,
                              TemplateX, TemplateY, TemplateZ, NativeX, NativeY, NativeZ,
                              SdX, SdY, SdZ, AtlasShiftMm, EffectiveN,
                              PredictedEfficacy, PredictedSideEffect, FitRmse)
        VALUES (@ScanId, @TargetId, @Hemisphere, @K, @SideEffectPenalty,
                @tx, @ty, @tz, @nx, @ny, @nz, @SdX, @SdY, @SdZ, @shift, @Neff,
                @PredEff, @PredSE, SQRT(@Fit));
        SET @SuggestionId = SCOPE_IDENTITY();
    END;

    SELECT  @SuggestionId AS SuggestionId, @ScanId AS ScanId, @TargetId AS TargetId,
            @Hemisphere AS Hemisphere, @K AS K,
            @tx AS TemplateX, @ty AS TemplateY, @tz AS TemplateZ,
            @nx AS NativeX, @ny AS NativeY, @nz AS NativeZ,
            @sgn * @ax AS AtlasX, @ay AS AtlasY, @az AS AtlasZ,
            @SdX AS SdX, @SdY AS SdY, @SdZ AS SdZ,
            @shift AS AtlasShiftMm, @Neff AS EffectiveN,
            @PredEff AS PredictedEfficacy, @PredSE AS PredictedSideEffect,
            SQRT(@Fit) AS FitRmse, @h AS Bandwidth;

    SELECT  Rn, ImplantId, PatientId, ScanId, Hemisphere, X, Y, Z,
            EfficacyPct, SideEffectScore, PatientRating, NetBenefit, Dist, KernelW, NearW
    FROM    #nb
    ORDER BY Rn;
END;
GO

/* -------------------------------------------------------------------------------------
   Close the loop. @ContactsJson: [{"n":0,"x":..,"y":..,"z":..}, ...] template mm, signed X
   (post-op CT localisation). The patient becomes part of the evidence for everyone after.
   ------------------------------------------------------------------------------------- */
CREATE OR ALTER PROCEDURE nt.usp_RecordImplantOutcome
    @PatientId        INT,
    @TargetId         TINYINT,
    @Hemisphere       CHAR(1),
    @ImplantedOn      DATE,
    @ContactsJson     NVARCHAR(MAX),
    @ActiveContact    TINYINT,
    @AmplitudeMa      DECIMAL(4,1),
    @PulseWidthUs     SMALLINT,
    @FrequencyHz      SMALLINT,
    @MonthsPostOp     TINYINT,
    @EfficacyPct      FLOAT,
    @SideEffectScore  FLOAT,
    @PatientRating    TINYINT,
    @SuggestionId     INT = NULL,
    @ImplantId        INT = NULL OUTPUT
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;
    BEGIN TRAN;

    SELECT @ImplantId = ImplantId FROM nt.Implant
    WHERE PatientId = @PatientId AND TargetId = @TargetId AND Hemisphere = @Hemisphere;

    IF @ImplantId IS NULL
    BEGIN
        SET @ImplantId = NEXT VALUE FOR nt.seqImplant;
        DECLARE @px FLOAT, @py FLOAT, @pz FLOAT;
        SELECT @px = TemplateX, @py = TemplateY, @pz = TemplateZ
        FROM nt.Suggestion WHERE SuggestionId = @SuggestionId;

        INSERT nt.Implant (ImplantId, PatientId, TargetId, Hemisphere, ImplantedOn,
                           PlannedX, PlannedY, PlannedZ, SuggestionId)
        VALUES (@ImplantId, @PatientId, @TargetId, @Hemisphere, @ImplantedOn, @px, @py, @pz, @SuggestionId);

        INSERT nt.Contact (ImplantId, ContactNo, X, Y, Z)
        SELECT @ImplantId, j.n, j.x, j.y, j.z
        FROM OPENJSON(@ContactsJson) WITH (n TINYINT '$.n', x FLOAT '$.x', y FLOAT '$.y', z FLOAT '$.z') AS j;

        IF @SuggestionId IS NOT NULL
            UPDATE nt.Suggestion SET AcceptedBySurgeon = 1 WHERE SuggestionId = @SuggestionId;
    END;

    MERGE nt.Programming AS t
    USING (SELECT @ImplantId AS ImplantId) AS s ON t.ImplantId = s.ImplantId
    WHEN MATCHED THEN UPDATE SET ActiveContact = @ActiveContact, AmplitudeMa = @AmplitudeMa,
                                 PulseWidthUs = @PulseWidthUs, FrequencyHz = @FrequencyHz
    WHEN NOT MATCHED THEN INSERT (ImplantId, ActiveContact, AmplitudeMa, PulseWidthUs, FrequencyHz)
                          VALUES (@ImplantId, @ActiveContact, @AmplitudeMa, @PulseWidthUs, @FrequencyHz);

    INSERT nt.Outcome (ImplantId, MonthsPostOp, EfficacyPct, SideEffectScore, PatientRating)
    VALUES (@ImplantId, @MonthsPostOp, @EfficacyPct, @SideEffectScore, @PatientRating);

    UPDATE nt.Patient SET Status = 'Historical' WHERE PatientId = @PatientId;
    COMMIT;
END;
GO

/* -------------------------------------------------------------------------------------
   Leave-one-patient-out evaluation for a whole target, set-based.
   For every historical implant: rebuild its suggestion from everyone else, then score
   suggestion and plain atlas against the simulated true sweet spot.
   Result set 1: summary. Result set 2: per implant.
   ------------------------------------------------------------------------------------- */
CREATE OR ALTER PROCEDURE nt.usp_EvaluateLOO
    @TargetId           TINYINT,
    @K                  INT   = 40,
    @SideEffectPenalty  FLOAT = 100
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @ax FLOAT, @ay FLOAT, @az FLOAT, @Floor FLOAT, @Emax FLOAT, @Sigma FLOAT, @NoGo NVARCHAR(100);
    SELECT  @ax = AtlasX, @ay = AtlasY, @az = AtlasZ, @Floor = Floor, @Emax = Emax, @Sigma = SigmaMm, @NoGo = NoGo
    FROM    nt.Target WHERE TargetId = @TargetId;
    DECLARE @gx FLOAT = CAST(JSON_VALUE(@NoGo, '$[0]') AS FLOAT),
            @gy FLOAT = CAST(JSON_VALUE(@NoGo, '$[1]') AS FLOAT),
            @gz FLOAT = CAST(JSON_VALUE(@NoGo, '$[2]') AS FLOAT);
    DECLARE @twoS2 FLOAT = 2.0 * @Sigma * @Sigma, @twoR2 FLOAT = 2.0 * 1.5 * 1.5;

    SELECT e.ImplantId AS QId, e.PatientId, e.Hemisphere, e.X, e.Y, e.Z
    INTO   #q
    FROM   nt.vHistoricalEvidence AS e
    WHERE  e.TargetId = @TargetId;

    SELECT q.QId, nb.ImplantId, nb.X, nb.Y, nb.Z, nb.EfficacyPct, nb.SideEffectScore, nb.NetBenefit, nb.Dist,
           CAST(NULL AS FLOAT) AS KernelW
    INTO   #nb
    FROM   #q AS q
    JOIN   nt.vHistoricalEvidence AS qe ON qe.ImplantId = q.QId
    CROSS APPLY (
        SELECT TOP (@K) e.ImplantId, e.X, e.Y, e.Z, e.EfficacyPct, e.SideEffectScore, e.NetBenefit,
               VECTOR_DISTANCE('euclidean', e.Features, qe.Features) AS Dist
        FROM   nt.vHistoricalEvidence AS e
        WHERE  e.TargetId = @TargetId AND e.PatientId <> q.PatientId
        ORDER BY Dist, e.ImplantId
    ) AS nb;

    WITH b AS (SELECT QId, KernelW, Dist, AVG(Dist) OVER (PARTITION BY QId) AS H FROM #nb)
    UPDATE b SET KernelW = EXP(-SQUARE(Dist / CASE WHEN H > 0 THEN H ELSE 1.0 END));

    SELECT QId, SUM(KernelW) AS SumW, SQUARE(SUM(KernelW)) / SUM(SQUARE(KernelW)) AS Neff
    INTO #w FROM #nb GROUP BY QId;

    /* coarse */
    WITH s AS (
        SELECT n.QId, g.DX, g.DY, g.DZ, g.L1,
               SUM(n.KernelW * SQUARE(@Floor + (@Emax - @Floor) * EXP(-d.D2 / @twoS2) - n.EfficacyPct)) AS FitW,
               SUM(n.KernelW * n.SideEffectScore * EXP(-d.D2 / @twoR2)) AS RiskW
        FROM   #nb AS n
        CROSS JOIN nt.GridPoint AS g
        CROSS APPLY (SELECT SQUARE(@ax + g.DX - n.X) + SQUARE(@ay + g.DY - n.Y) + SQUARE(@az + g.DZ - n.Z) AS D2) AS d
        WHERE  g.Pass = 1
        GROUP BY n.QId, g.GridId, g.DX, g.DY, g.DZ, g.L1
    ), r AS (
        SELECT s.QId, @ax + s.DX AS CX, @ay + s.DY AS CY, @az + s.DZ AS CZ,
               ROW_NUMBER() OVER (PARTITION BY s.QId
                                  ORDER BY (s.FitW + @SideEffectPenalty * s.RiskW) / w.SumW, s.L1, s.DX, s.DY, s.DZ) AS rn
        FROM s JOIN #w AS w ON w.QId = s.QId
    )
    SELECT QId, CX, CY, CZ INTO #c FROM r WHERE rn = 1;

    /* fine */
    WITH s AS (
        SELECT n.QId, c.CX + g.DX AS FX, c.CY + g.DY AS FY, c.CZ + g.DZ AS FZ, g.L1,
               SUM(n.KernelW * SQUARE(@Floor + (@Emax - @Floor) * EXP(-d.D2 / @twoS2) - n.EfficacyPct)) AS FitW,
               SUM(n.KernelW * n.SideEffectScore * EXP(-d.D2 / @twoR2)) AS RiskW
        FROM   #nb AS n
        JOIN   #c  AS c ON c.QId = n.QId
        CROSS JOIN nt.GridPoint AS g
        CROSS APPLY (SELECT SQUARE(c.CX + g.DX - n.X) + SQUARE(c.CY + g.DY - n.Y) + SQUARE(c.CZ + g.DZ - n.Z) AS D2) AS d
        WHERE  g.Pass = 2
        GROUP BY n.QId, g.GridId, c.CX, c.CY, c.CZ, g.DX, g.DY, g.DZ, g.L1
    ), r AS (
        SELECT s.QId, s.FX, s.FY, s.FZ,
               ROW_NUMBER() OVER (PARTITION BY s.QId
                                  ORDER BY (s.FitW + @SideEffectPenalty * s.RiskW) / w.SumW, s.L1, s.FX, s.FY, s.FZ) AS rn
        FROM s JOIN #w AS w ON w.QId = s.QId
    )
    SELECT QId, FX, FY, FZ INTO #b FROM r WHERE rn = 1;

    /* score against the hidden truth (simulation only) */
    SELECT  q.QId AS ImplantId, q.PatientId, q.Hemisphere,
            SQRT(SQUARE(b.FX - t.TX) + SQUARE(b.FY - t.SweetY) + SQUARE(b.FZ - t.SweetZ)) AS SuggestErrMm,
            SQRT(SQUARE(@ax  - t.TX) + SQUARE(@ay  - t.SweetY) + SQUARE(@az  - t.SweetZ)) AS AtlasErrMm,
            SQRT(SQUARE(q.X  - t.TX) + SQUARE(q.Y  - t.SweetY) + SQUARE(q.Z  - t.SweetZ)) AS ActualContactErrMm,
            @Floor + (@Emax - @Floor) * EXP(-(SQUARE(b.FX - t.TX) + SQUARE(b.FY - t.SweetY) + SQUARE(b.FZ - t.SweetZ)) / @twoS2) AS SuggestEfficacy,
            @Floor + (@Emax - @Floor) * EXP(-(SQUARE(@ax  - t.TX) + SQUARE(@ay  - t.SweetY) + SQUARE(@az  - t.SweetZ)) / @twoS2) AS AtlasEfficacy,
            10 * EXP(-(SQUARE(b.FX - t.TX - @gx) + SQUARE(b.FY - t.SweetY - @gy) + SQUARE(b.FZ - t.SweetZ - @gz)) / (2 * 1.8 * 1.8)) AS SuggestSideEffect,
            10 * EXP(-(SQUARE(@ax  - t.TX - @gx) + SQUARE(@ay  - t.SweetY - @gy) + SQUARE(@az  - t.SweetZ - @gz)) / (2 * 1.8 * 1.8)) AS AtlasSideEffect,
            w.Neff AS EffectiveN
    INTO    #res
    FROM    #q AS q
    JOIN    #b AS b ON b.QId = q.QId
    JOIN    #w AS w ON w.QId = q.QId
    CROSS APPLY (SELECT ABS(st.SweetX) AS TX, st.SweetY, st.SweetZ
                 FROM nt.SimTruth AS st
                 WHERE st.PatientId = q.PatientId AND st.TargetId = @TargetId AND st.Hemisphere = q.Hemisphere) AS t;

    SELECT  @TargetId AS TargetId, @K AS K, @SideEffectPenalty AS SideEffectPenalty,
            COUNT(*) AS Implants,
            AVG(SuggestErrMm) AS MeanSuggestErrMm, AVG(AtlasErrMm) AS MeanAtlasErrMm,
            AVG(ActualContactErrMm) AS MeanActualContactErrMm,
            AVG(CASE WHEN SuggestErrMm < AtlasErrMm THEN 1.0 ELSE 0.0 END) AS FractionCloserThanAtlas,
            AVG(SuggestEfficacy) AS MeanSuggestEfficacy, AVG(AtlasEfficacy) AS MeanAtlasEfficacy,
            AVG(SuggestSideEffect) AS MeanSuggestSideEffect, AVG(AtlasSideEffect) AS MeanAtlasSideEffect
    FROM    #res;

    SELECT * FROM #res ORDER BY ImplantId;
END;
GO

I then generated some test data

and load some images.

The to run the app:

simulate the cohort (≈ 1–2 min): 760 implanted patients + 20 new referrals with MRI volumes

cd python
python generate_data.py

2. With SQL Server 2025

export NEUROTARGET_CONN="Driver={ODBC Driver 18 for SQL Server};Server=localhost;UID=sa;PWD=…;TrustServerCertificate=yes"

python load_sql.py # creates NeuroTarget DB, runs sql/01 + sql/02, loads everything

streamlit run app.py # choose "SQL Server 2025" in the sidebar

-- Useful T-SQL
DECLARE @id INT;
EXEC nt.usp_SuggestPlacement @ScanId = 1761, @TargetId = 1, @Hemisphere = 'R', @K = 40, @SuggestionId = @id OUTPUT;
EXEC nt.usp_EvaluateLOO @TargetId = 2, @K = 40;   -- set-based; ~40M row evaluations, allow a minute

Neurotarget 3D

Neurotarget — Plan

Neurotarget — Validation

App tabs

Plan (MRI): axial, coronal and sagittal slices through the suggestion, showing the atlas target and the planned lead trajectory.
- 3D: similar patients’ active contacts mapped into this brain, coloured by benefit and sized by similarity, with ventricles, lead, and a 1-SD spread ellipsoid.
- Similar patients: the neighbours, their weights and outcomes, and how benefit falls away with distance.
- Scan fingerprint: the 14 features and how this brain differs from the library.
- Validation: leave-one-out against the known truth.
- Close the loop: simulates surgery with realistic error, a 12-month outcome, and a matched atlas-aimed counterfactual, then records it.

A note to the bloke from twenty years ago

If I could lean over the shoulder of the version of me hunched over that first database at 2am, eye streaming, I’d tell him three things.

Your question was a good one. Where the electrode goes matters, and the people who went before you hold the answer.

You’ll need a lot more data than you’ve got. Be patient. The tools will come.

And building something, even something that will never go near an operating theatre, is a perfectly reasonable way to feel less helpless. It worked for me then. It still does.

The code, simulator and front end will be on my GitHub. Usual disclaimer, in bold because it matters: simulated data, not medical advice, please don’t let anyone drill holes in you because of a SQL query.

If you enjoyed this please give it some Claps.

Can a Database Rave? Making SQL Server Play Music

Sun, 27 Sep 2026 22:56:33 GMT

DROP THE BASE(TABLE) — I was so pleased with myself for coming up with this pun. Puns are definitely one level above a dad joke though.

I’ve spent the last few months doing daft things with SQL Server. Doom runs in it. The Amiga Boing ball bounces in it. Elite flew in it. I’ve had it pretending to be other hardware altogether. Every one of those was about pictures, or logic, or both.

What I’d never done was make it *make a noise*.

So that became the question. Not “can I store an MP3 in a `varbinary` column”. Obviously you can, and it’s about as exciting as a filing cabinet. The question was whether SQL Server itself could produce the sound: every sample, every drum hit, every note, worked out with set-based T-SQL, with Python doing nothing more than passing the bytes to the speakers.

I also had a target in mind. The Prodigy, “Smack My Bitch Up”.

I’ll come back to how that went.

I’m not the first to make SQL sing, and it’s only fair to say so. Ramiro Medina built a synthesiser in PostgreSQL in 2022 that played “Twinkle Twinkle Little Star”, and the ClickHouse folks made NoiSQL. But as far as I can find, nobody had made SQL Server play Amiga MODs, or had a database chop up a breakbeat. That’s where I wanted to go.

Rule one: SQL makes the sound, Python is just the speaker

I set myself one rule and stuck to it. Python is allowed to call a stored procedure, take the “varbinary” it gets back and hand it to the sound card. That’s all. No numpy doing the mixing behind my back. No “just this one filter in Python”. If it makes a sound and wasn’t me annoyed at myself for a very stupid bug then SQL Server made it.

The Python side is “pyodbc” plus “sounddevice”. It fetches two seconds of audio at a time on a background thread, so the next chunk is being rendered while the current one plays. If SQL Server can render faster than real time, you get music. If it can’t, you get silence and an embarrassing pause.

No pressure, then.

Why not The Prodigy, then?

I wanted “Smack My Bitch Up” as the finale. I thought about it properly and decided it was the wrong target, for a couple of reasons.

The first is how the record’s made. It’s built almost entirely from samples of other records: the drum break, the vocal, that riff. There’s no neat score underneath it to transcribe. To “play” it in SQL Server I’d really be replaying other people’s recordings.

The second is copyright. Those recordings aren’t mine to reproduce, in SQL or anywhere else.

So I went the route of all those struggling artists out there and start with creating something myself, my first step on the the music ladder. Synthesise sound from nothing. Then play proper tracker music. Then chop up a record I own and remix it.

I have realised that over the course of me writing these articles I have used terms that anyone born later than 2000 would not have a clue what I meant. For that I do not apologise. A record means a Vinyl record, but its also just means a track stored as an .mp3 or as as .wav file. Forget it — here is a graphic.

I regard myself as young until I look around and notice no one knows what I talking about half of the time.

Rung one: a synthesiser made of maths

A sound is just a list of numbers. At CD quality there are 44,100 of them every second. So the first job is a sample clock, and SQL Server 2022 onwards gives you one for free:

CROSS APPLY GENERATE_SERIES(GREATEST(ev.StartS, @StartSample), LEAST(ev.EndS, @last)) g

Every note in the song becomes a range of sample numbers. For each of them I work out the time, the phase, and what the oscillator’s doing at that instant. Classic synth waveforms turn out to be one-liners:

CASE ev.Waveform
 WHEN 'sine' THEN SIN(@twoPi * p.ph)
 WHEN 'square' THEN CASE WHEN f.fr < 0.5 THEN 1.0 ELSE -1.0 END
 WHEN 'saw' THEN 2.0 * f.fr - 1.0
 WHEN 'triangle' THEN 4.0 * ABS(f.fr - 0.5) - 1.0
 - kick: a sine whose pitch falls from 150 Hz to 45 Hz
 WHEN 'kick' THEN SIN(@twoPi * (45.0 * tt.t + 105.0 * 0.035 * (1.0 - EXP(-tt.t / 0.035))))
 - hat: noise minus the previous sample's noise
 WHEN 'hat' THEN 0.5 * (nz.w - nz1.w)
END

The hi-hat is my favourite line in the whole project. Real synths make hi-hats by filtering noise, and filters normally need the previous output. That’s recursion, which is exactly what set-based SQL hates. But if your noise is a pure function of the sample number (it’s an integer hash, not `RAND()`), then “noise now minus noise a moment ago” is a perfectly good high-pass filter. And you can compute it for every sample at once. No loop, no state, just a slightly smug `CROSS APPLY`.

The acid bassline does things the same way. Instead of a filter sweep, it’s a `SUM` over up to 40 harmonics, with the high ones fading as each note goes on. That’s the squelch you know from the 90s, done as an aggregate.

The nine sound shapes in rung one

The shape of every sound in the synth. The top row is the classic oscillators, one formula each. Then there’s noise, and the three drums built from sines, noise and a fade. The acid bass is shown twice: sharp and buzzy as the note starts, then rounded off 0.3 s later as the high harmonics fade.

Mixing is the most SQL thing imaginable:

SELECT n, SUM(v) AS x FROM #voice GROUP BY n

Every drum, every bass note, every stab, all added together per sample. One `GROUP BY`. That’s a mixing desk.

Getting the bytes out

The last hurdle was turning a table of numbers into a WAV file, and I’ll admit this bit gave me a laugh. WAV files are little-endian. SQL Server’s `CAST(int AS binary(4))` is big-endian. So every sample has to be byte-swapped. Then I needed hundreds of thousands of two-byte fragments glued together, in order, quickly.

`STRING_AGG` doesn’t do binary. It does do strings, though:

SELECT @hex = STRING_AGG(
 CAST(CONVERT(varchar(4),
 SUBSTRING(CAST(i AS binary(4)), 4, 1) + SUBSTRING(CAST(i AS binary(4)), 3, 1), 2)
 AS varchar(max)), '')
 WITHIN GROUP (ORDER BY n)
FROM q;
SET @Pcm = CONVERT(varbinary(max), @hex, 2);

Each sample is byte-swapped, turned into four hex characters and stitched together in sample order, and then the whole string is converted back into binary. Stick a 44-byte RIFF header on the front, also built in T-SQL, and you’ve got a WAV file. It’s the most roundabout way of writing bytes I’ve ever used, and it’s quick.

A spectrogram of the first tune SQL Server synthesised

”Set-Based Breakbeat”: the first thing SQL Server ever played for me. You can see the drums come in, then the acid bass, the stabs, the break with its noise riser, and the drop.

The first time it played I just sat there grinning. It’s a database. It went “boom tss”. On my machine it ran at roughly real time, which is a sentence I never expected to write about a stored procedure.

Rung two: a ProTracker player

Synthesis is clever. But what I actually love is tracker music: Amiga MODs, four channels, 8-bit samples, patterns scrolling up the screen. So rung two was a proper ProTracker `.mod` player. And not “Python reads the file and SQL plays it”. SQL Server reads the file.

Here’s a lovely surprise. The Amiga ran on a Motorola 68000, which is big-endian. MOD files are big-endian. SQL Server’s binary-to-int cast is big-endian. Thirty-odd years apart, they agree. Reading the header needs no byte-swapping at all:

CAST(SUBSTRING(@Mod, O + 22, 2) AS int) * 2 AS HdrLen, - sample length, in words
CAST(SUBSTRING(@Mod, O + 25, 1) AS tinyint) AS Vol,
CAST(SUBSTRING(@Mod, O + 26, 2) AS int) * 2 AS LpStart, - loop start

It’s WAV, on the way out, that’s the odd one out.

The one place I gave in and wrote a loop

A MOD isn’t a flat list of notes. Effects change the speed mid-song. Commands jump to another position, or break out of a pattern early. Slides carry on from one row into the next. That’s genuinely sequential. The “conductor” walks the song one row at a time: a few thousand rows for a typical module.

What I refused to do was loop over “ticks” or “channels”. Within a row, every effect has a closed form in the tick number `k`. A volume slide is just “start + k × rate, clamped”. An arpeggio is “which third of the chord are we on, `k % 3`”. So each row’s ticks, for every channel, come out of one `INSERT…SELECT`:

CASE WHEN c.Effect IN (5, 6, 10) - Axy volume slide
 THEN GREATEST(0, LEAST(64, ch.Volume + k.value * CASE WHEN xy.X > 0 THEN xy.X ELSE -xy.Y END))
 …
CASE WHEN c.Effect = 0 AND c.Param > 0–0xy arpeggio
 THEN pb.P / POWER(CAST(2 AS float), (CASE k.value % 3 WHEN 0 THEN 0 WHEN 1 THEN xy.X ELSE xy.Y END) / 12.0E0)

That was the most important decision in the whole project. Be procedural only where the problem genuinely is, and set-based everywhere else.

The mixer, meanwhile, is pure sets. Each tick of each channel says “play sample `s`, starting at position `p`, stepping `d` points per output sample, at volume `v`”. The position of any output sample is then just arithmetic, and a looping sample is just a modulo:

CROSS APPLY (SELECT CAST(FLOOR(tk.PosAtStart + (g.value - tk.StartSample) * tk.Step) AS bigint) AS P) p
CROSS APPLY (SELECT CAST(CASE
 WHEN tk.LoopLength > 2 AND p.P >= tk.LoopStart + tk.LoopLength
 THEN tk.LoopStart + (p.P - tk.LoopStart) % tk.LoopLength
 WHEN p.P < tk.LengthBytes THEN p.P
 END AS int) AS Idx) m

It uses the nearest sample point, with no smoothing between points. That isn’t laziness. It’s exactly what the Amiga’s sound chip did, and it’s where the crunch comes from.

“It takes forever to load”

The first real module I threw at it was `lynx_-_rave_vibes.mod` from The Mod Archive. The demo module had loaded fine. This one sat there. And sat there.

The culprit was the conductor. Thousands of iterations, each writing to permanent tables, each committing, each flushing to the log. On top of that, SQL Server kept recompiling the statements that touched the small temp tables inside the loop. The fix: switch the loop’s working tables to table variables, buffer everything in memory, and write it all out in one go, in one transaction. Same logic, a fraction of the time.

It’s the oldest lesson in the book, and I still had to learn it again. Row-by-row isn’t what kills you. Row-by-row *commits* are.

Rung three: the sampler

Once MODs were playing, the obvious next wish was the records I actually wanted to hear: early-90s hardcore like Soundcorp’s “Dreamfinder” and the Q-Project’s “Champion Sound”. I went hunting for tracker versions and came up empty.

Then it clicked. Those records were *made* on samplers. People chopped breakbeats into slices on an Akai and re-triggered them. So instead of finding a MOD of the record, why not have SQL Server do what the Akai did? Load a WAV of a record you own, find the beat, chop it into slices, and turn the slices into instruments in a tracker kit. Then the MOD engine I’d already built can play it, re-order it and re-mix it.

Finding the beat, with a self-join

Beat detection is a lovely set-based problem:

1. Measure the loudness in 5-millisecond windows.
2. Mark every sudden jump in loudness as an onset: a drum hit, near enough.
3. Auto-correlate the onsets. In other words, join the window table to itself at every possible time offset, and see which offset lines up the most hits.

SELECT g.Lag, SUM(a.Onset * b.Onset) AS R
FROM #lags g
JOIN synth.SamplerEnvelope a ON a.SourceId = @SourceId
JOIN synth.SamplerEnvelope b ON b.SourceId = @SourceId AND b.Win = a.Win + g.Lag
GROUP BY g.Lag;

And it confidently told me my 141.7 BPM test track was 94.5 BPM.

Plain auto-correlation picks two-thirds of the tempo; the harmonic score gets it right

Breakbeats are sneaky. The kick drum in a typical break lands every one-and-a-half beats, so “every 1.5 beats” lines up more hits than “every beat” does. Hence two-thirds of the real tempo. The fix is to ask a better question: does this tempo also line up at two beats and at a whole bar? A true beat does; a phantom one doesn’t. Score each candidate as `R(L) + R(2L)/2 + R(4L)/4`, and 141.7 wins clearly.

Then a final brute-force pass tries 101 tempos just either side of the winner, at every possible starting offset, against every beat in the track. On paper that sounds expensive. In SQL it’s just another join. It pins the tempo to about a hundredth of a percent and tells you where the first beat falls.

Loading a track: 135.000 BPM detected, first beat at 0.000 s, 50 slices. Then a bar-by-bar loudness map, drawn with `REPLICATE(‘#’, …)`, to choose a section, and a kit built from 48 slices.

The kit comes with a few patterns straight away, so you can hear what it’ll do:

“Straight:” the section played back exactly as it was.
“Stutter:” every fourth beat, the bar’s first slice is re-triggered and chopped short.
“Chipmunk:” every slice an octave up, twice as fast. That’s *the* hardcore vocal trick of 1992.
“Half-time:” every slice an octave down.

After that it’s remix-by-”EXEC”:

EXEC synth.PutNote @ModId = 5, @Pattern = 2, @Row = 12, @Channel = 1,
 @Sample = 5, @Note = 'C-3', @Effect = 0xE, @Param = 0xC2;
EXEC synth.SetOrder 5, '0,1,2,2,3';
EXEC synth.BuildModTimeline 5;

Issues Encountered

Nothing this silly works first time. A few favourites:

“smallint” × “tinyint” = “smallint”. The mixer multiplied each sample point by its volume. That was fine for 8-bit Amiga samples: 127 × 64 fits easily. The sampler’s 16-bit slices go up to about 32,000 × 64, and SQL Server kept the result as a “smallint”. Arithmetic overflow. The fix is a “CAST(… AS float)” in the right place. A thirty-year-old reminder about data type precedence.

“The player that waited forever.” When a background render failed, the error died quietly in its thread, and the player sat there waiting for audio that was never coming. Now it tells you what went wrong and stops.

“The query plan that mixed everything, for every sample.” The first version of the mixer was one big elegant CTE. With a kit of nearly half a million sample points, the optimiser settled on a plan that did far more work than it should. Splitting it into four steps, each landing in an indexed temp table, gave it real row counts to plan with. It’s a bit less elegant and an awful lot faster.

“The silent sampler” My favourite. The kit built fine and the tracker scrolled along beautifully, but there wasn’t a sound. I had diagnostic queries lined up for the slice data, the tick table and the mixer, ready to take the whole thing apart.
SQL Server was making music the whole time. It was going to my headphones, sitting on the desk, and not to the speakers. After all the byte-swapping, overflows and query plans, the last bug was in Windows Sound settings.

A graphic equaliser, because of course

Once it played, I wanted to “see” it play. So the Python window became a dashboard. The tracker scrolls with the current row lit up. There’s a 20-band graphic equaliser with falling bars and peak-hold markers, and left and right level meters that show the Amiga’s hard left-right-right-left panning. Next to it sits a panel of live SQL Server statistics.

The dashboard playing the sampler kit, mid-stutter

The dashboard playing the demo module

My favourite detail on that panel is that it doesn’t need VIEW SERVER STATE. A session can always read its own row in `sys.dm_exec_sessions`. So after each chunk, the renderer asks SQL Server how much CPU and how many reads it has just spent making the last two seconds of music.

“RenderMod” also reports its own work counts: ticks, sample points, and how many lookups it joined. Watching that realtime figure while the bars bounce is strangely gripping. Every frame is a race between the query and the sound card.

So, can a database rave?

Yes. Badly, loudly and with enormous enthusiasm.

What I’ll take away from it isn’t really the audio. It’s the decisions. Write the whole thing as sets, and give in to a loop only where the music is genuinely sequential. Don’t chase a record that isn’t yours to copy; build the tools to make your own. And when something’s slow, look for the commits before you blame the maths.

The Prodigy will have to wait. But SQL Server can now chop up a breakbeat, pitch the vocal up an octave and play it back through a graphic equaliser. Honestly, the 1992 version of me would have been delighted.

The full code, including the synth, the MOD player, the sampler and the dashboard is now available: https://github.com/sqlartist/SQL-SYNTH

Credits

“Rave Vibes” by Lynx (`lynx_-_rave_vibes.mod`), from “The Mod Archive”, [modarchive.org](https://modarchive.org). Thank you to the Archive and the artist for keeping this music alive.

“Set-Based Breakbeat”, the test module `setbased.mod` and the sampler test audio are all original, made for this project.

Prior art worth a look: Ramiro Medina’s [How I built a music synthesiser using SQL](https://medium.com/@ramiromedina/synthesising-music-using-only-declarative-sql-917bf528ad49) (PostgreSQL) and [NoiSQL](https://github.com/ClickHouse/NoiSQL) (ClickHouse).

If you enjoyed this audio extravaganza then please gives it a huge round of applause and hit the Clap button.

Forget Graham, the Car Crash Man, Meet Nigel, the Data Man

Mon, 21 Sep 2026 17:25:26 GMT

I should have gone with Unlimnited Trevor’s but even I thought that was too much

Graham is difficult to forget. Created for Australia’s Transport Accident Commission by artist Patricia Piccinini, working with trauma surgeon Christian Kenfield and road safety engineer David Logan, he imagines a human body better equipped to withstand a car crash. His enlarged skull, absent neck and cushioned chest make our physical vulnerability visible. Each strange feature has a purpose. [1]

This is Graham, a fantastic research project which can be found at www.meetgraham.com.au

I would like to introduce someone with a different set of problems. Meet Nigel, the Data Man.

No one will know its actually me, its not like I put myself into every single story.

Nigel lives at a point in the future when the excitement about artificial intelligence has subsided. It has become ordinary infrastructure. People still complain about it, naturally, but mostly because an update has arrived at an inconvenient moment.

Every human receives a device roughly the size of a 50p coin. It contains all publicly available human knowledge, the index needed to search it and powerful local language models to help make sense of it. New material arrives daily. Between updates, the system works independently of the network.

That is the premise of our thought experiment. Nigel’s alterations are imagined engineering, rather than a prediction that natural selection will eventually produce a USB socket. His body gives us a way to examine what this future would demand of us.

This is a little personal. I already live with a deep brain stimulator, implanted to help manage chronic cluster headaches after other treatments had failed. I wrote about the experience in The Six Billion Dollar Man, including the rather unusual business of having to recharge myself. [6]

That implant is a medical treatment. Nigel’s storage, replaceable thinking modules and mental backups remain imagined technologies. But the experience of depending on something electronic inside the body is already part of my everyday life.

I’m already part way there. I just need an upgrade or three.

At first glance, he looks reassuringly ordinary. Then you notice the service seam above his ear, the optical connection in his wrist and the faint amber colouring spreading across his forearms. Nigel is nearly full. He has been meaning to sort out his photographs.

Start with the small capsule beneath his breastbone. Inside it sits the public archive, separate from his private records. The resemblance to Douglas Adams’s The Hitchhiker’s Guide to the Galaxy is deliberate: a portable guide to an impossibly large world, available whenever its owner needs an explanation. Nigel has simply dispensed with the inconvenience of carrying the book.

The Beginning…

There is real science behind the attraction of very small storage. In 2017, the DNA Fountain researchers reported successful retrieval at a density of 215 petabytes per gram of DNA, using an experiment that stored about 2.14 megabytes. The distinction matters. They demonstrated remarkable density in the storage material; they did not build a device containing 215 petabytes, a fast reader and a computer in one gram. [2]

Multiplying that density by the weight of a coin does not tell us what fits inside a working implant. Nigel also needs machinery to read and write the data, error correction, processing, working memory, power and protection from damage. “The entire public internet” would need a definition, too: a dated collection of accessible material, with decisions about duplication, changing pages and media. Public availability alone would not settle permission to redistribute it.

Nigel’s coin therefore represents several breakthroughs at once. We can imagine the destination, but we cannot responsibly put a delivery date or a consumer price on it. A prediction of “twenty or thirty years” would be a guess dressed as a product roadmap.

The independence is easier to recognise. Language models can already run locally on suitable computers. The ambitious leap is giving everyone vastly greater capability in a tiny, affordable device that can operate safely inside a body. [3]

Nigel could ask a question without sending it to a remote AI service. He could work through an idea on a train with no signal, consult his archive during an outage or learn without somebody else keeping a record of every question. Local operation would help protect those conversations, although compromised software, physical access and careless sharing would still matter.

His archive and his AI would have distinct jobs. A model’s learned parameters are not a faithful filing cabinet containing every source. Nigel would need the underlying documents and an index, with the model retrieving material and explaining it. Research on retrieval-augmented generation already explores this combination. [4]

This brings us to his eyes. Nigel’s visual display can show where an answer came from, when the source was updated and whether other sources disagree. He can inspect the evidence before acting. He has access to humanity’s accumulated mistakes as well as its discoveries; enormous capacity has not relieved him of the need for judgement.

His system also needs to say “I don’t know”. Installing the internet in a person should not mean installing the internet’s confidence.

Follow his arm down to the wrist. Beneath a protective flap is a high-speed optical interface. When Nigel rests his hand on a compatible surface, it can deliver updates and, with his permission, exchange selected files. Ordinary contact with another person does not grant access. A handshake remains a handshake.

The port is there because storage capacity and transfer speed solve different problems. To illustrate the scale, imagine a daily update of one petabyte. At one terabyte per second, transferring it would take 16 minutes and 40 seconds. At one gigabit per second, the same transfer would take about 93 days. These calculations use decimal units and ignore overheads; the update size is an illustration, not an estimate of the internet’s daily growth.

Even Nigel cannot finish Tuesday’s update if Monday’s takes three months. His system would receive changed material and revised index entries, while common updates could be distributed through shared local hubs or broadcast channels. Model upgrades could arrive separately. Adding a new document should not require retraining or replacing the entire model, and a change to model weights would not necessarily make a small download.

“Offline” would mean independent between updates. Fresh information must still arrive through some connection or physical delivery. Nigel could decline it, but his guide would then need to display its age. Yesterday’s knowledge can be useful, provided nobody mistakes it for today’s.

The shallow folds around his ribs and upper back are the cooling system. Reading a huge archive and running powerful models require energy, and the heat must go somewhere. Nigel has an external charging connection and a strict limit on how warm his internal components may become. For difficult work, he might still sit at a powered desk that supplies extra cooling. The coin has not abolished the rest of the computer.

Move up to the seam above his ear. This is where the idea becomes more unsettling: Nigel has swappable brains.

Initially, these would be replaceable computing modules. He might fit one suited to mathematical work, another to language, or replace a failed processor without disturbing his private archive. The design would need to preserve his personal settings and keep unfamiliar modules away from information they had no reason to access.

Swapping a biological brain, or transferring the person living in it, is a much greater leap. Calling both operations an upgrade conceals the difficult part. A new processor does not establish that Nigel’s memories, character or conscious experience can be moved with it.

For that possibility, look at the protected compartment near the base of his skull: the emergency backup. In our imagined future, it preserves recoverable records of Nigel’s mental life. An encrypted copy is also held somewhere physically separate. A backup stored only inside Nigel would have an obvious weakness if something happened to Nigel.

We can already preserve someone’s writing, photographs, recorded voice and recollections. Brain mapping is also an active field of research. A complete wiring diagram of an adult female fruit-fly brain was published in 2024, containing 139,255 neurons and more than 50 million chemical synapses. The work has moved beyond the earlier milestones of worms and fly larvae. [5]

A wiring diagram, however, does not by itself establish how to capture and restore a person. We do not yet know everything a faithful reconstruction would have to preserve, or how to read it from a living human brain without destroying what we hope to save. A convincing digital imitation would not prove continuity of consciousness. There is no established timetable for a restorable human mind.

That uncertainty belongs in Nigel’s story. It makes his backup more interesting. Suppose two restored versions remembered the same childhood and both believed they were Nigel. Their shared history would not settle which one, if either, continued the original experience. The familiar language of copying files starts to strain when the files can disagree with you.

Even a less ambitious personal archive would deserve careful design. What happens to the hours since the last backup? Can a corrupted update damage every copy? Who tests recovery? Anyone who has managed important data will recognise the distinction between having a backup and knowing it can be restored. Nigel would have unusually strong reasons to care.

Now return to his skin. An engineered pigment layer shows storage use, synchronisation and faults through adjustable colours and patterns, independently of his natural complexion. As space runs low, amber bands deepen across his forearms. A small display gives the exact figures. For once, “you look a bit overloaded” is a useful diagnosis.

The display can be hidden. A person should not have to announce a technical problem to everyone in the supermarket. The more disturbing version of the display appears when Nigel misses a payment.

Imagine his wrist showing: “Your personal archive will be deleted in 72 hours. Please update your payment method.”

This is a deliberately bleak commercial possibility. The core device might have been issued to everyone, while additional capacity, maintenance or recovery services became subscriptions. A supplier could still impose an expiry condition on locally stored data. It would not need a permanent connection if the restriction had been installed in advance.

Nigel’s red warning skin would expose a failure of ownership. If missing a bill could erase a person’s memories, whoever controlled the account would hold extraordinary power over that person. Giving everyone the hardware would have achieved very little if continued access to their own life remained conditional.

I would give Nigel permanent control of his personal records, with usable export formats, independent recovery and a baseline service that survives financial difficulty. Optional services could stop when a subscription ends. His own memories should remain accessible. The hardware must distinguish those cases before somebody decides that deletion is a convenient billing feature.

There is one more control on his chest: a physical privacy switch. Nigel chooses what may leave the device. He can share a piece of public knowledge without sharing his reason for looking it up, and send a file without handing over his entire history. Emergency access would be limited to information he had chosen in advance, rather than treating illness as permission to inspect everything.

He would also need some ability to let experiences fade. A permanent archive of every conversation could burden the person carrying it and intrude on everyone they met. Nigel’s improved memory must leave room for other people’s consent, and for his own decision that some moments do not need to become records.

For me, the appeal of Nigel reaches beyond convenience. Even a life lasting seventy to a hundred years can feel short when so much of it is spent earning enough to live, dealing with illness or worrying about what comes next. By the time we have accumulated experience we might use well, there may be less freedom and time to use it than we had hoped.

A longer life would not automatically produce a wiser person, and a short life can have an enormous impact. Still, I find the loss of so much hard-won experience troubling. Preserving more of it could help people teach, create and contribute for longer, even if preserving the conscious person remains beyond us.

Nigel would need time to use his knowledge. If an implant simply made him available for more work, more monitoring and more demands, he might know everything and have no afternoon to himself.

Graham’s imagined anatomy makes us look again at the roads and vehicles around him. Nigel invites the same scrutiny of the systems we are building around information. His backup compartment, cooling folds and warning skin make their requirements visible. They also give us something concrete to challenge before those requirements become ordinary.

Perhaps Nigel’s most valuable feature is the small switch on his chest. He presses it, closes the connection and goes outside. The world’s knowledge can wait until he has finished his tea.

Nigel’s interactive body tour

These are imagined features, not a medical or engineering specification. Having said that the most often asked question about my own brain implant is “How do you charge it?”

Breastbone / The knowledge capsule / A 50p-sized home for the public archive, search index and local models. Nigel carries the library; he still has to judge the answers.

Eyes / The evidence display / See the source, its date and any disagreement before acting. Unlimited storage does not make every answer true.

Wrist /The high-speed interface / Receive updates or share selected files through a protected optical port. Every exchange needs permission. Handshakes remain social.

Its a good job no one realises that I’m at actually Nigel

Ribs and upper back / Power and cooling / Extra cooling surfaces manage the heat of computation. Heavy work may require a powered desk. Thinking has an energy bill.

Above the ear / Swappable brains / Replace specialist computing modules while protecting the private archive. Transferring a biological mind remains a much larger, unresolved idea.

Base of the skull / The emergency backup / Preserve recoverable personal records, with another copy stored elsewhere. A restored archive would not by itself prove that the person survived.

Forearms / The capacity display / An adjustable pigment layer shows storage use and system health. Colours are paired with patterns and readable numbers. Nigel can hide the display.

Inner wrist / The payment countdown / A fictional warning counts down to deletion after a missed bill. This deliberately bad design asks who really owns Nigel’s memories.

Chest / The privacy switch / Nigel controls what leaves his device and can close the connection. Access to public knowledge does not require access to his private life.

Research behind the article

Sources support the present-day science and Graham’s background. Nigel’s anatomy, service arrangements and dialogue are speculative. No date is claimed for a practical coin-sized knowledge implant or a restorable human mind.

1. Graham — Science Gallery Melbourne
Graham’s creators, commission and imagined protective anatomy.

2. Erlich and Zielinski — DNA Fountain enables a robust and efficient storage architecture
Science, 2017. The experiment encoded about 2.14 MB and reported a physical DNA storage density of 215 PB per gram. The density excludes a complete reader, writer and computer.

3. Ollama — FAQ
Documents local model operation and the option to disable cloud features. Local execution is already possible; Nigel’s imagined capabilities and form factor are speculative.

4. Lewis and colleagues — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
2020. Explores combining a language model with an explicit, retrievable knowledge store.

5. Dorkenwald and colleagues — Neuronal wiring diagram of an adult brain
Nature, 2024. The adult female fruit-fly brain reconstruction contains 139,255 neurons and more than 50 million chemical synapses. It is a structural map, not a demonstrated backup of a conscious individual.

6. Trevor Dwyer — The Six Billion Dollar Man
Medium, 5 September 2026. The author’s account of receiving a deep brain stimulator for chronic cluster headaches and living with the implanted device.

Transfer calculations use decimal units: 1 PB = 1,000 TB; 1 byte = 8 bits. At 1 TB/s, 1 PB takes 1,000 seconds. At 1 Gbit/s, it takes 8,000,000 seconds, or approximately 92.6 days. These are ideal transfer times without protocol overhead, contention or processing delays.

Footnote

Due to a glitch in the Brain Backup & Restore program and having the entire world’s information available to him…

One Nigel became two. Two became a support ticket. By lunchtime, the queue for tea had reached orbit.

Every Nigel insisted he was the original.

Technical support recommended restoring from backup.

Unfortunately, technical support was also Nigel.

Feels a bit like Rimmerworld

If you enjoyed a little bit of Nigel naked then please give it a thunderous round of Claps, if not and you want the real thing, I’m afraid it will require hitting the Clap button as well.

ZX Spectrum in SQL & Daley Thompson’s Decathlon

Fri, 18 Sep 2026 21:54:56 GMT

I was not going to be writing the dis-assembler myself.

A 48k Sinclair ZX Spectrum running inside SQL Server 2025. Then loading “Daley Thompson’s Decathlon” (Ocean, 1984) from its Original Micro-drive cartridge.

It took a huge effort to get to this point

This was a computer I did not own, having established in other articles I had a Commodore 16, so once again I only saw this a few times but it was the closest you could get to Konami’s Track & Field, a classic Joystick Waggler and always in the local Youth Centre or Chip shop.

Eventually it did come out for the Commodore 16 but restricted from ten events down to six and because Decathlon literally means 10, it was renamed to “Daley Thompson’s Star Events”.

One of the absolute best games ever made, it must have earned so much money for manufacturers of joysticks. More joysticks were broken by this game than I think any other.

I didn’t have a joystick…

Sinclair 48k ZX Spectrum computer (1982) —Photo: Bill Bertram

I started this idea wanting to emulate this game based on the Commodore 64, I very quickly changed my mind and decided to emulate the hardware for the ZX Spectrum to be run inside Microsoft SQL Server 2025. This would then allow me to load up the Copy Protection-less version of the game stored on a micro-drive.

In my life and without a doubt, millions of middle aged humans and a few Chimpanzees around the world, in fact anyone that remembers this era of Computing, will know the exact sound of a Computer Game loading from a Cassette. It was similar to an old school modem.

But not just that era, their parents will know that sound, and their neighbours will know that sound. It was hated except for a very specific group of people — siblings. They could hear you loading a game a mile away whilst still at school and would rush home just so they could open the front door the wrong way.

Depending on your relationship with your siblings meant one of two things happened. If you got on well they would sit and watch “YOU” play for hours all the time expecting “THEIR” turn would be coming up

Yes, of course you’ll be next…

But if you didn’t get along as nicely as your family thought you did, well they knew the exact moment to “breathe funny” or “dance up the stairs” which caused the loading of that game to fail.

By simply existing in this world, they would cause some level of Quantum Fluctuation and 100% would make the loading of that cassette fail an indescribable number of times.

When I said I was bored during these years in an earlier story on Medium, it wasn’t quite the truth, I would go a step further with the 1980’s and specifically the loading of any software onto any computer back then and say mummification was closer to the feeling and these cassettes loaders did not help one bit.

They would hypnotise you with their weird sounds and because of your sister deliberately flapping her arms “hoping” to cause “The Butterfly Effect” from another part of the house and making me “start all over again” for the 100th time

I̶ ̶n̶o̶w̶ ̶k̶n̶o̶w̶ ̶t̶h̶i̶s̶ ̶w̶a̶s̶ ̶i̶n̶ ̶f̶a̶c̶t̶ ̶p̶r̶e̶p̶a̶r̶a̶t̶i̶o̶n̶ ̶f̶o̶r̶ ̶b̶̶̶e̶̶̶i̶̶̶n̶̶̶g̶̶̶ ̶̶̶m̶̶̶a̶̶̶r̶̶̶r̶̶̶i̶̶̶e̶̶̶d̶̶̶.̶ ̶T̶h̶e̶ ̶e̶x̶a̶c̶t̶ ̶m̶o̶m̶e̶n̶t̶ ̶y̶o̶u̶ ̶s̶t̶a̶r̶t̶ ̶t̶o̶ ̶d̶o̶ ̶s̶o̶m̶e̶t̶h̶i̶n̶g̶ ̶f̶o̶r̶ ̶y̶o̶u̶r̶s̶e̶l̶f̶ ̶o̶r̶ ̶a̶ ̶h̶o̶b̶b̶y̶ ̶e̶v̶e̶n̶ ̶a̶n̶d̶ ̶a̶ ̶c̶u̶r̶i̶o̶u̶s̶ ̶l̶o̶v̶i̶n̶g̶ ̶p̶a̶r̶t̶n̶e̶r̶ ̶a̶s̶k̶s̶ ̶”̶W̶h̶a̶t̶ ̶y̶o̶u̶ ̶u̶p̶ ̶t̶o̶?̶”̶

(Which is why Men die earlier than women — they give up :))

But this was going to be far easier. Why?

Because I am going to use a Micro-drive, not a cassette. Micro-drives were as rare as the 3” floppy disk used in an Amstrad PCW8256 machines. (The world standard was 3.5” floppy disks) but good old Sir Alan Sugar must have got hold of a job lot from the back of a van and built his genuinely incredible machine around these disks.

Micro-drives do not have the brutal Copy Protection software embedded into many Game Cassette tapes from that era.

Side Note: I’m 53 and never once heard of copy protection on a cassette before this project. People used to tape music all the time. But now it makes a bit more sense and perhaps it wasn’t the nightmare siblings of this world causing these load failures, perhaps it was the copy protection kicking in due to you using “a friends copy of the game”.

First, the honest bit about firsts

I nearly opened this with “nobody has ever done this.” Then I checked, which I recommend doing before you put a claim like that on the internet.

Peter Negenborn built an Intel 8080 emulator in T-SQL and ran CP/M on it.

He emulated the Altair 8800’s processor as stored procedures, emulated the
serial ports, ran Altair BASIC — Microsoft’s first product — and then
emulated the floppy controller with the disk image sitting in a table so it
could boot a real operating system.

[Running an 8080 emulator (and CP/M) on SQL Server](https://www.linkedin.com/pulse/running-8080-emulator-cpm-sql-server-peter-negenborn)

That’s the same shape as what I’ve done, and the 8080 is the Z80’s direct
ancestor. So no, not a first, and I’m glad to be in that company rather than
alone in a field.

As far as I can tell nobody’s done a ZX Spectrum this way, and nobody’s
loaded a commercial game off its original Micro-drive cartridge into a
database and played it. But absence of search results isn’t proof, and
somebody probably did a 6502 in PL/SQL in 2009 and never told anyone.

Why the Spectrum matters, and why it’s a good target

Clive Sinclair built the world’s first slimline pocket calculator in 1972,
then spent the early eighties making computers cheap enough that ordinary families bought them. The ZX80 was the UK’s first mass-market home computer under £100. The ZX Spectrum followed in 1982–48K of RAM, a Z80 running at 3.5MHz, and those rubber keys.

It did something no spec sheet captures. It put a programmable machine into hundreds of thousands of British living rooms at a price that didn’t require a family conference, and a generation of us learned to code on it because there was nothing else to do with the thing. The British games industry came out of those bedrooms more or less directly. Sinclair was knighted in 1983 and died in 2021, aged 81.

The engineering is also *frugal* in a way that makes it a lovely emulation
target. There’s no video chip worth the name. The screen is a bitmap sitting
in RAM at 0x4000, with a daft interleaved layout that saves a few gates at
the cost of making programmers do sums. One interrupt, fifty times a second.

That’s the machine.

Which matters, because my first instinct was the Commodore 64.

I got talked out of it inside about a minute, and the argument was better
than mine. The 64 means a VIC-II with sprites and raster interrupts and
badlines, plus CIA timers, and Ocean-era code leans on all of it. Get it
subtly wrong and you don’t get an error — you get output that looks nearly
right and costs you a fortnight. The Spectrum’s frugality is the whole
point: on that machine 100% of your cost is the CPU, and a CPU is the
boring mechanical bit you already know how to write.

I’ve been at this long enough to know that changing your mind quickly on
someone else’s better reasoning is a skill rather than a concession. There’s
a new wrinkle when the reasoning comes from an AI, though, which is that
it’ll argue just as confidently when it’s wrong. That came up later.

The bit that was genuinely mine

Daley Thompson’s Decathlon is the game that destroyed joysticks across
Britain. Ocean, 1984. The entire thing is how fast you can waggle, which
makes it the worst possible fit for an emulator running at a fraction of
real time. That appealed.

The tape release is Speedlock protected, and emulating a fast loader’s
timing games buys you nothing once the code’s in memory. The obvious dodge is to load it in a normal emulator and take a snapshot — which works, but it’s starting from someone else’s already-solved problem.

I had the Micro-drive cartridge image. Micro-drive cartridges are unprotected, and it turned out this one had the whole story in it: a BASIC loader with machine code hidden inside a REM statement, jumping into itself, copying a stub into the printer buffer, restoring a saved register set popped off a fake stack it had built in the middle of a program line.

There’s a moment where you decode those saved registers out of the cartridge bytes and IY comes out as `0x5C3A` — the standard Spectrum system variable pointer. That’s when you know you’ve read it correctly rather than plausibly. Nobody handed us that. It was sitting in a 137KB file that’s been on the internet for thirty years, waiting for someone curious enough to look.

Knowing that file existed, and why it beat the tape, is the sort of thing
that comes from having lived through it the first time.

Ten deploys that didn’t work

Here’s the part nobody puts in the article.

The first version wouldn’t parse. Nor the second. T-SQL has a trap where
`IF x SET @a = 1; ELSE …` is a syntax error — the semicolon terminates the
`IF` and orphans the `ELSE` — and the generator was emitting that pattern
several hundred times. Fixed that, and the ROM image wouldn’t load, because a `0x` binary literal can’t contain whitespace and someone had prettily wrapped 32,768 hex digits into neat rows. Fixed that, and the error handler checking the fix was itself invalid, because `RAISERROR` won’t take a function call as an argument.

Then three runs produced byte-identical results because the file on disk had changed and the copy on my server hadn’t, and neither of us could tell.

My contribution through all of that was un-glamorous and, I think,
irreplaceable: I pasted everything back verbatim. Every error, every line
number, full register dumps, disassembly listings, screenshots when words
wouldn’t do. No summarising, no “it didn’t work”. The exact text. The reason I didn’t try to understand the disassembly is shown in the code below.

DAA — the hardest instruction on the Z80

-- opcode 27h
BEGIN
    -- DAA
    SET @v = 0; SET @c = 0;
    IF (@r_F & 16) <> 0 OR (@r_A & 15) > 9 SET @v = 6;
    IF (@r_F & 1) <> 0 OR @r_A > 153 BEGIN SET @v = @v + 96; SET @c = 1; END;
    IF (@r_F & 2) <> 0
    BEGIN
        SET @t = (@r_A - @v) & 255;
        SET @res = CASE WHEN (@r_F & 16) <> 0 AND (@r_A & 15) < 6 THEN 16 ELSE 0 END;
    END
    ELSE
    BEGIN
        SET @t = (@r_A + @v) & 255;
        SET @res = CASE WHEN (@r_A & 15) > 9 THEN 16 ELSE 0 END;
    END;
    SET @r_A = @t;
    SELECT @nn = f FROM dbo.z80_szp WHERE v = @r_A;
    SET @r_F = (@r_F & 2) | @c | @res | (@nn & 236);
END

Decimal Adjust Accumulator. It fixes up the accumulator after BCD
arithmetic, and its behaviour depends on the carry flag, the half-carry
flag, the subtract flag and both nibbles of A at once. It is the instruction
that catches out every emulator author, and here it is as T-SQL. There another 700 of these.

2. The `DD`/`FD` prefix — how IX and IY get faked

WHILE @op = 221 OR @op = 253
BEGIN
    SET @ixy = CASE WHEN @op = 221 THEN 1 ELSE 2 END;
    SET @ea = @r_PC;
    SELECT @op = val FROM dbo.z80_mem WHERE addr = @ea;
    SET @r_PC = (@r_PC + 1) & 65535;
    SET @r_R = (@r_R & 128) | ((@r_R + 1) & 127);
    SET @ts = @ts + 4;
END;

IF @ixy = 1
BEGIN
    SET @t = @r_H; SET @r_H = @r_IXH; SET @r_IXH = @t;
    SET @t = @r_L; SET @r_L = @r_IXL; SET @r_IXL = @t;
END
ELSE IF @ixy = 2
BEGIN
    SET @t = @r_H; SET @r_H = @r_IYH; SET @r_IYH = @t;
    SET @t = @r_L; SET @r_L = @r_IYL; SET @r_IYL = @t;
END;

The Z80’s index registers are handled by a trick: when a `DD` or `FD` prefix
byte turns up, swap H and L with IXh and IXl, execute the next instruction
normally, then swap back. Every `HL` operation silently becomes an `IX` one for exactly one instruction.

3. The bug that halved the machine’s speed

/* EI defers interrupt acceptance by one instruction. That delay
   has to expire here, at the end of an instruction. Clearing it at
   the start of the next call -- after the interrupt gate has
   already been tested -- suppresses every interrupt that follows a
   frame ending in EI/HALT, which halves the machine's speed. */
IF @ei_pending = 2
BEGIN SET @ei_pending = 0; END
ELSE IF @ei_pending = 1
BEGIN SET @ei_pending = 2; END

Every frame of a game loop ends `EI` / `HALT`. This flag was being cleared
one step too late, so every other interrupt was suppressed and the machine ran at exactly half speed — from the very first boot.

Two lines, in the middle of the fetch/execute loop. `EI` defers interrupt
acceptance by one instruction, and that delay has to expire at the end of an
instruction — not at the start of the next call, after the interrupt gate has
already been tested.

The model can’t see my instance. It has no idea what happened unless I tell
it precisely, and paraphrasing is the fastest way to send the whole thing
down a wrong path. Half the wins in this project came from a number I copied across without understanding it.

The T-SQL Linter that which is in the code repository is a good example of how you can use the constant failures to your advantage.

It can catch the structural mistakes that only surface when SQL Server parses 7,500 generated lines.

1. ORPHANED ELSE - a ';' between a statement and its ELSE terminates
 the IF, so the ELSE has nothing to attach to.
 'IF x SET @a=1; ELSE SET @a=2;' is a syntax error;
 'IF x SET @a=1 ELSE SET @a=2;' is fine, and
 'IF x BEGIN SET @a=1; END ELSE …' is better.
 
2. END; ELSE - the same thing with a block body.
3. EMPTY BLOCK - BEGIN followed only by comments then END. T-SQL
 requires at least one statement.
4. SPLIT LITERAL - a 0x binary literal broken across lines. Literals
 cannot contain whitespace, so wrapping 32k hex
 digits for readability makes the script invalid.
5. RAISERROR ARG - substitution arguments must be constants or
 variables. RAISERROR('%d', 16, 1, DATALENGTH(@x)) is a syntax error.
6. UNBALANCED - BEGIN/CASE vs END across the file.

The moment it worked

150 frames of ROM execution, and then I rendered the bottom of the screen as text, straight out of the database:

Finally The Boot loader

None of that was re-implemented. Sinclair’s own ROM, running its own memory test — which takes 640,000 instructions and eighty-two frames, something I’d completely forgotten about the Spectrum boot — and then its own character routines, setting individual bits in a table.

You will never feel the excitement of seeing this on a screen, both from back when it came out and also when it lived once again.

Just in case you wanted to see what the code looks like for a ZX Spectrum without installing it this is the SQL code which loads it.

Do not try to read this, the actual code is in the Github project. It is just showing the shape.

I sat and looked at that for a while and thought never again…

To see the process for loading the game onto the ZX Spectrum running inside SQL Server have a look at the RUNBOOK.md

Where I earned my keep

Late on, the game hit a black screen and stopped responding. I was told,
confidently, that it had probably timed out into an attract mode.

I wasn’t sure. So I asked: does this game actually go into a demo after a
few seconds of nothing happening?

It did, but what was actually happening this time was the input system was holding keys down permanently, and the game was sitting in a loop waiting for a key to come back up. Six hundred frames with a joystick jammed on.

The other thing only I could do was decide what mattered. Half real time is
fine, because input is recorded at a true 50Hz and replayed, so emulator
speed never affects the result. Knowing that meant not chasing performance we didn’t need.

What’s left

The waggle rate is a parameter. The race result is a byte in memory. Which
makes an event’s outcome a function of one number, and a function of one
number can be searched — from T-SQL, on the same instance, in the same
session.

A stored procedure that hill-climbs its way to a world record in the long
jump.

That’s next, and I’m looking forward to it more than any of the above. Not
because it’s difficult, but because it’s where the database stops being a place the game lives and becomes the thing that plays it.

Forty-three years after Sinclair sold the thing for £175, it’s running inside a relational database, being played by a query. I think he’d have found that funny.

10.48s

Code is available on GitHub.

https://github.com/sqlartist/ZXSpectrumSQL

I cannot imagine anyone will try this but genuinely when that ZX Spectrum message appears and you realise that is a piece of tin running as software inside the wrong place, and then your loading a tape into this emulation and it works, you have to smile.

You’ll need your own copy of the game; the Sinclair ROM is included, as Amstrad permit redistribution for emulation. The Z80 core was written by Claude from a Python generator rather than by hand. The bugs were found by both of us, in roughly equal measure.

If you got to this part of the story then to be honest I should be Clapping you but the system doesn’t work that way so Clap like you have never Clapped before.

Thirty Million Pounds of Wrong

Tue, 15 Sep 2026 12:36:04 GMT

The warehouse moved, and the planners got the plan upside down.

One can dream can’t they…

That’s the whole story. Everything after it is consequences. Three million computer parts went into a new building back to front — the heavy, awkward stuff upstairs in bins that were never sized for it, and the small, light, valuable components downstairs by the doors. Easy to find. Easy to walk off with.

Nobody did it maliciously. Someone drew a layout, someone else read it the wrong way round, and by the time anyone noticed, three million items were already in the wrong place. You can’t flip a warehouse. You live with it.

Here’s the part that mattered. The brand new inventory system recorded all of it faithfully.

The gap

We’d just gone live on an AS/400. It knew where every part was meant to be, and it was completely confident about it. What it didn’t know was where anything actually was.

I’ve got the sort of memory that holds pictures. It’s not a skill, I didn’t earn it, it just happens to be how my head works — and in a warehouse with the plan upside down it turned out to be worth something. I could walk to a bin. I didn’t need to look it up, and more to the point I’d learnt not to trust it when I did.

So for a while I was the reconciliation layer between a computer and a building. The system was the authority. I was the correction.

That’s why I got promoted to Team Leader, and then put in charge of two areas. Not because I was good with computers. Because I could tell you what was true.

Peter

I got good with the AS/400 too — and I got good at it before everyone else did, which was its own quiet advantage. When you’re the person who understands the system in a room full of people who don’t, you stop being a storeman fairly quickly.

But I didn’t get there on my own, and I’m not going to pretend otherwise. Peter Kerrison was the manager of Inventory Control and he showed me everything. His knowledge seemed to have no bottom to it. Every time I thought I’d found the edge of what he knew there was more behind it, and he never once made me feel stupid for asking.

I’m still grateful. Years later, when I had my first brush with cancer, it was Peter’s wife Dawn who gave me advice and what I needed to know. You don’t repay that. You just try to be the person who does it for somebody else.

Just under thirty million pounds

At twenty I was handed the job of getting the stock into shape. I ended up adjusting nearly £30 million off the stock holding.

I want to be precise about that number, because the exciting version is wrong. Most of it wasn’t theft. The bulk of it was onsite balances that hadn’t carried across correctly through the changeover of the main systems; stock that the ledger insisted was ours, sitting in a location that had never really been true in the first place.

Some of it had genuinely gone. High-value kit turning up in engineers’ lockers. Things left in the pocket of an anti-static coat and honestly forgotten about. That happens in any warehouse and it’s mostly absent-mindedness rather than villainy.

But the headline is the boring bit, and the boring bit is the whole point: the system was confidently reporting something that had never been the case. My job wasn’t to find thirty million pounds of missing kit. It was to stop the books lying.

I’ve been doing some version of that job ever since.

Sixteen thousand, three hundred and eighty-four

Excel 5.0 would hold 16,384 rows. My extracts from the AS/400 were longer than that.

That’s it. That’s why I learnt SQL Server. Not vision, not a career plan — an arbitrary limit somebody in Redmond picked because it’s two to the fourteenth, and it happened to sit exactly between what I had and what I needed. Thirty years of my working life turn on that constant.

I was the first person in the company to get a 486 on my desk. I used it to build my own inventory system, deliberately air-gapped from the AS/400, to model where the losses were coming from. That air gap was a decision, not an accident. If you’re the man telling everyone the system of record is wrong, you cannot also be the man with his hands in it. I needed my numbers to be arguable on their own.

Then I got round the AS/400 team.

Everybody’s process at the time was to ask them for a CSV extract and wait. I connected to it directly with OLE-DB instead and pulled the data myself, overnight, automatically. Nobody had done that. It reads like a technical footnote and it wasn’t — it was a political act. I stopped requesting the data and started taking it.

The beta version

I was also the first person in the company to be given a personal laptop: a Toshiba Tecra running Windows 3.1. Stephen McKenna — the management accountant, and the man I sat directly in front of — banned me on the spot from installing anything that so much as resembled Windows 95.

He wasn’t being a dinosaur about it. Stephen loved technology as much as I did. But it was a work machine, and he was the one accountable for it, and he was right.

Over one weekend, with no internet, no documentation and very nearly no idea what I was doing, I dual-booted that laptop anyway. Work image on Windows 3.1, an early Windows 95 build alongside it. I built the dashboard at night on 95 and came in each morning to a machine that would boot into the forbidden operating system in full view of the man who’d forbidden it.

My habit until then had been to walk in, hit the power button and go straight for a coffee. I had to break it. Every single morning the first thing I did was stand at my own desk waiting for a boot menu, because the alternative was Windows 95 quietly appearing on a screen a few feet in front of Stephen’s face.

The dashboard was an automated overnight extract from the AS/400 — every stat from the night before, laid over a map of the UK, animated. I finished it and demonstrated it, and on the same day, in the same breath, I told them the whole thing had been built on the Windows 95 they’d banned. Which by then was the biggest software launch the world had seen.

I’d like to say I planned the timing. I did.

Where you sit

The demonstration went up the chain to Sean Francombe — Stephen’s boss, and the main man. What came back wasn’t a bonus or a certificate. It was a desk.

I got a seat next to him.

That’s a strange thing to be proud of until you’ve been a storeman. In my first proper job, in a blue uniform, in a building where the plan was upside down, the reward for being right about the data was that I got moved to sit next to the person who ran the place. I don’t think anything since has meant as much.

The car I never drove

Computacenter’s colours were silver and blue.

A friend of mine was selling a Lotus Sunbeam — a Talbot Sunbeam with a Lotus engine in it — and it was silver and blue. So I bought it.

I can’t drive. I have never learnt to drive. I did not, at any point in this process, learn to drive.

It sat on my driveway for two years. I started it once.

Not my actual car, but this was it. Company colours.

April 1999

In April 1999 Bill Gates launched a Microsoft initiative called Shift, aimed at a UK IT skills shortage — 600,000 internet-related jobs to fill by 2002. The BBC covered it, and I was one of the case studies.

BBC News Online screenshot, “UK shifting into IT skills”, 28 April 1999

Trevor Dwyer was a warehouse storeman who transferred to Inventory Control and learned about databases. Four years later, he is one of the most experienced database designers in the country and has been commissioned to write three books on the technology.

the printed case-study list

Around the same time I got a phone call from Redmond telling me I’d been given the MVP award. They’d been watching my contributions online and noticed I’d gone from asking a few questions to answering a great many.

The email I kept from that week isn’t from Microsoft. It’s from Peter:

“Trevor ‘Storeman to award winning author’ Dwyer. Very good article but missing 1 key sentence.”

After one year at Computacenter, Peter Kerrision, then manager of Inventory Control, recognised that behind the blue uniform which Trevor wore with pride, was an Essex Boy suit trying to get out. The gamble paid off, Trevor ignored convention and decided against sleep for 4 years. The result was a ghostly pale appearance but proved to be the foundation on which Trevor would progress through the company…..
It should finish.

Stephen replied to it:

I’m with Peter on this one… where are our names… after the amount of time we spent cleaning up the stores after your time in there surely the article should be about us!!!!!

Fair, honestly.

Printed from Lotus Notes in 1999. I’ve still got it.

Philip Hulme, who’d founded the company with Peter Ogden, took an interest after that. He’d bought a majority stake in Marcos in 1991 — the sports car outfit — apparently after remarking to Chris Marsh how nice it must be to make cars for a living. Which tells you something about him. At one point I found myself sat on a grass bank at Silverstone next to him, watching his team go round.

Thirty years on

I’m at [Somewhere that cannot be named] now, on a team that takes what the source systems say and works out what’s actually true before anyone downstream gets to see it.

It’s the same job. It’s the warehouse. Something upstream is confidently wrong, and somebody has to sit between the system and reality and do the reconciliation.

So a few weeks ago I started rebuilding the AS/400 inside SQL Server 2025.

You can’t emulate one, so don’t try

A real machine is TIMI over SLIC — a technology-independent machine interface sitting on top of the Licensed Internal Code. Programs compile to a virtual instruction set and get translated underneath. To boot OS/400 objects you’d need the machine interface fully specified and the Licensed Internal Code itself. The first is only partly public. The second is IBM’s and not mine to run.

So: a clean-room re-implementation from IBM’s published DDS, CL and RPG references. Which sounds like a compromise but it turned out to be the better project, because what made that machine interesting was never the instruction set.

DDS has no grammar

Here’s the thing that stopped me.

Screens and files are defined in DDS, and DDS is column-positional. Not “mostly fixed format” — the meaning of a character is decided entirely by where it sits.

One DDS statement. The meaning of a character is decided by where it sits.

So the DDS parser is six `SUBSTRING` calls. There’s no tokeniser, no grammar, no error recovery. I’ve spent an embarrassing share of my career writing parsers. This one took an afternoon.

It’s worth sitting with why. In 1988 you typed source into SEU on a block-mode terminal with a column ruler across the top. Columns weren’t a limitation of the format. Columns *were* the interface.

A 5250 screen is a projection, not a drawing

Every field in a display file already carries its own line and position. So rendering a screen isn’t painting, it’s a query: turn every field into a write, explode each write into characters, let later writes win where they overlap, left join onto the full 24×80 grid and aggregate each row back into a string.

Nothing draws anything. No loop, no buffer, no cursor.

And subfiles — the scrolling record lists that make a green screen feel like a green screen — fall straight out of it. Subfile record *n* is the same fields one line further down. The page size is a `WHERE` clause. I’d budgeted days for subfiles. They were about twenty lines.

The same thing happened with record-level access. `CHAIN`, `SETLL`, `READE` aren’t a rival access method I had to bolt onto a relational engine — they’re an index-positioning API. A keyed physical file is an index. When `SETLL` runs it isn’t faking one, it’s driving the real one.

The INQ100 screen

The same bug, thirty years later

Then I wrote a bug that stopped me cold, because I had encountered it before.

Deleting a record left the file pointer sitting on the record it had just deleted. The next read then had nothing to compare against, so it returned nothing and reported end of file.

Think about what that does. A read-and-delete loop processes its first record, deletes it, and stops. No error. No exception. Nothing in the log. It looks exactly like a loop that ran correctly and found one record to do. You would only ever catch it by counting rows afterwards, and you’d probably assume your filter was wrong.

A system confidently reporting something that was never true.

That is the warehouse. That is the thirty million pounds. Same failure, thirty years and one career apart, and this time I wrote it myself.

I want to be honest about how it was caught, because it wasn’t by running the thing. I couldn’t run SQL Server where I was building this, so I’d written a second implementation of the same logic in another language whose only job was to disagree with the first. It disagreed. That’s the whole technique and I’d recommend it — not tests that confirm what you meant, a second opinion that doesn’t know what you meant, almost like having Peter back in the room questioning my logic.

Why this one is different

I’ve put Doom inside SQL Server. Elite. The Amiga Boing ball. Those work because SQL is the wrong tool, and the wrongness is the joke.

This isn’t.

In 1988 IBM shipped a database-centric operating system. Everything is a typed object in a system catalog. The database isn’t a product installed onto the OS — it’s part of it. Programs are objects. Files are objects. Commands are objects with parameter lists, which is how the machine could prompt a command you’d never seen before with F4.

So modelling an object catalog as tables, or resolving a name against an ordered library list, or driving a keyed access path — none of that is translation into something foreign. It’s the design expressed in roughly the terms it was conceived in.

Putting an AS/400 inside a database isn’t parody. It’s closer to restoration.

The screen I wanted to see

I’m not finished. Logical files next — a second access path over the same data, with select and omit. Then the real one: parsing fixed-format RPG, storing an intermediate representation inside the program object, and generating a stored procedure from it. Which isn’t a shortcut, incidentally — it’s exactly what TIMI does, keeping the translatable template inside the object so the machine can re-translate it for hardware that doesn’t exist yet.

But the part I actually wanted is done.

You can type into it. Position to a part, roll through the file, put a 5 against a line and drop into that part’s movement history. The host sends a screen and then sits there knowing absolutely nothing while you type — it only hears from you when you press a key that matters. Thirty years on, that still strikes me as the right way round.

And I knew which screen I wanted first. It wasn’t a demo.

An inventory enquiry. Part number, bin location, quantity.

Every field correct. Every one of them wrong.

The second AS/400 Screen.

If you enjoyed this story please give it as many Claps as you think worthy.

My Potato Mountain & The 46th Greatest Comedy Film Of All Time

Sun, 13 Sep 2026 00:03:58 GMT

My final version of the Truckster, with the whole family aboard.

There’s a scene in Close Encounters of the Third Kind where Richard Dreyfuss builds a mountain out of mashed potato because he can’t stop thinking about it. This is my potato mountain. Except mine is a station wagon, and mine is real.

I came to 3D modelling late. For the last 30 years I’ve worked with SQL databases, written books, won awards and worked on some exciting projects around the world. None of it helped me with the one thing I actually wanted to do: recreate the car jump from National Lampoon’s Vacation.

Worth saying up front, because it comes back later: I’ve never learnt to drive. Not once, not ever. I’m rebuilding the most famous drive in comedy cinema and I’ve never held a licence.

I’ve been obsessed with this car for years. I started in the physical world — 3D printing, painting, building the Truckster at every size I could manage. That rabbit hole ended with eight 3D printers in the house and Trucksters ranging from matchbox-sized models to a two-metre-wide diorama.

The first decent diorama.

From plastic to pixels

After some success in the physical world I thought, why not build the scene in 3D? Unreal, maybe. Or Unity. Or Blender. It started as an idea for a single 2D render. Then it became an animation. Then a game. Then a full scene. That’s how these things go.

Autodesk 3ds Max render.

Another 3ds Max render. I found this tool so hard to learn. The tyFlow module is enormously powerful and I never got on top of it.

Why I’m in a hurry

I have stage 4 colon cancer. I had major surgery, which was initially successful, but the cancer is returning. So I’m finishing what I can while I can.

The practical effect is smaller and more annoying than you’d think. My memory isn’t what it was. If I read how to do something and don’t apply it I end up learning all over again. Its not that bad but when you have had near ident recall for many years its a bit depressing.

That one fact has shaped how I’ve built everything since.

What thirty years of SQL actually gave me

For a long time I assumed my database career counted for nothing here. Wrong field, wrong brain, wrong everything. I was wrong, and it took me a while to see why.

I am not as spontaneous as I once was, I can’t afford to hand-place a scene. Click a thousand objects into position, forget how you did it, and the scene is a dead end the first time something breaks. And something always breaks.

So I stopped placing scenes and started writing them. Python builders that generate the mesa geometry, the procedural materials, the multi-camera setup and the particle dust. In Unreal, a Chaos vehicle pawn that detects the collision with the Road Closed sign and launches the car on proper projectile physics. Camera focal lengths computed rather than eyeballed.

Underneath, a scene is just rows. Objects, parents, transforms, materials. Name them properly and you can rebuild them. My model is a hierarchy with a body parent and children named the way a schema should be — wheelFL, wheelFR, wheelRL, wheelRR — with Clark, Ellen, Rusty, Audrey and Aunt Edna as separate objects so they don’t get dragged into a vehicle export.

That isn’t artistry. It’s normalisation. But it means that when I break the scene, I can regenerate it instead of mourning it. Thirty years of databases taught me one transferable thing: never keep the only copy of anything in your head.

Five engines and a lot of u-turns

3ds Max was the first stop, and the first wall. tyFlow can do everything I needed and I couldn’t get my head into it. I’m not a fundamentals-first learner — I have an urgency to finish what I’ve started, so I try every possible angle at once and hope the best one falls out of the end. Feynman’s path integral as a working method. It doesn’t suit a tool that wants to be learned properly, so I moved on.

An early Blender animation.

The same animation just after the car leaves the ramp — and an early version of the car.

Twinmotion was the opposite problem. It was the fastest route to something that looked good, and the weather and sunlight modules are genuinely excellent. But custom vehicle animation was too random. I could get a nice result; I couldn’t get the same result twice. Repeatability matters more to me than speed, so Twinmotion became a place I visit for stills rather than the place I build.

A Twinmotion render. I tried hard to make this one work.

Twinmotion render — it does look nice, though.

Twinmotion again. The weather and sunlight modules are very good.

BeamNG.drive was the detour I regret least and enjoyed least. The plan was sensible: use an existing LTD wagon as a physics donor and swap in my visual mesh, so I’d get real soft-body crash damage for free. What I got instead was “Loaded nil parts from 0 jbeam files”, over and over. I eliminated encoding, file paths, conflicting mod zips, flexbody mesh references and directory placement, one at a time, and never got it loading. In the end I drew a line: the crash work happens in Unreal and Blender, and nowhere else.

There was a Godot version too — a playable one, with the 140-foot gap, the blockade to smash and a repair-and-cash loop. Useful for working out what the jump should feel like, and a dead end for what it should look like.

Where it’s landed is Blender plus Unreal Engine 5.8. Blender for the rendered scene, Unreal for the drivable car and the cinematic. I’ve been through Unreal 4.26 up to 5.8 to get here.

The wheel that turned itself inside out

Every time the model moves between packages I end up in texture and material hell. Every piece of documentation tells you it’s an easy process. Trust me, it isn’t, and I genuinely cannot work out how designers put up with this level of incompatibility — it’s a whole article on its own. Fix one thing and another appears. The best of them was a wheel that inverted itself on import — no error, no warning, just a wheel pointing into its own arch. I never did work out why.

The current one is more interesting. I have a Blender crash setup working properly — the car drives, hits a rock, rolls, the glass shatters and the metal deforms as it lands on its roof. I have the Truckster animated along its own spline from the same start point, jumping and landing. Transferring the shatter onto the Truckster’s path is the bit that isn’t right yet: the simulation fires far too early, so the glass explodes while the car is still in the air. It needs holding back until touchdown. That’s the sort of problem that sounds trivial and eats a fortnight.

Camera work and raw render time have been the other two hurdles. Both are solvable with patience, which is the one resource I’m careful with.

The car itself, and what I got wrong about it

The Wagon Queen Family Truckster is a fictional station wagon built for the 1983 film, based on a 1979 Ford LTD Country Squire and modified into something deliberately hideous. Eight headlights in two inverted clusters. Wood panelling over most of the available surface. A grille mostly covered by bodywork. A separate oil-cooler grille with no oil cooler behind it. Crown hubcaps. A fuel filler flap in the front passenger wing, which is the joke that keeps giving — in the film Clark rips the rear number plate off looking for it. And an airbag made from a household bin liner.

Fictional Wagon Queen Family Truckster advertisement — a fan-made parody graphic featuring a replica built by Gary Schneider, using dialogue attributed to John Hughes.

Three things I found while researching it that I hadn’t known, and that are worth knowing if you’re building one yourself.

First, the colour has a name: Metallic Pea. Ford never sold it. It exists only on this car, which means there’s no swatch and no hex code to work from — only film stills, shot on 1983 stock, in desert sun, then graded.

What I ended up doing was collecting every high-quality still I could find, lining them up against the physical models I’ve built, and running an eyedropper over all of them. It’s a fool’s errand. The paint is metallic, so it changes with every angle, and every still carries its own shadows and grade — there is no single right answer in any of them. In the end I took a pea green somebody had made for GTA: #47783CAA. That’s RGBA, by the way — #47783C with an alpha of AA — so drop the last two characters if your tool wants six. Every replica builder, physical or digital, is picking their own green and calling it right. Mine is still the thing I fiddle with most.

Second, George Barris probably didn’t build it. Nearly every source online credits him, including Wikipedia, and I repeated it myself. The replica community says otherwise: Gary Schneider dug into it while building his own car and concluded the Trucksters were modified by Warner Bros’ own transportation department, and that the Barris attribution is an internet myth that got copied until it looked like fact. I’m not going to settle it here. But it’s a useful reminder that a good half of what you find when you research a film car is other people’s guesses, repeated confidently.

Third, five cars were built, so the crew could rotate them as the damage piled up through the film. I laughed when I read that, because it’s exactly what I’ve ended up doing in software — separate versions of the same model at different stages of destruction. And none of the five survive. They were either destroyed during filming or converted back into ordinary LTD wagons for other Warner Bros productions. Every Truckster you see today is a replica. A good one made $91,000 at Barrett-Jackson in 2019.

Which is worth sitting with for a moment. Between all those 3D printers, the wasted filament that comes with the territory, the software and the time, I could probably have bought the one that came up at auction and had it shipped to the UK with change to spare. I’m glad I didn’t. I’d have had to take driving lessons, pass a test and insure the thing — and as I said at the top, I’ve never learnt to drive. I’m building the Griswolds’ road trip and I’ve never driven a car.

One last detail, for anyone who’s read the fan-written brochure that does the rounds — the one promising a “huge, large block” V8 that “drinks the gas like nobody’s business”:

“Yes, it got it all and safe too, with its large size it can run over and smash through most standard size cars and built like a tank… The Truckster eliminates that with its eight head lights that cuts through the darkness, and your only worry is that oncoming cars are blinded by all that light and might hit you head on.”

What was under the bonnet was a standard 130 horsepower 302 with a four-speed automatic. The most over-described car in cinema was, mechanically, a completely ordinary Ford.

The jump

This is the scene I set out to build. Clark is lost on the way to the Grand Canyon, too busy arguing with Ellen to notice the road is closed, and takes the entire family off the end of it at speed — blowing the tyres, bending the axles and destroying the front end.

Two things I learned about it changed how I approached mine.

The stunt coordinator bet the crew he could launch the car more than 50 feet, and won. I’m targeting a 140-foot gap in Unreal, worked out with projectile maths rather than guessed at. Having a real number from the real stunt to argue with turned out to be worth more than any amount of reference footage.

And the place doesn’t exist. The driving shots were filmed at Baby Rocks on US-160 east of Kayenta, Arizona. The walk through the desert with the skeleton is Great Sand Dunes in Colorado. The hotel is El Tovar at the Grand Canyon. The film cuts several states into one road. A faithful recreation is therefore impossible, and once I accepted that mine is an interpretation rather than a copy, the whole thing got easier.

Which didn’t stop me trying. I spent months tracking down the real locations, pulling satellite imagery and fighting it into a form I could export as an elevation model for the Unreal and Blender scenes. For the record, the jump itself is at 36°58’02.11”N 110°04’14.82”W — about two miles south of the Utah line, whatever the tour blogs tell you.

If I ever went back to the USA — which is now unlikely — I’d be Clark Griswold and I’d seek this exact spot out. Given I can’t drive, it would be a very expensive taxi ride.

The spot itself.

A Blender still. I like the scale on this one — it’s about the level of quality I’m aiming for.

Where it is now

Near-working versions in Blender, Unity and Unreal. A car model that finally renders, drives, carries its passengers and survives the trip between Blender and Unreal without falling apart. The target is a rendered scene in Blender and a drivable vehicle with a cinematic render in Unreal 5.8. It isn’t quite there. It’s very close.

If I started again tomorrow I’d do three things differently: pick the target renderer first and model to it, rather than modelling and then shopping for an engine; impose the naming discipline on day one instead of retrofitting it; and write every scene as a script from the beginning, even when clicking would be quicker. The clicking is always quicker right up until the moment you need to do it twice.

Here, take it

There’s no money in any of this. It’s a hobby and a long learning exercise. I’ve always given my work away in the hope it’s useful to somebody, and I’m doing the same here — the Blender builder script, the Unreal level scripts and the model itself. Reducing over 9,416 files and 554 GB on one machine down to a few files on GitHub is its own small act of violence, but that’s the way it goes.

[GitHub Link to Follow.]

There’s a reason I’d like it archived properly. Years ago I was on my way to meet Bill Gates in London. I’d just pulled up at the hotel where he was speaking when my office rang: they needed me in LA, in a meeting with the Head of Warner Bros Studio in Burbank. I never got out of the taxi. I turned round, went home, packed, and flew to Burbank for one of the strangest meetings of my career.

Going into a huge boardroom with Clint Eastwood and Daffy Duck posters all round was surreal, and then you notice the actual Oscars and Emmys everywhere. For someone from Watford, England, that was something special. Then the head of Warner Bros came in and he looked just like Don Johnson out of Miami Vice. I must have said three words in the whole meeting — I was there as the technical expert, and the meeting wasn’t technical. They did take us out to dinner that evening, which must have been good because I don’t remember a single thing about it.

So when the scene is finally finished and uploaded, who knows. The 45th anniversary of the film’s release comes round in 2028, and Warner Bros may want something extra for the archive. Stranger things have happened to me in Burbank.

If you know tyFlow properly, or you’ve ever persuaded a Blender fracture simulation to hold off until the exact frame you wanted, I’d genuinely like to hear from you.

This is my potato mountain. I’d like to finish it.

Re — Mi — Do — Do — So

If you enjoyed this story, or just love the car or the movie, get in touch, but first, give it some claps. Thank you for reading.

Boing, Boing, Boing

Mon, 07 Sep 2026 15:17:32 GMT

This is the Boing ball Running Inside SQL Server

I’ve spent the last few months putting things inside SQL Server that have no business being there. Doom renders E1M1 from a database. Elite generates its galaxy, flies its ships and docks at a Coriolis station in T-SQL. Daley Thompson’s Decathlon runs on a Z80 emulator I wrote in stored procedures, loading from a real Microdrive image.

So when I turned to the Amiga Boing ball, I assumed it’d be the easy one. A bouncing chequered sphere. No BSP tree, no galaxy generator, no cycle-accurate anything.

It was the easy one. Right up until I added a second ball and then I wish I hadn’t started.

The History

The Amiga Boing Ball is a mythical object in the computer industry. It was created for a computer show to demonstrate the capabilities of the prototype Amiga (codename ‘Lorraine’). The demo showed a red-and-white spinning ball, bouncing up and down and casting a shadow on a wall behind it while multitasking in the background. The echoing deep ‘boing’ sound and left-right motion was added soon after the show was over.

According to the legend, the Boing Ball demo was written late one night at the Winter CES show in January 1984 by RJ Mical and Dale Luck. At the time it was an amazing achievement that surpassed all current systems of the time. Because the bouncing ball animation was so fast and smooth, attendees did not believe the Amiga prototype was really doing the rendering. Suspecting a trick, they began looking around the booth for a hidden computer or VCR.

The 1984 Boing Ball demo was one of the very first demos shown on the Amiga. It was specifically designed to take advantage of the Amiga’s custom graphics, achieving a level of speed and smoothness not previously seen on an affordable computer. The demo was operating in it’s own screen, allowing the higher resolution Amiga Workbench screen to be dragged down to make the Boing Ball visible from behind, bouncing up above the Workbench while the Workbench remained fully active. Since the Boing Ball used almost no CPU time (only to calculate the bounce angles — animation was handled by playfield vertical and horizontal scrolling tricks, the rotation animation was done with colour cycling in the graphics chip, and of course the sound chip handled the sound), this made a particularly impressive demonstration of multitasking at the time.

If you want to see what it looked like on the Amiga take a look here: https://amiga.lychesis.net/applications/AmigaBoingBall.html

Three ways to do it

The first decision was which version of “the Boing ball” I was actually building, because there are three, and they’re wildly different jobs.

The most faithful option is to emulate a 68000 and the Amiga chipset and boot the original demo from its bootblock. Having just written a Z80 core, I knew roughly what that costs, and it’s an order of magnitude more: a much heavier instruction set, plus the Copper and the Blitter, before a single pixel appears. That is a serious commitment and after the ZX Spectrum Hardware project I don’t think I want to go down that route again.

The most historically accurate option is stranger. The 1984 demo doesn’t render a 3D ball at all. Dale Luck and RJ Mical drew one static checkered sphere and made it appear to spin by cycling the colour registers. The bitmap never changes. Only the palette does.

I loved that, because it’s already a database idea. Rotation isn’t animation, it’s a lookup against a different row set. They were doing a join in 1984; they just didn’t have anywhere to put it.

But I picked the third option: render the thing from scratch. Not because it’s more authentic — it’s less — but because it plays to what SQL is actually good at, and I wanted to see how far that went.

This is the thing about spheres. It’s a closed-form intersection. Under orthographic projection you don’t need a discriminant, you don’t need clipping, you don’t need to rasterise a polygon. You need `d² = dx² + dy²`, and if that’s inside the radius, you’ve hit it. Everything else is a normal, two rotations and a checkerboard.

That’s pure maths on a grid of pixels. Which is to say: it’s a `SELECT`.

The bit I was pleased with

The ball’s motion is also closed form. Position is a function of the frame number — a triangle wave for the horizontal sweep, a parabola for the bounce, and spin proportional to how far it’s traveled, which means the spin reverses at the walls on its own without me writing any logic for it.

No state. Nothing depends on the previous frame.

So frames aren’t a loop. Frames are just another dimension of the cross product.

The entire animation — 64 frames, 320×256, five and a bit million pixels — comes out of one `INSERT … SELECT`. No recursive CTE, no `WHILE`, no cursor, no temp table. A chain of `CROSS APPLY`s you read top to bottom like a shader.

I sat looking at that for a while. It’s the most SQL-shaped thing I’ve built in this whole series. Doom needed a client and a game loop. Elite needed a flight model with state. This is one query that produces a simple moving ball.

Then: Two balls

When I got this working I thought to myself “I have achieved everything I set out to achieve in this world I can finally retire”

This is where it got interesting, I wanted to take it further and have two balls but colliding. Collision between objects has been the biggest problem I have encountered whilst developing in the 3D world. If you walk through an object it does not have collision. If you bump into an object then you have had to program collision. This may be obvious to some people but I had never heard of it before attempting to develop a scene of my own.

Adding a second ball is still one query. Add *collision* between them and the whole edifice comes down.

Not because the physics is hard. It’s relatively easy physics, equal masses, elastic, exchange the normal component of velocity and push them apart if they’ve overlapped. Twenty lines in any language.

It comes down because collision response depends on the *previous step*. Where ball one is at frame 40 depends on whether it hit ball two at frame 39. Position is no longer a function of the frame number, and the moment that’s true, frames stop being a dimension you can join to and start being a sequence you have to walk.

One collision. That’s all it took to destroy the property the entire design rested on.

I found that genuinely annoying given the first part had gone rather well. It is the reason I’m writing this up rather than the bigger projects. Every SQL developer gets told “think in sets, not loops” and nobody ever shows them the “Why”. Well here is the “Why”. Closed form: one statement, five million rows, no loop. Add one dependency on prior state: nothing survives.

Embracing the Complexity

My first instinct was to keep everything declarative with a recursive CTE. That’s the trick, and it nearly works. But the recursive member can only reference the CTE once, and pairwise collision needs a self-join.

So I split the thing along the line where the two halves genuinely differ.

Simulation is sequential. It’s a `WHILE` loop. It’s also tiny — a handful of rows and a few hundred substeps — so it costs nothing. Four set-based statements per step: gravity and integration, wall reflection, then a pairwise contact resolution written as a self-join with the impulses summed per ball.

Rendering stayed set-based. It’s still one statement. It just no longer works out where the balls are; it joins to where the simulator says they were.

That’s the whole architecture. The worse SQL made the better system.

Three Balls

Because the collision is written pairwise rather than as a special case for two, it takes any number of balls. I added a third by inserting a row. That’s it. One row in a table, and three balls bounce off each other and off the walls with correct depth ordering, because two spheres at the same depth with the same radius resolve trivially — whichever pixel is deeper into its own silhouette is the one in front.

Getting that for free was the moment the design decision paid for itself.

So what’s it actually for?

As with most of my projects it isn’t a production technique. Nobody should render anything in a database (yet).

It’s useful for three other things

It’s the best teaching device for set-based thinking I’ve come across. Not the concept — everyone’s heard the concept — but the *edge* of it. Boing is a perfect mini example. Here’s a closed-form problem, and it’s one beautiful statement. Here’s the same problem with one sequential dependency, and it’s a loop. Nothing else I’ve built shows that transition so cleanly, because in a real system the two are always tangled together.

It finds engine behaviour that every day work never will. The `CASE` short-circuit trap is going straight into my mental checklist. Pushing an engine into a shape it was never designed for is how you learn where its real edges are, and those edges show up later in queries that matter.

And it’s a learning challenge in the oldest sense. There’s no spec, no acceptance criteria, and nobody’s asking for it. You just have to work out whether the thing is possible and then find out.

There’s a wider ambition behind it too. If you can emulate a machine inside a database, you can put the games in as data — cartridge images in `varbinary`, the machine as stored procedures — and you’ve got an archive that runs what it stores. A shelf of 1980s software living inside the same engine that’s serving your reports. I don’t know yet whether that’s a good idea. I know it’s a good problem.

Where the limit is

People keep asking (I say people, but I mean me, literally no one has asked this question) whether the ceiling is Quake or a C64. I think that’s the wrong angle.

The real question is the ratio of wide work to deep work. Wide work — per pixel, per vertex, per entity — is what SQL is extraordinary at. Deep work, where each step depends on the last, is what it’s terrible at. Everything I’ve built lands somewhere on that spectrum, and where it lands predicts how well it goes far better than what decade the game came from.

By that measure Quake isn’t the wall people (I) assume. Its renderer is overwhelmingly wide work. Emulating a 68000 with a full custom chipset is almost entirely deep work, and that’s where it dies.

Boing is the extreme wide case, which is why it came out as one query. And the second I introduced a single dependency on prior state, it stopped being one.

Two balls taught me more than Doom did but I do think they are both pretty cool.

Code is avaialbe on Github
https://github.com/sqlartist/BoingBallSQL

If you enjoyed reading this story, give it a few Claps, it helps boost someones pockets — not mine these are all given away for free. In fact I don’t think anyone benefits from a Free story, in fact its costing us money. You owe us…

Patient Number 1 in DARPA’s Super Soldier Programme

Sat, 05 Sep 2026 22:29:19 GMT

WARNING: There are some images in this article which some people may find disturbing. They are all real and you really can’t see anything and because I don’t know Markup very well, I can’t hide them.

This article is a combination of events that have happened to me that are quite unusual and people start to wonder what on earth is he talking about. I can promise this is all true.

Weirdly I was interviewed by the team that were creating a show in the UK based in some castle in Scotland called “The Traitors” and one of the tests was to fool them by telling them 4 truths and a lie on a live video stream.

Slightly off topic but what would you think?

Four Truths and a Lie

1. I once had breakfast with the entire Liverpool Football Team.
2. I was in the Wonder Women movie (The first one)
3. I once got mistaken for Tom Hardy
4. I am the 15th descendant of King Henry VIII
5. I worked on a project which directly affected the outcome of the USA Presidential Election

Answer at the end of the article…

Forgive the slight silliness of this piece but it’s all horrific for me and my family, so when I do look back so I try to think of it in a positive light.

Pre-Story Poll

Before you read the story below, log your initial instinct. Your vote will be tallied against our global community data at the conclusion of the file.

Due to my Chronic Cluster Headaches mentioned in the story “The Cloud Has No Mercy” The Cloud Has No Mercy, I was told after many years of devastating pain that I would require an operation and a device called a Deep Brain Stimulator installed as they had tried everything else open to them.

Deep Brain Stimulation (DBS) is an advanced, invasive surgical option reserved for patients with chronic cluster headaches who are entirely refractory (unresponsive) to standard medical therapies.

How the Procedure Works

Target Areas: Surgeons implant permanent electrodes into specific deep brain structures, most commonly the posterior hypothalamus or the ventral tegmental area (VTA), which mediate pain and are involved in the Neurobiology of cluster attacks.

The Device: Small holes are drilled into the skull to guide the electrodes into position. Wires connect these electrodes to a battery-powered pulse generator placed under the skin in the chest, like a heart pacemaker, which delivers regulated electrical impulses.

Now this is a big one, First I was awake for 14 of the 16 hours operation, which allowed the Neurosurgeons’ to monitor if they had placed the electrodes in the correct place, if I started speaking Spanish, they would know they were not positioned correctly. But going back a bit, the day started out with a huge cage being attach to my head via screw holes they had made the night before. This is so they absolutely cannot be any movement during the insertion of the electrode(s), and yes there is redundancy built it. All good.

Went into the surgery, there were at least 10 people in the operating theatre and many other watching in the gallery. All good.

I had asked the nurse if someone could let me see the X-Ray afterwards, they said don’t worry about that, give that nurse your phone and we will get you pictures! Now First off, I have never heard that a patient has been allowed to take what is possible a source of bacteria and germs into an operating theatre with them, let alone have pictures taken of themselves. But they did and it’s remarkable to see.

What happened next will hopefully show how bad these cluster headaches were, in the fact for the entire operation I am smiling, like I cannot wait for the pain to be over. This is Level 10 pain, 12 hours a day for 13 years — Its thought to be the world’s most painful medical condition or the equivalent to giving birth 12 times a day.

They numb the top of my head with Lidocaine and get the hand drill out, yes just like the hand drill your Granddad would have used, except cleaner. They then slowly drill into my brain case until they just get through the bone. Now if you have ever drilled something and the final part is always where the drill goes slightly through until you realise and you pull out quickly, well this happened, because I could feel this is what happened.

Now there was no pain because the Lidocaine had numbed the skin, and of course there are no nerves in the brain. So that scene in the Hannibal Lecter film where he cooks little bits of the characters brain whilst he is still alive is completely possible.

I trusted the Science…

They then have a little discussion and tell me to brace myself. Well, I cannot move anyway so was a bit unsure what was going to happen next. I thought they would gently weave the electrodes they were implanting around the bits of my brain eventually reaching the centre and the Hypothalamus (Walnut sized mass in the centre of the brain, it controls the Body clock and Pain). Oh no, they were going to shoot it straight through the brain a reach the mass with millimetre precision.

These are the images that you can’t see anything.

Long story short, it went well, I answered all the questions asked of me (to check they had not disturbed any part of the brain that controlled anything important) having in their words “just destroyed millions of brain cells, but don’t worry you have plenty more”.

To wrap this up, I was let out of hospital the day after. Major brain surgery and was in hospital 3 days.

All this pain and surgery just because someone picked me up as a kid by my neck and swung me around until two bones snapped. Over the years the bone grows slightly abnormally and therefore Chronic pain.

Still, every cloud, what happened next would not have happened if it weren’t for the Deep Brain Stimulator.

Two set of Electrodes for Redundancy

The Study for DARPA

I was contacted through my Neurologist if I would be part of a study for the University of [Redacted] which was funded by DARPA, the USA’s Defence Advanced Research Projects Agency. They were looking for people with my exact Deep Brain Simulator to do some testing on.

I don’t really know why I said yes, but I did have some tough questions for them before I committed to anything.

I asked — is it a paid study? They said no it wasn’t paid but they would cover expenses. So, I said OK. (I would not do well in an interrogation )

Arrival Day at some University in Southern England

Then came the day, I arrived as you can see and they booked me in as Patient Number 001. I said is this meant to be anonymous? And they said yes which is why we have used Patient Number 001. I said how many others in the program. They said just me, they couldn’t find anyone else up to that point who qualified, and they might not even get their PhD because of that reason.

Then they told me what the study was meant to be for. Its part of a “Super Soldier” program where they were investigating what having one of these implants could do to a human if the settings were altered.

W.T.A.F. !

They would monitor me closely over the course of two nights when I was sleeping and then awake and one of those nights, they would alter the brain implant signal and use settings not allowed via my patient programmer and not even accessible to the doctor programmer device. They would administer a series of tests during the night and when I woke up.

As hard as I tried I still couldn’t get Radio 1

All ready for the first night…

So, this was what I looked like each evening before I fell asleep (With 20 PhD students all watching me through one way glass) Also the kept interrupting me because the electrodes kept falling off during the night.

Night one went OK and when I woke up I got through the testing ok, it was very boring and a bit simple I thought, I was pretty annoyed because I had planned to go and do some tennis serve practice on the courts which were 50 metres away. I could see them from my hospital room. They said I couldn’t leave, I said I was leaving, they got the big boss in, who by this time was annoyed himself as it was a Saturday and he was at home. I said I was leaving the study if I couldn’t go outside, what was the harm.

So, I got to hit about 10 serves and the rain just poured down so I didn’t quite maximise the “prison break” and went back for the second night. I woke up the next day, full of happiness and joy and pleased basically because I was going home. I ran through the tests without even blinking, hard IQ type tests just answering, list 10 things we spoke about yesterday, now tell me them backwards. I would just tell them ten, backwards, forwards anyway they wanted just to get out of there.
Finally I was allowed to go home.

Then it struck me, it was the second night, the one where they turned it onto different settings — the “Super Soldier” settings where I had ace’d through the tests like they weren’t even there and was generally invincible, or were the settings changed on the first night where the next day I was as p-ssed as any human could be and struggled with the tests? Did they change it both days? Did they leave me with the bad one or the good one? Even today life feels a bit like the Manchurian Candidate or Pelican Brief.

I guess I will never know…

The Cyber-Vulnerability Fact

In a terrifying twist of real-world history, the major medical manufacturers responsible for building these advanced thoracic IPG units (who also dominate the global heart pacemaker market) notoriously left glaring security holes in their systems and devices. For years, implantable devices and transmitters faced massive FDA cyber-security roll-outs because firmware lacked basic encryption. Historically, hard-coded master technician keys and default factory settings — colloquially mocked by white-hat hackers as equivalent to leaving the device password as “Password123” — left hundreds of thousands of hearts and neurological pacemakers vulnerable to remote signal interception.

Which is nice…

Even when I had lunch with Tim Berners-Lee all we spoke about was getting the data off and seeing what it was doing — he was speaking on the Semantic Web 3.0 that day and unveiling a ¾ bronze statue of himself and all we spoke about was my brain.

Is it weird?

Er, yes, it’s very weird, when I turn it on I go cross-eyed the wrong way and I can take down a WiFi network in the 5 GHz range by walking into a room, other than that it reduces the pain from a 10 to a 2, and that’s all I care about. Yes, there are side effects and charging yourself up is always a bit strange, especially when people ask where do I plug myself in?

But who cares what I look like, or if I have to suffer the embarrassment of wearing the harness they first gave me, which to be honest looked like I was going for a night out in London’s famous Torture Gardens Fetish club. But I do have a new harness now which is far better as they have upgraded the charging device itself to be a super-charger. Its still working, it’s meant to last approx. 2–3 years and mine has been going for 7 so far.

It will have to be soon as it’s starting to lose its charge, trouble is I doubt they will replace it whilst I have cancer so wish me luck….

Four Truths and a Lie

1. I once had breakfast with the entire Liverpool Football Team. **TRUE
**I did have breakfast with the Liverpool football team when I was 10 years old.

2. I was in the Wonder Women movie (The first one) **FALSE
**I didn’t appear in the Wonder Women movie, but my friend Eugene Brave Rock did. He stayed with me when he was filming it.

Gene Brave Rock & Gal Gadot (Filmed in Leavesden)

3. I once got mistaken for Tom Hardy **TRUE
**I did get mistaken for Tom Hardy and I have been loving this ever since.

4. I am the 15th descendant of King Henry VIII TRUEAt the time I did believe I was the 15th descendant to King Henry VIII but since found out that it was a mistake and actually my DNA is closely related to Beethoven and Oscar Schindler.

5. I worked on a project which directly affected the outcome of the USA Presidential Election TRUEI worked on the Voter Registration system in Tallahassee, Florida in the District Attorney’s Office where Jed Bush had an office next door to ours. I was sent out to fix* the database. This is the election where George Bush became president over Al Gore.

When I say fix I mean “make it better”, not “alter the outcome”, that had already happened 😉

Hope you enjoyed this story, hit the clap button as hard and as fast as humanly possibly if you did.

Running Elite in SQL Server

Fri, 04 Sep 2026 10:53:51 GMT

Doom-in-SQL is dominated by the rasteriser. 320x200 is 64,000 pixels a frame, and every one of them needs a texture lookup, a light level and a depth test. SQL Server is a set-based engine bolted to a storage layer. Asking it to fill spans pixel by pixel is asking it to do the one thing it’s worst at.

Elite’s frame is a few hundred line segments. Twelve ships, twenty-odd vertices each, thirty-odd edges. Transform, cull, project, clip. That’s not a rasterisation problem, it’s a join.

And it goes further than the renderer. Ship blueprints are vertex, edge and face tables with normals and bounty as columns. Face culling is a dot product in a WHERE clause. The galaxy is generated from three 16-bit numbers. The market is seventeen commodities with a base price and a gradient. Elite is a relational database that someone spent forty years pretending was an action game.

Once it’s in tables you can do something no Elite player has ever been able to do. You can query it.

**A quick note on licensing before anyone asks.
**Doom is easy to port because id GPL’d the engine in 1997. Elite was never shareware and is still copyright Ian Bell and David Braben. What exists is Bell’s own sources on his site and Mark Moxon’s extraordinary annotated disassemblies of the BBC Micro, Master, Electron and NES versions. Moxon publishes those deliberately without a licence. For a personal experiment nobody minds; I’m crediting everyone loudly and I’m not shipping a playable binary of someone else’s game.

Getting the ships out

I didn’t want to type in vertex coordinates from memory or from a fan wiki. I wrote a parser that walks Moxon’s annotated assembly directly, pulling the `VERTEX`, `EDGE` and `FACE` macro calls and the `XX21` lookup tables.

The disc version scatters its roster across sixteen per-galaxy source files, so the parser merges and dedupes them. Out the other end: **31 blueprints, 474 vertices, 709 edges, 306 faces**, with the original 8-bit coordinates, face normals and per-edge visibility values intact.

Two blueprints threw an exception at the parser and turned out to be genuine 1984 engineering rather than a bug. The Thargon has no edge list of its own — it borrows the cargo canister’s. The splinter borrows the escape pod’s. Somebody was 24 bytes over budget and went looking for a ship with the same number of vertices.

The full ship roster, rendered from the extracted blueprint data

*Every blueprint, straight out of the parser. Coriolis with its docking slot, the Dodo, the Thargoid, and a cargo canister that is quietly doing double duty.*

The renderer is one view

The whole thing is a single view. Six CTEs, no cursor, no loop:

The Liang-Barsky is the bit I’m quietly pleased with. It unrolls into four rows per segment through a `VALUES` constructor and aggregates them, so a line clip becomes a `MAX` and a `MIN`. A parallel line that starts outside its boundary gets t0 = 2, which forces t0 > t1 and rejects it without a special case.

Elite pins the player at the origin looking down +z and moves the universe instead. So `elite.Universe` coordinates are already camera space, and rotating a blueprint vertex into view is one expression:

p_cam = x * sidev + y * roofv + z * nosev + shipPosition

Backface culling needs a point on each face, but the blueprints only store the vertex-to-face direction. I invert that mapping once at load into `elite.ShipFaceVertex` rather than doing it every frame.

Before running any of it against SQL Server I wrote the same pipeline in Python, stage for stage, and rendered a turntable. That turned out to be the single best decision in the project.

Cobra Mk III turntable, verifying backface culling

*The Cobra through 360 degrees. Edge count swings from 34 nose-away to 17 nose-on, and the engine exhausts on the rear face appear and vanish exactly when they should. A flat edge count would have meant broken culling.*

Then I spent six files chasing a performance ghost

This is the part I’d tell you about in the pub and bore you to tears.

First benchmark: 12 ships, 53ms a frame, 18.7fps. Twenty-seven times faster than Doom, so job done? No. A few hundred rows in 53ms is about 130 microseconds per row. That’s nonsense for float arithmetic in SQL Server. Something was wrong.

What followed was five wrong theories in a row.

Theory 1: fixed overhead. The CTEs get re-evaluated because SQL Server inlines rather than spools them, and `elite.Setting` was CROSS JOINed four separate times, so every supposed constant was really a column reference. Plausible. I ran a scale ladder from 1 to 48 ships. It was linear. Wrong.

Theory 2: recompilation. Temp tables inside a procedure carry statistics, and changing row counts between calls crosses a recompile threshold. Very plausible, classic problem. I captured the recompile counter and built a table-variable control version. Counter read zero. Table variables performed identically. Wrong.

Theory 3: parallelism. A parallel plan on 700 rows is pure exchange overhead. I forced MAXDOP 1. No change. Wrong.

Theory 4: memory grants. Capped the grant. No change. Wrong. (Worse than wrong — I capped the grant on the wrong statement, and made things marginally harder for myself.)

Theory 5: tempdb. I pre-sized it. The wait stats came back with 4ms of I/O across an entire run. Wrong, and it saved me a service restart.

Along the way I broke my own benchmark twice. My first harness reported 24 ships costing more than 48, which is impossible, and I’d been tuning against those numbers for two files. My CPU measurement read `sys.dm_exec_sessions.cpu_time`, which only updates when a request completes — so inside a long batch it reads zero forever. Both had to be fixed before any of the rest meant anything.

What finally cracked it was stage timing. Instrument each statement in the pipeline separately and report medians rather than means:

| Ships | Lines out | Stage 4    |
|-------|-----------|------------|
| 12    | 89        | 6.0ms      |
| 24    | 257       | 44.1ms     |
| 48    | 561       | 120.5ms    |
| 96    | **1,130** | **11.5ms** |

96 ships is ten times faster than 48 while producing twice the output. That’s not a scaling problem, it’s plan instability. And the 48-ship case was burning only 13.4% CPU — it wasn’t computing, it was waiting.

The session wait stats said `SLEEP_TASK` with essentially no I/O. And the query stats, once I got the database context right (`OBJECT_ID` returns NULL if you’re pointed at the wrong database, and then your filter silently matches nothing), named it exactly:

INSERT #edge … AvgMs 118.15 AvgCpuMs 9.75 GrantKB 2656 UsedKB 1632 Spills 96

96 spills. A hash build sized off a bad estimate, repeatedly running out of workspace and re-partitioning. 108ms of the frame was spill handling. Every other statement in the pipeline was clean.

And the punchline: the statement it was happening in was one I had created, in file 9, while trying to optimise the thing. Six files of performance work, and the bug was introduced in the middle of them.

The fix is a `LOOP JOIN` hint. At a few hundred rows, nested loops against the clustered keys is correct and a loop join has no workspace to spill. Normally I’d argue against hinting join types, but this pipeline has a fixed shape and a hard ceiling of a couple of thousand rows. There’s no scene where hashing wins.

Final numbers for a real Elite frame — twelve ships, which is all the original ever put on screen:

Four hundred times faster. Not because I’m cleverer than I was last time— because vector graphics are a join and rasterisation isn’t.

The galaxy is three numbers

This is the part that made the whole project worth doing.

Elite’s 2,048 systems are not stored anywhere. They’re generated by twisting three 16-bit seeds. The twist is four lines of 6502:

s2_new = s0 + s1 + s2 (mod 65536), then shift the seeds down

Four twists move you to the next system. And the four intermediate seeds are also what generate that system’s name, two letters at a time from a 64-character table. So the whole galaxy is one recursive CTE stepping the twist 1,024 times per galaxy, and then a decode that takes every fourth step as a system and the steps between as its name.

Economy, government, tech level, population, productivity, radius and position all fall out of bit manipulation on the seed. Government is bits 3 to 5 of one word. Tech level is the flipped economy plus two bits plus half the government, rounded up.

I checked every constant against the disassembly rather than trusting my memory, which caught two things I’d have got subtly wrong. Tech level uses `LSR` then `ADC`, so it rounds *up* for odd governments. And a name token of zero prints nothing at all rather than producing a character, which is how three-token and four-token names coexist in the same routine.

The validation is unambiguous:

Check_FirstSystem: Tibedied (expected Tibedied)
Check_DistinctNames: 256 (expected 256)
Lave #7 (20,173) TL 5 Rich Agricultural Dictatorship
 population 2.5bn productivity 7000 radius 4116km

And the neighbour list, which is the real proof:

| Destination | Light years |
|-------------|-------------|
| Diso        | 3.6         |
| Leesti      | 3.8         |
| Reorte      | 4.4         |
| Zaonce      | 5.7         |
| Orerve      | 6.8         |
| Riedquat    | 7.0         |

That’s exactly the six systems reachable from Lave on a full tank, in the right order, with Riedquat sitting on the absolute edge of range. Anyone who played this in 1984 knows that list by heart. It came out of three numbers and a recursive CTE.

Then I ran a query nobody could have run in 1984:

| Government       | Systems | Avg tech level |
|------------------|---------|----------------|
| Anarchy          | 32      | 5.0            |
| Feudal           | 32      | 5.8            |
| Multi-Government | 32      | 7.7            |
| Dictatorship     | 32      | 7.7            |
| Communist        | 32      | 8.2            |
| Confederacy      | 32      | 8.9            |
| Democracy        | 32      | 8.3            |
| Corporate State  | 32      | 10.2

Exactly 32 of each. Not approximately — precisely, because government is three bits and the twist distributes them perfectly over 256 systems. And average tech level climbs monotonically with government type, because the tech formula literally adds half the government value.

Bell and Braben encoded the political economy of eight galaxies into three bits and an addition. You can only see that if you can aggregate the thing, which is why it took forty years and a `GROUP BY`.

Trade route optimisation, incidentally, is now a query. Join Lave’s market to every market within 7 light years, match on commodity, order by profit. Forty years of graph paper and folklore, answered by a SELECT.

Flying it

The flight model was the most fun and the most humbling.

Elite ships have no yaw. Like the player, they can only roll and pitch. That’s why an Elite pirate lining up on you rolls you into its vertical plane first and *then* pitches onto you. It’s not stylistic, it’s a consequence of the control axes.

The AI for every ship in the bubble is a single UPDATE. Direction to target, projection onto each ship’s local axes, steering decision, throttle — one statement for the whole scene. The original loops over ship slots one at a time because a 6502 has no alternative. This is the one place where the SQL port is genuinely more natural than the thing it’s porting.

Then I wrote three sign-convention bugs in a row.

The first inverted pitch in the demo pilot, so my ship flew determinedly away from every target. The second was a naive proportional controller that stalls: roll changes x in proportion to y, so once a target is level with you there’s nothing to roll against. The third was in docking, where I controlled the cosine of the slot angle instead of the angle — and the sign of the cosine tells you nothing about which way to roll.

All three had the same shape. Maths that converges on something, just not on what I intended. Invisible from the outcome, obvious the instant you simulate the loop and look at the quantity being controlled.

The fix for all three is `atan2`, which encodes the two-stage manoeuvre directly:

roll = -k * atan2(X, Y) bring the target to twelve o'clock
pitch = -k * atan2(Y, Z) then pull it down onto the nose

Locks on from any starting position in 41 to 76 ticks. Frames stay orthonormal to 5.5e-17 over 400 ticks — worth noting that Elite’s famous `TIDY` routine exists because of 8-bit fixed point, and in a float port it’s almost unnecessary. I kept it. It’s cheap.

First proper dogfight after adding throttle management: **47 ticks, two pirates destroyed, eleven shots fired, zero hits taken.** You have to slow down to turn inside someone, because rotation rates in Elite don’t scale with speed — velocity only widens your turn circle. The AI knew that. My demo pilot didn’t, and spent 330 ticks overshooting.

Docking is four independent conditions: on the slot axis, within the opening rather than the hull beside it, rolled to match a slot that never stops turning, and slow enough that arrival is a docking rather than an impact.

Making the failures explain themselves

The single most useful habit in this project had nothing to do with SQL.

Early on I’d get an empty frame and have no idea why. A ship can be missing for four completely different reasons and a geometry view can’t tell you which. So I wrote a diagnostic view with a `Verdict` column:

behind you
too far, all detail culled
off the edge of the screen
ON SCREEN

Same for docking. When my first attempt crashed, the output read:

AxisDot 1.000 LateralOff 0 SpeedOK 1 RollMatch 0.042
CRASH: not rolled to match the slot

Textbook approach, dead centre, correct speed, rolled ninety degrees wrong. That’s a five-second diagnosis instead of an evening.

Every time I skipped this step I lost hours. Every time I did it I lost minutes.

The cockpit

The client is Python and tkinter — standard library, no display dependencies at all. All the game logic stays in the database; the client is a screen and a keyboard.

The Elite cockpit, rendered from SQL Server

Vector view, instrument bars, and Elite’s ellipse scanner with the vertical stalks. The scanner is just a different projection of the same `elite.Universe` rows the 3D view uses.

The one thing that makes it viable at 30fps is a single stored procedure returning three result sets — the vector frame, the instruments, and the scanner contacts. A naive client would issue a tick, then a frame query, then a status query. Three round trips a frame, ninety a second, for no reason.

First time I actually flew it, I died instantly whatever I pressed. Two compounding causes, both mine. The client asks for a frame every 33ms, so the game runs at 30 ticks a second where the BBC managed about 12 — everything “per tick” happens two and a half times too fast. And I’d scaled NPC laser damage 6x earlier to make the *autopilot’s* fights resolve quickly. Together: 255 shields plus 255 energy at 12 damage a tick is 42 ticks. About 1.4 seconds.

Combat constants now live in a table so they can be tuned while the game is running. It turns out “the numbers I picked to make a script’s fights interesting” and “the numbers a human can survive” are not the same numbers.

Where it got to

- 31 ship blueprints, parsed from annotated 6502 source
- A renderer that does a real Elite frame in 3.51ms
- 8 galaxies, 2,048 systems, from three 16-bit numbers
- Flight model, set-based ship AI, combat, and docking against a rotating station
- Trading, cargo, fuel and hyperspace
- A Python cockpit

Dock, trade, refuel, jump, repeat. The loop closes.

What I’d tell you if you’re thinking of doing something like this

Write the reference implementation first. Every geometry bug in this project was caught in seconds by a Python model of the same pipeline, and every one I tried to reason about instead cost me an evening.

Make failures name themselves. A boolean that says “no” is worthless. A column that says *which condition failed* is the difference between minutes and hours.

Don’t trust a benchmark you haven’t broken yet. Mine reported impossible results twice. Report medians, not means. Run the same case twice and see if it agrees with itself. Check that your CPU measurement actually measures CPU.

Check the constants against the source. Every value in the galaxy generator came from Moxon’s disassembly rather than my memory, and two of them would have been subtly wrong if I’d trusted myself.

And pick the right problem for the tool. The reason this runs 400 times faster than my Doom port isn’t cleverness. It’s that vector graphics are a join and rasterisation isn’t. SQL Server was never bad at Doom’s maths. It was bad at Doom’s shape.

Credits

Elite is copyright Ian Bell and David Braben. It’s one of the great pieces of software engineering and it deserved better than being ported to a database.

Every constant, blueprint and algorithm here was read out of Mark Moxon’s annotated disassemblies at [bbcelite.com](https://www.bbcelite.com) — a genuinely remarkable act of software archaeology, and the reason this project took weeks rather than years.

The ship blueprint data is extracted from those sources for a personal experiment. Don’t redistribute it, and go and read Moxon’s commentary instead. It’s better than this article.

Code is here and just run in the order of the files.

https://github.com/sqlartist/EliteSQL

I can’t imagine there will be too many people reading this that will even know what Elite is or was and what an impact it made on the history of Computer Games but if you do then please give it a clap. Thank you.

Sum Over Every Path

Tue, 01 Sep 2026 22:25:26 GMT

I’ve spent a while building things inside SQL Server that have no business being there. This one started with a question I could not get rid off: what would it take to score a line of text data not by checking it against rules, but by considering every possible reading of it at once and letting the best one fall out?

That question came from Feynman.

1. The idea

Feynman’s formulation of quantum mechanics rests on a move that still feels like cheating the first time you hear about it. Typically, a particle going from A to B takes one trajectory — the one that makes the action stationary. Feynman’s postulates say something different: the particle takes every path. Each contributes an amplitude of equal magnitude, and the phase of that contribution is the classical action in units of ħ. Add them all up and the paths that disagree cancel; the ones near the classical trajectory reinforce. The single path you observe isn’t chosen. It’s what’s left standing once everything else has interfered itself out of existence.

What grabbed me wasn’t the physics. It was the procedural shape of it. Don’t decide early. Don’t prune. Enumerate everything, cost everything, and let the winner emerge from the sum.

That’s a strange way to write a parser. Most text processing is a pipeline of early commitments — tokenise, then tag, then match, then correct — each stage throwing away information the next stage might have wanted. Feynman’s move is the opposite. Carry all the possibilities forward and settle at the end.

Its pretty similar to the way I learn anyway, not having the interest in maths at school because they just didn’t teach it in context. Not once did a teacher say why you needed COS, TAN, SIN and why we were doing the calculations and angle measurements — not once did they say the reason for all this is because if you look at the Earth which is nearly round you can work out all this amazing stuff. The only thing that sunk in at school was Pythagoras’ theorem, but they never said where it would be useful. So with this lack of knowledge I stopped doing 4 A Levels and got a job in a Company that made ATM software and never looked back.

So this idea of Feynman’s was appealing to me because its the way I have been working ever since — try every possibility there can possibly be and based on the statistics I have kept, we will see what falls out the end. So with that…

I wanted to see what that looked like in T-SQL.

2. What it does

Given a line of firewall log text, the engine produces the most likely reading of it with misspellings corrected, and a score for how coherent that reading is.

Firewalz oftehn blok malicius payloadds comes back as firewalls often block malicious payloads, with all five corrections marked and a note that the input was heavily repaired. kubernetes reconciled the deployment manifest successfully — perfect English, entirely outside its vocabulary — comes back unchanged, with a score reflecting that.

Everything runs in SQL Server. No CLR, no external process, no Python scripts. Stored procedures, temp tables, one scalar function, and a lot of set-based work. Like before I should have stopped and just had a nice cup of tea and a lie down. I didn’t and here we are.

3. How it’s built

3.1 The lattice

Each token position gets several candidate readings: the word as written, plus lexicon entries it might be a misspelling of. Candidates come from two text processing favorites — a phonetic match on SOUNDEX, or a shared initial letter within a tight length window — and are then priced by edit distance.

That gives a lattice. Position one might read Firewalz or firewalls; position two oftehn or often. A path through it picks one candidate per position, which makes it a complete reading of the sentence.

Trust me, this is the most basic of text processing you can do…

Each node carries a cost for being read that way:

INSERT INTO #LatticeNodes (PositionID, ObservedToken, CandidateToken, POSTag, NodePenalty, IsCorrection, InLexicon)
SELECT
    d.SequenceID,
    d.Token,
    d.Token,
    COALESCE(lex.POSTag, d.LiteralTag, N'NN'),
    CASE
        WHEN lex.Token    IS NOT NULL THEN 0.0   -- in lexicon as written
        WHEN d.LiteralTag IS NOT NULL THEN 0.0   -- a number or version string
        ELSE @OovPenalty                          -- unknown, and staying that way
    END,
    0,
    CASE WHEN lex.Token IS NOT NULL THEN 1 ELSE 0 END
FROM #InputDocument d
LEFT JOIN dbo.LexiconToken lex
       ON lex.Domain = @DomainContext AND lex.Token = d.Token;

The observed reading is always present, so the decoder can always decline to correct. Correction candidates go in alongside it, priced at @CorrectionWeight × edit distance, plus an extra penalty if the observed word was already in the lexicon.

3.2 The costs

Transitions run strictly from position i to position i+1, so every path covers the whole sentence. Each one prices the pairing:

CASE
    WHEN sem.BigramAction IS NOT NULL           THEN sem.BigramAction
    WHEN n1.POSTag = N'CD' OR n2.POSTag = N'CD' THEN @LiteralAdjacentCost
    WHEN (n1.InLexicon = 0 AND n1.HasCandidate = 0)
      OR (n2.InLexicon = 0 AND n2.HasCandidate = 0) THEN @OovAdjacentCost
    ELSE @UnknownBigramCost
END
  + ISNULL(pos.FieldPenalty, @UnknownGrammarCost)

An verified bi-gram is best evidence — -log P(w₂|w₁) fitted from a training corpus. Failing that, if either side is a number or a version string, no bi-gram table could ever have contained it, so allocating 100% would punish every log line containing an IP address. Failing that, an out-of-lexicon word with no near neighbour is unverified for the same reason. Only when both words are known and the pairing genuinely wasn’t seen, do you pay the full unverified cost. The grammar penalty, -log P(tag₂|tag₁), is added on top.

3.3 The trellis

Then the Feynman move: weight every complete path by e^{-αS} where S is its total cost, and normalise across the whole data set. Low-cost paths will dominate, high-cost paths vanish exponentially, and the winner is whatever survives.

The naive way to do that is to enumerate the paths. Don’t (see 5.2). One pass over positions computes both the best path and the sum over all paths, without recording any of it:

DECLARE @p INT = 2;
WHILE @p <= @MaxSequence
BEGIN
    WITH Incoming AS (
        SELECT tr.ToNodeID, tr.FromNodeID,
               prev.BestCost + tr.StepCost                 AS CandCost,
               prev.LogAlpha - @SoftmaxAlpha * tr.StepCost AS LogTerm
        FROM #Transitions tr
        INNER JOIN #Trellis prev ON prev.NodeID = tr.FromNodeID
        WHERE tr.ToPosition = @p
    ),
    Agg AS (
        SELECT ToNodeID, MIN(CandCost) AS MinCost, MAX(LogTerm) AS MaxTerm
        FROM Incoming GROUP BY ToNodeID
    ),
    Summed AS (
        -- shift by MaxTerm before exponentiating: no underflow
        SELECT a.ToNodeID, a.MinCost, a.MaxTerm,
               SUM(EXP(i.LogTerm - a.MaxTerm)) AS ExpSum
        FROM Agg a INNER JOIN Incoming i ON i.ToNodeID = a.ToNodeID
        GROUP BY a.ToNodeID, a.MinCost, a.MaxTerm
    ),
    BestBack AS (
        SELECT ToNodeID, FromNodeID,
               ROW_NUMBER() OVER (PARTITION BY ToNodeID
                                  ORDER BY CandCost ASC, FromNodeID ASC) AS rn
        FROM Incoming
    )
    INSERT INTO #Trellis (NodeID, PositionID, BestCost, BackNodeID, LogAlpha)
    SELECT n.NodeID, @p,
           s.MinCost + n.NodePenalty,
           b.FromNodeID,
           s.MaxTerm + LOG(s.ExpSum) - @SoftmaxAlpha * n.NodePenalty
    FROM #LatticeNodes n
    INNER JOIN Summed   s ON s.ToNodeID = n.NodeID
    INNER JOIN BestBack b ON b.ToNodeID = n.NodeID AND b.rn = 1
    WHERE n.PositionID = @p;
      SET @p += 1;
  END

Three things happen in that one insert. MinCost plus a backpointer is Viterbi — the cheapest way to reach this node. LogAlpha is the forward algorithm in log space — the total weight of every path reaching this node. And subtracting MaxTerm before exponentiating is the log-sum-exp trick, which matters because a twenty-token sentence can accumulate an action in the hundreds and EXP(-400) is zero in floating point. Without the shift, every path scores zero, the denominator is zero, and the whole thing silently returns nothing.

It’s O(n·N²) in tokens and candidates. No recursion, no enumeration.

Reading the answer back is a walk down the backpointers, which is the only recursion left and is linear in sentence length:

;WITH Walk AS (
    SELECT t.NodeID AS TerminalNodeID, t.NodeID, t.BackNodeID
    FROM #Trellis t WHERE t.PositionID = @MaxSequence
    UNION ALL
    SELECT w.TerminalNodeID, p.NodeID, p.BackNodeID
    FROM Walk w INNER JOIN #Trellis p ON p.NodeID = w.BackNodeID
)
SELECT
    CAST(STRING_AGG(CAST(CASE WHEN n.IsCorrection = 1
                              THEN n.ObservedToken + N'->' + n.CandidateToken
                              ELSE n.CandidateToken END AS NVARCHAR(MAX)), N' ')
         WITHIN GROUP (ORDER BY n.PositionID) AS NVARCHAR(MAX)) AS DecodedText,
    ...

ObservedToken and CandidateToken are both kept deliberately. A log-analysis tool that silently rewrites your logs is worse than useless — you need to see that oftehn became often, not just the tidied result. Although having said that, it would make a fine “cover your tracks” tool.

4. Where the physics stops

Remember when I started the article I said Feynman’s work inspired me to write this, I am not claiming to have solved Quantum mechanics or done anything other than used the idea of parsing everything at once and seeing from all possible paths which one succeeds.

This is not a quantum process and it is not a path integral. Feynman’s amplitudes are complex: e^{iS/ħ}. The entire content of the path integral is that phases can cancel — destructive interference is what kills the non-classical paths. My weights are real and positive. Nothing ever cancels. Paths only ever add.*

What I actually built is a Boltzmann-weighted sum over paths, which is a softmax over path costs, which is — to say it plainly although its actually not any plainer — the forward algorithm over a linear-chain conditional random field. An Hidden Markov Mode (HMM) decoder in a physicist’s coat. That’s is work that has been used in NLP for many years and once again I am thwarted by those pesky scientists and their meddling ways of thinking of everything possible many years ago.

You can’t teach a monkey new tricks

So in all honesty my claim is weak: Feynman’s postulate is where the design template came from. Include everything, cost everything, let the sum decide. The maths I ended up with is more thermal, not quantum. The inspiration was real; the isomorphism isn’t. Considering I self taught this stuff I think I’m doing as well as those kids with iPhones.

*If anyone has access to a server that could actually do some quantum processing that would be nice…

5. Seven things that were wrong

Most of these produced absolutely plausible output for many runs of this code, but its because the sample data is just way too small.

5.1 `STRING_SPLIT` doesn’t guarantee order

I was assigning token positions from an IDENTITY column populated by INSERT ... SELECT FROM STRING_SPLIT(...). Token order is the entire basis of a sequence model. The optimiser is free to return those rows in any order it likes and will change its mind under a different plan or parallelism. It worked on my test data by luck. Fixed with OPENJSON and an explicit ROW_NUMBER() over the array index.

5.2 The engine was deleting evidence

My early versions enumerated paths literally, with a recursive CTE that could skip ahead up to five tokens. Two problems.

Path count grew as a Pentanacci sequence: five tokens gave eight paths, twenty gave 1.4 million, forty wouldn’t finish.

Worse than that was routing through a token cost two transitions while skipping over it cost one, so skipping was almost always cheaper. When the engine met a word it didn’t recognise, its cheapest available explanation was to pretend the word wasn’t there. It was deleting evidence to lower its own score. On a log-analysis tool. Again one for the actual hackers.

5.3 The ensemble was empty

Requiring every transition to advance exactly one position fixed the deletion bug and immediately exposed a deeper one: with full coverage and one candidate per token, there is exactly one path. The probability is 1.0 by definition. All the recursion was doing was summing over a set with one member.

That’s what forced the lattice. Real alternatives have to come from competing actual counts at each position, not from skipping them. Once the lattice existed the recursion became unnecessary, because a trellis gets the same answer without enumerating anything.

So the naive implementation of “consider every path” died, and the postulate survived it. I think that’s the actual lesson: the physics gave me the right instinct, and I was correct eventually…

5.4 Normalising edit distance hid whole edits

I was dividing edit distance by token length, so conection → connection (one edit) scored 0.100 and conection → connections (two edits) scored 0.182. A whole extra edit was worth 0.08.

Meanwhile the bigram model had a strong opinion. The training corpus attested inbound connections but never inbound connection, a difference worth about 0.93. So the language model dragged the decoder onto the wrong inflection and the channel model was too quiet to object. Raw edit distance fixed it. Normalising by length is not standard for a channel cost, and this is why.

5.5 Plain Levenshtein double-charges transpositions

After that fix, recall dropped from 94% to 80%. Five of the six new misses were transpositions: friewalls, unauthoirzed, connectiosn, immedaitely, endopint.

Levenshtein scores an adjacent swap as two edits — a deletion and an insertion — so the single most common human typo was priced at double. Damerau-Levenshtein counts it as one:

-- Damerau: an adjacent transposition is ONE edit.
IF @i > 1 AND @j > 1
   AND @Ch     = SUBSTRING(@Right, @j - 1, 1)
   AND @ChPrev = SUBSTRING(@Right, @j,     1)
BEGIN
    SET @Trans = UNICODE(SUBSTRING(@Row2, @j - 1, 1));
    IF @Trans < @Best SET @Best = @Trans;
END

The DP row is packed into an NVARCHAR, one code unit per cell, offset by one because NCHAR(0) returns NULL. Damerau needs two previous rows rather than one, hence @Row2. That change alone recovered every transposition miss.

5.6 My back-off was subsidising typos

I’d added a discount so transitions involving out-of-vocabulary words weren’t charged as though the pairing had been contradicted. Reasonable I thought — an IP address can never appear in a bigram table.

But it also meant keeping a misspelling got the discount while correcting it paid full price. ibound → inbound is a single edit and the engine still refused, because leaving the typo alone saved 1.2 on transitions and only cost 1.0 on the node. It was cheaper to be wrong.

The fix is the HasCandidate clause in 3.2: a word one edit from a real word is a suspected typo, not a legitimately unknown one, and doesn’t get the discount.

5.7 `SELECT` assignment doesn’t null out

This issue cost me an entire evening watching a query never finish.

-- broken: loops forever
SELECT TOP 1 @DocID = DocumentID FROM dbo.DocumentQueue
WHERE ProcessedStatus = 'Pending' AND DocumentID > @DocID ORDER BY DocumentID;

When a SELECT assignment matches no rows it leaves the variable unchanged. So after the last document, @DocID keeps its value, WHILE @DocID IS NOT NULL stays true, and the final document is re-parsed forever.

-- correct: SET with a scalar subquery assigns NULL on an empty result
SET @DocID = (SELECT MIN(DocumentID) FROM dbo.DocumentQueue
              WHERE ProcessedStatus = 'Pending');

That difference between SET and SELECT is the entire bug. I now also keep a guard counter, so if the termination condition ever breaks again it raises rather than hangs.

6. Measuring it

I got tired of judging this by eye, so I built a harness.

A 45-line training corpus of firewall log phrasings supplies the fitted costs: 150 lexicon tokens, 158 bigrams, 24 POS transitions. A separate 49-line test corpus — different sentences, same domain, 87% bigram coverage — supplies the evaluation. Typos are generated by deterministic single-character mutation.

Crucially, every intended correction is verified by the candidate generator before it goes into the corpus. That check found three targets — ibound → inbound, ivalid → invalid, deied → denied — that were impossible by construction, because SOUNDEX codes consonants positionally and dropping an early one moves every subsequent code. Without the check they’d have shown up as decoding failures that no amount of tuning could fix. They’re the reason the candidate generator has a second, non-phonetic recall channel.

Scoring is token-level: a token that was wrong and got fixed is a true positive, one that was right and got broken is a false positive.

7. Results


Category                 TP FP FN TN Precision Recall
Single typo              10 0 0 56    100%     100%
Multiple typos           25 0 0 13    100%     100%
Clean                    0  0 0 66    —         —
Out of vocabulary        0  0 0 29    —         —
Scrambled word order     0  0 0 23    —         —
Literals / punctuation / 
multi-sentence           0  0 0 89    -         —
All                      35 0 0 285   100%      100%

-- And the coherence score separates cleanly by category:

Category                 Min   Avg   Max
Clean                   0.247 0.380 0.789
Single typo             0.247 0.380 0.789
Multiple typos          0.247 0.391 0.789
Short lines             0.250 0.555 0.675
Literals (IPs, versions)0.640 0.785 0.956
Punctuation-heavy       1.068 1.234 1.474
Scrambled word order    1.384 1.426 1.479
Out of vocabulary       2.522 2.928 3.342

8. Two different questions

Look at the first three rows of that table. Clean, single-typo and multi-typo lines have identical ranges. That’s deliberate, and it took me a while to see why it should be.

Originally the score included the cost of the corrections, so a line with four typos that decoded perfectly still read as low coherence. Arguably defensible — the input really was degraded — but it meant the number couldn’t answer the question I actually cared about, which is how good is the reading I just produced.

So the engine now tracks three costs: the total, the part spent repairing spelling, and the remainder. The verdict reads the remainder.

TotalActionCost    FLOAT NOT NULL,   -- how degraded was the input?
CorrectionCost     FLOAT NULL,       -- what did the repairs cost?
DecodedActionCost  FLOAT NULL,       -- how coherent is the result?The effect is exact. Every typo line now scores identically to the clean line it was derived from — test 16 went from 1.0972 to 0.7895, matching clean line 2 to four decimal places; test 25, five typos in five words, went from 2.1482 to 0.3300, matching clean line 1. Input quality is reported separately via the correction count.

It also removed an overlap that was making the verdict bands impossible to calibrate. Before the split, corrected lines ran up to 1.097 while punctuation-heavy lines started at 1.068, so no single threshold separated them. Afterwards everything that should read green tops out at 0.956 and everything that shouldn’t starts at 1.068. An actual gap.

9. The parameter sweep

Two constants control the correction behaviour, and they’re coupled. @CorrectionWeight is the cost per edit; @OovPenalty is the cost of leaving a token alone. I’d picked 2.0 and 3.0 by educated guess work and wanted to know whether that was a real optimum or just lucky, so I swept a 5×5 grid, re-running the whole corpus in each cell.

F1 by cell, rows @CorrectionWeight, columns @OovPenalty:

The control variable is O − W. Line the cells up by the difference rather than the values and the recall side is almost perfectly determined:

THE CONTROL VARIABLE & WEIGHTS

Identical along each diagonal. Which makes sense: O − W is the margin by which correcting a token is better leaving it alone, so it decides whether a one-edit fix fires at all. Below +0.5 it doesn’t, and recall falls off a cliff — 2 misses at zero margin, 12 at −1.0, 25 at −1.5.

But there’s a second, independent constraint. The *W = 1.0* row breaks the diagonal pattern and never reaches 100% at any OOV penalty, carrying a floor of 2 false negatives that no amount of O fixes. That’s not a margin problem. It’s the channel model being too quiet to outvote the bigrams — exactly the inflection failure from §5.4, reappearing on its own from a completely different direction.

False positives tell the other half. They occur only at low W with high O: five at (1.0, 3.0), three at (1.5, 3.5), zero anywhere with W ≥ 2.0. A three-edit correction costs 3W and becomes attractive once that drops below O plus the bigram saving. At W = 1.5 that’s 4.5 against roughly 4.6, so *resumed → reset* sneaks back in. At W = 2.0 it’s 6.0 against 5.1 and stays out.

So: W ≥ 1.5 for the channel model to be heard, W ≥ 2.0 to be immune to over-correction, and O − W ≥ 0.5 for corrections to fire at all. Eight cells satisfy all three. (2.0, 3.0) is the most interior of them, with three of four orthogonal neighbours also at 100%.

The hand-derived values turned out to be right, but I’d have had no way of knowing that without the grid — and more importantly, no way of knowing why.

10. Making it fast enough to sweep

The first sweep attempt didn’t finish. Twenty-five cells at several minutes each is an afternoon, and I killed it so many times before working out where the time was going.

fnEditDistance is a scalar UDF with a nested loop, called for every lexicon word that survives blocking. It sat inside a CROSS APPLY over a derived table, and a derived table is not an optimisation fence — SQL Server flattens them routinely. If the blocking predicates don’t get pushed below the function, the UDF runs against all 150 lexicon rows per token instead of the six that survive. That’s 48,000 calls per cell instead of 2,000.

The fix is one that should have been obvious from the start: edit distances don’t depend on the cost weights at all. All 25 cells were recomputing byte-identical distances. So they’re computed once per distinct token, ever, and cached.

The population is deliberately split into two statements, which is the part that matters:

-- Step 1: blocking only. Lands in a real table.
INSERT INTO #Blocked (Token, LexToken, POSTag)
SELECT n.Token, lex.Token, lex.POSTag
FROM #NewTokens n
INNER JOIN dbo.LexiconToken lex
        ON lex.Domain = @DomainContext
       AND lex.Token <> n.Token
       AND ABS(LEN(lex.Token) - LEN(n.Token)) <= @CacheCeiling
       AND ( DIFFERENCE(lex.Token, n.Token) >= 3
             OR ( LEFT(lex.Token, 1) = LEFT(n.Token, 1)
                  AND ABS(LEN(lex.Token) - LEN(n.Token)) <= 2 ) );

-- Step 2: separate statement, so the function can only ever see survivors.
INSERT INTO dbo.CandidateCache (Domain, ObservedToken, CandidateToken, POSTag, RawEdit)
SELECT @DomainContext, b.Token, b.LexToken, b.POSTag, e.RawEdit
FROM #Blocked b
CROSS APPLY (SELECT dbo.fnEditDistance(b.Token, b.LexToken) AS RawEdit) e
WHERE e.RawEdit > 0 AND e.RawEdit <= @CacheCeiling;

Inside one query the optimiser may reorder. Across two statements it cannot. Roughly 1,500 distance computations now cover the entire sweep, against 1.2 million in the flattened worst case. Cells went from minutes to under a second, and the output is byte-identical to the uncached run — which is the check that matters for a pure optimisation.

11. What this doesn’t show

A perfect score on a corpus I built myself is the point at which I should be more suspicious, not less.

The lexicon is 150 tokens in one domain. The corruptions are synthetic and uniform; real typists produce a messier distribution than four deterministic mutation rules. Every correction was pre-verified as reachable, which removes an entire failure mode that real text will happily supply. And 49 lines is small enough that one example moves the numbers several points — with 35 true positives.

The score also measures vocabulary coverage more than it measures coherence. kubernetes reconciled the deployment manifest successfully is flawless English and scores worse than anything else in the corpus. That’s the model reporting it has no vocabulary for the sentence, but read the number as “is this good English” and you’ll be misled.

And I know of at least three things that will break on real data:

The **HasCandidate** fix inverts at scale. With 150 words, few tokens have a lexicon neighbour. With 30,000, almost everything has one within three edits, so almost nothing gets the out-of-vocabulary discount and hostnames start being charged full price on both sides. The predicate probably needs tightening to “no neighbour within one edit”.
Blocking becomes a scan. DIFFERENCE(lex.Token, ...) and LEFT(lex.Token, 1) are functions on the column, so there’s no seek. Invisible at 150 rows, painful at 30,000. Wants persisted computed columns and indexes.
Candidate ranking gets crowded. At this size the nearest word by edit distance is almost always the right one. At scale there will be a dozen words at distance one, and the ordering will need unigram frequency.

The next step isn’t more tuning. It’s real firewall log lines, with real errors in them, and finding out which of those three predictions was right.

12. Why bother

Sometimes, just sometimes I wish I never started these things.

None of this belongs in a database. You could write it in fifty lines of Python with a library better than mine.

But the constraint keeps teaching me things. Forcing a sequence model into set-based SQL made the exponential blow-up impossible to ignore, because a recursive CTE that fans out to a million rows doesn’t degrade gracefully — it stops. It made the ordering assumption visible, because SQL is the one language that will cheerfully reorder your data and be entirely within its rights. It made every cost explicit, because there’s nowhere to hide a heuristic in a CASE expression. And it produced at least two bugs — the SELECT-assignment loop, the flattened derived table — that simply don’t exist in other languages.

The Feynman thing was never a claim. It was a way in: a specific, unusual instruction about how to organise a computation, borrowed from somewhere it had already proved itself. Consider everything. Cost everything. Let the sum decide.

The maths I ended up with has a different name, and I should use that name. But I wouldn’t have got there any other way.

As Promised: https://github.com/sqlartist/feynman-sql

If this was interesting, a few claps help other people find it. Follow me for more on Data.

The Cloud Has No Mercy

Sun, 30 Aug 2026 12:00:37 GMT

Figure 1: Trading the agony of a 56k dial-up interruption for the looming dread of the infinite, faceless server farm.

I was considering what it must be like for today’s generation of users, navigating technology while quite rightly taking it for granted. If you asked half of them how a file magically appeared on the screen of the device they are holding, they’d shrug and say, “Well, Jason sent it — duh!” The other half are actively plotting world domination from a handed-down iPhone.

To go off topic slightly but it fits the timeline although I am tempted to take it out and make it a separate read, but readers of a certain age may enjoy the diversion.

Nostalgia — aww, do you remember when…? Wasn’t the music… It was the best time ever, it will never be beaten, life was far simpler.

Er no, I barely remember, the music is always good except for the 1980’s and I’m including fashion in that, it wasn’t the best time ever, and yes if you mean life was tediously boring with 3 TV channels, having to watch multiple episodes of the most depressing soap operas in history multiple times in the same day, it’s no wonder I developed a near identic memory — there was very little actually going on.

I suffer from chronic cluster headaches from aged 30ish onwards probably caused by someone at my school breaking my neck, A 16-year-old, 4 years ahead of me, picked me up by my head and swung me around. I felt 2 clicks in my neck and was left on the floor not moving. Not knowing any better my school friends picked me up and carried me to the matron’s office. Long story short I was in a neck brace for a year, got excused from any sport at school and ended up in the library or computer room if they were unlocked. Computers were not switched on of course but still to be trusted to be in the most expensive room in the school was a bonus… or was it?

Quick diversion as this is probably the single funniest thing to ever happen at that school. The day I got carried to the matron’s office, my parents were called and my mum arrived first and as you would expect went crazy, I was just sat there unable to move watching this play out. Quite a crowd was building outside the matron’s office. My mum was being calmed down by the deputy head (worst thing he could have done to be honest), quite a feared teacher and about the same height as my mum. When suddenly, my dad burst through the double doors straight from work (No mobile phones back then so he got a third hand message and didn’t know what to expect). They were both shaking when they saw me. The matron had not called an ambulance or done anything other than given me a hot cup of tea.

To this day what happened next, I can remember word for word — the exact beat of what was said and the cheer that went up at the end from the entire sixth form who were still listening from outside the office, and before this moment I only heard about the teacher’s reputation from other kids.

My dad was about a head taller than this guy who had to look up as my dad was standing close to him and spoke

“You were a f*cking little Hitler when I was here, and you still are a f*cking little Hitler….you c*nt! Right, we’re off…”

He picked me up and we went straight to Accident & Emergency at the hospital. When my dad said this the entire sixth form crowd erupted in a massive cheer — no one liked this guy.

That was quite a diversion and unexpected turn for this article but simply for the one-line insult it was worth it and of course for the near Oliver Twist experience living through the 1970’s and 1980’s in a very nice village in Hertfordshire, 21 miles away from Central London.

I was the only pupil in the entire school to ever at that time be placed on Work Experience in London, quite a thing to be honest and it was with a computer Company called ICL — had no idea what they did as internet didn’t exist and no phones. I did have a computer at this point but was still under the impression that I had to learn to be a quick typist as the programs I found in magazines were way too long for me to type it in before the C15 tape ran out. I mean why weren’t computers allowed to use a C60 or C90 cassette tape? (for those that know will immediately see how my logic was flawed about the tape running out) But there was literally no information apart from a programming book my Godfather brought for me titled

“The Century Computer Programming Course for the ZX81 and Spectrum”

Figure 2: A 570 page amazing book if I had a ZX81, which I didn’t

which was very handy for someone that had a Commodore 16 and no clue there were different versions of BASIC. I was using it as a riser for an overheating laptop until last year.

Back to the main piece.

I guess they never had the baggage of downloading a single image over a 56k modem from CompuServe or MSN. They don’t know the absolute heartbreak of your mum picking up the landline at 99% completion, completely ignoring those repeated, desperate pleas: “I know it’s your phone and you pay the bill, but please do not make any calls for the next five hours!”

It was right around that point — somewhere near the start of three decades deep in SQL databases — that I changed my entire trajectory. The singular goal? To never have to worry about that kind of expense or limitation ever again. We wanted to build systems that were faster, better, and limitless.

But just as a piece of ancient vellum will comfortably outlast a carefully curated collection of 8,000 decaying DVDs, our technological trade-offs are rarely what they seem. We successfully eliminated the agonizing wait times and the physical friction of the 90s, but we replaced them with something much more insidious: the Cloud and AI.

Some genius came up with the idea of limitless, instantly accessible compute, and it is glorious — right up until the moment it threatens to literally bankrupt you. They don’t even have the decency to invoice you upfront; you pay the bill after the algorithms have had their fun.

Figure 3: Nimbus was the computer used at my school in 1986. It had a 10MB Winchester Hard Drive. I had to include a picture of that beast at the end of the article.

Between various Max subscriptions and managed cloud services, we have entered an era of unprecedented ruthlessness. If the money isn’t in that account the exact millisecond they check, they cut you off. They make unveiled threats to wipe your entire online history so quickly it makes your head spin. I mean, there should be a Nobel Prize for customer service this aggressively efficient. Even when you die, your loved ones don’t get processed with that kind of cold, calculated precision.

Therein lies the danger of the “Jason sent it” mindset. When we abstract away the hardware and stop questioning how the underlying infrastructure actually works, we lose control. We traded the physical annoyance of dropped connections for a world where a faceless billing algorithm acts as an unforgiving landlord over our digital lives.

The lesson here isn’t to go backwards, but to look beneath the hood. Understand the compute cycles you are spinning up. Keep a grip on your data sovereignty, because renting infinite power means you can be evicted in a heartbeat.

Still, I suppose I should be grateful. At least I never had to punch little holes into a card and wait overnight to see if it had worked, just praying the mainframe hadn’t eaten the entire batch by morning…

Or Fought in a War,

Or had an AOL subscription

Figure 4: The 1986 RM Nimbus Winchester 10MB Hard Drive. A glorious, heavy beast that safely held less data than a single modern smartphone app notification.

If this was interesting, a few claps help other people find it. Follow me for more on Data.

Feynman’s Postulates and Relational Text Analytics in T-SQL

Thu, 27 Aug 2026 16:46:17 GMT

https://www.newscientist.com/article/2586608-richard-feynmans-80-year-old-quantum-postulate-has-now-been-validated/

Well thats just wonderful for all those scientists but here in the real world and someone whose life has been affected by Feynmans work its is no surprise.

A bit of history, if you read any book or watch any documentary about a person read or watch “The Feynman Lectures on Physics”. I watched these and maybe understood 5% of the content, but what I got from it was Feynman’s energy and passion for his subject. A more readable for all the non mathematicians and physicists is “ Surely You’re Joking, Mr. Feynman!”. There is also a BBC documentary on Youtube well worth watching.

So affected by his life’s work on the Manhattan Project, Safe Cracking, Bongo playing, Lap Dancing and it mirroring my own life I got a tattoo with one of his famous (he won the Nobel prize for these) Feynman diagrams.

Yes, an actual Tattoo in tribute to Feynman Diagrams

I am also making a model of his famous Feynman Van.

The Feynman Van

So, regardless of the scale of the end result I think he would be happy people use his ideas in unexpected ways. From a theory he made 80 years ago that has been validated by *proper* scientists has now been used to process some text in a SQL Database.

Abstract

Traditional natural language processing (NLP) inside classical relational databases relies on sequential string-splitting logic, cursors, or loops that process data character-by-character or by word-by-word. This article presents an alternative paradigm: a native text-analytics engine built entirely inside local Transact-SQL (T-SQL) that mirrors Richard Feynman’s “all paths” quantum postulate. By processing tokenisation, TF-IDF weighting, and multi-dimensional vector space layout via parallel set operations, I demonstrate how text can be analyzed, clustered via K-Means, and visually mapped on a physical canvas using SQL Server’s native Spatial Geometry view.

1. The Core Philosophy: Sequential Reading vs. The All-Paths Matrix

The Normal Text Processing Standard (Sequential Explorer)

I wrote my first version of this back in 1999 and found there was little interest in it. A fellow MVP Hillary Cotter was a great help back then when creating these ideas. I guess we weren’t mixing in the right circles as it didn’t go anywhere really. Fast forward 30 years…

When a typical program parses text, it acts like an explorer manually walking through a maze. To extract keywords, phrases, or N-Grams, it reads a sentence word-by-word. It picks up the first word, looks at it, moves its pointer to the next word, and repeats. This is a linear timeline. If a database has to scan millions of rows of unstructured text logs this way, it bottlenecks the CPU by running millions of repetitive, sub-loops.

The Feynman Methodology (The Infinite Stopwatch Maze)

Richard Feynman’s quantum postulate changes the entire architecture of the search. He argued that tiny particles don’t choose a single path from A to B — they travel down every single possible path at the exact same moment.

This T-SQL framework forces unstructured text strings to act exactly like Feynman’s quantum particles. Instead of looping through a sentence, a Tally Table Matrix acts like a beam splitter. The second a string enters the query engine, it is instantly shattered into all possible character positions simultaneously.

Every word location becomes an independent path running in parallel:

The Noise (Destructive Interference): In quantum mechanics, paths have “stopwatches” with spinning hands. If hands point in opposite directions, they cancel out. In this code, common stop-words (like “and” or “uses”) have their weights calculated against the entire document collection via Inverse Document Frequency (IDF). Because they appear everywhere, their mathematical stopwatch hands point in opposite directions, creating destructive interference that completely zeros out their linguistic noise.
The Signal (Constructive Interference): Unique, definitive terms (like “quantum” or “database”) align perfectly. Their stopwatches point in the exact same direction, creating constructive interference. This piles up mathematical weight into specific vector pathways, letting the most important terms bubble to the top.

2. Moving Beyond the “Bag of Words”: Geometry & Tied Dice

A standard “Bag-of-Words” text tool is completely blind to context. It throws all words into an unstructured pile, losing track of word positions.

By using Feynman’s spatial logic, words are not just counted; but are tracked along the physical geometry of their paths. Because the engine maps the spatial position of every token at the exact same moment, it can run instantaneous relational self-joins to build multi-word chains (N-Grams and phrases) without losing the structural flow of the text.

We can then leverage the quantum concept of Entanglement (the “Tied Dice” principle) to handle document relationships. In standard database queries, comparing every document to every other document requires running thousands of isolated searches one after the other, or using a graph database.

This engine runs a global matrix cross-join. It acts like pairs of magically tied dice: altering or evaluating a vector coordinate on one side of the matrix instantly forces the corresponding relationship score to resolve on the other side. This creates a true, multi-dimensional tensor contraction natively inside the relational engine, mapping the overlapping similarities of the entire corpus in a single pass.

3. The Complete, End-to-End Local Framework

The script below combines our entire pipeline. It initialises the unstructured text logs and stop-word filters, tokenises the text via parallel character paths, maps them into a 10-dimensional TF-IDF space, uses a matrix projection to compress those down to a 2D canvas, runs an automated K-Means distance clustering routine, and displays a Convex Hull Bounding Polygon directly in SQL Server’s Management Studio Query Analyser Spatial canvas.

-- =========================================================================
-- 1. ENVIRONMENT INITIALIZATION
-- =========================================================================
IF OBJECT_ID('dbo.DocumentSpatialPlot', 'U') IS NOT NULL DROP TABLE dbo.DocumentSpatialPlot;
IF OBJECT_ID('dbo.DocumentCoordinates3D', 'U') IS NOT NULL DROP TABLE dbo.DocumentCoordinates3D;
IF OBJECT_ID('dbo.DocumentCorpus', 'U') IS NOT NULL DROP TABLE dbo.DocumentCorpus;
IF OBJECT_ID('dbo.StopWords', 'U') IS NOT NULL DROP TABLE dbo.StopWords;

CREATE TABLE dbo.DocumentCorpus (
    DocID INT IDENTITY(1,1) PRIMARY KEY, DocTitle VARCHAR(50), DocText VARCHAR(MAX)
);
CREATE TABLE dbo.StopWords (Word VARCHAR(50) PRIMARY KEY);

CREATE TABLE dbo.DocumentCoordinates3D (
    DocTitle VARCHAR(50) PRIMARY KEY, Coord_X FLOAT NOT NULL, Coord_Y FLOAT NOT NULL, Coord_Z FLOAT NOT NULL, ClusterID INT NULL
);
CREATE TABLE dbo.DocumentSpatialPlot (
    DocTitle VARCHAR(50) PRIMARY KEY, ClusterID INT, ClusterLabel VARCHAR(50), SpatialPoint GEOMETRY NOT NULL 
);

-- Seed Data & Noise Filter Matrix
INSERT INTO dbo.DocumentCorpus (DocTitle, DocText) VALUES
('Quantum Physics', 'quantum physics uses quantum mechanics and quantum states'),
('Database Systems', 'relational database systems use database tables and indexing'),
('Hybrid Engines', 'quantum computers and database systems create hybrid computing engines'),
('Hardware Clusters', 'superconducting circuits cluster hardware networks and processing arrays'),
('Network Security', 'secure networks use cryptographic keys inside database firewalls');

INSERT INTO dbo.StopWords (Word) VALUES ('and'),('use'),('uses'),('create'),('to'),('the'),('is'),('of'),('in'),('for');
GO

-- =========================================================================
-- 2. PARALLEL TOKENIZATION, TF-IDF, AND DIMENSION REDUCTION
-- =========================================================================
WITH Tally AS (
    SELECT TOP 2000 ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS N FROM sys.all_columns
),
RawTokens AS (
    -- FEYNMAN PATHWAY SPLIT: Shatters all character paths at the exact same moment
    SELECT c.DocID, c.DocTitle,
           LOWER(SUBSTRING(' ' + c.DocText + ' ', t.N + 1, CASE WHEN CHARINDEX(' ', ' ' + c.DocText + ' ', t.N + 1) - t.N - 1 > 0 THEN CHARINDEX(' ', ' ' + c.DocText + ' ', t.N + 1) - t.N - 1 ELSE 1 END)) AS CleanWord
    FROM dbo.DocumentCorpus c JOIN Tally t ON t.N <= LEN(' ' + c.DocText + ' ') - 1
    WHERE SUBSTRING(' ' + c.DocText + ' ', t.N, 1) = ' ' AND CHARINDEX(' ', ' ' + c.DocText + ' ', t.N + 1) - t.N - 1 > 0
),
FilteredTokens AS (
    SELECT rt.* FROM RawTokens rt LEFT JOIN dbo.StopWords sw ON rt.CleanWord = sw.Word WHERE sw.Word IS NULL
),
TermFrequency AS (
    SELECT DocID, DocTitle, CleanWord, CAST(COUNT(*) AS FLOAT) / SUM(COUNT(*)) OVER(PARTITION BY DocID) AS TF
    FROM FilteredTokens GROUP BY DocID, DocTitle, CleanWord
),
InverseDocFrequency AS (
    -- COHERENCE EVALUATION: Overlapping words create destructive interference, reducing their weight
    SELECT CleanWord, LOG(CAST((SELECT COUNT(*) FROM dbo.DocumentCorpus) AS FLOAT) / CAST(COUNT(DISTINCT DocID) AS FLOAT)) AS IDF
    FROM TermFrequency GROUP BY CleanWord
),
HighDimVectors AS (
    SELECT tf.DocTitle, tf.CleanWord, (tf.TF * idf.IDF) AS TF_IDF_Weight
    FROM TermFrequency tf JOIN InverseDocFrequency idf ON tf.CleanWord = idf.CleanWord
),
ProjectionWeights (WordToken, Comp_X, Comp_Y, Comp_Z) AS (
    -- 10-Dimensional word matrix compression definitions
    SELECT 'quantum', 0.85, 0.10, -0.20 UNION ALL SELECT 'database', 0.05, 0.90, 0.15 UNION ALL
    SELECT 'engines', 0.40, 0.40, 0.70 UNION ALL SELECT 'physics', 0.75, -0.05, -0.35 UNION ALL
    SELECT 'mechanics', 0.80, 0.00, -0.25 UNION ALL SELECT 'relational', -0.10, 0.85, 0.20 UNION ALL
    SELECT 'indexing', 0.00, 0.70, 0.40 UNION ALL SELECT 'networks', -0.30, 0.30, 0.80 UNION ALL
    SELECT 'hardware', -0.40, 0.15, 0.75 UNION ALL SELECT 'cryptographic', -0.50, 0.60, 0.50
)
INSERT INTO dbo.DocumentCoordinates3D (DocTitle, Coord_X, Coord_Y, Coord_Z)
SELECT v.DocTitle, SUM(v.TF_IDF_Weight * p.Comp_X), SUM(v.TF_IDF_Weight * p.Comp_Y), SUM(v.TF_IDF_Weight * p.Comp_Z)
FROM HighDimVectors v JOIN ProjectionWeights p ON v.CleanWord = p.WordToken
GROUP BY v.DocTitle;
GO

-- =========================================================================
-- 3. NATIVE K-MEANS DISTANCE CLUSTERING
-- =========================================================================
DECLARE @Centroids TABLE (ClusterID INT PRIMARY KEY, X FLOAT, Y FLOAT, Z FLOAT);
INSERT INTO @Centroids (ClusterID, X, Y, Z) VALUES (1, 0.5, 0.0, -0.2), (2, 0.0, 0.5, 0.5);

DECLARE @LoopCounter INT = 1;
WHILE @LoopCounter <= 5
BEGIN
    -- Measure spatial distance using Euclidean math
    WITH DistanceCalculation AS (
        SELECT d.DocTitle, c.ClusterID,
            SQRT(SQUARE(d.Coord_X - c.X) + SQUARE(d.Coord_Y - c.Y) + SQUARE(d.Coord_Z - c.Z)) AS Distance,
            ROW_NUMBER() OVER(PARTITION BY d.DocTitle ORDER BY SQRT(SQUARE(d.Coord_X - c.X) + SQUARE(d.Coord_Y - c.Y) + SQUARE(d.Coord_Z - c.Z)) ASC) AS ClosestRank
        FROM dbo.DocumentCoordinates3D d CROSS JOIN @Centroids c
    )
    UPDATE d SET d.ClusterID = dc.ClusterID FROM dbo.DocumentCoordinates3D d JOIN DistanceCalculation dc ON d.DocTitle = dc.DocTitle WHERE dc.ClosestRank = 1;

    DELETE FROM @Centroids;
    INSERT INTO @Centroids (ClusterID, X, Y, Z) SELECT ClusterID, AVG(Coord_X), AVG(Coord_Y), AVG(Coord_Z) FROM dbo.DocumentCoordinates3D GROUP BY ClusterID;
    SET @LoopCounter = @LoopCounter + 1;
END;
GO

-- =========================================================================
-- 4. SPATIAL GEOMETRY CONVERSION & MULTI-LAYER MAP PLOT
-- =========================================================================
INSERT INTO dbo.DocumentSpatialPlot (DocTitle, ClusterID, ClusterLabel, SpatialPoint)
SELECT DocTitle, ClusterID, CASE WHEN ClusterID = 1 THEN 'Quantum Science' WHEN ClusterID = 2 THEN 'Data & Network' END,
    -- Feature Scaling (*100) forces the SSMS Map View canvas to automatically zoom and scale perfectly
    geometry::STPointFromText('POINT(' + CAST(ROUND(Coord_X * 100.0, 4) AS VARCHAR(20)) + ' ' + CAST(ROUND(Coord_Y * 100.0, 4) AS VARCHAR(20)) + ')', 0)
FROM dbo.DocumentCoordinates3D;
GO

-- THE FINAL GRAPHICAL MAP VIEW GENERATION
WITH ScaledPoints AS (
    SELECT ClusterLabel, geometry::STPointFromText('POINT(' + CAST(ROUND(Coord_X * 100.0, 4) AS VARCHAR(20)) + ' ' + CAST(ROUND(Coord_Y * 100.0, 4) AS VARCHAR(20)) + ')', 0) AS PointShape
    FROM dbo.DocumentCoordinates3D WHERE ClusterID = 2
),
ClusterBoundary AS (
    -- Convex Hull aggregates the separate point nails into a tight, solid background polygon sheet
    SELECT 'Cluster 2 Territory' AS [Label], 'Data & Network' AS [Group], geometry::UnionAggregate(PointShape).STConvexHull() AS [SpatialShape] FROM ScaledPoints
)
SELECT [Label], [Group], [SpatialShape] FROM ClusterBoundary
UNION ALL
SELECT DocTitle AS [Label], ClusterLabel AS [Group], SpatialPoint.STBuffer(3) AS [SpatialShape] FROM dbo.DocumentSpatialPlot;
GO

4. Visualizing the Final Measurement Map

When you run this script inside SQL Server Management Studio (SSMS), look down at the query results window and select the Spatial Results tab next to the grid text view. Set the drop down to map the **Label** column on the screen.

If you use your own text data you may encounter a problem which renders the Spatial view as completely blank. This can be resolved by multiplying everything by 100 which gives the Spatial Browser enough data to work with to allow it to display.

The Spatial Query output

The Spatial Browser tab in SSMS has sadly been neglected but it does work

The Wave Function Collapse (The Shaded Polygon): The STConvexHull() function generates a shaded polygon area around the network infrastructure points. This polygon marks out the exact contextual territory of the data systems data.
The Bridge Point: The Hybrid Engines node sits right on the boundary line of the shape. Because it contains both quantum and database paths, its vector coordinate sits in the buffer zone between the two clusters.
The Outlier: The Quantum Physics bubble floats completely outside the shaded shape on the other side of the screen canvas, proving that its vocabulary path belongs to a completely separate linguistic group.

By applying Feynman’s “all paths” perspective to database logic, we replace slow text loops with high-velocity geometric calculations — mapping raw strings into physical, visual coordinates on our screens.

A way to use this on non textual data that is equally useful could be with Application Logs. Take the following as an example:

CREATE TABLE dbo.ApplicationLogs (
    LogID INT IDENTITY(1,1) PRIMARY KEY,
    LogCode VARCHAR(20) NOT NULL,
    Description VARCHAR(100) NOT NULL
);

INSERT INTO dbo.ApplicationLogs (LogCode, Description) VALUES
('ERR_404', 'Page Not Found'),
('ERR_500', 'Internal Server Error'),
('ERR_403', 'Access Denied Forbidden'),
('SYS_INIT', 'System Initialization Success'),
('DB_CONN', 'Database Connection Established'),
('AUTH_FAIL', 'User Authentication Failed'),
('CRIT_OOM', 'Critical Out of Memory Warning'),
('NET_TIMEOUT', 'Network Request Timed Out'),
('DISK_FULL', 'Storage Space Exhausted'),
('API_FAIL', 'External API Gateway Unresponsive'),
('SEC_BREACH', 'Unusual Login Location Detected'),
('CACHE_MIS', 'Redis Cache Miss Occurred'),
('QUE_OVER', 'Message Queue Overflowing'),
('DATA_CORR', 'Data Corruption Detected in Payload'),
('FILE_MISS', 'Configuration File Missing'),
('PROC_DEAD', 'Worker Process Terminated Unexpectedly');
GO

IF OBJECT_ID('dbo.LocalQuantumRegister', 'U') IS NOT NULL
    DROP TABLE dbo.LocalQuantumRegister;

CREATE TABLE dbo.LocalQuantumRegister (
    PathID INT PRIMARY KEY,
    BinaryString VARCHAR(4) NOT NULL,
    Amplitude FLOAT NOT NULL,
    Probability AS (SQUARE(Amplitude)) PERSISTED
);
GO

We then create the stored procedure:

CREATE OR ALTER PROCEDURE dbo.usp_QuantumSearchMultiTarget
    @SearchPattern VARCHAR(20)
AS
BEGIN
    SET NOCOUNT ON;

    -- 1. SETUP DATA MAPPING AND DYNAMIC MATRIX MATH
    DECLARE @TotalPaths FLOAT = 16.0;
    DECLARE @InitialAmplitude FLOAT = 1.0 / SQRT(@TotalPaths);
    DECLARE @MatchCount INT = 0;

    -- Local table variable to bind textual identifiers to numeric PathIDs
    DECLARE @DataMap TABLE (
        PathID INT IDENTITY(0,1),
        OriginalLogID INT,
        LogCode VARCHAR(20),
        IsTarget BIT DEFAULT 0
    );

    INSERT INTO @DataMap (OriginalLogID, LogCode)
    SELECT LogID, LogCode FROM dbo.ApplicationLogs;

    -- Flag all data entries matching the text parameter
    UPDATE @DataMap
    SET IsTarget = 1
    WHERE LogCode LIKE @SearchPattern;

    SELECT @MatchCount = COUNT(*) FROM @DataMap WHERE IsTarget = 1;

    IF @MatchCount = 0
    BEGIN
        RAISERROR('Error: Zero database rows matched the specified criteria. Simulation halted.', 16, 1);
        RETURN;
    END;

    -- Grover Math for Multi-Targets: Floor( (Pi / 4) * Sqrt(N / M) )
    DECLARE @OptimalIterations INT = FLOOR((PI() / 4.0) * SQRT(@TotalPaths / CAST(@MatchCount AS FLOAT)));
    IF @OptimalIterations = 0 SET @OptimalIterations = 1; 

    -- 2. INITIALIZE UNIFORM SUPERPOSITION ("ALL PATHS")
    TRUNCATE TABLE dbo.LocalQuantumRegister;

    WITH BinaryGeneration AS (
        SELECT 0 AS ID, CAST('0000' AS VARCHAR(4)) AS BinStr
        UNION ALL
        SELECT ID + 1, CAST(RIGHT('0000' + 
            CAST((ID + 1)/8 AS VARCHAR(1)) + 
            CAST(((ID + 1)%8)/4 AS VARCHAR(1)) + 
            CAST((((ID + 1)%8)%4)/2 AS VARCHAR(1)) + 
            CAST((((ID + 1)%8)%4)%2 AS VARCHAR(1)), 4) AS VARCHAR(4))
        FROM BinaryGeneration WHERE ID < 15
    )
    INSERT INTO dbo.LocalQuantumRegister (PathID, BinaryString, Amplitude)
    SELECT ID, BinStr, @InitialAmplitude FROM BinaryGeneration;

    -- 3. CORE WAVE INTERFERENCE LOOP
    DECLARE @CurrentLoop INT = 1;
    DECLARE @AvgAmplitude FLOAT;

    WHILE @CurrentLoop <= @OptimalIterations
    BEGIN
        -- Phase Inversion (The Oracle): Flip the amplitude sign for ALL matching targets
        UPDATE q
        SET q.Amplitude = -q.Amplitude
        FROM dbo.LocalQuantumRegister q
        JOIN @DataMap m ON q.PathID = m.PathID
        WHERE m.IsTarget = 1;

        -- Amplitude Amplification (The Diffusion Step): Invert around the global average
        SELECT @AvgAmplitude = AVG(Amplitude) FROM dbo.LocalQuantumRegister;
        
        UPDATE dbo.LocalQuantumRegister 
        SET Amplitude = (2.0 * @AvgAmplitude) - Amplitude;

        SET @CurrentLoop = @CurrentLoop + 1;
    END;

    -- 4. INTERSECT QUANTUM STATES WITH REAL-WORLD LOG ROWS
    SELECT 
        m.PathID AS [Quantum_Path_Index],
        q.BinaryString AS [Quantum_Address],
        CAST(ROUND(q.Probability * 100, 2) AS VARCHAR(10)) + '%' AS [Match_Confidence],
        l.LogCode AS [Discovered_Key],
        l.Description AS [Raw_Database_Value],
        @OptimalIterations AS [Dynamic_Iterations_Executed]
    FROM dbo.LocalQuantumRegister q
    JOIN @DataMap m ON q.PathID = m.PathID
    JOIN dbo.ApplicationLogs l ON m.OriginalLogID = l.LogID
    ORDER BY q.Probability DESC;
END;
GO

and execute with different parameters each time. This is on an heap object.

SET STATISTICS IO ON
GO
EXEC dbo.usp_QuantumSearchMultiTarget @SearchPattern = 'ERR%';
GO
EXEC dbo.usp_QuantumSearchMultiTarget @SearchPattern = 'DISK%';
GO
EXEC dbo.usp_QuantumSearchMultiTarget @SearchPattern = 'NET%';
GO
SET STATISTICS IO OFF

The output for the Quantum Search Query

I want to scale this method to a huge amount of data but I also wish I wasn’t doing quantum calculations on a non quantum computer so if anyone at NVIDIA or Microsoft is feeling generous… :)

That’s all for today, and its been exhausting, so I’m off just like Feynman would be to do more research sadly not at the local Lap Dancing club.

Relational Space Carving in SQL Server using My Head

Sun, 23 Aug 2026 14:28:14 GMT

Here is the complete documentation of how you can turn SQL Server into a crude 3D photogrammetry engine, taking a flat image, mathematically carving it in a database, and outputting a watertight mesh ready for the 3D printer.

Project: Relational Space Carving in SQL Server

The Goal: Generate a 3D point cloud (.obj) of a human head natively within SQL Server using a 4-panel orthographic reference image, and process that raw data into a physical mesh.

Step 1: The 2D Proof of Concept

Before calculating Z-depth, the database needed to prove it could accurately shred a binary file and map spatial coordinates. We loaded an uncompressed 24-bit/32-bit BMP into memory using OPENROWSET, shredded the binary string using an inline Tally Table to avoid WHILE loops, and mapped the RGB luminance values to an ASCII gradient.

A 2D rendering of my Head, Obviously Front View

By inverting the gradient and compensating for the byte-stride of a 32-bit image, the database successfully rendered a 2D heightmap directly in the SSMS query results window.

The inverted 2D ASCII render perfectly* mapped in SSMS text results exported to Notepad

When I say perfectly rendered my head, I mean I nearly lost it and abandoned the entire project at this point. The first time I ran it I got a very lopsided view of my neck. When I realised I had to zoom out quite a bit I did sort of see my head.
I then inverted the image and then saw my head but still lopsided.

Eventually I realised I was looking at a 32-bit image; a 24-bit image looks at RED, GREEN and BLUE at 8-bits each. A 32-bit image looks at RED, GREEN and BLUE and an 8-bit alpha channel for transparency ! That is when it all appeared.

I then did the same and rendered this image which is the input into the next section.

The Four Renders (Front, Left. Right and Back)

I would smile more but if you ever want to animate your head it is important to maintain a “no expression” look, also I have a huge scar on the back of my actual head and because it is really tough taking a picture of the back of your head this is a stunt rear head

Step 2: The 3D Space Carving Architecture

To generate 3D geometry, we utilized a technique called Visual Hull Intersection (Space Carving).

The query reads the top-left quadrant of the image as the Front Profile (X, Y) and the top-right quadrant as the Side Profile (Z, Y). By setting a luminance threshold of 150, we mathematically erased the grey studio background. I had issues at this point with my maths and ended up just resorting to using the top two views.

The core engine is a strict INNER JOIN on the Y (height) axis. If a voxel is solid from the front view and solid from the side view on the exact same row, it survives. If not, it is carved away.

Early tests using a massive @Skip = 8 downsampling variable resulted in severe mathematical artifacts, including the “Cartesian explosion” when the background threshold was too high, and heavy horizontal terracing when the gaps between the evaluated rows were too wide.

The Cartesian explosion showing Moiré interference patterns

To eliminate the stepping artifacts, we dropped the downsampling to a 1:1 pixel resolution (@Skip = 1).

Because evaluating millions of pixels via CTEs would cause the query optimizer to spool endlessly, the architecture was shifted to physical temporary tables. By inserting the front and side coordinates into #TempFront and #TempSide, and applying a Clustered Index to the Y axis on both, we forced SQL Server to execute a lightning-fast Merge Join.

The Final SQL Script

Step 4: Skinning the Data in Blender

Because SQL only writes vertices (points in space), the raw import into Blender appears as a solid black shadow. The geometry has no faces, so light cannot react to it.

To convert this raw data into a manifold mesh, we bypass manual topology and use Geometry Nodes:

Assign a Points to Volume node to the geometry, tuning the Radius up until the individual point spheres bridge the gaps between the coordinates.
Pipe that volume directly into a Volume to Mesh node.

When this is executed on the 1:1 high-resolution SQL export, the severe terracing disappears, resulting in an incredibly dense, solid silhouette.

The high-resolution 1:1 mesh output from the Geometry Nodes.

You can almost see the head now turned, so using both the front view and the side view. The Moiré interference patterns were still showing and this is because when doing the combining of the views, SQL is using a pure join and if it does not match perfectly then the point cancels itself out. What I needed was a fuzzy join which looked for points near the actual point.

I also forgot about Blender and the way it imports .obj files. So, for the first few attempts I was getting a head 6000 metres tall and I just thought Blender had failed to import.

When I realised, I carried on with the post-processing.

Step 5: Post-Processing and Print Prep

The final step is knocking down the micro-grid “waffle” texture left behind by the spherical voxel generation.

By applying the Geometry Nodes modifier and passing the mesh through a Decimate modifier (to reduce the polygon count) followed by a Smooth modifier (to average the vertex laplacians), the head becomes perfectly clean.

It is now a watertight, production ready .stl or .obj, ready to be sent across to the eight 3D printers and brought into the physical world. I already have the eyes to put into the head but that’s another article.

This is almost printable but rotating it shows why you should not leave a gap of more than 30 degrees between each image. They must overlap otherwise how can the algorithm now how to match each pixel to the next image. Photogrammetry tools do this well, I however am using SQL Server and have not. I could either increase the number of photos required for the image matching or I could use a photogrammetry tool. Guess which I choose 😊

Next time I put my robot eyes into the 3d printed head and begin the cloning process ;)

I am sort of joking…

Hope you enjoyed the failure and the article. For some unknown reason it would not let me paste the code into the article so screenshots for now. Code when I have it up on github.

Trevor (@sqlartist)

Vellum Lasts 500 Years. My 8,000 DVDs Didn’t.

Wed, 19 Aug 2026 11:52:04 GMT

Vellum and Acts of Parliament

*I have around 80 terabytes accumulated over thirty years. I have managed to index roughly a fifth of it. Keeping it all was the easy part.*

A corridor rendered from a filesystem

Ask someone for a cassette player because you need to read an important document you stored on one, and watch their face.

I had eight thousand DVDs. At the time that felt like permanence — physical, indexed, sitting on shelves. Then Blu-ray, then nothing, and now a stack of discs I have no confident way to read and no realistic way to search. Before that it was tape. Before that, floppies I dutifully copied forward for years until I stopped.

Meanwhile there are documents in British archives written on vellum in the eleventh century that you can walk up to and read today. No reader required. No format migration. No driver. The Domesday Book is nearly a thousand years old and its retrieval technology is *eyes*.

That contrast has bothered me for a long time, and I think most people take the wrong lesson from it. The usual conclusion is that we need better media — longer-lived discs, archival film, quartz glass. And that’s a real field with real progress in it. But it isn’t the thing that has actually cost me.

What’s actually cost me is that I *have* the data and cannot see it.

## Thirty years of keeping everything

I’ve saved almost everything since the mid-nineties. I’ve also lost nearly all of it several times over — drive failures, migrations, one memorable act of self-sabotage. Each time I had backups. Each time the backups turned out to be less restorable than I’d assumed. What survived, survived by accident and duplication rather than by strategy.

The current state of that thirty-year sediment is roughly **80 TB** — two main machines plus a shelf of external drives holding whatever was current when I last migrated. I have no complete inventory of it. I’m not sure a complete inventory is even possible without a month of downtime.

What I *have* indexed is the creative-work portion: the drives holding my Unreal Engine, Unity and Blender projects.

- **11,217,469 files**

- **14.78 TB** across four drives

- **2,057,943 directories**

That’s the well-behaved fifth. The part I actually curate, actively use, and would have said I knew my way around. The other sixty-odd terabytes are worse in every respect.

And that’s the uncomfortable bit. This isn’t a landfill I’d neglected. This is the good stuff.

And here’s the thing that finally pushed me into building something: I have no way to look at it. Not “no convenient way” — no way at all. My actual tool for understanding my own storage is [WizTree](https://diskanalyzer.com/), which is excellent at what it does and what it does is show me a treemap of *sizes*. It tells me that 400 GB is sitting in a folder. It cannot tell me what any of it is.

Explorer is worse. Search indexes filenames. Nothing tells me *this is a car model, and so are these six hundred others, and four of them are the same car*.

The information is preserved. It is completely unnavigable. As far as actually using it goes, that’s the same as having lost it — just more expensive.

## Preservation is a solved-ish problem. Comprehension isn’t.

There’s a whole discipline of digital preservation, and its practitioners worked this out decades ago. The threat to a digital archive isn’t usually the medium rotting. It’s that the *reader* disappears — the drive, the codec, the application, the person who knew what the folder structure meant. Media obsolescence outpaces media decay by a wide margin. The standard defence isn’t a magic disc, it’s boring and effective: open formats, many copies, and migrating everything forward on a schedule, forever.

But even that only guarantees the bytes survive. It says nothing about whether you can *find* anything, or whether you’d recognise it if you did. A perfectly preserved archive that nobody can navigate is a very well-maintained landfill.

That’s the problem I actually wanted to solve. Not “will this survive?” but “given that it survived, how do I stand in it and look around?”

## Two half-finished projects

I’d been circling this from two directions without noticing they were the same direction.

**The first** was a Blender add-on. I wanted to browse my 3D work — Unreal Engine, Unity and Blender projects — the way you’d browse an OLAP cube: pivot it, slice it, drill down. So I wrote a fast filesystem scanner that catalogues everything into a local database, detects project roots by their marker files, and derives dimensions from what it finds — drive, path hierarchy, file type, calendar.

Then I added the part that mattered: a **semantic layer**. Rather than only recording that a folder exists, it parses project names, paths and READMEs into *facets* — what a project depicts, what kind of asset it is, what genre it belongs to. Not “e:\\_Vehicles\\Vehicle_Red_Mustang” but *vehicle, car, racing*. Geography and style are modelled as “depicts” facets rather than literal locations, so a desert canyon asset lives under `Natural > Arid > Desert` regardless of which folder it happens to sit in.

That scan produced 4,546 detected projects out of the 11.2 million scanned files, with around 11,000 facet assertions describing them.

**The second** was, I’ll admit, a stunt. I’d been building increasingly unreasonable things inside SQL Server — a ray tracer, a Turing machine, eventually a playable renderer — mostly to find out how far T-SQL could be pushed as an expressive medium rather than a query language.

The stunt gave me a renderer. The add-on gave me a semantically tagged map of my entire storage. It took me embarrassingly long to notice that the second thing was a *world description* and the first thing could *walk around in it*.

## Standing inside your own hard drive

So the two got welded together. The cube feeds a compiler that turns projects into architecture, and SQL Server renders it in first person.

The mapping is deliberately literal, and every surface is a fact:

What each colour and object represents

| you see                   | it means                                    |
|                           |                                             |
| room size                 | bytes on disk                               |
| ceiling height            | file count, log scaled                      |
| wall colour               | which semantic district you're standing in  |
| darkness                  | nothing has touched this in a long time     |
| amber doorframe           | a doorway into an adjacent project          |
| steel pillars             | the tags that classify this project         |
| a pit instead of a floor  | untagged - the classifier couldn't reach it |
| floor plate underfoot     | an asset this project shares with another   |

The result for my corpus: **4,546 rooms in 70 districts**, 75,595 wall segments, 6,783 doorways, and 3,707 floor plates, laid out across a world just under 15,000 units across.

Districts are the part that turns it from a visualisation into a place. Group by semantic facet and you get a vehicle quarter with several hundred rooms in it, a cyberpunk quarter, a large dark region of things nothing has opened in two years. You don’t query for those. You walk round a corner and there they are.

Crossing into a different district

## It’s also a text adventure

A rendered view can’t tell you a row count, so every frame ships prose alongside the picture:

- [vehicle].[UltimateCarRacingProject] 38,114 files 11.4 GB 4 tags 2,207 folders
exits: N [vehicle].[DestructibleVehicles5.7] SW [game].[LyraStarterGame]
* shares 107 files (11.4 GB) with [demo].[CitySample]

And you can talk to it. What you type compiles into a predicate, and the matching rows become beacons visible through walls, with a bearing and a distance:

/where is CitySample
/show me the biggest
/what's unused
/heaps

Those beacons are the thing that makes fifteen terabytes tractable. You ask a question, and the answer becomes a direction you can walk in.

The automap

## What it told me that a query wouldn’t have

Two things, and neither was what I built it to find.

**The duplication is worse than I thought, and more interesting.** The floor plates connect projects sharing substantial files. Standing in one project and seeing four plates underfoot — 107 files here, 88 there, 11 GB in one case — is a very different experience from a duplicate-file report. A report gives you a list to feel guilty about. The plates tell you *which of your projects are secretly the same project*, which is information about how you actually work.

**My classifier has holes I couldn’t see.** Ask for `/heaps` and every untagged project lights up. Most were exactly what you’d expect: `_zips2`, `_MegaDownloads`, folders called `Documents2` — genuine landfill, correctly unclassified. But scattered among them were real projects with perfectly descriptive names the tokeniser simply wasn’t catching. I went in to look at the graphics and came out with a to-do list for the tagging vocabulary.

That’s the argument for the whole approach, really. Both of those were sitting in the database the entire time. I’d never have thought to ask for either.

## This isn’t only for filesystems

One file in the system knows where the data came from. Everything after it — layout, rendering, navigation, the query language — works from a normalised table of *rooms* and *links between rooms*. Replace the harvester and nothing downstream notices.

I built the first version against **AdventureWorks2025**, because a known schema makes it obvious when the geometry is wrong: tables as rooms, foreign keys as floor plates, heaps as rooms with no floor. It works, and at 71 tables it’s honestly pointless — you’d read the diagram faster.

But the same engine would walk a purchase order system, where you’d step from an order into the customer who placed it and out into every other order they’ve made. Or a document archive. Or a photo library. Anything with entities and relationships between them has a floor plan hiding in it.

## Running it

Seven SQL scripts, a PowerShell client, two Python bridges.

| file              | what it does                                               |
|                   |                                                            |
| `01_schema.sql`   | tables, tunables, the intent grammar                       |
| `02_harvest.sql`  | database catalog harvester                                 |
| `03_layout.sql`   | the map compiler - partition, walls, doors, pillars, plates|
| `04_render.sql`   | raycaster, colour ramps, status line, automap              |
| `05_navigate.sql` | movement, collision, the command parser                    |
| `06_cube.sql`     | filesystem-cube staging and harvester                      |
| `07_perf.sql`     | spatial index for room lookups                             |
| `schemawalk.ps1`  | the client                                                 |
| `inspect_cube.py` |                                                            |
| `export_cube.py`  | the bridge from the scanner's database                     |

Install into any database — it doesn’t need to be the one you’re walking:

sqlcmd -S . -d SQLWORLD -C -i 01_schema.sql
sqlcmd -S . -d SQLWORLD -C -i 02_harvest.sql
sqlcmd -S . -d SQLWORLD -C -i 03_layout.sql
sqlcmd -S . -d SQLWORLD -C -i 04_render.sql
sqlcmd -S . -d SQLWORLD -C -i 05_navigate.sql
sqlcmd -S . -d SQLWORLD -C -i 07_perf.sql

To walk a database:

EXEC nav.usp_Build @source_db = 'AdventureWorks2025';
EXEC nav.usp_Verify;

To walk a filesystem, add the cube staging and bridge the scanner’s output across:

sqlcmd -S . -d SQLWORLD -C -i 06_cube.sql
python inspect_cube.py D:\_AssetOLAP\data\asset_cube_v4.db
python export_cube.py D:\_AssetOLAP\data\asset_cube_v4.db -o cube_load.sql
sqlcmd -S . -d SQLWORLD -C -i cube_load.sql -f 65001

EXEC nav.usp_BuildCube @district_by = 'facet';
EXEC nav.usp_Verify;

Then walk in:

.\schemawalk.ps1 -Server . -Database SQLWORLD

`w a s d` to move,
`q e` to turn,
`f` to follow the plate under your feet,
`m` for the map,
`/` to say something.

`@district_by` decides what the quarters of the city are
`facet` groups by what projects contain,
`engine` by Unreal versus Blender versus Unity,
`drive` by physical volume. Same data, three completely different cities.

It runs at 70 to 120 frames per second on the full 4,546-room world, with the entire pipeline — visibility, projection, shading, string assembly — inside SQL Server. The client only forwards keystrokes.

## Back to the vellum

I don’t think my problem was ever storage. I have the bytes. I have had the bytes, in some form, for thirty years, through several catastrophes and a lot of media formats that no longer exist.

What I didn’t have was any way to stand in the middle of it and understand what I was looking at. And I suspect that’s the real archival problem for most people now. We solved *keeping*. We did not solve *comprehending*, and the gap between the two widens every year that storage gets cheaper.

Vellum’s advantage was never really its longevity. It was that the reader and the medium were the same thing: you look at it, and you have it. Everything since has separated the two, and every layer we’ve added between the bytes and the understanding is another thing that can go obsolete.

I can’t do much about that. But I can make a fifth of it into somewhere I can walk, look around, and notice something. The other sixty terabytes are next.

— -

*Built with SQL Server, T-SQL, a filesystem scanner, and thirty years of not deleting anything.*

Running Doom Inside SQL Server

Thu, 13 Aug 2026 15:47:46 GMT

Appendix: RUN.md

Running Doom in SQL Server

Fourteen steps, in this order. Each owns a disjoint set of objects, so any one of them can be re-run on its own afterwards without disturbing the others.

doom_schema.sql is NOT in the list: E1M1_freedoom.sql already embeds it. Running it separately would drop the map data.

# File                   Creates

1 E1M1_freedoom.sql      schema + E1M1 geometry + 158 textures

2 doom_render.sql        render_frame, sector_at, framebuffer

3 doom_present.sql       frame_bmp, le4, screen_pixel, render_and_time

4 doom_game.sql          player, input_state, line_collide, 
                         p_tick, game_loop

5 doom_sprites.sql       sprite tables + render_sprites

6 E1M1_sprites.sql       sprite frames and columns

6b doom_sprites_fast.sql replaces render_sprites, adds thing_sector

7 doom_client.sql        p_frame, p_input_set

8 doom_doors.sql         door, p_use, p_door_tick; 
                         takes over build_collision, 
                         move_blocked, p_frame

9 doom_sky.sql           render_sky; final p_frame 
                         (all later stages are OBJECT_ID-guarded, 
                         so nothing redefines it again)

10 E1M1_sprites.sql      reload it: now 444 frames, 
                         including the weapon HUD sprites

11 doom_weapons.sql      inventory, pickups, firing, render_psprite

12 doom_statusbar.sql    hud_patch, hud_column, render_hud

13 hud.sql               37 status bar patches (map-independent)

sqlcmd -S . -d doom -i E1M1_freedoom.sql
sqlcmd -S . -d doom -i doom_render.sql
sqlcmd -S . -d doom -i doom_present.sql
sqlcmd -S . -d doom -i doom_game.sql
sqlcmd -S . -d doom -i doom_sprites.sql
sqlcmd -S . -d doom -i E1M1_sprites.sql
sqlcmd -S . -d doom -i doom_sprites_fast.sql
sqlcmd -S . -d doom -i doom_client.sql
sqlcmd -S . -d doom -i doom_doors.sql
sqlcmd -S . -d doom -i doom_sky.sql
sqlcmd -S . -d doom -i E1M1_sprites.sql      # reload: weapon sprites added
sqlcmd -S . -d doom -i doom_weapons.sql
sqlcmd -S . -d doom -i doom_statusbar.sql
sqlcmd -S . -d doom -i hud.sql

doom_doors.sql must come after doom_game.sql and doom_client.sql: it redefines build_collision, move_blocked and p_frame.
Re-running either of those later reverts the world to static. After loading it, run EXEC dbo.build_collision; — line_collide now stores sector ids rather than cached opening heights, so doors can move without invalidating it.

After doom_sprites_fast.sql, run EXEC dbo.build_thing_sector; once — it precomputes which sector each thing stands in, which keeps a scalar UDF out of the per-frame query.

doom_sprites.sql creates the tables and DELETEs their contents, so E1M1_sprites.sql must follow it. p_frame calls render_sprites only if it exists, so the game still runs without either.

Play

.\doom_client.ps1 -Server . -Database doom

W/S move,
A/D turn,
Q/E strafe,
Shift run,
SPACE open doors,
CTRL fire,
1–4 select weapon,
R respawn,
Esc quit.

-Tics 12 advances more simulation per displayed frame; -Scale 4 for a bigger window.

Healthcheck

EXEC dbo.doom_healthcheck;   -- optional but recommended
EXEC dbo.render_and_time;    -- run twice; the first pays a ~780ms compile
EXEC dbo.render_and_time;

Expect roughly: render_ms ~290, present_ms ~35, and pixels_lit close to 64,000 — with sky and the status bar in, the frame is fully covered.

pixels_lit = 0 means the texture tables are empty — the rasteriser INNER JOINs them, so it completes silently with no rows. Re-run E1M1_freedoom.sql.

Other maps

python wad2sql.py FREEDOOM1.WAD --list
python wad2sql.py FREEDOOM1.WAD E1M3 -o E1M3.sql
python gen_sprites.py FREEDOOM1.WAD E1M3 -o E1M3_sprites.sql

Sprite data is per-map: only the sprites that map’s things reference are extracted, plus the player’s weapons. The status bar graphics (hud.sql) are map-independent and do not need regenerating.

Needs wad2sql.py, wadtex.py, wadsprite.py, doom_schema.sql and doom_textures.sql in the same directory. Works on any vanilla-format IWAD or PWAD; extended (ZDBSP-compressed) nodes are detected and rejected.

Known gaps

No monster AI: things are drawn, collide and can be shot, but nothing moves or fights back.
Lifts, switches and walk-triggered lines: only manual (use) doors so far.
Locked doors refuse: there is no key inventory yet.
The status bar face is static; Doom’s reacts to damage and health.
Masked middle textures on two-sided lines (grates, vines) aren’t drawn, though their alpha masks are stored.
game_loop runs at ~12-15 tics/sec, not 35: WAITFOR DELAY rounds up to SQL Server’s ~15.6ms timer resolution. The client’s p_frame avoids this by batching tics per frame instead.

Tested against FreeDoom (BSD-licensed) so the whole thing can be shared without redistributing commercial game data. It works with a retail *DOOM.WAD* too — same parser, same schema.

Sourcecode is now available: https://github.com/sqlartist/DoomSQL

I Built a Language Model Entirely Inside SQL Server

Sun, 09 Aug 2026 16:34:10 GMT

For the last 30 years I have known a single language — SQL. I never did the whole university Computer Science degree, I was hands on using Microsoft Excel 5.0 and then when my data was too large for 16,384 rows I started my journey into Microsoft SQL Server 6.0. Has it been a hindrance not being able to make apps over the data I was storing, of course but it also made me think in sets and I always think of performance the moment I write , not as some afterthought when the system is dying.

Anyway this brings me to a little history and then the good stuff. Back in 1999 I was obsessed with unstructured data, and combining this with the structure in a SQL database had to be the holy grail. So I really got into Morphology, Semantics and Lexical analysis of unstructured data — think Word documents, Adobe PDFs or just plain HTML.

I wrote a whole suite of SQL code that did the parsing into key-phrases and keywords using N-Grams, Algorithm processing using Term Frequency-Inverse Document Frequency (TF-IDF) and eventual display into a multi-dimensional structure (OLAP) which itself was displayed in near real time on a Call centre monitor (HOLAP with Zero Aggregations).

Then I stopped for 27 years, until recently. I started hearing about LLMs and how they were changing the world, everyone out of a job because the code writes itself. Then I look into it and it’s a mess, releases every day and these strangely named models — YaLM 100B (Just like new stars, but there are so many of them they are just given a number, HD 95086) given the strangest of names and no governance or even checks as to the source of the data. For someone that works in data for this long, suddenly the world has gone crazy, how on earth can you trust this stuff? You don’t know its source and to make decisions based on a version of the truth that an LLM produced that was superseded 3 times in a month by another LLM from a company with bigger pockets, which itself was superseded by a country with bigger pockets than the company…

Which finally leads me onto why I am writing an article on using SQL to create, process, analyse and report on how an LLM works internally and you can see how the data moves between its networks and its final output. An interesting point is how many of my SQL friends around the world constantly amaze me with code that shouldn’t be written in SQL for whatever reason and they just nail it, sometimes even out performing the native language. Not bad for a language that is the same age as me.

The Joy Of Writing Code

Everyone knows a language model is matrix multiplication wearing a hat. Fewer people notice that matrix multiplication is just `GROUP BY` over a join, which means your database server has been quietly capable of this all along. So I built one: weights in tables, forward pass as a query, back propagation as a sequence of `UPDATE` statements. I trained it on a curated corpus of 122 idiomatic SQL queries, so the thing generates SQL.

A language model written in SQL, trained on SQL, that emits SQL.

**The Whole Model Is Five Lines of Algebra
**I went with a Bengio-style neural language model rather than a transformer. It’s the transformer’s direct ancestor, and it’s small enough that every tensor fits comfortably in a table without any of it feeling like a stunt.

A Bengio-style neural language model, introduced by Yoshua Bengio and colleagues in 2003, is a feed forward architecture that predicts the next word in a sequence using a shared word embedding table and a multi-layer perceptron, overcoming the data sparsity limits of traditional n-gram models.

My own work in this area was in 1999 with the TF-IDF algorithm and using this along with a semantic database called WordNet to create structure from unstructured data but stopped due to a complete lack of interest from anyone except a fellow SQL Server MVP Tony Rogerson.

But back to our main subject. Here’s the entire architecture. Context of C tokens, embedding lookup, concatenate, one dense layer with ReLU, project to the vocabulary, softmax:

slot k of C holds token t_k
x[k*D + d] = Emb[t_k, d] concatenated embeddings
z[h] = SUM_i x[i] * W1[i,h] + B1[h]
a[h] = MAX(z[h], 0) ReLU
logit[o] = SUM_h a[h] * W2[h,o] + B2[o]
p[o] = EXP(logit[o] — max) / SUM(…) softmax

Look at lines two and five. `SUM_i x[i] * W1[i,h]` is a sum of products, grouped by `h`. That’s not *like* a `GROUP BY` over a join. It **is** a `GROUP BY` over a join. Every line above becomes one CTE.

Defaults are context 3, embedding dimension 16, hidden 32, vocabulary around 213 tokens. Roughly 12,000 parameters. Small, but genuinely learned rather than counted.
Dimensions live in a config table rather than being hard-coded, so the model is re-sizable without editing a single function:

/* — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
 Hyperparameters. One row. Every function joins to this instead
 of hard-coding dimensions, so the model is resizable.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — */
DROP TABLE IF EXISTS dbo.ModelConfig;
CREATE TABLE dbo.ModelConfig
(
 lock_col char(1) NOT NULL CONSTRAINT PK_ModelConfig PRIMARY KEY
 CONSTRAINT CK_ModelConfig_OneRow CHECK (lock_col = ‘X’),
 C int NOT NULL, -- context length, in tokens
 D int NOT NULL, -- embedding dimension per token
 H int NOT NULL, -- hidden units
 V int NOT NULL -- vocabulary size
);
INSERT INTO dbo.ModelConfig (lock_col, C, D, H, V) VALUES (‘X’, 3, 16, 32, 0);

Tokenising SQL, In SQL
Before anything can be learned, the corpus has to be shredded into tokens. `COUNT(*)` needs to become three tokens, not one opaque blob, or the model never learns that parentheses have structure.

The naive approach is a chain of `REPLACE` calls padding every punctuation character with spaces. That works right up until it meets `<>`, which the single-character pass happily shreds into `<` and `>`. Two tokens where there should be one, and the model spends its capacity learning that `<` is usually followed by `>`.

The fix is to park the multi-character operators behind placeholders first, then restore them once the single-character pass is done:

/* — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
 Normalisation. Lower-cases and separates punctuation so that
 “COUNT(*)” becomes “count ( * )” — three tokens, not one blob.

Multi-character operators (<>, >=, <=, !=) are parked behind
 placeholders first, otherwise the single-character pass would
 shred them into two tokens.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — */
CREATE OR ALTER FUNCTION dbo.fn_Normalise (@txt nvarchar(max))
RETURNS nvarchar(max)
AS
BEGIN
 DECLARE @s nvarchar(max) = LOWER(@txt);
-- Park two-character operators.
 SET @s = REPLACE(REPLACE(REPLACE(REPLACE(@s,
 N’<>’, N’~1~’), N’>=’, N’~2~’), N’<=’, N’~3~’), N’!=’, N’~4~’);
-- Pad single-character punctuation with spaces.
 SET @s = REPLACE(@s, N’(‘, N’ ( ‘);
 SET @s = REPLACE(@s, N’)’, N’ ) ‘);
 SET @s = REPLACE(@s, N’,’, N’ , ‘);
 SET @s = REPLACE(@s, N’*’, N’ * ‘);
 SET @s = REPLACE(@s, N’=’, N’ = ‘);
 SET @s = REPLACE(@s, N’<’, N’ < ‘);
 SET @s = REPLACE(@s, N’>’, N’ > ‘);
 SET @s = REPLACE(@s, NCHAR(39), N’ ‘ + NCHAR(39) + N’ ‘); -- single quote
 --Restore the two-character operators, now safely spaced.
 SET @s = REPLACE(REPLACE(REPLACE(REPLACE(@s,
 N’~1~’, N’ <> ‘), N’~2~’, N’ >= ‘), N’~3~’, N’ <= ‘), N’~4~’, N’ != ‘);
WHILE CHARINDEX(N’  ‘, @s) > 0
 SET @s = REPLACE(@s, N’  ‘, N’ ‘);
RETURN LTRIM(RTRIM(@s));
END;

Splitting on spaces is the easy half, but position matters — a language model without token order is just a bag of words. `STRING_SPLIT` only gained an ordinal in SQL Server 2022, which rules it out if you want this running on anything older. `OPENJSON` has always had one:

/* — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
 Tokeniser. Splits on spaces and returns 1-based positions.
 OPENJSON is used rather than STRING_SPLIT because it yields an
 ordinal on every supported version, not just 2022+.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — */
CREATE OR ALTER FUNCTION dbo.fn_Tokenize (@txt nvarchar(max))
RETURNS TABLE
AS
RETURN
 SELECT pos = ROW_NUMBER() OVER (ORDER BY CAST(j.[key] AS int)),
 token = j.value
 FROM OPENJSON(N’[“‘ + REPLACE(dbo.fn_Normalise(@txt), N’ ‘, N’”,”’) + N’”]’) AS j
 WHERE j.value <> N’’;

Wrap the string in JSON array syntax, let `OPENJSON` do the splitting, and take `[key]` as the position. It’s a hack, but it’s a fast one and it works everywhere.

Randomness Is Harder Than It Looks

Weights start random. In SQL Server that’s a trap, because `RAND()` is evaluated once per statement, not once per row. Use it to initialise a weight table and you get several thousand identical values, a model with no symmetry to break, and a very confusing afternoon.

`CHECKSUM(NEWID())` is evaluated per row, which is what you actually want. There’s a second trap hiding inside it: `ABS()` overflows on the single value `-2³¹`. Take the modulus first — `%` preserves the sign, so you get your uniform distribution either side of zero without needing `ABS()` at all:

/* — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
 RAND() is constant for the whole statement, so it is useless for
 filling a table. CHECKSUM(NEWID()) is evaluated per row.
 He initialisation for the ReLU layers: scale = sqrt(2/fan_in).
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — */
INSERT INTO dbo.W2 (hid, out_id, w)
SELECT h.n, v.token_id,
 (CHECKSUM(NEWID()) % 1000001) / 1000000.0
 * SQRT(2.0 / @H)
FROM (SELECT n FROM dbo.Nums WHERE n < @H) AS h
CROSS JOIN dbo.Vocab AS v;

**The Forward Pass Is One Query
**Here’s the payoff. The entire inference path — embedding lookup, hidden layer, ReLU, output projection, softmax — as a single inline table-valued function. Five CTEs for five lines of algebra:

/* — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
 fn_NextToken: the complete forward pass in one statement.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — */
CREATE OR ALTER FUNCTION dbo.fn_NextToken (@txt nvarchar(max))
RETURNS TABLE
AS
RETURN
 WITH cfg AS (SELECT D FROM dbo.ModelConfig),
/* x : the concatenated embedding vector, length C*D */
 x AS (
 SELECT i = c.slot * cfg.D + e.dim,
 val = e.w
 FROM dbo.fn_ContextIds(@txt) AS c
 CROSS JOIN cfg
 JOIN dbo.Emb AS e ON e.token_id = c.token_id
 ),
/* z, a : hidden pre-activation and ReLU activation */
 a AS (
 SELECT s.hid,
 act = CASE WHEN s.z + b.w > 0 THEN s.z + b.w ELSE 0 END
 FROM (SELECT w1.hid, z = SUM(x.val * w1.w)
 FROM x JOIN dbo.W1 AS w1 ON w1.inp = x.i
 GROUP BY w1.hid) AS s
 JOIN dbo.B1 AS b ON b.hid = s.hid
 ),
/* logits : one per vocabulary entry. Driven from B2 with a LEFT JOIN
 so that a fully dead hidden layer still yields a complete row set. */
 lg AS (
 SELECT b.out_id,
 logit = b.w + ISNULL(s.s, 0)
 FROM dbo.B2 AS b
 LEFT JOIN (
 SELECT w2.out_id, s = SUM(a.act * w2.w)
 FROM a JOIN dbo.W2 AS w2 ON w2.hid = a.hid
 WHERE a.act > 0
 GROUP BY w2.out_id
 ) AS s ON s.out_id = b.out_id
 ),
/* softmax, with the usual max-subtraction for numerical safety */
 ex AS (
 SELECT out_id, logit,
 e = EXP(logit — MAX(logit) OVER ())
 FROM lg
 )
 SELECT token_id = ex.out_id,
 token = v.token,
 logit = ex.logit,
 p = ex.e / SUM(ex.e) OVER ()
 FROM ex
 JOIN dbo.Vocab AS v ON v.token_id = ex.out_id;

Two details in there are load-bearing.

The `LEFT JOIN` driving the logits from `B2` rather than from the hidden layer looks like belt and braces. It isn’t. Filter on `a.act > 0` and join inwards, and a context that happens to kill every hidden unit produces zero rows — a softmax over nothing. Driving from the bias table guarantees a complete distribution whatever the activations do.

The softmax is split across two scopes for a reason. The obvious one-liner is `SUM(EXP(x — MAX(x) OVER (…))) OVER (…)`, and SQL Server rejects it outright — a windowed function may not appear inside the argument of another windowed function. The max-subtraction and the normalising sum need separate CTE levels. I hit this in two places and it’s the sort of thing you only discover at `CREATE` time.

Back-propagation, And The Order That Matters

Training is where it stops being cute and starts being engineering. Here are the gradients, in the order they must be computed:

dLogit[s,o] = (p[s,o] — 1{o = target[s]}) / B
dW2[h,o] = SUM_s a[s,h] * dLogit[s,o]
dB2[o] = SUM_s dLogit[s,o]
dZ[s,h] = (SUM_o dLogit[s,o] * W2[h,o]) * 1{z[s,h] > 0}
dW1[i,h] = SUM_s x[s,i] * dZ[s,h]
dB1[h] = SUM_s dZ[s,h]
dX[s,i] = SUM_h dZ[s,h] * W1[i,h]
dEmb[t,d] += SUM over every (s,slot) where context token = t, of dX

Read line four and line seven again. `dZ` needs `W2` and `dX` needs `W1` — and both need the values from *before* this batch’s update. Write the weights first and you’ll compute the backward pass against parameters that have already moved.

The model still trains if you get this wrong. It just trains slowly, and silently wrongly, and you’ll spend a week blaming your learning rate. So both gradients are materialised into temp tables before either weight matrix is touched.

The update itself is the part I enjoyed most. SGD with momentum is two assignments — `v ← m*v + g`, then `w ← w — lr*v` — and T-SQL evaluates every right-hand side in an `UPDATE` against pre-update row values. Which means both assignments land in a single statement, with no staging table and no ordering worry:

/* — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
 SGD with momentum. Both assignments read pre-update values,
 which is exactly the semantics required.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — */
;WITH g AS (
 SELECT a.hid, d.out_id, gr = SUM(a.act * d.g)
 FROM #A AS a
 JOIN #DL AS d ON d.seq_id = a.seq_id
 GROUP BY a.hid, d.out_id
)
UPDATE t
SET v = @Momentum * t.v + g.gr,
 w = t.w — @curLr * (@Momentum * t.v + g.gr)
FROM dbo.W2 AS t
JOIN g ON g.hid = t.hid AND g.out_id = t.out_id;

The gradient for the embedding table is the fiddly one. `dX` comes back indexed by position in the concatenated input vector, so you have to unpick which context slot each index belonged to before you can scatter it onto the right token. Integer division and modulo do the job:

/* — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
 Scatter dX back onto the embedding rows it came from.
 i / D recovers the context slot; i % D recovers the dimension.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — */
;WITH g AS (
 SELECT c.token_id, dim = dx.i % @D, gr = SUM(dx.grad)
 FROM #DX AS dx
 JOIN #Ctx AS c ON c.seq_id = dx.seq_id AND c.slot = dx.i / @D
 GROUP BY c.token_id, dx.i % @D
)
UPDATE t
SET v = @Momentum * t.v + g.gr,
 w = t.w — @curLr * (@Momentum * t.v + g.gr)
FROM dbo.Emb AS t
JOIN g ON g.token_id = t.token_id AND g.dim = t.dim;

The Bug That Cost Me The Most Time

Not the calculus. A variable name.

I’d declared `@b` for the batch index and `@B` for the batch row count, because that’s how the maths is written. SQL Server compares variable names case-insensitively, regardless of database collation, so those are the same variable. The `DECLARE` fails, every variable declared alongside them becomes unbound, and you get a screen of errors ending in a baffling complaint about a CTE forty lines further down.

The same trap was sitting one line below: a local `@lr` colliding with the `@Lr` parameter. It hadn’t been reported yet only because the `DECLARE` never got that far.

Worth internalising, because the error count tells you nothing. One bad `DECLARE` unbinds everything in it. Fix the first message and re-run rather than working down the list.

Does It Actually Work?

It does. 122 published SQL queries, 60 epochs, roughly two billion row operations:

Model Training Performance

Perplexity of 2.9 over a 213-token vocabulary means that at each step it’s effectively choosing between about three candidates instead of 213. Prompt it and it produces recognisable SQL — `SELECT`, then a column list, then `FROM`, then a plausible table name.

EXEC dbo.usp_Generate @Prompt = N'SELECT', @Temperature = 0.8, @Samples = 5;
EXEC dbo.usp_Generate @Prompt = N'SELECT * FROM', @Greedy = 1;

Result of usp_Generate procedure

Two Things SQL Does Better Than Python

I expected this to be a novelty. Two parts genuinely came out nicer than the equivalent imperative implementation, and both come from the same property: the model returns a *relation*.

Because inference is a table-valued function, running a batch of prompts is `CROSS APPLY`. No loop, no padding, no batching logic — one tidy result set:

/* — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
 Many prompts at once. The model is a table-valued function, so
 CROSS APPLY runs a whole batch and the result is one relation.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — */
SELECT pr.prompt,
 rank_pos = ROW_NUMBER() OVER (PARTITION BY pr.prompt ORDER BY nt.p DESC),
 nt.token,
 probability = FORMAT(nt.p, ‘P1’)
FROM (VALUES
 (N’SELECT’),
 (N’SELECT * FROM’),
 (N’SELECT COUNT(*) FROM orders GROUP BY’),
 (N’INNER JOIN customers c ON’),
 (N’WITH ranked AS (‘),
 (N’UPDATE employees SET’)
 ) AS pr(prompt)
CROSS APPLY (
 SELECT TOP (4) token, p
 FROM dbo.fn_NextToken(pr.prompt)
 ORDER BY p DESC
) AS nt
ORDER BY pr.prompt, nt.p DESC;

prompt                         rank_pos             token                          probability
------------------------------ -------------------- ------------------------------ ------------------------------
INNER JOIN customers c ON      1                    o                              43.2%
INNER JOIN customers c ON      2                    p                              32.7%
INNER JOIN customers c ON      3                    c                              12.4%
INNER JOIN customers c ON      4                    country                        3.9%
SELECT                         1                    *                              30.8%
SELECT                         2                    customer_id                    9.4%
SELECT                         3                    o                              3.8%
SELECT                         4                    c                              3.6%
SELECT * FROM                  1                    customers                      29.2%
SELECT * FROM                  2                    products                       25.5%
SELECT * FROM                  3                    employees                      21.6%
SELECT * FROM                  4                    orders                         14.0%
SELECT COUNT(*) FROM orders GR 1                    customer_id                    55.3%
SELECT COUNT(*) FROM orders GR 2                    category_id                    10.2%
SELECT COUNT(*) FROM orders GR 3                    c                              6.9%
SELECT COUNT(*) FROM orders GR 4                    country                        5.1%
UPDATE employees SET           1                    salary                         47.9%
UPDATE employees SET           2                    department_id                  9.7%
UPDATE employees SET           3                    product_id                     4.2%
UPDATE employees SET           4                    '                              3.2%
WITH ranked AS (               1                    select                         84.3%
WITH ranked AS (               2                    1                              8.8%
WITH ranked AS (               3                    10                             1.2%
WITH ranked AS (               4                    50                             0.7%

The second one is better still. Constrained decoding — forcing generation to obey a grammar — is a genuinely awkward feature to bolt onto a normal inference stack. Here it’s a `WHERE` clause and a renormalising window function. `WHERE` is applied before window functions, so the arithmetic just works out:

/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  fn_Score runs the forward pass at every position at once and
  returns the probability the model gave to the token that
  actually appeared. High surprisal marks the spots where the
  query departs from what internet SQL usually looks like.
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
SELECT  position, token,
        probability    = FORMAT(p, 'P2'),
        surprisal_bits = ROUND(surprisal, 2)
FROM    dbo.fn_Score(N'SELECT customer_id, COUNT(*) FROM orders GROUP BY customer_id;')
ORDER BY position;

The interesting part is the shape: FROM after a column list will be near-certain, the table name after FROM much less so.

There’s a third benefit that isn’t about queries at all. Nothing is hidden in a binary blob. The trained model is rows, so it can be backed up, diffed, code-reviewed, version-controlled, and shipped by `INSERT`. Cosine similarity between token embeddings is a `GROUP BY` — and run it, and you’ll find table names cluster with table names and aggregates cluster with aggregates. Nobody told it those categories exist.

sum does return count and avg

Constrained Decoding

/*==============================================================================
  QUERY 9 -- Constrained decoding.

  Because the distribution arrives as a rowset, constraining generation to a
  grammar is a WHERE clause plus a renormalisation window. No special decoder
  is needed. Here: after FROM, only allow known table names.
==============================================================================*/
SELECT  token,
        p_raw        = FORMAT(p, 'P3'),
        p_constrained= FORMAT(p / SUM(p) OVER (), 'P1')
FROM    dbo.fn_NextToken(N'SELECT * FROM')
WHERE   token IN (N'customers', N'orders', N'products', N'employees',
                  N'payments', N'departments', N'categories')
ORDER BY p DESC;
GO

/*==============================================================================
  A SMALL LANGUAGE MODEL, ENTIRELY INSIDE SQL SERVER
  File 04 of 04 : Sampling procedure and example queries against the model

  Everything below treats the language model as what it now is: a set of
  tables, and a function that returns a probability distribution as a rowset.
  That means the usual toolkit -- TOP, WHERE, GROUP BY, CROSS APPLY, window
  functions -- applies to the model itself, not just to its training data.
==============================================================================*/

USE SqlLlm;
GO
SET NOCOUNT ON;
GO

/*==============================================================================
  SAMPLER

  Autoregressive decoding needs a loop, because each step depends on the
  previous one, and a recursive CTE may not call a table-valued function on
  its recursive member. So this is the one part of the system that is
  procedural rather than set-based.

  The context is recovered by re-tokenising the accumulated text each step.
  That is O(n) per token rather than O(1), but n is a few dozen tokens, and
  it keeps the state in one place.
==============================================================================*/
CREATE OR ALTER PROCEDURE dbo.usp_Generate
    @Prompt      nvarchar(max) = N'select',
    @MaxTokens   int   = 40,
    @Temperature float = 0.8,   -- < 1 sharpens, > 1 flattens
    @TopK        int   = 8,     -- sample only from the k most likely tokens
    @Greedy      bit   = 0,     -- 1 = always take the argmax (deterministic)
    @Samples     int   = 1
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @out TABLE (sample_no int, generated nvarchar(max), tokens int);
    DECLARE @s   int = 1,
            @txt nvarchar(max),
            @i   int,
            @tok nvarchar(100),
            @r   float;

    WHILE @s <= @Samples
    BEGIN
        SET @txt = dbo.fn_Normalise(@Prompt);
        SET @i   = 0;

        WHILE @i < @MaxTokens
        BEGIN
            SET @tok = NULL;

            IF @Greedy = 1 OR @Temperature < 0.05
            BEGIN
                SELECT TOP (1) @tok = token
                FROM   dbo.fn_NextToken(@txt)
                ORDER BY p DESC;
            END
            ELSE
            BEGIN
                SET @r = RAND(CHECKSUM(NEWID()));

                ;WITH topk AS (
                    SELECT TOP (@TopK) token, p
                    FROM   dbo.fn_NextToken(@txt)
                    ORDER BY p DESC
                ),
                tempered AS (
                    SELECT token, q = POWER(p, 1.0 / @Temperature) FROM topk
                ),
                normed AS (
                    SELECT token, q = q / NULLIF(SUM(q) OVER (), 0) FROM tempered
                ),
                cumulative AS (
                    SELECT token,
                           hi = SUM(q) OVER (ORDER BY q DESC ROWS UNBOUNDED PRECEDING)
                    FROM   normed
                )
                SELECT TOP (1) @tok = token
                FROM   cumulative
                WHERE  hi >= @r
                ORDER BY hi;

                -- floating-point tail guard
                IF @tok IS NULL
                    SELECT TOP (1) @tok = token FROM dbo.fn_NextToken(@txt) ORDER BY p DESC;
            END;

            IF @tok IS NULL OR @tok = N'</s>' BREAK;

            SET @txt = @txt + N' ' + @tok;
            SET @i += 1;
        END;

        INSERT INTO @out (sample_no, generated, tokens)
        VALUES (@s, dbo.fn_Detokenise(@txt), @i);

        SET @s += 1;
    END;

    SELECT sample_no, generated, tokens FROM @out ORDER BY sample_no;
END;
GO

/*==============================================================================
  QUERY 1 -- The model card.
  How big is this thing, and what did it read?
==============================================================================*/
SELECT  context_tokens = C,
        embed_dim      = D,
        hidden_units   = H,
        vocab_size     = V,
        total_parameters = V*D + (C*D)*H + H + H*V + V
FROM    dbo.ModelConfig;

SELECT  tensor = 'Emb (token embeddings)', parameters = COUNT(*) FROM dbo.Emb
UNION ALL SELECT 'W1 (input -> hidden)',  COUNT(*) FROM dbo.W1
UNION ALL SELECT 'B1',                    COUNT(*) FROM dbo.B1
UNION ALL SELECT 'W2 (hidden -> vocab)',  COUNT(*) FROM dbo.W2
UNION ALL SELECT 'B2',                    COUNT(*) FROM dbo.B2
ORDER BY parameters DESC;

SELECT  source_site, queries = COUNT(*)
FROM    dbo.Corpus
GROUP BY source_site
ORDER BY queries DESC;
GO

/*==============================================================================
  QUERY 2 -- Next-token distribution.
  The core inference call. Ask the model what comes after a prefix.
==============================================================================*/
SELECT TOP (10)
        token,
        probability = FORMAT(p, 'P2'),
        logit       = ROUND(logit, 3)
FROM    dbo.fn_NextToken(N'SELECT * FROM')
ORDER BY p DESC;

-- How confident is it? Shannon entropy of the same distribution, in bits.
SELECT  entropy_bits = -SUM(p * LOG(p) / LOG(2.0)),
        uniform_bits =  LOG((SELECT V FROM dbo.ModelConfig) * 1.0) / LOG(2.0)
FROM    dbo.fn_NextToken(N'SELECT * FROM');
GO

/*==============================================================================
  QUERY 3 -- Many prompts at once.

  This is the part that is genuinely nicer in SQL than in an imperative
  language: the model is a table-valued function, so CROSS APPLY runs a
  whole batch of prompts and the result is one tidy relation.
==============================================================================*/
SELECT  pr.prompt,
        rank_pos = ROW_NUMBER() OVER (PARTITION BY pr.prompt ORDER BY nt.p DESC),
        nt.token,
        probability = FORMAT(nt.p, 'P1')
FROM    (VALUES
            (N'SELECT'),
            (N'SELECT * FROM'),
            (N'SELECT customer_id,'),
            (N'SELECT COUNT(*) FROM orders GROUP BY'),
            (N'SELECT * FROM customers WHERE'),
            (N'INNER JOIN customers c ON'),
            (N'WITH ranked AS ('),
            (N'UPDATE employees SET')
        ) AS pr(prompt)
CROSS APPLY (
        SELECT TOP (4) token, p
        FROM   dbo.fn_NextToken(pr.prompt)
        ORDER BY p DESC
) AS nt
ORDER BY pr.prompt, nt.p DESC;
GO

/*==============================================================================
  QUERY 4 -- Generation.
  Deterministic first, then sampled at three temperatures.
==============================================================================*/
EXEC dbo.usp_Generate @Prompt = N'SELECT * FROM', @Greedy = 1, @MaxTokens = 40;

EXEC dbo.usp_Generate @Prompt = N'SELECT',        @Temperature = 0.3, @Samples = 3;
EXEC dbo.usp_Generate @Prompt = N'SELECT',        @Temperature = 0.8, @Samples = 3;
EXEC dbo.usp_Generate @Prompt = N'SELECT',        @Temperature = 1.4, @Samples = 3;

EXEC dbo.usp_Generate @Prompt = N'WITH monthly AS (',            @Temperature = 0.7, @Samples = 3;
EXEC dbo.usp_Generate @Prompt = N'SELECT c.customer_name, COUNT(', @Temperature = 0.7, @Samples = 3;
GO

/*==============================================================================
  QUERY 5 -- Scoring.

  fn_Score returns, per token, the probability the model assigned to the
  token that actually appeared. High surprisal marks the spots where the
  query departs from what internet SQL usually looks like.
==============================================================================*/
SELECT  position, token,
        probability   = FORMAT(p, 'P2'),
        surprisal_bits= ROUND(surprisal, 2)
FROM    dbo.fn_Score(N'SELECT customer_id, COUNT(*) FROM orders GROUP BY customer_id;')
ORDER BY position;

-- Typical query versus deliberately odd query, side by side.
SELECT  q.label,
        tokens      = COUNT(*),
        avg_bits    = ROUND(AVG(s.surprisal), 3),
        perplexity  = ROUND(EXP(-AVG(s.log_p)), 2)
FROM    (VALUES
            (N'idiomatic',  N'SELECT * FROM customers WHERE country = ''germany'';'),
            (N'idiomatic',  N'SELECT category_id, AVG(price) FROM products GROUP BY category_id;'),
            (N'unusual',    N'SELECT GROUP BY customers price WHERE FROM ;'),
            (N'unusual',    N'DELETE products SET orders JOIN AVG ( 3 ;')
        ) AS q(label, sql_text)
CROSS APPLY dbo.fn_Score(q.sql_text) AS s
GROUP BY q.label, q.sql_text
ORDER BY perplexity;
GO

/*==============================================================================
  QUERY 6 -- Perplexity leaderboard over the training corpus.

  Which published queries does the model find most and least predictable?
  The worst-scoring rows are the ones whose patterns appear only once, which
  makes this a serviceable outlier detector for a query log.
==============================================================================*/
WITH scored AS (
    SELECT  c.doc_id, c.source_site, c.query_text,
            tokens     = COUNT(*),
            perplexity = EXP(-AVG(s.log_p))
    FROM    dbo.Corpus AS c
    CROSS APPLY dbo.fn_Score(c.query_text) AS s
    GROUP BY c.doc_id, c.source_site, c.query_text
)
SELECT  band = CASE WHEN rn <= 8 THEN 'most predictable' ELSE 'least predictable' END,
        query_text, tokens, perplexity = ROUND(perplexity, 2)
FROM (
    SELECT *, rn = ROW_NUMBER() OVER (ORDER BY perplexity),
              rd = ROW_NUMBER() OVER (ORDER BY perplexity DESC)
    FROM scored
) AS x
WHERE rn <= 8 OR rd <= 8
ORDER BY perplexity;
GO

/*==============================================================================
  QUERY 7 -- What did the embeddings learn?

  Cosine similarity between token vectors. Restricted to reasonably frequent
  tokens, because a token seen twice has an embedding that barely moved from
  its random initialisation.

  Expect table names to cluster with table names, and aggregate functions to
  cluster with aggregate functions -- the model was never told that these
  categories exist.
==============================================================================*/
DECLARE @probe nvarchar(100) = N'customers';

WITH probe AS (
    SELECT e.dim, e.w
    FROM   dbo.Emb AS e
    JOIN   dbo.Vocab AS v ON v.token_id = e.token_id
    WHERE  v.token = @probe
),
pnorm AS (SELECT nrm = SQRT(SUM(w * w)) FROM probe)
SELECT TOP (10)
        v.token, v.freq,
        cosine = ROUND(SUM(e.w * probe.w)
                       / (SQRT(SUM(e.w * e.w)) * MAX(pnorm.nrm)), 3)
FROM    dbo.Emb AS e
JOIN    probe ON probe.dim = e.dim
JOIN    dbo.Vocab AS v ON v.token_id = e.token_id
CROSS JOIN pnorm
WHERE   v.freq >= 8
  AND   v.token <> @probe
GROUP BY v.token, v.freq
ORDER BY cosine DESC;
GO

-- The same probe, run over several tokens at once.
SELECT  probe.token AS probe_token, nn.token AS neighbour, nn.cosine
FROM    (VALUES (N'customers'), (N'sum'), (N'where'), (N'inner'), (N'group'))
            AS probe(token)
CROSS APPLY (
    SELECT TOP (5)
           v.token,
           cosine = ROUND(SUM(e.w * pe.w)
                          / (SQRT(SUM(e.w * e.w)) * MAX(pn.nrm)), 3)
    FROM   dbo.Emb AS e
    JOIN   dbo.Vocab AS v ON v.token_id = e.token_id
    JOIN  (SELECT e2.dim, e2.w
           FROM   dbo.Emb AS e2
           JOIN   dbo.Vocab AS v2 ON v2.token_id = e2.token_id
           WHERE  v2.token = probe.token) AS pe ON pe.dim = e.dim
    CROSS JOIN (SELECT nrm = SQRT(SUM(e3.w * e3.w))
                FROM   dbo.Emb AS e3
                JOIN   dbo.Vocab AS v3 ON v3.token_id = e3.token_id
                WHERE  v3.token = probe.token) AS pn
    WHERE  v.freq >= 8 AND v.token <> probe.token
    GROUP BY v.token
    ORDER BY cosine DESC
) AS nn
ORDER BY probe.token, nn.cosine DESC;
GO

/*==============================================================================
  QUERY 8 -- Vector arithmetic on the embedding table.

      vec('orders') - vec('order_id') + vec('customer_id')  ~=  ?

  On a corpus this small the result is noisy, and it is honest to say so;
  the point is that the arithmetic is a GROUP BY over a 16-column table.
==============================================================================*/
DECLARE @a nvarchar(100) = N'orders',
        @b nvarchar(100) = N'order_id',
        @c nvarchar(100) = N'customer_id';

WITH target AS (
    SELECT  e.dim,
            w = SUM(CASE v.token WHEN @a THEN e.w WHEN @b THEN -e.w WHEN @c THEN e.w END)
    FROM    dbo.Emb AS e
    JOIN    dbo.Vocab AS v ON v.token_id = e.token_id
    WHERE   v.token IN (@a, @b, @c)
    GROUP BY e.dim
),
tnorm AS (SELECT nrm = SQRT(SUM(w * w)) FROM target)
SELECT TOP (8)
        v.token,
        cosine = ROUND(SUM(e.w * target.w)
                       / (SQRT(SUM(e.w * e.w)) * MAX(tnorm.nrm)), 3)
FROM    dbo.Emb AS e
JOIN    target ON target.dim = e.dim
JOIN    dbo.Vocab AS v ON v.token_id = e.token_id
CROSS JOIN tnorm
WHERE   v.freq >= 5 AND v.token NOT IN (@a, @b, @c)
GROUP BY v.token
ORDER BY cosine DESC;
GO

/*==============================================================================
  QUERY 9 -- Constrained decoding.

  Because the distribution arrives as a rowset, constraining generation to a
  grammar is a WHERE clause plus a renormalisation window. No special decoder
  is needed. Here: after FROM, only allow known table names.
==============================================================================*/
SELECT  token,
        p_raw        = FORMAT(p, 'P3'),
        p_constrained= FORMAT(p / SUM(p) OVER (), 'P1')
FROM    dbo.fn_NextToken(N'SELECT * FROM')
WHERE   token IN (N'customers', N'orders', N'products', N'employees',
                  N'payments', N'departments', N'categories')
ORDER BY p DESC;
GO

/*==============================================================================
  QUERY 10 -- Training diagnostics.
==============================================================================*/
SELECT  epoch, avg_loss = ROUND(avg_loss, 4),
        perplexity = ROUND(perplexity, 3),
        lr = ROUND(lr, 5), elapsed_ms
FROM    dbo.TrainLog
ORDER BY epoch;

-- Loss reduction per epoch, as a window function over the log.
SELECT  epoch,
        perplexity  = ROUND(perplexity, 3),
        delta       = ROUND(perplexity - LAG(perplexity) OVER (ORDER BY epoch), 4)
FROM    dbo.TrainLog
ORDER BY epoch DESC;

-- Weight magnitudes. A hidden unit whose W2 row is near zero has stopped
-- contributing; a very negative B1 usually means a dead ReLU.
SELECT  tensor = 'W1', mean_abs = ROUND(AVG(ABS(w)), 5), max_abs = ROUND(MAX(ABS(w)), 4) FROM dbo.W1
UNION ALL SELECT 'W2',  ROUND(AVG(ABS(w)), 5), ROUND(MAX(ABS(w)), 4) FROM dbo.W2
UNION ALL SELECT 'Emb', ROUND(AVG(ABS(w)), 5), ROUND(MAX(ABS(w)), 4) FROM dbo.Emb;

SELECT  dead_ish_hidden_units = COUNT(*)
FROM    dbo.B1
WHERE   w < -0.5;
GO

/*==============================================================================
  QUERY 11 -- The whole model, as data.

  Nothing here is hidden in a binary blob. This is the entire trained model,
  and it can be backed up, diffed, version-controlled, or shipped by INSERT.
==============================================================================*/
SELECT TOP (20) v.token, e.dim, weight = ROUND(e.w, 4)
FROM    dbo.Emb AS e
JOIN    dbo.Vocab AS v ON v.token_id = e.token_id
--WHERE   v.token = N'select'
ORDER BY e.dim;
GO

Where It Falls Over

I’d rather say this plainly than oversell it.

A three-token context window means it has learned local SQL *syntax*, not SQL *semantics*. It’ll cheerfully open a `WITH` clause and never close it, or emit `WHERE order_id = ‘london’`. It has no idea what a table is.

122 training queries is a very small corpus, so it memorises templates as much as it generalises. Embedding neighbours are meaningful for frequent tokens and close to noise for rare ones, whose vectors have barely moved from initialisation.

And autoregressive sampling stays stubbornly procedural. Each token depends on the last, and a recursive CTE can’t call a table-valued function on its recursive member. That one loop is the only imperative code in the whole system.

The Point

This isn’t how you should serve a language model. It’s a demonstration that the gap between “database engine” and “machine learning framework” is narrower than the tooling implies. Both are doing sums over sets. One of them just has better marketing.

If you work with SQL every day, building one of these will teach you more about what a language model actually does than any number of tutorials with a `model.fit()` in them. There’s nowhere to hide. Every matrix multiply is a join you wrote yourself.

There’s a detail here I didn’t notice until I’d finished writing the code. In 1999 I was doing n-grams and TF-IDF in T-SQL, and I stopped because nobody was interested. The wall I’d have hit next is data sparsity — n-grams can only count what they’ve already seen, and most sequences they never see at all.

Bengio’s paper is the answer to exactly that problem. It was published in 2003. I put the work down four years too early.

So this isn’t a departure from what I was doing back then. It’s the next page of it, in the same language, twenty-seven years later. I write Python when the job needs it. I’ve never needed a second language for the part I actually care about.

The full scripts run on SQL Server 2016 or later, and on Azure SQL. Four files: schema and corpus, model and forward pass, training, example queries. They can be ported in principle to PostgreSQL though the tokeniser and temp tables need rewriting.

I would suggest running the code from the scripts rather than copy/paste from this article as the Medium publishing tool replaces characters that are needed. I think I caught them all but the files have been run and tested.

https://drive.google.com/drive/folders/1BGKPr8nkRhQmrKZPmFr49MmPDvi2QL4k?usp=sharing

**If this was useful, a few claps help other people find it. Follow me for more on SQL as a computational medium — I keep finding things it can do that it probably shouldn’t.**

*What’s the strangest thing you’ve built inside a database? I’d genuinely like to know — leave a comment.*