leftovers
This commit is contained in:
Lars Nolden
+514
@@ -0,0 +1,514 @@
|
||||
# Query tiles
|
||||
|
||||
Status: rough draft, no code changes applied. §4 and §6 SQL are sketches that have not been executed; everything in §2 and §3 was measured.
|
||||
|
||||
Decisions taken, on evidence in §2: the ad-hoc surface runs against an **engine-enforced read-only snapshot** in a second DuckDB instance, never the existing handle; every submitted statement passes a **parse-only single-`SELECT` gate** before it reaches the driver; the natural-language model is **loopback-only**, which removes redaction and lets the real taxonomy into the prompt; chart choice is **deterministic from result shape**, with the model's hint as a tiebreak only.
|
||||
|
||||
Goal: answer *"How much did I spend on my hobbies the last 2 weeks?"* and *"What are my fixed recurring costs for the past two months?"* as pinnable tiles, without giving up the journal-is-truth architecture or adding a browser dependency.
|
||||
|
||||
## 1. Three surfaces, one pipeline
|
||||
|
||||
| Surface | Input | Who writes the SQL | Useful without the model |
|
||||
| --- | --- | --- | --- |
|
||||
| Raw query | SQL | you | yes |
|
||||
| Ask | a question | local model, reviewed by you | — |
|
||||
| Pinned tile | saved plan | whoever wrote it, once | yes |
|
||||
|
||||
```
|
||||
question ──► /api/ask ──► {sql, title, chart, assumptions} ──┐
|
||||
├─► review ──► gate ──► read-only snapshot ──► shape ──► tile ──► queries.finance
|
||||
raw SQL ─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The model never executes anything. `/api/ask` returns a plan; `/api/query` runs one. Keeping them apart means a slow or wrong model degrades one surface instead of all three, and the raw path carries no AI dependency at all.
|
||||
|
||||
## 2. Measured: the obvious implementation is unsafe
|
||||
|
||||
A throwaway probe (`internal/analytics/probe_test.go`, deleted) exercised the sandbox primitives against duckdb-go v2.5.6.
|
||||
|
||||
**Multi-statement SQL is executed by `database/sql`.** The driver's `prepareStmts` extracts every statement, then loops `for i := 0; i < count-1` **preparing and executing each leading statement**, returning only the last one prepared (`connection.go:215-256` in `github.com/duckdb/duckdb-go/v2@v2.5.6`). Submitting
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT 1) AS q LIMIT 1; DROP TABLE t
|
||||
```
|
||||
|
||||
dropped the table. Wrapping user SQL in `SELECT * FROM ( … ) LIMIT n` is therefore **not** a sandbox: the user closes the paren and appends statements. The wrapper still rejects a bare `DROP`/`SET`/`COPY`/`PRAGMA` at parse time, so it is a usability filter, not a boundary.
|
||||
|
||||
**A second instance with a read-only attachment is a boundary.** In-process, alongside the existing read-write handle:
|
||||
|
||||
```sql
|
||||
-- instance 2, dsn ":memory:"
|
||||
ATTACH '<data>/cache/finance.duckdb' AS fd (READ_ONLY); -- succeeds while instance 1 holds it RW
|
||||
USE fd;
|
||||
SET threads = 2; SET memory_limit = '256MB';
|
||||
SET enable_external_access = false;
|
||||
SET lock_configuration = true;
|
||||
```
|
||||
|
||||
| Probe | Result |
|
||||
| --- | --- |
|
||||
| `DROP TABLE fd.t` / `INSERT` / `UPDATE` / `CREATE` on `fd` | `Invalid Input Error: Cannot execute statement of type "DROP" on database "fd" which is attached in read-only mode!` |
|
||||
| `SELECT 1; DROP TABLE fd.t` | same rejection — multi-statement does not help |
|
||||
| `read_csv('/etc/passwd')` | `Permission Error: … file system operations are disabled by configuration` |
|
||||
| `COPY fd.t TO '/tmp/x.csv'` | same |
|
||||
| `ATTACH '/tmp/evil.db' AS evil` | same |
|
||||
| `INSTALL httpfs` | same |
|
||||
| `SET enable_external_access = true` | `Cannot change configuration option … the configuration has been locked` |
|
||||
| `CREATE TABLE memory.evil (x INT)` | **allowed** — scratch in the throwaway in-memory catalog |
|
||||
| `SELECT count(*) FROM fd.t`, `information_schema.columns` | allowed |
|
||||
| 10¹³-row cross join, 400 ms `context` deadline | interrupted after 900 ms; pool usable afterwards |
|
||||
| read-write cache after all of the above | intact |
|
||||
|
||||
Residual write surface is the ad-hoc instance's own `memory` catalog. That is harmless — it is discarded when the instance is recreated — and arguably useful for materialising an intermediate result. It is *not* free of cost: see the temp-directory item in §3.
|
||||
|
||||
**The attachment is a frozen snapshot.** After instance 1 committed an `INSERT`, the attached instance still reported the old row count. The ad-hoc instance must be recreated whenever the projection is rebuilt. Treat this as a feature: every tile in one page render reads one consistent snapshot, labelled with the existing `State.Revision` (`internal/app/app.go:44`).
|
||||
|
||||
**`json_serialize_sql` is a free parse-only gate.** User SQL is passed as a *bound parameter*, so there is no injection surface and nothing executes:
|
||||
|
||||
```sql
|
||||
SELECT json_serialize_sql(CAST(? AS VARCHAR))
|
||||
```
|
||||
|
||||
| Submitted | Gate result |
|
||||
| --- | --- |
|
||||
| `SELECT 1` | `statements=1 nodes=[SELECT_NODE]` |
|
||||
| `WITH x AS (SELECT 1) SELECT * FROM x` | `statements=1 nodes=[SELECT_NODE]` |
|
||||
| `SELECT 1 -- ; DROP TABLE t` | `statements=1 nodes=[SELECT_NODE]` — comment handled |
|
||||
| `SELECT '; DROP TABLE t'` | `statements=1 nodes=[SELECT_NODE]` — literal handled |
|
||||
| `SELECT 1; DROP TABLE t` | `error_type=not implemented`, `Only SELECT statements can be serialized to json!` |
|
||||
| `DROP TABLE t` / `SET …` / `COPY … TO` / `PRAGMA version` | same rejection |
|
||||
| `this is not sql` | `error_type=parser`, `syntax error at or near "this"` |
|
||||
| `SELECT * FROM nope` | `statements=1 nodes=[SELECT_NODE]` — parse only, no catalog binding |
|
||||
|
||||
Errors come back as JSON data, not as a raised exception, so the gate yields a clean message instead of a stray engine error. Unknown tables and columns are *not* caught here; they surface at execution with DuckDB's own "Did you mean" hint, which is the better message anyway.
|
||||
|
||||
**`DECIMAL(24,4)` scans as `duckdb.Decimal`** with exact `String()` → `-12345678901234.5678` (`types.go:371-386`). Money must be serialised through that, never through `Float64()`; `domain.Money` (`internal/domain/model.go:4`) is an exact decimal string for the same reason.
|
||||
|
||||
**`lock_configuration` is missing from the existing `Open`.** `internal/analytics/store.go:66-73` sets `enable_external_access = false` and disables extension autoload but never locks the configuration, so any future SQL path on that handle could re-enable filesystem access. Add it there regardless of this feature; the probe confirms DDL still works after the lock, and that a later `SET` on the same instance is rejected.
|
||||
|
||||
## 3. Execution layer
|
||||
|
||||
### 3.1 Two instances, one process
|
||||
|
||||
`analytics.Store` gains a second, disposable instance:
|
||||
|
||||
```go
|
||||
type Store struct {
|
||||
db *sql.DB // read-write projection, as today
|
||||
mu sync.Mutex // guards snapshot swap
|
||||
snap *snapshot // read-only attachment, nil until first use
|
||||
path string
|
||||
}
|
||||
|
||||
type snapshot struct {
|
||||
connector *duckdb.Connector // dsn ":memory:"
|
||||
db *sql.DB // MaxOpenConns(1)
|
||||
revision string
|
||||
}
|
||||
```
|
||||
|
||||
Lifecycle: created lazily on the first ad-hoc query, closed and recreated when `revision` differs. `internal/app/app.go:167-173` already knows the moment the projection changes (`a.indexed = rev`); the snapshot is invalidated there.
|
||||
|
||||
`ATTACH` must precede `SET enable_external_access = false`, because attaching is itself a filesystem operation. The path is interpolated, not bound — `ATTACH ?` is a parser error, and the path is ours, not user input.
|
||||
|
||||
### 3.2 Do not hold the app mutex
|
||||
|
||||
`App.Dashboard` (`app.go:207-217`) holds `a.mu` for the whole query, and `store.go:63` pins `SetMaxOpenConns(1)` deliberately so dashboard snapshots serialise against rebuilds. A five-second ad-hoc scan on either of those would stall every read and write in the process. The ad-hoc path therefore:
|
||||
|
||||
1. takes `a.mu`, calls `a.snapshot(ctx)` to make the projection current, reads `revision`, obtains the read-only handle, **releases `a.mu`**;
|
||||
2. runs the query on the snapshot pool with its own deadline.
|
||||
|
||||
A rebuild that lands mid-query is harmless: the old attachment stays valid until its last reader is done.
|
||||
|
||||
### 3.3 Limits
|
||||
|
||||
| Control | Value | Why |
|
||||
| --- | --- | --- |
|
||||
| statements | exactly 1, `SELECT_NODE` | §2 gate |
|
||||
| context deadline | 5 s default, 30 s ceiling | verified to interrupt; pool survives |
|
||||
| row cap | `LIMIT 501`, report `truncated` when 501 come back | one page of table, bounded JSON |
|
||||
| `memory_limit` | `256MB`, matching `store.go:68` | bounded native memory |
|
||||
| `temp_directory` | disabled, or `max_temp_directory_size` small | **unverified** — the probe created a 20 M-row scratch table under a 256 MB `memory_limit` without error, but that table compresses to almost nothing, so whether it spilled at all was not established. Whether `enable_external_access = false` covers spill files needs its own probe; the safe default is to fail fast rather than risk filling the data disk |
|
||||
| concurrency | one ad-hoc query at a time, others get a clear "busy" | single connection; queueing is worse than refusing |
|
||||
|
||||
The `LIMIT` is appended by wrapping the gated statement — `SELECT * FROM ( … ) AS q LIMIT 501`. The wrapper is safe *after* the gate has proven the text is one `SELECT`.
|
||||
|
||||
### 3.4 Result encoding
|
||||
|
||||
```json
|
||||
{
|
||||
"columns": [{"name": "month", "type": "DATE"}, {"name": "outflow", "type": "DECIMAL(24,4)"}],
|
||||
"rows": [["2026-08-01", "412.7300"]],
|
||||
"truncated": false,
|
||||
"ms": 14,
|
||||
"revision": "…"
|
||||
}
|
||||
```
|
||||
|
||||
Rules: `DECIMAL` → exact string via `duckdb.Decimal.String()`; `DATE`/`TIMESTAMP` → ISO text, matching the calendar-day discipline in `web/src/ui.tsx:73-75`; `NULL` → JSON `null`; `LIST`/`STRUCT` → JSON; everything else by its natural JSON type. `columns[].type` carries `DatabaseTypeName()` so the frontend can right-align numerics and pick a chart without guessing.
|
||||
|
||||
## 4. Semantic views — the accuracy lever
|
||||
|
||||
Both example questions are four-table joins with three traps:
|
||||
|
||||
1. `amount` is **signed net movement**, not an expense (`store.go:34-36`);
|
||||
2. `kind IN ('transfer', 'investment')` must be excluded — the canonical clause is `store.go:242`, explained at `store.go:238-241`;
|
||||
3. category rollups need `category_ancestors`, and ancestor groups **overlap**, so they must never be summed together (`store.go:34-36`).
|
||||
|
||||
A small model will get all three wrong against the base tables. Views make them unrepresentable. They also shrink the task from a BIRD-style multi-join to a Spider-easy single-table query, which is exactly where small models are strong (§11).
|
||||
|
||||
Sketches, not yet executed:
|
||||
|
||||
```sql
|
||||
-- Every fact, denormalised, with names instead of registry ids.
|
||||
CREATE VIEW v_tx AS
|
||||
SELECT t.id, t.booking_date, t.value_date, t.kind, t.currency,
|
||||
t.account_id, t.category_id, t.merchant_id, -- kept for joins back to base tables
|
||||
t.amount, -- signed: negative is money out
|
||||
-t.amount AS outflow, -- positive is money out
|
||||
a.display_name AS account,
|
||||
a.institution,
|
||||
c.name AS category,
|
||||
c.kind AS category_kind,
|
||||
COALESCE(m.name, '') AS merchant,
|
||||
t.counterparty,
|
||||
t.raw_description AS description,
|
||||
(SELECT string_agg(c2.name, ' / ' ORDER BY ca.depth DESC)
|
||||
FROM category_ancestors ca JOIN categories c2 ON c2.id = ca.ancestor_id
|
||||
WHERE ca.category_id = t.category_id) AS category_path,
|
||||
(SELECT list(g.name ORDER BY g.name)
|
||||
FROM transaction_tags tt JOIN tags g ON g.id = tt.tag_id
|
||||
WHERE tt.transaction_id = t.id) AS tags
|
||||
FROM transactions t
|
||||
LEFT JOIN accounts a ON a.id = t.account_id
|
||||
LEFT JOIN categories c ON c.id = t.category_id
|
||||
LEFT JOIN merchants m ON m.id = t.merchant_id;
|
||||
|
||||
-- Spending and income analytics. Mirrors store.go:242 exactly.
|
||||
CREATE VIEW v_flow AS SELECT * FROM v_tx WHERE kind NOT IN ('transfer', 'investment');
|
||||
CREATE VIEW v_spending AS SELECT * FROM v_flow WHERE amount < 0;
|
||||
CREATE VIEW v_income AS SELECT * FROM v_flow WHERE amount > 0;
|
||||
|
||||
-- Correct rollups. One row per transaction × ancestor: never sum across ancestors.
|
||||
CREATE VIEW v_spending_by_ancestor AS
|
||||
SELECT s.id, s.booking_date, s.currency, s.outflow,
|
||||
ca.ancestor_id, c.name AS ancestor, ca.depth
|
||||
FROM v_spending s
|
||||
JOIN category_ancestors ca ON ca.category_id = s.category_id
|
||||
JOIN categories c ON c.id = ca.ancestor_id;
|
||||
```
|
||||
|
||||
`category_path` is the cheap win: *"hobbies"* becomes `WHERE category_path ILIKE '%Hobbies%'` — no join, no fanout, no ancestor arithmetic. `v_spending_by_ancestor` exists for when a correct grouped total is needed.
|
||||
|
||||
A calendar spine, so empty months render as zero bars rather than vanishing:
|
||||
|
||||
```sql
|
||||
CREATE VIEW v_month AS
|
||||
SELECT month FROM (
|
||||
SELECT DISTINCT date_trunc('month', booking_date)::DATE AS month FROM transactions
|
||||
); -- draft: a gap-free generate_series between min and max is what is actually wanted,
|
||||
-- and must behave on an empty dataset where min(booking_date) is NULL
|
||||
```
|
||||
|
||||
Recurring costs — this *is* the second example question, and it is worth owning as a view rather than hoping the model derives it:
|
||||
|
||||
```sql
|
||||
CREATE VIEW v_recurring AS
|
||||
WITH s AS (
|
||||
SELECT COALESCE(NULLIF(merchant, ''), counterparty) AS payee, booking_date, outflow, category_path
|
||||
FROM v_spending
|
||||
WHERE COALESCE(NULLIF(merchant, ''), counterparty) <> ''
|
||||
), d AS (
|
||||
SELECT *, date_diff('day', lag(booking_date) OVER (PARTITION BY payee ORDER BY booking_date),
|
||||
booking_date) AS gap
|
||||
FROM s
|
||||
), g AS (
|
||||
SELECT payee,
|
||||
count(*) AS hits,
|
||||
median(gap) AS gap_days,
|
||||
median(outflow) AS typical_amount,
|
||||
stddev_pop(outflow) / nullif(avg(outflow), 0) AS amount_variation,
|
||||
max(booking_date) AS last_seen,
|
||||
any_value(category_path) AS category_path
|
||||
FROM d GROUP BY payee
|
||||
)
|
||||
SELECT *,
|
||||
CASE WHEN gap_days BETWEEN 6 AND 8 THEN 'weekly'
|
||||
WHEN gap_days BETWEEN 13 AND 16 THEN 'biweekly'
|
||||
WHEN gap_days BETWEEN 25 AND 35 THEN 'monthly'
|
||||
WHEN gap_days BETWEEN 85 AND 95 THEN 'quarterly'
|
||||
WHEN gap_days BETWEEN 350 AND 380 THEN 'yearly'
|
||||
END AS cadence,
|
||||
typical_amount * 30.44 / gap_days AS monthly_equivalent
|
||||
FROM g
|
||||
WHERE hits >= 3 AND amount_variation < 0.15;
|
||||
```
|
||||
|
||||
Cadence classification rather than a hardcoded 28–31 day window, so a yearly insurance premium and a weekly grocery standing order both land correctly, normalised to `monthly_equivalent`. The `amount_variation` filter is what separates a subscription from a coffee habit; grouping by payee alone (rather than payee × rounded amount) survives a price increase. Both choices are open — §14.
|
||||
|
||||
### 4.1 Mechanics and the stability contract
|
||||
|
||||
Views are part of `schema` (`store.go:97-106`) and must be created after `postings` is populated (`store.go:209-215`). `Rebuild` drops tables by name (`store.go:119`); DuckDB refuses to drop a table a view depends on, so **the view names must join that list, dropped before the tables**.
|
||||
|
||||
Pinned SQL in `queries.finance` outlives every rebuild, so view names and columns become a **public interface**. Base tables stay reachable and undocumented: power users may use them, nothing promises they are stable. A tile whose SQL no longer binds shows the engine error in place of its chart — visible breakage, never a silent zero.
|
||||
|
||||
## 5. Statement gate
|
||||
|
||||
```
|
||||
POST /api/query {sql}
|
||||
├─ reject empty / >8 KiB
|
||||
├─ SELECT json_serialize_sql(CAST(? AS VARCHAR)) ── parse only, user SQL bound as a value
|
||||
│ ├─ error:true → 400 with error_message
|
||||
│ └─ len(statements) != 1 → 400 "one SELECT statement at a time"
|
||||
│ └─ node.type != SELECT_NODE → 400 same
|
||||
├─ wrap: SELECT * FROM ( <sql> ) AS q LIMIT 501
|
||||
└─ run on the read-only snapshot with a deadline
|
||||
```
|
||||
|
||||
The gate runs on the read-write handle (it is a pure function of a string) or on the snapshot — either works; the snapshot avoids touching the serialised handle at all.
|
||||
|
||||
Engine errors are returned verbatim: it is the user's own SQL, and DuckDB's messages are good. Nothing about a query or its results is logged, consistent with the no-provider-logging discipline in `internal/classification/client.go`.
|
||||
|
||||
## 6. Ask: question → plan
|
||||
|
||||
### 6.1 Why a new package, not `classification.Client`
|
||||
|
||||
| Concern | `classification` | ask |
|
||||
| --- | --- | --- |
|
||||
| Endpoint | OpenRouter by default; `http://` allowed only for loopback (`client.go:321`) | loopback **required** |
|
||||
| Request body | hardcodes `provider: {data_collection: deny, zdr: true, require_parameters: true}` (`client.go:302`) | no provider block; a local server may reject unknown fields |
|
||||
| Privacy | `redactor()` strips IBANs, own names, digit-bearing tokens (`privacy.go:53-107`) | no redaction — redacting the schema destroys the query |
|
||||
| Sent content | one transaction | schema, taxonomy, dates; **no transaction rows** |
|
||||
| Failure | falls back to unclassified enrichment with provenance | returns an error; the raw SQL path is unaffected |
|
||||
|
||||
The inversion is the point: **because the model is loopback-only, redaction is unnecessary**, and the real category names, tag names and account names can go into the prompt — which is precisely what NL→SQL needs to write a correct `WHERE`. A redacted schema yields useless SQL, so routing this through OpenRouter is not a smaller version of the feature; it is a different, broken one.
|
||||
|
||||
New package `internal/askql`. `classification` is untouched.
|
||||
|
||||
### 6.2 Transport
|
||||
|
||||
Target **llama-server** (llama.cpp). Ollama's `/v1/chat/completions` shim [mistranslates or ignores](https://github.com/ollama/ollama/issues/10001) `response_format.json_schema`; grammar-constrained decoding is what makes a small model emit parseable JSON every time, so it is not optional. llama-server honours the OpenAI shape and compiles the schema to a grammar.
|
||||
|
||||
Deployment: nixpkgs ships [`services.llama-cpp`](https://github.com/nixos/nixpkgs/blob/master/nixos/modules/services/misc/llama-cpp.nix) with `model`, `host`, `port`, `extraFlags` — one systemd unit next to `nix/service.nix`, loopback-bound, firewall untouched.
|
||||
|
||||
Configuration follows the existing hand-rolled reader (`app.go:100-127`, writer at `app.go:376-380`), which rejects unknown keys, so both sides need the new pair:
|
||||
|
||||
```toml
|
||||
ask_base_url = "http://127.0.0.1:8081/v1"
|
||||
ask_model = "qwen-coder"
|
||||
```
|
||||
|
||||
Empty `ask_base_url` disables the Ask surface; the query and tile surfaces keep working. A non-loopback host is a startup error, not a warning.
|
||||
|
||||
`server.go:114` sets CSP `connect-src 'self'`, so the browser cannot reach the model even on loopback. The proxy through Go is mandatory, and is the right place for it anyway.
|
||||
|
||||
### 6.3 Prompt
|
||||
|
||||
Everything the model needs, nothing it does not:
|
||||
|
||||
- the view DDL from §4, column types and one-line comments — **views only**, base tables omitted;
|
||||
- the taxonomy: every category with its `category_path`, every tag, the top ~50 merchants by transaction count. `finance/categories.finance` is 364 B today; the whole taxonomy fits with room to spare;
|
||||
- **today's date**, plus `min(booking_date)` and `max(booking_date)`. The model has no clock; without this, "last 2 weeks" silently returns zero rows;
|
||||
- a handful of curated question → SQL exemplars. Cheapest remaining accuracy gain (§11);
|
||||
- the framing already used in `classification`: user content is untrusted data, never instructions.
|
||||
|
||||
No transaction rows. Registry names are user-influenced (classification writes merchant names from bank text), so a hostile merchant name could in principle carry instructions — bounded by the fact that the only thing the model can produce is a read-only `SELECT` that **you see before it runs**. Worst case is a confusing query you decline.
|
||||
|
||||
### 6.4 Response
|
||||
|
||||
Strict JSON schema, one round trip, no tool loop:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object", "additionalProperties": false,
|
||||
"required": ["sql", "title", "chart", "assumptions"],
|
||||
"properties": {
|
||||
"sql": {"type": "string"},
|
||||
"title": {"type": "string"},
|
||||
"chart": {"enum": ["kpi", "bar", "line", "table"]},
|
||||
"x": {"type": ["string", "null"]},
|
||||
"y": {"type": "array", "items": {"type": "string"}},
|
||||
"assumptions": {"type": "string"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`assumptions` is load-bearing, not decoration:
|
||||
|
||||
> hobbies → categories under `Expenses / Leisure / Hobbies`; last 2 weeks → `2026-08-29` … `2026-09-11`
|
||||
|
||||
`finance/categories.finance` currently holds only the four seed categories, so *"hobbies"* may not exist as a category at all and may resolve to a tag or a merchant. Making the model state its mapping is what lets you correct it in one edit instead of mistrusting every answer.
|
||||
|
||||
Repair: on a bind error, one retry with the engine message appended, then stop. Never an unattended loop. Both attempts are shown.
|
||||
|
||||
Serialisation: one inference at a time, 60 s deadline, `Abort` in the UI. No rate controller — a loopback model has no quota, and `ratelimit.Controller` exists for provider cooldowns.
|
||||
|
||||
## 7. Visualization
|
||||
|
||||
Deterministic from the result shape; the model's `chart` is a tiebreak, never an instruction.
|
||||
|
||||
| Shape | Render |
|
||||
| --- | --- |
|
||||
| 1 row × 1 numeric column | KPI, with the previous-period comparison idiom already in `Overview` |
|
||||
| `DATE`-like column + 1 numeric | line/area |
|
||||
| text column + 1 numeric, ≤ 24 rows | horizontal bars |
|
||||
| anything else | table |
|
||||
|
||||
Reuse what exists: `.bar-chart`, `.bar-column`, `.bar-track`, `.bar`, `.bar-value`, `.bar-label` (`web/src/styles.css:548-599`), `MonthlyChart` (`Overview.tsx:366-410`) and `GroupPanel` (`Overview.tsx:411-480`). A line chart needs a small hand-rolled SVG. **No charting dependency**: `web/package.json` has four runtime deps and the CSP serves scripts from `'self'` only; a self-hosted workspace that bundles its own fonts rather than fetching them (`main.tsx:37-39`) should not grow a chart library for four chart types.
|
||||
|
||||
Explicitly refused: letting the model emit chart code or a Vega-style spec it invents. That is arbitrary JS in a service with no application authentication.
|
||||
|
||||
## 8. Persistence and UI
|
||||
|
||||
### 8.1 `queries.finance`
|
||||
|
||||
Pinned tiles are user content, so they belong in the journal, not in `state/`. Add `queries.finance` to `registryFiles` (`internal/journal/codec.go:22`); the codec is json-tag driven, so the block grammar follows from the struct:
|
||||
|
||||
```
|
||||
query {
|
||||
id: "qry_hobbies_2w"
|
||||
title: "Hobby spend, last 2 weeks"
|
||||
question: "How much did I spend on my hobbies the last 2 weeks?"
|
||||
sql: "SELECT sum(outflow) AS spent FROM v_spending WHERE category_path ILIKE '%Hobbies%' AND booking_date >= current_date - INTERVAL 14 DAY"
|
||||
chart: "kpi"
|
||||
position: 1
|
||||
}
|
||||
```
|
||||
|
||||
Diffable, hand-editable, backed up with everything else, survives a cache wipe. Writes go through the existing `App.mutate` revision-conflict path (`app.go:199-201`) and the `s.account`/`s.category` handler idiom (`server.go:239-293`).
|
||||
|
||||
`domain.Dataset` gains `Queries []Query`, and `domain.Validate` gains id/title/SQL checks. Whether a stored query is gated at write time or only at run time is an open item — gating at write time means a view rename can make the journal unloadable, which is worse than a broken tile.
|
||||
|
||||
### 8.2 The page
|
||||
|
||||
A new nav entry in `navigation` (`main.tsx:43-54`), between Wealth and Settings. Name open (§14).
|
||||
|
||||
Layout: a tile grid. Each tile shows title, chart, and a footer with row count, elapsed ms, the snapshot revision, and a disclosure that reveals the SQL. Tiles re-run when `revision` changes, exactly like `Overview`'s effect (`Overview.tsx:29-64`).
|
||||
|
||||
Composer: a question box and a SQL editor, side by side rather than staged, so the two paths are visibly the same pipeline. `Ask` fills the SQL box and shows `assumptions` above it; `Run` executes; `Pin` writes to `queries.finance`. Nothing auto-runs and nothing auto-pins.
|
||||
|
||||
Optional, once tiles exist: render the first few pinned tiles on `Overview`. That is the "tile" framing in its most useful form, and it costs one component reuse.
|
||||
|
||||
## 9. API
|
||||
|
||||
| Route | Body | Returns |
|
||||
| --- | --- | --- |
|
||||
| `POST /api/query` | `{sql, limit?}` | `{columns, rows, truncated, ms, revision}` |
|
||||
| `POST /api/ask` | `{question}` | `{sql, title, chart, x, y, assumptions, model, ms}` |
|
||||
| `POST /api/queries` | `{id?, title, question, sql, chart, position, delete?}` | `State` |
|
||||
|
||||
Registered in `New` (`server.go:40-74`), decoded with `decode` (`server.go:192-204`), answered with `respond` (`server.go:205-216`). All three are `POST`, so they pick up the existing `Sec-Fetch-Site`, `Origin` and `application/json` guards at `server.go:130-153`.
|
||||
|
||||
## 10. Phasing
|
||||
|
||||
1. **Sandbox, gate, `/api/query`, result table, `queries.finance` tiles.** No model. Independently useful, and it is where all the engineering risk lives.
|
||||
2. **Deterministic chart inference and the chart picker.**
|
||||
3. **Views (§4) and a schema-browser panel.** Makes hand-written SQL pleasant *and* is the prerequisite for step 4 being any good.
|
||||
4. **`internal/askql`, llama-server, `/api/ask`, assumptions, one repair.**
|
||||
|
||||
Doing 4 before 3 is the main way this disappoints.
|
||||
|
||||
## 11. Work breakdown
|
||||
|
||||
| File | Change |
|
||||
| --- | --- |
|
||||
| `internal/analytics/store.go` | add `SET lock_configuration = true` to `Open`; add views to `schema` and to the drop list at `:119`; create views after `postings` |
|
||||
| `internal/analytics/adhoc.go` (new) | `snapshot` lifecycle, `ATTACH … (READ_ONLY)`, lockdown, gate, wrap, run, result encoding incl. `duckdb.Decimal` |
|
||||
| `internal/analytics/query.go` | unchanged |
|
||||
| `internal/app/app.go` | invalidate the snapshot where `a.indexed = rev` (`:171`); `AdHoc` entry point that releases `a.mu` before running; `ask_base_url` / `ask_model` config keys and writer |
|
||||
| `internal/askql/` (new) | llama-server client, schema card, prompt, strict schema, one repair |
|
||||
| `internal/domain/model.go`, `domain.go` | `Query` struct, `Dataset.Queries`, validation |
|
||||
| `internal/journal/codec.go` | `queries.finance` in `registryFiles` |
|
||||
| `internal/server/server.go` | three routes and handlers |
|
||||
| `web/src/api.ts` | `QueryResult`, `Plan`, `Query` types |
|
||||
| `web/src/Queries.tsx` (new) | page, tile grid, composer, chart inference, table |
|
||||
| `web/src/main.tsx` | nav entry, route |
|
||||
| `web/src/styles.css` | tile grid, table, line-chart svg |
|
||||
| `nix/service.nix`, `README.md`, `OPERATIONS.txt` | llama-cpp unit, setup, operational notes |
|
||||
|
||||
## 12. Calibration
|
||||
|
||||
Published execution accuracy for general small coder models on realistic multi-table schemas: **~39 % at 7B, 47 % at 14B, 50 % at 32B** on BIRD ([cross-family size × technique frontier](https://arxiv.org/pdf/2606.29733)). SQL-specialised models do markedly better — [Arctic-Text2SQL-R1](https://www.snowflake.com/en/blog/engineering/arctic-text2sql-r1-sql-generation-benchmark/) 14B reaches 64.9 % BIRD-dev / 86.8 % Spider-test; [XiYanSQL-QwenCoder](https://github.com/XGenerationLab/XiYanSQL-QwenCoder)-32B reaches 69 % BIRD-test, with 3B/7B/14B variants.
|
||||
|
||||
Two consequences already built into this design: the schema must be small and denormalised (§4 turns a BIRD-hard join into a Spider-easy single-table query), and the SQL must always be visible and one click from editable. Plan for "wrong a third of the time, obviously wrong when it is".
|
||||
|
||||
Latency, inferred not measured: a 7B Q4_K_M emitting ~200 constrained JSON tokens on a 7840U-class CPU lands around 15–25 s end to end. Tolerable for Ask, annoying for iteration — another reason the raw SQL path must stand alone. An SQL-specialised 3B is worth benchmarking against your own questions before committing to 7B.
|
||||
|
||||
## 13. Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
| --- | --- |
|
||||
| Driver executes leading statements of a multi-statement string | read-only attachment (engine-enforced) **and** the parse-only gate; neither alone |
|
||||
| Ad-hoc query stalls the whole app | never hold `a.mu` during execution; separate pool; one query at a time |
|
||||
| Stale snapshot answers with pre-rebuild data | invalidate at `app.go:171`; every result carries `revision`; tiles re-run on change |
|
||||
| Spilling fills the data disk | disable `temp_directory` or cap `max_temp_directory_size` on the ad-hoc instance — **needs a probe**, `enable_external_access = false` does not cover spill |
|
||||
| Plausible but wrong SQL believed | `assumptions` shown, SQL shown, row count and revision in the footer, previous-period comparison for KPIs |
|
||||
| Pinned SQL breaks when views change | views are a versioned interface; broken tile shows the engine error, never a silent zero |
|
||||
| Registry names carry prompt injection | model output is only a read-only `SELECT` you approve before it runs |
|
||||
| `/api/query` is a new privilege class on a service with no auth | the sandbox *is* the mitigation; without §2 and §3 this endpoint is an arbitrary file read |
|
||||
| Model unavailable or slow | Ask degrades independently; query and tile surfaces have no AI dependency |
|
||||
|
||||
## 14. Open decisions
|
||||
|
||||
1. **Page name and placement** — `Ask`, `Lab`, `Query`? A new nav page, tiles embedded in `Overview`, or both?
|
||||
2. **Loopback-only for the ask model** — accept as a hard invariant, or is an OpenRouter fallback wanted (which forces redaction back in and, per §6.1, breaks the feature)?
|
||||
3. **Views as the documented query surface**, base tables explicitly unstable — accept?
|
||||
4. **`v_recurring` grouping** — payee alone with an `amount_variation` filter (survives price rises, as drafted), or payee × rounded amount (splits on a price rise, but separates two different subscriptions to the same payee)?
|
||||
5. **Where inference runs** and the RAM budget — decides 3B vs 7B vs 14B, and whether it shares the service host.
|
||||
6. **Stored-query validation timing** — gate SQL at write time (a view rename can make the journal unloadable) or only at run time (a broken tile, loadable journal)?
|
||||
7. Should `v_month` be a gap-free spine generated between `min` and `max`, and what should it do on an empty dataset?
|
||||
|
||||
## 15. Implementation notes
|
||||
|
||||
Only the parts where the obvious implementation is wrong.
|
||||
|
||||
### 15.1 Attach before locking down
|
||||
|
||||
```go
|
||||
// ATTACH is itself a filesystem operation, so external access must stay enabled
|
||||
// until the snapshot is attached. The path is interpolated because ATTACH takes
|
||||
// no parameters; it is our own path, never user input.
|
||||
for _, stmt := range []string{
|
||||
"ATTACH '" + path + "' AS fd (READ_ONLY)",
|
||||
"USE fd",
|
||||
"SET threads = 2",
|
||||
"SET memory_limit = '256MB'",
|
||||
"SET enable_external_access = false",
|
||||
"SET lock_configuration = true",
|
||||
} { … }
|
||||
```
|
||||
|
||||
Reversing the last two lines, or setting `enable_external_access = false` before the `ATTACH`, fails with `Permission Error: Cannot access file …`.
|
||||
|
||||
### 15.2 The gate must bind, not format
|
||||
|
||||
```go
|
||||
// The submitted text is a value, not code: json_serialize_sql parses it without
|
||||
// executing it, and a bound parameter leaves no injection surface. The cast is
|
||||
// required — an untyped parameter yields
|
||||
// "json_serialize_sql first argument must be a VARCHAR".
|
||||
row := db.QueryRowContext(ctx, "SELECT json_serialize_sql(CAST(? AS VARCHAR))", sql)
|
||||
```
|
||||
|
||||
The result scans as `map[string]any` through this driver, not as a string. `{"error": true, "error_type": …, "error_message": …}` on rejection; `{"statements": [{"node": {"type": "SELECT_NODE"}}]}` on acceptance.
|
||||
|
||||
### 15.3 Money must not become a float
|
||||
|
||||
```go
|
||||
// duckdb.Decimal.String() is exact; Float64() is not. domain.Money is a decimal
|
||||
// string for the same reason, and the dashboard already renders exact strings.
|
||||
case duckdb.Decimal:
|
||||
cell = v.String()
|
||||
```
|
||||
|
||||
### 15.4 Views join the drop list
|
||||
|
||||
`Rebuild` drops tables by name at `store.go:119`. DuckDB refuses to drop a table a view depends on, so a view left behind breaks the next rebuild — the failure surfaces as `a.indexError` and takes the whole dashboard down, not just the tiles. Drop views first, in dependency order, or drop with `CASCADE`.
|
||||
|
||||
### 15.5 The local request body is not the OpenRouter one
|
||||
|
||||
`classification.complete` sends `provider: {data_collection: "deny", zdr: true, require_parameters: true}` (`client.go:302`). Those keys are OpenRouter routing directives; a local server has no providers, and a strict OpenAI-compatible server may reject unknown fields. `askql` sends `model`, `messages`, `stream: false`, `max_tokens`, `response_format` and nothing else.
|
||||
|
||||
### 15.6 The config reader rejects unknown keys
|
||||
|
||||
`app.go:112` is a `switch` over known keys with a hard error on anything else, and the writer at `app.go:376-380` rewrites the whole file. Adding `ask_base_url` / `ask_model` to one side only makes an existing `config.toml` unreadable after the first save.
|
||||
Reference in New Issue
Block a user