Run classification previews in the background with live progress

The preview endpoint held one HTTP request open while classifying
serially at three-second pacing, so any real range meant minutes of a
grayed-out button and per-row errors were invisible until the loop
ended. Analyse now starts a single background run against its own
snapshot; a progress endpoint reports analysed counts, proposed
changes and errors as they happen, and the page polls it with a
progress bar, pace-based estimate and a Stop button. Navigating away
no longer orphans the run: the page re-attaches to it on return.

A run that has produced no successful proposal and fails three times
in a row with the identical error stops early and reports that error,
so a wrong key or unsupported model surfaces in seconds instead of
repeating across the whole paced range.

Also normalize a null settings.private_names, which crashed the whole
UI on a workspace that had never saved preferences.
This commit is contained in:
Lars Nolden
2026-09-12 12:24:50 +02:00
parent 635c11be56
commit 9092c5721d
10 changed files with 449 additions and 91 deletions
+8 -1
View File
@@ -673,7 +673,14 @@ canonical data and the index error is surfaced rather than serving stale totals.
Reclassification Reclassification
---------------- ----------------
AI / Classification: choose dates, model and independent Merchant/Category/Tags AI / Classification: choose dates, model and independent Merchant/Category/Tags
fields. Analyse produces a read-only preview. Apply all/selected writes all fields. Analyse starts a background run and reports live progress: analysed
count, proposed changes, and per-transaction errors as they happen. Requests
stay paced seconds apart, so a large range takes minutes; the page may be left
and revisited, and Stop abandons the run without writing anything. A run that
has produced no successful proposal and fails three times in a row with the
same error stops early and reports that error instead of repeating it across
the whole range. Only one run exists at a time.
The finished run is a read-only preview. Apply all/selected writes all
selected changes in one canonical commit; financial facts never change. A selected changes in one canonical commit; financial facts never change. A
manual edit, external journal change or taxonomy change invalidates old previews. manual edit, external journal change or taxonomy change invalidates old previews.
Previews are kept in memory for up to one hour and disappear on restart. Cancel Previews are kept in memory for up to one hour and disappear on restart. Cancel
+2
View File
@@ -417,6 +417,8 @@ Every AI classification requests `provider.data_collection = "deny"`, `provider.
Classification spaces request starts by at least **three seconds**, including successful requests, rather than sending a burst between 429s. This is a conservative application policy, not a published quota for every model. On HTTP 429, backoff starts at **15 seconds** and increases across consecutive failures; `Retry-After` seconds or HTTP dates can extend the wait. Successful retries retain the learned spacing (up to **30 seconds**) instead of immediately bursting again. Each operation makes at most **four attempts**, with at most **two minutes of automatic retry waiting**, preserving the same model, sanitized prompt, and privacy controls. Imports and previews share this pacing and cooldown. Long or exhausted limits leave records unclassified with a retry-time error; local merchant rules still work. After the cooldown, run **AI classification → Analyse** again for previously failed records—repeating a bank import does not reclassify existing transactions. Classification spaces request starts by at least **three seconds**, including successful requests, rather than sending a burst between 429s. This is a conservative application policy, not a published quota for every model. On HTTP 429, backoff starts at **15 seconds** and increases across consecutive failures; `Retry-After` seconds or HTTP dates can extend the wait. Successful retries retain the learned spacing (up to **30 seconds**) instead of immediately bursting again. Each operation makes at most **four attempts**, with at most **two minutes of automatic retry waiting**, preserving the same model, sanitized prompt, and privacy controls. Imports and previews share this pacing and cooldown. Long or exhausted limits leave records unclassified with a retry-time error; local merchant rules still work. After the cooldown, run **AI classification → Analyse** again for previously failed records—repeating a bank import does not reclassify existing transactions.
**AI classification → Analyse** runs in the background: the page shows how many transactions have been analysed, proposed changes, and every per-transaction failure as it happens, with a **Stop** button that abandons the run without writing anything. You can navigate away and return; the run keeps building and the page re-attaches to it. A run that has produced no successful result and fails **three times in a row with the same error** stops early and reports that error — a wrong key or an unsupported model surfaces within seconds instead of repeating across the whole range.
## Data, backups, and recovery ## Data, backups, and recovery
Back up the **entire canonical finance directory**, including registry files, journals, `config.toml` when present, and operational/recovery state, plus any separately stored environment-managed secrets. `state/openrouter.json` and `state/enablebanking.json` contain UI-managed credentials: protect backups accordingly, including the matching banking session state. Stop the service for a consistent filesystem backup. DuckDB under `cache/` can be excluded and rebuilt. Back up the **entire canonical finance directory**, including registry files, journals, `config.toml` when present, and operational/recovery state, plus any separately stored environment-managed secrets. `state/openrouter.json` and `state/enablebanking.json` contain UI-managed credentials: protect backups accordingly, including the matching banking session state. Stop the service for a consistent filesystem backup. DuckDB under `cache/` can be excluded and rebuilt.
+1
View File
@@ -71,6 +71,7 @@ type App struct {
bank banking.Provider bank banking.Provider
classifier classification.Client classifier classification.Client
previews map[string]Preview previews map[string]Preview
previewRun *previewJob
taxonomies map[string]TaxonomyPreview taxonomies map[string]TaxonomyPreview
csvImports map[string]CSVImport csvImports map[string]CSVImport
authStates map[string]authorization authStates map[string]authorization
+51 -10
View File
@@ -129,6 +129,32 @@ func TestFailedClassificationStillImportsAndRetryIsIdempotent(t *testing.T) {
} }
} }
// runPreview drives the background preview job to completion the way the UI
// does: start the run, then poll progress until it reports done.
func runPreview(t *testing.T, a *App, r PreviewRequest) (Preview, error) {
t.Helper()
start, err := a.StartPreview(context.Background(), r)
if err != nil {
return Preview{}, err
}
deadline := time.Now().Add(15 * time.Second)
for {
p, err := a.PreviewProgress(start.ID)
if err != nil {
return Preview{}, err
}
if p.Done {
if p.Error != "" {
return Preview{}, errors.New(p.Error)
}
return *p.Preview, nil
}
if time.Now().After(deadline) {
t.Fatal("preview run did not finish")
}
time.Sleep(5 * time.Millisecond)
}
}
func TestPreviewCooldownProtectsLaterPreviewsAndImports(t *testing.T) { func TestPreviewCooldownProtectsLaterPreviewsAndImports(t *testing.T) {
a, s := testApp(t) a, s := testApp(t)
s = seed(t, a, s) s = seed(t, a, s)
@@ -145,7 +171,7 @@ func TestPreviewCooldownProtectsLaterPreviewsAndImports(t *testing.T) {
defer cancel() defer cancel()
for _, model := range []string{"test/model", "test/another-model"} { for _, model := range []string{"test/model", "test/another-model"} {
preview, err := a.Preview(ctx, PreviewRequest{ preview, err := runPreview(t, a, PreviewRequest{
Revision: s.Revision, From: "2026-09-01", To: "2026-09-30", Revision: s.Revision, From: "2026-09-01", To: "2026-09-30",
Model: model, Fields: Fields{Category: true}, Model: model, Fields: Fields{Category: true},
}) })
@@ -185,24 +211,39 @@ func TestPreviewCooldownProtectsLaterPreviewsAndImports(t *testing.T) {
} }
} }
func TestCancelledLastClassificationDoesNotProducePreview(t *testing.T) { func TestCancelledPreviewRunProducesNoPreview(t *testing.T) {
a, s := testApp(t) a, s := testApp(t)
s = seed(t, a, s) s = seed(t, a, s)
ctx, cancel := context.WithCancel(context.Background()) ids := make(chan string, 1)
defer cancel()
provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Stop the run from within its first provider call, as the UI's Stop
// button would mid-request.
a.CancelPreview(<-ids)
w.Header().Set("Retry-After", "60") w.Header().Set("Retry-After", "60")
w.WriteHeader(http.StatusTooManyRequests) w.WriteHeader(http.StatusTooManyRequests)
cancel()
})) }))
defer provider.Close() defer provider.Close()
a.classifier = classification.Client{APIKey: "test", Model: "test/model", BaseURL: provider.URL} a.classifier = classification.Client{APIKey: "test", Model: "test/model", BaseURL: provider.URL}
p, err := a.Preview(ctx, PreviewRequest{ start, err := a.StartPreview(context.Background(), PreviewRequest{
Revision: s.Revision, From: "2026-09-09", To: "2026-09-09", Revision: s.Revision, From: "2026-09-09", To: "2026-09-09",
Model: "test/model", Fields: Fields{Category: true}, Model: "test/model", Fields: Fields{Category: true},
}) })
if !errors.Is(err, context.Canceled) || p.ID != "" { if err != nil {
t.Fatalf("cancelled final record produced a preview: id=%q, error=%v", p.ID, err) t.Fatal(err)
}
ids <- start.ID
deadline := time.Now().Add(10 * time.Second)
for {
if _, err := a.PreviewProgress(start.ID); err != nil {
break // the cancelled run is gone, never a finished preview
}
if time.Now().After(deadline) {
t.Fatal("cancelled preview run still reports progress")
}
time.Sleep(5 * time.Millisecond)
}
if _, err := a.ApplyPreview(context.Background(), start.ID, s.Revision, []string{"any"}); err == nil {
t.Fatal("cancelled run produced an applicable preview")
} }
after, err := a.Snapshot(context.Background()) after, err := a.Snapshot(context.Background())
if err != nil { if err != nil {
@@ -261,7 +302,7 @@ func TestPreviewIsReadOnlySelectedApplyPreservesFactsAndOtherFields(t *testing.T
} }
mockClassifier(t, a) mockClassifier(t, a)
before := domain.Clone(s.Data) before := domain.Clone(s.Data)
preview, err := a.Preview(context.Background(), PreviewRequest{Revision: s.Revision, From: "2026-09-01", To: "2026-09-30", Model: "improved/model", Fields: Fields{Category: true}}) preview, err := runPreview(t, a, PreviewRequest{Revision: s.Revision, From: "2026-09-01", To: "2026-09-30", Model: "improved/model", Fields: Fields{Category: true}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -306,7 +347,7 @@ func TestStalePreviewCannotOverwriteManualCorrection(t *testing.T) {
a, s := testApp(t) a, s := testApp(t)
s = seed(t, a, s) s = seed(t, a, s)
mockClassifier(t, a) mockClassifier(t, a)
p, err := a.Preview(context.Background(), PreviewRequest{Revision: s.Revision, From: "2026-09-01", To: "2026-09-30", Model: "test/model", Fields: Fields{Category: true}}) p, err := runPreview(t, a, PreviewRequest{Revision: s.Revision, From: "2026-09-01", To: "2026-09-30", Model: "test/model", Fields: Fields{Category: true}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+1 -1
View File
@@ -12,7 +12,7 @@ import (
func checkOpenRouterPreview(t *testing.T, a *App, s State, auth <-chan string, key string) { func checkOpenRouterPreview(t *testing.T, a *App, s State, auth <-chan string, key string) {
t.Helper() t.Helper()
p, err := a.Preview(context.Background(), PreviewRequest{Revision: s.Revision, From: "2026-09-01", To: "2026-09-30", Model: "test/model", Fields: Fields{Category: true}}) p, err := runPreview(t, a, PreviewRequest{Revision: s.Revision, From: "2026-09-01", To: "2026-09-30", Model: "test/model", Fields: Fields{Category: true}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+173 -25
View File
@@ -3,11 +3,13 @@ package app
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"reflect" "reflect"
"slices" "slices"
"strings" "strings"
"time" "time"
"finance-duck/internal/classification"
"finance-duck/internal/domain" "finance-duck/internal/domain"
) )
@@ -44,6 +46,33 @@ type Preview struct {
created time.Time created time.Time
} }
// PreviewProgress is the live state of one preview run. Errors accumulate as
// they happen so a failing provider is visible after seconds, not after the
// whole paced range. Preview is set only when Done with an empty Error.
type PreviewProgress struct {
ID string `json:"id"`
Total int `json:"total"`
Analysed int `json:"analysed"`
Changes int `json:"changes"`
Unchanged int `json:"unchanged"`
Errors []ClassificationError `json:"errors"`
Done bool `json:"done"`
Error string `json:"error,omitempty"`
Preview *Preview `json:"preview,omitempty"`
}
func (p PreviewProgress) clone() PreviewProgress {
p.Errors = append([]ClassificationError{}, p.Errors...)
return p
}
// previewJob is the single in-flight (or most recently finished) preview run.
// status is guarded by App.mu; cancel stops the goroutine cooperatively.
type previewJob struct {
cancel context.CancelFunc
status PreviewProgress
}
func validRange(from, to string) error { func validRange(from, to string) error {
f, e := time.Parse("2006-01-02", from) f, e := time.Parse("2006-01-02", from)
if e != nil { if e != nil {
@@ -58,49 +87,162 @@ func validRange(from, to string) error {
} }
return nil return nil
} }
func (a *App) Preview(ctx context.Context, r PreviewRequest) (Preview, error) { func validatePreviewRequest(r PreviewRequest) error {
if err := validRange(r.From, r.To); err != nil { if err := validRange(r.From, r.To); err != nil {
return Preview{}, err return err
} }
if !r.Fields.Merchant && !r.Fields.Category && !r.Fields.Tags { if !r.Fields.Merchant && !r.Fields.Category && !r.Fields.Tags {
return Preview{}, errors.New("select at least one enrichment field") return errors.New("select at least one enrichment field")
} }
if strings.TrimSpace(r.Model) == "" { if strings.TrimSpace(r.Model) == "" {
return Preview{}, errors.New("model is required") return errors.New("model is required")
}
return nil
}
func previewEligible(t domain.Transaction, r PreviewRequest) bool {
return t.Facts.BookingDate >= r.From && t.Facts.BookingDate <= r.To &&
t.Enrichment.Kind != "transfer" && t.Enrichment.Kind != domain.KindInvestment
}
// StartPreview validates the request against the current journal and starts a
// background classification run. The provider is paced to one request every
// few seconds, so any real range takes minutes: the caller polls
// PreviewProgress instead of holding an HTTP request open for the duration.
// Only one run exists at a time; the run owns its own snapshot and never
// touches canonical data.
func (a *App) StartPreview(ctx context.Context, r PreviewRequest) (PreviewProgress, error) {
if err := validatePreviewRequest(r); err != nil {
return PreviewProgress{}, err
} }
a.mu.Lock() a.mu.Lock()
defer a.mu.Unlock()
if a.previewRun != nil && !a.previewRun.status.Done {
return PreviewProgress{}, errors.New("a preview is already being generated; stop it first")
}
s, err := a.snapshot(ctx) s, err := a.snapshot(ctx)
client := a.classifier.WithModel(r.Model)
a.mu.Unlock()
if err != nil { if err != nil {
return Preview{}, err return PreviewProgress{}, err
} }
if r.Revision != s.Revision { if r.Revision != s.Revision {
return Preview{}, errors.New("revision conflict: reload before analysing") return PreviewProgress{}, errors.New("revision conflict: reload before analysing")
} }
p := Preview{ID: domain.NewID("preview"), Revision: s.Revision, Changes: []Change{}, Errors: []ClassificationError{}, created: time.Now()} client := a.classifier.WithModel(r.Model)
baseMerchants := len(s.Data.Merchants) total := 0
for _, t := range s.Data.Transactions { for _, t := range s.Data.Transactions {
if t.Facts.BookingDate < r.From || t.Facts.BookingDate > r.To || t.Enrichment.Kind == "transfer" || t.Enrichment.Kind == domain.KindInvestment { if previewEligible(t, r) {
total++
}
}
runCtx, cancel := context.WithCancel(context.Background())
job := &previewJob{cancel: cancel, status: PreviewProgress{ID: domain.NewID("preview"), Total: total, Errors: []ClassificationError{}}}
a.previewRun = job
go a.runPreview(runCtx, cancel, client, s, r, job)
return job.status.clone(), nil
}
func (a *App) runPreview(ctx context.Context, cancel context.CancelFunc, client *classification.Client, s State, r PreviewRequest, job *previewJob) {
defer cancel()
p, err := classifyRange(ctx, client, s, r, job.status.ID, func(u PreviewProgress) {
a.mu.Lock()
if a.previewRun == job {
job.status = u
}
a.mu.Unlock()
})
a.mu.Lock()
defer a.mu.Unlock()
if a.previewRun != job {
return // stopped by CancelPreview; discard the result
}
job.status.Done = true
if err != nil {
job.status.Error = err.Error()
return
}
for id, old := range a.previews {
if time.Since(old.created) > time.Hour {
delete(a.previews, id)
}
}
if len(a.previews) >= 20 {
job.status.Error = "too many active previews; cancel one first"
return
}
a.previews[p.ID] = p
job.status.Analysed = p.Analysed
job.status.Changes = len(p.Changes)
job.status.Unchanged = p.Unchanged
job.status.Errors = append([]ClassificationError{}, p.Errors...)
job.status.Preview = &p
}
// PreviewProgress reports the current (or most recently finished) preview run.
// An empty id re-attaches to whatever run exists, so navigating away from the
// page does not orphan a run that is still spending provider requests.
func (a *App) PreviewProgress(id string) (PreviewProgress, error) {
a.mu.Lock()
defer a.mu.Unlock()
job := a.previewRun
if job == nil || (id != "" && job.status.ID != id) {
return PreviewProgress{}, errors.New("no matching preview run; analyse again")
}
return job.status.clone(), nil
}
// classifyRange proposes enrichment for every eligible transaction in the
// snapshot, reporting progress after each one. It stops early when the run has
// produced no successful proposal yet and the same error message repeats three
// times in a row: an identical repeated failure is a configuration or provider
// problem, and grinding through the rest of the paced range would only repeat
// it a few seconds apart.
func classifyRange(ctx context.Context, client *classification.Client, s State, r PreviewRequest, id string, report func(PreviewProgress)) (Preview, error) {
p := Preview{ID: id, Revision: s.Revision, Changes: []Change{}, Errors: []ClassificationError{}, created: time.Now()}
baseMerchants := len(s.Data.Merchants)
total := 0
for _, t := range s.Data.Transactions {
if previewEligible(t, r) {
total++
}
}
progress := func() {
if report != nil {
report(PreviewProgress{ID: id, Total: total, Analysed: p.Analysed, Changes: len(p.Changes), Unchanged: p.Unchanged, Errors: append([]ClassificationError{}, p.Errors...)})
}
}
succeeded := false
repeated := 0
for _, t := range s.Data.Transactions {
if !previewEligible(t, r) {
continue continue
} }
if err = ctx.Err(); err != nil { if err := ctx.Err(); err != nil {
return Preview{}, err return Preview{}, err
} }
p.Analysed++ p.Analysed++
proposal, e := client.Classify(ctx, t.Facts, s.Data, true) proposal, e := client.Classify(ctx, t.Facts, s.Data, true)
if err = ctx.Err(); err != nil { if err := ctx.Err(); err != nil {
return Preview{}, err return Preview{}, err
} }
if e != nil { if e != nil {
if n := len(p.Errors); n > 0 && p.Errors[n-1].Error == e.Error() {
repeated++
} else {
repeated = 1
}
p.Errors = append(p.Errors, ClassificationError{t.Facts.ID, e.Error()}) p.Errors = append(p.Errors, ClassificationError{t.Facts.ID, e.Error()})
if !succeeded && repeated >= 3 {
return Preview{}, fmt.Errorf("stopped after %d identical failures — %s — with %d of %d transactions not analysed", repeated, e.Error(), total-p.Analysed, total)
}
progress()
continue continue
} }
succeeded = true
after := t.Enrichment after := t.Enrichment
if r.Fields.Merchant { if r.Fields.Merchant {
after.MerchantID = proposal.Enrichment.MerchantID after.MerchantID = proposal.Enrichment.MerchantID
if e = addProposal(&s.Data, proposal, t.Facts); e != nil { if e = addProposal(&s.Data, proposal, t.Facts); e != nil {
p.Errors = append(p.Errors, ClassificationError{t.Facts.ID, e.Error()}) p.Errors = append(p.Errors, ClassificationError{t.Facts.ID, e.Error()})
progress()
continue continue
} }
} }
@@ -112,6 +254,7 @@ func (a *App) Preview(ctx context.Context, r PreviewRequest) (Preview, error) {
} }
if e = domain.ValidateEnrichment(s.Data, t.Facts, after); e != nil { if e = domain.ValidateEnrichment(s.Data, t.Facts, after); e != nil {
p.Errors = append(p.Errors, ClassificationError{t.Facts.ID, e.Error()}) p.Errors = append(p.Errors, ClassificationError{t.Facts.ID, e.Error()})
progress()
continue continue
} }
beforeComparable, afterComparable := t.Enrichment, after beforeComparable, afterComparable := t.Enrichment, after
@@ -123,23 +266,14 @@ func (a *App) Preview(ctx context.Context, r PreviewRequest) (Preview, error) {
slices.Sort(afterComparable.TagIDs) slices.Sort(afterComparable.TagIDs)
if reflect.DeepEqual(beforeComparable, afterComparable) { if reflect.DeepEqual(beforeComparable, afterComparable) {
p.Unchanged++ p.Unchanged++
progress()
continue continue
} }
after.Classification = proposal.Enrichment.Classification after.Classification = proposal.Enrichment.Classification
p.Changes = append(p.Changes, Change{t.Facts.ID, t.Facts.RawDescription, t.Enrichment, after}) p.Changes = append(p.Changes, Change{t.Facts.ID, t.Facts.RawDescription, t.Enrichment, after})
progress()
} }
p.NewMerchants = append([]domain.Merchant{}, s.Data.Merchants[baseMerchants:]...) p.NewMerchants = append([]domain.Merchant{}, s.Data.Merchants[baseMerchants:]...)
a.mu.Lock()
defer a.mu.Unlock()
for id, old := range a.previews {
if time.Since(old.created) > time.Hour {
delete(a.previews, id)
}
}
if len(a.previews) >= 20 {
return Preview{}, errors.New("too many active previews; cancel one first")
}
a.previews[p.ID] = p
return p, nil return p, nil
} }
func (a *App) ApplyPreview(ctx context.Context, id, rev string, ids []string) (State, error) { func (a *App) ApplyPreview(ctx context.Context, id, rev string, ids []string) (State, error) {
@@ -198,6 +332,20 @@ func (a *App) ApplyPreview(ctx context.Context, id, rev string, ids []string) (S
return State{}, err return State{}, err
} }
delete(a.previews, id) delete(a.previews, id)
if job := a.previewRun; job != nil && job.status.ID == id {
a.previewRun = nil
}
return state, nil return state, nil
} }
func (a *App) CancelPreview(id string) { a.mu.Lock(); defer a.mu.Unlock(); delete(a.previews, id) }
// CancelPreview stops a running preview job and discards a finished preview.
// A run and its stored preview share one id, so a single cancel covers both.
func (a *App) CancelPreview(id string) {
a.mu.Lock()
defer a.mu.Unlock()
if job := a.previewRun; job != nil && job.status.ID == id {
job.cancel()
a.previewRun = nil
}
delete(a.previews, id)
}
+12 -1
View File
@@ -68,6 +68,7 @@ func New(a *app.App, assets fs.FS, publicURL string) (http.Handler, error) {
respond(w, v, e) respond(w, v, e)
}) })
s.mux.HandleFunc("POST /api/reclassify/preview", s.preview) s.mux.HandleFunc("POST /api/reclassify/preview", s.preview)
s.mux.HandleFunc("POST /api/reclassify/progress", s.previewProgress)
s.mux.HandleFunc("POST /api/reclassify/apply", s.apply) s.mux.HandleFunc("POST /api/reclassify/apply", s.apply)
s.mux.HandleFunc("POST /api/reclassify/cancel", s.cancel) s.mux.HandleFunc("POST /api/reclassify/cancel", s.cancel)
s.mux.HandleFunc("POST /api/taxonomy/propose", s.taxonomyPropose) s.mux.HandleFunc("POST /api/taxonomy/propose", s.taxonomyPropose)
@@ -490,7 +491,17 @@ func (s *Server) preview(w http.ResponseWriter, r *http.Request) {
if !decode(w, r, &b) { if !decode(w, r, &b) {
return return
} }
v, e := s.app.Preview(r.Context(), b) v, e := s.app.StartPreview(r.Context(), b)
respond(w, v, e)
}
func (s *Server) previewProgress(w http.ResponseWriter, r *http.Request) {
var b struct {
ID string `json:"id"`
}
if !decode(w, r, &b) {
return
}
v, e := s.app.PreviewProgress(b.ID)
respond(w, v, e) respond(w, v, e)
} }
func (s *Server) apply(w http.ResponseWriter, r *http.Request) { func (s *Server) apply(w http.ResponseWriter, r *http.Request) {
+174 -53
View File
@@ -1,6 +1,12 @@
import { useState } from "react"; import { useEffect, useRef, useState } from "react";
import { Sparkles, ShieldCheck, Check, X, ArrowRight } from "lucide-react"; import { Sparkles, ShieldCheck, Check, X, ArrowRight } from "lucide-react";
import type { Dataset, Enrichment, Preview, State } from "./api"; import type {
Dataset,
Enrichment,
Preview,
PreviewProgress,
State,
} from "./api";
import { categoryPath, request } from "./api"; import { categoryPath, request } from "./api";
import { DateField, Empty, ErrorMessage, Field, Modal } from "./ui"; import { DateField, Empty, ErrorMessage, Field, Modal } from "./ui";
export function Classification({ export function Classification({
@@ -24,17 +30,96 @@ export function Classification({
const [busy, setBusy] = useState(false); const [busy, setBusy] = useState(false);
const [error, setError] = useState(""); const [error, setError] = useState("");
const [confirm, setConfirm] = useState(false); const [confirm, setConfirm] = useState(false);
const cancel = async () => { const [running, setRunning] = useState<PreviewProgress | null>(null);
if (!preview) return; const runStart = useRef({ time: 0, analysed: 0 });
const finalize = (result: Preview) => {
result.changes ??= [];
result.errors ??= [];
result.new_merchants ??= [];
for (const change of result.changes) {
change.before.tag_ids ??= [];
change.after.tag_ids ??= [];
}
const confidenceRank: Record<string, number> = {
low: 0,
medium: 1,
high: 2,
};
result.changes.sort(
(a, b) =>
(confidenceRank[a.after.classification.confidence || "low"] ?? 0) -
(confidenceRank[b.after.classification.confidence || "low"] ?? 0),
);
setPreview(result);
setSelected(
result.changes
.filter((change) => change.after.classification.confidence !== "low")
.map((change) => change.id),
);
};
// A run keeps going on the server while this page is closed; re-attach to
// it on mount instead of presenting a fresh, contradictory setup form.
useEffect(() => {
let stale = false;
(async () => {
try {
const p = await request<PreviewProgress>("/api/reclassify/progress", {
id: "",
});
if (stale) return;
if (!p.done) {
runStart.current = { time: Date.now(), analysed: p.analysed };
setRunning(p);
} else if (
!p.error &&
p.preview &&
p.preview.revision === state.revision
) {
finalize(p.preview);
}
} catch {
// No run to re-attach to.
}
})();
return () => {
stale = true;
};
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
useEffect(() => {
if (!running || running.done) return;
const timer = setTimeout(async () => {
try {
const p = await request<PreviewProgress>("/api/reclassify/progress", {
id: running.id,
});
p.errors ??= [];
if (!p.done) {
setRunning(p);
return;
}
setRunning(null);
if (p.error) setError(p.error);
else if (p.preview) finalize(p.preview);
} catch (err) {
setRunning(null);
setError(err instanceof Error ? err.message : String(err));
}
}, 1200);
return () => clearTimeout(timer);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [running]);
const cancel = async (id: string) => {
setBusy(true); setBusy(true);
setError(""); setError("");
try { try {
const response = await request<{ ok: boolean }>( const response = await request<{ ok: boolean }>(
"/api/reclassify/cancel", "/api/reclassify/cancel",
{ id: preview.id }, { id },
); );
if (!response.ok) if (!response.ok)
throw new Error("The server did not confirm cancellation."); throw new Error("The server did not confirm cancellation.");
setRunning(null);
setPreview(null); setPreview(null);
setSelected([]); setSelected([]);
} catch (err) { } catch (err) {
@@ -77,7 +162,65 @@ export function Classification({
</p> </p>
</div> </div>
</div> </div>
{!preview ? ( {running ? (
<section className="panel">
<div className="panel-heading">
<div>
<h3>Classifying transactions</h3>
<p>
{running.analysed} of {running.total} analysed ·{" "}
{running.changes} proposed changes · {running.unchanged}{" "}
unchanged · {running.errors.length} errors
</p>
</div>
</div>
<div className="form-body">
<div
className="progress-track"
role="progressbar"
aria-valuemin={0}
aria-valuemax={running.total}
aria-valuenow={running.analysed}
>
<div
className="progress-fill"
style={{
width: running.total
? `${Math.round((running.analysed / running.total) * 100)}%`
: "100%",
}}
/>
</div>
<p role="status" className="muted">
Provider requests are spaced several seconds apart to respect rate
limits
{remainingEstimate(running, runStart.current)}. You can leave this
page; the preview keeps building and will be here when you return.
</p>
{running.errors.length > 0 && (
<div className="alert error">
<div>
<strong>
{running.errors.length} transaction
{running.errors.length === 1 ? "" : "s"} failed so far
</strong>
<p>{running.errors[running.errors.length - 1].error}</p>
</div>
</div>
)}
<div className="form-actions">
<button
className="button secondary"
disabled={busy}
onClick={() => cancel(running.id)}
>
<X size={16} />
Stop
</button>
</div>
</div>
</section>
) : !preview ? (
<section className="panel classification-setup"> <section className="panel classification-setup">
<div className="panel-heading"> <div className="panel-heading">
<div> <div>
@@ -101,7 +244,7 @@ export function Classification({
setBusy(true); setBusy(true);
setError(""); setError("");
try { try {
const result = await request<Preview>( const start = await request<PreviewProgress>(
"/api/reclassify/preview", "/api/reclassify/preview",
{ {
revision: state.revision, revision: state.revision,
@@ -111,46 +254,13 @@ export function Classification({
fields, fields,
}, },
); );
if ( if (!start.id)
!result.id ||
!result.revision ||
!("changes" in result) ||
!("errors" in result) ||
!("new_merchants" in result)
)
throw new Error( throw new Error(
"The server returned an incompatible preview.", "The server returned an incompatible preview run.",
); );
result.changes ??= []; start.errors ??= [];
result.errors ??= []; runStart.current = { time: Date.now(), analysed: 0 };
result.new_merchants ??= []; setRunning(start);
for (const change of result.changes) {
change.before.tag_ids ??= [];
change.after.tag_ids ??= [];
}
const confidenceRank: Record<string, number> = {
low: 0,
medium: 1,
high: 2,
};
result.changes.sort(
(a, b) =>
(confidenceRank[
a.after.classification.confidence || "low"
] ?? 0) -
(confidenceRank[
b.after.classification.confidence || "low"
] ?? 0),
);
setPreview(result);
setSelected(
result.changes
.filter(
(change) =>
change.after.classification.confidence !== "low",
)
.map((change) => change.id),
);
} catch (err) { } catch (err) {
setError(err instanceof Error ? err.message : String(err)); setError(err instanceof Error ? err.message : String(err));
} finally { } finally {
@@ -223,14 +333,8 @@ export function Classification({
} }
> >
<Sparkles size={17} /> <Sparkles size={17} />
{busy ? "Classifying transactions…" : "Generate preview"} {busy ? "Starting…" : "Generate preview"}
</button> </button>
{busy && (
<p role="status" className="muted">
This can take a while for a large date range. Keep this page
open.
</p>
)}
{!state.data.transactions.length && ( {!state.data.transactions.length && (
<p className="muted"> <p className="muted">
Import transactions from Accounts before generating a preview. Import transactions from Accounts before generating a preview.
@@ -338,7 +442,7 @@ export function Classification({
<button <button
className="button secondary" className="button secondary"
disabled={busy} disabled={busy}
onClick={cancel} onClick={() => cancel(preview.id)}
> >
<X size={16} /> <X size={16} />
{busy ? "Working…" : "Cancel preview"} {busy ? "Working…" : "Cancel preview"}
@@ -474,3 +578,20 @@ function EnrichmentView({
</div> </div>
); );
} }
// remainingEstimate projects the finish time from the pace observed since
// this page attached to the run; the server paces provider requests, so the
// first sample is meaningless and re-attaching mid-run must not count work
// done before it.
function remainingEstimate(
p: PreviewProgress,
start: { time: number; analysed: number },
): string {
const sampled = p.analysed - start.analysed;
const remaining = p.total - p.analysed;
if (remaining <= 0 || sampled < 2 || !start.time) return "";
const seconds = Math.round(
((Date.now() - start.time) / sampled / 1000) * remaining,
);
if (seconds < 90) return ` — roughly ${seconds} seconds remaining`;
return ` — roughly ${Math.round(seconds / 60)} minutes remaining`;
}
+15
View File
@@ -200,6 +200,20 @@ export interface Preview {
unchanged: number; unchanged: number;
errors: { id: string; error: string }[]; errors: { id: string; error: string }[];
} }
// PreviewProgress is the live state of a background classification run.
// Errors accumulate as they happen; preview is present only when done
// without a fatal error.
export interface PreviewProgress {
id: string;
total: number;
analysed: number;
changes: number;
unchanged: number;
errors: { id: string; error: string }[];
done: boolean;
error?: string;
preview?: Preview;
}
export interface ProposedCategory { export interface ProposedCategory {
name: string; name: string;
parent?: string; parent?: string;
@@ -395,6 +409,7 @@ export function normalizeState(state: State): State {
else if (!Array.isArray(state.data[key])) else if (!Array.isArray(state.data[key]))
throw new Error(`The server state has invalid ${key}.`); throw new Error(`The server state has invalid ${key}.`);
} }
state.settings.private_names ??= [];
for (const tx of state.data.transactions) tx.enrichment.tag_ids ??= []; for (const tx of state.data.transactions) tx.enrichment.tag_ids ??= [];
for (const merchant of state.data.merchants) { for (const merchant of state.data.merchants) {
merchant.aliases ??= []; merchant.aliases ??= [];
+12
View File
@@ -1389,6 +1389,18 @@ summary .badge {
transform: rotate(360deg); transform: rotate(360deg);
} }
} }
.progress-track {
height: 8px;
border-radius: 4px;
background: #e1e9e5;
overflow: hidden;
}
.progress-fill {
height: 100%;
border-radius: 4px;
background: var(--emerald);
transition: width 0.6s ease;
}
footer { footer {
display: flex; display: flex;
justify-content: space-between; justify-content: space-between;