Allow the rounding a broker's own printed figures propagate
A real Scalable export refused to import at record 148: "buy gross -808.5599 does not equal quantity 6 times price 134.76, which is -808.56". Six NVIDIA shares settled at 808.5599 against a printed price of 134.76, because the fill was 134.759983 and the export printed the price to two places. One ten-thousandth out, and the whole file was rejected. The check held a gross to its own stated precision, which is only half the story: the price is rounded too, and the file never says by how much. So the allowance is now half a unit of the gross's stated precision plus one part in a hundred thousand of the gross, compared against a product kept exact at 1e-16 rather than rounded first. Measured over the complete export - 88 security rows - exactly one deviates at all, by one part in eight million, eighty times inside the new bound. What the bound still refuses is unchanged in kind: a price taken from the wrong share class, and the misplaced decimal separator the check exists for, which misses by four orders of magnitude. What it now accepts is the broker's own rounding, including a whole cent once a gross stated to the cent passes about five hundred euro, where a genuine one-cent error cannot be told from that rounding anyway. The row is kept as a regression test alongside four grosses that must still be refused: a cent, a euro, a wrong instrument's price, and a factor of ten.
This commit is contained in:
+17
-8
@@ -443,17 +443,26 @@ The share column is signed only for corporate actions and depot transfers. Buys
|
|||||||
and sells are unsigned and take their direction from the type. Both conventions
|
and sells are unsigned and take their direction from the type. Both conventions
|
||||||
are resolved at import, once.
|
are resolved at import, once.
|
||||||
|
|
||||||
Every security row is checked against shares times price, to the precision the
|
Every security row is checked against shares times price, allowing for the
|
||||||
export stated the amount at and no further. One export prints the exact product
|
rounding the export's own printed figures propagate. Both ends are rounded and
|
||||||
to nine places, and the check is then exact. Another prints the notional rounded
|
neither states by how much: one export prints the notional to the cent, so
|
||||||
to cents, where demanding exactness rejects every trade whose product does not
|
0,426581 shares at 63,06 settle as 26,90 where the product is 26,90019786;
|
||||||
land on a whole cent - measured on a real export, 29 of 59 of them. One unit of
|
another prints a price to fewer places than the fill actually had, settling six
|
||||||
the stated precision is still four orders of magnitude tighter than the
|
NVIDIA shares at 808,5599 against a printed 134,76 whose product is 808,56.
|
||||||
misplaced decimal separator this check exists to catch.
|
The allowance is half a unit of the gross's stated precision plus one part in a
|
||||||
|
hundred thousand of the gross. Measured over a complete real export of 88
|
||||||
|
security rows, exactly one deviates at all, by one part in eight million.
|
||||||
|
|
||||||
|
What that still refuses: a price taken from the wrong share class, and the lost
|
||||||
|
decimal separator the check exists for, four orders of magnitude out. What it
|
||||||
|
accepts: the broker's own rounding, including a whole cent once a gross stated
|
||||||
|
to the cent passes about five hundred euro, where a real one-cent error cannot
|
||||||
|
be told from that rounding.
|
||||||
|
|
||||||
It cannot catch a separator lost uniformly across a row: 1 x 25,795 and
|
It cannot catch a separator lost uniformly across a row: 1 x 25,795 and
|
||||||
1 x 25795 both satisfy it. A price cross-check against an outside provider is
|
1 x 25795 both satisfy it. A price cross-check against an outside provider is
|
||||||
the only remedy and is deliberately not implemented.
|
the only remedy and is deliberately not implemented. A spreadsheet round-trip
|
||||||
|
is what strips those separators, so import the broker's original file.
|
||||||
|
|
||||||
Rejected whole, with the record number: an unknown status, an unknown type, a
|
Rejected whole, with the record number: an unknown status, an unknown type, a
|
||||||
classifying column that disagrees with its type, an account type other than the
|
classifying column that disagrees with its type, an account type other than the
|
||||||
|
|||||||
@@ -202,7 +202,7 @@ Because a cash row's amount already includes the tax the broker withheld or refu
|
|||||||
|
|
||||||
Two more traps there: a `DIVIDEND` row fills the share column with **the holding the dividend was paid on**, so adding it would double the position; and crypto carries a bare ticker like `DOGE` in `symbol`, with its real identifier only in the description. Both are handled, and a position row that resolves to neither is refused.
|
Two more traps there: a `DIVIDEND` row fills the share column with **the holding the dividend was paid on**, so adding it would double the position; and crypto carries a bare ticker like `DOGE` in `symbol`, with its real identifier only in the description. Both are handled, and a position row that resolves to neither is refused.
|
||||||
|
|
||||||
Only `Executed` rows import from Scalable: a cancelled retry is all zeros, so it passes every arithmetic check and would otherwise become a phantom trade. Every security row is verified against shares × price **to the precision the broker stated the amount at** — exactly, where the export prints the full product; to within a cent, where it prints the notional rounded. An unknown row type, a mismatched classifying column, a foreign settlement currency, an unresolvable security, or a failed check rejects the **whole file** with the record number, because each of those can move money that never moved.
|
Only `Executed` rows import from Scalable: a cancelled retry is all zeros, so it passes every arithmetic check and would otherwise become a phantom trade. Every security row is verified against shares × price, **allowing for the rounding the export's own figures propagate** — both the gross and the price are printed rounded, and neither says by how much. Across a complete real export of 88 security rows exactly one deviates at all, by one part in eight million; a misplaced decimal separator is four orders of magnitude outside the allowance. An unknown row type, a mismatched classifying column, a foreign settlement currency, an unresolvable security, or a failed check rejects the **whole file** with the record number, because each of those can move money that never moved.
|
||||||
|
|
||||||
Securities are registered by **ISIN** in **Instruments**. The ISIN is the identity; the name is editable display text, because one ISIN appears under several broker names over the years. Crypto is held under the ISIN-shaped identifier the broker issues for it. Set the account's **settlement IBAN** for an export that names no counterparty of its own, so deposits from your bank pair with the funding account instead of staying unpaired. They never become income either way — a broker record is excluded from spending and income analytics, from bulk reclassification, and from the AI entirely.
|
Securities are registered by **ISIN** in **Instruments**. The ISIN is the identity; the name is editable display text, because one ISIN appears under several broker names over the years. Crypto is held under the ISIN-shaped identifier the broker issues for it. Set the account's **settlement IBAN** for an export that names no counterparty of its own, so deposits from your bank pair with the funding account instead of staying unpaired. They never become income either way — a broker record is excluded from spending and income analytics, from bulk reclassification, and from the AI entirely.
|
||||||
|
|
||||||
|
|||||||
@@ -228,6 +228,38 @@ func TestSingleShareRowCatchesOnlyInconsistentArithmetic(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A broker's own gross can disagree with its own printed shares times price,
|
||||||
|
// because the price is printed to fewer places than the fill actually had.
|
||||||
|
// Six NVIDIA shares settled at 808.5599 against a printed 134.76, whose
|
||||||
|
// product is 808.56: one ten-thousandth out, and the whole file was refused.
|
||||||
|
// The rounding the printed figures propagate is allowed; anything above one
|
||||||
|
// part in a hundred thousand still is not.
|
||||||
|
func TestRoundedPriceDoesNotRejectTheBrokersOwnGross(t *testing.T) {
|
||||||
|
const row = `2025-01-09;10:37:32;Executed;SCALixkS3TomjQv;NVIDIA;Security;Buy;US67066G1040;6;134,76;-808,5599;0,00;0,00;EUR`
|
||||||
|
result := readBroker(t, row)
|
||||||
|
inv := result.Facts[0].Investment
|
||||||
|
if inv.Gross != "-808.5599" || inv.Price != "134.76" || inv.Quantity != "6" {
|
||||||
|
t.Fatalf("trade read as %+v", inv)
|
||||||
|
}
|
||||||
|
if got := result.Facts[0].Amount; got != "-808.5599" {
|
||||||
|
t.Errorf("settled %s, want -808.5599", got)
|
||||||
|
}
|
||||||
|
for name, gross := range map[string]string{
|
||||||
|
"one cent out": "-808,5699",
|
||||||
|
"factor of ten": "-8.085,599",
|
||||||
|
"a euro out": "-809,5599",
|
||||||
|
"wrong instrument": "-908,5599",
|
||||||
|
} {
|
||||||
|
file, err := ReadCSV(strings.NewReader(scalableHeader + strings.Replace(row, ";-808,5599;", ";"+gross+";", 1) + "\n"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: %v", name, err)
|
||||||
|
}
|
||||||
|
if _, err := ParseScalableCSV(file, brokerAccount(), nil); err == nil {
|
||||||
|
t.Errorf("%s: accepted a gross its own shares times price does not support", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// A reinvested distribution settles shares times price, so it carries as many
|
// A reinvested distribution settles shares times price, so it carries as many
|
||||||
// decimal places as the two together need. A real export reinvests to nine,
|
// decimal places as the two together need. A real export reinvests to nine,
|
||||||
// which is past what money holds and past what a share count holds, so reading
|
// which is past what money holds and past what a share count holds, so reading
|
||||||
|
|||||||
+44
-22
@@ -639,7 +639,14 @@ func validateInvestment(f Facts, a Account, instruments map[string]Instrument) e
|
|||||||
return fmt.Errorf("%s requires a nonzero quantity", inv.Event)
|
return fmt.Errorf("%s requires a nonzero quantity", inv.Event)
|
||||||
}
|
}
|
||||||
// A position-only valuation carries the sign of the position change; a
|
// A position-only valuation carries the sign of the position change; a
|
||||||
// settled trade carries the sign of the cash, which is the opposite.
|
// settled trade carries the sign of the cash, which is the opposite. The
|
||||||
|
// product is kept exact at 1e-16 so the comparison never rounds first.
|
||||||
|
product := new(big.Int).Mul(big.NewInt(quantity), big.NewInt(price))
|
||||||
|
if inv.Settling() {
|
||||||
|
product.Neg(product)
|
||||||
|
}
|
||||||
|
difference := new(big.Int).Sub(product, new(big.Int).Mul(big.NewInt(gross), productPerMoney))
|
||||||
|
if difference.Abs(difference).Cmp(grossSlack(gross, inv.Gross)) > 0 {
|
||||||
expected, ok := RoundedProduct(quantity, price)
|
expected, ok := RoundedProduct(quantity, price)
|
||||||
if !ok {
|
if !ok {
|
||||||
return fmt.Errorf("%s quantity times price is out of range", inv.Event)
|
return fmt.Errorf("%s quantity times price is out of range", inv.Event)
|
||||||
@@ -647,18 +654,6 @@ func validateInvestment(f Facts, a Account, instruments map[string]Instrument) e
|
|||||||
if inv.Settling() {
|
if inv.Settling() {
|
||||||
expected = -expected
|
expected = -expected
|
||||||
}
|
}
|
||||||
// The gross is checked to the precision the broker stated it at, and no
|
|
||||||
// further. One broker prints the exact product to nine places, and the
|
|
||||||
// check is then exact. Another prints the notional rounded to cents, where
|
|
||||||
// demanding exactness rejects every trade whose product does not land on a
|
|
||||||
// whole cent - measured on a real export, 29 of 59 of them. One unit of
|
|
||||||
// the stated precision is still four orders of magnitude tighter than the
|
|
||||||
// misplaced decimal separator this check exists to catch.
|
|
||||||
difference := expected - gross
|
|
||||||
if difference < 0 {
|
|
||||||
difference = -difference
|
|
||||||
}
|
|
||||||
if difference >= statedUnit(inv.Gross) {
|
|
||||||
return fmt.Errorf("%s gross %s does not equal quantity %s times price %s, which is %s", inv.Event, inv.Gross.String(), inv.Quantity.String(), inv.Price.String(), Money(formatScaled(expected, moneyScale, 2)))
|
return fmt.Errorf("%s gross %s does not equal quantity %s times price %s, which is %s", inv.Event, inv.Gross.String(), inv.Quantity.String(), inv.Price.String(), Money(formatScaled(expected, moneyScale, 2)))
|
||||||
}
|
}
|
||||||
if inv.PositionOnly() {
|
if inv.PositionOnly() {
|
||||||
@@ -689,14 +684,41 @@ func settles(inv *Investment, gross, fee, tax, amount int64) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// statedUnit is one unit of the last decimal place a money figure was written
|
// productPerMoney converts money's ten-thousandths to the 1e-16 units a
|
||||||
// with, in exact ten-thousandths. Money always renders at least two places, so
|
// quantity times a price lands in.
|
||||||
// a whole-euro figure counts as stated to the cent.
|
var productPerMoney = new(big.Int).Exp(big.NewInt(10), big.NewInt(productScale-moneyScale), nil)
|
||||||
func statedUnit(m Money) int64 {
|
|
||||||
_, fraction, _ := strings.Cut(string(m), ".")
|
const productScale = quantityScale * 2
|
||||||
unit := int64(1)
|
|
||||||
for range moneyScale - len(strings.TrimRight(fraction, "0")) {
|
// grossSlack is how far a printed gross may sit from the product of the printed
|
||||||
unit *= 10
|
// quantity and price before the row is refused. Both ends are rounded, and
|
||||||
|
// neither states by how much.
|
||||||
|
//
|
||||||
|
// The gross is rounded to its own last decimal place: one broker prints the
|
||||||
|
// notional to the cent, so 0.426581 shares at 63.06 settle as 26.90 where the
|
||||||
|
// product is 26.90019786, and demanding exactness there rejects half a
|
||||||
|
// portfolio. The price is rounded to a precision the file does not state: the
|
||||||
|
// same export settles six NVIDIA shares at 808.5599 while printing the price
|
||||||
|
// as 134.76, whose product is 808.56, because the real fill was 134.759983.
|
||||||
|
// So the slack is half a unit of the gross's stated precision, plus one part
|
||||||
|
// in a hundred thousand of the gross itself.
|
||||||
|
//
|
||||||
|
// Measured over a complete real export of 88 security rows, exactly one
|
||||||
|
// deviates at all, by one part in eight million - eighty times inside this
|
||||||
|
// bound. What it refuses: any deviation above one part in a hundred thousand,
|
||||||
|
// which covers a price taken from the wrong share class and the lost decimal
|
||||||
|
// separator this check exists for, four orders of magnitude out. What it
|
||||||
|
// accepts: the broker's own rounding. On a gross stated to the cent the slack
|
||||||
|
// reaches a whole cent at around five hundred euro, above which a genuine
|
||||||
|
// one-cent error is indistinguishable from that rounding and is allowed.
|
||||||
|
func grossSlack(gross int64, printed Money) *big.Int {
|
||||||
|
_, fraction, _ := strings.Cut(string(printed), ".")
|
||||||
|
places := len(fraction)
|
||||||
|
if places > moneyScale {
|
||||||
|
places = moneyScale
|
||||||
}
|
}
|
||||||
return unit
|
half := new(big.Int).Exp(big.NewInt(10), big.NewInt(int64(productScale-places)), nil)
|
||||||
|
half.Quo(half, big.NewInt(2))
|
||||||
|
relative := new(big.Int).Abs(new(big.Int).Mul(big.NewInt(gross), productPerMoney))
|
||||||
|
return half.Add(half, relative.Quo(relative, big.NewInt(100_000)))
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user