Automation KPIs: measuring what the ERP did on its own

Every ERP promises automation. Few can answer the question that arrives three months after go-live: what exactly did it do on its own, how many times, and what would have happened if it hadn't? Without an answer, automation is a matter of trust — and in finance, trust is not evidence.

The difficulty is structural. While automation works it is invisible: a posting that happened by itself creates no task and enters nobody's day. It becomes visible only when it fails. So users remember it by its failures and vendors describe it by its intent. Neither is a measurement.

The unit of measure is an event, not an impression

An automation KPI is meaningful only when it reduces to a countable event: one document, one statement line, one posting, one approval. Everything above — faster processes, a team with room to breathe — is a derived conclusion. It may rest on those numbers, not replace them.

To count an event, the system has to know its own provenance. Every record answers who created it, and that is not a username but a class of actor. Without that field, written at creation and immutable afterwards, every percentage is a reconstruction from logs — and reconstructions favour whoever performs them.

• Manual — a person entered the record from scratch.
• Rule — deterministic automation (posting scheme, reference matching, recurring invoice) produced the record untouched.
• Draft accepted unchanged — system proposed, person confirmed, no field altered.
• Draft accepted with edits — at least one field changed before confirmation; which field is the most valuable data point here.
• Draft rejected — the proposal was discarded and the work done by hand.
• Automated, then reversed — the rule fired, someone reversed the result.

The last two rows are why this exercise exists. Reversal rate is the one KPI automation cannot game: if 12% of automated postings end in a reversal, the system is not saving time — it moves time from entry to correction, onto more expensive people. The field most edited in accepted drafts does not measure the user; it shows where the rule misunderstands the business.

Four families of numbers that mean something

1. Straight-through processing rate

The share of documents that went from intake to posting without a human edit. The numerator is easy. The argument lives in the denominator, and that is where this KPI is falsified — not by inventing figures, but by quietly removing inconvenient cases from the base.

• A document rejected at validation stays in the denominator. It arrived, and somebody dealt with it.
• A document the rule handled and a human corrected in one field is NOT straight-through. Partly automated is not a category; only untouched and touched.
• Document types are never averaged. A utility invoice and an import costing with a customs declaration and landed costs are not the same job.
• The period is calendar-based, not measured from the day the rule went live. Choosing the start date is the oldest trick in reporting.

2. Bank statement matching

The bank statement is the best proving ground in this market: the input is structured and reality is not. Serbian model 97 references with a check digit, and the IPS QR payments carrying them, give a clean path — validate the reference, find the document, close the line. Then the remainder begins: a payment with no reference, one transfer covering four invoices, a partial payment with a disputed discount, a customer paying from a related party's account. That remainder consumes the day and is what a blended percentage hides, so the rate is reported by tier:

1. Deterministic: a valid reference resolves to exactly one open document, the amount agrees. The system closes the line and asks nothing.
2. Deterministic with a remainder: reference valid, amount short. The line closes partially, the balance stays open — without guessing at the difference.
3. Heuristic with evidence: no reference, but counterparty, amount and value date resolve to one candidate. This is a draft for confirmation. A system that posts here without asking will show a beautiful percentage and an expensive year.
4. Multiple candidates: two open documents of identical value. The system does not choose. A person does, and the system records the choice.
5. No candidate: the line goes to a resolution queue. An honest outcome, counted as unmatched — not vanished from the denominator.

3. Hours saved

The most abused number in the category. One honest formula: standard handling time per event type, measured at baseline, times automated events, minus time spent on corrections and reversals. The standard is a median, never a maximum — the slowest user on their worst day gives a figure that is comfortable and useless.

If posting a supplier invoice takes a median of 4 minutes, and in one month the system posted 620 documents untouched while 40 needed a 3-minute correction, the saving is 620 × 4 − 40 × 3 = 2,360 minutes, roughly 39 hours. Every factor has a source: the median from baseline, the count from the database, the corrections from the provenance register.

4. Errors prevented

The hardest KPI, because it measures what did not happen. It is honest only when it counts a block that stopped a genuinely wrong record, confirmed afterwards: a duplicate supplier invoice caught on tax ID, document number and amount counts only if a check proved the original was already posted. Otherwise it is a false positive. Classes are measured separately: duplicates; a line in the wrong box of the Serbian VAT ledger (POPDV), caught before the PP PDV return; a mismatch between the electronic VAT records (EEO/EPP) and the ledger; issuing from a longer-dated lot when FEFO required another; a PPP-PD payroll figure deviating from the contract. Each has a known cost of correction — and only that cost makes the KPI usable.

Four ways an automation KPI lies

1. Counting attempts, not outcomes. The rule fired 1,400 times — how many results survived? Without a reversal rate, the first number means nothing.
2. Silent fallback. Automation fails, the system hands the work to a human without a trace, and the event leaves the statistics. A failure that is not recorded turns every percentage into a success rate.
3. Reclassifying the denominator. When the number drops, the definition of a processed document is adjusted. So KPI definitions are versioned, and reports carry their version.
4. Averaging across types. Ninety per cent on utility invoices and thirty on import costings give seventy overall — a figure describing work nobody does, hiding the part that hurts.

How to set this up

1. 1

   Freeze the baseline

   For one month, measure median handling time per document type while everything is still manual.

2. 2

   Define the denominator before the numerator

   Write down what enters the base, per document type, before the first rule goes on. A denominator agreed after a disappointing number is never honest.

3. 3

   Introduce one rule at a time

   Two rules released the same day give one percentage and no conclusions. An effect belongs to a rule only if it was the sole change.

4. 4

   Measure corrections, not just successes

   The reversal rate and the most edited field in drafts show where the rule misunderstands the business, and what to fix next.

5. 5

   Show the numbers to the people doing the work

   A KPI visible only to management becomes a slide. The same figure must reach the accountant handling the consequences — otherwise nobody reports that the percentage is pretty and the job still hard.

How NG One approaches it

NG One treats automation KPIs as part of the core, not a report bolted on later. Provenance is a field on the record, written at creation: manual, rule, draft accepted, accepted with edits, rejected, reversed. The Automation Center, in the Insights, automation and AI space, presents active rules, execution history and hours saved over that field — data born with the work, not reconstructed afterwards. The same figures surface on the home screen: since the last sign-in, 47 documents were posted automatically and 83% of the bank statement matched. Behind each sentence sits a drill-down. A number without a path back to the records is a slogan, not a KPI.

The conclusion is not to measure more. It is to measure fewer things honestly, with the denominator written down and the reversal rate in plain sight. A company that knows 78% of its statement lines close deterministically, 9% through a confirmed proposal and 13% stay manual knows where the next hour of work sits. A company that knows the ERP helped a lot knows nothing — and finds out when choosing the next system.