Title: How to Test a Bank Statement Extraction API Before You Buy
Canonical URL: https://bankstatement.ai/blog/how-to-test-a-bank-statement-extraction-api-before-you-buy
Updated: 2026-07-18
Category: api-revenue-competitors-buying-decisions

# How to Test a Bank Statement Extraction API Before You Buy

## TL;DR

- Run every candidate against the same synthetic or properly authorized statements.
- Freeze expected transactions and pass/fail thresholds before uploading files.
- Score extraction correctness, operational reliability, and human-review burden separately.
- Reject candidates that miss a mandatory correctness or workflow gate.
- Date documentation, access terms, test conditions, and observations so results remain reproducible.

A parser can return a plausible transaction array from a three-page statement while omitting one debit and reversing another transaction’s sign. The request may appear successful even though either error could corrupt reconciliation or posting. A reproducible test matrix exposes these failures by comparing every candidate with the same known transactions, fixed thresholds, and workflow experiments.

## Table of Contents

- [Build a representative statement test corpus](#build-a-representative-statement-test-corpus)
- [Establish ground truth and thresholds before uploading](#establish-ground-truth-and-thresholds-before-uploading)
- [Run the same API experiments for every candidate](#run-the-same-api-experiments-for-every-candidate)
- [Score extraction correctness without hiding critical errors](#score-extraction-correctness-without-hiding-critical-errors)
- [Measure reliability and review burden separately](#measure-reliability-and-review-burden-separately)
- [Apply the matrix to a synthetic multi-page statement](#apply-the-matrix-to-a-synthetic-multi-page-statement)
- [Use mandatory gates to shortlist or reject candidates](#use-mandatory-gates-to-shortlist-or-reject-candidates)
- [FAQ](#faq)

## Key Takeaways

- Representative coverage matters more than raw sample count. Each fixture should add a meaningful format, layout, page pattern, or scan condition.
- Ground truth must define the expected transaction count, field values, and debit or credit signs before testing begins.
- Correctness, operational reliability, and review burden need separate gates. A blended score can conceal a missing transaction or failed job.
- Documentation describes intended behavior. Dated observations record what happened under stated test conditions.
- Unavailable documentation or test access remains unresolved until verified. It is neither an assumed pass nor an assumed failure.

## Build a Representative Statement Test Corpus

Use synthetic statements or financial documents your organization is authorized to process. Give every fixture a stable ID, then record its origin, authorization status, creation date, file format, page count, layout, and reason for inclusion. Send the same unchanged files to every candidate.

Aim for representative coverage, not the largest possible folder. Add a fixture only when it introduces a material condition found in the intended workload.

| Fixture | Pages | File format | Layout condition | Scan condition | Purpose |
|---|---:|---|---|---|---|
| S-01 | 1 | Digital PDF | Single transaction table | Clear | Baseline |
| S-02 | 3 | Digital PDF | Repeated headers | Clear | Multi-page continuity |
| S-03 | 4 | Scanned PDF | Dense rows and page breaks | Readable | OCR coverage |
| S-04 | 2 | Scanned PDF | Known layout | Controlled low quality | Failure and review behavior |
| S-05 | 1 | CSV or XLSX | Known columns and values | Not applicable | Structured-file normalization |

Include CSV or XLSX only when the candidate’s current primary documentation lists that format. The low-quality PDF should be controlled, such as a degraded copy of a known fixture. Record what happens, but reserve detailed defect diagnosis and scan remediation for a separate OCR assessment.

Coverage and effort pull in different directions. More fixtures represent more layouts, but every addition requires labeling, execution, and review. Start with the smallest set covering distinct formats, page patterns, and scan conditions. Expand it around high-volume layouts or failures that need confirmation.

## Establish Ground Truth and Thresholds Before Uploading

Create a canonical transaction file for each fixture. Record the expected row count and every transaction’s date, description, amount, and debit or credit sign. Assign a stable test ID to each transaction so rows can be matched even when a parser changes spacing or punctuation.

![Frozen ground truth is compared with API output, while ambiguous matches go to manual review.](https://files.trafficwins.com/generated-images/0371334e-670e-4ffe-a66a-c61eca5d47f4/5431238c-60cb-5fce-9d5e-bff1ff08e479/725ba211-7b39-4f05-8647-0b3047931029/dfa2a8a9-8b7f-4456-b7cc-e569fe4f71cb/inline-1.png)


Define matching rules before viewing any result. Dates and amounts can usually use exact normalized comparisons. Descriptions may need approved rules for whitespace, capitalization, abbreviations, or harmless punctuation changes. Send ambiguous matches to manual review instead of silently counting them as correct.

Separate mandatory gates from tolerated variation. An accounting import test might require:

- Zero missing transactions.
- Zero duplicated transactions.
- Zero incorrect debit or credit signs.
- Exact normalized dates and amounts.
- Description differences only when the remaining text supports the intended matching or review task.

These thresholds are illustrative. A historical search tool may tolerate a description mismatch that a posting workflow cannot. Record who approved each threshold and why it fits the downstream use.

Freezing the rules prevents an appealing result from changing the definition of success. If a threshold proves unsuitable, revise it explicitly and rerun every candidate with the new version.

## Run the Same API Experiments for Every Candidate

Obtain each vendor’s current primary API documentation and test-access terms before execution. Save the source URL, documented version when available, and checked-on date. If documentation or access terms cannot be verified, mark the affected experiment unresolved rather than relying on a secondary comparison.

Each experiment should record a prerequisite, documented expectation, observed behavior, captured evidence, and pass/fail criterion.

1. **Authentication:** Send one request with valid credentials and one with controlled invalid credentials. Remove secrets before saving responses.
2. **Upload:** Submit the same supported fixture. Capture the request shape, accepted response, job or result identifier, and validation response.
3. **Asynchronous completion:** When processing is asynchronous, poll through the documented route until a terminal state or the buyer’s test timeout. Record state changes, request count, and elapsed time as test-specific observations.
4. **Malformed file:** Submit the designated malformed or unreadable PDF. Apply a predeclared rule, such as requiring a clear rejection or explicit failed state instead of an unexplained stall.
5. **Result retrieval:** Retrieve the structured result and preserve the raw response used for scoring.
6. **Exports:** When documentation promises CSV or XLSX, retrieve each documented export and compare its rows and key values with the structured result.
7. **Observable retention:** Attempt retrieval at planned observation points. Record whether results remain available or expire as documented. This experiment tests availability only, not security or compliance.

Use the same polling ceiling, retry policy, network environment, and evidence-capture method where vendor contracts permit. Record retries and failures as raw observations without converting them into pricing conclusions.

## Score Extraction Correctness Without Hiding Critical Errors

Compare every returned transaction with frozen ground truth. Report error counts for each fixture before calculating any aggregate.

![A scorecard keeps missing rows, duplicates, sign errors, and field mismatches as separate gates.](https://files.trafficwins.com/generated-images/0371334e-670e-4ffe-a66a-c61eca5d47f4/5431238c-60cb-5fce-9d5e-bff1ff08e479/725ba211-7b39-4f05-8647-0b3047931029/dfa2a8a9-8b7f-4456-b7cc-e569fe4f71cb/inline-2.png)


| Check | Pass condition | Evidence to record |
|---|---|---|
| Transaction count | Matches ground truth | Expected and returned counts |
| Missing rows | Within mandatory threshold | Count and transaction IDs |
| Duplicate rows | Within mandatory threshold | Count and matched rows |
| Transaction signs | Direction is correct | Incorrect-sign count |
| Date | Meets normalization rule | Mismatch count |
| Description | Meets approved rule or manual decision | Mismatch count |
| Amount | Exact normalized value | Mismatch count |

Do not let averages conceal critical failures. A candidate with thousands of correct fields may still be unsuitable if one transaction is absent or a credit becomes a debit. Keep missing rows, duplicates, and sign errors visible as independent gates.

Automated matching is useful for counts, normalized dates, amounts, and signs. Manual review remains necessary when descriptions are reordered, a row is split, or similar transactions make matching uncertain. Preserve the automated result and reviewer decision so another evaluator can reproduce the classification.

Any percentage derived from this corpus describes only these fixtures and conditions. It is not a universal accuracy rate.

## Measure Reliability and Review Burden Separately

Correct transactions do not prove that the workflow is predictable. For operational reliability, record completed, failed, and stalled jobs; polling requests; terminal states; raw retries; retrieval failures; and export failures. Apply the workflow gates set before execution.

Measure human-review burden independently:

- Minutes spent reviewing each fixture.
- Rows inspected.
- Fields or rows corrected.
- Ambiguous cases escalated.
- Whether the result could enter the intended downstream workflow after review.

Reviewer experience affects timing. Use the same reviewer or comparably trained reviewers following identical instructions. Treat review minutes as comparative evidence for the shared corpus, not as a general labor forecast.

Verification or confidence signals may help route work, but they do not establish correctness. Ground-truth comparison is still needed to identify missing rows, duplicates, and incorrect signs.

For example, suppose two candidates pass every mandatory correctness and workflow gate. Candidate A requires 18 review minutes across the corpus, while Candidate B requires 47 minutes and several description corrections. Those illustrative measurements do not alter correctness, but they provide a reason to investigate Candidate A further.

## Apply the Matrix to a Synthetic Multi-Page Statement

Create an illustrative three-page PDF containing 24 known transactions. Include a repeated header, a transaction positioned near a page break, two similar merchant descriptions, debits, credits, and decimal amounts. Store the canonical 24-row transaction file separately.

For every returned result:

- Confirm that all 24 transactions appear exactly once.
- Compare each normalized date and amount with its canonical row.
- Confirm that every debit and credit has the expected sign.
- Review description differences under the frozen normalization rule.
- Record missing rows, duplicates, sign errors, and field mismatches separately.
- Record polling, terminal state, retrieval outcome, and review minutes outside the correctness score.

As a documented-capability example, bankstatement.ai lists PDF, CSV, and XLSX as accepted file formats, with OCR extraction for PDFs ([bankstatement.ai llm.txt](https://bankstatement.ai/llm.txt)). Its documented API behavior includes bearer authentication, asynchronous polling, normalized transaction JSON, CSV and XLSX export retrieval, and an overall conversion verification summary with confidence and issues. The summary applies to the conversion as a whole, so it should not be treated as field-level confidence.

Place product facts under **documented capability**, with the source and checked-on date. Keep **observed result** blank until the fixture has been processed and evidence saved. Use **unresolved** for facts that cannot be verified. Reserve **buyer judgment** for thresholds such as zero missing rows or zero sign errors.

No public bankstatement.ai accuracy benchmark or processing-time SLA is available. Recheck current product and API documentation before a procurement decision because published capabilities can change. A completed synthetic test would describe only its fixture, configuration, conditions, and date, not general performance.

Apply the same evidence standard to every candidate. Support capability claims with current primary documentation and performance claims with executed tests. If primary documentation or test-access terms are unavailable, leave the affected scorecard cells unresolved.

## Use Mandatory Gates to Shortlist or Reject Candidates

Evaluate mandatory gates before comparing convenience or review effort. Reject or pause a candidate when it exceeds a non-negotiable threshold for missing rows, duplicates, signs, fields, authentication, completion, retrieval, or exports. A strong average elsewhere should not rescue a failed gate.

![Candidates must pass correctness and workflow gates before review burden is compared.](https://files.trafficwins.com/generated-images/0371334e-670e-4ffe-a66a-c61eca5d47f4/5431238c-60cb-5fce-9d5e-bff1ff08e479/725ba211-7b39-4f05-8647-0b3047931029/dfa2a8a9-8b7f-4456-b7cc-e569fe4f71cb/inline-3.png)


Among candidates that pass, compare operational observations and human-review burden. Document every accepted exception, its owner, downstream consequence, and retest condition. A candidate without usable test access remains unresolved unless access itself violates a predeclared buying requirement.

The scorecard measures sample performance and the tested workflow. Pricing, security, and full contract fit require separate evaluations. Freeze the matrix, obtain and date primary documentation and access terms for every candidate, compare each contract with the intended workflow, then run the same synthetic statement through every accessible API.

## FAQ

### How many statements are enough for an initial test?

Start with the smallest corpus covering every material format, layout, page pattern, and scan condition in the intended workload. Add fixtures for a distinct condition, a high-volume layout, or a failure that needs confirmation. Raw document count does not prove coverage.

### Should production bank statements be used?

Prefer synthetic statements when they can reproduce the required conditions. If production files are necessary, use only documents your organization is authorized to process and follow its data-handling rules. Remove unnecessary sensitive information only when doing so will not invalidate the test.

### Can a verification summary replace manual checking?

No. A verification summary can direct attention to possible issues, but it cannot confirm the result against the canonical transaction set. Missing rows, duplicates, incorrect signs, and ambiguous matches still require ground-truth comparison.

### What if a vendor’s documentation or trial is unavailable?

Mark the affected capability or candidate as unresolved, record the date and access obstacle, and avoid assigning an inferred score. Decide separately whether unavailable test access violates a buying requirement.

### When should the evaluation be rerun?

Rerun it when the statement mix changes materially, a vendor changes its API or extraction system, acceptance thresholds change, or an unresolved access condition becomes testable. Preserve corpus and scorecard versions so results from different runs are not mixed.