Document indexing and metadata, explained properly

An organisation finishes a scanning project, opens the delivered archive, and cannot find the lease it needs. The images are excellent. Every page is crisp, straight and searchable. The problem is that four thousand documents were indexed by box number and date of scanning, and nobody knows which box the lease was in. Capture was never the weak point.

Three things, often confused

Index fields

Structured values attached to a document as a whole: invoice number, supplier, contract reference, employee ID, date of issue, document type. They are typed, selected from a list, or extracted and verified. Each one is a deliberate decision to spend effort so that a future search is fast and exact. Index fields support filtering, sorting, range queries, deduplication and retention rules. Full-text search cannot do any of those reliably.

Full text

Everything OCR read from the page, stored as searchable body content. It is cheap to produce and covers words nobody thought to index. It is also noisy: it will match a supplier name that appears once in a footer, and it inherits every OCR error on a degraded page. Full text finds documents you did not anticipate needing. It is a poor foundation for anything that must be exact.

The metadata schema

The definition rather than the data: which fields exist for each document type, which are mandatory, what data type each holds, what format a date takes, whether a value comes from a controlled list, and how documents relate to one another. The schema is written once, before indexing starts, and it is the artefact that most projects skip. Skipping it is why archives end up with dates in four formats and a supplier spelled three ways.

Indexing depth decides what is findable

Depth is the single biggest cost variable in most digitization projects, and it scales with fields per document rather than with pages. Choosing it is a business decision about retrieval, not a technical one.

A common and sensible pattern is to mix levels within one project. Index the active, frequently-retrieved series at document level with a full field set, and the dormant series at file level with full text. Uniform depth across a whole archive usually means somebody either overspent on dead records or underspent on live ones.

Designing fields people will actually use

1. 1Watch how retrieval happens now. Sit with the people who currently pull files and note what they say out loud when they look for something. Those phrases are your field list. A schema designed in a meeting reflects how the organisation describes itself, not how it searches.
2. 2Separate identity fields from descriptive ones. Identity fields, such as a contract number or an employee ID, must be exact, validated and unique. Descriptive fields, such as a subject or a category, can tolerate looser values because nothing depends on them being unique.
3. 3Use a controlled list wherever a field has fewer than a few dozen legitimate values. Free text for document type produces Invoice, INV, invoice and Tax Invoice in the same column within a week.
4. 4Store dates in one machine-readable format, and store what was printed on the page separately if the printed form matters. Never let an indexer decide how to interpret an ambiguous numeric date. Define the rule in the schema.
5. 5Define the document boundary rule before anything is scanned. What makes one document end and the next begin: a separator sheet, a header, a change of reference number. Boundary errors are the hardest defect to fix after delivery because they corrupt every field on both documents.
6. 6Decide how versions and duplicates behave. If the same contract exists in three departments' files, is it one indexed document with three locations, or three documents? Either answer works. No answer produces a mess.
7. 7Keep provenance. Record the source container, the capture date and the batch. When an anomaly surfaces two years later, provenance is what lets you trace it back to a physical shelf.

Verifying that the index is right

Index accuracy is measured at field level, on a random sample, by someone re-keying blind and comparing. Document-level accuracy is a misleading measure: a document with one wrong field out of ten is not ninety per cent useful, because the one wrong field may be the only one anyone searches by.

Three mechanisms are worth specifying, and they layer.

• Validation at entry. Format masks on reference numbers, date ranges that reject impossible values, and mandatory-field rules stop a large share of errors before they are ever stored.
• Double-key entry on critical fields. Two operators key the same value independently and the system flags disagreements for adjudication. Expensive, so reserve it for the fields that carry the retrieval weight.
• Sampled audit on delivered batches, with a defined lot size, sample size and acceptance rule, and rework of the whole lot when a batch fails. Patching only the found defects leaves the same error rate in the pages nobody sampled.

Ask for the audit results per batch as part of delivery. An index accuracy figure quoted without a stated measurement method, sample size and field list is not a figure at all.