Define the unit represented by a row
A dataset can contain one row per event, premises, animal, sample or test result. Before counting duplicates, read the data dictionary and inspect how the identifiers are used. The same event ID can legitimately appear on several specimen rows.
Write down the intended relationship between event, observation and result. If that relationship is missing from the supplied documentation, mark it as unresolved. Do not enforce a one-row-per-event rule merely because it makes a spreadsheet easier to sort.
Distinguish three common patterns
An exact duplicate repeats the same record without adding information. A revision changes the account of a previously recorded event. A linked observation adds a separate finding to that event. These patterns need different treatment.
Compare stable fields, source records and revision indicators. Identical animal counts are not enough to prove two events are the same, and different counts do not prove they are different. A corrected total can change while the event identity remains stable.
A small fictional example
Suppose event E-42 appears three times. Row A records an initial notification, row B records a later laboratory result and row C repeats row B after an export error. Removing every repeated E-42 would discard the result. Counting all three as separate events would inflate the total.
The appropriate structure preserves A and B as linked records and handles C as an identified duplicate of B. Keep a record of the decision so the next import does not reintroduce the same duplicated row.
Build a reconciliation table
| Incoming record | Current identity | Relationship | Reason |
|---|---|---|---|
| Initial notification | E-42 | Observation within event | Original source |
| Laboratory update | E-42 | Later observation | New result, same event |
| Repeated export row | Earlier result record | Duplicate | Same documented content |
This is an illustrative structure, not a replacement for the dataset owner's definitions. Retain source identifiers even if the public system assigns its own stable IDs. Row numbers should not become permanent identities because sorting changes them.
Recalculate at the right level
After reconciliation, count events using event identities and tests using test identities. If the dataset permits several species, samples or premises within an event, keep those relationships visible. A single deduplicated table may not answer every reporting question.
Document exclusions and corrections alongside totals. If a public figure changes, explain whether it reflects a new event, a revised count or removal of a duplicate. Avoid interpreting a database repair as a sudden change in animal disease occurrence.
Preserve the incoming release
Keep the original file, its release label and the reconciliation decisions. Apply corrections in a derived working copy rather than silently rewriting the received file. Repeatable rules should be explicit, and unusual cases should have their own reasons.
Use the release-changelog guide to explain the change and the data-release reference to check differences between releases before publishing a new total.
Preserve the rule as well as the corrected file
Document the fields compared, the relationship assigned and the evidence behind the decision. A repeatable import can then distinguish exact duplicates from revisions and linked observations without deleting every row that shares an event identifier. Keep exceptional cases explicit when the general rule cannot resolve them.
In a second fictional dataset, one event contains separate test rows for two animals. A uniqueness check designed for an event table deletes the second row. Restoring the observation identities preserves one event with two represented animals and their actual tests. The repair concerns data structure, not an additional disease occurrence.
When a source release changes its identifier convention, compare the documented meaning before joining it to older files. Identical strings can have different scopes, and different strings can refer to a revised account of the same event. Keep the original release and crosswalk rather than forcing either relationship from text alone.
Frequently asked questions
Does a repeated event ID always mean duplicate data?
Read the dataset's row definition and identifier relationships. Several animals, observations or tests may legitimately belong to one event. Deduplicate at the represented record level.
Can different counts prove separate events?
A revision may change a count while retaining the same event identity. Preserve source versions and the actual update relationship. Do not make a new event merely because a number differs.
How should a corrected public total be explained?
State whether it reflects a new occurrence, a revised account or removal of an identified duplicate. Name the counted unit and retain the reconciliation evidence. Readers should be able to understand why the database total changed without mistaking a record repair for a sudden rise or fall in disease. Keep the original incoming file available under the appropriate release label.
Cite this reference
LivestockDefense. “Resolve duplicate event identifiers in disease datasets.” Updated 2026-11-03. https://livestockdefense.com/guides/public-disease-datasets-with-duplicate-event-ids/
Add the date you accessed the page. This identifies the current reference; keep a permitted saved copy if your work depends on its exact wording. A citation alone does not establish support for a claim.
How to cite a page and its revision ↗