Dedupe CSV

Duplicate rows creep in everywhere: a customer signs up twice, two exports overlap by a day, a list is merged from several sources, or someone pastes the same block twice. Duplicates inflate counts, send people the same newsletter twice and distort totals. This tool finds them and removes them, while showing you exactly what it removed.

Deciding what counts as a duplicate

The most important choice is which columns must match. Matching on every column only catches rows that are identical in every cell. Usually you want a key: the email address for a mailing list, an order number for orders, an ID for records. If your file has an Email column, it is selected for you; otherwise all columns are. Click the column chips to change the selection, for example email plus date for event registrations.

Loose matching

“Ignore upper/lower case” treats “[email protected]” and “[email protected]” as the same. “Ignore extra spaces” removes spaces at the start and end and squeezes repeated spaces in the middle, so “Ann Lee ” matches “Ann Lee”. Both are on by default, because these differences are almost always typing accidents. The comparison is loose, but the rows you keep are not altered.

First or last copy

When a value repeats, one row survives. “First copy” keeps the earliest row in the file, which suits sign-up lists where the original registration matters. “Last copy” keeps the latest, which suits logs and exports where later rows are updates.

Reviewing the duplicates

The counts tell you how many rows the file has, how many were removed, how many distinct values repeat, and how many rows are kept. Switch the preview to “Removed duplicates” to see exactly which rows will go. You can even download those removed rows on their own, to check them or to keep a record of the clean-up.

Common pitfalls

Leading zeros and formats can make identical records look different: “007” and “7”, or “+1 415 555 0132” and “(415) 555-0132”. Clean such columns first, for example phone numbers with Clean Phone Numbers, then deduplicate. Also check that the key really is unique per record; two different people can share a family email address.

Saving

Download the cleaned rows as CSV, Excel or TSV. The original file on your device stays untouched, so nothing is lost if you want to try different settings.

When rows are text lines

For plain lists without columns, such as a list of email addresses one per line, Remove Duplicate Lines is the simpler tool.

A quick check before you trust the result

Compare the counts with what you expect. If a 5,000-row newsletter list loses 2,000 rows, the key is probably wrong, for example an empty Email column that makes every blank row look like a duplicate of every other blank row. Open the removed rows, read a few of them, and only then use the cleaned file. A second pair of eyes on a sample of removed rows is cheap insurance for lists that matter.

Related tools

After merging several exports with Merge CSV Files, deduplicate the result here. To see what changed between two versions of a list instead, use Compare CSV Files. The CSV Data Profiler shows how many distinct values each column has.