Remove Duplicate Lines From Text
Paste a list, and every line that appears more than once collapses to its first occurrence. The original order survives by default, capitalisation and stray spacing are yours to control, and there is a mode that shows you the duplicates instead of removing them when you need to audit rather than clean.
Use this without the search next time. Prathom Workbench puts Prathom's tools in your toolbar.
Add to Chrome — freeWhat it does
- Preserves the original order of the list
- Optional case-insensitive matching
- Blank-line and surrounding-space handling
- Inverted mode that reveals duplicates instead of removing them
How to use Remove Duplicate Lines
- 1
Paste your list
One item per line, up to 1,000,000 characters — well beyond any list you would realistically paste in.
- 2
Decide how lines are compared
Turn off case sensitivity to treat Apple and apple as the same entry. Leave "Ignore surrounding spaces" on if the list came out of a spreadsheet or a copied column.
- 3
Choose what comes out
Keep the original order, or sort the result alphabetically. Switch on "Show duplicates instead" to see only the entries that were repeated.
- 4
Copy or download
Take the cleaned list to your clipboard, or download it as a .txt file. The counts tell you how many lines went in and how many were removed.
How it works
The text is split on any newline convention — Windows, Unix, or classic Mac — so a file that has traveled between systems does not produce phantom duplicates from stray carriage returns.
Each line is then turned into a comparison key. If "ignore surrounding spaces" is on, the key is trimmed; if case sensitivity is off, the key is lowercased. The key is only ever used for matching. What gets written back out is the line exactly as you supplied it.
That separation is the whole design. It means you can match loosely without your data being rewritten — trim for comparison and still keep the original indentation, or match case-insensitively and still keep the original capitalisation.
Keys go into a Map. A Map in JavaScript preserves insertion order, so the
result comes out in the order the first occurrences appeared, with no sorting
step and no extra cost. This is why "keep original order" is the default rather
than an option you pay for: it is what the natural implementation already does.
Sorting is the thing that costs extra, and it is opt-in.
Because lookup is by hash rather than by comparison, the work grows in proportion to the number of lines rather than to the square of it. Doubling the list roughly doubles the time instead of quadrupling it.
When you'd use this
Exported lists are the everyday case — a mailing list assembled from three sources, a CSV of leads pulled twice, a column of SKUs copied out of a report. Duplicates in these are usually harmless right up until something charges per record or emails per address.
The audit case is the more interesting one. Turning on "show duplicates instead" turns the tool into a detector: which invoice numbers repeat, which usernames were registered twice, which URLs are in the sitemap more than once. The cleaning use answers "give me a clean list"; the inverted use answers "tell me what went wrong", and the second question is often the one actually worth asking.
What happens to your list
Lists are among the most sensitive things people paste into online tools. Customer emails, employee numbers, patient identifiers, subscriber exports — all routinely dropped into whatever result comes up first in search, and it is a fair question to ask what happens to them.
Deduplication runs on this site's own server rather than in your browser, so your list is sent over an encrypted connection to do the work. What matters is what happens after: there is no file, no database row, and no log line that keeps it. The list exists only for the moment it takes this server to deduplicate it, then the request ends and it is gone — nothing is retained, and nothing is available for anyone to retrieve afterward.
Examples
Cleaning an exported email list
With case sensitivity off, [email protected] and [email protected] collapse into one entry and the first spelling is the one kept. The blank line was discarded before counting, which is why four lines went in rather than five.
Auditing a keyword list without losing variants
Here case sensitivity is switched on, so the two capitalisations survive as separate entries. That is the behavior you want when the question is how a term is actually being written across a set of pages, rather than which terms exist.
Frequently asked questions
Does it keep the first or the last occurrence?
The first. If a line appears at positions 3, 40, and 91, position 3 survives and the later two are dropped. This matters whenever the list is already in a meaningful order — by priority, by date added, or by relevance — because it means deduplicating never promotes a later entry above an earlier one.
What happens to my original capitalisation?
It is preserved. Comparison and output are deliberately separate: the tool builds a normalized key to decide whether two lines match, but writes back the line exactly as you typed it. So you can match case-insensitively, or ignore surrounding spaces, without the result being silently reformatted.
Can it handle a CSV file?
It treats every line as one unit, so it will correctly remove fully identical CSV rows. It will not deduplicate on a single column — two rows that share an ID but differ anywhere else are different lines and both survive. For column-level work, sort by that column in a spreadsheet first, or extract the column and dedupe that.
What does "Show duplicates instead" do?
It inverts the question. Rather than returning the list with repeats removed, it returns only the entries that appeared more than once, one example of each. This is the mode to use when you are investigating a list rather than cleaning it — finding double-booked slots, repeated invoice numbers, or a product imported twice.
Is there a size limit?
1,000,000 characters. A request over that limit is refused outright rather than silently truncated, so you would see a clear error instead of a result quietly built from only part of your list. Within that limit, lists into the hundreds of thousands of lines process quickly — deduplication is a single pass with hash lookups rather than a comparison of every line against every other.
Further reading
- Two Words That Look Identical and Are Not the Same TextA search that finds nothing, a duplicate that will not deduplicate, a login that fails with the right password. Often the cause is that two pieces of text that look exactly the same on screen are different sequences of bytes — and Unicode allows both spellings on purpose. Here is what is happening and the one-line fix.
- Why item10 Sorts Before item2Sort a list of files and item10 lands before item2. This is not a bug and it is not a bad sort — it is the correct result of comparing text as text. Natural sort is the fix, and knowing when not to use it matters as much as knowing how.
- How to Remove Duplicate Lines From a ListDeduplicating a list sounds like one operation with one right answer. It is actually four decisions, and getting any of them wrong either leaves duplicates behind or silently deletes rows you needed. The decisions are more interesting than the operation.