Remove Duplicate Lines
Remove, isolate or list duplicate lines. Free, and runs entirely in your browser — nothing you enter is uploaded, stored or sent to a server.
About Duplicate Remover
Three modes that answer different questions: keep the first occurrence of each line, show only the lines that appeared more than once, or show only the lines that appeared exactly once. Case and surrounding whitespace can be ignored, and the output can be sorted.
This is list arithmetic without a spreadsheet — merging two exports and removing repeats, finding the ids that appear in a log more than once, or isolating the entries unique to one of two lists.
Where it helps
- The duplicate and unique-only modes turn deduplication into an analysis tool.
- Ignoring case and surrounding spaces catches near-duplicates from messy exports.
- Original order is preserved unless you ask for sorting.
- Runs locally, so customer lists and access logs stay on your machine.
Limits
- Comparison is exact after the optional case and trim adjustments — no fuzzy matching for typos.
- Whole lines only: you cannot deduplicate on one column of a CSV.
- Duplicate counts are not reported, only which lines are duplicated.
- Very large lists are held in memory, so a huge log is better handled by
sort -u.
Frequently asked questions
Which mode removes duplicates?
“First occurrence of each line”. The other two are for inspecting duplicates rather than removing them.
How do I find entries in one list but not the other?
Paste both lists together and choose “only lines that appear once”. Anything present in both disappears.
Why are two apparently identical lines both kept?
They differ invisibly — trailing whitespace, a non-breaking space, or CRLF against LF. Enable trimming, or clean them with the Whitespace Remover.
Can I deduplicate on one CSV column?
Not directly. Convert to JSON, or extract the column first; comparison here is on the whole line.
Related tools
- Whitespace Remover — Trim, collapse and normalise whitespace
- Text Comparison — Line-by-line diff of two blocks of text
- Structure Converter — Convert between JSON, XML, YAML, TOML, TOON, HCL and CSV
- Word Counter — Count words, characters, sentences and reading time
- Markdown Viewer — Read a Markdown file the way it is meant to look, with its contents beside it
- Markdown to HTML — Convert Markdown to HTML you can paste somewhere, with the rendered page beside it
More Text Tools tools
- Markdown Viewer — Read a Markdown file the way it is meant to look, with its contents beside it
- Markdown to HTML — Convert Markdown to HTML you can paste somewhere, with the rendered page beside it
- Case Converter — Switch between sentence, camel, snake and other cases
- Word Counter — Count words, characters, sentences and reading time
- Text Comparison — Line-by-line diff of two blocks of text
- Whitespace Remover — Trim, collapse and normalise whitespace
- Sort Lines — Sort lines alphabetically, naturally, by number or length, reverse them, or shuffle them