How to use the remove duplicate lines
- Paste your list with one item per line.
- Choose options: ignore case, trim spaces, remove empty lines, and an optional sort.
- The cleaned list appears instantly with a count of removed lines.
- Copy or download the result.
How it works
The tool reads the lines once from top to bottom. For each line it builds a comparison key — trimmed if you chose to trim, lowercased if you chose to ignore case — and checks a set of keys it has already seen. The first line with a given key is kept; later ones are dropped and counted. Because only the first occurrence is kept, the original order is preserved.
Sorting, if selected, happens afterwards using natural order, so “item2” comes before “item10”. Windows (CRLF), Unix (LF) and old Mac (CR) line endings are all recognised.
Worked example
The list “banana, apple, banana, cherry, apple” (one per line) becomes “banana, apple, cherry” with 2 duplicates removed. With “Ignore case” on, “Apple” and “APPLE” would also be treated as the same line, keeping whichever appears first.
Limitations
- Lines must match exactly (after the trim and case options); near-duplicates such as typos are not detected.
- Ignoring case uses simple lowercase folding, which may not merge every language's special cases.
- Output always uses Unix line breaks (LF).
Frequently asked questions
Which copy of a duplicate is kept?
The first one, in its original position. Later repeats are removed.
Can I remove duplicates from a CSV column?
Yes, if you paste just that column. If you paste whole rows, a line only counts as a duplicate when the entire row matches.
How many lines can it handle?
Hundreds of thousands of lines on a typical laptop, because the check uses a hash set. Performance depends on your device's memory.
Does “trim” change my output?
Yes. With trim on, leading and trailing spaces are removed from the lines that are kept, not only when comparing.