First and last policies differ in placement
For “A, B, A” on separate lines, keeping first produces A then B. Keeping last produces B then A. The selected occurrence stays at its original relative position.
0 input lines · 0 retained · 0 removed · 0 repeated groups
Repeated-line frequency report
No repeated lines match the current settings.
Maximum 100,000 UTF-16 code units and 10,000 lines. Matching uses NFC normalization; retained text preserves the selected original line. CRLF and CR become LF. CSV formula-like cells receive an apostrophe prefix. Processing stays in this browser; text is not submitted to a server.
Text is processed locally in your browser. No tool input is sent to a remote processing service.
How to use the Duplicate Line Remover
- 01
Paste one item per line. Line order is preserved among the occurrences that survive.
- 02
Choose first occurrence, last occurrence, lines that appear only once, or one representative of each repeated group.
- 03
Adjust case, edge-space, and blank-line rules; inspect the retained-line count.
- 04
Copy or download the result. Export the frequency report to inspect counts and original positions.
Examples
Unique-only is not ordinary deduplication
For red, blue, red, green, unique-only keeps blue and green. Repeated-only keeps one red. These policies help isolate one-off entries or investigate repeats.
Match loosely, preserve original text
Enable case and edge-space matching to group “ Coffee ” with “coffee”. Keeping first retains the spaces and capitalization of “ Coffee ”; matching rules do not rewrite the retained line.
Pick a retention policy deliberately
Keep first is a useful default for preserving the earliest list order. Keep last favors later edits or a more recent spelling while leaving each surviving entry in the position of its last appearance. Unique-only discards every member of a repeated group, so it can remove much more than an ordinary deduplication. Repeated-only returns one first occurrence per repeated group. The removed count reflects all exclusions under the selected policy, including intentionally removed blank lines.
Understand what counts as the same line
All matching keys use NFC normalization, so canonically equivalent composed and decomposed accents compare together. English case-insensitive matching lowercases the key when enabled. Ignoring edge whitespace trims the key, but not the output line. Internal spaces and punctuation still matter: “New York” and “New York” remain separate. No fuzzy spelling, phone formatting, URL canonicalization, or linguistic case-folding is attempted.
Audit repeated groups before replacing your list
The frequency table shows a representative line, total occurrences, and the first and last original line numbers. It is sorted by descending count and then first occurrence. The page displays the first 100 repeated groups to keep the report manageable; CSV includes every repeated group. The report describes matching groups regardless of which retention policy you choose. A representative is always the first original spelling, even when the retained output uses the last occurrence.
Blank lines, exports, and spreadsheet cells
A trailing newline is an empty final line. Unless Remove blank lines is enabled, blank lines participate in matching like other entries. The tool accepts 100,000 UTF-16 code units and 10,000 lines and normalizes CRLF and CR to LF. Text downloads preserve retained line contents. In the CSV report, cells beginning with common spreadsheet formula markers receive an apostrophe prefix so that a line such as “=SUM(A1:A2)” is exported as text data.
What to know
- Maximum 100,000 UTF-16 code units and 10,000 lines.
- Exact normalized matching only; no fuzzy deduplication. The on-screen report shows at most 100 groups.
Common questions
Will the tool remove duplicate words inside a sentence?
No. It compares complete lines. Put each candidate entry on a separate line before processing.
Can I preserve all copies of a repeated entry?
The repeated-only policy returns one representative per repeated group, not every copy. Use the frequency report to see the full occurrence count.
Are visually similar Unicode characters merged?
Only canonical NFC equivalence is applied. A Latin a and a Cyrillic а remain different, as do ordinary spaces and internal nonbreaking spaces.