Remove Duplicate Lines

Order is preserved

The first occurrence of each line is kept where it was; later copies are dropped. That matters when the order means something — a priority list, a sequence of steps, a ranking.

Tools that deduplicate by sorting first will also reorder everything, which is fine for a set of values and destructive for a list.

The two options exist for real reasons

• Ignore case — "Apple" and "apple" are usually the same entry in a list of names or tags, and usually different in a list of identifiers or passwords.
• Ignore surrounding spaces — data pasted from a spreadsheet or a PDF very often carries trailing spaces that are invisible and make identical-looking lines differ.

If a list looks like it has duplicates and nothing is removed, turn both on. That combination catches nearly every case.

What it cannot catch

Only exact matches, once those options are applied. "J. Smith" and "John Smith" are different lines, as are "St." and "Street", and an address with a comma in a different place. Deduplicating records rather than strings is fuzzy matching, which is a genuinely hard problem and not what this is.

Common uses

• Cleaning an email or mailing list.
• Merging several lists into one.
• Tidying log output with repeated lines.
• Finding the unique values in a column pasted from a spreadsheet.

Questions people ask

Is the order preserved?

Yes. The first occurrence stays where it was and later copies are removed. Nothing is sorted.

Why are my duplicates not being removed?

Usually invisible trailing spaces, or a difference in case. Turn on both options — that catches nearly every case.

Can it catch near-duplicates?

No. 'J. Smith' and 'John Smith' are different lines. Matching records rather than strings is fuzzy matching, which is a much harder problem.

Are blank lines removed?

Only the repeats. The first blank line is kept, like any other line.

Your text stays in the page

This matters more here than on most tools. People paste API keys into hash generators, customer data into CSV converters, contract wording into diff checkers and configuration into JSON validators. Almost every site offering these sends your input to a server to process it.

Nothing here is transmitted. The work happens in your browser, this site is static files with no server behind it, and the about page lists every request the site makes, read live from your own browser so you can check rather than trust.