URL List Deduplicator

Deduplicate a list of URLs

This tool removes duplicate URLs from a list, with optional normalisation so that addresses which are effectively the same are treated as duplicates. You can ignore trailing slashes, lowercase the host, ignore the www prefix, and ignore the difference between http and https. These options matter because a site often serves the same page at several URL variants, and without normalisation a naive deduplication would keep them all as if they were different pages.

The first occurrence of each unique URL is kept in its original form, and the tool reports how many duplicates it removed.

Finding truly unique pages

When auditing a site, crawling, or merging URL lists from different sources, you often need the set of genuinely distinct pages. URL variants from trailing slashes, protocol differences and the www prefix inflate counts and distort analysis. Normalising before deduplicating gives an accurate list of unique pages, which is the right basis for counting, comparing against a sitemap, or planning a migration.

Choose normalisation options that match how your site actually serves content: only ignore www or protocol if those variants truly resolve to the same page. All processing happens locally in your browser, keeping your URL lists private.

Related tools

• Meta Tag Analyzer — Check a page's meta tags for the problems that actually affect indexing and click-through.
• Redirect Chain Checker — Trace the full path of redirects (301/302) to find loops or lost link juice.
• Robots.txt Validator — Validate robots.txt syntax and check whether a specific URL is allowed or blocked for a given crawler.
• Sitemap Generator (Lite) — Generate a valid XML sitemap with lastmod dates and correct structure, and learn which URLs belong in it and which do not.