Bulk Text Cleaner

The characters you cannot see

Text copied from PDFs, web pages and word processors arrives carrying non-breaking spaces, zero-width spaces, soft hyphens, byte order marks and directional marks. They look like nothing, or like ordinary spaces, and they break string comparisons, defeat search, corrupt CSV imports and cause database lookups to fail against data that appears identical. A cleaner's most valuable job is removing what you cannot see.

Non-breaking spaces are the usual offender

U+00A0 renders exactly like a normal space and is inserted constantly by word processors and web content. Two strings differing only by one are not equal, which produces the maddening bug where a value visibly matches and the comparison returns false. It also survives most naive trimming, since trim functions typically strip only ASCII whitespace.

Smart quotes and dashes

Word processors substitute curly quotes for straight ones and em dashes for double hyphens as you type. That is correct for prose and fatal for code, CSV and anything parsed: a curly quote is not a string delimiter, and pasting such code produces a syntax error whose cause is invisible in most editors. Converting them back to ASCII equivalents is standard practice before using copied text as data.

Line endings differ by platform

Windows ends lines with carriage return plus line feed, Unix and macOS with line feed alone. Mixed endings in one file cause diffs to show every line as changed, break shell scripts with a trailing carriage return in the shebang, and confuse parsers. Normalising to one convention is usually the right first step for any file that has travelled between systems.

PDF text carries hard line breaks

Copying a paragraph from a PDF gives one line break per rendered line, plus hyphens where words were split across lines. Re-joining requires removing breaks inside sentences while keeping genuine paragraph breaks, and rejoining hyphenated fragments. Doing it by hand across a long document is tedious; doing it wrong merges paragraphs that should stay separate.

Clean before you compare, not after

Cleaning is most valuable applied at the point data enters a system. Cleaning after a mismatch has already been investigated wastes the investigation, and cleaning inconsistently across two datasets you intend to join guarantees they will not match.