Word Frequency Counter

Raw counts are dominated by function words

Run any English text through a naive counter and the top ten will be the, of, and, to, a, in, that, is, it, for. These carry almost no topical information. Filtering stop words is what makes the output useful, though the right stop-word list depends on the task: for authorship analysis, function words are the signal, not the noise, because their frequencies are habitual and hard to disguise.

Zipf's law and why the tail matters

Word frequencies follow a power law: the most common word appears roughly twice as often as the second, three times as often as the third, and so on. The practical consequence is that a small handful of words accounts for most of the text, while the words that actually distinguish this document from any other sit far down the list. Reading only the top twenty tells you the text is in English, not what it is about.

Stemming changes the answer

Are run, runs, running and ran one word or four? Stemming crudely chops suffixes; lemmatisation maps words to dictionary forms properly. Counting without either scatters a concept across several entries and understates it. Counting with aggressive stemming merges words that should stay apart. Neither is correct in general — the choice depends on whether you care about concepts or about exact forms.

Keyword density is not an SEO strategy

Frequency analysis is often reached for to hit a target keyword density. There is no such target: Google has not used keyword density as a ranking signal for many years, and writing to a percentage reliably produces text that reads badly. Frequency analysis is genuinely useful for the opposite purpose — spotting a word you have unconsciously repeated forty times, or confirming a page never actually says the thing it is about.

Where it does real work

Finding filler and crutch words in your own writing. Checking that a translation preserved terminology consistently. Comparing two documents' vocabularies to see what is distinctive to each. Building a glossary from a corpus. In every case the interesting output is a comparison, not an absolute count.

Tokenisation is where errors enter

Hyphenated words, apostrophes, numbers, URLs and non-Latin scripts all need decisions. Is "don't" one token or two? Is "state-of-the-art" one or four? Different tools answer differently, which is why the same text counted by two tools produces two different totals. Consistency within one analysis matters more than which convention you pick.