There are a number of common issues in corpus linguistics papers.
- an extremely common tendency for authors to primarily cite frequencies normalised per million or thousand words (i.e. a per word baseline or multiple thereof),
- data is usually plotted without confidence intervals, so it is not possible to spot visually whether a perceived change might be statistically significant, and
- significance tests are often employed without a clear statement of what the test is evaluating.
The first issue may be unique to corpus linguistics, deriving from its particular historical origins.
It concerns the experimenter attempting to identify counterfactual alternates or select baselines. This is an experimental design question.
In the beginning was the Word.
Linguists examining volumes of plain text data (later supported by computing and part-of-speech tagging) invariably concentrated on the idea of the word as the unit of language. Collocation and concordancing sat alongside lexicography as the principal tools of the trade. “Statistics” here primarily concerned probabilistic measures of association between neighbouring words in order to find common patterns. This activity is of course perfectly fine, and allowed researchers to make huge gains in our understanding of language.
Without labouring the point (which I do elsewhere on this blog), the corollary of the statement that language is grammatical is that if, instead of describing the distribution of words, n-grams, etc, we wish to investigate how language is produced, the word cannot be our primary focus. Continue reading