Abstract
The use of Project Gutenberg (PG) as a text corpus has been extremely popular
in statistical analysis of language for more than 25 years. However, in
contrast to other major linguistic datasets of similar importance, no
consensual full version of PG exists to date. In fact, most PG studies so far
either consider only a small number of manually selected books, leading to
potential biased subsets, or employ vastly different pre-processing strategies
(often specified in insufficient details), raising concerns regarding the
reproducibility of published results. In order to address these shortcomings,
here we present the Standardized Project Gutenberg Corpus (SPGC), an open
science approach to a curated version of the complete PG data containing more
than 50,000 books and more than $3 10^9$ word-tokens. Using different
sources of annotated metadata, we not only provide a broad characterization of
the content of PG, but also show different examples highlighting the potential
of SPGC for investigating language variability across time, subjects, and
authors. We publish our methodology in detail, the code to download and process
the data, as well as the obtained corpus itself on 3 different levels of
granularity (raw text, timeseries of word tokens, and counts of words). In this
way, we provide a reproducible, pre-processed, full-size version of Project
Gutenberg as a new scientific resource for corpus linguistics, natural language
processing, and information retrieval.
Description
A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics
Links and resources
Tags