Multilingual corpora for the study of new concepts in the social sciences and humanities:
By: Revekka Kyriakoglou, Anna Pappa
Potential Business Impact:
Helps computers understand new ideas from company websites.
This article presents a hybrid methodology for building a multilingual corpus designed to support the study of emerging concepts in the humanities and social sciences (HSS), illustrated here through the case of ``non-technological innovation''. The corpus relies on two complementary sources: (1) textual content automatically extracted from company websites, cleaned for French and English, and (2) annual reports collected and automatically filtered according to documentary criteria (year, format, duplication). The processing pipeline includes automatic language detection, filtering of non-relevant content, extraction of relevant segments, and enrichment with structural metadata. From this initial corpus, a derived dataset in English is created for machine learning purposes. For each occurrence of a term from the expert lexicon, a contextual block of five sentences is extracted (two preceding and two following the sentence containing the term). Each occurrence is annotated with the thematic category associated with the term, enabling the construction of data suitable for supervised classification tasks. This approach results in a reproducible and extensible resource, suitable both for analyzing lexical variability around emerging concepts and for generating datasets dedicated to natural language processing applications.
Similar Papers
Leveraging LLMs to Create Content Corpora for Niche Domains
Computation and Language
Builds helpful lists from internet information.
A large-scale, unsupervised pipeline for automatic corpus annotation using LLMs: variation and change in the English consider construction
Computation and Language
Lets computers sort words for language study.
Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora
Computation and Language
Builds better AI for languages other than English.