BPEmb is a collection of pre-trained subword unit embeddings in 275 languages, based on Byte-Pair Encoding (BPE). In an evaluation using fine-grained entity typing as testbed, BPEmb performs competitively, and for some languages better than alternative subword approaches, while requiring vastly fewer resources and no tokenization.

This dataset is split into 275 archives, one for each language. Languages are identified via their Wikipedia ID, e.g. "en" for English or "de" for German. Each archives contains three kinds of files:

.w2v.txt: Pretrained embeddings in word2vec plain text format, for different vocabulary sizes (vs1000 to vs100000) and embedding dimensionalities (d25 to d300). .vocab: The byte-pair vocabulary for the given vocabulary size (vs1000 to vs100000) in plain text format. .model: A byte-pair segmentation model in binary SentencePiece format. .model files contain the same information as *.vocab files and are provided for convenience.