Plaintext Wikipedia dump 2018

PID

Wikipedia plain text data obtained from Wikipedia dumps with WikiExtractor in February 2018.

The data come from all Wikipedias for which dumps could be downloaded at [https://dumps.wikimedia.org/]. This amounts to 297 Wikipedias, usually corresponding to individual languages and identified by their ISO codes. Several special Wikipedias are included, most notably "simple" (Simple English Wikipedia) and "incubator" (tiny hatching Wikipedias in various languages). For a list of all the Wikipedias, see [https://meta.wikimedia.org/wiki/List_of_Wikipedias].

The script which can be used to get new version of the data is included, but note that Wikipedia limits the download speed for downloading a lot of the dumps, so it takes a few days to download all of them (but one or a few can be downloaded fast). Also, the format of the dumps changes time to time, so the script will probably eventually stop working one day. The WikiExtractor tool [http://medialab.di.unipi.it/wiki/Wikipedia_Extractor] used to extract text from the Wikipedia dumps is not mine, I only modified it slightly to produce plaintext outputs [https://github.com/ptakopysk/wikiextractor].

Identifier
PID http://hdl.handle.net/11234/1-2735
Metadata Access http://lindat.mff.cuni.cz/repository/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:lindat.mff.cuni.cz:11234/1-2735
Provenance
Creator Rosa, Rudolf
Publisher Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL)
Publication Year 2018
Rights Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0); http://creativecommons.org/licenses/by-sa/3.0/; PUB
OpenAccess true
Contact lindat-help(at)ufal.mff.cuni.cz
Representation
Language Abkhazian; Abkhaz; Achinese; Adyghe; Adygei; Afrikaans; Akan; Amharic; English, Old (ca.450-1100); Arabic; Official Aramaic (700-300 BCE); Imperial Aramaic (700-300 BCE); Aragonese; Assamese; Asturian; Bable; Leonese; Asturleonese; Avaric; Aymara; Azerbaijani; Bashkir; Bambara; Belarusian; Bengali; Bangla; Bislama; Tibetan; Tibetan Standard; Central; Bosnian; Breton; Buginese; Bulgarian; Catalan; Valencian; Cebuano; Czech; Chamorro; Chechen; Cherokee; Church Slavic; Old Slavonic; Church Slavonic; Old Bulgarian; Old Church Slavonic; Chuvash; Cheyenne; Cornish; Corsican; Cree; Crimean Tatar; Crimean Turkish; Kashubian; Welsh; Danish; German; Dinka; Divehi; Dhivehi; Maldivian; Lower Sorbian; Dzongkha; Greek, Modern (1453-); Greek; English; Esperanto; Estonian; Basque; Ewe; Faroese; Persian; Farsi; Fijian; Finnish; French; Northern Frisian; Western Frisian; Fulah; Fula; Pulaar; Pular; Friulian; Gaelic; Scottish Gaelic; Irish; Galician; Manx; Gothic; Guarani; Guaraní; Gujarati; Haitian; Haitian Creole; Hausa; Hawaiian; Hebrew; Herero; Hindi; Hiri Motu; Croatian; Upper Sorbian; Hungarian; Armenian; Igbo; Ido; Inuktitut; Interlingue; Occidental; Iloko; Interlingua (International Auxiliary Language Association); Indonesian; Inupiaq; Icelandic; Italian; Javanese; Lojban; Japanese; Kara-Kalpak; Kabyle; Kalaallisut; Greenlandic; Kannada; Kashmiri; Georgian; Kanuri; Kazakh; Kabardian; Central Khmer; Khmer; Kikuyu; Gikuyu; Kinyarwanda; Kirghiz; Kyrgyz; Komi; Kongo; Korean; Karachay-Balkar; Kurdish; Ladino; Lao; Latin; Latvian; Lezghian; Limburgan; Limburger; Limburgish; Lingala; Lithuanian; Luxembourgish; Letzeburgesch; Ganda; Marshallese; Maithili; Malayalam; Marathi; Marāṭhī; Moksha; Minangkabau; Macedonian; Malagasy; Maltese; Mongolian; Maori; Māori; Malay; Creek; Mirandese; Burmese; Erzya; Neapolitan; Nauru; Nauruan; Navajo; Navaho; Ndonga; Low German; Low Saxon; German, Low; Saxon, Low; Nepali; Nepal Bhasa; Newari; Dutch; Flemish; Norwegian Nynorsk; Nynorsk, Norwegian; Norwegian; Pedi; Sepedi; Northern Sotho; Chichewa; Chewa; Nyanja; Occitan (post 1500); Provençal; Oriya; Oromo; Ossetian; Ossetic; Pangasinan; Pampanga; Kapampangan; Panjabi; Punjabi; Papiamento; Pali; Pāli; Polish; Portuguese; Pushto; Pashto; Quechua; Romansh; Romanian; Moldavian; Moldovan; Rundi; Kirundi; Aromanian; Arumanian; Macedo-Romanian; Russian; Sango; Yakut; Sanskrit; Saṁskṛta; Sicilian; Scots; Sinhala; Sinhalese; Slovak; Slovenian; Slovene; Northern Sami; Samoan; Shona; Sindhi; Somali; Sotho, Southern; Southern Sotho; Spanish; Castilian; Albanian; Sardinian; Sranan Tongo; Serbian; Swati; Sundanese; Swahili; Swedish; Tahitian; Tamil; Tatar; Telugu; Tetum; Tajik; Tagalog; Thai; Tigrinya; Tonga (Tonga Islands); Tok Pisin; Tswana; Tsonga; Turkmen; Tumbuka; Turkish; Twi; Tuvinian; Udmurt; Uighur; Uyghur; Ukrainian; Urdu; Uzbek; Venda; Vietnamese; Volapük; Waray; Walloon; Wolof; Kalmyk; Oirat; Xhosa; Yiddish; Yoruba; Zhuang; Chuang; Chinese; Zulu
Resource Type corpus
Format text/plain; charset=utf-8; application/octet-stream; application/x-gzip; downloadable_files_count: 299
Discipline Linguistics