README.md

# Awesome collections on DataHub

The awesome section presents collections of high quality datasets organized by topic.

Home page for awesome collections is located in the [frontend](https://github.com/datahq/frontend) repo and should be modified from there. See the live page here:

https://datahub.io/awesome

## Collections

* [Air Pollution data](https://datahub.io/awesome/air-pollution)
* [Bibliographic data](https://datahub.io/awesome/bibliographic-data)
* [Broadband data](https://datahub.io/awesome/broadband-data)
* [Climate Change](https://datahub.io/awesome/climate-change)
* [Demographics (population)](https://datahub.io/awesome/demographics)
* [Economic Data and Indicators](https://datahub.io/awesome/economic-data)
* [Education](https://datahub.io/awesome/education)
* [Football](https://datahub.io/awesome/football)
* [GeoJSON](https://datahub.io/awesome/geojson)
* [Health Care Data](https://datahub.io/awesome/healthcare-data)
* [Inflation](https://datahub.io/awesome/inflation)
* [Linked Open Data](https://datahub.io/awesome/linked-open-data)
* [Logistics](https://datahub.io/awesome/logistics-data)
* [Machine Learning / Statistical](https://datahub.io/awesome/machine-learning-data)
* [Movies and TV](https://datahub.io/awesome/movies-and-tv)
* [Open Corporates](https://datahub.io/awesome/opencorporates)
* [Property Prices](https://datahub.io/awesome/property-prices)
* [Reference Data](https://datahub.io/awesome/reference-data)
* [Stock Market Data](https://datahub.io/awesome/stock-market-data)
* [War and Peace](https://datahub.io/awesome/war-and-peace)
* [Wealth, Income and Inequality](https://datahub.io/awesome/wealth-income-and-inequality)
* [World Bank](https://datahub.io/awesome/world-bank)
* [YAGO](https://datahub.io/awesome/yago)


air-pollution.md

---
title: Air Pollution Data
description: Air pollution datasets and other sources of related data.
keywords: air pollution data
date: 2018-07-17
modified: 2020-03-20
---

## Policy

* Country or sub-region level monitor airpollution
* CO2 "price" in European ETS http://datahub.io/core/eu-emissions-trading-system (European Union Emissions Trading System (EU ETS) data from EUTL)

## State of the World

* On a global level there are open data on airpollution around the globe https://www.epa.gov/outdoor-air-quality-data/download-daily-data
with parameters of datasets: type of pollutants, year, geographical areas
* On a loval level there are open data on the level of each country (see below for some countries)

Parameters

* Pollutants level
* Citizen science contributions and independent measurements

## Sources

* World https://www.epa.gov/outdoor-air-quality-data/download-daily-data
* France https://www.airparif.asso.fr/en/#
* UK https://www.gov.uk/government/collections/air-quality-and-emissions-statistics




bibliographic-data.md

---
title: Bibliographic data
description: Existing databases or services providing substantial bibliographic data
keywords: DBLP data
date: 2018-07-17
modified: 2018-07-17
image: bibliographic-data.png
---

## The DBLP Computer Science Bibliography

The DBLP computer science bibliography contains the metadata of over 1.8 million publications, written by over 1 million authors in several thousands of journals or conference proceedings series.

Although DBLP started with a focus on database systems and logic programming (hence the acronym), it has grown to cover all disciplines of computer science.

### Data

Resources list the full dump of the DBLP XML records (see http://dblp.uni-trier.de/xml/ - a [simple DTD](http://dblp.uni-trier.de/db/about/dblp.dtd) is available.

The paper "[DBLP - Some Lessons Learned](http://dblp.uni-trier.de/xml/docu/dblpxml.pdf)" documents technical details of this XML file. In the appendix ["DBLP XML Requests"][paper] you may find the description of a primitive DBLP API.

[paper]: http://dblp.uni-trier.de/xml/docu/dblpxml.pdf

### Openness: OPEN

As of 2011-12-09 this data is open (relased under ODC-By). See  the license information in the [Readme.txt](http://dblp.uni-trier.de/xml/README.txt) and the announce post: http://openbiblio.net/2011/12/09/dblp-releases-its-1-8-million-bibliographic-records-as-open-data/

### Data and Resources

* [DBLP XML records (Full dump in xml (gzipped))](http://dblp.uni-trier.de/xml/dblp.xml.gz)
* [DBLP DTD - The XML file references this DTD.](http://dblp.uni-trier.de/xml/dblp.dtd)


broadband.md

---
title: Broadband Data
description: Data on broadband access and usage around the world
keywords: broadband data
date: 2021-05-14
modified: 2021-05-14
---

* United States Broadband Usage Percentages Dataset
(published by Microsoft) in 2021 https://github.com/microsoft/USBroadbandUsagePercentages


climate-change.md

---
title: Climate Change Data
description: A collection of the most important "general" datasets on climate change.
keywords: climate data, world climate data, historical climate data, CO2 levels globally, CO2 emission from fossil fuels
date: 2018-04-12
modified: 2018-06-12
image: climate-change.png
---

What are the most important "general" datasets to have on climate change?

[[toc]]

## Policy

* Country or sub-region commitments (e.g. emission amounts, cuts, temperature targets)
* Which countries / regions have emission trading schemes in place and their status
* CO2 "price" in European ETS http://datahub.io/core/eu-emissions-trading-system (European Union Emissions Trading System (EU ETS) data from EUTL)

## State of the World

Green house gases including C02:

* CO2 Levels globally (PPM over time) - http://datahub.io/core/co2-ppm
* CO2 emissions:
  * From Fossil Fuels: http://datahub.io/core/co2-fossil-global
  * From Fossil Fuels By Nation: https://datahub.io/core/co2-fossil-by-nation
* Human population growth / Urban population growth
* Volcano emissions

Impacts

* Global temperature
  * http://datahub.io/core/global-temp (NASA GISTEMP and GCAP)
  * http://datahub.io/core/global-temp-anomalies (NASA GISTEMP anomalies)
* [#27](https://github.com/datasets/registry/issues/27) - CO2 per country (what about other GHG)
* Soil temperature
* Satelite measurement temperature datasets
* Ocean temperature datasets
* Glacier size - http://datahub.io/core/glacier-mass-balance
* Sea Level http://datahub.io/core/sea-level-rise (Global Average Absolute Sea Level Change, 1880-2014)

## Sources

* IPCC Data - http://www.ipcc-data.org/
* NOAA
* http://nsstc.uah.edu/climate/
* http://data.worldbank.org (search for 'climate change')
* http://cdiac.ornl.gov/climate/temp/temp_table.html
* http://www.climate4you.com/GlobalTemperatures.htm
* https://www.ncdc.noaa.gov/data-access
* http://catalog.data.gov/dataset
* https://climatedataguide.ucar.edu/climate-data
* https://climatedataguide.ucar.edu/data-vertical-levels/surface-data-set


climate-data.md


# Climate open data

Everyone can have access to climate data, however many people were not able to access the data due to various reasons: failing register on the website, 
formulating and sending request to get the right data they need.
Here we write precise 

We use several sources for open data on temperature:
1. reanalysis data from GES DISC MERRA from https://disc.gsfc.nasa.gov/
2. observational data: example data available for downloading after login http://data.ceda.ac.uk/badc/ukmo-hadobs/data/insitu/MOHC/HadOBS/HadUK-Grid/v1.0.0.0/60km/  and downloadable data link url = 'http://dap.ceda.ac.uk/thredds/fileServer/badc/ukmo-hadobs/data/insitu/MOHC/HadOBS/HadUK-Grid/v1.0.0.0/60km/tas/mon/v20181126/tas_hadukgrid_uk_60km_mon_188401-188412.nc' and https://services.ceda.ac.uk/cedasite/myceda/user/ 

# How to download the data 
1. Create account and log in to https://www.ceda.ac.uk/ 
2. Depending on the variables and time-scale of the data you want to download setup request, for example, 2010 taxmax (temperature): http://dap.ceda.ac.uk/thredds/fileServer/badc/ukmo-hadobs/data/insitu/MOHC/HadOBS/HadUK-Grid/v1.0.0.0/60km/tas/mon/v20181126/tas_hadukgrid_uk_60km_mon_201001-201012.nc



# Climate variables  
Examples of climate variables:
    surface_air_temperature 

    surface_layer_height 

    surface_pressure 

    surface_specific_humidity 

    surface_wind_speed

# Code for parcing data 

For parcing data one needs to install ´xarray´ python package.
The code for parcing the data is uploaded to the folder https://github.com/Liyubov/climate_open_data/blob/master/parse_hadukgrid.py 

You can also open and look into the climate data using command line via:
*ncdump -t tasmax_hadukgrid_uk_60km_mon_188601-188612.nc*


coronavirus.md

---
title: Coronavirus Data and Resources
description: A collection of the most important, relevant datasets on Coronavirus (COVID-19) outbreak.
keywords: coronavirus, covid-19, health, epidemic
date: 2020-03-25
modified: 2020-03-25
image: climate-change.png
---

Coronavirus disease 2019 (COVID-19) time series listing confirmed cases, reported deaths and reported recoveries. Data is disaggregated by by country (and sometimes subregion). Coronavirus disease (COVID-19) is caused by the [Severe acute respiratory syndrome Coronavirus 2 (SARS-CoV-2)][sars2] and has had a worldwide effect. On March 11 2020, the World Health Organization (WHO) declared it a pandemic, pointing to the over 118,000 cases of the coronavirus illness in over 110 countries and territories around the world at the time.

[covid]: https://en.wikipedia.org/wiki/Coronavirus_disease_2019
[sars2]: https://en.wikipedia.org/wiki/Severe_acute_respiratory_syndrome_coronavirus_2

[[toc]]

## Cases, Deaths and Recoveries

Primary Dataset: https://datahub.io/core/covid-19

This dataset includes time series data tracking the number of people affected by COVID-19 worldwide, including:

* confirmed tested cases of Coronavirus infection
* how many have reportedly died while sick with Coronavirus
* the number of people who have reportedly recovered from it

Data is in CSV format and updated daily. It is sourced from [this upstream repository](https://github.com/CSSEGISandData/COVID-19) maintained by the amazing team at [Johns Hopkins University Center for Systems Science and Engineering](https://systems.jhu.edu/) (CSSE) who have been doing a great public service from an early point by collating data from around the world.

Data is cleaned and normalized, for example tidying dates and consolidating several files into normalized time series. There is also metadata such as column descriptions.

### Sources

The upstream dataset currently lists the following upstream datasources:

- World Health Organization (WHO): https://www.who.int/
- DXY.cn. Pneumonia. 2020. http://3g.dxy.cn/newh5/view/pneumonia.
- BNO News: https://bnonews.com/index.php/2020/02/the-latest-coronavirus-cases/
- National Health Commission of the People’s Republic of China (NHC):
- http://www.nhc.gov.cn/xcs/yqtb/list_gzbd.shtml
- China CDC (CCDC): http://weekly.chinacdc.cn/news/TrackingtheEpidemic.htm
- Hong Kong Department of Health: https://www.chp.gov.hk/en/features/102465.html
- Macau Government: https://www.ssm.gov.mo/portal/
- Taiwan CDC: https://sites.google.com/cdc.gov.tw/2019ncov/taiwan?authuser=0
- US CDC: https://www.cdc.gov/coronavirus/2019-ncov/index.html
- Government of Canada: https://www.canada.ca/en/public-health/services/diseases/coronavirus.html
- Australia Government Department of Health: https://www.health.gov.au/news/coronavirus-update-at-a-glance
- European Centre for Disease Prevention and Control (ECDC): https://www.ecdc.europa.eu/en/geographical-distribution-2019-ncov-cases
- Ministry of Health Singapore (MOH): https://www.moh.gov.sg/covid-19
- Italy Ministry of Health: http://www.salute.gov.it/nuovocoronavirus
- ECDC dataset https://www.ecdc.europa.eu/en/coronavirus
- Metadata on publications about coronavirus https://github.com/nties/COVID19-open-research-dataset 
- Open data resources for fighting COVID https://arxiv.org/abs/2004.06111v1

Additional background is available in the [CSSE blog](https://systems.jhu.edu/research/public-health/ncov/), and in the [Lancet paper](https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30120-1/fulltext) ([DOI](https://doi.org/10.1016/S1473-3099(20)30120-1)), which includes this figure:

![](https://i.imgur.com/X32lUEU.png)

## Cases By Country

For now see https://datahub.io/core/covid-19 as that includes aggregate figures by country. For additional sources (verified and crowd-sourced) https://github.com/open-covid-19/data


### US

Data sources: COVID Tracking Project in USA https://covidtracking.com/  
New York Times data from https://github.com/nytimes/covid-19-data 


Data information: 
COVID Tracking Project in USA provides the information from state/district/territory public health authorities—or, occasionally, from trusted news reporting, official press conferences, or (very occasionally) tweets or Facebook updates from state public health authorities or governors.
Data from New York Times is State-Level Data

State-level data can be found in the states.csv file. (Raw CSV file here.)

date,state,fips,cases,deaths
2020-01-21,Washington,53,1,0

### Italy

Data source: https://github.com/pcm-dpc/COVID-19


Data information:
In order to inform citizens and make available the data collected, useful only for communication and information purposes, the Civil Protection Department has developed an interactive geographical dashboard that can be reached at http://arcg.is/C1unv (desktop version) and http://arcg.is/081a51 (mobile version) and makes available, under license CC-BY-4.0, the following information updated daily at 6:30 p.m. (following the press conference of the Head of Department):

    National trend
    Json data
    Provinces data
    Regions data
    Summary sheets
    Areas
    Notes

### United Kingdom

Data source: https://github.com/tomwhite/covid-19-uk-data


Data information: 
The following CSV files are available:

    data/covid-19-cases-uk.csv: daily counts of confirmed cases for (upper tier) local authorities in England, health boards in Scotland and Wales, and local government district for Northern Ireland.
        Note that prior to 18 March 2020 Wales data was broken down by local authority, not heath board, and prior to 27 March 2020 there were no breakdowns by area for Northern Ireland.
    data/covid-19-totals-uk.csv: daily counts of tests, confirmed cases, deaths for the whole of the UK
    data/covid-19-totals-england.csv: daily counts of tests, confirmed cases, deaths for England
    data/covid-19-totals-northern-ireland.csv: daily counts of tests, confirmed cases, deaths for Northern Ireland
    data/covid-19-totals-scotland.csv: daily counts of tests, confirmed cases, deaths for Scotland
    data/covid-19-totals-wales.csv: daily counts of tests, confirmed cases, deaths for Wales
    data/covid-19-indicators-uk.csv: daily counts of tests, confirmed cases, deaths for the whole of the UK and individual countries in the UK (England, Scotland, Wales, Northern Ireland). This is a tidy-data version of covid-19-totals-*.csv combined into one file.

### France

Data source: https://github.com/cedricguadalupe/FRANCE-COVID-19


Data information: 
Data is updated daily according to the main sources of information: 
Agence Regionale de Sante;    Santé Publique France : https://www.santepubliquefrance.fr/;   Geodes: https://geodes.santepubliquefrance.fr/#c=indicator&view=map2

The github repository wih data contains three main files with confirmed cases, number of death and number of recovered people.

    france_coronavirus_time_series-confirmed.csv
	france_coronavirus_time_series-deaths.csv 	
	france_coronavirus_time_series-recovered.csv

## Models
There are various types of models in epidemiology, which are used for prediction of prevalence (total number of infected) or the duration of an epidemic spreading processes. One of the most commonly used types is compartmental models, where population is divided into compartments of people with the same properties, e.g. people who are susceptible to virus, who are recovered etc.
There are three main types of compartmental models in epidemiology based on which many other models can be built upon: 
    Susceptible-Infected (SI), where each person can be just in two states, either susceptible (S) or infected (I); 
     Susceptible-Infected-Recovered (SIS), where each person can be in one of two states, susceptible (S) or infected (I), but then can become susceptible again;
     Susceptible-Infected-Recovered (SIR), where each person can be  in one of three states, susceptible (S), infected (I) or recovered (R).
     

In **SIR model** one can study the number of people in each of three compartments: susceptible, infected and recovered, denoted by variable S, I and R correspondingly. SIR system can be  expressed by the set of ordinary differential equations proposed by [O. Kermack and Anderson Gray McKendrick](https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology). 

The simulations of SIR model are shown in the python script [here](https://github.com/Liyubov/heterogeneous-dynamics-on-networks/blob/master/code_network_heterogen_models/spreading_SIR.py).

The main take-away message from the SIR model simulations is that we observe so-called peak of epidemic, when the number of infected people reaches its maximum. This moment of when size of epidemic outbreak is maximal depends on SIR model parameters: probability of epidemic transmission and recovery from the disease. 

Another model, which has been proven to be much more realistic for some epidemics spreading models is **SEIR model** [1]. 
The main idea of this model is that for many important infections there is a significant incubation period during which individuals have been infected but are not yet infectious themselves. During this period the individual is in a special state E (exposed), additional state to three states of parts of population in SIR model. 

The SEIR model studies number of people between four states: susceptible (S), exposed (E), infected (I), and resistant (R) , each being the number of people in those groups. There are several model parameters, which partially control how fast people move from being susceptible to exposed, from exposed to infected, and from infected to resistant. This model can have additional parameters; one is the background mortality, which is unaffected by disease-state, while the other is vaccination. The vaccination moves people from the susceptible to resistant directly, without becoming exposed or infected.
The SEIR model is the most widely used tool for predicting epidemics and acting to suppress them. In 2020, the model was further refined by Richard Neuer [3].


## References:
1. [Maslov et al. "Window of Opportunity for Mitigation to Prevent Overflow of ICU capacity in Chicago by COVID-19"](https://arxiv.org/abs/2003.09564)
2. [Pastor-Satoras et al. "Epidemic processes in complex networks"](https://arxiv.org/abs/1408.2701)
3. [COVID model by Neuer](https://neherlab.org/covid19/)
4. [O. Kermack and Anderson Gray McKendrick](https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology)
5. Simulations of SIR and SEIR models [here](https://github.com/Liyubov/heterogeneous-dynamics-on-networks/blob/master/code_network_heterogen_models/spreading_SIR.py)


demographics.md

---
title: Demographics (population)
description: Population data and data analytics.
keywords: world population, city population, historical population, demographic, demographics data
date: 2018-06-11
modified: 2018-06-11
image: demographics.png
---

## Population figures for countries, regions (e.g. Asia) and the world

Data comes originally from World Bank and has been converted into standard CSV:

[core/population](/core/population)

## City Population Annual Time Series (UN Statistics Division)

UNSD Demographic Statistics: City population by sex, city and city type:

[core/population-city](/core/population-city)

## Global Historical Population

The population data starts from -1000000 BC to 1990 with the average number of people. There are several population data from the different reports such as Deevey, McEvedy and Jones 1978, Durand Low, Durand High, Clark,Biraben, Blaxter, UN, Kremer:

[core/population-global-historical](/core/population-global-historical)

## Population Growth

Population growth estimates and projections:

[core/population-growth-estimates-and-projections](/core/population-growth-estimates-and-projections)


economic-data.md

---
title: Economic Data
description: A collection of economic indicators available on DataHub.
keywords: economic statistics, examples of economic indicators, explore data by country, basic economic indicators, US economic data, UK economic data, World Bank data
date: 2018-06-06
modified: 2018-06-12
image: economic-data.png
---

On this page, we summarize all available economic data (or economic statistics) on DataHub. Users can find various examples of economic indicators or explore data by country. The page is curated and updated on regular basis so don't forget to bookmark it!

[[toc]]

## Basic economic data

Following datasets can be considered as basic economic indicators:

* [GDP](/core/gdp) - Country, Regional and World GDP (Gross Domestic Product).
* [CPI](/core/cpi) - Annual Consumer Price Index (CPI) for most countries in the world.
* [Inflation](/core/inflation) - Annual inflation by GDP deflator and consumer prices.
* [Population](/core/population) - Population figures for countries, regions (e.g. Asia) and the world.
* [Cash Surplus Deficit](/core/cash-surplus-deficit) - Cash Surplus/Deficit, in % of GDP, from 1990 to 2013.
* [Other Economic Data](#other-economic-data-and-indicators)

## US economic data

Economic indicators for the United States:

* [GDP](/core/gdp-us) - Gross Domestic Product (GDP) of the United States (US) both nominal and real on an annual and quarterly basis.
* [Employment](/core/employment-us) - US Employment and Unemployment rates since 1940.
* [CPI](/core/cpi-us) - US Consumer Price Index and Inflation (CPI).
* [Bond Yields](/core/bond-yields-us-10y) - 10 year US Government Bond Yields (long-term interest rate).
* [House Prices](/core/house-prices-us) - US House Price Index (Case-Shiller).
* [Household Incomes](/core/household-income-us-historical) - Income Limits for Each Fifth and Top 5 Percent of All Households: 1967 to 2016.
* [Investor Flow of Funds](/core/investor-flow-of-funds-us) - US Investor Flow of Funds into Investment Classes (Bonds, Equities etc).
* [Education Budget](/core/usa-education-budget-analysis) - US Education Budget Analysis.

## UK economic data

Economic indicators for the United Kingdom:

* [GDP](/core/gdp-uk) - UK Real GDP since 1948 from the Office of National Statistics.
* [Bond Yields](/core/bond-yields-uk-10y) - 10 year nominal yields on UK government bonds from the bank of England.
* [House Prices](/core/house-prices-uk) - UK house prices since 1953 as monthly time-series.
* [Interest Rates](/core/interest-rates-gb) - Interest Rate since 1694 from Bank of England.

## World Bank data

World Bank is a one of the best sources of different datasets including economic data. Please, read the following article about World Bank data collections, API and more:

[awesome/world-bank](/awesome/world-bank)

## International Monetary Fund (IMF)

IMF World Economic Outlook (WEO) database. The source database is made of annual values for each country on 45 indicators since 1980. In addition the database includes the IMF projects approximately 6 years into the future.

[core/imf-weo](/core/imf-weo)

## Other economic data and indicators

* [Corruption Perceptions Index](/core/corruption-perceptions-index)
* [Euribor](/core/euribor) - Euribor rates by year and granularity. Only monthly granularity is provided.
* [Exchange Rates](/core/exchange-rates) - Foreign exchange rates from US Federal Reserve in daily, monthly and yearly basis.
* [Expenditure on Research and Development](/core/expenditure-on-research-and-development)
* [GINI Index](/core/gini-index) - Historic values of the GINI Index.
* [Global House Prices](/core/house-prices-global) - Residential property price statistics from different countries.
* [Population Growth Estimates and Projections](/core/population-growth-estimates-and-projections)
* [Purchasing Power Parity (PPP)](/core/ppp)
* [Brent and WTI Spot Prices](/core/oil-prices) - Europe Brent and WTI (Western Texas Intermediate) Spot Prices (Annual/ Monthly/ Weekly/ Daily).
* [Natural gas prices](/core/natural-gas) - Time series of major Natural Gas Prices including US Henry Hub.
* [Gold Prices](/core/gold-prices) - Monthly gold prices since 1950 in USD (London market).


education.md

---
title: Education Data
description: US education data
keywords: education, data, data analytics, US education budget analysis, CIRP freshmen survey
date: 2018-06-07
modified: 2018-06-07
image: education.png
---

## United States of America education budget analysis

United States of America Education budget to GDP analysis:

[core/usa-education-budget-analysis](/core/usa-education-budget-analysis)

## CIRP Freshmen Survey

CIRP survey of the american freshmen conducted in 2014. Contains student’s opinions on various matters as well as information about institutions participating in a survey:

[core/cirp-survey-of-freshmen](/core/cirp-survey-of-freshmen)


football.md

---
title: Football Data
description: A collection of awesome football datasets including national teams, clubs, match schedules, players, stadiums, etc.
keywords: football data and API, world cup, stadium data, football apps
date: 2017-12-13
modified: 2018-06-12
image: football.png
---

A collection of awesome football datasets including national teams, clubs, match schedules, players, stadiums, etc.

*Contributions welcome. Anything missing? Send in a pull request. Thanks.*

Note: :octocat: stands for the GitHub page, :gem: stands for the RubyGems page and :datahub: stands for dataset on DataHub.

[[toc]]

# Football Data Guides / Articles

_Where's the open football data?_

- [Guide to Football Data and APIs](http://www.jokecamp.com/blog/guide-to-football-and-soccer-data-and-apis/) - The Definite Football Data List collected by Joe Kampschmid  
- [Article: Using open football data - Get ready for the World Cup in Brazil 2014 @ The Data Wrangling Blog (Open Knowledge Foundation (OKFN) Labs)](http://okfnlabs.org/blog/2014/05/06/open-data-world-cup.html) by Gerald Bauer

# Football Datasets

## Football datasets on DataHub

- [sports-data/english-premier-league :datahub:](https://datahub.io/sports-data/english-premier-league)
- [sports-data/spanish-la-liga :datahub:](https://datahub.io/sports-data/spanish-la-liga)
- [sports-data/italian-serie-a :datahub:](https://datahub.io/sports-data/italian-serie-a)
- [sports-data/german-bundesliga :datahub:](https://datahub.io/sports-data/german-bundesliga)
- [sports-data/french-ligue-1 :datahub:](https://datahub.io/sports-data/french-ligue-1)

## World Cup

- [openfootball/world-cup :octocat:](https://github.com/openfootball/world-cup)
- [lsv/fifa-worldcup-2018 :octocat:](https://github.com/lsv/fifa-worldcup-2018) - FIFA Worldcup 2018 - json data (The data file will be updated within a hour after a match is over)
- [kidehen/worldcup2014 :octocat:](https://github.com/kidehen/worldcup2014) - World cup data
- [estiens/world_cup_json :octocat:](https://github.com/estiens/world_cup_json) - rails backend for a scraper that outputs World Cup data as JSON
- [sanand0/fifadata :octocat:](https://github.com/sanand0/fifadata) - scraping FIFA world cup data
- [pratapvardhan/FIFAWorldCup :octocat:](https://github.com/pratapvardhan/FIFAWorldCup) - FIFA World Cup data includes teams data, squad formations, clubs dominance


## England

- [engsoccerdata :octocat:](https://github.com/jalapic/engsoccerdata) - all top 4 tier football matches in England 1888-2014; collected by James Curley


## Misc

- [jokecamp/FootballData :octocat:](https://github.com/jokecamp/FootballData) - a hodgepodge of JSON and CSV football data
- [llimllib/soccerdata :octocat:](https://github.com/llimllib/soccerdata) - a collection of soccer results
- [milkysunshine91/sport_db.Football :octocat:](https://github.com/milkysunshine91/sport_db.Football) - general purpose football database
- [orlandoaleman/FootballAppResources :octocat:](https://github.com/orlandoaleman/FootballAppResources)


# Stadium Datasets

- [openfootball/stadiums :octocat:](https://github.com/openfootball/stadiums)


# Football Apps

_Open source apps for match scores, picks, predictions, office pools, and more_

- [worldcup-2014 gem :octocat:](https://github.com/hpoydar/worldcup-2014), [:gem:](https://rubygems.org/gems/worldcup-2014) - provides command line access to World Cup 2014 information and results
- [world_cup_cli gem :octocat:](https://github.com/jameswilliamiii/world_cup_cli), [:gem:](https://rubygems.org/gems/world_cup_cli) - a command line interface that provides you the latest group table standings, scores, and see upcoming matches from the 2014 World Cup

- [fatiherikli/worldcup :octocat:](https://github.com/fatiherikli/worldcup) - World cup results for hackers; uses Soccer For Good API
- [Huang-Wei/2014 :octocat:](https://github.com/Huang-Wei/2014)
- [rtopitt/bolao2014 :octocat:](https://github.com/rtopitt/bolao2014) - Bolão PiTTlândia Copa do Mundo 2014
- [rtopitt/bolao :octocat:](https://github.com/rtopitt/bolao) - Bolão Copa 2010
- [threefunkymonkeys/funky-world-cup :octocat:](https://github.com/threefunkymonkeys/funky-world-cup) - a match predictions website for the FIFA World Cup, that allows you to create groups so you can play with your friends defining prices
- [malagant/tipptop :octocat:](https://github.com/malagant/tipptop) -  world cup 2010 betting game; W-JAX Challenge

- [soccer_league :octocat:](https://github.com/mrjabba/soccer_league) - a rails application designed to manage soccer leagues, specifically teams, players and their stats
- [standings gem :octocat:](https://github.com/scottluptowski/standings), [:gem:](https://rubygems.org/gems/standings) - view European football (e.g. the English Premier League, English Championship, Scottish Premier League, La Liga, Ligue 1, Serie A, and Bundesliga) standings from your terminal.
- [ahs85/bundesliga_predictions :octocat:](https://github.com/ahs85/bundesliga_predictions) - predictions of the Deutsche Bundesliga (football league) season 2012/13
- [architv/soccer-cli](https://github.com/architv/soccer-cli) - command line tool for league table standings, match scores and more (in Python) using an HTTP JSON API


- [4teamwork/ftw.footballchallenge :octocat:](https://github.com/4teamwork/ftw.footballchallenge) - an online football bet game based on plone
- [sigi/bookie :octocat:](https://github.com/sigi/bookie) - a rails application to manage a soccer betting community or office pool
- [kdungs/tippspiel :octocat:](https://github.com/kdungs/tippspiel) - bet on football games with your friends
- [chipsmachine/bltippspiel :octocat:](https://github.com/chipsmachine/bltippspiel) - Bundesliga betting game (tippspiel)
- [chrenkot/Austrian-Bundesliga :octocat:](https://github.com/chrenkot/Austrian-Bundesliga) - a little open source android app for gathering information about the austrian bundesliga


geojson.md

---
title: GeoJSON
description: GeoJSON datasets available on DataHub.
keywords: geojson, European NUTS boundaries, Country Polygons, Natural Earth Admin1 Polygons
date: 2018-06-07
modified: 2018-06-07
image: geojson.png
---

[[toc]]

## European NUTS boundaries as GeoJSON at 1:60m

Geo Boundaries for NUTS administrative levels 1, 2 and 3 edition 2013:

[core/geo-nuts-administrative-boundaries](/core/geo-nuts-administrative-boundaries)

If you don't know what NUTS (Nomenclature of Territorial Units for Statistics) are, see the [related Wikipedia article](https://en.wikipedia.org/wiki/Nomenclature_of_Territorial_Units_for_Statistics).

## Country Polygons as GeoJSON

Geodata data package providing geojson polygons for all the world’s countries. Perfect for use in apps and visualizations:

[core/geo-countries](/core/geo-countries)

## Natural Earth Admin1 Polygons as GeoJSON

Geodata data package providing geojson polygons for the largest administrative subdivisions in every countries:

[core/geo-ne-admin1](/core/geo-ne-admin1)

## Natural Earth Admin1 Polygons as GeoJSON US

Geodata data package providing geojson polygons for the states in the USA:

[core/geo-admin1-us](/core/geo-admin1-us)


healthcare-data.md

---
title: Health Care Data
description: Ready-to-use datasets on DataHub about Health Care.
keywords: Healthcare, DNA Sequencing Costs, Pharmaceutical Drug Spending by countries and Data Analytics
date: 2018-06-07
modified: 2018-06-07
image: healthcare-data.png
---

## Costs assosiated with DNA sequencing since 2001

DNA Sequencing Costs:

[core/genome-sequencing-costs](/core/genome-sequencing-costs)

## Pharmaceutical Drug Spending by countries (data analytics)

Pharmaceutical Drug Spending by countries with indicators such as a share of total health spending, in USD per capita (using economy-wide PPPs) and as a share of GDP. Plus, total spending by each countries in the specific year:

[core/pharmaceutical-drug-spending](/core/pharmaceutical-drug-spending)


inflation.md

---
title: Inflation
description: Datasets re Inflation
date: 2017-12-13
modified: 2018-06-06
image: inflation.png
---

* Annual inflation by GDP deflator and consumer prices: [core/inflation](/core/inflation)
* US Consumer Price Index and Inflation (CPI): [core/cpi-us](/core/cpi-us)


linked-open-data.md

---
title: Linked Open Data
description: An overview of the Linked Open Data datasets.
keywords: Open Data, DBpedia, Lexvo, General Multilingual Environmental Thesaurus, Freebase, Linked Railway Data Project, DBTune
date: 2018-05-10
modified: 2018-06-12
image: linked-open-data.png
---

A group for Linked Open Data datasets. The initial import of data for this group was done in October 2009 from the list of [RDF datasets dumps](https://www.w3.org/wiki/DataSetRDFDumps) provided by the W3C Linked Open Data Interest Group.

[[toc]]

## DBpedia

DBpedia.org is a community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to ask sophisticated queries against Wikipedia and to link other datasets on the Web to Wikipedia data.

The English version of the DBpedia knowledge base currently describes 6.0M entities of which 4.6M have abstracts, 1.53M have geo coordinates and 1.6M depictions. In total, 5.2M resources are classified in a consistent ontology, consisting of 1.5M persons, 810K places (including 505K populated places), 490K works (including 135K music albums, 106K films and 20K video games), 275K organizations (including 67K companies and 53K educational institutions), 301K species and 5K diseases. The total number of resources in English DBpedia is 16.9M that, besides the 6.0M resources, includes 1.7M skos concepts (categories), 7.3M redirect pages, 260K disambiguation pages and 1.7M intermediate nodes.

Altogether the DBpedia 2016-04 release consists of 9.5 billion (2015-10: 8.8 billion) pieces of information (RDF triples) out of which 1.3 billion (2015-10: 1.1 billion) were extracted from the English edition of Wikipedia, 5.0 billion (2015-04: 4.4 billion) were extracted from other language editions and 3.2 billion (2015-10: 3.2 billion) from DBpedia Commons and Wikidata. In general, we observed a growth in mapping-based statements of about 2%.

Thorough statistics can be found on the DBpedia website and general information on the [DBpedia datasets here](https://old.datahub.io/dataset/dbpedia).

## Lexvo

Linguistic Data Size of dump and dataset: ~40MB

Download dump: CC-BY-SA 3.0 license

The web service additionally provides some parts that are not fully open, e.g. English language names taken from the Ethnologue Language Codes database are subject to specific redistribution conditions. Please, see [lexvo dataset](https://old.datahub.io/dataset/lexvo).

## General Multilingual Environmental Thesaurus

A thesaurus in 20+ languages for terms related to the environment and environmental data. Published by the European Environment Agency.

[Available in RDF](https://old.datahub.io/dataset/gemet) without reuse constraints.

## Freebase

### Description

"Freebase is an open database of the world?s information. It is built by the community and for the community?free for anyone to query, contribute to, built applications on top of, or integrate into their websites."

### Openness: OPEN

* License: CC-BY for data. Variety of open source licenses (or PD) for text blurbs & images.
* Access: API + bulk.
  * bulk: yes.
  * api: yes.

Triple count and link statistics provided by Freebase contributor Tom Morris

### RDF data and URIs

Freebase has [an RDF service](https://developers.google.com/freebase/) that exposes URIs and generates RDF descriptions for all Freebase topics.

## Linked Railway Data Project

Bringing together data on the United Kingdom's railway network under Linked data principles.
Please, see [dataset](https://old.datahub.io/dataset/linked-railway-data-project).

## DBTune – Music-Related RDF

This effort has started in the context of the Linked open data community project of the Semantic Web Education and Outreach Interest Group.

Its main purpose is to make available freely available data concerning music on the semantic-web, such as Magnatune, Jamendo, Dogmazic, Mutopia, and to create links between them and other available semantic web repositories, such as Frederick Giasson's Musicbrainz dump, or DBPedia and GeoNames."
Please, see [DBTune – Music-Related RDF](https://old.datahub.io/dataset/dbtune).

## schema.org

Schema.org markup schema to provide structured metadata for websites and beyond. Linked data variant provided by http://schema.rdfs.org/
Please, see [schema.org dataset](https://old.datahub.io/dataset/schema-org)

## The CIA World Factbook

US government profiles of countries and territories around the world. Information on geography, people, government, transportation, economy, communications, etc.

Please, see  [The CIA World Factbook dataset](https://old.datahub.io/dataset/cia-world-factbook)

## U.S. Census

2000 U.S. Census converted into over a billion RDF triples.

Population statistics at various geographic levels, from the U.S. as a whole, down through states, counties, sub-counties (roughly, cities and incorporated towns)

Notes: also found in the of SPARQL Endpoints.

Please, see [U.S. Census dataset](https://old.datahub.io/dataset/us_census_data)

## SIMILE Data Collection

Various data sets including CIA's World Factbook, Library of Congress' Thesaurus of Graphic Materials, National Cancer Institute's cancer thesaurus, Web Consortium's Technical Reports

Please, see [SIMILE Data Collection](https://old.datahub.io/dataset/simile_data_collection)

## OpenVocab

From home page:
>OpenVocab is a community maintained vocabulary intended for use on the Semantic Web
OpenVocab is ideal for properties and classes that don't warrant the effort of creating or maintaining a full schema. OpenVocab allows anyone to create and modify vocabulary terms using their web browser. Each term is described using appropriate elements of RDF, RDFS and OWL. OpenVocab allows you to create any properties and classes; assign labels, comments and descriptions; declare domains and ranges and much more.

Please, see [OpenVocab dataset](https://old.datahub.io/dataset/openvocab)

## The Mondial Database

From home page:

>The MONDIAL database has been compiled from geographical Web data sources listed below:
* CIA World Factbook,
* a predecessor of Global Statistics which has been collected by Johan van der Heijden.
* additional textual sources for coordinates,
* the International Atlas by Kümmerly & Frey, Rand McNally, and Westermann,
* and some geographical data of the Karlsruhe TERRA database.

Please, see [The Mondial Database](https://old.datahub.io/dataset/mondial)

## Linked ISO 3166-2 Data

Linked ISO 3166-2 Data. ISO-3166-2 gives codes for countries and their principal subdivisions.

Please, see [Linked ISO 3166-2 Data](https://old.datahub.io/dataset/iso-3166-2-data)

## Historical Events Markup Language

Historical Event Markup and Linked Project (Heml) provides an XML schema for historical events and a Java Web app which transforms conforming documents into hyperlinked timelines, maps and tables. It aims to provide a most information-rich interchange format for historical data, and thus add a historical component to the growing movement for a 'Semantic Web.'

For our purposes main item of interest is the data provided with the project available from: . About 30 timelines in the form of RDF files on historical themes ranging from Alexander the Great to the Meiji Restoration.

Please, see [Historical Events Markup Language](https://old.datahub.io/dataset/heml)

## A Short Biographical Dictionary of English Literature (RKBExplorer)

About
>This repository contains data processed by Hugh Glaser as a birthday present to Ian Davis.

>It is a very quick and dirty job of processing the data provided by Project Gutenberg from A Short Biographical Dictionary of English Literature by John W. Cousin.

Please, see [A Short Biographical Dictionary of English Literature (RKBExplorer)](https://old.datahub.io/dataset/biolit)

## Billion Triples Challenge Dataset 2008

* Data exposed: various dumps
* Size of dump and data set: 1 billion triples

Please, see [Billion Triples Challenge Dataset 2008](https://old.datahub.io/dataset/billion_triples_challenge_dataset2008)


logistics-data.md

---
title: Logistics Data
description: Ready-to-use logistics datasets - explore, download and use in your tool!
keywords: Logistics Data, Reference Data
date: 2018-06-06
modified: 2018-06-06
image: logistics-data.png
---

Summary list of datasets available on DataHub that are related to Logistics.

[[toc]]

## List of countries

ISO 3166-1-alpha-2 English country names and code elements. This list states
the country names (official short names in English) in alphabetical order as
given in [ISO 3166-1][] and the corresponding ISO 3166-1-alpha-2 code elements:

[core/country-list](/core/country-list)

[ISO 3166-1]: http://www.iso.org/iso/home/standards/country_codes.htm

## Airport codes

Airport codes from around the world:

[core/airport-codes](/core/airport-codes)

## Country codes

Comprehensive country code information, including ISO 3166 codes, ITU dialing codes, ISO 4217 currency codes, and many others:

[core/country-codes](/core/country-codes)

## Continent codes

A list of the seven continents with English names and short, unique and permanent identifying codes:

[core/continent-codes](/core/continent-codes)

## UN Locode

The United Nations Code for Trade and Transport Locations is a code list maintained by UNECE, United Nations agency, to facilitate trade:

[core/un-locode](/core/un-locode)

## ISO 6346 Container Type Codes

Coded list of ISO 6346 shipping containers, used in international trade and electronic shipping messages:

[core/iso-container-codes](/core/iso-container-codes)

## World cities

List of major cities in the world that have above 15,000 inhabitants. Each city is associated with its country and subcountry to reduce the number of ambiguities. Subcountry can be the name of a state (eg in United Kingdom or the United States of America) or the major administrative section (eg ‘‘region’’ in France’’):

[core/world-cities](/core/world-cities)


machine-learning-data.md

---
title: Machine Learning / Statistical Data
description: Examples of machine learning datasets.
keywords: machine learning datasets, examples of ml data
date: 2018-06-06
modified: 2018-06-06
image: machine-learning-data.png
---

Machine learning is used as a general term for computational data analysis: using data to makes inferences and predictions. Interpreted broadly it includes computational statistics, data analytics, data mining and a good portion of data science.

Machine learning algorithms are often categorized as supervised or unsupervised ("data mining").

> * Supervised machine learning algorithms can apply what has been learned in the past to new data using labeled examples to predict future events. Starting from the analysis of a known training dataset, the learning algorithm produces an inferred function to make predictions about the output values. The system is able to provide targets for any new input after sufficient training. The learning algorithm can also compare its output with the correct, intended output and find errors in order to modify the model accordingly.
>
> * In contrast, unsupervised machine learning algorithms are used when the information used to train is neither classified nor labeled. Unsupervised learning studies how systems can infer a function to describe a hidden structure from unlabeled data. The system doesn’t figure out the right output, but it explores the data and can draw inferences from datasets to describe hidden structures from unlabeled data.
>
> * Semi-supervised machine learning algorithms fall somewhere in between supervised and unsupervised learning, since they use both labeled and unlabeled data for training – typically a small amount of labeled data and a large amount of unlabeled data. The systems that use this method are able to considerably improve learning accuracy. Usually, semi-supervised learning is chosen when the acquired labeled data requires skilled and relevant resources in order to train it / learn from it. Otherwise, acquiring unlabeled data generally doesn’t require additional resources.
>
> * Reinforcement machine learning algorithms is a learning method that interacts with its environment by producing actions and discovers errors or rewards. Trial and error search and delayed reward are the most relevant characteristics of reinforcement learning. This method allows machines and software agents to automatically determine the ideal behavior within a specific context in order to maximize its performance. Simple reward feedback is required for the agent to learn which action is best; this is known as the reinforcement signal.
>
> [source](http://www.expertsystem.com/machine-learning-definition/)

## Datasets

There are a variety of machine-learning datasets on the DataHub under the @machine-learning account: https://datahub.io/machine-learning

* Seismic Bumps: https://datahub.io/machine-learning/seismic-bumps. This is a classification problem. The data describe the problem of high energy (higher than 10^4 J) seismic bumps forecasting in a coal mine. Data come from two of longwalls located in a Polish coal mine.


## Existing collections

* [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets.html). 404 datasets.
* [OpenML datasets](https://www.openml.org/search?q=tags.tag%3AOpenML100&type=data&table=1&size=100)
* [Kaggle datasets](https://www.kaggle.com/datasets)
* [Academic Torrents](http://academictorrents.com/)
* [TU Berlin/ MLdata.org](http://mldata.org/repository/data/)
* [AWS Public Datasets](https://aws.amazon.com/public-datasets/)
* [BigQuery Public Datasets](https://cloud.google.com/bigquery/public-data/)


movies-and-tv.md

---
title: Movies and TV
description: Various resources for Movies and TV data
keywords: movies data, tv dataset, imdb
date: 2018-07-21
modified: 2018-07-21
image: movies-and-tv.png
---

## Internet Movie Database

Large film/movie database claiming:

* 425,000+ titles
* 1,700,000 + filmographies of cast and crew members
* Films from 1891 to Present
* Foreign and independent movies, television movies and shows, new and future releases, and more

Information on database structure can be found here: . Individual files are gzipped an in a bespoke text format structure for which standard libraries exist e.g. [IMDbPy](https://imdbpy.sourceforge.io/).

Information on history and development (including 'enclosure') on wikipedia:

### Openness: NOT OPEN

See conditions of usage - https://www.imdb.com/conditions

### Data and Resources

ftp index of all the IMBD dump files (which are themselves gzips of a bespoke plain text format):

ftp://ftp.fu-berlin.de/pub/misc/movies/database/


opencorporates.md

---
title: OpenCorporates - The Open Database Of The Corporate World
description: Open Database of corporate entities.
keywords: access-nobulk, corporations, database, ecommerce, format-rdf, government, lod, lodcloud-diagram-2011-09-19, no-deref-vocab, opendatachallenge, published-by-third-party, scraped, size.xlarge
date: 2018-07-17
modified: 2018-07-17
image: opencorporates.png
---

Open Database of corporate entities. As of 2011-04-09 has information on 7,841,828 companies from around the world. Jurisdictions covered include:

* 41,292 Bermuda
* 3,886,733 United Kingdom
* 96,104 Gibraltar
* 105,640 Isle of Man
* 77,693 Iceland
* 60,827 Jersey
* 92,795 Luxembourg
* 2,188,873 Netherlands
* 97,653 Alaska (US)
* 197,798 District of Columbia (US)
* 996,420 Michigan (US)

There is good API access but currently but no bulk availability.

## License

See https://opencorporates.com/info/licence. However, should note that most data in OpenCorporates is scraped from elsewhere so this license only covers the 'IP' that OpenCorporates has obtained as a result of their efforts (and license of original databases, e.g. Companies House in the UK, is unclear).

## Data and Resources

* [Example JSON record from the API (for Google)](http://opencorporates.com/companies/gb/03977902.json)
* [Example RDF record](http://opencorporates.com/companies/us_ak/124437.rdf)


property-prices.md

---
title: Property Prices
description: Property Prices Datasets available on DataHub.
keywords: house prices, global cross-country property prices, UK house prices, US house prices
date: 2018-06-06
modified: 2018-06-06
image: property-prices.png
---

Property prices including house prices

* "Global" - comparable cross-country from BIS: http://datahub.io/core/house-prices-global - from BIS data http://www.bis.org/statistics/pp.htm
* UK - various data series: http://datahub.io/core/house-prices-uk -
* US - Case/Shiller series: http://datahub.io/core/house-prices-us

Interesting paper with long run time series across the world: http://voxeu.org/article/home-prices-1870 - however no data link afaict.


reference-data.md

---
title: Reference Data
description: Easy-to-use reference data in CSV and JSON formats.
keywords: Country list, S&P 500 companies, Language codes, World cities, Airport codes, Country codes, Currency codes, UN Locode
date: 2018-06-06
modified: 2018-06-19
image: reference-data.png
---

This page is the place where you can find all reference data available on DataHub. Each dataset is easy-to-use and includes instructions on how to use it in different tools and programming languages.

What is reference data? Here is the quote from Wikipedia that answers that question:

> Reference data are data that define the set of permissible values to be used by other data fields. Reference data gain in value when they are widely re-used and widely referenced. Typically, they do not change overly much in terms of definition, apart from occasional revisions. Reference data are often defined by standards organizations, such as country codes as defined in ISO 3166-1... [read full article](https://en.wikipedia.org/wiki/Reference_data)

[[toc]]

## The most popular datasets

In this section, we have included the datasets that are most popular among our users.

### List of countries

ISO 3166-1-alpha-2 English country names and code elements. This list states
the country names (official short names in English) in alphabetical order as
given in [ISO 3166-1][] and the corresponding ISO 3166-1-alpha-2 code elements:

[core/country-list](/core/country-list)

[ISO 3166-1]: http://www.iso.org/iso/home/standards/country_codes.htm

### S&P 500 companies

List of companies in the S&P 500 (Standard and Poor’s 500). The S&P 500 is a free-float, capitalization-weighted index of the top 500 publicly listed stocks in the US (top 500 by market cap). The dataset includes a list of all the stocks contained therein:

[core/s-and-p-500-companies](/core/s-and-p-500-companies)

### Language codes

Comprehensive language code information, consisting of ISO 639-1, ISO 639-2 and IETF language types:

[core/language-codes](/core/language-codes)

### World cities

List of major cities in the world that have above 15,000 inhabitants. Each city is associated with its country and subcountry to reduce the number of ambiguities. Subcountry can be the name of a state (eg in United Kingdom or the United States of America) or the major administrative section (eg ‘‘region’’ in France’’):

[core/world-cities](/core/world-cities)

## Other useful reference data

### Airport codes

Airport codes from around the world:

[core/airport-codes](/core/airport-codes)

### Country codes

Comprehensive country code information, including ISO 3166 codes, ITU dialing codes, ISO 4217 currency codes, and many others:

[core/country-codes](/core/country-codes)

### Currency codes

List of currencies and their 3 digit codes as defined by ISO 4217. The data provided here is the consolidation of Table A.1 “Current currency & funds code list” and Table A.3 “Historic denominations”:

[core/currency-codes](/core/currency-codes)

### Continent codes

A list of the seven continents with English names and short, unique and permanent identifying codes:

[core/continent-codes](/core/continent-codes)

### SMDG Master Terminal Facilities List

List mantained by the [SMDG Secretariat](http://smdg.org/) to specify the port terminal facilities in UN/EDIFACT messages. The list is directly linked with the [UN/LOCODE](#un-locode) codelist:

[core/smdg-master-terminal-facilities-list](/core/smdg-master-terminal-facilities-list)

### IMO IMDG Classification Codes

Official IMDG Codes for use in transport of dangerous goods as described by the IMO:

[core/imo-imdg-codes](/core/imo-imdg-codes)

### IPv4 geolocation

Database of IPv4 address networks with their respective geographical location:

[core/geoip2-ipv4](/core/geoip2-ipv4)

### Top Level Domain Names

This Data Package contains the delegation details of top-level domains:

[core/top-level-domain-names](/core/top-level-domain-names)

### Classification of the Functions of Government

Classification of the Functions of Government (COFOG) is a classification defined by the United Nations Statistics Division:

[core/cofog](/core/cofog)

### UNECE/CEFACT package codes

Coded representations of the package type names used in International Trade (UNECE/CEFACT Trade Facilitation Recommendation No.21):

[core/unece-package-codes](/core/unece-package-codes)

### UNECE Units of measure

Standardised codes from Recommendation 20, mantained by UNECE:

[core/unece-units-of-measure](/core/unece-units-of-measure)

### UN Locode

The United Nations Code for Trade and Transport Locations is a code list maintained by UNECE, United Nations agency, to facilitate trade:

[core/un-locode](/core/un-locode)

### Membership to Copyright Treaties

This data provides the details of the membership by states to WIPO administered treaties on the subject matter of copyright:

[core/membership-to-copyright-treaties](/core/membership-to-copyright-treaties)

### List of FIPS

List of FIPS (Federal Information Processing Standards) region codes:

[core/fips-10-4](/core/fips-10-4)

### Media Types

This dataset lists all the Media Types (MIME types), Media Subtypes, and their file extensions:

[core/media-types](/core/media-types)

### ISO 6346 Container Type Codes

Coded list of ISO 6346 shipping containers, used in international trade and electronic shipping messages:

[core/iso-container-codes](/core/iso-container-codes)

### International Chamber of Commerce Incoterms

International Commercial Terms (‘Incoterms’) are internationally recognised standard trade terms used in sales contracts:

[core/icc-incoterms](/core/icc-incoterms)

### DAC and CRS code lists

The DAC Secretariat maintains various code lists which are used by donors to report on their aid flows to the DAC databases. In addition, these codes are used to classify information in the DAC databases:

[core/dac-and-crs-code-lists](/core/dac-and-crs-code-lists)

### UK Condensed SIC 2007

UK condensed standard industrial classification of economic activities (SIC) 2007 codes:

[core/uk-sic-2007-condensed](/core/uk-sic-2007-condensed)

### S&P 500 Companies with Financial Information

List of companies in the S&P 500 (Standard and Poor’s 500). The dataset includes a list of all the stocks contained therein and associated key financials such as price, market capitalization, earnings, price/earnings ratio, price to book etc.:

[core/s-and-p-500-companies-financials](/core/s-and-p-500-companies-financials)

### NYSE and Other Listings

List of companies in the NYSE, and other exchanges:

[core/nyse-other-listings](/core/nyse-other-listings)

### NASDAQ listings

List of companies in the NASDAQ exchanges:

[core/nasdaq-listings](/core/nasdaq-listings)


stock-market-data.md

---
title: Stock Market Data
description: Find and Explore ready-to-use Stock Market Datasets.
keywords: stock data, s&p 500, finance vix, oil prices, natural gas prices, gold prices, nyse listings, nasdaq listings
date: 2018-06-07
modified: 2018-06-07
image: stock-market-data.png
---

[[toc]]

## S&P 500 companies

List of companies in the S&P 500 (Standard and Poor’s 500). The S&P 500 is a free-float, capitalization-weighted index of the top 500 publicly listed stocks in the US (top 500 by market cap). The dataset includes a list of all the stocks contained therein:

[core/s-and-p-500-companies](/core/s-and-p-500-companies)

## S&P 500 Companies with Financial Information

List of companies in the S&P 500 (Standard and Poor’s 500). The dataset includes a list of all the stocks contained therein and associated key financials such as price, market capitalization, earnings, price/earnings ratio, price to book etc.:

[core/s-and-p-500-companies-financials](/core/s-and-p-500-companies-financials)

## Standard and Poor's (S&P) 500 Index Data including Dividend, Earnings and P/E Ratio

S&P 500 index data including level, dividend, earnings and P/E ratio on a monthly basis since 1870:

[core/s-and-p-500](/core/s-and-p-500)

## VIX - CBOE Volatility Index

CBOE Volatility Index (VIX) time-series dataset including daily open, close, high and low:

[core/finance-vix](/core/finance-vix)

## NYSE and Other Listings

List of companies in the NYSE, and other exchanges:

[core/nyse-other-listings](/core/nyse-other-listings)

## NASDAQ listings

List of companies in the NASDAQ exchanges:

[core/nasdaq-listings](/core/nasdaq-listings)

## Brent and WTI Spot Prices

Europe Brent and WTI (Western Texas Intermediate) Spot Prices (Annual/ Monthly/ Weekly/ Daily) from EIA U.S. (Energy Information Administration):

[core/oil-prices](/core/oil-prices)

## Natural gas prices

Time series of major Natural Gas Prices including US Henry Hub. Data comes from U.S. Energy Information Administration EIA:

[core/natural-gas](/core/natural-gas)

## Gold Prices

Monthly gold prices since 1950 in USD (London market). Data is sourced from the Bundesbank:

[core/gold-prices](/core/gold-prices)


war-and-peace.md

---
title: War and Peace - Inter-state Conflicts, Associated Fatalities etc
description: Data on inter-state conflicts, international relations and other correlates of war including fatalities.
keywords: Uppsala Conflict Data Program (UCDP) dataset, Correlates of War, Brecke Conflict Catalog
date: 2018-02-19
modified: 2018-06-06
image: war-and-peace.png
---

Data on inter-state conflicts, international relations and other correlates of war including war-related deaths and injuries.

[[toc]]

## Uppsala Conflict Data Program (UCDP) dataset

A whole variety of datasets on armed conflicts 1946-present

http://ucdp.uu.se/downloads/

Some of most relevant:

* UCDP/PRIO Armed Conflict Dataset version 17.2 A conflict-year dataset with information on armed conflict where at least one party is the government of a state in the time period 1946-2016.
  * CSV: http://ucdp.uu.se/downloads/ged/ged172-csv.zip
  * CodeBook: http://ucdp.uu.se/downloads/ucdpprio/ucdp-prio-acd-172.pdf


:::info
Note: PRIO dataset https://www.prio.org/Data/Armed-Conflict/Battle-Deaths/ merged into Uppsala Conflict Data Program (UCDP) dataset
:::

## Correlates of War

Correlates of War - http://www.correlatesofwar.org/

> COW seeks to facilitate the collection, dissemination, and use of accurate and reliable quantitative data in international relations. Key principles of the project include a commitment to standard scientific principles of replication, data reliability, documentation, review, and the transparency of data collection procedures. More specifically, we are committed to the free public release of data sets to the research community, to release data in a timely manner after data collection is completed, to provide version numbers for data set and replication tracking, to provide appropriate dataset documentation, and to attempt to update, document, and distribute follow-on versions of datasets where possible.

Dataset on old datahub https://old.datahub.io/dataset/cow (from 2007!)

### Data

* http://www.correlatesofwar.org/data-sets/COW-war
  * CSV http://www.correlatesofwar.org/data-sets/COW-war/intra-state-war-data-v4-1/at_download/file
* Lots of code books!

### License: Not OPEN

Data is under click-wrap with a bunch of restrictions that mean this is **not** [open data][] (last checked 2018-02-19):

* Data are not to be used for any commercial activity; users of COW data agree not to sell or use the data as part of any financial profit-making activity.
* Users agree to cite each of the datasets they use in the manner described on each specific dataset web page.
* Users agree not to distribute the dataset to any third party without written permission of the COW director and data host.
* Users agree to ask permission for any dissemination, posting, or other use of the data that is not covered by the above restrictions.

[open data]: https://opendefinition.org/

## Brecke Conflict Catalog

Conflict Catalog:(Violent Conflicts 1400 A.D. to the Present in Different Regions of the World) Authors: Peter Brecke Contents: 3708 conflicts, data on parties, fatalities, date and duration.

* http://www.cgeh.nl/sites/default/files/Conflict%20Catalog%2018%20vars.xls
* Notes: http://www.cgeh.nl/sites/default/files/Notes%20about%20Conflict%20Catalog.pdf
* Homepage: http://www.cgeh.nl/data#conflict

TODO: data package!


wealth-income-and-inequality.md

---
title: Wealth, Income and Inequality
description: Data and analysis on Wealth, Income and Inequality
keywords: household incomes in the US, income data in the US
date: 2018-06-06
modified: 2018-06-06
image: wealth-income-and-inequality.png
---

Qu: how have incomes of different social groups changed over time?

More specifically: Real income changes by quintile (plus top 5% and top 1%) in last 40+ years in the US?

An: check out this dataset - https://datahub.io/core/household-income-us-historical

## World Inequality Database

https://wid.world/

Download full dataset from https://wid.world/data/

> The World Inequality Database (WID.world) aims to provide open and convenient access to the most extensive available database on the historical evolution of the world distribution of income and wealth, both within countries and between countries
>
> During the past fifteen years, the renewed interest for the long-run evolution of income and wealth inequality gave rise to a flourishing literature. In particular, a succession of studies has constructed top income share series for a large number of countries (see Thomas Piketty 2001, 2003, T. Piketty and Emmanuel Saez 2003, and the two multi-country volumes on top incomes edited by Anthony B. Atkinson and T. Piketty 2007, 2010; see also A. B. Atkinson et al. 2011 and Facundo Alvaredo et al. 2013 for surveys of this literature). These projects generated a large volume of data, intended as a research resource for further analysis, as well as a source to inform the public debate on income inequality. To a large extent, this literature follows the pioneering work of Simon Kuznets 1953, and A. B. Atkinson and Alan Harrison 1978, and extends it to many more countries and years.

## World Income Inequality Database

https://www.wider.unu.edu/database/world-income-inequality-database-wiid4

Excel file: https://www.wider.unu.edu/sites/default/files/WIID/WIID_19Dec2018.xlsx

The World Income Inequality Database (WIID) contains information on income inequality in various countries, and is maintained by the United Nations University-World Institute for Development Economics Research (UNU-WIDER). The database was originally compiled during 1997-99 for the research project Rising Income Inequality and Poverty Reduction, directed by Giovanni Andrea Corina. A revised and updated version of the database was published in June 2005 as part of the project Global Trends in Inequality and Poverty, directed by Tony Shorrocks and Guang Hua Wan. The database was revised in 2007 and a new version was launched in May 2008.

The database contains data on inequality in the distribution of income in various countries. The central variable in the dataset is the Gini index, a measure of income distribution in a society. In addition, the dataset contains information on income shares by quintile or decile. The database contains data for 159 countries, including some historical entities. The temporal coverage varies substantially across countries. For some countries there is only one data entry; in other cases there are over 100 data points. The earliest entry is from 1867 (United Kingdom), the latest from 2003. The majority of the data (65%) cover the years from 1980 onwards. The 2008 update (version WIID2c) includes some major updates and quality improvements, in fact leading to a reduced number of variables in the new version. The new version has 334 new observations and several revisions/ corrections made in 2007 and 2008.

## US

> The most widely used sources of data and statistics on household income and its distribution are the annual survey of households conducted as part of the Census Bureau’s Current Population Survey (CPS) and the Internal Revenue Service’s (IRS) Statistics of Income (SOI) data compiled from a large sample of individual income tax returns.  The Census Bureau publishes annual reports on income, poverty, and health insurance coverage in the United States based on the CPS data,[2] and the IRS publishes an annual report on individual income tax returns based on the SOI.[3]  While the Federal Reserve also collects income data in its triennial Survey of Consumer Finances (SCF),[4] the SCF is more valuable as the best source of survey data on wealth.

> Each agency produces its own tables and statistics and makes a public-use file of the underlying data available to other researchers.  In addition, the Congressional Budget Office (CBO) has developed a model that combines CPS and SOI data to estimate household income both before and after taxes, as well as average taxes paid by income group back to 1979.[5]  Economists Thomas Piketty and Emmanuel Saez have used SOI data to construct estimates of the concentration of income at the top of the distribution back to 1913.[6]  That work has been expanded recently to examine trends in wealth concentration.[7]  CBO and Piketty-Saez regularly release reports incorporating the latest available data. [source](https://www.cbpp.org/research/poverty-and-inequality/a-guide-to-statistics-on-historical-trends-in-income-inequality)

* US Census data: https://www.census.gov/topics/income-poverty/income.html
  * Data tables: https://www.census.gov/topics/income-poverty/income/data/tables.html
  * Historical Income Tables: Households https://www.census.gov/data/tables/time-series/demo/income-poverty/historical-income-households.html
* Internal Revenue Service, “SOI Tax Stats — Individual Income Tax Returns Publication 1304,” multiple years available

Emmanuel Saez and Gabriel Zucman, “Wealth Inequality in the United States since 1913: Evidence from Capitalized Income Tax Data,” Quarterly Journal of Economics, Vol. 131, No. 2, May 2016, http://eml.berkeley.edu/~saez/SaezZucman2016QJE.pdf.


world-bank.md

---
title: World Bank Data
description: An overview of the World Bank data holdings.
keywords: Open Data Catalog, World Development Indicators, DataBank, Open finances, Global Consumption Database, Microdata Library
date: 2018-02-09
modified: 2018-06-06
image: world-bank.png
---

The World Bank has a wide array of high quality data and much of it is [open data][]. It covers several distinct collections and databases.

* [Open Data Catalog][] - a listing of available World Bank datasets. You can browse from https://data.worldbank.org/

* [World Development Indicators][] - It includes over 15 major databases with more than 2000 indicators.

* [DataBank][] - collections of time series data on a variety of topics. Total: 69 databases.

* [Open finances][] - World Bank Group’s financials datasets. There are 93 datasets in 8 categories and one dataset not belonging in any of these categories
  * [Loans and credits][] - 5 datasets
  * [Financial Reporting][] - 33 datasets
  * [Shareholder Equity][] - 4 datasets
  * [Procurement][] - 4 datasets
  * [Projects][] - 9 datasets
  * [Budget][] - 3 datasets
  * [Trust Funds and FIFs][] - 12 datasets
  * [Other][] - 23 datasets

* [Projects & operations][] - information on all of the World Bank's lending projects since 1947. Around 13,119 projects in 173 countries.

* [Global Consumption Database][] - datasets on household consumption patterns in developing countries. Data can be derived by country, sector or product.

* [Microdata Library][] - collection of sample surveys of households, business establishments or other facilities. As of August 28, 2018 the Library contains 2,696 surveys and 2,577 citations.

[Open Data Catalog]: http://datacatalog.worldbank.org/
[World Development Indicators]:https://data.worldbank.org/products/wdi
[DataBank]:http://databank.worldbank.org/data/home.aspx
[Open finances]:https://finances.worldbank.org/
[Loans and credits]:https://finances.worldbank.org/browse?category=Loans+and+Credits&limitTo=datasets
[Financial Reporting]:https://finances.worldbank.org/browse?category=Financial+Reporting&limitTo=datasets
[Shareholder Equity]:https://finances.worldbank.org/browse?category=Shareholder+Equity&limitTo=datasets
[Procurement]:https://finances.worldbank.org/browse?category=Procurement&limitTo=datasets
[Projects]:https://finances.worldbank.org/browse?category=Projects&limitTo=datasets
[Budget]:https://finances.worldbank.org/browse?category=Budget&limitTo=datasets
[Trust Funds and FIFs]:https://finances.worldbank.org/browse?category=Trust+Funds+and+FIFs&limitTo=datasets
[Other]:https://finances.worldbank.org/browse?category=Other&limitTo=datasets
[Projects & operations]:http://projects.worldbank.org/
[Global Consumption Database]:http://datatopics.worldbank.org/consumption/
[Microdata Library]:http://microdata.worldbank.org/catalog
[open data]: https://opendefinition.org/
[featured indicators page]:https://data.worldbank.org/indicator?tab=featured

[[toc]]

## Data

We have created [world-bank](https://datahub.io/world-bank) user specifically for World Bank datasets so they can all be found in one place.

### World Development Indicators

One of the major open ones are the [World Development Indicators][] which provide a variety of important indicators ranging from GDP and population to CO2 emissions and literacy in a standardized cross country form. We added 308 datasets from [featured indicators page][] to datahub. These datasets can be found in several main categories:

* Agriculture & Rural Development

* Aid Effectiveness

* Climate Change

* Economy & Growth

* Education

* Energy & Mining

* Environment

* External Debt

* Financial Sector

* Gender

* Health

* Infrastructure

* Poverty

* Private Sector

* Public Sector

* Science & Technology

* Social Development

* Social Protection & Labor

* Trade

* Urban Development

Some of the most popular world-bank indicators featured on our website:

* Total population of various world regions based on the de facto definition of population, which counts all residents regardless of legal status or citizenship. The values shown are midyear estimates https://datahub.io/world-bank/sp.pop.totl

* Cause of death, by non-communicable diseases (cancer, diabetes mellitus, cardiovascular diseases, digestive diseases, skin diseases, musculoskeletal diseases, and congenital anomalies) https://datahub.io/world-bank/sh.dth.ncom.zs

* Prevalence of HIV, female (% ages 15-24) https://datahub.io/world-bank/sh.hiv.1524.fe.zs

## API

The World Bank has a data API. You can use this to access a variety of data including the world development indicators.

You can get data in CSV, JSON and XML (default).

Per indicator:

```bash
https://api.worldbank.org/indicator/GC.DOD.TOTL.GD.ZS?format=csv

# for some reason json format just yields metadata
https://api.worldbank.org/indicator/GC.DOD.TOTL.GD.ZS?format=json
```

More elaborate queries documented in http://blogs.worldbank.org/opendata/first-steps-in-integrating-open-data, for example:

http://api.worldbank.org/en/countries/KE;XF;XM/indicators/EN.ATM.CO2E.PC?date=1961:2011&format=csv

http://api.worldbank.org/en/countries/KE;XF;XM/indicators/EN.ATM.CO2E.PC?date=1961:2011&format=json

### Scripts

Here is a script that automates access to World Bank data and conversion to data packages.

https://github.com/rufuspollock/world-bank-data


## License

License: OPEN (mostly)

There is currently no easy way to automatically obtain that listing though.

As of April 20th 2010 data is now open (subject to some reservations for specific datasets). See this [blog post](http://blog.okfn.org/2010/04/20/world-bank-opens-up-development-data/).

### License Details

[Terms of use summary](http://data.worldbank.org/summary-terms-of-use) state:

> You are free to copy, distribute, adapt, display or include the data in other products for commercial and noncommercial purposes at no cost subject to certain limitations summarized below.
>
> You must include attribution for the data you use in the manner indicated in the metadata included with the data.
>
> You must not claim or imply that The World Bank endorses your use of the data by or use The World Bank’s logo(s) or trademark(s) in conjunction with such use.
>
> Other parties may have ownership interests in some of the materials contained on The World Bank Web site. For example, we maintain a list of some specific data within the Datasets that you may not redistribute or reuse without first contacting the original content provider, as well as information regarding how to contact the original content provider. Before incorporating any data in other products, please check the list: Terms of use: Restricted Data
>
> The World Bank makes no warranties with respect to the data and you agree The World Bank shall not be liable to you in connection with your use of the data.


yago.md

---
title: The DBLP Computer Science Bibliography
description: YAGO3 is a huge semantic knowledge base, derived from Wikipedia WordNet and GeoNames.
keywords:   YAGO, ckanupload.esw.200910, crossdomain, format-rdf, linkeddata, lod, lodcloud-diagram-2011-09-19, lodcloud-diagram-2014-08-30, no-deref-vocab, no-license-metadata, no-provenance-metadata, ontology, published-by-producer
date: 2018-07-17
modified: 2018-07-17
image: yago.png
---

YAGO3 is a huge semantic knowledge base, derived from Wikipedia WordNet and GeoNames. Currently, YAGO3 has knowledge of more than 10 million entities (like persons, organizations, cities, etc.) and contains more than 120 million facts about these entities.

## Data and Resources

* [The entire YAGO in RDF/TTL/Turtle format](http://resources.mpi-inf.mpg.de/yago-naga/yago3.1/yago3.1_entire_ttl.7z)
* [The entire YAGO in TSV format](http://resources.mpi-inf.mpg.de/yago-naga/yago3.1/yago3.1_entire_tsv.7z)
* [Schema of YAGO in TTL/Turtle/RDF](http://resources.mpi-inf.mpg.de/yago-naga/yago3.1/yagoSchema.ttl.7z)
* [rdf:type facts of YAGO in RDF/TTL/Turtle](http://resources.mpi-inf.mpg.de/yago-naga/yago3.1/yagoTypes.ttl.7z)
* [Taxonomy of YAGO in RDF/TTL/Turtle](http://resources.mpi-inf.mpg.de/yago-naga/yago3.1/yagoTaxonomy.ttl.7z)

Go to the Web page of YAGO to check individual downloads: [Simplified taxonomy, multilingual, links to DBpedia, geonames, WordNet](http://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/yago/downloads/). These include:

* TAXONOMY: All types of entities, and the class structure of YAGO2s. Moreover, it has formal definitions of YAGO relations.
* SIMPLETAX: An alternative, simpler taxonomy of YAGO.
* CORE: Core facts of YAGO2s, such as the facts between entities, the facts containing literals, i.e., numbers, dates, strings, etc.
* GEONAMES: Geographical entities, classes taken from GeoNames.
* META: Temporally and spatially scoped facts together with statistics and extraction sources about the facts.
* MULTILINGUAL: The multilingual names for entities.
* LINK: The connection of YAGO2s to Wordnet, DBPedia, etc.
* OTHER: Miscellaneous features of YAGO2s, such as Wikipedia in-outlinks, etc.


Topics