In this post, Cameron Hurst, Data Engineer at the UK Data Service, shares the inside information about the International Energy Association (IEA) datasets now available to users, and the process of making this data accessible.
Last week the Census Aggregate and International Macrodata team in the UK Data Service, based in Jisc, reached a milestone in their work to provide IEA data to our users. Working hard over the first few months of the year to keep up with a significant amount of UK Census data releases, the team has also been hard at work to get these significant international datasets into their .Stat platform.
International Data in the Service collection
One of the UK Data Service’s major resources is that of international socio-economic macrodata, vast amounts of data provided by a selection of gold standard international agencies to be made use of by the UK’s higher and further education institutes. The library of data covers a wide variety of subjects, from environmental data provided by the Organisation for Economic Cooperation and Development (OECD) to energy production information provided by the IEA and even balance of payments statistics from the International Monetary Fund (IMF). This wide selection of data makes the international offerings of the UKDS very useful to a wide range of fields.
We aim to provide this data to be openly and freely available to any user in a selection of formats. The OECD’s Main Economic Indicators is a great example of this, providing a wide range of indicators on recent economic developments in OECD member countries and a selection of key non-member economies as well.
The IMF’s International Financial Statistics is another prominent dataset made available to our users, containing data on over 200 countries for over 1500 separate indicators of financial statistics such as exchange rates, international liquidity and interest rates, some of the datasets reach back as far as 1948 providing a wealth of historical research as well as up-to-date statistics.
IEA World Energy Data
In addition to offering these open datasets we also provide free access for UK HE and FE institutions to data from providers such as the IEA and the United Nations Industrial Development Organization (UNIDO), data providers who normally charge to access their data.
Over the past 6 months we have been releasing the latest IEA datasets and it’s great to be able to say that we have now got all the 2022 datasets out, a total of 74 different datasets and metadata covering everything in the world of energy production and consumption.
The following datasets are now available and can be accessed via the links or through searching on the UK Data Service catalogue:
- The IEA Greenhouse Gas Emissions records data on CO2 and other greenhouse gases being emitted from the production and consumption of energy each year.
- The IEA Renewables Information provides key figures on balance, supply, consumption, exports and imports of energy from renewable sources for countries across the globe.
- The IEA Coal Information holds data on the usage, supply, exports and imports of various coal products for many countries.
- The IEA Electricity Information records data regarding electricity generation, supply and consumption, exports and imports and monthly supply.
- The IEA Natural Gas Information holds data regarding natural gas usage along with details on imports, exports, supply and consumption for each country data is made available for.
- The IEA Oil Information provides data on the usage of various oil products including conversion factors, imports, exports, supply, consumption and demand.
The IEA Energy Efficiency Indicators provides detailed analysis of drivers of energy consumption, split down into sectors such as industrial, residential, transport and services. - The IEA Energy Prices and Taxes provides not only data on the end use price of energy usage split by currencies and sectors but also the spot market and crude oil costs for such usages.
- The IEA Energy Technology Research and Development dataset covers the budgets of countries work into energy research split into various factors such as budget per region, budget per thousand units of GDP and a summary of such data.
- The IEA Projections – Energy Policies of IEA Countries contains data projecting into the future – up to 2050 – on items such as the coal supply, economic indicators and energy balances.
- The IEA World Energy Balances holds data on a wider selection of world countries, rather than just IEA member countries – regarding energy balances, conversion factors and indicators.
- The IEA World Energy Prices datasets contain data for energy prices on a wider selection of countries than are available in the Energy Prices and Taxes dataset. These data contain monthly oil prices and taxes and subnational transport costs.
- And finally the IEA World Energy Statistics contains energy statistics, conversion factors and oil demand for a wide selection of countries.
Making the data accessible
Each of these datasets are made available by their corresponding provider in a raw format that we then take and shape into a format for our platform, allowing us to give users the ability to subset exactly which data they wish to get out right in the browser, rather than having to download entire sets of data and then remove all of the unwanted data they might not need for their purposes. As part of this process we include a a robust Quality Assurance process to make sure that the data going in to the system is of the highest quality and a sanity check for each individual release to make sure we are providing the proper data to our users.
If you are interested in seeing what data is available, you can browse the entire collection via our catalogue. Additionally, if you are a member of a HE or FE institution you can log into the website using your institutional credentials and gain access to those non-open datasets mentioned earlier.
What next?
Currently we have a significant focus on releasing ‘Census 2021’ data for England, Wales and Northern Ireland as it is made available by the National Statistical Institutes. These releases will likely continue throughout 2023. On top of that we also expect to see the first releases of Scottish 2022 census data, this census was chosen to be held a year later due to the effects of COVID-19 and subsequent lockdowns.
However, we will still be aiming to deliver new releases of key international macrodata at the same time. We hope to be able to also get out new editions of the latest IMF data, along with key OECD data seeing updates where our schedule allows.
And of course when the IEA begin releasing their data for the 2023 period we will be once again moving to get them available to you all as soon as we can.
In addition we are continuing to upgrade and improve our platforms throughout this year and beyond, having recently rolled out such an update to our Census aggregate data platform which we’ll be sharing information on in a further blog post in the coming weeks.
We are always keen to meet the needs of users, whether it be bugs or issues found or positive changes you notice or improvements you’d like to see. You can contact us via this form to submit any queries you might have.
About the author
Cameron Hurst is a Data Engineer working in the Census Aggregate and International Macrodata team for the UK Data Service, which is based in Jisc. He works on the ingesting of international data to our platforms.