Dave Rawnsley, Senior Technical Co-ordinator for the UK Data Service, gives an overview of a new data platform that will make it easier to access International and Census data.
In case you didn’t know, DotStat was the name of our old International Data platform. It’s a piece of software developed by the SIS-CC, a collaboration of statistical organisations around the world such as the OECD, UNICEF, Stats NZ, Statistics Tunisia, the International Labour Organization and for a long time the UK Data Service.
The SIS-CC drive the development of the DotStat platform, support the standardization of statistical initiatives, advance the reporting of the UN’s Sustainable Development Goals and provide training in data modelling.
Recently the UK Data Service’s team based at Jisc has been working to upgrade our version of Dotstat to the latest release and with that we can do so much more.
DotStat used to just be a single web application to deliver data, but it is now a suite of programs to create, manage and disseminate data with the use of SDMX from the start to the finish. SDMX is a standard way of describing statistical data and metadata and a way of exchanging that information and allowing it to be shared across statistical organisations. This allows our data to be better described, to be interchangeable, to have higher quality and therefore be more relevant to our users.
Search and Browse
Because all our data is now described in SDMX we can search for keywords across all our data holdings, a search for “Age” currently throws up hits from the OECD Economic Outlook, World Bank Development Indicators, three IMF datasets and the UK Census for 1971 and 2021.
Alternatively, you can browse the datasets available by each provider or for census data you can look at the different topic categories across multiple censuses.
What can Data Explorer do?
Present the metadata
All our datasets in the Data Explorer come with an overview page that tell you the edition details, how to cite the dataset, links to source metadata and any terms and conditions of use.
Our census data overviews have links to the dimension definitions and any quality notices that the Census suppliers have created. We also have links to geography lookups, official documentation and the original census questionnaire forms.
Present the data
All the data in the Data Explorer is presented in a tabular format:
You can change between dimension names, labels or both.
You can change the layout of the table, so if you prefer your along the top row and your years as columns, Data Explorer can do that for you.
If you want to change the number of years in your data, we can change the start year, end year or go for a ‘last xxxx years’ approach. All we need to do is alter the filters for the data:
We can also change any of the other filters from the default, to add in or take out other countries or to change the subject under review for those large datasets that contain numerous subjects.
Data Visualisation
Data Explorer can create a number of different charts based on the type of data you select.
Data Explorer provides ‘Chart’ functionality which allows some primary Exploratory Data Analysis (EDA) to be executed by visualisation. Charts can be further modified with colour highlights and by editable labels. Charts can then be downloaded as PNG format.
Download the data
We know you really want to get your hands on the data to analyse it in your own allocations on your own computers, so we offer a simple choice of downloads – Excel (.xlsx) or Comma Separated Variable (csv) in either filtered or unfiltered forms.
The API
If you know what an API is then it will be very easy to use the one built into Data Explorer. The API allows program access to the data with queries built using URL calls. All our API calls are built on the SDMX standard with various response formats including JSON. Take a look at our API documentation for more information.
There is also integration with PowerBI, an SDMX PowerBI connector is available online for those wanting to create an automated link with which to drive their data analysis.
What’s next?
We’re currently loading in all our 2021 census data and will be doing that for the next few months.
Then we’ll be moving on to historical census data from 2011 to 1981 to go with the 1971 data we already have in the Data Explorer.
On the international data front, we will be restarting our rolling calendar of updates plus looking for new sources of data that can complement our existing range of suppliers.
On the functionality front, we are hoping to have a mapping option to allow the creation of maps that have data overlays.
We are also pushing for ways to incorporate huge datasets that will allow the Data Explorer to deal with census datasets at the lowest geographical levels.
If you have any thoughts or comments on our new platform, you can email the team.
About the author
Dave Rawnsley is Senior Technical Co-ordinator for the UK Data Service.