Historically, we have invested in statistical stovepipes, systems to deliver a statistic but prevent integration and reuse, says Chris Buxton, chief digital officer at Stats NZ.

Buxton says his team was set up to support large scale development along these lines, but lacked the ability to leverage enterprise platforms.

Working on the mantra of “unleashing the power of data to change lives”, he says the team established a theme in their Information Systems Strategic Plan around data access and integration. 

“The vision was to create the ability for analysts to access and integrate data as they needed, when they needed, unhindered by the technology.”

This was the backdrop for the creation of Datahub.

The new Datahub is a centralised data management platform to store, verify, and analyse data, he explains.

The Datahub, based on the Cloudera technology stack, feeds data into new and existing statistical and analytic systems, such as R, SAS PowerBI, and Python.

Data storage costs are reduced by moving from multiple data storage systems to a single, centralised storage system.

Buxton says errors in data are detected early through an automated validation and checking systems that also extracts relevant metadata and prepares the data for use. 

With no restrictions on how data is formatted, the Datahub provides increased flexibility around how data is stored and used. 

“It’s important to note the Datahub isn’t a database or tool, but a new framework for management and using data,” he says.

“As we explored this organisational need we had multiple avenues open to us.”

Buxton, whose role evolved from CTO to Chief Digital Officer, says, “We chose to take a bold step to reimagine data management, integration and analytics at Stats NZ and invest in an innovative approach.”

The concept, he adds, was to provide a ubiquitous capability to support both the traditional statistical production process, enabling innovation through data, while keeping costs manageable.

Stats NZ’s Integrated Data Infrastructure, a research database holding microdata about people and households across New Zealand, is Datahub’s first tenant. 

“The Datahub enables Stats NZ to take ownership and empower our users of our data management by monitoring and enabling others to access data,” he says.

Stats NZ is also able to implement effective data governance by easily reporting and auditing data. Tracking enables understanding of where, when, why, and how data is used, he states.

“They use data to drive innovation and insights, while sustaining the integrity of the Statistical pipeline,” he says.

The Datahub also includes built-in and mandatory data encryption, data security, user-access management, and full data and user auditing. 

Full data tracking and data lineage is provided, and all data access attempts (successful and unsuccessful) are tracked and reported. 

Buxton explains the data brokers and custodians now have full visibility (through interactive MicroSoft PowerBI reports) of how data is loaded, processed and accessed. 

“The innovative use of technology, enhanced through our own development capabilities, is transforming the value we can gain from data, and enhancing the data services we provide for New Zealand.”

He says while Stats NZ was not the first organisation to attempt to leverage this technology, he believes its implementation is unique in the manner it’s creating a standardised data management pipeline in a data driven statistical agency.

“The diversity of data being ingested will enable a rich innovation space for Stats NZ to explore data insights, while the use of data projections will maintain the integrity of our traditional statistical mission,” he states.

“This approach makes our implementation unique.”

The implementation also enables the use of machine learning and AI in the standardised pipelines to improve efficiency and effectiveness. 

The incorporation and learning from these technologies is progressing and we are already seeing some benefits around metadata extraction and processing

‘Be comfortable with risk’

Concurrent with this is Buxton’s focus on ensuring a culture that is in step with an increasingly digital environment.

In the past year, he initiated DBS 2.0, which aims to reshape the culture at Stats NZ to an agile mindset.

“We recognised we did not have the right digital culture to maximise the benefits of our new ‘IT as a service’ world,” says Buxton.

DBS 2.0 stems from the Digital Business Services (DBS), which was established in mid-2016. The goal was to build a team that could go further than the delivery of technology, and build an open, collaborative and innovative information technology organisation, one that would deliver new approaches and methodologies to the way data is used.  

To build an open, collaborative and innovative information technology organisation, with teams centred around services and outcomes, he says they used a clear set of organisational design principles, to develop the design of the overall organisation structure of the new DBS, in particular the service lines and DevOps team groupings. 

DBS 2.0 was empowered to run with these principles and help shape a culture to leverage the ITaaS environment and help us maximise the benefits of cloud.

“Reflecting on what we have done in the last three years we’ve come a long way in achieving this goal, but cultural work is never finished” he says.

Today, he says, management focuses on different metrics and KPIs. These include:

Embracing transparency: “Embedding a digital change for us meant everyone in the company - regardless of role - is aware of the impact digital can have on themselves and productivity,” says Buxton.

“We achieved this by ‘working out loud’”.

This entails managers using social mechanisms (Yammer) to talk about what they are working on and what is on their minds. They also held regular “town hall” meetings and team showcases so people could see what others were doing.

Encouraging collaboration: “When our employees enjoy spending time and working together, the workflow improves considerably,” says Buxton. “Ideas fly around the room at high speeds, and the progress is measured in real time.”

This was achieved by co-location and the use of collaboration capabilities, like MS Teams and Azure.

Being comfortable with risk: “Risk doesn’t have to be a scary concept,” he says.

“The beauty of digital is it lends itself to experimentation and the wins can be built on to bring great rewards to a business.”

Buxton says the team published an open dashboard. “Rather than worry about the detail, we monitor the overall performance and manage risk through that. This empowers the teams to deliver with a performance expectation measured through real performance data.”

Giving staff a voice: “The squeaky wheel tends to get the oil,” says Buxton.

“That voice had tended to prevent change in the past and often undermined initiatives through incorrect information.”

Thus, the team tapped collaboration tools to provide everyone a voice and a chance to contribute to discussions.

“This has had a positive impact to enable us to see beyond the ‘noise’ but also enable those more open to change to have a greater contribution,” he states.

“We stopped focusing on the negativity and focused on those already more engaged and wanting to make positive contributions. That has encouraged more to turn negatives around into a positive opportunity.”