State agencies generate a massive amount of data. This Midwest state generates about 4 petabytes of data, spread across 1,600 databases and 120 agencies, boards, and commissions, and its governor recognized a need to leverage all this data. He tasked an agency to unlock the power of the state’s data resources by removing the siloes and leveraging advanced data analytics.
Although a massive amount of data was available, it was extremely underutilized and used almost exclusively for traditional reporting with no analytics within each agency. Each agency had access solely to their data, and there was no precedent or process for sharing data across agencies. Without insights from data analytics or data from other agencies to create a more holistic picture, each agency was left stifled and unable to use even their own data in a meaningful way.
And although the solution might sound simple — “just share the data!” — it is far more complicated than that, especially for the highly regulated industries within the state government. Privacy concerns and implications of data sharing were huge concerns that also had to be addressed and had kept the state from pursuing similar initiatives in the past.
The state was also burdened with the reality of its current technology landscape, as these agencies are not known for having cutting-edge technology platforms and highly customized solutions. Instead, they were saddled with legacy platforms with long-term contracts or uphill battles to institute change.
Although there were significant roadblocks to consider, the governor wanted to push forward. He maintained that unlocking data was imperative to allow the state to identify and drive meaningful social change and tackle complex problems facing residents’ health, security, and well-being.
This agency engaged with Fusion to help establish a data sharing and analytics platform that would allow the state to unlock its data’s full potential through analytics.
We worked with the state agency to develop a robust data-sharing platform that would allow the agencies to share data securely. We created a big data platform that allowed the different agencies to share resources, tools, and commons services to execute their analytical use cases. Developing the platform included:
- Enabling platform administration services for care and feeding of the platform
- Establishing data governance processes and policies, including data lineage, metadata management, data quality assessments, and usage monitoring
- Defining a data governance program
- Configuring security services to address concerns around security and privacy, including authorization, authentication, auditing/monitoring, and encryption
- Developing deidentification capabilities to protect critical data
- Defining the framework that allowed for shared capabilities for ingestion and analytics, along with the flexibility to enable agencies to integrate uniquely with their internal systems
- Establishing policies, procedures, and data sharing agreements to allow agencies to best manage their data
- Defining policies to enable agencies to launch and solicit new projects
- Defining the lifecycle for onboarding data that includes planning, ingestion, governance, processing, enrichment, and consumption
- Socializing, evangelizing, and consulting with different agencies to help them leverage this analytics platform with their use cases