State leverages analytics to improve quality of life for residents

Challenge

State agencies generate a massive amount of data. This Midwest state generates about 4 petabytes of data, spread across 1,600 databases and 120 agencies, boards, and commissions, and its governor recognized a need to leverage all this data. He tasked an agency to unlock the power of the state’s data resources by removing the siloes and leveraging advanced data analytics.

Although a massive amount of data was available, it was extremely underutilized and used almost exclusively for traditional reporting with no analytics within each agency. Each agency had access solely to their data, and there was no precedent or process for sharing data across agencies. Without insights from data analytics or data from other agencies to create a more holistic picture, each agency was left stifled and unable to use even their own data in a meaningful way.

And although the solution might sound simple — “just share the data!” — it is far more complicated than that, especially for the highly regulated industries within the state government. Privacy concerns and implications of data sharing were huge concerns that also had to be addressed and had kept the state from pursuing similar initiatives in the past.

The state was also burdened with the reality of its current technology landscape, as these agencies are not known for having cutting-edge technology platforms and highly customized solutions. Instead, they were saddled with legacy platforms with long-term contracts or uphill battles to institute change.

Although there were significant roadblocks to consider, the governor wanted to push forward. He maintained that unlocking data was imperative to allow the state to identify and drive meaningful social change and tackle complex problems facing residents’ health, security, and well-being.

This agency engaged with Fusion to help establish a data sharing and analytics platform that would allow the state to unlock its data’s full potential through analytics.

Solution

We worked with the state agency to develop a robust data-sharing platform that would allow the agencies to share data securely. We created a big data platform that allowed the different agencies to share resources, tools, and commons services to execute their analytical use cases. Developing the platform included:

Enabling platform administration services for care and feeding of the platform
Establishing data governance processes and policies, including data lineage, metadata management, data quality assessments, and usage monitoring
Defining a data governance program
Configuring security services to address concerns around security and privacy, including authorization, authentication, auditing/monitoring, and encryption
Developing deidentification capabilities to protect critical data
Defining the framework that allowed for shared capabilities for ingestion and analytics, along with the flexibility to enable agencies to integrate uniquely with their internal systems
Establishing policies, procedures, and data sharing agreements to allow agencies to best manage their data
Defining policies to enable agencies to launch and solicit new projects
Defining the lifecycle for onboarding data that includes planning, ingestion, governance, processing, enrichment, and consumption
Socializing, evangelizing, and consulting with different agencies to help them leverage this analytics platform with their use cases

Outcomes

The state can now share data between agencies to create a more comprehensive information set and pull valuable insights from the newly available analytics. The different agencies can also rest easy with the assurance that their data is being shared privately and securely, maintaining compliance with regulatory requirements.

With updated technology and business processes, the agencies can operate in a secure, flexible, and scalable way as they continue to build out use cases and include additional agencies, boards, and commissions. Also, with Cloudera’s Hadoop distribution, they can customize the platform if their needs change in the future.

The state has already seen a significant impact on the lives of its residents from the data analytics solution, including valuable insights for big data and analytics projects, including:

Infant Mortality (Life Sciences/Public Health)
Opioid and Criminal Justice (Life Sciences, Crime/Corrections, and Recidivism)
Workers Compensation Managed Care (Life Sciences, Insurance/Risk Management)
DODD Claims Fraud (Govt Process Automation, Waste/Fraud Abuse)

Now, the state can use their data to drive meaningful social and economic change and deliver services to those most in need — identifying inefficiencies, optimizing programs, creating jobs and economic growth, adjusting policies, and predicting and preparing for otherwise unexpected events and challenges — a win for both the state and its residents.