Federal agencies are challenged with managing the tidal wave of data that is required for enhancing decision-making and operational performance. Hadoop is emerging as a key solution for managing this data in accelerated and more effective ways.
GovTransformer had the opportunity to speak with Audie Hittle, Federal Chief Technology Officer (CTO), Emerging Technologies Division, EMC Corporation, who discussed this new frontier in big data analytics for government.
GovTransformer: Tell us about some of the recent trends with regards to Hadoop and the federal government.
Hittle: Absolutely. The trends that we are seeing in the federal government are running parallel with what we are seeing in the commercial sector, which is that data creation is growing at an astounding rate.
A recent IDC report showed that, from 2005 to 2020, the rate of data creation and analysis will grow by a factor of 300, which is essentially from 130 exabytes to 40,000 exabytes, or 40 trillion gigabytes. This translates into more than 5,200 gigabytes for every man, woman, and child in 2020.
With all of this data growth, there is an increased need for new ways to manage structured and unstructured data, and leverage the information to better support mission goals.
According to a recent Transparent Market Research study, the global Hadoop market was worth $1.5 billion in 2012 and is expected to reach $20.9 billion by 2018.
The federal government falls right in line with this. We are seeing a high level of interest from federal agencies for Hadoop solutions. Although there has been an increase in Hadoop pilot programs, the overall adoption rate has been slow. I believe this is due to the confusion over which distribution providers are the most effective. In addition, there is a perception that these kinds of solutions are expensive.
However, there are ways for leveraging Hadoop solutions in more efficient and cost-effective ways.
GovTransformer: Can you provide some examples of how Hadoop can help agencies best support their mission goals and requirements?
Hittle: There are plenty of great examples to discuss – from healthcare to the defense and intelligence arenas. From the healthcare side, the use of Hadoop can significantly cut down on fraud and false benefit claims. In fiscal year 2013 alone, the Medicaid program covered about 71.7 million individuals at a cost of $431.1 billion – the Centers for Medicare and Medicaid (CMS) estimated that $14.4 billion were improper payments. This is astounding.
By using Hadoop, agencies can sift through all of the payment data to easily and quickly find discrepancies, hundreds of times faster than before. It is also estimated that there is a trillion dollar hidden economy that can develop thanks to using these types of solutions for uncovering improper payments and other big data analytics applications.
At the GEOINT Conference last year, we saw many intelligence leaders discussing how Hadoop and big data analytics will help provide the real-time actionable information that is needed for supporting mission goals and requirements.
GovTransformer: Tell us about EMC’s efforts in this area.
Hittle: Certainly. EMC is a leader in data storage and information management, and we provide the most innovative solutions for leveraging big data analytics, as well as accelerating the usage of Hadoop.
Our Isilon Scale-out Storage Solution for Hadoop combines a highly efficient storage platform with native Hadoop integration. This solution allows federal agencies to accelerate analytics, gain new flexibility, and avoid the costs of a separate Hadoop data storage infrastructure.
We can help agencies leverage Hadoop deployments that reduce time, as well as allow customers to take advantage of technologies that are already in place. By leveraging their existing technologies and our rapid implementation guidelines, Hadoop deployments can actually be up and running in a matter of hours.
GovTransformer: Where is government headed in 2015 when it comes to big data and storage?
Hittle: This is the old-fashioned “crystal ball” question. To quote Yogi Berra, “It’s tough to make predictions, especially about the future.”
That said, I believe that 2015 is going to be an exciting and fast-paced year when it comes to big data and storage for the federal government. We will see an expanded need for these kinds of solutions as agencies truly embrace big data for enhancing operational performance based on demonstrated efficiencies and examples.
In addition, the cyber threat landscape will keep growing, which will further underscore the need for data-driven models of operation. I believe that Hadoop will be critical for helping agencies meet these expanded needs.
GovTransformer: Anything else to add?
Hittle: I would like to thank GovTransformer for giving me the opportunity to share these insights. My parting thought would be that I firmly believe that Hadoop is generating a situation that could positively disrupt the federal government.
We would like to thank Audie for taking the time to speak with GovTransformer. To learn more about EMC, click here.