Spend much time around data storage discussions and you’re likely to end up contemplating a version of the chicken-or-the-egg question: Did the rise of new kinds and amounts of data lead to the demand for new storage, or did the promise of new storage open up possibilities for new kinds and amounts of data?
In a recent commentary, Ari Berman, Ph.D., vice president and general manager of consulting at BioTeam, Inc., a life science informatics and bio-IT consulting service, noted that the practice of designing experiments to output only the most relevant data (has) shifted to the general sentiment that researchers should collect all information, regardless of its direct relevance.”
What drove the change? The arrival on the scene of Big Data.
According to Berman, “Big Data promised to enable computer-aided discoveries that could not be anticipated by careful planning of experiments, suggesting that humans alone were not capable of making the discoveries of the 21st century. Well-designed algorithms, analytics platforms, and a large amount of computing power would yield new discoveries that weren’t part of the original hypotheses. Big Data drove the plausibility of this hypothesis-generating form of research into overdrive.”
Viewed more specifically from the perspective of data, IT and data storage managers have had to accept that fact that storing data, including health data, is no longer enough.
To that end, tech writer Drew Robb at Datamation, recently spoke to a number of data management stakeholders about how data is increasingly being viewed as a source for mining new insights.
As Will Hayes, CEO of Lucidworks, a data management provider, put it to Robb, successful data implementation isn’t just about providing access and arrange data in tidy dashboards. “It is about insight, and how to analyze rapidly as the volume of data growth accelerates. The quantity of data, then, means traditional storage systems need help. Machine learning is being looked to as the answer to this problem.”
Back on the research side, Ari Berman explains that “for one of the first times in human history, the promise of scientific computing and the ability to find clues in data that were otherwise unfindable, created a revolution in how research was done. Collect as much data on a subject as possible, save it all, analyze it in bulk, find the needle in the haystack, wipe hands on pants, publish, profit, repeat.”
In his view, infrastructure abstraction paradigm shifts, as well as the ongoing development of new storage techniques, have transformed the way the world works, the result of which, he hopes, will result in increasing the frequency of scientific discovery and the quality of life in general.