Expert: storage must evolve to support AI and other new tech

Without storage hardware and software working effectively in tandem, says one expert, data cannot be used to full effect, which in turn limits the effectiveness of AI systems and other emerging technologies.

Jeff Rowe | Nov 05, 2017 11:00 pm

As Artificial Intelligence (AI) and machine learning, among other technological developments, make increasing inroads into healthcare and other sectors of the economy, they inevitably mean organizations must up their game when it comes to sifting through their data storage options.

That’s according to Robert Lee, VP and chief architect at Pure Storage, in a recent interview at ZDNet

"Historically, the limit to how much data has been able to be processed, the limit to how much insight we've been able to garner from data has been bottlenecked by storage's ability to keep the compute fed," said Lee. "Somewhere around the early 2000s, the hardware part of compute, CPUs started getting more parallel. It started doing multi-socket architectures, hyper threading multi-core. Fast-forward a couple of years beyond that, applications, software started getting more parallel. Things like distributed computing, scale-out systems, parallelization started becoming more prevalent."

He added that enterprises increasingly realized that building out larger compute clusters does not generate better results because the additional compute hardware just sits idly behind storage. Indeed, he went on, some of the biggest challenges enterprises face today are around amassing large datasets and feeding them into compute for analysis and pattern recognition.

"Fundamentally, the more varied data sets that you can provide into AI systems and machine learning and training systems, the better results you're going to get ... in any space, whether it's autonomous driving, natural language processing, facial recognition," he said.

According to Le, the challenge of building high-performance storage systems that work with flash media is really one that needs to be solved using software.

"Removing all of the extra components that you find typically within an SSD and directly writing software to work with hardware that is giving us direct access to those flash chips, has allowed us to drive much better performance as well as much better longevity and efficiency out of the flash usage," he said.

"You need to design software systems, you need to rethink how storage controller software is written for that media. The performance characteristics that we're able to drive out of our products ... there's a delicate dance and tight integration between software that's purpose-built for the media and hardware that's [designed] to accelerate the software."