Even without Artificial Intelligence and Machine Learning, the growth rate of data in Enterprise Information Technology has pushed us to a point where we need a different and better approach to building Data Storage Infrastructure. Smarter Storage Arrays, Improvements in Flash and increased speed in data access have gotten us to this point – but future proofing our Storage Infrastructure requires a broader design scope.
We need a single and consistent data management framework that is specially-designed for the business needs. We need to enable frictionless access and sharing of data in a distributed data environment as opposed to siloed storage. The “Data Fabric” will be designed on the desired business outcomes as they relate to the following elements:
- Performance
- Availability and Reliability
- Centralized and Instrumented Data Management with Integrated Tools
- Optimized Cost of Ownership
- Intelligently serving the hybrid cloud design
- Data Protection
- Data Security
- Parallel Architecture
- Data Locality
- Data Lifecycle Management
The Data Fabric Design for an enterprise is already a vital exercise for any company. The design needs to incorporate the actual data flow – from collection to organization to integration to processing to output. The design will need to incorporate the volume and velocity requirements at each point along the flow.
I think too many times the Infrastructure and Architecture teams get involved in defining the “Data Storage Requirements” far too late in the development cycle. In today’s world (and even more in the world of the near future), the volume and velocity of enterprise data will create problems if we simply look at each application as incremental capacity requirements. Ideally, we can create an environment where the Data Fabric Design is clearly understood across the IT organization and becomes a foundational element for Developers and others who create the services that will use that Fabric.
When you add in the enormous amounts of data that will be required to effectively power AI/ML, and the fact that the data needs to be stored and be available for high-demand processing in the AI/ML engine – our infrastructure will break without an intelligent Data Fabric Design.
