Data management challenges in a regulated, hybrid, multi-cloud world

The capacity of raw storage is growing exponentially (see my Figure), which shows quarterly growth by the 3 main types – disk drives, NAND and DRAM (the latter 2, solid state drives). In 2017 a total of 1,500EB (1ZB) were shipped globally. While my data here includes all raw storage (including that used in mobile phones, tablets, laptops and desktop PCs), traditionally the majority has been used in servers and storage systems installed in the data centers and branch offices of commercial and government users. However the dramatic adoption of cloud computing has changed things in the last few years, with a growing proportion of raw storage devices now housed in cloud service provider data centers.

Many users are struggling to keep on top of the ever-increasing volumes of data. Getting a working data governance strategy in place is essential in order to become compliant with the EU’s GDPR and other similar data protection legislation.

The growing challenges of data management

There are many challenges for those responsible for storage systems and data. In particular:

• Structured v unstructured data – there are differences between the information we hold in traditional databases and that associated with the monitoring of devices or interactions on social media; data storage has to be able to handle both types of data effectively.
• Cloud storage is different – especially in the use of AWS’s Simple Storage Service (s3) object storage, which has to be integrated with the file and block storage more traditionally used in on-premise systems; a hybrid cloud strategy requires users to be able to handle multiple types of data and storage devices.
• Shadow IT – the early adoption of cloud services as been by departments, often without the approval (and sometimes even the knowledge) for their organisation’s IT department; integrating these applications and their data is essential to have a holistic view of the total data estate.
• Primary v secondary storage – traditionally users have made copies of their live data for security (backup) and access of older data which doesn’t need to be constantly available (archive). Tape drives and juke boxes are still used by many companies to store secondary data, despite the slow speed of access; solid state storage gives us the speed needed to handle the influx of masses of unstructured data, but we need to find more efficient ways of accessing backups and archives.
• Object, file and block storage – these are the 3 generic ways of storing information; many companies haven’t yet come to grips with managing on premise object storage.
• Software defined storage (SDN) – to be able to make the most of the advances in the cost per gigabyte of storage a number of suppliers offer software solutions running on industry standard x86 servers; many users still rely on storage systems hardware and are confused about the new opportunities and cheaper prices of an SDN approach.
• The legal necessities of data governance and management – the EU’s GDPR legislation gives citizens the right to know what private data an organisation holds on them, a requirement for the supplier to seek permission every time their data is processed (and ‘opt in’ rather than ‘opt out’), the right ‘to be forgotten’ (have their data removed from databases) and their use of services not to be dependent on providing their personal data; becoming compliant with the regulation is a non-trivial task for those organisations with poor data governance and the large potential fines for non-compliance makes it an urgent issue for all.

The storage industry is offering a bewildering number of new solutions, but it’s no longer possible to take a piecemeal approach due to GDPR and similar regulations being enacted in a number of countries outside the EU.

ITCandor’s storage management research

I’ve been following the storage market for many years and produce a mass of information on suppliers of raw storage, systems and storage, as readers of this site know well. In 2018 I’m refocusing my activities to cover more suppliers and their approaches. I’m keen to guide users to make intelligent choices in this vital area of the IT industry.

Please contact me if you’d like to get involved.