In our digital world the most important asset for most organisations and individuals is information, which we hold and process as data in a huge variety of storage devices. However spending on storage systems has been in a long term decline despite the growing number of regulations that require data retention and the valiant marketing efforts of established vendors. There are many reasons for the decline including the unbundling of storage software from hardware arrays (often described as Software Defined Storage) and the increasing efficiency of new solutions, which cut down the need to over provision storage capacity. The Figure above shows the quarterly revenues of storage systems compared with those of NAND and DRAM solid state and disk drives. My numbers include some significant overlap since the main component of storage systems comes from the other 3 categories. You’ll want to know more about how this market is developing and how it’s likely to develop in the next few years.
Market shares
Declining spending on storage systems has had a devastating effect on suppliers (see the bottom left of the Figure for their 2016 market shares). You know things are going badly when the leading supplier (EMC) is acquired (by Dell), but this was just one of many acquisitions such as 3Par and Nimble by HPE, SolidFire by NetApp and TMS by IBM). The lack of market growth and high prices of flash arrays have fuelled the acquisition of a number of start-up suppliers; the 3-year profit cycle in the ITC industry will drive more in 2018. The on-going challenge for the established vendors is to farm profits from large and medium sized customers of their older hardware products, while keeping up to date with newer, cheaper solutions. Users know this, so there has been a vast difference between the list and actual prices they pay. The perennial problem for users is the cost of change; new technologies promise huge savings, but swapping takes time and money and, even if the new solution works, it’s difficult to entrust the most important corporate assets to an unknown supplier, whose status is liable to change if it is a start-up.
The disk drive market is fully consolidated with just 3 companies holding 67% of its $17.0b 2016 value (see the bottom right of the Figure opposite, where a number of other motorised storage devices such as tape drives are included). The leading disk drive suppliers here have also been experimenting in storage systems areas through acquisition (Western Digital of SANDisk, Seagate of Dot Hill for instance).
The solid state disk market has more suppliers (see the middle top pie chart) than the spinning drive one, but it too is going through a period of consolidation – and you shouldn’t believe anyone who glibly suggests that there’s an exponential capacity growth even in this area; I’ve measured declines in some recent quarters, due perhaps to oversupply, overpricing and the wait for new capacity drives.
Solid state flash v spinning disk
It’s logical that the massive scale economies of solid state storage chip production (the first one costs billions, but all the rest are free, as someone put it), lower running costs and lower operating temperature will lead to the death of the spinning disk market, but it’s not going to happen anytime soon. The Figure opposite shows the quarterly revenues of disk and flash drives between 2003 and 2016. Spending on disk drives has fallen significantly since 2010, while that on flash drives grew steeply (despite the Credit Crunch causing a short hiatus in 2008-09); however the 2 areas have shown very similar absolute spending patterns since 2013, with a decline in both areas in 2015. For as long as disk drive producers manage to introduce higher capacity drives and the price differential between the 2 technologies exists, users can store more data, more cheaply on disk than flash. In addition there are many more choices in how to attach and manage flash than disk. Virtually the only technologies our industry has given up are mercury relays and punch cards. Some thought that IBM’s introduction of the world’s first Winchester disk drive in 1956 would lead to the death of the magnetic tape market started by IBM a year earlier – so, if we still use tape drives 60 years later, it may be a long time before we see the last disk drive.
Storage systems, software and maintenance
While spending on storage systems – mainly hardware – has been declining, there has been an increase in hardware maintenance and storage software (see the Figure opposite). The longer life cycles of storage systems means that it costs users more to keep them going, hence the increase in hardware maintenance, while the disruption of software-only approaches of a multitude of specialist (mostly start-up) vendors accounts for the growth of that area.
There are many suppliers of storage software (often start-up companies owned by experienced engineers from established storage system vendors). They typically use ‘Software Defined Storage’ campaigns and promises of greater efficiency and much lower costs. I’ve talked to many during my time at ITCandor, such as Storpool (which targets cloud service providers), Stormagic (branch offices), Datacore, Falconstor and Virsto (storage hypervising – the latter bought by VMware in 2013).
Storage as a Service and the cloud
Storage systems are a major capital investment for large and medium sized organisation, especially when you add the cost of buildings, storage administrators, data scientists and insurance to the hardware, software and maintenance costs. Consequentially there has been a strong uptake of cloud storage – a market led by Amazon, whose solutions use the S3 protocol for connections. The Figure opposite compares the revenues of storage systems, software and maintenance with the total spent on IaaS on an annual basis between 2005 and 2016. Although the 2 lines approach each other over time, the real world is more complicated; there’s an overlap lowering the physical spending because an increasing amount is being bought by cloud service providers, while IaaS revenues for storage are lower once you exclude the amount spent on processing and networking. There are also additional risks of adopting cloud storage – your supplier may suffer from occasional outages, your communications costs will be higher, you’ll need to work on hybrid ways of bridging between external and internal resources and you need to adhere to regulations which may prevent you from using cloud services at all for mission critical applications, or restrict which countries in which your data can be stored.
Storage systems – the regional perspective
There have been strong variations in the regional growth of spending over the last few years (see the Figure opposite which shows the quarterly totals spent on storage systems in the Americas, Asia Pacific and EMEA in current $US values. To a certain extent the drop-off in spending from 2011 in Asia Pacific and EMEA (but not the Americas) can be linked to the growth of cloud service providers, most of which started by building data centres in the US; it is also due to the shift away from buying solutions from the established storage system vendors, which occurred more slowly in the Americas than it did internationally.
The US dominates spending in the Americas, accounting for 78% of the total in 2016 and due partially to the predominance of cloud service providers there (despite the largest ones buying raw storage, rather than storage systems). In Asia Pacific Japan is the largest country market (26% of the total in 2016), followed by China (11%) and Australia (10%). In EMEA the biggest country markets were the UK (18%), Germany (17%) and France (12%) in 2016.
Massive changes in exchange rates account for some of the dramatic falls in spending in Asia Pacific and EMEA when calculated in current $US shown in the Figure above. The Figure opposite normalises growth rates by using local currencies. I’ve used the Japanese Yen for Asia Pacific shown in pink and the Euro for EMEA in red. It’s interesting that regional growth rates were similar up to 2009, but very different and more dynamic afterwards. In particular there was a deeper fall in Asia Pacific in 2012 (perhaps in the aftermath of the devastating earthquake in Japan in 2011. In EMEA there was relatively strong spending growth in 2015 followed by an equally strong decline last year.
The dip in spending in 2008-9 was associated with the Credit Crunch, which affected almost all IT and Communications hardware markets. I discovered that the aftermath accelerated the adoption of newer, cheaper solutions in other areas (a stronger movement towards x86-based products in the server market for instance). In the storage market it accelerated the move towards the use of both cloud services and ‘Software Defined’ storage adoption.
The changing attachment markets
There are literally billions of IT and Communications devices which include storage. I show the unit shipments of the main ones in the Figure opposite on an annual basis for the period between 2005 and 2016. Of course each device type uses different sorts of storage – almost always a single solid state device in phones, tablets and – increasingly – laptop PCs; multiple and highest capacity drives in servers.
Raw storage suppliers planning for the type and capacity of the solid state and spinning disk drives they will build in the future need to predict the fall off in phone and server shipments, the speed of take-off of new devices (especially IoT, smart wearables, solid printers, etc.) and adoption of certain applications (analytics, blockchain, artificial and augmented reality, data mining, Web apps, SAP, Microsoft SharePoint and Exchange, etc.). They also need to think about servers, which will increasingly hold the data associated with many of the newer mobile applications and devices. In many cases the development of new technology will be dependent on the future availability of higher capacity, lower voltage storage, which in turn is dependent on a continuous shrinking of die sizes.
In addition to the devices to which storage will be attached, it is vital for users and vendors to predict the changing landscape of storage protocols. There has been a strong movement away from Fibre Channel towards iSCSI for large storage arrays for instance, towards Amazon’s S3 for cloud-attached storage and from SAS to SATA disks.
Storage is a vital technology at the heart of the IT and Communications market and you shouldn’t take it for granted. Whether you’re a manufacturer, integrator, channel player or user there are many consequences of the development of the market and restrictions in the use you can make of computing in coming years which need to be planned for carefully. Don’t be hoodwinked by generic ‘Big Data’ or cloud marketing campaigns, or their claims of exponential capacity growth making your choices increasingly difficult. Don’t be surprised if the current trends towards Op Ex via the cloud or disaggregation into hardware, software and service elements are reversed in the future.
As always I’m eager to help you – so please contact me if you’d like more details at a product, service or country basis, or help in planning and forecasting.. and/or let me know whether you agree or disagree with my analysis by contacting me at info@itcandor.com.