Moore in Storage
My old friend Peter Karlsson (formerly of Sun) recently wrote to say he had gotten married in Singapore (which is itself of rite of passage) and had also joined the ranks at Micro$oft. I was pleased to learn of his wedded bliss, and naturally a little saddened by the M$-gig news ... having once evangelized with him on behalf of Sun and Java, it was troubling to learn that Peter had had to ultimately succumb to the dark side, as it were. Alas, we must do what we must do.
Peter also enclosed a link to this article which reports that the average cost for disk storage is now less than 10 cents per Mb. Moore's Law, or a derivative thereof, continues to rapidly reduce the cost of storage, which over time gives rise to interesting results.
Gmail, for instance, has raised the bar (or lowered the cost, as the case may be) for large storage-capacity email accounts. Yahoo, in response to Google's efforts, was forced to do likewise ... my fee-based Yahoo account now gets me 2.0 Gb for the same price I paid for 100 Mb just a few months ago.
When Peter and I were still evangelizing on behalf of Sun, I made a (not so bold) prediction in early 2001 that the cost for a Tb of storage would fall below (US)$100 before the end of the decade. The "Drive Capacity / Cost" chart was taken from that presentation. At the time, an 80 Gb disk drive sold for a little less than $600.
Note that today (2004), the predicted capacity/cost of a disk drive is pretty close to that of the Western Digital Caviar, which boasts a 250 Gb capacity. At an MSRP of $210 ... about half what was predicted ... it should be clear that if anything, the prediction I made those many moons ago was too conservative. Clearly, if present trends continue, a Tb of storage will fall well below $100, probably well before 2010. Perhaps as soon as 2007 or 2008.
A Tb of storage should suffice to store (in mp3) every audio album produced in the world this year ... about 20,000 albums, give or take a few thousand. It's no wonder the RIAA is getting excited ... or a few hundred movies with trailers, outtakes and all the extra stuff we're used to getting with a Wal-Mart DVD. At less than $100 per unit, that's a whole lot more bang for the buck than we get today ...
So what are the ramifications of vast amounts of cheap storage? Beyond super-sized email accounts and gargantuan song-swappers, what should we expect? What should we hope for? What should we fear?
When I started with Sun in early 1994, as a program manager for a product called "Solstice Backup" (a network-software based backup solution sourced from Legato, which Sun sold before the emergence of SANs), one constraint to consider was the available network bandwidth juxtaposed to the amount of data to be backed up over a given time. With vast quantities of storage becoming so cheap, the implications for SANs strategies, backup solutions and exponentially-increasing mission-critical data will require a complete rethinking of current approaches to data management and storage.