DDN Announces the Biggest Big Data Object System
From Wikibon
Posting in Progress - please do not edit or download until this line is removed
Big Data Growth and Challenges
The growth in data is coming from machines, not humans, and the biggest growth is coming from sensors. Video, acoustic, pressure, heat, chemical, proximity, speed and many other sensors are flooding in, in addition to the computer generation of text, tables, and graphics. The biggest growth is coming from video, surveillance, high performance computing, life sciences, cloud & web content, environmental monitoring, rich media and government intelligence.
The problems of storing this big data tsunami are the normal ones of writing, indexing, provenance, security, protection and retrieval, only on a massive scale. In traditional IT, file systems have been built to handle this. The list of file systems has grown, and traditional Networked files systems (NAS) have improved dramatically with global names spaces and better metadata management. However there is general computer science agreement that these types of systems cannot scale to meet the performance and availability requirements of petabyte/exabyte with billions/trillions of records, at least not cost effectively. World’s Fastest File Systems in 2012 is a DARPA Lustre system, which achieves about 3 billion reads and writes/day.
The biggest big data systems are now object based rather than file based. One key advantage of object storage is that the data and metadata are stored together, which eliminates locking, metadata traversals, directory crawling, and file allocation tables. For example, Google claims that the Google Megastore achieves 3 billion writes and 20 billion reads per day. As of Q2 2011, the Amazon S3 system stores about half a trillion objects and reads a peak of 290,000 0bjects per second (25 billion per day).
Action Item:
Footnotes:
