DDN Announces the Biggest Big Data Object System
From Wikibon
David Floyer (Talk | contribs) |
David Floyer (Talk | contribs) |
||
| Line 4: | Line 4: | ||
The growth in data is coming from machines, not humans, and the biggest growth is coming from sensors. Video, acoustic, pressure, heat, chemical, proximity, speed and many other sensors are flooding in, in addition to the computer generation of text, tables, and graphics. The biggest growth is coming from video, surveillance, high performance computing, life sciences, cloud & web content, environmental monitoring, rich media and government intelligence. | The growth in data is coming from machines, not humans, and the biggest growth is coming from sensors. Video, acoustic, pressure, heat, chemical, proximity, speed and many other sensors are flooding in, in addition to the computer generation of text, tables, and graphics. The biggest growth is coming from video, surveillance, high performance computing, life sciences, cloud & web content, environmental monitoring, rich media and government intelligence. | ||
| - | The problems of storing this big data tsunami are the normal ones of writing, indexing, provenance, security, protection and retrieval, only on a massive scale. In traditional IT, file systems have been built to handle this. [http://en.wikipedia.org/wiki/Comparison_of_file_systems The list of file systems has grown], and traditional Networked files systems (NAS) have improved dramatically with global names spaces and better metadata management. However there is general computer science | + | The problems of storing this big data tsunami are the normal ones of writing, indexing, provenance, security, protection and retrieval, only on a massive scale. In traditional IT, file systems have been built to handle this. [http://en.wikipedia.org/wiki/Comparison_of_file_systems The list of file systems has grown], and traditional Networked files systems (NAS) have improved dramatically with global names spaces and better metadata management. However there is a general computer science consensus that these types of systems cannot scale to meet the performance and availability requirements of petabyte/exabyte with billions/trillions of records, at least not cost effectively. The World’s Fastest POSIX File Systems in 2012 is a DARPA Lustre system, which achieves about 3 billion reads and writes/day. |
| - | The biggest big data systems are now object based rather than file based. One key advantage of object storage is that the data and metadata are stored together, which eliminates many of the locking, metadata traversals, directory crawling, and file allocation tables issues. For example, [http://highscalability.com/blog/2011/1/11/google-megastore-3-billion-writes-and-20-billion-read-transa.html Google claims that the Google Megastore achieves 3 billion writes and 20 billion reads per day]. As of Q2 2011, the [http://aws.typepad.com/aws/2011/07/amazon-s3-more-than-449-billion-objects.html Amazon S3 system] stores about half a trillion objects and reads a peak of 290,000 0bjects per second (25 billion per day). | + | The biggest big data systems are now object based rather than file based. One key advantage of object storage is that the data and metadata are stored together, which eliminates many of the locking, metadata traversals, directory crawling, and file allocation tables issues of traditional file systems. For example, [http://highscalability.com/blog/2011/1/11/google-megastore-3-billion-writes-and-20-billion-read-transa.html Google claims that the Google Megastore achieves 3 billion writes and 20 billion reads per day]. As of Q2 2011, the [http://aws.typepad.com/aws/2011/07/amazon-s3-more-than-449-billion-objects.html Amazon S3 system] stores about half a trillion objects and reads a peak of 290,000 0bjects per second (25 billion per day). |
<!--|body|--> | <!--|body|--> | ||
Revision as of 14:26, 12 October 2011
Posting in Progress - please do not edit or download until this line is removed
DataDirect Networks has announced WOS 2.0, which is positioned as the world's fastest object storage system. WOS 2.0 is a fully integrated system including DDN storage, erasure coded data protection mechanisms, a replication strategy for distributing object data, and extensions to the interface options with WOS 2.0 to include S3 APIs and a NAS interface.
Big Data Growth and Challenges
The growth in data is coming from machines, not humans, and the biggest growth is coming from sensors. Video, acoustic, pressure, heat, chemical, proximity, speed and many other sensors are flooding in, in addition to the computer generation of text, tables, and graphics. The biggest growth is coming from video, surveillance, high performance computing, life sciences, cloud & web content, environmental monitoring, rich media and government intelligence.
The problems of storing this big data tsunami are the normal ones of writing, indexing, provenance, security, protection and retrieval, only on a massive scale. In traditional IT, file systems have been built to handle this. The list of file systems has grown, and traditional Networked files systems (NAS) have improved dramatically with global names spaces and better metadata management. However there is a general computer science consensus that these types of systems cannot scale to meet the performance and availability requirements of petabyte/exabyte with billions/trillions of records, at least not cost effectively. The World’s Fastest POSIX File Systems in 2012 is a DARPA Lustre system, which achieves about 3 billion reads and writes/day.
The biggest big data systems are now object based rather than file based. One key advantage of object storage is that the data and metadata are stored together, which eliminates many of the locking, metadata traversals, directory crawling, and file allocation tables issues of traditional file systems. For example, Google claims that the Google Megastore achieves 3 billion writes and 20 billion reads per day. As of Q2 2011, the Amazon S3 system stores about half a trillion objects and reads a peak of 290,000 0bjects per second (25 billion per day).
Action Item:
Footnotes:
