Speaker, Paul McCullagh
Blobstreaming.org

This session explained why BLOB streaming may be relevant to your project, and how you can get started streaming BLOBs.

For ordinary types of data, MySQL sends the buffer from the client to the server. Streaming involves making a connection from the source to the destination continuously, storing and streaming simultaneously. But BLOBs are not ordinary data.

BLOBs were invented by Jim Starkey. A BLOB is backronym for Basic Large Object, or Binary Large Object, and is a collection of binary data stored as a single entity in a database management system. Blobs are typically images, audio, or other multimedia objects, though sometimes binary executable code is stored as a BLOB. Database support for BLOBs is not universal.

The problem of storing BLOBs in a database is multifaceted.  BLOBs create big rows in memory, and sequential scans are not possible. The database can become too big to handle, and then the database won't scale well. In addition, BLOBs slows down replication, and BLOB data must be written to the binary log.

So, why put BLOBs in the database? So that BLOB operations are transactional and have valid references, and so you have the data all in one place, which is good for testing, Small BLOBs are handled better by databases, and it's convenient to handle all your data the same way. Additionally, with BLOBs in a database, your backups are more consistent. Lastly, putting the BLOBs in a database makes replication possible.

To overcome these issues, the MySQL Project streams BLOB data directly in and out of the database.You can store Blobs of any size in the database, and create a scalable back-end that can handle any throughput and storage requirement. The object is to provide an open system that can be used by all engines.

The Scalable BLOB Streaming Project solves the above problems with a BLOB repository, so a collection of BLOBs are stored outisde of database rows, and the references are stored in the table. The advantages of this is that it allows for incremental backup, automatic defragmentation and compastion, and repository blob data that will not be written to the binary log and can be scaled out.

The goal of the Scalable BLOB Streaming Project for MySQL is to provide a scalable, stream-based, infrastructure for the storage and retrieval of pictures, films, MP3 files, and other binary and text objects (BLOBs) to and from the database.

The first step has been taken with the creation of the BLOB Streaming engine for MySQL, a storage engine that allows BLOB data to be uploaded and downloaded directly to and from a MySQL table, using the standard HTTP protocol. Read the rest of this blog