Shigure – A high-performance object storage system with rich metadata provided by a distributed ledger

Abstract

Networked computer systems often require massive data storage systems for powering the input needs of the system and providing for a place to store computation results. However, not all networked systems have such high data demands -- sometimes a networked system is just one that holds a family's media and data backups. No matter what the size of the system, a need for data storage will still exist. Where there is data, there is also metadata to describe and help search through the data. Metadata management within networked data storage is a messy proposition, and is often a woefully overlooked component of the system. Typical computer filesystems provide little in the way of metadata support beyond basic file attributes without relying on external software to maintain some sort of index for the user. Modern object storage systems do provide additional metadata support, but do not provide any sort of indexing beyond what a typical filesystem does. I have developed, implemented, and tested Shigure to address the problems of providing adaptable storage with rich metadata support, including indexing. Shigure combines a modern, high-performance object storage system with a distributed ledger for keeping track of and indexing metadata and providing access control to the data stored within the system. This allows Shigure to provide scalable storage with rich metadata indexing support that allows unique access patterns that typical filesystems and object storage systems do not provide without the need for an external database. Shigure also implements a hierarchical model for user permissions that goes beyond the typical idea of users and groups that filesystems provide, allowing for a user-friendly way of sandboxing client programs that wish to make use of the data stored within the system. Shigure has been tested on a variety designs of storage systems to evaluate its performance in various situations -- from a single-node system that might be used by an individual in a home network to a system approximating a big data cloud storage environment with multiple storage nodes communicating over the Internet.