SeaweedFS is a simple and highly scalable distributed file system. There are two objectives:
- to store billions of files!
- to serve the files fast!
SeaweedFS started as an Object Store to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages volumes on volume servers, which manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (O(1), usually just one disk read operation).
There is only 40 bytes of disk storage overhead for each file’s metadata. It is so simple with O(1) disk reads that you are welcome to challenge the performance with your actual use cases.
SeaweedFS started by implementing Facebook’s Haystack design paper. Also, SeaweedFS implements erasure coding with ideas from f4: Facebook’s Warm BLOB Storage System and has a lot of similarities with Facebook’s Tectonic Filesystem
On top of the object store, the optional Filer can support directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata stores, e.g., MySql, Postgres, Redis, Cassandra, HBase, MongoDB, Elastic Search, LevelDB, RocksDB, Sqlite, MemSql, TiDB, Etcd, CockroachDB, YDB, etc.
For any distributed key-value stores, the large values can be offloaded to SeaweedFS. With the fast access speed and linearly scalable capacity, SeaweedFS can work as a distributed Key-Large-Value store.
SeaweedFS can transparently integrate with the cloud. With hot data on the local cluster and warm data on the cloud with O(1) access time, SeaweedFS can achieve both fast local access time and elastic cloud storage capacity. What’s more, the cloud storage access API cost is minimized. More rapid and Cheaper than direct cloud storage!
- Can choose no replication or different replication levels, rack and data center aware.
- Automatic master servers failover – no single point of failure (SPOF).
- Automatic compaction to reclaim disk space after deletion or update.
- Automatic entry TTL expiration.
- Any server with some disk space can add to the total storage space.
- Adding/Removing servers does not cause any data re-balancing unless triggered by admin commands.
- Automatic Gzip compression depends on file MIME type.
- Optional picture resizing.
- Support ETag, Accept-Range, Last-Modified, etc.
- Support in-memory/leveldb/read-only mode tuning for memory/performance balance.
- Support rebalancing the writable and read-only volumes.
- Customizable Multiple Storage Tiers: Customizable storage disk types to balance performance and cost.
- Transparent cloud integration: unlimited capacity via tiered cloud storage for warm data.
- Erasure Coding for warm storage Rack-Aware 10.4 erasure coding reduces storage cost and increases availability.
- Filer server provides “normal” directories and files via http.
- File TTL automatically expires file metadata and actual file data.
- Mount filer reads and writes files directly as a local directory via FUSE.
- Filer Store Replication enables HA for filer meta data stores.
- Active-Active Replication enables asynchronous one-way or two-way cross cluster continuous replication.
- Amazon S3 compatible API accesses files with S3 tooling.
- Hadoop Compatible File System accesses files from Hadoop/Spark/Flink/etc or even runs HBase.
- Async Replication To Cloud has extremely fast local access and backups to Amazon S3, Google Cloud Storage, Azure, BackBlaze.
- WebDAV accesses as a mapped drive on Mac and Windows, or from mobile devices.
- AES256-GCM Encrypted Storage safely stores the encrypted data.
- Super Large Files stores large or super large files in tens of TB.
- Cloud Drive mounts cloud storage to local cluster, cached for fast read and write with asynchronous write back.
- Gateway to Remote Object Store mirrors bucket operations to remote object storage, in addition to Cloud Drive
By default, the master node runs on port 9333, and the volume nodes run on port 8080. Let’s start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We’ll use localhost as an example.
SeaweedFS uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.