Laniakea turns a group of storage devices into block volumes, shared filesystems, and S3-compatible object storage. It is designed for self-hosted clusters that need explicit durability, predictable failure handling, and the option to keep S3 data at more than one geographic site.

Warning

Laniakea is experimental and has not published a stable release. Its protocols and on-disk format may still change without compatibility shims. Use disposable data while evaluating it.

At a glance

Applications can use the same cluster in several ways:

Storage service	What a client sees	Typical use	Current durability
Block	An NVMe/TCP or NBD disk	Virtual machines, databases, filesystems	Replication
Shared filesystem	A FUSE mount	Shared directories and ordinary file access	Replication
Object storage	An S3-compatible endpoint	Backups, archives, applications, data platforms	Replication or erasure coding
Kubo/IPFS profile	An S3-backed /blocks datastore	Durable immutable IPFS blocks	Replication or erasure coding
S3 federation	A local S3 endpoint backed by multiple sites	Geographic copies and geo-aware write acknowledgement	Independent site durability

These are separate frontends over the same storage foundation. S3 is not implemented as files on the shared filesystem, and neither S3 nor filesystem payloads are stored inside NATS.

How it works

At a high level, clients talk to a frontend, and that frontend reads or writes the storage daemons that own the data:

The main pieces are:

• An OSD is a storage daemon responsible for one formatted device.
• A pool describes how data is placed and protected across eligible OSDs.
• A gateway presents a client protocol such as NVMe/TCP, NBD, or S3.
• An MDS coordinates filesystem metadata while file data flows directly to storage.
• A controller applies cluster changes and coordinates maintenance.
• NATS JetStream is the site-local source of truth for cluster state, leases, audit records, and resumable progress.
• A fence is a generation number that prevents an old process from continuing to write after ownership has moved elsewhere.

One Laniakea cluster belongs to one site-local NATS cluster. A separate Core-NATS overlay can connect several independent sites for S3 federation without extending JetStream consensus across the WAN.

What Laniakea is designed to guarantee

• Acknowledged writes satisfy the durability policy selected for their pool.
• Only the current fenced owner may mutate a volume or namespace partition.
• Checksums detect corrupt metadata, chunks, and erasure-coded fragments.
• Reads, repair, scrub, recovery, replacement, and garbage collection fail closed instead of inventing data.
• Placement can keep copies or fragments in different devices, hosts, racks, or zones.
• Maintenance work is bounded, observable, restartable, and safe to retry.
• Device replacement and power-of-two placement-shard growth use staged, verified handoffs rather than an immediate global remap.
• JetStream capacity is monitored and new control-plane mutations stop before reserved recovery headroom is consumed.

There are deliberate limits:

• Block volumes and filesystem data are replicated; erasure coding currently protects immutable S3 payloads only.
• NBD is useful for development and a VM pinned to one host, but an existing kernel NBD connection cannot move to another gateway. Use NVMe/TCP multipath when the client must survive gateway failover.
• A verified geographic S3 replica is read-only. Automatic home-site promotion and multi-writer object namespaces are not implemented.
• A geo-active S3 policy makes covered writes depend on remote durability receipts, so WAN or destination failure can intentionally stop those writes.
• Ordinary management routes are not authenticated. Keep them on loopback or a trusted management network protected by firewall policy.

The active roadmap and remaining release work are tracked in docs/ROADMAP.md.

Try the local lab

The quickest way to explore Laniakea is the disposable local lab. It creates six sparse image-backed OSDs and exposes a block volume, a shared filesystem, and an S3 service. It does not attach or mount them automatically.

You need Go 1.25 or newer, Make, Docker, curl, jq, and uuidgen. The first run may also build the pinned SPDK image. The S3 example below additionally uses the AWS CLI.

make help
make lab

laniakea status --fail-on never
laniakea ls
laniakea df

The lab prints the commands for connecting NBD or NVMe/TCP, mounting the filesystem, and using its S3 credentials. For example:

source .local-laniakea/s3.env
aws --endpoint-url "$LANIAKEA_S3_ENDPOINT" s3 mb s3://demo
aws --endpoint-url "$LANIAKEA_S3_ENDPOINT" s3 cp README.md s3://demo/README.md
aws --endpoint-url "$LANIAKEA_S3_ENDPOINT" s3 ls s3://demo/

Clean up with:

make clean

Cleanup refuses to continue while a VM, NBD/NVMe connection, or mounted FUSE filesystem still depends on the lab. Disconnect the consumer normally and run the command again; do not force-detach live storage.

For a manual walkthrough, follow docs/GETTING-STARTED.md. docs/NBD.md continues from there to a raw /dev/nbdX suitable for a virtual-machine disk.

Build from source

go install ./cmd/laniakea
laniakea --help
laniakea version

Run focused package tests while developing. The complete host and federation qualification suites are intentionally separate because some build containers, create large sparse images, require host kernel facilities, or run for a long time.

go test ./cluster ./control ./placement
make help

make release runs the complete integration suite first. Only a successful run produces .build/linux-ARCH/laniakea, stamped with the tag-aware Git revision and a UTC build timestamp, stripped, and compressed with upx --best --lzma. ARCH defaults to go env GOARCH; set it explicitly to cross-build another Linux architecture. The packaging target requires UPX.

make install builds and packs the selected ARCH, then installs it as laniakea in the first GOPATH entry's bin directory. Override INSTALL_DIR to select another destination. A normal go install remains a dev build and reports the invocation time from laniakea version.

This convenience target does not replace the clean, exact-revision evidence, performance, and soak requirements described below.

Basic operator workflow

A production-shaped deployment starts with an existing three- or five-member NATS JetStream cluster and at least three Laniakea controllers. Each host uses one configuration file:

/etc/laniakea/<cluster>.toml

Select it once for the current shell:

export LANIAKEA_CONFIG=/etc/laniakea/production.toml
laniakea status --fail-on never

1. Initialize the site-local authority

laniakea admin cluster init --name production --replicas 3

This creates the bounded JetStream streams and key-value buckets used for cluster state. Size those resources together with the NATS account and server storage budgets before admitting production data.

2. Provision and admit storage

Provisioning formats the selected device and is destructive. Always use a stable /dev/disk/by-id/... path and verify the serial shown by the command.

sudo laniakea device provision /dev/disk/by-id/nvme-VENDOR_MODEL_SERIAL

laniakea admin device admit http://storage-01.example.net:8420 \
  --host storage-01 --rack rack-a --zone dc-a

The first command operates locally on the host that owns the device. The second records that running OSD and its physical failure-domain labels in the cluster. Repeat this for enough independent devices or hosts to satisfy the durability policy you intend to create.

3. Create storage by name

Create a replicated block volume:

laniakea admin pool create nvme --replicas 3 --failure-domain host
laniakea admin volume create database --pool nvme --size 8TiB

Create a shared filesystem and a replicated S3 service:

laniakea admin fs create projects --replicas 3 --failure-domain host
laniakea admin s3 create archive --replicas 3 --failure-domain host

On a topology with enough independent failure domains, preview and create a capacity-oriented S3 service with erasure-coded immutable payloads:

laniakea admin s3 create backups \
  --durability capacity --failure-domain host --plan

laniakea admin s3 create backups \
  --durability capacity --failure-domain host

Creation commands are idempotent: retrying the same request converges on the same derived IDs and policy. A same-named request with different geometry fails rather than silently changing existing storage.

See docs/CLUSTER.md for the full deployment and operations guide and docs/SYSTEMD.md for service installation.

Observe and manage a cluster

The everyday command surface is split deliberately:

• Top-level commands inspect state and do not mutate the cluster.
• laniakea admin ... submits audited changes.
• laniakea device provision is the guarded host-local media operation.
• The hidden laniakea service ... tree is for systemd and controlled test environments, not normal administration.

Common read-only commands are:

Command	Answers
laniakea status	Can the cluster safely serve data?
laniakea ls	Which devices, pools, volumes, filesystems, and S3 stores exist?
laniakea df	How much physical and shared logical capacity is available?
laniakea device status DEVICE	Why may this device receive—or not receive—data?
laniakea pool status NAME	How is this pool protected and placed?
laniakea pool shard ls --state not-clean	Which placement shards need attention?
laniakea maintenance status	Which repair, movement, scrub, or cleanup jobs are active?
laniakea federation status	Which S3 stores have geographic policy, and in what phase?

Inventory and capacity commands accept --json. Status, inventory, and capacity views also support --watch or -w for an interactive two-second refresh.

Capacity needs careful interpretation. Volumes are thin-provisioned, and pools draw from shared devices. Therefore a pool's available capacity is an estimate, not a reservation, and availability from several pool rows must not be added together. device df is the physical view; pool df and volume df explain the logical view.

Maintenance and lifecycle

Laniakea exposes maintenance as resumable cluster work rather than a command that must remain attached to one terminal. Operators can:

• recover missing replicated chunks and misplaced fragments;
• scrub metadata and payload integrity;
• reclaim unreachable data through certified garbage collection;
• add, relabel, reweight, drain, replace, decommission, and finally purge a device identity;
• double a pool's placement-shard count online; and
• migrate standalone block volumes between replicated pools.

Before changing topology, inspect laniakea maintenance status. Follow the dedicated replacement, pool resize, and erasure-coded recovery runbooks for the required preconditions and exit checks.

S3 and geographic federation

Laniakea's S3 service has its own ordered object namespace and immutable payload representation. Its implemented profile includes SigV4, multipart uploads, versioning, Object Lock, coherent snapshots and restore, quotas, storage classes, lifecycle, notifications, replication, compression, and replicated or erasure-coded payloads. The precise supported API—not a vague compatibility percentage—is generated in docs/S3-COMPATIBILITY.md.

Federation joins otherwise independent Laniakea sites:

Attaching a federator is passive and does not interrupt local I/O or move data. An operator enrolls an S3 store through explicit phases:

• geo-ready means the destination has verified data and may serve read-only S3 traffic.
• geo-active means covered home-site mutations return success only after the configured independent sites have supplied durability receipts.

Pre-activation work can be cancelled. An active policy must first be suspended with an explicit reduced-durability acknowledgement, then retired to return to local operation. Only a local policy can be reconfigured or detached.

Replica endpoints remain read-only, and Laniakea does not automatically change DNS, promote a new home site, or create a multi-writer namespace. See the federation guide for the operating model and lifecycle, then the implementation record and security boundary for deeper detail.

To explore the full three-site path locally, make supercluster/s3-up creates three OSDs and one S3 endpoint per site. make supercluster/s3-ready enables verified geo reads; make supercluster/s3-activate separately strengthens US writes to require geographic receipts. The fixture ports, public development credentials, persistence behavior, and smoke commands are documented in its operator README. make supercluster/s3-e2e runs isolated, sustained geo-active traffic across failover and WAN faults. It also checks queue drain, recovery latency, and container memory without taking over the stable development fixture's ports.

Verification and project status

The repository contains fast package tests plus opt-in qualification for real OSD paths, NVMe/TCP multipath, S3 clients, erasure coding, federation, rolling upgrades, recovery, corruption, and resource exhaustion. Use make help to see the available targets and their scope.

Passing development tests does not make a release. Replicated S3 has a 15-gate, exact-revision evidence contract; long-running soak, performance, and comparison evidence remains release work. The EC release handoff begins only after that replicated contract passes. See docs/S3-RELEASE-QUALIFICATION.md and docs/ROADMAP.md for the current boundary.

Documentation map

If you want to…	Read…
Build a disposable cluster by hand	Getting started
Design or operate a real cluster	Cluster deployment
Attach a block volume through NBD	NBD guide
Run services under systemd	systemd guide
Configure every daemon and network plane	Configuration and network planes
Use the native S3 service	S3 guide
Understand the supported S3 API	S3 compatibility profile
Operate erasure-coded S3 data	EC operations
Understand the EC design	Erasure coding
Store Kubo/IPFS blocks	Kubo/IPFS guide
Understand or operate multi-site federation	Federation guide
Study the federation protocol and implementation	Supercluster implementation record
Review architectural decisions	ADR index
See current priorities and deferrals	Roadmap
Run performance harnesses	Performance guide
Work on the SPDK bridge	SPDK guide

License

Laniakea is licensed under the GNU Affero General Public License, version 3 only (AGPL-3.0-only).