# Active Storage on several nodes, each with its own disk

Active Storage's Disk service assumes one machine. Run the same Rails app on three nodes, each with its own disk, and a file uploaded through node A returns a 404 when a request for it lands on node B.

The usual fixes are to move to S3 or another object store, mount a shared NFS volume, or pin requests to nodes. Sometimes none of those fit. The files must stay on hardware you control, the network share is the thing you are trying to retire, or the load balancer isn't yours to configure.

We hit that on a client app and wrote [activestorage-multi_node_disk](https://github.com/clearstackio/activestorage-multi_node_disk). Version 0.1.0 is now on rubygems.org.

## The idea

A file stays on the disk of the node that received it. Any node can read, check or delete it: the service looks on its own disk first, then asks the other nodes and keeps the first copy that matches the blob.

Nothing is recorded about where a file lives. There is no new column, no lookup table and no backfill, so the blobs you already have keep working the day you switch.

## Setup

Add the gem:

```ruby
gem "activestorage-multi_node_disk"
```

Change one line in `config/storage.yml` and list the nodes:

```yml
local:
  service: MultiNodeDisk
  root: <%= Rails.root.join("storage") %>
  nodes: [app1.internal, app2.internal, app3.internal]
```

Give every node the same token in credentials:

```sh
bin/rails credentials:edit   # multi_node_disk: { token: <the same random value on every node> }
```

And turn on token checks in an initializer (`bin/rails g active_storage:multi_node_disk:install` writes it for you):

```ruby
ActiveStorage::MultiNodeDisk.configure do |config|
  config.auth = :token
end
```

That's the whole setup. `has_one_attached`, `blob.download`, `blob.open`, `purge` and `purge_later` work as they do now.

## How it works

`MultiNodeDiskService` is a subclass of Active Storage's `DiskService`, and Active Storage loads it from `service: MultiNodeDisk` the same way it loads Disk or S3. It overrides only what has to change:

- **Upload** writes to this node's disk, exactly as Disk does.
- **Read** (`download`, `download_chunk`, `open`) checks this disk first. On a miss, it asks every other node at once, at most 8 at a time, under one deadline. The first copy whose size and checksum match the blob wins, and a copy that doesn't match is skipped and reported.
- **`exist?`** checks this disk, then whether any node has the file.
- **Delete** removes the local copy, then asks every other node to delete theirs. A 404 counts as done.

Everything else stays local, as with Disk: `url` (use Active Storage's proxy mode, which reads through `download`), `compose`, and `delete_prefixed` for variants, which any node can regenerate.

Each node answers the others at one endpoint:

```
GET | HEAD | DELETE  /rails/active_storage/multi_node_disk/blobs/:key
```

It lives in an isolated engine under Active Storage's routes prefix and is mounted ahead of your routes, so a catch-all can't hide it. The endpoint serves only from its own disk. A request that arrives from another node never fans out again, so there are no loops and no request storms.

### What it costs

A local hit costs one `File.exist?`, plus one indexed blob lookup for a whole-file `download`. A miss on N nodes sends N − 1 requests. Only the node that has the file sends any bytes; the others answer 404 after a `File.exist?`. For the three to five nodes a typical stage runs, that is cheap enough that we dropped our first plan to record each file's node: saving a few 404s didn't justify an extra write on every upload and a backfill.

## Errors Rails already understands

The service raises subclasses of Active Storage's own errors, so your existing rescues keep working:

| Error | Superclass | When |
| --- | --- | --- |
| `NotHere` | `ActiveStorage::FileNotFoundError` | every node said "not here" |
| `Unavailable` | `ActiveStorage::Error` | a node that may have it didn't answer (retry) |
| `Mismatch` | `ActiveStorage::IntegrityError` | the only copies found don't match the blob |

## Deletes: inline or in a job

The local copy is always deleted at once. For the other nodes, pick a mode:

- `:sync` (default) deletes inline and raises `Unavailable` naming any node that didn't confirm.
- `:job` hands the remote deletes to `ActiveStorage::MultiNodeDisk::DeleteJob`, which retries until every node confirms and is safe to run twice.

```ruby
config.delete_mode = :job
config.delete_job_queue = :maintenance
```

## Who is calling

The gem checks *who is calling* its endpoint. It never decides *who may see a file*; that belongs to your app, on your app's own pages.

| `auth` | The endpoint needs |
| --- | --- |
| `:none` (default) | nothing |
| `:token` | `Authorization: Bearer <token>`, else 401 |
| `:ip` | the caller in `allowed_ips`, else 403 |
| `[:ip, :token]` | both |

`:none` is the default so a first install works on a closed network. The README warns about it and the gem logs a warning at boot in production. Use `:token` anywhere else.

Under `:ip`, the allowlist defaults to the nodes' own addresses, resolved from DNS and refreshed every 60 seconds. The caller is the socket's peer, and `X-Forwarded-For` counts only when that peer is one of your `trusted_proxies`, because a forwarded header is something any caller can write.

## An audit trail

When files are payroll exports or member lists, someone eventually asks who pulled which file, and from where. Every answered request and every remote fetch or delete becomes one versioned JSON document:

```json
{"v":1,"at":"2026-10-17T12:00:00.000Z","action":"read","result":"served","status":200,
 "key":"abc","filename":"members.csv","byte_size":22,"node":"app2","from_node":"app1",
 "remote_ip":"10.0.0.1","auth":["token"],"hop":true,"duration_ms":3.1,
 "requester":{"id":42,"label":"hr@example.com"},"requester_verified":true}
```

It goes to a JSON Lines file by default (`tmp/multi_node_disk/audit.log`), or to a table, a callable, or nowhere. It never contains a token, file contents or a signed URL. The same document is also sent as an `audit.multi_node_disk` notification.

The requester is up to you:

```ruby
config.audit_requester = ->(request) { {id: Current.user&.id, label: Current.user&.email} }
```

and for jobs:

```ruby
ActiveStorage::MultiNodeDisk.with_requester(id: user.id, label: user.email) do
  blob.open { |file| import(file) }
end
```

The requester travels with the request to the node that serves the file, so the audit line on node B says which user on node A asked. Node B marks it verified only when token or IP auth passed.

If you want the audit table, `bin/rails g active_storage:multi_node_disk:install --audit` writes the migration. It has one JSON `payload` column (`jsonb` on PostgreSQL, `json` on MySQL 8, a CLOB on Oracle), an index on `created_at` only, and names that fit Oracle's 30-character limit. Nothing runs until you migrate.

## Testing your app

The gem ships a test helper that runs fake nodes in process, each answered by the gem's real endpoint:

```ruby
RSpec.configure { |config| config.include ActiveStorage::MultiNodeDisk::TestHelper }

it "reads a file another node holds" do
  node_b = multi_node_disk_node("app2")
  multi_node_disk_move(blob, to: node_b)

  multi_node_disk_cluster("app2" => node_b, "app3" => :missing) do |requests|
    expect(blob.download).to eq(bytes)
  end
end
```

A node can also be `:wrong`, `:timeout` or `:down`, so you can spec what your app does when a node is out.

## Rules we kept

- **Extend Active Storage, don't wrap it.** One service class and one changed line in `storage.yml`, with no adapter layer in between.
- **No runtime dependencies beyond Rails.** `Net::HTTP`, system DNS, Rails credentials, `Socket.gethostname`. If you want HTTPX, IPv4 pinning or your own token scheme, there is a hook for it in the initializer.
- **No database unless you ask for one.** Only the opt-in audit table touches it.
- **Nothing at `/`.** No root route, no catch-all, no changes to `default_url_options`.

The README's [Advanced section](https://github.com/clearstackio/activestorage-multi_node_disk#advanced) lists every hook, including the full config of the app the gem was written for.

## Try it

```ruby
gem "activestorage-multi_node_disk"
```

It needs Ruby 3.3+ and Rails 8.1. The source, issues and changelog are on [GitHub](https://github.com/clearstackio/activestorage-multi_node_disk). It's MIT-licensed, and bug reports and pull requests are welcome.