Operations and recovery¶

Operate the node directly through its official Compose project.

A container marked running is not enough. Use chain ID, advancing head, head freshness, gossip, peers, restart state, API probes, disk growth, memory pressure, and logs as operational evidence.

Routine health¶

Run the complete synchronization and health procedure. It fails when a required service is absent, a restart count increases, a container is recreated, the head is stale or does not advance, gossip is disabled, or no connected peer is visible.

For a public Seed Node, also monitor external P2P reachability, DNS stability, inbound peer activity, and continuity of the advertised Peer ID.

For a quick view from the official checkout:

cd /opt/koinos
docker compose ps
docker compose logs --tail 100 chain p2p block_store
df -h /var/lib/koinos
du -sh /var/lib/koinos/*
docker stats --no-stream

Use docker compose logs --since and --until for a specific incident window. Koinos also writes service logs below each service's configured log-dir. Locate and measure them instead of assuming Docker contains the only history:

find /var/lib/koinos -type f -path '*/logs/*' -print
du -sh /var/lib/koinos/*/logs 2>/dev/null
docker info --format 'Docker logging driver: {{.LoggingDriver}}'

Set a capped retention policy for both the service log directories and the active Docker logging driver. Alert before the filesystem reaches the documented warning threshold, when the head stops advancing, gossip disables unexpectedly, peers disappear, or a container repeatedly restarts.

Clean service control¶

Stop all configured services cleanly:

docker compose stop
docker compose ps

Start or recreate them after a confirmed configuration change:

docker compose up -d
docker compose ps

A plain restart neither pulls images nor necessarily applies Compose changes. Restart a single service only when its dependencies and data contract make that safe.

Update and rollback¶

Prepare a new release or recorded deployment commit in a separate checkout. Do not overwrite the running checkout before comparison.

Create the proposed checkout and select the release tag or commit:

sudo git clone https://github.com/koinos/koinos.git /opt/koinos-next
sudo chown -R koinos:koinos /opt/koinos-next
read -r -p 'Selected release tag or commit: ' next_revision
test -n "$next_revision"
git -C /opt/koinos-next checkout "$next_revision"
git -C /opt/koinos-next rev-parse HEAD

Record both revisions:

git -C /opt/koinos rev-parse HEAD
git -C /opt/koinos-next rev-parse HEAD

Compare the operator-controlled files:

diff -u /opt/koinos/env.example /opt/koinos-next/env.example || true
diff -ru /opt/koinos/config-example /opt/koinos-next/config-example || true
diff -u /opt/koinos/docker-compose.yml \
  /opt/koinos-next/docker-compose.yml || true
diff -u /opt/koinos/env.example /opt/koinos/.env || true
diff -ru /opt/koinos/config-example /opt/koinos/config || true

The first three comparisons show upstream changes. The final two show the current operator's local changes. Use both sets as review input; do not copy the old configuration blindly.

Create the new active files from the proposed release or commit, then use your editor to apply only the confirmed local values:

cd /opt/koinos-next
cp env.example .env
cp -R config-example config
sudoedit .env
sudoedit config/config.yml
docker compose config

Transfer the absolute basedir, loopback bindings, profiles, credentials, peer settings, and producer configuration carefully. Preserve the new genesis, descriptors, RabbitMQ configuration, and required version-specific settings unless the confirmed upstream diff says otherwise.

Before downtime:

1. record current and proposed revisions, profiles, tags, and image digests;
2. review Compose, .env, config, genesis, descriptors, and RabbitMQ changes;
3. validate the proposed checkout with docker compose config;
4. pull its exact pinned images;
5. verify free disk space;
6. preserve configuration, encrypted keys, P2P identity, and recoverable data;
7. define rollback triggers and a maintenance window.

The current checkout at /opt/koinos naturally uses the Compose project name koinos. Pass that name explicitly from both directories so the proposed checkout updates the same project instead of creating a second koinos-next project.

Pull the selected images, stop the current services, and activate the proposed checkout:

cd /opt/koinos-next
docker compose --project-name koinos config
docker compose --project-name koinos pull

cd /opt/koinos
docker compose --project-name koinos stop
docker compose --project-name koinos ps --all

cd /opt/koinos-next
docker compose --project-name koinos up -d --remove-orphans
docker compose --project-name koinos ps

Re-run every health and protocol check. Keep the old checkout, images, and snapshot until the new deployment has remained healthy.

If a software or configuration check fails and the data format remains compatible, roll back to the prior checkout directly:

cd /opt/koinos-next
docker compose --project-name koinos stop

cd /opt/koinos
docker compose --project-name koinos config
docker compose --project-name koinos up -d --remove-orphans
docker compose --project-name koinos ps

Repeat every health and protocol check. If the selected release changes an on-disk format incompatibly, stop and follow that release's documented data rollback procedure before starting the prior version; do not improvise by mixing data directories.

Choose the smallest recovery action¶

Before reindexing, resynchronizing, or replacing data:

1. identify the absolute basedir and network;
2. record the chain ID and active configuration;
3. stop the node cleanly;
4. verify free space;
5. preserve configuration, P2P identity, and every key;
6. move old data to a dated rollback directory instead of deleting it;
7. define the validation and rollback steps.

Reindex¶

Reindex replays stored blocks to rebuild derived state. Add reset: true only under the affected service in a copy of the active config, review the diff, and start only the required service and dependencies. Remove reset: true immediately after initialization consumes it, so a later restart cannot reset the service again.

Never use a global reset when only chain, transaction_store, contract_meta_store, or account_history needs rebuilding.

The following direct procedure reindexes chain from the existing block_store. It replaces the active chain state and assumes the standard paths used throughout this guide.

Stop cleanly, preserve the active configuration and chain state, and record the recovery directory:

cd /opt/koinos
docker compose stop
docker compose ps

reindex_backup="/var/lib/koinos-before-reindex-$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 750 -o koinos -g koinos "$reindex_backup"
sudo cp -a /opt/koinos/.env /opt/koinos/config "$reindex_backup/"
sudo cp -a /var/lib/koinos/chain "$reindex_backup/"
printf '%s\n' "$reindex_backup" |
  sudo tee /var/lib/koinos/reindex-rollback-directory.txt

Do not continue unless the stopped state and copied directories have been inspected. Edit config/config.yml and set both values only under chain:

chain:
  reset: true
  verify-blocks: true

Review and start the affected service:

cd /opt/koinos
diff -u config/config.yml \
  "$(cat /var/lib/koinos/reindex-rollback-directory.txt)/config/config.yml" ||
  true
docker compose config
docker compose up -d chain
docker compose logs --tail 100 --follow chain

Wait for the verified Resetting database... message. Then remove reset: true immediately, keep verify-blocks: true, and validate the normal configuration without restarting the reindex:

sudoedit /opt/koinos/config/config.yml
cd /opt/koinos
docker compose config
docker compose up -d
docker compose logs --tail 100 --follow chain p2p

The reindex is complete only after Chain state is synchronized with block store appears and the full health procedure passes. If it fails, stop the node and roll back directly:

cd /opt/koinos
docker compose stop
reindex_backup="$(cat /var/lib/koinos/reindex-rollback-directory.txt)"
failed_reindex="/var/lib/koinos-failed-reindex-$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 750 -o koinos -g koinos "$failed_reindex"
sudo mv /var/lib/koinos/chain "$failed_reindex/"
sudo mv /opt/koinos/.env "$failed_reindex/env.failed"
sudo mv /opt/koinos/config "$failed_reindex/config.failed"
sudo cp -a "$reindex_backup/chain" /var/lib/koinos/
sudo cp -a "$reindex_backup/.env" /opt/koinos/.env
sudo cp -a "$reindex_backup/config" /opt/koinos/config
docker compose config
docker compose up -d
docker compose ps

Keep both chain-state copies until the restored node passes the full health procedure.

Full resynchronization¶

Use a full P2P resynchronization only when a narrower repair or a verified backup restore is unsuitable. With the node stopped, move core and optional state into a dated preservation directory. Retain config, genesis, descriptors, P2P identity, and keys. Start the required services against empty state and monitor the network, chain ID, peers, gossip, head, disk, and logs throughout synchronization.

Stop the node, verify the intended basedir, and create a recoverable destination:

cd /opt/koinos
docker compose stop
docker compose ps

koinos_basedir=/var/lib/koinos
test "$koinos_basedir" = /var/lib/koinos
test -d "$koinos_basedir"
resync_backup="/var/lib/koinos-before-resync-$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 750 -o koinos -g koinos "$resync_backup"
printf '%s\n' "$resync_backup" |
  sudo tee "$koinos_basedir/resync-rollback-directory.txt"

Move only public and rebuildable service state:

resync_backup="$(cat /var/lib/koinos/resync-rollback-directory.txt)"
for state_dir in \
  chain block_store mempool transaction_store contract_meta_store account_history
do
  if sudo test -e "/var/lib/koinos/$state_dir"; then
    sudo mv "/var/lib/koinos/$state_dir" "$resync_backup/"
  fi
done

Do not move p2p, block_producer, the selected checkout, .env, or config. Confirm that neither the global section nor any service retains reset: true, then validate and start:

cd /opt/koinos
docker compose config
docker compose up -d
docker compose ps
docker compose logs --tail 100 --follow p2p block_store chain

Run the full synchronization and health procedure until the head is fresh and advancing. If rollback is required, use the recorded path:

cd /opt/koinos
docker compose stop
resync_backup="$(cat /var/lib/koinos/resync-rollback-directory.txt)"
failed_resync="/var/lib/koinos-failed-resync-$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 750 -o koinos -g koinos "$failed_resync"
for state_dir in \
  chain block_store mempool transaction_store contract_meta_store account_history
do
  if sudo test -e "/var/lib/koinos/$state_dir"; then
    sudo mv "/var/lib/koinos/$state_dir" "$failed_resync/"
  fi
  if sudo test -e "$resync_backup/$state_dir"; then
    sudo mv "$resync_backup/$state_dir" /var/lib/koinos/
  fi
done
docker compose config
docker compose up -d
docker compose ps

Keep the failed and restored data copies until the node passes the full health procedure.

Restore¶

Use the staged public backup procedure only when its network, date, layout, and source fit the recovery. It accelerates restoration of public chain data; it is not a source of local identity or authority.

Incident evidence¶

Do not repeatedly restart a failing database. Preserve:

• the exact error and the relevant last log lines or incident time range;
• docker compose ps;
• image tags and digests;
• selected release or commit and configuration diff;
• disk, filesystem, and kernel status;
• the last clean shutdown and latest valid head.

Clone or snapshot affected data before experimenting. Report reproducible evidence through the Koinos issue tracker, omitting credentials, private keys, private network details, and sensitive archives.