Operating the fleet server

How the fleet server is deployed on genossec.com, and how to change, update and troubleshoot it.

How it's deployed

flowchart LR
  net(["Internet"]) -- "HTTPS :443" --> nginx["nginx<br/>Let's Encrypt cert"]
  nginx -- "/" --> site["genossec.com site<br/>127.0.0.1:8000"]
  nginx -- "/dashboard/, /v1/, /baseline" --> fleet["Fleet server<br/>127.0.0.1:8003"]
  fleet --> db[("SQLite<br/>/var/lib/genos-fleet")]

• nginx terminates TLS and routes three paths to the fleet server. The routes are in fleet/deploy/nginx-genos-fleet.conf, included in the genossec.com site.
• The fleet server is a separate process, genos-fleet.service (gunicorn), listening only on localhost. It shares nothing with the website except the domain.
• Data is one SQLite file plus the session key, in /var/lib/genos-fleet/ (mode 0700). The server refuses to start if that directory is readable by other users or on a shared path like /tmp.

Install or update

cd ~/genos/genos-server
sudo fleet/deploy/install.sh

Safe to re-run after every pull. It: 1. Writes /etc/genos-fleet/config.toml if missing. 2. Installs Python dependencies. 3. Installs the systemd unit and restarts the service. 4. Adds the nginx routes, after backing up the site config. It restores the backup if nginx -t fails. 5. Prints the certificate pin.

Code-only changes don't need sudo. Reload the running service:

kill -HUP $(systemctl show -p MainPID --value genos-fleet)

A reload also re-reads dashboard.env and republishes the catalog. It does not pick up a changed systemd unit or nginx config; re-run install.sh for those.

Configuration

/etc/genos-fleet/config.toml

The file is owned by root. Edit with sudo, then reload.

Expected hashes

Tell the server which versions you expect, and any node that differs raises hash drift:

[expected]
scanner_version = "0.3.0"
policy_wasm_sha256 = "…"
bundle_sha256 = "…"

# Override for one node, by hostname or agent_id:
[agents."web-01"]
baseline_canonical_hash = "…"

Leave a value out to skip that check.

Who can sign in

Dashboard sign-in uses Clerk. Settings live in dashboard.env at the repo root (git-ignored, mode 0600):

NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_...
FLEET_ADMIN_USER_IDS=user_abc...,user_def...

Only the Clerk users listed in FLEET_ADMIN_USER_IDS get in. To add someone: have them sign in once. The page shows their user ID and the line to add. Add the ID, separated by a comma, then reload. The server verifies sign-ins with Clerk's public keys, so it doesn't need a Clerk secret key.

Adding a node

The dashboard's Install tab mints a one-time key and shows the exact command to run on the host:

curl -fsSL https://genossec.com/install/<key> | sudo bash

GET /install/<key> returns the installer only for a valid, unused, unexpired key — that's the "an admin minted this" check — and it changes no state (the key is consumed later by the daemon at /v1/enroll). The installer writes the host's agent identity and config, pins this server's certificate, and offers to run the baseline scan. The key is a normal enrollment token, so create-token works too.

The /install/ path must be routed to the fleet app in nginx. deploy/install.sh adds it; after pulling this change, re-run it once:

sudo fleet/deploy/install.sh

Everyday tasks

All of these run from ~/genos/genos-server/fleet with FLEET_CONFIG set:

cd ~/genos/genos-server/fleet
export FLEET_CONFIG=/etc/genos-fleet/config.toml
FLASK=".venv/bin/flask --app fleetmon:create_app()"

Removing a host's baseline

Each host gets one baseline, and a second upload is refused with 409. When a host is rebuilt, remove its old baseline so the new install can upload:

$FLASK remove-baseline web-01

Also delete /var/lib/genos-inventory/baseline.json on the host, or its scanner refuses to run.

Certificate pin

Daemons pin the public key of genossec.com's certificate. Let's Encrypt renews the certificate every ~60 days. The installer's --pin-key option set reuse_key = True for genossec.com, so renewals keep the same key and the pin stays valid.

Don't remove reuse_key. A new key means every daemon refuses to connect, and every node goes red at once. To rotate the key on purpose, first roll a second (backup) pin out to every node, then rotate.

Publishing the catalog

/dashboard/catalog serves the catalog bundle built from genos_inventory/catalog at startup. After changing the catalog, reload the service. The SHA-256 on the dashboard's Catalog page changes, and scanners pinned to the old value will refuse the new one until you update their pin.

Backups

Everything is in /var/lib/genos-fleet/. To copy the database safely while the service runs:

$FLASK backup ~/fleet-$(date +%F).db

The copy is created with mode 0600.

Keep backups private: they hold host metadata. secret_key in the same directory only signs dashboard sessions; losing it just signs everyone out.

Troubleshooting