Skip to main content

Monitoring credentials the server can't read

· 4 min read
Florent Clairambault
Maintainer of SolidPing

A monitoring tool needs your secrets. To check a Postgres primary it needs the password. To check an authenticated endpoint it needs the token.

Most tools keep those secrets encrypted in their database and decrypt them on the server. That protects you from a stolen backup. It does not protect you from the server itself.

I wanted a stronger option for checks that run inside your own network. This is how SolidPing does it, and where it stops working.

The setup​

You can run an agent inside your network. It's the same binary, started with SP_NODE_ROLE=agent. It dials out to the server over a WebSocket, so no inbound port is open.

On first start the agent generates two keypairs locally:

  • Ed25519, to identify itself
  • X25519, to receive secrets

The private halves stay in a file (mode 0600) or in an env var. The server only ever gets the public keys.

Sealing​

When you save a check that runs only on that agent's region, the server encrypts its secret fields to the agents' X25519 public keys, using age. It then stores the ciphertext and forgets the plaintext.

The stored envelope is a small JSON blob:

{"v": 2, "alg": "age-x25519", "fingerprints": ["..."], "ct": "<base64 age file>"}

The server holds no matching identity, so it can't decrypt this. When the agent claims a job, the server ships the envelope as is. The agent unseals it in memory and runs the check.

If you have three agents in a region for HA, the blob is sealed to all three. Any of them can open it.

Failing closed​

What if no agent has enrolled yet? My first version quietly fell back to a key the server could read. I reverted that. A private-only check with nothing to seal to is now rejected at write time with a validation error.

A guarantee that silently turns off is not a guarantee.

What it costs​

Two things are worse than a normal setup.

When an agent is added or revoked, the recipient list changes. The server can't re-seal a sealed-only check, because it can't read it. You re-save the credentials. Checks that also run from cloud regions are fine: the server keeps a second copy it can read, and re-seals that one.

And if you lose the agent's keys, you lose the secrets. You enroll a new agent and re-enter them.

What it doesn't protect against​

Be clear about this part.

The server sees the secret once, when you submit it. It seals it and keeps only the ciphertext. So a stolen database, a backup, or someone with read access to the tables gets nothing usable for these checks.

It does not help if the server is actively malicious. The public keys come from the server, so a hostile operator could seal to a key of their own, or just log the request. If you self-host, you are the operator and this doesn't matter. If you use a hosted service, it's a protection against what is stored, not against whoever runs it.

The dashboard shows a fingerprint for each agent, and the agent logs one when it enrolls. But it identifies the agent's signing key, and nothing forces you to compare the two. Pinning the sealing key at enrollment would be the real fix. It's not built yet.

The agent also holds your secrets in memory while it runs checks. That's unavoidable, it needs them.

Why bother​

Because the usual answer to "can your staff read my database password" is "we promise they can't". I'd rather have "the data isn't there".

The code is in server/internal/crypto/credentials/sealing.go, about 230 lines.