Server troubleshooting

What to check first

Four commands, in this order. Most problems are identified by the second one, and running them in order saves you from fixing the wrong thing.

kernctl status      # is it up, and what does it think it is?
kernctl health      # every check, grouped, with what to do about each
kernctl doctor      # this machine, and the path from here to the server
kernctl logs        # what it said while going wrong

health is the one that pays off: it covers the database and its migrations, the connection pool, both secrets, whether PUBLIC_URL matches reality, whether the management API is exposed, tokens, mail, expired sessions, stale invitations, unused whiteboard images, the stream log, record sequence integrity, orphaned records and table sizes. Anything repairable, --fix repairs.

doctor is for when you can't reach the server at all — it tests the path rather than the server, so it tells you whether the problem is between you and it.

Then, from the server itself, the one request that bypasses everything else:

curl -s http://127.0.0.1:8787/kern/v1/health

If that answers and your app can't, the problem is the proxy or TLS between them — not the server. That one split decides which half of this page you need.

Can't reach the server

Symptom. The app says it can't reach the server; the address does nothing in a browser.

Confirm which half is broken, from the server's own shell:

curl -s http://127.0.0.1:8787/kern/v1/health   # the server
kernctl service status                          # is it even running?
curl -sI https://kern.example.com/kern/v1/health  # the proxy, from outside

If the server died at startup, read the first twenty lines of its log rather than the last — a missing DATABASE_URL or an unreachable database says so immediately and then everything after it is noise.

Answered, but not Kern

Symptom. The app says something replied at that address, but it isn't a Kern server.

Why. Before anything else, the app asks the address what it is. A Kern server identifies itself; anything else fails this check rather than failing later in a way nobody can diagnose.

Confirm:

curl -s https://kern.example.com/kern/v1/instance

You should get JSON naming the service, a protocol number and the instance name.

Addresses are tidied on the way in: a bare hostname gets https://, and /kern/v1 is appended if it isn't there. So kern.example.com and https://kern.example.com/kern/v1/ are the same server — but https://kern.example.com/kern is not.

Refuses plain HTTP

Symptom. Kern won't sign in to your server over http://.

Why. The session token is sent as a bearer on every request. Over plain HTTP to a public address, that token and everything synced would cross the internet in the clear, so the app refuses rather than letting you do it by accident.

Fix. Either put TLS in front of it, or make the address genuinely local:

ssh -L 8787:127.0.0.1:8787 you@server

Then sign in to http://127.0.0.1:8787, which is true: that is this machine, and the traffic is inside SSH. Caddy will get you a certificate in two lines if you have a hostname — see putting TLS in front of it.

Protocol too new

Symptom. The app refuses the server and names two numbers — the protocol the server speaks and the one the app implements.

Why. The server reports a protocol version. An app refuses a server whose protocol is higher than its own, deliberately and immediately, instead of connecting and then failing somewhere incomprehensible.

Fix. Update the apps. To see where the server stands:

curl -s https://kern.example.com/kern/v1/instance

When you run the server for other people, update apps before the server, or tell people first with kernctl notice. See the handshake.

No live collaboration

Symptom. Sync works — changes arrive, eventually — but you never see anyone's cursor, whiteboards aren't multiplayer, and edits only appear after a reload or a delay.

Why. This is the most common self-hosting mistake, and it is nearly always one thing: the reverse proxy isn't passing Upgrade and Connection. Sync is ordinary HTTP and keeps working, so nothing looks broken; the realtime socket is a WebSocket upgrade and silently doesn't happen.

Confirm — ask for an upgrade and look at the status:

curl -sI -o /dev/null -w '%{http_code}\n' \
  -H "Connection: Upgrade" -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
  https://kern.example.com/kern/v1/realtime

A 400 or 401 means the request reached Kern — the socket path works, and it is only refusing because there is no ticket. A 200 with an HTML body, or a 426, means the proxy answered instead of forwarding the upgrade.

Fix, in nginx:

proxy_http_version 1.1;
proxy_set_header Upgrade    $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 300s;

Caddy needs none of this. If you have Cloudflare in front, WebSockets are on by default for tunnels but check they aren't disabled on a proxied hostname.

proxy_read_timeout matters too: with a short timeout the socket connects and then drops every minute or so, which looks like flaky collaboration rather than a configuration problem.

One more cause, if everything above is right: running several API instances. Realtime rooms live in one process, so two instances behind a load balancer means people in the same space may be connected to different ones and never see each other. Run one instance, or add shared pub/sub.

Links go to the wrong place

Symptom. Invitation emails, share links or stream addresses point at localhost, a bare IP, or a hostname that doesn't resolve for the person who got them.

Why. Those links are built from PUBLIC_URL. The server cannot infer its public address from a request, so if that setting still says http://localhost:8787, that is what people receive.

Confirm:

kernctl settings | grep -i url
kernctl health            # it compares PUBLIC_URL against what the proxy serves

Fix. Set it to the origin people actually use, and restart:

PUBLIC_URL=https://kern.example.com

Already-sent invitations keep the old address, so reissue them: kernctl invites list, then resend from the app. Existing stream endpoints change address with PUBLIC_URL unless STREAMS_URL pins them — anything already posting to the old address will need updating.

Nobody gets codes

Symptom. Sign-up or invitation emails never arrive, so nobody can verify an address.

Why. Either no SMTP is configured — in which case codes are printed to the server log rather than sent — or SMTP is configured and the mail is being rejected or filtered.

Confirm:

kernctl health            # reports mail as a warning while SMTP is unset
kernctl logs --level warn
curl -s https://kern.example.com/kern/v1/instance   # features.email

kernctl users list --unverified is the list of people who got stuck, which is what to work through once mail is fixed.

Can't sign up

Symptom. Someone reaches the server, but it won't create their account.

Confirm what the instance is telling apps:

curl -s https://kern.example.com/kern/v1/instance   # registration
kernctl settings | grep -i registration

On a brand-new server, the first account is always allowed whatever the mode says — so if even the first sign-up fails, the problem is mail or connectivity, not registration.

Management API refuses me

Symptom. kernctl or Kern Platform is refused, often with admin_local_only.

Why. There are two gates, in this order: where the request came from, and then the token. The refusal names which one stopped you, and the difference matters — admin_local_only means the address list refused you, not your token. Nothing is wrong with your credentials.

If you put the management API behind a reverse proxy, the proxy is what the allow list sees, because the client address is taken from the socket and never from a header — trusting a header there would undo the gate. Restrict access at the proxy instead.

Lost the token

Symptom. No management token to hand, and the one printed at first start is long gone.

Fix. On the server's own machine, you don't need one:

sudo cat /var/lib/kern/admin-token    # or wherever ADMIN_TOKEN_FILE points
kernctl token list                     # reads that file automatically
kernctl token new "laptop"             # mint a fresh one, shown once

A token's secret half is stored only as a hash, so an existing token genuinely cannot be recovered — mint a new one and revoke what you've lost:

kernctl token revoke <id>

If the token file is gone too, it is written on first start when no tokens exist — so a server with tokens won't recreate it. Mint one from the server's shell, which reads the file if present and is otherwise the local path in.

Logs are empty

Symptom. kernctl logs shows nothing, or nothing from before a restart.

Why. It reads a ring buffer in the server's memory (ADMIN_LOG_LINES, 2000 by default), so a restart clears it. That is a feature — no log files to rotate — but it is not history.

For history, read the process log:

docker compose logs api            # Docker
journalctl -u kern-server          # native

Raise ADMIN_LOG_LINES if you keep running out of buffer, and remember the process log is where the startup messages are — the ring buffer may have scrolled past them.

Image uploads fail

Symptom. Dropping an image on a shared whiteboard fails, or fails only for larger images.

Unused shared images are cleaned up on their own after a week, and kernctl health reports how many are waiting — so a space near its quota is usually holding images nothing references any more.

Streams refusing things

Symptom. A form or webhook posts to a stream and gets a refusal, or nothing appears in the space.

First, read the delivery log. Every submission is logged with why it was refused, in the app under the space menu → Streams…, and across the instance with:

kernctl streams list

The streams documentation covers each refusal in detail. The ones specific to self-hosting:

A stream being abused from outside: kernctl streams disable <id> stops it accepting anything without deleting it or losing its settings.

Running out of disk

Symptom. The disk fills, or the database is much larger than you expected.

Confirm:

kernctl health          # table sizes, unused images, stream log, old deleted spaces
kernctl status          # how much is in the instance
kernctl spaces list     # per-space sizes
df -h

Whiteboard images are what grow. Everything else — tasks, notes, boards — is small text. Then:

kernctl health --fix    # expired sessions, stale invitations, unused images, stream log
kernctl heal            # repair everything, then check again

These repairs are safe by construction: each deletes something that has already expired or lost its parent. To stop it recurring, cap the inputs — ASSET_QUOTA_MB, ASSET_MAX_BYTES, STREAM_LOG_MAX_DAYS on the configuration page.

Spaces deleted more than 30 days ago are reported by health and can be purged with kernctl spaces delete <id> --purge. Keep an eye on backup files too — a nightly pg_dump with no pruning is a common way to fill a disk.

Database problems

Symptom. The server won't start, or health reports the database or its migrations.

kernctl query "SELECT …" gives a read-only look at the database when you need to see for yourself. It cannot write, so it is safe to use while diagnosing.

After a restore

Symptom. You restored a backup, and clients behave oddly — changes from after the backup have gone, or a space resynchronises from scratch.

Why, and it is the right outcome. Restoring replaces the database. A space whose record sequence has gone backwards is restored from the server on the app's next sync, so everyone converges on what the server now holds — which means changes made after the backup are gone.

What to do:

• Tell people before you restore, with kernctl notice.
• Restore .env alongside the database — specifically STREAMS_ENCRYPTION_KEY, without which restored stream credentials are unreadable, and BETTER_AUTH_SECRET, without which everyone is signed out.
• Run kernctl health --fix afterwards to clear anything the restore orphaned.

If people have work from after the backup still on their machines, have them export it before they next sync — that window is the only chance.

An upgrade went wrong

Symptom. After an upgrade the server won't start, or apps now refuse it.

Back up first. It makes an upgrade reversible:

kernctl backup /srv/backups/pre-upgrade-$(date +%F).sql.gz
kernctl service upgrade

Ordinary upgrades need no downtime window and no migration step. Send a notice first if people are connected, since the service restarts.

Reporting a problem

If none of the above fits, collect these before opening an issue:

kernctl status --json
kernctl health --json
kernctl doctor
kernctl logs --level warn
curl -s https://kern.example.com/kern/v1/instance

Say what you expected, what happened, and whether the server is Docker or native. Check the output before posting it — settings and logs can include addresses and email addresses, though none of it carries anybody's tasks or notes.

Server issues belong in the Kern-Cloud-Self-Host repository; problems with the desktop app itself go to Kern.