A server stopped reporting

If a server's graphs have gone flat, or you received an email saying it stopped reporting, this article helps you work out why and fix it. It's for anyone who manages servers in InfraNest.

Overview

• Either the server itself is down, or the agent on it can't get its readings to us.
• Your dashboard's Server metrics widget shows Not reporting with a number whenever any server has gone quiet — click it to see which ones.
• This article helps you tell the two apart, and fix whichever it is.

See when we last heard from it

1. Go to Monitoring, select the Servers tab, and open the server.
2. On the Overview tab, find the Agent panel. It says last heard from followed by a time.

That time is the one to trust. If it's more than a few minutes ago, nothing is arriving.

3. If the panel says Readings refused, the agent is reaching us but we can't keep what it sends. The panel tells you why, and how to fix it. Today that's always the server's clock: it says how far ahead or behind the clock is, and gives you the command that switches on time sync. Once the clock is right, readings arrive again with the agent's next report.

Ask the agent itself

If the Agent panel doesn't explain it, the agent can tell you what's wrong, even when it can't reach us.

1. On the server, run:

sudo infranest-agent status

2. On Windows, run this in PowerShell as Administrator instead:

& "C:\Program Files\InfraNest\infranest-agent.exe" status

3. Read the result. It says when the agent last delivered a reading, how many readings are waiting, and why it's stuck.

NoteIf you get no answer at all, the agent isn't installed or isn't running. Check that the server is up, then restart the agent with sudo systemctl restart infranest-agent on Linux.

What the answer means, and what to do

The token was rejected. The agent's token was deleted in InfraNest, or it belongs to a different server. InfraNest doesn't show this on its side, so the agent's own status is the only place you'll see it.

1. Open the server.
2. In the Agent panel, select Install the agent, then Install it again. This creates a new token.
3. Run the new command on the server.
4. Delete the old token with the bin icon next to it.

Monitoring is switched off. Monitoring is off for this server in InfraNest. Nothing is wrong with the agent or its token, so there's nothing to reinstall. Switch monitoring back on with Switch monitoring on and the agent picks up again by itself.

The clock is off. The server's time is wrong, and we turn away readings that claim to come from the future, or that look too old to keep. The Agent panel shows this too, as Readings refused, with how far off the clock is. Switch on time sync with sudo timedatectl set-ntp true on most Linux systems, or w32tm /resync in PowerShell as Administrator on Windows.

It can't connect. The agent can't reach InfraNest. It only needs to make outgoing secure web connections (HTTPS, port 443) to ingest.infranest.io. Check that a firewall, security group or proxy on the server's network allows that. No incoming ports are needed.

If none of these fit, run sudo infranest-agent flare and paste the result into a support ticket. It contains everything we need, and leaves out your token.

Nothing is lost while it's quiet

The agent keeps its readings on the server for about eight hours and sends them once the connection is back. So if you fix a network problem within that time, the gap in your graphs fills itself in.

Get told the next time

1. Go to Monitoring → Alert defaults.
2. Under Is anything broken?, switch on The server has gone quiet.
3. Set how long InfraNest should wait before alerting. Allow a few minutes of silence, not one — ten is a comfortable choice. If you pick less than your plan can judge, InfraNest tells you the shortest you can set.

You can change this for one server on that server's Alerts & rules tab.

TipThis alert also covers a server whose readings we're turning away. It's still in touch, but nothing it sends is kept, so after the same few minutes it counts as gone quiet, and the alert says it's the clock.

Good to know

• Deleting a server's last agent token doesn't uninstall the agent. It keeps running and retrying until you remove it from the machine.
• Each server needs its own token. If two servers share one, we can't tell when one of them goes quiet.