Proxmox tasks, backups and alerts

Tasks#

Infrastructure > Hypervisors, the Tasks tab, lists what Proxmox did on every host (backups, updates, migrations, snapshots) and every action started from Tenvara, newest first. Each row shows the task, its status, the guest it concerned, the cluster and node, the customer, who started it and when.

Filter by Status, Cluster, Node, Customer and Started from (Proxmox itself or Tenvara), or type in the filter box to find a task, guest, node or cluster. Click a task to open it in the side panel with its log. A cluster page's All tasks link opens the same list for that cluster.

Tenvara keeps the logs of failed tasks from the last seven days and of every task it started. Proxmox forgets task logs when a node restarts; Tenvara does not. Tasks are kept for 90 days.

Actions you start from Tenvara appear in the list straight away as queued, then running, then OK or with the error Proxmox gave. The user shown is the Proxmox user that ran the task (the agent runs as root@pam); Tenvara records which technician asked on its own action record.

Backups#

Tenvara reads the results of vzdump backups from each node's task logs, one row per guest per backup run, including the archive name, size and storage, or the error. You see them:

• on each guest's page under Backups
• on the cluster page under Backup jobs (the jobs scheduled in Proxmox, their schedule, target and next run)
• on the overview under Backups that need a look
• in the Failed backups view of Infrastructure > Guests

A guest's backup is overdue when it is in an enabled backup job and has had no successful backup within the job's interval plus 12 hours (a day for a daily job, a week for a weekly one).

Tip: Tenvara reports on Proxmox's own backups. To back up guests with Tenvara Backup instead, see Backup.

Alerts#

Proxmox alerts appear in Alerts with the source Infrastructure, and can open tickets through your alert rules like any other alert. They are only raised for adopted clusters. Each kind can be switched off or on, for everyone or per customer, under Alert types.

Under Alert types these are grouped as Hypervisor host problem, Cluster problem, Hypervisor storage filling up, ZFS pool in trouble, ZFS pool scrub overdue or too fast, Virtual machine in error, Virtual machine stopped, Hypervisor backup failed, Virtual machine action failed and Hypervisor connection failing (an API connection to a cluster that keeps failing).

A storage alert also says when the storage's volumes were given more space than it has, for example "650 GB provisioned on 464 GB", because thin volumes can then fill it. Storage, node and quorum alerts are raised and cleared only by a real reading: a report taken while the cluster had no quorum, or one that came without figures for a storage, leaves the alert as it is, so an alert and its ticket do not close and reopen while the problem is still there.

ZFS pool alerts#

A ZFS pool alert says what is wrong in words that fit the pool's layout, and when it started:

• What and since when. A mirror or raidz that lost a disk says how many more disk failures it can take ("can take 1 more disk failure", or "it can take no more disk failures"). A pool of a single disk, or of striped disks, never says "lost redundancy": it says nothing protects its data. The alert's start is the time of the first ZFS error event since the pool was last cleared, so a pool in trouble for days does not look new.
• Why. For each disk in trouble, the kernel's last disk error (a SATA link error, interface CRC errors from a cable or connector, a command timeout, a controller reset or a media error), the disk's SMART basics, and a reading such as "So: a link error, the drive itself is healthy." or "So: the drive is failing."
• What to do. When the disk itself is healthy and the pool has no damaged files, the alert offers Clear the errors and scrub: the node's agent clears the error counts, starts a scrub and follows it. A clean scrub closes the alert; a scrub that finds errors leaves it open and says what it found. It is not offered while a scrub or resilver is running, while a disk in trouble is unreachable or failing, or while the pool has damaged files. Otherwise the alert names the disk to replace (model and serial) or the damaged files to restore first.

For a cluster read through an API connection there is no shell on the node, so the alert tells you to run the clear and scrub on the node yourself once the disk checks out.

What the agent reports and how often#

Each node's agent sends a report on connect, every five minutes once the cluster is adopted, and after every action. A report covers:

• the cluster (name, quorum, members) and every node, guest and storage pool
• each guest's configuration (operating system type, cores, memory, disks, networks, start at boot, guest agent), addresses from the guest agent and snapshots
• backup jobs and the recent tasks, with logs of failed tasks
• the health, layout and error counts of every ZFS pool, and the updates waiting on each node
• per-minute CPU, memory, network and disk figures, which Tenvara rolls into hourly averages for the charts

If a part of the report fails (for example Proxmox took too long to answer), Tenvara keeps what it already had for that part rather than blanking it.