Proxmox clusters and nodes

Tenvara reads and manages Proxmox VE in one of two ways:

• The agent on each node (recommended). The agent talks to the node's own local API as root, so there is no API user, token or secret to create on Proxmox, nothing listens on the network, and there is no self-signed certificate to trust. Because every node runs an agent, the cluster stays visible when any one node is down. Node maintenance mode and host updates need this way.
• An API token, for a cluster where you cannot or do not want to install the agent on the nodes. Tenvara reads it itself, or a device of the customer on the same network reads it.

With the agent on each node#

1. Install the Tenvara agent on every node of the cluster, the same way as any Linux server. See Installing the agent.
2. Check the Hypervisor role is on for those devices in Settings > Devices and agent. It is on by default.

Within a few minutes of the agent connecting, the node reports the cluster (or itself, if it is a standalone node), every guest, storage, backup jobs and recent tasks. The agent reports on connect and then every five minutes (every 15 minutes before adoption), and again after every action.

With an API token#

1. In Proxmox, create an API token for a user (for example monitor@pve!rmm) and give it the permissions below.
2. In Tenvara, go to Infrastructure > Hypervisors, open the Proxmox VE connections tab and click Add a Proxmox VE cluster.
3. Choose the Customer and enter a Name (shown until the cluster reports its own).
4. Enter the Host (any node, by name or address) and Port (8006 by default), the Token ID and the Secret. The secret is kept encrypted and never shown again.
5. Under Certificate, choose System trust, Cluster CA, Trusted certificate (compare and pin its fingerprint) or Do not check.
6. Under Read by, choose The server (Tenvara reads it every 5 minutes unless changed in Settings > Infrastructure; it must be able to reach the node on that port) or A device of the customer on the same network. When the server reads a node on a private address, tick Inside the network on purpose; the choice is recorded in the audit log.
7. Click Test connection. It shows the version, the cluster name, the nodes online, the guests, and what the token may and may not do.
8. Click Add and read. A cluster connected this way is adopted from the start.

Token permissions#

Tenvara reads what the token may do and lists it, feature by feature, on Test connection. A read Proxmox refuses becomes a warning on the connection and the cluster page naming what is hidden and the permission that gives it; it never fails the read or removes a guest. An action the token may not run is refused before anything is sent.

With privilege separation on (the Proxmox default), the token's own permissions count, not its user's. A token with far more rights than needed gets a warning.

Adopting a cluster#

A newly reported cluster appears as Detected. You can look at everything it reported, but Tenvara will not queue any action on it or raise alerts for it until it is adopted. Adoption is for the whole cluster, not node by node.

1. Open Infrastructure > Hypervisors and its Detected tab, which shows how many hosts are waiting.
2. Open the detected cluster or standalone node.
3. Press Adopt.
4. Confirm the Customer. It defaults to the customer whose devices reported it; change it only if the hardware belongs to someone else.
5. Confirm. Adopting tells every node's agent it may now accept actions.

You need the Manage permission for Infrastructure to adopt. From the hypervisor list you can also press a on a detected row to adopt it.

Ignoring a host you do not manage#

Not every detected host is yours to look after: a customer's own lab, or a box on its way out. Press Ignore beside Adopt (on the host's page, on the list for the rows you pick, or on the node's device card) and add a reason if you like. An ignored host leaves the list, the Detected tab, the overview's counts and the guest list, and later reports never bring it back. The ignored hosts are listed with who ignored them, when and why, each with Un-ignore. Adopting an ignored host takes the ignore off first.

An adopted host can go back the other way: Release back to detected in the page's More actions menu clears its alerts and its agents refuse actions again. A cluster added through an API connection was added on purpose and is not released: remove the connection instead.

Note: A guest page on a detected host says so: its figures are shown as reported, and power actions, snapshots and backups start once the host is adopted for its customer.

The hypervisor list#

Infrastructure > Hypervisors opens on the Clusters tab, which lists every cluster and standalone node with its Health, Platform, Customer, Nodes online, Guests up, CPU, Memory and Fullest storage. Filter by Health, Platform, Adoption and Customer, save views and choose columns like any other list.

Health is Healthy, Needs attention or Critical, and the reasons show under the name: a node offline, the cluster losing quorum, storage filling up, a failed backup and so on.

The page's tabs are Clusters, Detected, Proxmox VE connections, vSphere connections and Tasks (see Proxmox tasks, backups and alerts). A cluster with a node that needs a restart shows a badge such as "1 to restart" in its Platform column; it opens Restart a host for that cluster. See Restarting and patching hypervisor hosts.

The cluster page#

Open a cluster to see:

• Customer, Nodes online, Guests running, Last report and who adopted it and when.
• The reasons for its current health, for example "Node hwa-pve01 is offline" or "Storage usb-backup on hwa-pve01 is 91.3% full".
• Nodes: each node's status, its device in Tenvara, address, CPU, memory, guests, cores and uptime.
• Guests: every virtual machine and container, filterable by status, type, node and last backup.
• Recent tasks, with a link to All tasks for the cluster.
• Storage: shared storage once, local storage per node, with type and space used.
• Backup jobs: the jobs scheduled in Proxmox (Datacenter > Backup), their schedule, target storage and next run.

The header has Refresh (ask for a fresh report now) and the More actions menu. Storage says what each store holds (disks, containers, ISO images, container templates, backups) and how much is free.

Renaming and pausing#

• Rename gives a cluster a name of your own in Tenvara. Clear it to go back to the name it reports, which stays visible as Reported name.
• Pause stops reading the cluster, raising alerts and running actions until you Resume. Open alerts are cleared, history is kept, and a Paused badge shows on the list and the page. Use it while a cluster is being rebuilt or moved.

Removing a cluster#

Removing (confirm with Remove) deletes the cluster's nodes, guests, tasks, backup history and charts from Tenvara and clears its alerts. Nothing changes on Proxmox. If one of its nodes still runs the agent, the cluster comes back as detected with its next report. Uninstall the agent from the nodes first if you want it gone for good.

Host updates#

Hypervisor hosts are patched like any other device, through the built-in Hypervisor hosts group and patch policy, which install updates in the patch window but never restart the host. The node then shows Restart needed until a person restarts it with Restart this host. Restarting and patching hypervisor hosts covers both. Update hosts, below, is the other way: a run you start yourself that updates and, if you choose, restarts the nodes of one cluster in turn.

The cluster page's Host updates section shows the updates waiting on each node, from its package lists, and whether a node has a newer kernel installed than the one running. Update hosts updates the nodes one at a time through each node's own agent, so a cluster is patched without signing in to every node.

1. On the cluster page (or a standalone node's), press Update hosts.
2. Under Nodes, in this order, tick the nodes to update and put them in the order you want. Each shows how many updates it has and whether a reboot is pending.
3. Under Running guests, choose what happens to the guests on each node:
   • Move them off live: each node's running guests move live to the online node with the most free memory that can take them, and with Move the guests back afterwards they come back once the node is done.
   • Leave them where they are: nothing moves. The node installs its updates with its guests running. Only when it reboots are its running guests shut down cleanly first, one at a time in reverse start order, and once it is back exactly those guests start again in their start order.
4. Choose Reboot: When needed (only when a newer kernel was installed), Always or Never. With Never, no guest is touched and a node with a new kernel is left with a reboot pending.
5. Set Each guest has this long to shut down (a guest's own shutdown timeout in Proxmox comes first), and tick Force a guest off if it has not shut down in time if the run should not pause on a guest that hangs.
6. Read What the run will do, then press Start updating.

What the run will do checks for room before anything starts. For each node it lists every running guest with what will happen to it (Moves live to a node, HA moves it, Shut down and started again or Left running), whether the node is expected to reboot, and the expected downtime for the guests that are shut down (about, and at most). A guest that cannot fit on any other node is listed under Room check and is shut down for the reboot and started again instead, so the run never pauses half way on a guest that cannot move.

The run follows the cluster's high availability set-up. A cluster with no HA resources says "No HA resources on this cluster" and no node goes into maintenance mode. With HA resources, HA guests follow HA (maintenance mode moves them away and back) and your choice applies to the rest; with Leave them where they are, maintenance mode is used only for a reboot.

While it runs, the section shows each node's progress and what happened to each guest (moved, moved back, moved by HA, shut down and started again, forced off, left running). A failed step pauses the run: Try again repeats it, Cancel stops the run and leaves what has run (moved guests stay where they are, guests shut down for a reboot stay off until you start them, and a node in maintenance mode stays in it until you take it out on its page).

If a package fails to install part way, the node's agent repairs the half-installed packages once and tries the upgrade again before the run pauses. A node still left with packages half-installed is never rebooted: the run pauses with the reason, and the node page shows Upgrade interrupted: packages half-installed with Repair packages. A paused run is named wherever it blocks something (the node page, the update dialog and the restart dialog) with Resume, Cancel the run and a link to the run. Every step is a task of the cluster with its log, and the audit log records each node when it is done, with how many guests moved and how many were shut down and started again.

A run does not start while another is open on the cluster, when the cluster has lost quorum, or when a chosen node is offline or has no connected agent. A node that reboots rejoins its cluster as soon as it is back in touch with the other nodes.

Tip: Mark guests that should never move, such as one with disks on local storage or a passed-through device, as Never move. See Managing virtual machines and containers.

The node page#

Open a node from the cluster page to see:

• Host, its Device in Tenvara, Up for and Last report, with Maintenance mode and Restart this host in the header, and a Restart needed callout when the node is waiting for one.
• Performance: hourly CPU, memory and network charts over 24h, 7d, 30d or 90d. History is kept for 400 days.
• Guests on this node.
• Now: CPU in use out of its cores, memory, swap, root disk, load average, CPU model and sockets, and kernel.
• Package updates waiting on the node, Storage and its Recent tasks.

When a node stops reporting, the page says so ("It has stopped reporting. The figures below are the last ones it sent.") and keeps showing its last figures. On the lists and the overview the host shows as not reporting, and it is left out of the totals for guests and capacity until it reports again.

How nodes, guests and devices link up#

• A node is linked to the device whose agent reports as that node.
• A guest that runs the Tenvara agent itself is linked to its own device, matched by its SMBIOS UUID, then a MAC address, then its hostname. The guest page shows it as Own device, and the guest's device page shows which host it runs on.

Standalone nodes and clusters#

A node that was never clustered shows as a Standalone Proxmox VE node. If it later joins a cluster, its standalone record is merged into the cluster. A node removed from the cluster in Proxmox (pvecm delnode) is removed from Tenvara with its metrics and alerts.

Consoles#

Every running guest has a Console on its page, opened in a tab of its own: the screen for a virtual machine, a terminal for a container. See Managing virtual machines and containers. Proxmox's own web interface still works alongside Tenvara for the host's shell.

Related#

• Restarting and patching hypervisor hosts
• Proxmox tasks, backups and alerts
• Hyper-V hosts and virtual machines