By the end of this article you have three HVM hosts installed from HPE’s own medium, a VME Manager appliance running and licensed, a cluster built with the management and compute interfaces the right way round, a clustered datastore on a multipathed array LUN, a heartbeat target set, a virtual machine running on a tagged VLAN, and a validated end-to-end test that proves the thing actually moves workloads when a host goes away.
What this is derived from. Every step, field name, value and command below is taken from HPE’s current documentation set, read on 13 August 2026: the Deployment Guide and the VM Essentials 9.0.0 user manual. I have not built this cluster on hardware. Where the documentation states something I state it; where I have sequenced or gated a step from how the components behave rather than from an explicit instruction, I mark it as derived in the step itself. The validation commands are the vendor’s own, taken from its documented validation procedure. That framing is the whole of the caveat and I am not going to repeat it at every step.
The worked estate. Bramhall Instruments, a manufacturer, retiring three ageing ESXi hosts at its Manchester site. Three HPE ProLiant DL360 Gen11 servers, each with four network ports across two cards, connected to two switches. An HPE Alletra MP B10000 already in the rack, presenting one LUN over iSCSI on two separate subnets. Management on tagged VLAN 110, virtual machine traffic on VLANs 220 to 224, storage untagged on two non-routed subnets at a nine-thousand-byte MTU. External block storage rather than converged, because the array exists and is paid for.
Prerequisites. A VM Essentials entitlement, or the willingness to run on the thirty-day evaluation the manager applies automatically. Out-of-band management access to all three servers so you can mount virtual media. Hosts meeting the Deployment Guide’s requirements: 64-bit x86 with Intel VT or AMD-V enabled, a minimum of 8 GB RAM for the HVM runtime, a minimum of 100 GB of operating system storage, and ten-gigabit networking for anything you intend to call production. Forward and reverse DNS records for every host and for the manager appliance. A working NTP source. An administrative account on the array able to create and map a LUN. And a maintenance window, because step 3 destroys what is on those servers now.
One documentation conflict to resolve before you size anything: the Deployment Guide asks for 8 GB of RAM and 100 GB of operating system storage, while the user manual’s hardware requirements page still says 4 GB and 15 GB. Both are current pages in the 9.0 documentation set. Build to the higher numbers.
The naming convention, decided before anything is created
Everything this build creates gets a name, and the names outlive the build. Agree them now, because renaming a cluster after datastores are attached to it is the kind of task that is technically possible and practically nobody does.
| Artefact | Convention | Bramhall value |
|---|---|---|
| HVM host | hvm- then site code, then a two-digit ordinal. Site first because the second site is always coming. | hvm-mcr-01, hvm-mcr-02, hvm-mcr-03 |
| Management and compute bond | One bond, OS-default name. The VLAN sub-interface name is generated by the operating system from the bond name and the VLAN ID; never type it from memory. | bond0, with the management sub-interface read from the host rather than assumed |
| Storage interfaces | Left unbonded and left with their predictable OS names. Read them, do not guess them. | Read with ip address show on each host |
| VME Manager appliance | vme-mgr- then site code, then ordinal, with matching forward and reverse DNS | vme-mgr-mcr-01 |
| Group | grp- then the business unit that owns the resources | grp-manufacturing |
| Cloud | pc- then site code. Private Cloud type, one per site. | pc-mcr |
| Cluster | hvm- then site, then purpose, then ordinal | hvm-mcr-prod-01 |
| iSCSI target objects | Not named in the product. The Add dialog on the iSCSI sub-tab takes a portal address and a port and nothing else, so the two path objects are distinguished by their portal addresses. Record the path-to-portal mapping in your runbook, because the product will not hold it for you. | Path A is the portal on the first storage subnet, path B the portal on the second. Both recorded against the cluster in the build record. |
| Master tenant account name | The organisation’s name as it should appear in the manager. A label, not a boundary, because VM Essentials does not support multi-tenancy. | Bramhall Instruments |
| First administrative user | A named individual, never a shared account, with the same identifier the person uses elsewhere in your estate. | A named engineer, recorded in the build record against the date |
| Router object, the OVS bridge domain | rtr- then the bond it sits on, then the site. Only applies to routers you create; the cluster build produces one for the compute bond and names it itself. | rtr-bond0-mcr, if you ever add a second bridge |
| Virtual machines | Existing estate convention. This sheet creates two test workloads, and test workloads get names that say so, because an unnamed test VM survives for four years. | test-mcr-auto-01 and test-mcr-pinned-01 |
| Datastore | ds- then cluster name, then type, then ordinal | ds-hvm-mcr-prod-01-cds-01 |
| Network port group, created by hand | net-vl then VLAN ID, then purpose. VLAN first because that is what you search on at two in the morning. | net-vl220-app |
| Network port group, created by the cluster wizard | Not yours to name. Every VLAN you enter in the COMPUTE VLANS field at cluster creation produces a port group under a name the product generates. Read those names off the cluster’s Networks sub-tab and record them; do not assume they follow your convention. | Read after step 10, not predicted before it |
The files that move, and where each one lands
Five artefacts cross between machines in this build. Three of them are versioned by HPE, which means their filenames carry a version stamp you must read rather than retype. Printing a literal filename in a runbook is how a build fails on a step nobody suspects.
| Artefact | Comes from | Travels to | What it is for on arrival |
|---|---|---|---|
The HVM OS install ISO, named HVM_Install_ followed by a version stamp | My HPE Software Center | Mounted as virtual media on each server’s out-of-band management console, all three | Boots the installer that lays down the HVM operating system. Used once per host, then unmounted before the first reboot. |
The manager appliance ISO, named HPE_VM_Essentials_SW_image_ followed by a version stamp | My HPE Software Center | Copied by SCP to the home directory of hvm-mcr-01 only | Carries the manager appliance disk image. Mounted loopback on the host and read in place. |
The appliance disk image, a .qcow2 file inside that ISO | Inside the mounted ISO, at whatever mount point you chose | Nowhere. It is consumed in place. | The manager virtual machine’s disk. Supplied to the installer as a path, not copied out and not renamed. |
| The licence key | Your HPE entitlement | Typed into the manager’s initial setup wizard, or applied afterwards from the licence settings | Converts the automatic thirty-day evaluation into an entitled installation. |
| The first guest image, a cloud-init Linux qcow2 | Your own image build, or a distribution’s cloud image | Uploaded into the manager’s image library, which stores it on whichever datastore is marked as the image target | The source for the first virtual machine in step 14. This is the reason IMAGE TARGET is ticked in step 12, and the reason it is ticked on exactly one datastore. |
Step 1. Fix the four commitments in writing
Four answers determine what you can change later, and they are all given during this build. [VME 4] argues why each one is expensive to reverse; the column below says only what each choice does to the steps that follow. Write them down and have somebody sign them off before you mount anything.
| Decision | Bramhall value | Why this one |
|---|---|---|
| Manager size | Large, 4 vCPU and 32 GB | Small is documented for one cluster and Medium for three. Large is documented for ten. The Manchester build is one cluster today and the second site is funded, and resizing a manager that holds live inventory is worse than over-provisioning by 16 GB. |
| Cluster type | Non-converged, external block storage | Determines what steps 11 and 12 do. This build presents a LUN rather than giving local disks to Ceph. |
| Datastore type | Clustered datastore, formerly and still widely called GFS2, on an iSCSI LUN | Sets the host floor at three on this build, and makes step 11’s two-path registration a hard prerequisite of step 12. The 9.0 generation does allow a two-node variant backed by an external quorum witness, which is a different design with a different failure conversation and is not what this sheet walks. |
| Network topology | Converged management and compute on one bond, storage separate and multipathed | Determines what you build in step 3 and what you enter in step 10. Four ports across two cards is the shape this sheet walks; six or more would decouple management from compute and change step 3. |
Step 2. Prove the prerequisites before anything becomes destructive
Run these from a workstation on the management network. Every one of them is cheap now and expensive after step 3.
# Forward and reverse DNS for the manager appliance and every host.
# Reverse matters: the manager installer asks for a resolvable appliance URL
# and the hosts have to resolve it back.
for n in vme-mgr-mcr-01 hvm-mcr-01 hvm-mcr-02 hvm-mcr-03; do
ip=$(dig +short "$n.bramhall.internal" A | tail -n1)
if [ -z "$ip" ]; then echo "$n: NO A RECORD"; continue; fi
ptr=$(dig +short -x "$ip")
echo "$n -> $ip -> ${ptr:-NO PTR}"
done
# Expected: an address for every name, and a PTR that resolves back to the same
# name. Note that dig returns the PTR fully qualified and with a trailing dot,
# so compare it to the full name, not to the short one.
# Failure mode: NO PTR is the one that hurts. The manager installs cleanly
# without it and the hosts then fail to register against the appliance URL,
# which surfaces as a cluster that provisions and never completes post-provision.Confirm the ports the platform needs are open between the segments involved. This is the subset that matters for a non-converged build; the Ceph daemon ports are irrelevant here.
| Setting | Value | Why |
|---|---|---|
| Host to manager | 443/TCP | The host agent’s channel to the manager. Blocked, and hosts appear in the inventory and never report health. |
| Manager to host | 7443/TCP | Hypervisor console. Blocked, and everything works except the console, which nobody tests until an outage. |
| Manager to host, and host to host | 22/TCP | Cluster provisioning and inter-host operations. This is how the manager builds the cluster at all. |
| Host to host | 5404 to 5406/UDP, 2224/TCP, 21064/TCP | Membership, cluster service and distributed lock manager. These carry the machinery that keeps a clustered datastore from being corrupted by two hosts writing at once. The port table is written for the platform generally and 2224 belongs to the Pacemaker-era cluster service, so on a layout 1.3 cluster it may go unused. Open it with the rest rather than reasoning about which layout you will end up on. |
| Host to host | 9929/TCP and 9929/UDP, 3121/TCP | Documented in the platform’s port table for inter-host communication. Open them with the rest rather than discovering which one you needed. |
| Manager to guest | 22/TCP, 5985/TCP, 5986/TCP | Post-provision access into deployed Linux and Windows guests. Not needed to build a cluster, needed the first time you provision anything with automation. |
Step 3. Install the HVM OS on each host
Download the HVM OS install ISO from My HPE Software Center. Its filename carries a version stamp, so read the name off the download rather than typing one from a runbook. Mount it as virtual media through each server’s out-of-band management console and set the boot order to take it. At the boot menu, select the HVM Install entry for Ubuntu 24.04. The installer is a text interface driven with Tab and the arrow keys.
On the network configuration screen, build the bond before you configure any address. Create the bond, add the two management and compute ports to it, and set the mode. Then, because Bramhall’s switch ports are trunks with the management VLAN tagged, add a VLAN tag to the bond and put the address on the VLAN interface rather than on the bond itself.
| Setting | Value | Why |
|---|---|---|
| Bond membership | The two ports that reach both switches, one from each network card | One port per card and per switch is what makes the bond survive losing either. Two ports on one card is a bond that survives a cable and not a card. |
| Bond mode | active-backup | Requires nothing of the switch pair. LACP is supported and needs the switches joined into one logical switch first, which turns a host task into a network project. |
| Create a bond even with one interface | Yes | The documentation recommends it explicitly. Adding a second interface to an existing bond later is trivial; converting a bare interface into a bond on a host running workloads is not. |
| VLAN tag on the bond | The management VLAN, 110 here | The switch ports are trunks and are not tagging for you. If your management port carries a native VLAN instead, skip the tag and address the bond directly. |
| IPv4 method on the VLAN interface | Manual, with address, mask, gateway and DNS | A static management address is required for the cluster build to succeed. DHCP here fails later, not now. |
| Storage disk layout | Use an entire disk, set up as an LVM group, then edit the logical volume to the maximum size | The installer leaves the logical volume short of the disk by default. Growing it later is possible and is one more thing to remember on three hosts. |
Verification gate, immediately before the destructive step. The installer picks a disk for you, the documentation tells you to check that it picked the right one, and it does not always pick the one you meant. The selection itself is read off the Guided storage configuration screen’s Used Devices list, not from a command. What a command gives you is the inventory to check that selection against. From the installer’s shell, or from the server’s out-of-band storage inventory before you boot:
lsblk -o NAME,SIZE,TYPE,MODEL,SERIAL
# Expected: a device list in which you can identify, by model and serial, the
# internal boot device you intend to install onto. Write that serial down and
# confirm the Used Devices list on the Guided storage configuration screen names
# the same device before you accept the Confirm Destructive Action prompt.
# Failure mode: accepting the default on a host whose array LUNs are already
# zoned to it formats storage that belongs to something else. That prompt is the
# last reversible moment in the whole build.Set proxy details if your environment needs them, complete the profile screen, and let the installation run. Reboot, and make sure the ISO is unmounted or removed so the server does not boot the installer again. Repeat for all three hosts.
Step 4. Patch each host and fix its clock
Log in over SSH and bring the host up to date. Everything comes from HPE’s own curated repositories, which are already configured on the image, and only packages from those repositories are supported. This is the same discipline that governs day-2 patching, and [VME 5] takes it up properly.
sudo apt update && sudo apt upgrade -y
# Expected: packages resolve from the HPE-hosted repositories configured on the image.
# Failure mode: an apt source you added yourself will resolve happily and put you
# outside support. If you see a third-party origin in the output, stop and remove it.
timedatectl status
# Expected: "System clock synchronized: yes" and an NTP service active.
# Failure mode: unsynchronised clocks across hosts break the membership and locking
# layer in ways that present as random storage faults rather than as a time problem.Step 5. Finish the network with the hpe-vm console
The installer built the bond. The platform’s own host utility is what tunes it and what adds the storage interfaces. Open it on each host:
sudo hpe-vmSelect Configure Network. Under device type, select bond, select your bond, and edit the parameters. Then, with ethernet selected under device type, use Add to create each storage interface, entering its name under Device ID and editing the addresses row. Interface names come from the host, so read them first with ip a and use what you read.
| Setting | Value | Why |
|---|---|---|
| Bond mii-monitor-interval | 100 | Without a monitor interval an active-backup bond does not actually fail over. The bond looks correct and does nothing when a link drops. |
| Bond up delay | 200 | Stops the bond promoting a link that has come up electrically before the switch port is forwarding. |
| Bond down delay | 200 | Stops a momentary flap being treated as a failure. |
| Storage interfaces | Added individually as ethernet devices, not bonded | The storage layer handles path redundancy through multipathing. Bonding storage interfaces fights the multipath layer for the same job. |
| Storage addressing | A static address on each, on two different subnets | Two paths on one subnet is one path with extra steps, and the documentation warns that a shared subnet causes storage access problems. |
| Storage gateway | None | iSCSI traffic here is not routed. A gateway on a storage interface is an invitation for storage traffic to leave the storage network. |
| Storage MTU | 9000 | Jumbo frames end to end. Switch ports and array ports must match, and mismatch is the single most common cause of a datastore that mounts and then stalls under load. |
| Storage VLAN tagging | None, access ports | Storage switch ports are normally untagged access ports in this design. Tag only if your switches present a trunk. |
Save from the main network menu and confirm. Repeat on all three hosts.
Step 6. Prove the network before you build anything on it
Derived step. The documentation does not instruct you to test the fabric at this point, and an MTU mismatch is the classic storage fault that is not a storage fault: standard-sized traffic works, the datastore mounts, and the failure arrives later under write load. Run these on every host, against both array portals.
# Jumbo frames end to end. 8972 bytes of payload plus 28 bytes of headers is 9000.
# Replace the addresses with your array's two iSCSI portals.
ping -M do -s 8972 -c 3 10.60.10.50
ping -M do -s 8972 -c 3 10.60.11.50
# Expected: three replies from each portal.
# Failure mode: "Message too long" means the local interface MTU is wrong.
# Silence with no error means something in the path between here and the array
# has a smaller MTU and is dropping the frame without telling anyone. That is the
# dangerous case, because standard-sized traffic still works and the datastore
# will mount, run, and then fall over under real write load.
# Bond failover actually works. Read the bond name rather than assuming bond0;
# you named it in the installer and it need not be the default.
shopt -s nullglob
bonds=(/proc/net/bonding/*)
if [ "${#bonds[@]}" -eq 0 ]; then
echo "NO BOND EXISTS on this host"
else
for b in "${bonds[@]}"; do echo "== $b"; cat "$b"; done
fi
# Expected: MII Status "up" on the bond and on both slave interfaces, and a
# non-zero MII Polling Interval.
# Failure mode: a polling interval of 0 means step 5's parameter did not save,
# and an active-backup bond with no monitor interval looks healthy and does not
# fail over. The NO BOND EXISTS line means the installer's bond was never built.Step 7. Install the VME Manager on the first host
The manager appliance ships as a qcow2 image inside an ISO. Copy the ISO to hvm-mcr-01 with any SCP client, then mount it loopback and let the installer read the image in place. The ISO’s filename carries a version stamp, so resolve it rather than typing it.
# Resolve the real filename instead of retyping a version, and stop rather than
# guess when there is more than one candidate. Do not reduce this to the first
# match: glob and ls both sort lexically, so v8 wins over v9 and you mount the
# wrong appliance without being told.
shopt -s nullglob
isos=(~/HPE_VM_Essentials_SW_image_*.iso)
for i in "${isos[@]}"; do echo "$i"; done
if [ "${#isos[@]}" -ne 1 ]; then
echo "Resolve to exactly one ISO before continuing."; return 2>/dev/null || exit 1
fi
ISO="${isos[0]}"; echo "Using $ISO"
# Expected: one path listed, then the Using line.
# Failure mode: more than one path means an older download is still in place.
# Delete it rather than picking. Zero paths means the SCP landed elsewhere.
# In both cases the block stops here rather than mounting an empty argument.
sudo mkdir -p /mnt/vme
sudo mount -o loop "$ISO" /mnt/vme
sudo find /mnt/vme -name '*.qcow2' -print
# Expected: exactly one path, which is the value the manager installer wants for
# its image field. Use find rather than a top-level glob: the documentation says
# to browse to the file, which means it is not guaranteed to sit at the root of
# the mount. The filename is generated by HPE's packaging, so read it here.Verification gate, before the install. The manager’s appliance URL has to resolve from the hosts, forward and reverse, or the hosts and the manager will not talk once the cluster is built. You checked this from a workstation in step 2. Check it again from the host, because the host is what has to resolve it.
getent hosts vme-mgr-mcr-01.bramhall.internal
dig +short -x 10.60.1.10
# Expected: the address you are about to assign the appliance, and the name back again.
# Failure mode: a missing record here produces a manager that installs cleanly and a
# cluster that never completes its post-provision phase, with nothing in the UI that
# points at DNS as the cause.Open sudo hpe-vm again and select Install VME Manager. Fill the fields as follows.
| Setting | Value | Why |
|---|---|---|
| Appliance IP address | The static address you reserved on the management VLAN | This becomes the manager’s address permanently. It is the one thing here that is genuinely painful to change afterwards. |
| DNS servers | A comma-delimited list | The installer does not currently validate the delimiter. A wrong separator is accepted, stored, and produces name resolution failures inside the appliance later. |
| Appliance URL | The fully qualified name, resolvable forward and in reverse from the hosts | Everything the hosts do with the manager goes through this name. An address here works until you move something. |
| Admin username | A named administrative account for the appliance’s own command line | This is the appliance’s shell account, not the first UI user. They are different accounts created in different places and people conflate them. |
| Image URI | The qcow2 path you resolved above | Read from the mounted ISO. Never a literal from a runbook. |
| Network interface | The same tagged management interface the host uses | The manager sits on the management network with the hosts. Leave the compute VLAN option unchecked unless you have a deliberate reason to put the manager somewhere else. |
| Appliance size | Large | The console offers three predefined sizes and this is where the choice is made. Fixed in step 1. If your build of the console does not present a size at this point, the appliance is deployed at a default shape and you resize the manager virtual machine afterwards, before it holds anything. |
The install takes a while and the progress bar is not a reliable narrator. Once the progress display reports that Morpheus services are starting, the appliance URL will begin answering, and any response at all, including one that says the appliance is still loading, means the web tier is up. The documentation is explicit that the loading graphic timing out is not a failure and that you should allow at least an hour before treating the install as irrecoverably stuck. That is a floor, not a deadline, so do not tear down a healthy install at fifty-nine minutes. Watch the real thing instead, over SSH to the appliance:
sudo morpheus-ctl tail morpheus-uiStep 8. Complete initial setup and apply the licence
Browse to the appliance URL. The first-run workflow asks for a master tenant account name, the first administrative user, an appliance name, a backups toggle and a licence.
| Setting | Value | Why |
|---|---|---|
| Master tenant account name | Bramhall Instruments | An identifier, not a boundary. VM Essentials does not support multi-tenancy and the wizard says so; treat this as a label and do not design delegation around it. |
| First user | A named system administrator, not a shared account | This user gets every permission. Audit trails that lead to admin lead nowhere. |
| Appliance name | vme-mgr-mcr-01 | Matches DNS and the naming convention, so the appliance identifies itself the same way everything else refers to it. |
| Enable backups | On | Turns on both manager database backups and the native virtual machine backup feature. They land on the manager’s local storage by default, which is not a backup, so pointing them at an external repository belongs on your list before this cluster carries anything real. |
| Licence | Your entitlement key | Leaving it blank starts a thirty-day evaluation automatically. That is fine for a pilot and a diary entry you will forget. |
Step 9. Create the group and the cloud that will hold the cluster
A cluster has to live in a Cloud, and a Cloud has to belong to a Group. Both are created in the Infrastructure area before the cluster exists.
Create a Group named grp-manufacturing. Groups are how roles are scoped to resources, so this is a permissions boundary rather than a folder, and it is worth naming after the thing that owns the workloads rather than after the site. Then create a Cloud of type Private Cloud named pc-mcr and assign it to that Group. Private Cloud is the type that houses HVM clusters; the VMware type is for connecting an existing vCenter, and anything beyond those two requires the Enterprise tier.
Step 10. Create the cluster, and get the tagging the right way round
This is the step the whole build turns on. Create a cluster of type HVM in the Clusters area, assign it to grp-manufacturing, and name it hvm-mcr-prod-01.
Verification gate, before you submit. Two interface names go into this wizard and swapping them produces a cluster that builds successfully and cannot pass virtual machine traffic. Read both from a host and write them down rather than working from memory.
ip -br address show
# Expected: the bond with no address of its own, and a VLAN sub-interface on that
# bond carrying the management address. The sub-interface name is generated by the
# OS from the bond name and the VLAN ID; take it from this output.
#
# The tagged sub-interface is the MANAGEMENT NET INTERFACE.
# The bare bond, with no tagging, is the COMPUTE NET INTERFACE.
#
# Failure mode: entering the bare bond as management and the tagged interface as
# compute produces a cluster that provisions green and then cannot carry any VLAN
# to any VM. The documentation calls selecting the untagged bond for compute
# crucial, and it is the only field in this wizard that fails silently.| Setting | Value | Why |
|---|---|---|
| Layout | Read the list the wizard offers before you decide | The Deployment Guide’s walkthrough says to leave the default, which it names as HVM 1.2 on Ubuntu 24.04. The user manual documents layout 1.3, introduced at 9.0.0, which replaces Pacemaker as the resource manager with the platform agent, and documents an upgrade from 1.2 whose second phase cannot be rolled back. Both pages are current, which is why this row tells you to look rather than to type. Where 1.3 is offered on a new build, take it, because the alternative is performing that one-way upgrade later on a cluster carrying workloads. Where only 1.2 is offered, take it and put the upgrade on your roadmap. |
| SSH hosts | Three entries, named hvm-mcr-01 to -03, each with its management address | The name is only a display label in the interface. Making it match the real hostname is free and stops the inventory diverging from DNS. |
| SSH username and password | A regular account with sudo on every host | Not root. The same account must exist on all three or the build stops on the host that lacks it. |
| MANAGEMENT NET INTERFACE | The tagged VLAN sub-interface read above | Inter-host cluster communication rides this. |
| COMPUTE NET INTERFACE | The bare, untagged bond | The virtual switch has to do the tagging itself for VM VLANs to flow. This is the failure named in the gate above. |
| COMPUTE VLANS | 220-224 | Takes a single value or a range. Each becomes a port group you can select when provisioning. |
| CPU TYPE | x86_64 | The only supported value today. |
| CPU MODEL | host-passthrough | The default, and correct when cluster nodes share a CPU family. Mixed generations are the one case for pinning a specific model, and doing so costs you instruction sets on the newer hosts. |
| DYNAMIC PLACEMENT | Enabled | Balances on memory utilization, with aggressiveness profiles and migration cooldowns at 9.0. Start on Moderate and leave the cooldowns alone until you have a month of behaviour to look at. |
| POWER POLICY | Balanced | The default. Performance mode drives CPUs harder and reduces network latency, and requires a host agent at 3.0.0 or above to take effect at all, so setting it on a fresh build before agents are current changes nothing and looks like it did. |
Submit, then open the cluster’s History tab and watch it. The manager installs KVM and Open vSwitch and prepares the hosts. The cluster is ready when the provision and post-provision activities both complete with a green check.
Step 11. Present the LUN and register both paths
On the array, create the LUN and map it to an iSCSI target, enable concurrent connections so every host can attach at once, and decide about CHAP. The platform supports CHAP on clustered datastores but has no interface for enabling it, so if you need it, configure it outside the wizard and plan for it to be undocumented in your own runbook unless you write it down.
Then register each path in the cluster’s storage configuration, on the iSCSI sub-tab, using Add and giving the portal address and the port. The dialog asks for nothing else, so the two objects are distinguished by their portal addresses and by nothing you can type. Register the portal on the first storage subnet, then the portal on the second, and write the path-to-portal mapping into your build record now, because the product will not hold it for you. Refresh the cluster afterwards.
Verification gate, and this one is the sharpest in the build. Multipathing cannot be established after the datastore is created. Both paths must be registered before step 12, or the only route to a multipathed datastore is destroying this one and starting again. Prove both paths are live on every host before you go on.
sudo multipath -ll
# Expected: one multipath map, with two paths under it, both in state "active ready".
# Run it on all three hosts and expect the same map identifier on each.
#
# Failure mode 1: two separate maps instead of one means the two portals are
# presenting different LUNs. Fix the array mapping, not the host.
# Failure mode 2: one map with one path means only one target registered.
# Go back and add the second before creating anything.
# Failure mode 3: nothing at all means the LUN is not mapped to this host's
# initiator. Check the array's host group.
sudo lsblk -o NAME,KNAME,SIZE,TYPE,HCTL,WWN
# Take the LUN's identity from the WWID in the multipath map header above, not
# from this listing: on the dm- device the datastore wizard actually consumes,
# lsblk's WWN column is frequently empty because it is populated from udev on the
# underlying SCSI paths. This listing is here to show which paths belong to which
# map. Match the multipath WWID against the volume identifier on the array.
#
# Device names of the /dev/sdX form are assigned in discovery order, differ
# between hosts and change across reboots. Never record one in a runbook and
# never select storage by one.Step 12. Create the clustered datastore
On the cluster’s Storage tab, Data Stores sub-tab, click Add.
| Setting | Value | Why |
|---|---|---|
| NAME | ds-hvm-mcr-prod-01-cds-01 | Cluster, then type, then ordinal. The second datastore on this cluster is coming and it should sort next to this one. |
| TYPE | GFS2 Pool, the clustered filesystem type | HPE has renamed these to Clustered Datastores in its release material while the interface and most documentation still say GFS2. Same object. |
| BLOCK DEVICE | The multipath device you identified by its WWID in step 11 | Selected by identity, never by a device name you memorised. This is the field where a wrong answer formats something else. |
| ALLOW REFORMATTING | Ticked, once and only once you are certain | Writes a filesystem if none exists, and destroys everything on the disk. On a genuinely new LUN this is what you want. On a LUN somebody re-presented from another system, it is an incident. |
| IMAGE TARGET | Ticked | Makes this datastore the store for virtual images. A cluster should have exactly one image target, so tick it here and not on the second datastore. |
Verification gate, immediately before saving. Reformatting is irreversible and there is no undo. Confirm the device selected in the block device field is the multipath map whose WWID you matched against the array volume in step 11, and not a device that arrived from somewhere else, and confirm the array shows no other host group attached to that volume. Then save.
When it completes, set a heartbeat target. Edit the datastore on the Data Stores sub-tab and mark the box labelled HEARTBEAT TARGET. Hosts write to a heartbeat file that the others watch, and a file that stops being updated is what triggers failover. It is not on by default, and a cluster without one has a thinner availability story than its architecture diagram implies. Where you have two datastores, mark both.
Step 13. Create the network the first workload will use
Verification gate, and this one saves you from creating something twice. Step 10 entered five VLANs in the COMPUTE VLANS field, and the documentation states that a port group is created for each. So before you create anything here, look at what already exists on the cluster’s Network tab. The Routers sub-tab should show the Open vSwitch bridge and libvirt network the cluster build produced for the compute bond, and the Networks sub-tab should show five port groups the product has named for itself. Record those generated names in your build record. On the path this sheet walks, both sub-tabs are populated when you arrive, and your task here is to edit the existing port group for VLAN 220 so that it carries addressing, not to add a second object on the same VLAN. Two port groups on one VLAN is a configuration that saves cleanly and produces a network list nobody can reason about six months later.
Create objects here only in the two cases where nothing suitable exists: the Routers sub-tab is empty, or you are adding a VLAN outside the range you entered at cluster creation. In those cases build them in order, because the order is not optional. A router object comes first, added as an OVS Bridge Domain, which from the KVM side is an Open vSwitch bridge and a libvirt network with no port groups on it. A network object comes second, added as an OVS Port Group, which is a libvirt port group attached to that bridge and carrying a VLAN tag. Creating the port group first is not possible, because its ROUTER field has nothing to point at.
| Setting | Value | Why |
|---|---|---|
| GROUP | grp-manufacturing, or Shared | Shared makes the object available to every Group. Naming the owning Group instead is the more defensible default, because a network is a resource somebody is accountable for. |
| NAME | net-vl220-app on an object you create; on the wizard-generated port group, whatever the field will accept | The convention applies to objects you create. For the port groups step 10 generated, apply the convention if the NAME field is editable and record the generated name if it is not. Either way the build record carries both names, because the generated one is what appears in logs. |
| CIDR | The subnet carried on VLAN 220 | The manager uses this to allocate and to report, so a wrong value here produces addressing that looks right in the interface and does not work. |
| GATEWAY | The gateway on that subnet | Handed to guests. This is a compute network and it does route, unlike the storage interfaces in step 5. |
| DNS PRIMARY and DNS SECONDARY | Your internal resolvers | Both fields exist. Filling only the first is a decision, so make it deliberately rather than by leaving a box alone. |
| ROUTER | The existing OVS bridge object for the compute bond | If this list is empty, go back and add the OVS Bridge Domain first. A new bridge needs a free physical interface or bond, and the manager does not discover interfaces the host has not already surfaced. |
| VLAN ID | 220 | The tag the port group applies. This is the value that has to match what the switch trunk carries. |
| NETWORK POOL | Empty on this build | Only relevant once you have configured an IP pool and want the platform allocating addresses. Leaving it empty means guests get addressing from DHCP or from cloud-init, which is the right answer for a first cluster. |
If you later add a physical interface to a host, the manager will not see it. The interface has to exist in the host’s netplan configuration and be applied there, then be assembled through the host utility, and only then does refreshing the cluster surface it. That sequence is the platform’s answer to who owns physical connectivity, and it is worth putting in your runbook before somebody spends a morning on it.
Step 14. Provision the first virtual machine
First get an image in. Upload a cloud-init Linux qcow2 into the manager’s image library; it lands on whichever datastore is marked as the image target, which is why exactly one datastore was marked in step 12. A small cloud image is the right first workload because it proves image handling, storage and networking in one pass and boots fast enough to iterate on.
Provision two instances into pc-mcr, on hvm-mcr-prod-01, both with their disks on ds-hvm-mcr-prod-01-cds-01 and their interfaces on the port group carrying VLAN 220. Name them test-mcr-auto-01 and test-mcr-pinned-01. Two, not one, because step 15 needs to observe both halves of the failover contract and a single workload can only demonstrate one of them.
Then set placement on each explicitly: on the virtual machine’s detail page, open ACTIONS and select Manage Placement. The three strategies are Auto, Failover and Pinned, and only the first two are eligible for an automatic move when a host is lost. Set test-mcr-auto-01 to Auto and test-mcr-pinned-01 to Pinned, on the same host. Step 15 takes that host away and expects the two to behave differently.
Step 15. The end-to-end test
Four checks in sequence. The first proves the cluster’s plumbing, the second proves live migration, the third proves graceful evacuation, and the fourth, which is derived rather than documented and needs a change window, proves the failover contract itself.
Check one, the cluster’s own health. This is HPE’s documented validation set for a layout 1.3 cluster. Run it on each host.
# Everything here except the mount check needs root. Running the rest
# unprivileged is worse than failing: virsh without an explicit connection URI
# falls back to the per-user session daemon, which returns an empty pool list
# and no error at all.
sudo morpheus-node-ctl status
sudo systemctl status morpheus-morphd
sudo morpheus-node-ctl version
sudo corosync-quorumtool
sudo dlm_tool ls
sudo dlm_tool ls -n
mount | grep gfs2
sudo virsh -c qemu:///system pool-list --all
# Expected: services up; quorum reporting all three nodes; one lockspace per
# clustered filesystem with all three node IDs in its members line and a change
# line showing no failed nodes; the filesystem mounted; the storage pool listed
# and active.
# Failure mode: a pool active on two hosts and not the third is the classic
# partial build. That host cannot receive a migration and will not tell you so
# until you try one.
# The pool name is generated by the platform, so read it from the list above and
# set it explicitly. Do not take the first line unseen: a host may also carry a
# local or default pool, and pool-info on the wrong object tells you nothing.
read -rp 'Clustered pool name, copied from the list above: ' POOL
sudo virsh -c qemu:///system pool-info "$POOL"Check two, live migration under load. Start a continuous ping to test-mcr-auto-01 from a workstation, then move it to another host through the cluster’s migrate action.
ping -i 0.2 10.60.220.31
# Expected: the migration completes and the ping shows either no loss or a single
# dropped packet at cutover.
# Failure mode: sustained loss, or a migration that fails outright, almost always
# means the destination host cannot see the datastore. Go back to check one and
# look at which host is missing the pool.Before check three, know which host carries the manager. The manager appliance was installed on hvm-mcr-01 in step 7 and it is a virtual machine like any other, running on that host’s local storage. It cannot be live migrated off, so that host will not enter maintenance mode, and pulling its power takes your management plane down in the middle of the test. Run checks three and four against hvm-mcr-02 or hvm-mcr-03. If either test virtual machine is on hvm-mcr-01, move it first, and re-pin test-mcr-pinned-01 to whichever host you are about to test. Making the manager’s own host survivable is a separate design conversation and [VME 4] has it.
Check three, maintenance mode and evacuation. On the detail page of the host now running the test virtual machine, open ACTIONS and select Enter Maintenance. The interface shows you which workloads it intends to move before it acts. The expected result is that test-mcr-auto-01 migrates away and that the host does not enter maintenance while test-mcr-pinned-01 is still running on it, which is the documented behaviour rather than a fault. Stop the pinned workload, let the host complete its entry into maintenance, then select Leave Maintenance and confirm the host returns to service. Start the pinned workload again before check four.
Check four, if your change window permits it. The harder test is worth doing exactly once, on a host that is not hvm-mcr-01. Pull its power and confirm that test-mcr-auto-01 comes up elsewhere within a short time, that test-mcr-pinned-01 stays down, and that when the host returns the pinned workload is present and in a stopped state rather than lost. That is the failover contract in one experiment, and doing it on day one is worth more than believing it for two years.
The consolidated per-host verification block
Be honest about what this is. Host network configuration on this platform happens through hpe-vm, an interactive console, and HPE documents no command line equivalent for building a bond or adding a storage interface, so there is no script that replaces step 5. What is scriptable is every check around it, and running those as one block on each host beats three sessions of typing and produces a record you can attach to the build.
The cluster build has no scripted equivalent here either, and that is a deliberate omission rather than an oversight. The platform publishes a REST API reference and a command line client, and the 9.0 generation added support for multiple authentication tokens per user, so an automated build is achievable. I have not run one, and inventing endpoint paths for an article would be a worse failure than saying so. Build the cluster through the manager, and if you are going to automate it, take the endpoint shapes from the API reference in your own installation rather than from anybody’s blog.
#!/usr/bin/env bash
# Run on each HVM host after the OS install and after hpe-vm has been used to
# set the bond parameters and the storage interfaces, and before creating the
# cluster. This verifies; it does not configure. Read every derived value.
set -euo pipefail
echo "== time =="
timedatectl status | grep -E 'synchronized|NTP service' || echo "NO NTP SYNC
echo "== interfaces, for the cluster wizard =="
ip -br address show
echo "The tagged sub-interface above is MANAGEMENT NET INTERFACE."
echo "The bare bond above is COMPUTE NET INTERFACE."
echo "== bond health =="
shopt -s nullglob
bonds=(/proc/net/bonding/*)
[ "${#bonds[@]}" -gt 0 ] || echo "NO BOND EXISTS on this host"
for b in "${bonds[@]}"; do
echo "== $b"
grep -E 'MII Status|MII Polling Interval|Slave Interface' "$b" || true
done
echo "== jumbo frames to both array portals =="
PORTALS=(10.60.10.50 10.60.11.50) # your array's two iSCSI portals
for portal in "${PORTALS[@]}"; do
ping -M do -s 8972 -c 3 "$portal" || echo "FAILED at MTU 9000 to $portal"
done
echo "== storage paths =="
sudo multipath -ll || echo "no multipath maps yet; expected before targets are registered"
sudo lsblk -o NAME,SIZE,TYPE,WWNCompletion checklist
Three hosts installed from the HPE medium, patched from the HPE repositories, and clock-synchronised. A bond on each with a monitor interval that is not zero, and two unbonded storage interfaces on separate non-routed subnets at a nine-thousand-byte MTU, proven with a do-not-fragment ping to both array portals. A manager appliance at Large, resolvable forward and reverse from every host, licensed or knowingly on evaluation, with backups enabled and an external repository on your list. A Group and a Private Cloud. A cluster on the highest available layout, with the tagged interface as management and the untagged bond as compute, green on both provision phases. Both iSCSI paths registered before the datastore existed, with one multipath map and two active paths on all three hosts. One clustered datastore, selected by multipath WWID rather than by device name, marked as image target, with a heartbeat target set. One port group on the VLAN your workloads use. Two named test workloads, one on Auto placement and one Pinned. The Auto workload live migrated with at most one dropped packet. One host, not the one carrying the manager, that entered and left maintenance mode with its evacuable workloads moved and its pinned workload correctly blocking entry until it was stopped.
That is a cluster. What it is not yet is an operated platform: nothing here says how you patch it next month, what the manager upgrade does to your change window, or where the native backup stops being a backup. [VME 4] gave you the reasoning behind the four decisions this build committed to, and [VME 5] takes up what owning them costs, in hours and in discipline, against the vSphere routine you are leaving behind.
VME
‹ Previous: [VME 4] How VME Is Built: Clusters, Storage, and Networks
