Case Study: Virtualisation Platform Evolution & Tape‑Based Backup Architecture

Over more than a decade of continuous operation, Kent ITS engineered and maintained a highly resilient enterprise virtualization and backup platform built entirely on secondary-market hardware and open-source software. Supporting critical multi-tenant business workloads—including core identity services, corporate email, file storage, and active application servers—the infrastructure was designed to counter tight capital constraints and aging hardware. By replacing restrictive, vendor-locked solutions with an agile XCP-ng hypervisor cluster and establishing a bare-metal, fully CLI-driven dual LTO tape rotation architecture, the platform successfully delivered continuous high-availability, complete data sovereignty, and a reliable, physically air-gapped disaster recovery pipeline without relying on costly commercial software licensing.

This long-term engineering initiative demonstrates the viability of executing high-availability infrastructure entirely within strict capital constraints. By shifting away from vendor-locked hypervisors to open-source XCP-ng virtualization, Kent-ITS established a resilient, three-node enterprise cluster utilizing secondary-market hardware. To solve the critical requirement for secure, cloud-independent data sovereignty, a custom CLI-driven dual-drive LTO tape backup pipeline was engineered. This architecture successfully sustained continuous multi-tenant business workloads, proving that rigorous, root-cause systems design can deliver absolute uptime and definitive air-gapped ransomware immunity without relying on costly commercial software licenses.

The Challenge

Operating over a ten-year evolutionary lifecycle, this infrastructure faced severe, compounding real-world constraints across compute, storage, and licensing. The initial environment relied on secondary-market enterprise hardware (HP ProLiant DL380 G5 and DL360 G6 hosts) backed by a modest QNAP NAS, which severely restricted iSCSI/NFS storage capacity and VDI retention depth. This hardware limitation was further exacerbated by restrictive vendor licensing; VMware’s free hypervisor actively blocked essential API access, native backup capabilities, and live migrations.

With zero budget allocated for commercial enterprise backup utilities, all data protection workflows had to be engineered entirely from scratch using native Linux CLI tools. As the virtualized workloads expanded, standard LTO4 and LTO5 tape capacities became insufficient to hold a rolling 30 days of unmanaged VDI snapshots. The core engineering challenge demanded maintaining a highly reliable, long-term retention platform with true, cloud-independent, air-gapped physical media security—all while enforcing strict zero-downtime requirements that mandated hypervisor upgrades and hardware migrations be performed entirely live on production systems.

The Solution

1. Open-Source Hypervisor Migration & Cluster Architecture

To bypass restrictive vendor limitations, the entire platform was migrated from VMware to XenServer 7, followed by seamless, in-place upgrades to XCP-ng while critical production workloads remained entirely online. This migration established an agile, licensing-free infrastructure with open storage formats, granular command-line interface (CLI) control, and native live-migration capabilities. The underlying hardware topology was structured as a resilient three-node HP ProLiant DL360 G6 compute cluster, backed by a central QNAP NAS delivering shared iSCSI block storage and NFS data volumes for VM disks and transactional backups.

2. Hardware Lifecycle Evolution

As workload compute demands increased, Kent-ITS executed a live rolling hardware refresh, swapping out two legacy G6 nodes for high-density DL360 G9 enterprise servers. This mixed-generation host pool remained completely stable due to XCP-ng’s robust legacy and modern hardware abstraction layer. This targeted upgrade substantially optimized the infrastructure's aggregate CPU performance, available RAM capacity, and thermal/power efficiency without requiring a complete, high-capital system replacement.

3. Bare-Metal, CLI-Driven Tape Backup Architecture

A dedicated DL360 G6 node was decoupled and re-engineered into an isolated, bare-metal backup server. This server was fitted with a dual-drive physical tape array comprising an LTO4 tape drive mapped to /dev/st0 and an LTO5 tape drive mapped to /dev/st1. Completely avoiding the overhead of commercial backup suites, the entire automation engine was custom-built using standard Linux utilities (tar, mt, cron), direct NFS/iSCSI storage mounts, and native Xen Orchestra API streams.

Key engineered features include:

  • Direct-to-Tape Stream Optimization: Xen Orchestra streamed full and incremental Virtual Disk Image (VDI) chains at a sustained ~110 MB/s directly into the raw tar container interface. This completely bypassed local filesystem overhead, eliminated local scratch-disk requirements, prevented the creation of millions of unmanageable small metadata files, and maintained optimal tape transport speeds to stop destructive tape "shoe-shining."
  • Localized File-Level Compression: High-churn application data sets (Zimbra mail spools and Samba file sharing data) were compressed locally into consolidated backup.tar.gz archives, maximizing tape block capacity and streamlining bare-metal file restoration.
  • Dual-Drive Workload Segregation: To achieve maximum hardware throughput, backup routines split the parallel tasks across both tape drives simultaneously. The LTO4 drive handled the compressed local file archives, while the higher-capacity LTO5 drive independently managed continuous VDI streams and hypervisor metadata.
  • Low-Touch Automation Engine: Specialized cron shell scripts handled low-level tape operations—dynamically setting variable tape block sizes via mt, generating point-in-time archives, running the data stream, and triggering physical media ejection upon task verification. This restricted on-site human intervention to a simple, foolproof daily task: pulling the ejected tape and transferring it to off-site archival storage.

4. Pragmatic Retention & Capacity Management

The storage architecture implemented a tiered retention policy designed to balance physical storage limitations with strict business continuity requirements:

  • Near-Line: 14-day native hypervisor snapshots combined with rolling incremental VDI chains within Xen Orchestra for immediate local recoveries.
  • Air-Gapped: Daily full backups and months of chronological VDI history preserved across a strict physical rotation of offline LTO tape cartridges.

Retention depths were dynamically tuned and optimized over time as VDI structural chains expanded, safely operating within the physical constraints of the QNAP NAS while guaranteeing an uncompromised, long-term historical recovery pipeline.

Project Outcomes

  • Zero Data-Loss Incidents: Achieved total data preservation and absolute recovery verification over a continuous ten-year operational lifecycle.
  • Continuous System Availability: Maintained uninterrupted uptime for business-critical services during live hypervisor cluster migrations and major enterprise hardware refreshes.
  • Air-Gapped Ransomware Immunity: Established a definitive, vendor-independent backup format with a true physical air-gap, keeping historical archives completely safe from network-based encryption attacks.
  • Maximum Lifecycle Value: Extracted peak utility and enterprise performance from secondary-market hardware, completely eliminating the need for recurring commercial licensing fees.

Equipment Used

XCP-ng
VMware ESXi