Case Study: Virtualisation Platform Evolution & Tape‑Based Backup Architecture

Read more about Case Study: Virtualisation Platform Evolution & Tape‑Based Backup Architecture

Over more than a decade of continuous operation, Kent ITS engineered and maintained a highly resilient enterprise virtualization and backup platform built entirely on secondary-market hardware and open-source software. Supporting critical multi-tenant business workloads—including core identity services, corporate email, file storage, and active application servers—the infrastructure was designed to counter tight capital constraints and aging hardware. By replacing restrictive, vendor-locked solutions with an agile XCP-ng hypervisor cluster and establishing a bare-metal, fully CLI-driven dual LTO tape rotation architecture, the platform successfully delivered continuous high-availability, complete data sovereignty, and a reliable, physically air-gapped disaster recovery pipeline without relying on costly commercial software licensing.

Section

Case-Studies

This long-term engineering initiative demonstrates the viability of executing high-availability infrastructure entirely within strict capital constraints. By shifting away from vendor-locked hypervisors to open-source XCP-ng virtualization, Kent-ITS established a resilient, three-node enterprise cluster utilizing secondary-market hardware. To solve the critical requirement for secure, cloud-independent data sovereignty, a custom CLI-driven dual-drive LTO tape backup pipeline was engineered. This architecture successfully sustained continuous multi-tenant business workloads, proving that rigorous, root-cause systems design can deliver absolute uptime and definitive air-gapped ransomware immunity without relying on costly commercial software licenses.

Operating over a ten-year evolutionary lifecycle, this infrastructure faced severe, compounding real-world constraints across compute, storage, and licensing. The initial environment relied on secondary-market enterprise hardware (HP ProLiant DL380 G5 and DL360 G6 hosts) backed by a modest QNAP NAS, which severely restricted iSCSI/NFS storage capacity and VDI retention depth. This hardware limitation was further exacerbated by restrictive vendor licensing; VMware’s free hypervisor actively blocked essential API access, native backup capabilities, and live migrations.

With zero budget allocated for commercial enterprise backup utilities, all data protection workflows had to be engineered entirely from scratch using native Linux CLI tools. As the virtualized workloads expanded, standard LTO4 and LTO5 tape capacities became insufficient to hold a rolling 30 days of unmanaged VDI snapshots. The core engineering challenge demanded maintaining a highly reliable, long-term retention platform with true, cloud-independent, air-gapped physical media security—all while enforcing strict zero-downtime requirements that mandated hypervisor upgrades and hardware migrations be performed entirely live on production systems.

1. Open-Source Hypervisor Migration & Cluster Architecture

To bypass restrictive vendor limitations, the entire platform was migrated from VMware to XenServer 7, followed by seamless, in-place upgrades to XCP-ng while critical production workloads remained entirely online. This migration established an agile, licensing-free infrastructure with open storage formats, granular command-line interface (CLI) control, and native live-migration capabilities. The underlying hardware topology was structured as a resilient three-node HP ProLiant DL360 G6 compute cluster, backed by a central QNAP NAS delivering shared iSCSI block storage and NFS data volumes for VM disks and transactional backups.

2. Hardware Lifecycle Evolution

As workload compute demands increased, Kent-ITS executed a live rolling hardware refresh, swapping out two legacy G6 nodes for high-density DL360 G9 enterprise servers. This mixed-generation host pool remained completely stable due to XCP-ng’s robust legacy and modern hardware abstraction layer. This targeted upgrade substantially optimized the infrastructure's aggregate CPU performance, available RAM capacity, and thermal/power efficiency without requiring a complete, high-capital system replacement.

3. Bare-Metal, CLI-Driven Tape Backup Architecture

A dedicated DL360 G6 node was decoupled and re-engineered into an isolated, bare-metal backup server. This server was fitted with a dual-drive physical tape array comprising an LTO4 tape drive mapped to /dev/st0 and an LTO5 tape drive mapped to /dev/st1. Completely avoiding the overhead of commercial backup suites, the entire automation engine was custom-built using standard Linux utilities (tar, mt, cron), direct NFS/iSCSI storage mounts, and native Xen Orchestra API streams.

Key engineered features include:

Direct-to-Tape Stream Optimization: Xen Orchestra streamed full and incremental Virtual Disk Image (VDI) chains at a sustained ~110 MB/s directly into the raw tar container interface. This completely bypassed local filesystem overhead, eliminated local scratch-disk requirements, prevented the creation of millions of unmanageable small metadata files, and maintained optimal tape transport speeds to stop destructive tape "shoe-shining."
Localized File-Level Compression: High-churn application data sets (Zimbra mail spools and Samba file sharing data) were compressed locally into consolidated backup.tar.gz archives, maximizing tape block capacity and streamlining bare-metal file restoration.
Dual-Drive Workload Segregation: To achieve maximum hardware throughput, backup routines split the parallel tasks across both tape drives simultaneously. The LTO4 drive handled the compressed local file archives, while the higher-capacity LTO5 drive independently managed continuous VDI streams and hypervisor metadata.
Low-Touch Automation Engine: Specialized cron shell scripts handled low-level tape operations—dynamically setting variable tape block sizes via mt, generating point-in-time archives, running the data stream, and triggering physical media ejection upon task verification. This restricted on-site human intervention to a simple, foolproof daily task: pulling the ejected tape and transferring it to off-site archival storage.

4. Pragmatic Retention & Capacity Management

The storage architecture implemented a tiered retention policy designed to balance physical storage limitations with strict business continuity requirements:

Near-Line: 14-day native hypervisor snapshots combined with rolling incremental VDI chains within Xen Orchestra for immediate local recoveries.
Air-Gapped: Daily full backups and months of chronological VDI history preserved across a strict physical rotation of offline LTO tape cartridges.

Retention depths were dynamically tuned and optimized over time as VDI structural chains expanded, safely operating within the physical constraints of the QNAP NAS while guaranteeing an uncompromised, long-term historical recovery pipeline.

Zero Data-Loss Incidents: Achieved total data preservation and absolute recovery verification over a continuous ten-year operational lifecycle.
Continuous System Availability: Maintained uninterrupted uptime for business-critical services during live hypervisor cluster migrations and major enterprise hardware refreshes.
Air-Gapped Ransomware Immunity: Established a definitive, vendor-independent backup format with a true physical air-gap, keeping historical archives completely safe from network-based encryption attacks.
Maximum Lifecycle Value: Extracted peak utility and enterprise performance from secondary-market hardware, completely eliminating the need for recurring commercial licensing fees.

XCP-ng

VMware ESXi

Offices

Meta Tag Description

Learn how Kent ITS engineered a high-availability XCP-ng cluster with custom, CLI-driven dual LTO tape backups on refurbished hardware to deliver absolute uptime and air-gapped ransomware protection.

Title Card Summary

A deep dive into a decade-long virtualisation platform that achieved zero data loss on refurbished hardware. Combines XCP-ng hypervisors with a custom, CLI-driven dual LTO tape architecture to deliver resilient, air-gapped ransomware protection.

Case Study : Self‑Hosted Zimbra Email Platform for UK Distillery – Private Cloud Case Study

Read more about Case Study : Self‑Hosted Zimbra Email Platform for UK Distillery – Private Cloud Case Study

Kent ITS engineered and deployed a high-performance, private cloud email and directory services platform for a UK distillery, replacing an unstable and non-scalable third-party hosted architecture. Designed to overcome severe local infrastructure limitations—including early dependencies on highly latent connectivity—the system was built on dedicated, refurbished enterprise hardware using open-source workloads and robust application-aware backup utilities. Over more than a decade of active production, the platform underwent continuous live evolution, migrating hypervisors, hardening security boundaries, and integrating independent mail gateways to provide near-instant performance, absolute data sovereignty, and an 86% reduction in long-term licensing costs compared to public cloud alternatives.

Section

Case-Studies

This long-term private cloud initiative demonstrates the immense economic and operational value of engineered system longevity over short-term public cloud dependency. Tasked with modernizing a failing, high-latency hosted mail setup for a UK distillery, Kent ITS built an independent, on-premise application environment. By decoupling core workloads into specialized virtual machines and transitioning from commercial VMware to open-source XCP-ng, the architecture eliminated vendor lock-in while vastly increasing compute efficiency. Featuring multi-layered threat mitigation, an application-aware backup matrix, and resilient LTE failover boundaries, this deployment has sustained business-critical communications for over ten years—proving that strict root-cause systems design can deliver unparalleled speed, absolute control, and massive capital savings.

The distillery's legacy email infrastructure, hosted externally via TSOhost, had devolved into a severe operational bottleneck and an escalating risk to business continuity. Because the site originally lacked a fixed broadband connection and relied on unstable early mobile links, external IMAP latency caused routine email actions to take minutes instead of seconds. This performance degradation was compounded by unstructured shared mailbox habits and exponential growth in message volumes, which the hosted platform could not scale to support even after basic ADSL lines were introduced.

Administratively, the organization possessed zero control over system configurations, lacked any native performance-tuning mechanisms, and had no reliable, independent backup or recovery pipeline. The enterprise was entirely dependent on a black-box third-party provider, creating an unacceptable operational environment where business data lacked sovereignty, visibility, and basic disaster resilience.

1. Bare-Metal Infrastructure & Hypervisor Lifecycle Evolution

To achieve absolute performance control, Kent ITS established an on-site, rack-mounted private cloud environment utilizing structured cabling. The initial compute foundation relied on a refurbished HP ProLiant DL380 G5 server subjected to complete firmware lifecycle updates prior to production. The environment was initially virtualized using VMware ESXi, but as the platform evolved, it was seamlessly migrated in-place to an open-source XCP-ng ecosystem running on newer G6 hosts, achieving total hypervisor independence with zero operational downtime.

5. Decoupled Workload Architecture

To ensure systemic stability and simple administrative scaling, workloads were strictly isolated into dedicated Ubuntu Server virtual machines:

Directory Services: Centralized identity and user access control.
File Services: Secure local storage repositories.
Email Core: A dedicated Zimbra Collaboration deployment.

3. High-Performance Mail Engineering & Migration

Kent ITS executed an IMAP-based migration from the legacy TSOhost server, transferring historical mail volumes with zero data loss. By hosting the Zimbra instance locally within the private cloud network, external IMAP latency was completely eliminated, collapsing multi-minute mail delays down to near-instantaneous client responsiveness.

4. Storage Optimization & Application-Aware Backups

While shared storage originally allowed for virtual machine mobility, it introduced dangerous file locks and snapshot-related system instability. An engineering decision was made to prioritize absolute data integrity over live-migration capabilities by migrating the VM disks onto high-speed, local RAID-backed storage arrays—completely eliminating read-only disk state risks. For data protection, the architecture integrated Zextras, an application-aware backup suite that handles continuous incremental protection and allows granular object-level restores (from a single message up to a full disaster recovery).

5. Multi-Layered Security Hardening & Threat Reduction

Following a credential reuse event, a rigorous hardening framework was applied across the perimeter:

Brute-Force Mitigation: fail2ban was deployed across both the primary mail core and edge gateways.
Access Control: System-wide credential resets paired with strict, complex password policies.
Proxmox Mail Gateway (PMG) Integration: A dedicated, isolated edge mail gateway layer was built in front of the mail server (Internet → Mail Gateway → Mail Server), shielding it from direct exposure. This gateway enforced strict security filters: greylisting ( $450$ temporary rejections), Reverse DNS validation, DNSBL blocklists (including Spamhaus), explicit recipient verification, and firewall-level blacklists.

6. Resilient Business Continuity Engineering

During a severe, week-long broadband outage caused by a physical ISP infrastructure failure, Kent ITS activated an automated failover path. Utilizing a specialized MikroTik LTE gateway, inbound and outbound mail traffic was instantly rerouted over cellular networks, maintaining continuous, uninterrupted business operations with zero mail drops or data loss.

Near-Instant Performance: Transformed the user experience by lowering email processing latencies from multi-minute delays to instant, real-time access.
86% Licensing Capital Reduction: Saved the business over £18,000 across a 10-year horizon by opting for a self-hosted Zimbra/Zextras environment (~£3,000 total) over an equivalent 25-seat Microsoft 365 licensing model (~£21,000 total).
Definitive Threat Reduction: Drastically shrunk the mail network's visible attack surface and slashed inbound spam volumes via a multi-layered Proxmox Mail Gateway topology.
Proven Operational Resilience: Verified absolute site survivability during a catastrophic 7-day ISP broadband failure through rapid MikroTik LTE failover orchestration.
Engineered System Longevity: Validated a long-term architecture that seamlessly survived complete hypervisor changes (VMware to XCP-ng), underlying hardware generation upgrades (G5 to G6), and multiple major operating system updates over more than a decade without requiring a platform replacement.

Zimbra Collaboration Suite

Meta Tag Description

Discover how Kent ITS built a self-hosted Zimbra email platform on XCP-ng for a UK distillery, cutting licensing costs by 86% while ensuring near-instant performance.

Title Card Summary

A look into a ten-year private cloud email platform built on refurbished hardware for a UK distillery. Details the migration from VMware to XCP-ng, a Proxmox Mail Gateway threat defense layout, and an 86% cost reduction over public cloud alternatives.

Subscribe to VMware ESXi