Let’s Start A Home Lab … Said Tony One Day

Home Lab, , , , ,

Last week I finally started building the home lab I have been thinking about for the better part of 15 years. It took a lot of deliberations, justifying why I need one, what makes sense and what does not, make a decision, just to finally have the end result quietly humming in my hallway while I come up with the right workloads to make it melt. This is the first part of a series of blog posts on the building process, challenges, and decisions along the way.

Some Background

Spending fifteen years at VMware, I almost always had compute resources to spare for anything I need. Our internal cloud gave out as many VMs as your team’s quota allowed, and you could keep them alive for a long time (that a vCenter got decommissioned). Furthermore the vSphere UI team owned and operated their own rack which was unjustly underutilized by engineers so I almost had the system at my disposal.

What did I need compute for, you might ask?! Many thinks as it turns out:

  • Dev VMs During Corona, I was at the mercy of the network connection to the office to code on my workstation which per company policy had to be either a windows PC or a Mac. I would never choose Mac and Windows Subsystem for Linux (WSL) was not reliable yet so my only other option was to run Linux on a Dev VM – either on my office machine or in the Sofia datacenter. Since I was doing frontend development, I could debug in the browser, and I only needed SSH to the VM to connect Visual Studio Code to it. The VM I used was a monster for its time – 24 Xeon cores, 32 GB of RAM – juuust enough so that Node.js do not crash with our build.
  • VMware and Partner Appliances in my role as the architect for vSphere Client extensibility it was critical to constantly keep track on the plugin ecosystem and learn all the ways the team and I missed how partners are using our product and APIs. Especially for solutions for software-defined networking, having better access to the networking fabrics and potentially even to the physical servers was important.
  • Kubernetes clusters while we had a k8s-as-a-service devops offering, for certain projects it was insufficient. A project such as KEDA for orchestrating distributed k8s workloads requires access to the cluster, not just a namespace. Obviously not an easy thing to allow to a single engineer if you’re the team supporting a few tens of thousands of engineers.

A few things drove my interest in building my own home lab as an alternative to this: In order:

  1. For years, I was toying with the idea. William Lam was a great design partner for many projects I was doing over the years. Everyone in the virtualization community knows him as one of the most prolific evangelists and advocates not only for VMware, but also for home-labbing. For years I was following his updates in the field trying to justify to my own frugality why I need a lab.
  2. In 2024 Broadcom finalized their acquisition of VMware. Despite all the good things the acquisition triggered there were not so great: they stopped the Thursday beer bashes and disturbed my corporate social life immensely … and they put severe restrictions on the amount of hardware the VCF division got, greatly reducing the bandwidth for CI/CD automation, never mind our ability to play with distributed workloads.
  3. In September of 2025 I was laid off.

The Purpose

For some time after my sudden unbusying I just enjoyed having infinite time to spare on hobbies and sport, but gradually, personal projects creeped back into my brain and the compute I had at home proved inadequate for everything that laid ahead. What I did have was:

  • A desktop PC that was quite good and I had built during the Corona to build the chromium codebase quickly:
    • AMD Ryzen 9 5990X, 12/24 core/thread processor on a ASUS TUF Gaming B550+ motherboard.
    • 64 GiB DDR4 RAM – one stick of the two unfortunately failing just as RAM prices hit record high due to the AI bubble.
    • 1TB Western Digital WD_BLACK SN850 – and another older SSD and a 16TB HDD in addition to these.
    • NVIDIA GeForce GTX 970 4GiB VRAM, that just got upgraded to one of the cheapest GeForces available on the market – the RTX 5060 8GiB.
  • Synology DiskStation DS918+ NAS I have mostly used to run Plex Media Server, streaming all my totally-not-illegal-I-swear content, a backup location, and a replacement Dropbox solution for my dad as I could not live with the thought of him actually paying the salaries of software engineers.
    • Intel Celeron J3455 Quad Core
    • 4 GB SO-DIMM DDR3L.
    • 4×4 TB Seagate Barracuda in a RAID5 configuration for a total of roughly 10TB of storage space.
    • 2x Gbit ethernet adapters in link aggregation.

I started playing with the idea of just using the NAS since it could do VMs and docker containers but it became apparent quickly that it will be inadequate. For starters, nobody runs plain containers for microservices, we expect Kubernetes. Luckily, it is possible to run rancher on the machine … somewhat. Basically it took four or five attempts between Gemini and me to figure out which version of the rancher image can even boot on Synology DSM OS and would not get confused by the system’s “exoticness”. Then when the system finally ran and I could get the Rancher UI up I realized that I had – with great effort and to an even greater success – managed to create a k8s cluster with just a control-plain node and no actual worker nodes! With Rancher per my understanding, running a single node for both control plane and workloads is somewhere between impossible and inadvisable. Now, I could have spun another VM on the NAS and have it be the workload node but I already saw the CPU spike to 30% and RAM to over 45% so I didn’t have enough headroom for anything meaningful.

So, I was looking for an alternative. I could run a VM on my desktop and set up k8s inside it but it had to be Hyper-V as I had enabled it to run WSL and I disliked the idea of expanding the usage beyond this. I was pleasantly surprised at how much Docker Desktop for Windows had evolved over the years. Since the last time I tried it not only did it learn to use WSL but it could run a Kubernetes node as well. After a few hiccups, Gemini and I managed not only to get it working, but to connect it to the Rancher control plane mode as well, so that … finally … I could run a single demo nginx pod.

So, I had a way to run k8s workloads but only that. I wanted to learn how to administer a cluster not only a k8s pone but also Proxmox, Nutanix CE, or Harvester. Running multiple VMs with nested hypervisors is can help you how to configure virtual infrastructure but is not a recipe for a performant production environment. On top of learning infrastructure software I wanted to run a full multi-node k8s cluster on top of and next to which I wanted to deploy distributed multi-service and multi-instance apps of my own making and better understand their performance characteristics in a proper computer cluster.

It was obvious I need to spend some money. But for what exactly?

I knew I wanted at least a three-node cluster to be able to set up proper HA / failover policies with handing of network partition. Before you say anything: e should not let the non-criticality of the workloads go in the way of proper infrastructure design, right?

The Constraints

Starting with – surprise! – the budget. This is not my startup project so I did not want to spend what used to be a month’s salary on the project, especially given that in a few years the setup is going to be obsolete. This ruled out purchasing brand new servers, AI rigs, or gold-plated micro PCs such as Minisforum. I would have to go refurbished. Luckily, my Synology NAS could serve as the shared storage for the cluster, so I had one thing less to worry about. Once I capped the … capex, I had to think about the opex as well. If I was not careful, I can easily double my electricity bill if picking machines that are too many generations behind. The more efficient the better, especially if my suspicion that I’d be running sporadic heavy workloads proves correct.

Another thing to consider was location. When I bought my current apartment and remodeled it, I had to sacrifice the pantry to expand the second bathroom. What I was left with was 45 cm / 20″-deep recess for a washing machine and a water heater. Next to it I added a long row of ceiling-high cupboards / wardrobes with shelves of configurable height. I had planned up front to have electricity and communication cables passing through all of the sections. Also, all rooms’ ethernet cables end up in the section closest to the entrance, and this is where my router sits near the entrance to the apartment, where all communication cables go in.

The problems with the cupboard are two: depth and ventilation.

Cupboards are 38cm / 15″ deep. Obviously fitting a standard rack is not possible. I have friends that keep a blade tilted to rest on the wall, and this could possibly be a solution, but then came the second concern. I had anticipated that whatever I install in the cupboard is bound to generate some heat and had the cupboard maker cut out pairs of 425 mm x 65 mm / 16.73″ x 2.56″ vents in the doors of the cupboards to facilitate airflow.

Still, without air conditioning, a regular set of servers would either melt themselves or burn the furniture to the ground pretty quickly.

Moving the servers to the study was not an option too. First, between the computers, huge desk, and musical instruments there’s hardly any spot left.

Note that this is an old, uncluttered version of the study, when I didn’t keep two more guitars around.

Leaving lack of space aside, the rack-mount servers within my budget eat electricity as if I have a nuclear reactor in the garage, and reach the noise levels of a jet fighter taking off. Even in a hallway cupboard this is going to be too much. And while I can proooobably put up with it – after all I once slept by an open window 50 metres from the second busies highway in the country, and I spent five years living with a beeping burglar alarm with dead battery – my significantly-better half, who wakes up from the sound of my typing on the iPad on-screen keyboard, will certainly not be OK with the setup.

The BOM

What I was left with is buying a set of refurbished tiny PCs I’ve verified are a good fit for Proxmox and/or Nutanix CE. Research with Gemini narrowed the choice down to a few Dell OptiPlex, HP EliteDesk, and Lenovo ThinkCentre Tiny models. Dells and HPs are generally sold at a premium so I pushed down my distrust of Lenovo and their loyalty to the Chinese Communist Party. The model I picked is ThinkCentre M80q Tiny. The reason I picked it was that the Intel i5-10500T 12 thread CPU and the ability to further upgrade the hardware with more RAM and storage – last two both critical for Nutanix CE. Also, the offer I got from Impacto included the choice to upgrade RAM to 32 GB and I got a 12% discount for a total cost of 1,305 € with VAT. More or less this was the total cost of the lab since the only thing I bought on top of that was a smart plug, a patch cable, and another power strip.

I must admit, the machines look pretty cool and at 204.2 mm x 186.6 mm x 64 mm / 8.04″ x 7.35″ x 2.52″ these things are worthy of their Tiny branding.

The final inventory used is as follows:

The End Result

The first version of the server rack is shown below. It’s dirty, messy and invigorating. The machines are running Proxmox the installation of which I would cover in another post.

Power draw is stable around 49 W and daily consumption is around 1.2 kWh. Using the TP-Link P110M smart plugs and their Tapo app for iOS to monitor the usage and control the uptime. The plugs are also paired with Apple Home but I can only turn the plugs on or off from it.

The temperature of this version is 26°C / 79°F with the cupboard door open and raises up to 33°C / 91°F with the door closed. Given that this is the same temp at which my desktop idles it is good enough. Still, the middle two servers are tightly packed and heat up more than the Synology NAS and the top-most server which can lead to some CPU throttling. Fortunately, there is room for improvement based on what I plan to do next.

A Note On the Title

Not wasting time chasing originality, I poached the title off the third album by UK punk legends The Exploited. And while I do personally love Maggie in contrast to Wattie and company, there is no denying the album is is a classic. Give it a listen, then checkout The Massacre and Beat the Bastards which are even stronger crossover thrash masterpieces.

トニー

      

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.