maxmobley

My obsession with server hardware began like many others my age: hosting Minecraft servers. As a middle schooler, I begged my parents to let me buy an old Dell server to put in our basement. I wasn’t able to convince them at the time, so instead, I made my own “server” the best way I knew how: installing Linux on my hand-me-down iMac. This went poorly as I was a 12 year old that had no idea what bash was, so I quickly reverted back to OS X and gave up on hosting my minecraft servers for the time being.

However, this started a deep interest in server hardware for me: I loved the idea of having my own environment instead of paying a cloud hosting company to do it for me. Eventually, I just got a Raspberry Pi 3 to run my minecraft servers. Fortunately, I quickly outgrew the performance the Pi offered (a Pi 3 could barely run a single vanilla server..), leaving me no choice but to again beg my parents to let me buy a server. This time, though, I was more prepared: I proposed that I would purchase server hardware on government surplus sites with the intent to refurbish and then resell them on eBay. This initially started with my parents fronting the first purchase, but it actually became quite profitable to the point I was running a self-sufficient business mostly on my own as a 15 year old. Instead of pocketing the cash, though, I would just keep some of the servers that I had repaired for myself. I went through many different phases of hardware and software, from base Ubuntu, to UnRaid, FreeNAS, and more. I eventually settled on a 4-node Proxmox + Ceph cluster, which has been running smoothly for over 4 years with only some minor changes.

On each node, I am running a slightly different hardware configuration, but each is based on a Ryzen processor and Supermicro chassis. I settled on Ryzen because of AM4’s interoperability-first approach for its hardware products. At the time, AM4 also had a lot more options for memory and motherboard selection than the Intel ecosystem supported. This was important to me because I intended to be tinkering and switching components between systems often. Ryzen also offered better performance for the money, so it was an easy choice overall. I could have gone with older Xeons as is so common in homelabs, but I wanted to focus on more modern hardware as well as keeping power draw to a minimum wherever possible.

My eventual goal for my entire setup was N+1 redundancy wherever financially and technically feasible. I was not about to source redundant power sources for my residential home, but I doubled up on other systems like networking, power distribution, etc. I can lose any one component, cable, PDU, network switch, server, or UPS before any downtime occurs. For power, this is accomplished through redundant power supplies, with each node primarily running on wall power via a PDU, but also being connected to the UPS for power outages. Each node has a 10GbE SFP+ connection and a 1GbE RJ45 connection on separate switches. The 10GbE is rather necessary for Ceph to function properly, so losing my 10G Mikrotik switch for more than a few hours would be very bad, as nodes would fail to synchronize their data under any larger workload.

My Mikrotik switch is the sole link to the Internet by default (the 1GbE switch will fail-over if the MikroTik turns off), so I can track all incoming/outgoing traffic from the cluster via its per-port statistics. I have never actually checked these until I wrote this article (this switch is about 2 years old), so I was shocked to see I had sent ~240TB outgoing from the cluster and ~200TB incoming. It makes sense with my data intensive workflows, but it’s still just a Very Big Number (especially when the UI shows it in bytes!).

I will never be able to cover all the services I run on this cluster, but it’s mainly web hosting, massive data storage for various projects (primarily web scraping), media servers, game servers, random selfhosted services, course-specific VMs, and much, much more. Having the ability to just spin up a VM or container with whatever specs I want is awesome. The main benefit of having this infrastructure was what I learned while building and maintaining it, though. I cannot emphasize how much I have learned by growing it all out over 6+ years. I learned what systemd was before I learned trigonometry. I was building multi-hundred-TB storage architectures before I knew what a derivative was. I’ve never not known that computers were “my thing”. I’ve never been entirely certain what niche of computer science I wanted to go in to, but over the course of building out my homelab, I learned about embedded programming, AI, cryptography, network protocols, plain old software engineering, and more. If I hadn’t banged my head against a wall all those times while trying to make the many different aspects of a server cluster work, I would not be where I am today.

Today, I’ve mostly moved past the role of Home SysAdmin. My cluster is in a stable enough state, and I have no need for extra compute. Additionally, I’m now in college and simply don’t have the time to tinker with a server rack 700 miles away. I rely on my homelab now to run some services I use daily, but more than that, I rely more on the experience and knowledge it has given me. I’d guess that many software engineers have not been in the situation of deciding whether the slower chipset-linked PCIe lanes will connect their NVMe drive or Infiniband NIC. I certainly feel like the wholistic experience of configuring, building, and fixing the server hardware that my code will be running on makes me a much better software engineer. With that, I have to highly recommend two things: First, parents, let your kids follow their interests (within reason, lol). I am eternally thankful my parents eventually gave in and let me buy that first Raspberry Pi, as I’ve never faltered since in knowing that something with computers would be my career. Second, knowing hardware as a software engineer immediately gives you a leg up. Gaining a deeper understanding of the hardware your code runs on is one of the best things you can do to create, at the very least, better thought-out software.