^(OP reply with the correct URL if incorrect comment linked)
[Jump to Post Details Comment](/r/homelab/comments/1bopa7z/a_short_followup_video_tour_of_my_homelab_i_know/kwqhi10/)
This is just a short walk around of my homelab from the previous post I did a few days ago. I did some rewiring work over the last few weeks so it is a tad be more organized than it was before.
As I mentioned in the video the server room is on a lower floor underground which has the benefit of really containing the noise! By far the most common question is power - With everything running the room peaks at about 7kw, however most of the time it is around 5kw.Those kw are of course almost entirely converted to heat, and that heat is removed with both a minisplit as well as an economizer (outside air exchange).
Since it is in the 50s here in Portland more than half the year the outside air cooling help a lot.
Damn, that power consumption is crazy! If i convert it to the kWh price i currently have then i would pay alone for the consumption between €43.800,,- and €61.320,- a year!
Portland OR?? And you have a basement??
That is intense! I have 2 4U supermicro servers for storage, you have like 10 of them! My God!
Also, I checked out your garage journal thread on the whole house build. I'm trying to pick my jaw up off the floor.
Seeing as we have similar educations, one of us is severely getting underpaid! You guy's hiring? I'm tired of working at big I for garbage salary! I see you used to work there too, and your clearly better off now that you're NOT working there - DAMN!
Also I know hindsight is 20/20, but you should've run a lot more single mode OC2 fiber in your house, and a lot less CAT6 - I ran CAT6 in my house too, and I just ordered a 10Gig copper switch, but to do anything over 10Gig (or even for 10Gig), fiber is the way to go. At least I put conduit in my walls, so I can pull some fiber in pretty easily after the fact!
As u/Bellegr4ine mentioned below, I am the CTO of Surescripts, which is a healthcare technology company that handles the majority of prescriptions in the US. >30 billion transactions, so lots of tech and network. I came to Surescripts back in 2013 when they acquired the company I co-founded.
I started my career fresh out of college as an engineer at Intel at the start of the Pentium era, which was both an incredible experience and really gave me an appreciation for serious complexity in engineering. As you can probably infer from my videos I am on the more 'super technical nerd' side of the envelope... more so because I am curious and willing to invest some time to turn that into learning.
More than anything else, my biggest reason for success has been the ability to take advantage of opportunity. Opportunity comes and goes, and either you are prepared to take advantage of it or not.
OP, please feel free to step in and correct me if I'm wrong at all here, but I think I can provide some perspective. No disrespect is meant at all (you definitely have an awesome setup that I'm super jealous of), I'm just trying to bridge the gap between homelabing and what people like you and I are used to working with at the datacenter/enterprise level.
From a home user or home lab perspective, yes this quote about this being "simple" seems a bit wild. It's likely not that much more *complex* than most homeland, but it's definitely much *larger* and has more redundancy. For people who work in data centers where this kind of gear normally lives, this is a very standard/simple setup (it's just unusual and awesome to see it in a home).
Look at power, for example, A/B power with UPSes seems like an excessive thing to your average homelabber, but it's pretty standard in a datacenter. Most homelabbers use single UPSes to provide power for enough time to shut things down, or maybe they'll run some mission critical things until the batteries die. Datacenters run UPSes only for a few seconds until the diesel generators kick in and take over.
A/B power is typically load balanced, but each phase is able to take the full load if you lose one side, so there's a backup for the backup. Now add trio of diesel generators. One for A, one for B, and one as backup for when you're servicing the other two. Add enough diesel fuel to last a week, and you might qualify for a MEF certification. In theory you could lose mains power, two generators, and one side of the power distribution (including the UPS) and still be running at 100%.
OP has solar for backup and it sounds like it works pretty well, but I'm assuming he doesn't have a backup for it if there's a problem with the mains power and the solar, and then another backup for that backup.
(I regularly work in a medium sized single-story MEF 3 certified datacenter, and the power room (batteries, panels, breakers, etc) is bigger than OP's house, and the generator yard is as well.)
Same thing with the networking. It looks legit and has redundancy (links to multiple switches, which is standard in datacenters), and is definitely pretty epic. But it's "simple" in the sense that it's probably fairly flat. I'm assuming there are a good handful of VLANs, but he doesn't mention anything unusual happening at layer 3. Perhaps he has a routing protocol (ISIS, OSPF, maybe iBGP) running between the switches, perhaps not. OP did say in another comment that he has a fairly standard cable internet connection with Starlink as a backup, so he likely doesn't have his own AS and is likely not peering with any IXes via BGP.
So yes OP does have an awesome and drool-worthy setup, but it's probably a lot more "simple" than he's used to being around at work.
OP, thanks for sharing this project/adventure with us here. Your post history, house build blog, and YouTube channel have been fun rabbit holes to dive into! You definitely have some fun and expensive hobbies that most of us can only afford to dream about 😅
Those are fantastic questions, and very accurate assumptions. This is a homelab, and while I have done some things that are above average, this is in no way comparable to real high level datacenter operations. I professionally manage extremely high reliability and high scalability operations doing billions of transactions so I am very familiar with what that entails..and some of those environments are just fantastic in complexity. ( and can occasionally have that complexity bite you as happened with our racks in Flex a few months ago that had a failure that also crippled Cloudflare).
I did aim for simplicity in places that I could. Network, as you suggest, is pretty flat. VLANs of course, separation for different IOT things, but on the whole not that complex. If any thing some complexity is just for the sake of experimenting and learning. No external BGP, and overall simply mechanism for reliability. (Not that I would not like to have multiple real connections, I just can't get them where I live yet!).
On the power side, that also has a basic approach - Keep things running without a lot of intervention. The UPS provides good isolation, but only 10 mins of runtime. The UPS is backed up by a second house wide 42kwh battery system (Enphase), and that system is backed up by both the grid, solar, and a generator. The generator can run off natural gas, and propane as a backup. One big advantage of the battery+generator setup is the generator does not have to run all the time. When the batteries get low, the genset will fire up and run the generator at 100% to charge the batteries.
The single most important thing - This is a playground and a lab. The moment it becomes more than a place I can learn it becomes less useful. For example - It was really fun to automate the power failure detection so certain machines auto power down if I am running on battery/generator. There are always cases where it does't work like I thought it would, and I learn more, regroup, and try again.
Looking great. Do you have written what the rack contain in detail somewhere? How many Petabyte and Compute power do you have? :)
Is it running 110 or 240v? 10-11amp is crazy. It would cost me 3.484 dollars/month to run in Denmark, if its 11a @ 240v. (11amp \* 24 hours \* 30 days \* 0,44$/kwh) \^\^
My homelab use only 0,6a @ 240v =\~ 200$ month :)
I'll preface with I'm not super hip to server hardware. Are there 80+ titanium server PSUs you can get to ease your power draw a little bit more? Have you gotten rid of every mechanical HDD for SSD? Have you considered upgrading to newer CPU architectures that have better idle power draw?
I know small to medium Businesses who live of a 4 bay QNAP NAS. What are you doing with als those Machines 🤣🤣🤣
It is really nice tho, ngl, I´m kind of jealous. 😜 Great Work!
Don’t get my question wrong. I absolutely love it from a nerd tech side. My question is why though. Like I have a small proxmox running but it basically runs the home nas and a Plex server and it runs HA.
I can’t wrap my head around what one can do with all that in a home.
So it's just creeping features. You have a plex server on proxmox now imagine wanting to never delete anything, add storage, now image wanting to back that up add more storage, now imagine wanting that to be highly available add a cluster, now you want better networking add switches, add routers, now you want to make sure that they don't just turn off in an emergency add a ups. It's a rabbit hole you can go down and if you have the knowledge and the money before you know it you've converted a whole basement too it haha.
Wow 🤯 I’m very impressed, not only by the equipment but by how organized you’ve managed to keep everything!
That energy usage makes my eyes water though 😅 I’m in the Netherlands, where my average cost per KW is €0.29/$0.31 and I don’t have solar because I live in an apartment so your setup would cost me €73.70/$79.52 p*er day*
For context, that’s more electricity than I use in an entire month for my whole apartment 😳
My homelabing started with, I want a My Cloud Home, couple years later I have two Dell workstation Towers that are already overkill for my current needs, I know what my setup will look like in a couple more years 😃
Awesome setup, loved the monitoring monitors so much.
Absolut great!! But would not call it a homelab.. just the storage capacity way out of home use …
Would you tell us the specs and what you are running it for?
^(OP reply with the correct URL if incorrect comment linked) [Jump to Post Details Comment](/r/homelab/comments/1bopa7z/a_short_followup_video_tour_of_my_homelab_i_know/kwqhi10/)
This is just a short walk around of my homelab from the previous post I did a few days ago. I did some rewiring work over the last few weeks so it is a tad be more organized than it was before. As I mentioned in the video the server room is on a lower floor underground which has the benefit of really containing the noise! By far the most common question is power - With everything running the room peaks at about 7kw, however most of the time it is around 5kw.Those kw are of course almost entirely converted to heat, and that heat is removed with both a minisplit as well as an economizer (outside air exchange). Since it is in the 50s here in Portland more than half the year the outside air cooling help a lot.
You might want to consider installing an air-source water heater and duct the DC air to it. Will take some load off the AC in winter :)
The economizer pulls in cold air from outside, and the warm air is pushed into the upper garage, so that heat keeps that garage warm.
So much this! HP water heaters thrive in warm(ish) rooms.
Came here to offer similar observation. A preheat tank before the water heater with off heating from electronics would be economically feasible here
Damn, that power consumption is crazy! If i convert it to the kWh price i currently have then i would pay alone for the consumption between €43.800,,- and €61.320,- a year!
At my rate in Canada its $11,826 Cad/year... much cheaper but still hard to swallow.
Portland OR?? And you have a basement?? That is intense! I have 2 4U supermicro servers for storage, you have like 10 of them! My God! Also, I checked out your garage journal thread on the whole house build. I'm trying to pick my jaw up off the floor. Seeing as we have similar educations, one of us is severely getting underpaid! You guy's hiring? I'm tired of working at big I for garbage salary! I see you used to work there too, and your clearly better off now that you're NOT working there - DAMN! Also I know hindsight is 20/20, but you should've run a lot more single mode OC2 fiber in your house, and a lot less CAT6 - I ran CAT6 in my house too, and I just ordered a 10Gig copper switch, but to do anything over 10Gig (or even for 10Gig), fiber is the way to go. At least I put conduit in my walls, so I can pull some fiber in pretty easily after the fact!
Why would you live in Portland
Why do you live where you live??
Because I don’t risk getting killed or robbed easily.. nor would want laws against victims and which protect criminals..
How do you manage to live in Portland? The biggest shithole…
All that for Minecraft?
Love it! What do you do for a living if you don’t mind sharing
he has AN OF
more probably hosts it... :)
He's the CTO of Surescripts most likely.
As u/Bellegr4ine mentioned below, I am the CTO of Surescripts, which is a healthcare technology company that handles the majority of prescriptions in the US. >30 billion transactions, so lots of tech and network. I came to Surescripts back in 2013 when they acquired the company I co-founded. I started my career fresh out of college as an engineer at Intel at the start of the Pentium era, which was both an incredible experience and really gave me an appreciation for serious complexity in engineering. As you can probably infer from my videos I am on the more 'super technical nerd' side of the envelope... more so because I am curious and willing to invest some time to turn that into learning. More than anything else, my biggest reason for success has been the ability to take advantage of opportunity. Opportunity comes and goes, and either you are prepared to take advantage of it or not.
im interested too
"So, some pretty simple stuff here..." funniest quote. Amazing setup!
OP, please feel free to step in and correct me if I'm wrong at all here, but I think I can provide some perspective. No disrespect is meant at all (you definitely have an awesome setup that I'm super jealous of), I'm just trying to bridge the gap between homelabing and what people like you and I are used to working with at the datacenter/enterprise level. From a home user or home lab perspective, yes this quote about this being "simple" seems a bit wild. It's likely not that much more *complex* than most homeland, but it's definitely much *larger* and has more redundancy. For people who work in data centers where this kind of gear normally lives, this is a very standard/simple setup (it's just unusual and awesome to see it in a home). Look at power, for example, A/B power with UPSes seems like an excessive thing to your average homelabber, but it's pretty standard in a datacenter. Most homelabbers use single UPSes to provide power for enough time to shut things down, or maybe they'll run some mission critical things until the batteries die. Datacenters run UPSes only for a few seconds until the diesel generators kick in and take over. A/B power is typically load balanced, but each phase is able to take the full load if you lose one side, so there's a backup for the backup. Now add trio of diesel generators. One for A, one for B, and one as backup for when you're servicing the other two. Add enough diesel fuel to last a week, and you might qualify for a MEF certification. In theory you could lose mains power, two generators, and one side of the power distribution (including the UPS) and still be running at 100%. OP has solar for backup and it sounds like it works pretty well, but I'm assuming he doesn't have a backup for it if there's a problem with the mains power and the solar, and then another backup for that backup. (I regularly work in a medium sized single-story MEF 3 certified datacenter, and the power room (batteries, panels, breakers, etc) is bigger than OP's house, and the generator yard is as well.) Same thing with the networking. It looks legit and has redundancy (links to multiple switches, which is standard in datacenters), and is definitely pretty epic. But it's "simple" in the sense that it's probably fairly flat. I'm assuming there are a good handful of VLANs, but he doesn't mention anything unusual happening at layer 3. Perhaps he has a routing protocol (ISIS, OSPF, maybe iBGP) running between the switches, perhaps not. OP did say in another comment that he has a fairly standard cable internet connection with Starlink as a backup, so he likely doesn't have his own AS and is likely not peering with any IXes via BGP. So yes OP does have an awesome and drool-worthy setup, but it's probably a lot more "simple" than he's used to being around at work. OP, thanks for sharing this project/adventure with us here. Your post history, house build blog, and YouTube channel have been fun rabbit holes to dive into! You definitely have some fun and expensive hobbies that most of us can only afford to dream about 😅
Those are fantastic questions, and very accurate assumptions. This is a homelab, and while I have done some things that are above average, this is in no way comparable to real high level datacenter operations. I professionally manage extremely high reliability and high scalability operations doing billions of transactions so I am very familiar with what that entails..and some of those environments are just fantastic in complexity. ( and can occasionally have that complexity bite you as happened with our racks in Flex a few months ago that had a failure that also crippled Cloudflare). I did aim for simplicity in places that I could. Network, as you suggest, is pretty flat. VLANs of course, separation for different IOT things, but on the whole not that complex. If any thing some complexity is just for the sake of experimenting and learning. No external BGP, and overall simply mechanism for reliability. (Not that I would not like to have multiple real connections, I just can't get them where I live yet!). On the power side, that also has a basic approach - Keep things running without a lot of intervention. The UPS provides good isolation, but only 10 mins of runtime. The UPS is backed up by a second house wide 42kwh battery system (Enphase), and that system is backed up by both the grid, solar, and a generator. The generator can run off natural gas, and propane as a backup. One big advantage of the battery+generator setup is the generator does not have to run all the time. When the batteries get low, the genset will fire up and run the generator at 100% to charge the batteries. The single most important thing - This is a playground and a lab. The moment it becomes more than a place I can learn it becomes less useful. For example - It was really fun to automate the power failure detection so certain machines auto power down if I am running on battery/generator. There are always cases where it does't work like I thought it would, and I learn more, regroup, and try again.
This is so overkill. I love it 😄
Homelab final boss
So this is what rich people do. I like that.
Wait... you have a second SECRET lab. You can't tease us like that. Cool video, I with my datacenter looked half as good as this.
Here :) https://www.youtube.com/watch?v=XHvdqB6LSg0
JFC I thought I had an OK retro collection. I kneel.
Now this is a setup to be jealous of. Love all the storage blocks. This is a home data center!
What the actual fuck
Looking great. Do you have written what the rack contain in detail somewhere? How many Petabyte and Compute power do you have? :) Is it running 110 or 240v? 10-11amp is crazy. It would cost me 3.484 dollars/month to run in Denmark, if its 11a @ 240v. (11amp \* 24 hours \* 30 days \* 0,44$/kwh) \^\^ My homelab use only 0,6a @ 240v =\~ 200$ month :)
Everything is 240V except the lights, and the KVMs.
That's a lot of power. Can you share some more what hardware you have? :)
I'll preface with I'm not super hip to server hardware. Are there 80+ titanium server PSUs you can get to ease your power draw a little bit more? Have you gotten rid of every mechanical HDD for SSD? Have you considered upgrading to newer CPU architectures that have better idle power draw?
I know small to medium Businesses who live of a 4 bay QNAP NAS. What are you doing with als those Machines 🤣🤣🤣 It is really nice tho, ngl, I´m kind of jealous. 😜 Great Work!
Don’t get my question wrong. I absolutely love it from a nerd tech side. My question is why though. Like I have a small proxmox running but it basically runs the home nas and a Plex server and it runs HA. I can’t wrap my head around what one can do with all that in a home.
So it's just creeping features. You have a plex server on proxmox now imagine wanting to never delete anything, add storage, now image wanting to back that up add more storage, now imagine wanting that to be highly available add a cluster, now you want better networking add switches, add routers, now you want to make sure that they don't just turn off in an emergency add a ups. It's a rabbit hole you can go down and if you have the knowledge and the money before you know it you've converted a whole basement too it haha.
"Pretty simple Stuff" Nice and tidy, love it!
#lifegoals
How are you getting telemetry from your UPSes? Everything looks SO organized…it’s terrifying!
I am directly measuring the power in and out of the UPS , as well as the power for each rack, using CTs
🤩 awesome ! Where do u have buy the light on your wall ?
Wow 🤯 I’m very impressed, not only by the equipment but by how organized you’ve managed to keep everything! That energy usage makes my eyes water though 😅 I’m in the Netherlands, where my average cost per KW is €0.29/$0.31 and I don’t have solar because I live in an apartment so your setup would cost me €73.70/$79.52 p*er day* For context, that’s more electricity than I use in an entire month for my whole apartment 😳
Amazing, I have a full 46U cabinet, wiring diagrams, etc. But this makes me feel lazy and like I only have a Chromebook by comparison. Super cool.
Master. Show me the path and ill follow you :D
Do you analyze data files for a living or something holy shit
My homelabing started with, I want a My Cloud Home, couple years later I have two Dell workstation Towers that are already overkill for my current needs, I know what my setup will look like in a couple more years 😃 Awesome setup, loved the monitoring monitors so much.
Holly smokes!! Now that’s a home lab!!!!
Absolut great!! But would not call it a homelab.. just the storage capacity way out of home use … Would you tell us the specs and what you are running it for?
Tell me you're the PERFECT house to rob without telling me you're the perfect house to rob.