T O P

  • By -

Daveid

If it's brand new, make it HP's problem and give them a call ([1-800-633-3600](tel:1800-633-3600)). Also I think you're in the wrong sub if you're looking for legit advice.


cisco_bee

I put in a support ticket before this post. Also, in my experience, this sub is *much* more helpful than *that other sub*. And when it's not helpful, it's hilarious. Win-win.


capn_doofwaffle

Here's your grain of salt . Take it and use it wisely. šŸ¤£


TPlays

Fuck it, Iā€™ll give him mine too. .


orangekrate

Howany grains till we have enough for margaritas?


capn_doofwaffle

Shitty Sys Admins don't even bother mixing, we drink straight from the bottle... šŸ¤£


askaboutmy____

the mixer bottle...


1isntprime

Well if we are trying to have enough for everyone then sadly one grain donated per person will never be enough


Yikes_Hmm

And my axe!


Br0k3Gamer

Brah, on my screen, thatā€™s pepper.Ā 


zer0moto

šŸ˜‚


Eviscerated_Banana

Those 3 flashy lights at the point of failure look relevant, much cooler looking than the usual single LED's I get to deal with too :)


Platocalist

life hack


espero

You need to figure out how to reset the firmware and other things onboard like "pram" if present.Ā  Before that take out all disks, all memory and all power supplies and put it back in. Reseat all riser cards if present too. There should be a hardware diagnostics tool in the HP Integrated lights out menus, at least with a hardware event log... Godspeed


cisco_bee

I like to push the pram a lot.


espero

There may be small reset buttons, push them all


Haunting_Tax_

have you been having issues with the fans running full blast and ilo going unresponsive? Got a brand new DL380 with the same behaviour.


dagbrown

I hear ya. The other guys would probably come up with a solution involving donating large amounts of money to their favorite charity, Microsoft.


liebeg

Annoing somebody else sis always a solution


cisco_bee

Annoying stupid people is just a bonus. All the cool people in this thread had fun.


CryptographerLow9657

Lol - sysadmin is full of shit sysadmins and shittysysadmin is full of decent sysadmins šŸ˜†


Sethdarkus

Checks out some people just wanna get a good laugh lol


cisco_bee

I was able to "manually" flash the firmware via the iLO and afterwards it booted. So it was definitely something the ~~Smart~~ Crummy Update Manager did.


archiekane

Ah yes, when the update requires another update to allow access to your hardware.


mawesome4ever

In before you gotta pay for updates to not brick your phone (uhh ransomeware?)


Mach3Tech

Well, just like the American prison system. Criminals learn from other criminals. (Cell phone company taught the hackers a new trick)


derfmcdoogal

Or your reseller if you bought it through one. Got an HP laptop in that would have a fuzzy screen on boot, turned around and gave it back to them within a week, and they said they would fix it. Invoice came in and I denied payment then they got all Pikachu faced when I told them we weren't paying until the issue was resolved. Resellers, HP, not sure which one I hate worse.


rustytrailer

I hate being sent to the manufacturer by the reseller. Like you sold me this shit and it was doa. You get on the phone with them we paid you a lot a of money


derfmcdoogal

Luckily my "**Value Added** Reseller" is pretty good about it.


edmonton2001

Wash it with soap and water. Tell company you need a very expensive pressure washer if bucket doesnā€™t work.


cisco_bee

It still doesn't work but the CEO was really impressed with how shiny everything is now!


MyClevrUsername

CEOs are easily impressed by shiny things.


lesusisjord

that includes little tiny lights. The more that are illuminated, the better!


wimpunk

Especially by shiny dashboards.


cisco_bee

something something single pane of glass!


Breitsol_Victor

New data center opens, bishop comes thru to flick holy water on things - yikes.


KadahCoba

Install in the rack first so you can do the other servers at the same time. I've seen a video of exactly that before, so clearly it normal practice.


jjaAK3eG

I loathe having to wait for these "update managers" to finalize. It's best to unplug the machine way before the process ends to speed it up.


Tricky_Fun_4701

But it needs to be done exactly 12 minutes after the update starts. That way a significant about of programming has been written. This makes it easier for the machine to not boot.


cisco_bee

Turns out it was something the ~~Smart~~ Crummy Update Manager did. I was able to "manually" flash the firmware via the iLO and afterwards it booted.


jjaAK3eG

Glad you got it back up!


cisco_bee

Yeah, I waited too long to brick this one. I'll definitely do the next one faster.


cisco_bee

This is after running the "Smart Update Manager". Either it's not as smart as it thinks it is or I'm not. iLO works fine. There is a physical monitor connected and it never gets any video. No fans spin up. Nothing. edit: I did open a support case. I'm just hoping some other shittysysadmin can help more quickly.


Cocopower9

Open it up and void as many warranties as possible (Call hp)


cisco_bee

I had to open it already to powerwash it as suggested by another user. Thanks.


capn_doofwaffle

Akthully... I've seen some settings that wont allow a machine to boot if the cover is still showing "open". Doubt thats your case but might be worthwhile to make sure any buttons or sensors are plugged in and the cover is closed correctly. -shittyadmin out.


askaboutmy____

additional water cooling, nice


mitspieler99

> iLO works fine. There is a physical monitor connected That's why this is the correct sub. ILO should tell you the needfuls. Prepare to download some shitty file from ILO to upload it into a shitty HP website. Then wait. Upload again because it's not working. Then wait again and send the file with dropbox to the tech. Then wait again while said tech opens up a ticket BECAUSE THE SHITTY SYSTEM ISN'T WORKING. A few days later they might send you a replacement for whatever.


cisco_bee

Thank you for this valuable insight into my future. Can you tell if I'll find true love?


joefleisch

Wrong Reddit for this reply butā€¦ iLO should allow you to see the BIOS or IPMI logs. Failure to post could be incorrectly seated RAM or PCIe cards. They get knocked loose by the FedEx crew. Check videos on YouTube about how they handle boxes. I had a Cisco UCS arrive where I saw the guy throw the box out of the truck into the parking lot. He put a hole in the side of the box. I would have refused shipment if I did not need it that day. I made the guy wait while I booted the server and ran a diagnostic disk. When the guy protested I pointed to the camera by our entrance.


cisco_bee

It was a failed firmware update via Smart Update Manager. I "manually" flashed it via iLO and the server boots now.


mitspieler99

> Can you tell if I'll find true love? Not like this. However the eastern euro DELL accountants you get to know when you don't pay the bills are quite tenacious. Olga, if you're reading this. I like you.


gjpeters

It's time to retry the update manager to see if it works now.


cisco_bee

I did that. Except just to see WTF it was doing. It wanted to install system firmware from early 2023 which was at least one major revision behind. It wanted to *downgrade it*. I said no thanks and deleted the ISO.


edmonton2001

Lenovo tells me to do the same. I tell them that I canā€™t load an OS if the raid card is dead. And they reluctantly send me the part.


pretendinglikeimbusy

Does it post in the iLO? Done iLO updates? Heard a story the other day of someone's HPE array rebooting 12 times while going through iLO updates


cisco_bee

iLO is accessible and acts like everything is cool. I think it's in denial.


lysergic_tryptamino

Did you slam it with the palm of your hand? Next step would be to kick it. Usually resolves right away. Itā€™s like a vending machine.


RiceFieldsAreSafe

You forgot the elbow smash


gjpeters

If all else fails, try a rack shake


Skeesicks666

Rack shake is amateur hourā€¦Pros, like me, build their datacenter directly on a geological fault line!


edmonton2001

Take the drives out and blow on them. Itā€™s NES right?


CompletelyInadequate

I thought you installed RGB lights in that bitch


cisco_bee

homestar\_thesystemisdown.gif


randomquote4u

take the drives out. close your eyes and mix them about on a table. then slap them back in bays 1 2 3. repeat until it boots


cisco_bee

I'm completely blind and deaf. Can I skip step 2?


briantforce

In that case, wear winter gloves so that your heightened sense of touch doesnā€™t interfere with RNG.


cisco_bee

sfdjha oisu;h;hiufd hsfdaf;dlkasfd;jl uirhef SezsdXm,, vmcnesdZXCmujaSEZWDgnthcbesdZF


GlitteringAd9289

Honestly, to me, it looks like a short somewhere causing it to power down. I would try removing as many devices as you can and try again trial and error style. I would start with those drives. As another user put it elegantly, make sure to shuffle them all up and put them into random drive bays. Raid cards love this, but only after that power washing. EDIT: I don't have any hands-on experience with HP servers, only Dell, but is there an iDRAC alternative you can log into to check for system errors? EDIT: OP was able to fix it by using iLO to flash a fresh BIOS image, looks like the BIOS image was corrupted thus causing the instant power-down with no errors.


cisco_bee

Yes, it's called iLO (I assume that stands for Incredibly Lazy Operator). It is working fine. It shows no issues in the logs. p.s. Shout out to ChatGPT for the iLO joke. I really am incredibly lazy.


GlitteringAd9289

Thats a great joke, why even leave a chair when you can press the power button remotely! Does the iLO have some sort of diagnostic sequence you can run? It may at least give more information


cisco_bee

Tried it. Just like the goggles, zey do nothing!


cisco_bee

I removed the drives and it does the same thing. I was going to shuffle them, but I accidentally dropped them in a bucket of water we keep in the server room for the leaky roof.


GlitteringAd9289

Ahaha. Wondering if it's a borked BIOS? Saw the other comment where you mentioned the smart update manager may have not been so smart in this case. Is there an option in iLO to flash a new BIOS image?


cisco_bee

>Is there an option in iLO to flash a new BIOS image? FYI this was the fix. You win a cookie! I ran Smart Update Manager again after I fixed it just to SEE what it was doing (without committing the changes). It was trying to "update" the firmware to a version that was over a year old and a major revision behind.


GlitteringAd9289

Yay! I'm not sure if you can, but you may want to edit the original post with this fix in case there is that one person scrambling to try and fix the same issue.


FilthyStatist1991

If no post, you canā€™t access iDRAC. I think he either installed bad firmware or the memory needs to be reseated. Or a short on a PCI bus somewhere


cisco_bee

Incorrect, in this case at least. iLO works fine.


FilthyStatist1991

So it does post?


cisco_bee

It did not POST at the time of this post (ha). However, It was a botched firmware update by Smart Update Manager. I installed firmware "manually" via iLO and everything works now.


GlitteringAd9289

This is incorrect, the whole point of iDRAC or iLO is to access the server even if it's powered off or having issues remotely, if you couldn't do this while it was powered off, it would sort of defeat the purpose. It's a dedicated computer to control a larger computer in Leymens terms.


FilthyStatist1991

Yeah, I get it with like a network KVM. But last I knew, your Dell server had to get past post before it could use iDRAC. Itā€™s intended for getting to pre-boot settings. When itā€™s powered off, how would its internal hardware function without power. IDRAC NIC card is on the server itself, not like a network KVM.


GlitteringAd9289

I know for sure our 3 poweredge servers with iDRAC 9 enterprise are accessible as long as at least 1 PSU has power, even if the server is completely powered off. iDRAC actually has the capabilities to work even without a CPU or any RAM present, as it has its own dedicated hardware. https://www.dell.com/support/kbdoc/en-us/000179517/dell-poweredge-how-to-configure-the-idrac-system-management-options-on-servers#:\~:text=The%20iDRAC%20is%20a%20piece,the%20server%20is%20turned%20off.


cisco_bee

Think about Wake-on-LAN. Just because the server is "off" doesn't mean it's not powered.


Newbosterone

Have you tried turning it off and on again? Have you tricked a user into touching it, so you can blame them? Try putting a sign on it that says, ā€œDo Not Touchā€.


cisco_bee

>Have you tried turning it off and on again? You think this is my first rodeo? >Have you tricked a user into touching it, so you can blame them?Ā  Holy shit, how many rodeos have you been to?!


Newbosterone

Iā€™m not as old as dirt, but when I was born it was still a new idea.


ExpressDevelopment41

Have you tried threats? If you power it on with a hammer in hand, it boots 90% of the time.


cisco_bee

It worked!


machacker89

it should. it knows you'd struck 6he fear of god(s) into it. lol


avowed

Ain't got no gas in it.


Maxplode

Take disks out. Go home. But seriously, have you tried taking the disks out and booting it up? Get a monitor on it??


cisco_bee

There is a monitor on it. It does nothing. I tried taking all 5 drives out (The OS is on NVME drives not pictured). Same result. Just lots of blinkyflashy. Never any video. Never any fan noise. No beeps.


Maxplode

Just noticed that the first 2 disks on the left don't flash green like the others. Tried reseating the RAM or swapping them over?


cisco_bee

It was a botched firmware update by Smart Update Manager. I installed firmware "manually" via iLO and everything works now.


edmonton2001

Blow on the disks and rub you finger along that connection piece


andovinci

Does it blink? Yes? Good. No? Good


thepfy1

The iLO should have any errors and possibly diagnostics.


cisco_bee

iLO is in hard denial. [https://imgur.com/HxFBYEX](https://imgur.com/HxFBYEX)


Whyd0Iboth3r

The top one has a ! on it. What does that say? Does it give you any info when you mouse over it?


cisco_bee

No, it has an (i). It just means informational. Not good or bad, man, it like, just is.


Whyd0Iboth3r

Well, I tried. Good luck.


cisco_bee

I was able to "manually" flash the firmware via the iLO and afterwards it booted. So it was definitely something the ~~Smart~~ Crummy Update Manager did.


mrhorse77

those SMART managers are either amazing, or garbage, no in between. varies with models of course... most likely it failed applying the firmware update, or got disrupted and then that killed the boot process. id just avoid using it at this point, or download a new one and try that instead... I also typically only put my boot drives in at first when running the SMART crap. anything complicated makes it become a STUPID manager


Chiller252

Try taking it down to minimum config and see if it boots with proc 1 and dimm 1. If not, your board might be toast. Also, see if there's any nonsense in the iLO log. Then proceed to hang the mobo on your wall as a decorative piece


Absolute_Peril

Isn't being broke normal for HP?


Practical-Alarm1763

Nah, just unplug everything, make sure the cases are closed, take out all the drives, plug them back in, then blow into the USB ports. After that, say a prayer in spanish, plug the power back in, then turn it on. If it doesn't work, rinse and repeat 3 times. Then call Eightch Pee.


photosofmycatmandog

Holy crap I thought this sub was satire.


cisco_bee

A real shitty sysadmin can find the truth in any satire.


mikmeh

RGB is out


NSA_Chatbot

Sounds like one or more of your fans got knocked loose in shipping. There's an undocumented feature in some servers that if one fan fails, the others go to the noisiest profile the company could figure out. They've found that when that happens, every competent sys admin immediately investigates.


HITACHIMAGICWANDS

The first two drives didnā€™t blink, Iā€™d take the other ones out, probably do less blink then at least


cisco_bee

Thanks. Tried taking all 5 out. It does the same thing. (Note that there are NVME drives in the back for the sys OS)


HITACHIMAGICWANDS

Probably wipe those next


HITACHIMAGICWANDS

Theyā€™ll fix it on the RMA


biggt620

Try taking the system down to minimum config and see if it will boot. From there, add 1 part a time and test to see if you can isolate the problem. Also check for loose cables on the inside.


cisco_bee

Right now everything is unplugged except power. Removed the 5 drives seen. It still does the same thing.


biggt620

Do you have any pcie cards installed or risers? Remove all dimms except for 1 or 2 of them, drop it down to 1 processor as well. Then test.


cisco_bee

Fuck that sounds like actual work :( Also keep in mind it was working before running the Smart Update Manager, which they totally should have called the Crummy Update Manager.


Newbosterone

We need a bot for this sub. After you respond to a post, it asks, ā€œAre you sure this comment is shitty enough? It appears to contain useful informationā€.


cisco_bee

Nah, I like it the way it is because you *never really know* if it's useful. Especially if you're truly a shitty admin, like me.


eldudelio

Well its an hp so...


cisco_bee

I know I just wanted to put a macbook air in their but my boss said no :(


thatguywhoreddit

Did you try matching rgb ram?


perlapr

I had a similar problem with dl380 gen11. The problem was the faulty native network card.


cisco_bee

hmmm, you might be on to something... [https://imgur.com/Jxr2Lk7](https://imgur.com/Jxr2Lk7) It's the only bad thing I find in the logs. Pretty insane if this keeps it from even POSTing...


perlapr

My server suddenly shut down for this problem after 5 hours of life. You can remove the network card and try again.


Platocalist

judging from the sound the other one is working fine, the users probably won't notice so just go take the rest of the day off. You have earned it.


Superb_Raccoon

Have you tried unplugging it, waiting 30 seconds, and plugging it back in?


cisco_bee

I had tried that only about 10 times. It turns out it was a botched firmware update by Smart Update Manager. I installed firmware "manually" via iLO and everything works now.


ArtisticVisual

Sometimes I intentionally break my servers because I donā€™t like green


cisco_bee

Amber is the color of your energy. Whooa-oa.


erinxcv

I recommend you run a defrag job. If itā€™s RAID, your data is striped across each drive and you want all the data to be re-organized so that it runs more efficiently. The power supply causes issues too if itā€™s not working so before you shut it down, make sure you replace both PSUā€™s at the same time. Do not turn it off first - shutdown should always be the last step. Run a second defrag afterwards just to be safe.


Pbart5195

Reseat the memory. Make sure itā€™s in there good when you do. I always use my memory reseating tool. The maintenance guy keeps calling it a ball peen for some reason.


xandaar337

Try pressing the "any" key the kids keep talking about.


Skeesicks666

This is why I buy my servers on ebay, 1/3 the price for at least 50% working, so it is a net-winā€¦and the non working ones get resold as ā€žuntestedā€œ for a little money on the side!


deanm11345

This is going to sound stupid but give it a nice little smack on top as itā€™s booting. No lol but more to the point, check all your riser cards are seated properly. I had Dell that would do this, though I wouldnā€™t put it past HP, that the *proprietary* PCIe riser card *had* to be installed for it to boot. However it never quite sat right so I had to give it a little smack on boot.


WhiskeyBeforeSunset

The problem ia because it has HP stickers. Slap an apple sticker on there and it'll just work.


Moist-Chip3793

It is fixable, but you need to sacrifice a rooster! Then you have to thread the fine line of giving it just enough fresh blood to work, without it shorting out, but in my experience it helps immensely, if you have a good firm grasp around the neck of the rooster, when you shuffle it off its mortal coil! Also, a good pair of scissors is better than an axe, if you cut yourself you will have to start over, since mixing the blood spoils the process!


Melodic-Matter4685

From what I recall, a lot of hp server stuff won't work without warranty contract. But... I haven't ever really used hp hardware at home.... so I'm probably wrong.


That_Redditor_Smell

New? Lol


cryan7755

Man, this reminded me why I started wearing sound canceling ear buds in the DC. Also, i'd be monitoring ilo during power up to see what's up.


kebobs22

I'm a HUGE dip switch enjoyer when hpe crap doesn't wanna work


flakey_nob_cheese

No help but that 380 is identical looking to the ones I installed at airbusā€¦. In 2004! Cool story bro I know


rayruest

Ah, good old HP. I remember a time I had a new HP that wouldnt boot. Opened it up and the cables weren't plugged into the drive chassis, raid battery, etc. Its like someone gave up halfway through assembly. That and all of the DOA laptops we were getting from them, we switch to Dell servers and Lenovo laptops. Havent had a problem since.


jeannaukhc

Vielleicht hilft da ein bisschen Lametta! (Maybe some tinsel could do the job!) -Loriot


cashew76

Updated firmware: ok now 100% confidence. Production ready. /s


Cheapbubucko

You sure it isn't the ram configuration


fireduck

Can you reach the BMC? What does it say?


CTL-ALT

Open up the chassis, poor water all over the circuit board turn it back on let run. Turn off, dry the circuit board. RMA.


New-Incident267

Rofl. You think this is shifty admin. I watched a Senior Technical implement MPIO in 1 year. It's just funny.


No-Drink2529

This is why we have backups. I use VMware and Veeam and can do instant recoveries in minutes.


Next_Information_933

Just call your vendor or hpe directly.. It's not a reddit issue


cisco_bee

Go back to r/sysadmin. The rest of us are having fun.