Postmortem September 17th outage and rollback

[Buoyant Armiger]

Spooky. Thank you for doing the needful.

 

[Alaric the Visigoth]

@Null If you're at liberty to say which raid where you using for your server?

 

[Onni Kalsarikännit]

Thanks for sacrificing your Sunday so we can say nigger and faggot, Null. You are my favorite niggerfaggot


It hasn't been adressed in this thread so far but tor is very broken.
See here (.st link)

Edit: posting and quoting works fine on tor again. all stickers also work again. Thx, Jersh!

 

[Pill Cosby]

Here's me right now thinking Liz-Fong-Jones who is a self admitted rapist; who also admitted to scum journalists being part of a collective who have committed to crimes to prevent a lawful US business from operating using tactics which are (I believe) a federal crime.

 

[Pedophobe]

Kiwi Farms going down is like 9/11 for autistic people.

All those shitposts lost forever... like sneeds in the rain...

 

[Nisse Hult]

Nobody is better at taking down Kiwi Farms than Kiwi Farms.

 

[Dollar Store Sentai]

If four SSDs fail at the exact same time I would suspect that it's a problem with the firmware, controller, backplane or software.

I had two HDD's die within days of each other because they were the same batch, bought and installed at the same time. It is more common than you think.

 

[Null]

chat up

 

[Nottafed]

Assuming its a pci-e x8 nvme controller, that likely shit the bed, which would cause all 4 to show dead simultaneously. Sucks because it's not something people just have laying around.

 

[Kendall Motor Oil]

Now someone needs to buy him Optane sticks when they flood the used market.

 

[keytar]

I CAN sneed

 

[Moths]

Throw a coin to your slobbermutt
Great to see that your hardware always seems to have a lifetime of a few years

 

[Don Yagon]

Can't post normally since the post form doesn't load because of borked CSP. Had to use an addon that temporarily disables CSP just to post.




Upd: seems to be fixed now.

 

[Patrick Bait-man]

9/17
Never forget the day the quadruple drives failed and the posts that died.

 

[potatoman]

All of the drives at once is bad luck. Glad Null wasn’t so retarded he didn’t do backups.

 

[Lake]

Ayyyy the quick fix was an unexpected surprise. Thanks, Jersh.

 

[WelperHelper99]

Nobody is better at taking down Kiwi Farms than Kiwi Farms.

The only thing that can kill the farms is itself

 

[IT Dude]

Made an account to clarify things. I'm the dude that helps Null with this stuff when he needs it. No, I don't monitor the server cause it's Null's. But maybe I'll setup some stuff for Null to monitor stuff including drive health.

The storage on the server use ZFS pools. The SATA SSD array (SNEED Pool) bypasses the RAID Controller and is entirely JBOD passthrough.
The 4 x NVMe drives are U.2 drives in the front and are in FEED ZFS Pool. The backplane handles SATA, SAS, and U.2. U.2 has it's own area for those drives and connects to the motherboard with a OCuLink cable to a JNVMe header.
The 4 x 1.6TB WD Ultrastar DC SN620 NVMe U.2 drives disappeared from the server last night. But before that, the kernel reported write errors to one of them.

There are a few reasons this could have happened, from most likely to less likely:
- BIOS/UEFI Firmware stopped communicating with the NVMe drives. This happened with a certain BIOS setting when it was initially setup
- The drives actually died from the workload. Unlikely considering these can handle 1.7 Drive Writes per day. But very feasible. These are 2nd hand enterprise drives
- The backplane/JNVMe headers exploded. Super unlikely

The drives are likely still alive, and the server's firmware probably took a shit.
We need to inspect the server's BIOS settings or possibly even update the firmware. Then we can determine if the drives are toast or useless.
There was no foul play at hand here. At best a firmware bug, at worse, the drives sudoku'd.

 

[Toolbox]

Can't post normally since the post form doesn't load because of borked CSP. Had to use an addon that temporarily disables CSP just to post.
Ver archivo adjunto 5343500
Ver archivo adjunto 5343503
Ver archivo adjunto 5343501

That brokeness appears to be affecting the onion site and only the onion site. Tried to login multiple times there before hopping on .st, everything's good on the clear web version.

 

[Quandaries Sage]

Ty dear feeder for your hard work
We appreciate it