T O P

  • By -

-Archivist

Stickied. Let's talk biggest deletion vs accidental loss. --------- I'll go first as it (deletion) was this week, I freed up 720TB (those who've known me awhile, it was one of those WD black boxes) of content that had been sitting around for a few years, after confirming everything had found new _forever_ homes be it with other hoarders or institutions I consider the data _safe_ and not something I need to keep around, so I get that feeling, it was very nice to gain space on my network without spending monies!!


TheOtherPete

I do enjoy a clean-up session using a graphic file manager like SpaceMonger where all the directories/files (and free space) are represented by blocks whose size is proportionate to their disk size. Then as you delete things you watch your free space block get bigger and bigger.


TheAJGman

`ncdu` here, but it's equally satisfying. Though with ZFS snapshots the gratification is delay a week or so depending on the dataset.


Immortalbob

Reminds me of diskpie back in the day. Drive space pie chart


s_i_m_s

> SpaceMonger WizTree here still prefer WinDirStat but really I can find and delete the offending file(s) in WizTree and go on to something else before WinDirStat has ever finished scanning. Wiztree works like Everything file search and cheats by reading the NTFS MFT rather than spending the next hour crawling the file system like normal software.


SumoSect

Interesting to learn about wiztree. WinDirStat has always been my go to but fuck. Turn it on and make some coffee and maybe it's done. Probably not.


Weary-Fix-9152

Loving the punctuation here.


SumoSect

Just how I talk unfortunately haha.


Weary-Fix-9152

No disrespect meant. Just had to crack up like an idiot about your "go to but fuck". You'd think the 12 year old in you would go away by 40. Guess not lol


SumoSect

Hahaha. Ya you're right. I didn't take it as disrespect. Sometimes it's just the things you know work so you're ok with its' faults.


svidrod

I love opening up spacemonger and going woah I accidentally downloaded 200 tb of different versions of the same movie


Weary-Fix-9152

Well that statement just sold me on checking it out. I found the other day I have a 2 or 3 copies of the same films in different qualities. Forgot to delete the lesser quality ones when I got the 4k lol. Thank you. I'm looking forward to this.


Signal-Ad-8333

Try wizztree. You like blocks? It'll give you colorful blocks


ymgve

bro NSFW this post


zik

You don't belong here. Turn in your badge and gun and show yourself out.


Snoo44080

Wait, you weren't supposed to have a gun, where'd you get that.


CivBEWasPrettyBad

Downloaded that 3D printed gun blueprint. Don't worry, it didn't get deleted.


ryfromoz

It's to go with my 3d printed car!


Inode1

Oh shit, someone downloaded a car after all.


TotallyNotADoggy

I downloaded an entire bear ^^^^genome ^^^^sequence once!


ChairAmazing5422

Would you download a car? Yes, yes I would


That_Acanthisitta305

I got human DNA, for compression contest, now I dunno where is it, havent delete it yet, might be useful to aliens someday if they found my HDD and we are extinct.


TotallyNotADoggy

Human DNA contains really important data. It's good to know that someone on this planet has a backup in case we need it!


ryfromoz

Id love to sequence and have a copy of my dna


Captain_Starkiller

You guys have guns and badges?


metalwolf112002

I have a gun and a badger. Works just as well. Nobody argues once they see you have a honey badger on a leash.


Captain_Starkiller

Legit.


ryfromoz

They just want to badger you too much and risk disease


Due_Bass7191

Wait, you all get guns?


Lords_of_Lands

Are your PC fans not driven by combustion engines? Cars are too big to retrofit into a PC case so you have to hookup automatic guns as mini engines to keep those fans spinning. Electricity is too unreliable. You don't want to risk your hard drives overheating during an extended power outage. If there's anything I learned on this subreddit it's that heat kills. Controlling your heat is the most important thing you can do.


Due_Bass7191

I use aqueducts to liquid cool my rig


-Aquatically-

Hard drive and NAS.


Grouchy-Structure888

I really hope that you were referencing a car and a gun there bc that's just witty af to me if so :)


-Aquatically-

I was


ByteArchivist

I understand chief, can I still keep my pension for all of my good years of work on the force?


DeepIndigoSky

Get out before I make you restore ALL THE DATA you got rid of with nothing but punch cards


ByteArchivist

You wont take me alive!


darktalos25

Wait, I'd like my badge, never got it. I'd take another gun too.


drfusterenstein

Mr. Treehorn downloads a lot of data in this town. You don't download shit, Lebowski


LINUXisobsolete

absolute data curator CHAD. I bet you give it metadata too.


ByteArchivist

Is it even really data if it does not have metadata?


_kazza

New job just dropped : Data Philosopher


H9419

In my faculty the subject is called information theory


ThatBlokeYouKnow

Has anyone sent a reddit care thing to OP yet?


ByteArchivist

Yes they have!


blackmine57

How can I do that?


ThatBlokeYouKnow

I dunno, saw a lot of people getting them the other day


blackmine57

Used to get a lot of them too


runningblind77

My favorite way to see free space go up is `sudo btrfs device add`. To each their own.


ByteArchivist

The true path of enlightenment. A path I can never tread due to my painful affliction of "just not really wanting to spend that much money"-itus. A terrible incurable curse of mine.


runningblind77

Refurbished enterprise drives are pretty cheap. I recently picked up a 12TB HGST (w/ 25,000 power on hours) and a 14TB Toshiba (w/ 3000 power on hours) for like $100 each shipped. Both combined were cheaper than the brand new 12TB Seagate I bought last year.


Zatchillac

I just got a refurbished Exos 16TB for $130 (as well as a 5 year warranty) and it showed 0 hours. I ran the extended SMART test in HDD Scan and it claimed no issues


runningblind77

Yeah, I ran a long S.M.A.R.T. test on each drive as well as an [alternative](https://wiki.archlinux.org/title/badblocks#Alternatives) bad blocks test before adding mine to the pool.


christophocles

oh look at the high roller here, with your $100 hard drives. I just bought a set of ten 8tb drives for about tree fiddy. Seriously, $35 apiece, $4.375 per TB. They passed the SMART long test, that's good enough for me! 11-wide raidz3, let's go https://ibb.co/qpfTLvb


runningblind77

Damn, where'd you find that deal?


christophocles

Look up e-waste disposal wholesalers. You could find even better deals than this at auctions.


AbjectKorencek

*cries in european prices*


SakuraKira1337

I buy mainly 20TB Toshibas new for 300€ or less. I think prices are okay here


christophocles

$16 per TB is pretty damn steep, man. You don't have ebay in europe?


SakuraKira1337

I won’t buy used. Had a lot of mixed results with that


christophocles

For 25-50% of the cost of new, I'm willing to take the risk. I only use enterprise grade SAS disks, with raidz2 or z3. I haven't had any of them fail yet.


SakuraKira1337

Okay it’s rare to find ones for 50% of the cost. 25% I have never seen. So maybe used market is more expensive


AbjectKorencek

Unethical pro tip buy used drives, run data recovery software on them, find something valuable on it.


ByteArchivist

Those are not terrible prices. I always have to buy pairs of same-sized drives any time I buy a new one, and I try to make sure they are not from the same manufacturing batch.


Pmur0479

Where do you get these reliably refurbished?


runningblind77

I got mine from [serverpartdeals](https://www.ebay.ca/str/serverpartdeals)


Pmur0479

Thanks for the info


Lords_of_Lands

Sure each individual, previous generation drive isn't much but how are you storing everything? There's only so much space in a PC case which requires buying higher capacity drives when full. In triplicate those newer drives costs far more. The racks I've seen take up far more physical space and seem expensive with non-consumer components for everything. I don't know where I'd put one. Setup in the basement and mount as a networked share?


runningblind77

My case has room for 7 3.5" drives as well as 3 5.25" bays that I could add adapters to which ought to be plenty for me for the foreseeable future. With btrfs I can swap out one drive at a time and the drives don't need to be the same size. I currently have 3 4TB drives, 2 12TB drives and a 14TB drive giving me about 24 TiB usable with every byte mirrored, and my case still has room for an additional drive without any adaptors or convertors.


Lords_of_Lands

Thanks.


pongo1231

Mine is compress-force=zstd:15, `sudo btrfs fi defrag -czstd -r` followed by `sudo duperemove -dhr --dedupe-options=same,partial -b4k`. Only takes half a century to complete usually.


yourzero

> btrfs device add "In order to subtract, you must add."


AbjectKorencek

Yes, yes, yes. That's the way. (except I never bothered with the sudo stuff when linux was my primary os, I just used su before doing anything that needed it)


heff66

Unpopular opinion: "Curated" data hoards are often more useful than "Complete" data hoards. (Put down the pitchforks and torches, people!)


ByteArchivist

The difference between hoarding and archiving is labeling and organizing things.


heff66

You make a good point. That's a hair worth splitting. The first two definitions of "hoard" found in the Cambridge Dictionary could certainly apply to data hoarders. Hoard (verb): to collect large amounts of something and keep it for yourself, often in a secret place. Hoard (verb): to keep a large number of things that are not needed or have no value, because you are suffering from a mental condition. Dang! Feeling called out on that one. Thanks word nerds!


tinnitushaver_69421

It hurts to delete (for most), but the less data there is the longer it's gonna last and the cheaper it is to make it last.


ByteArchivist

Deleting the first byte is always the hardest.


purgedreality

I hear you brother, curated hoarding is still hoarding! At least you're not one of these lossy2lossy re-encoders! That would be embarrassing.


Puzzled-Ad-3504

I used to enjoy doing that, while also altering the fps and anything I could just to see how it would turn out. (This was like 15 year ago, when windows media encoder was amazing as dealing with corrupted data) But do that for archiving stuff 😳 I would never.


ByteArchivist

I could never! Except for all the times that I have done that and will do that again in the future. Sometimes I don't need my 30 minute cell phone quality video to be 128GB. Or my rasterized SVG to be 400MB.


Dull_Wasabi_5610

Daring today, arent we op?


SomeoneHereIsMissing

If I want to delete things, I put them in the "cemetery", it's a RAID 5 array of very old drives on my secondary/testing NAS.


ByteArchivist

It's like being on death row and not knowing your execution date


TheG00GLE

[https://i.imgur.com/z5xe73L.jpeg](https://i.imgur.com/z5xe73L.jpeg)


itsbentheboy

Going through a purge myself. Early in my collecting i was disorganized, and careless. I am now ensuring that everything is sorted, organized, named properly, and of usable quality. All my Media is going to formats like h264 for direct-play on all my connected devices. All text files are going to plaintext or markdown formats to be application agnostic. Document scans are going to PDF for OCR scanning. Certain things saved for interests / hobbies I no longer have are getting pruned. I had so many duplicates, re-downloads, or trash content.... Its spring cleaning time in the data hoard.


sonicrings4

Maybe if you didn't delete things, you'd have records of who asked


mineturte83

😂😂


BuonaparteII

An alternative to deleting: encode to AV1, AVIF, and Opus. It will probably be more blurry but at least you'll have it rather than not... Imagemagick `convert` is quite a bit better than `ffmpeg` for converting images to AVIF and it works with transparent images too. Here's a [script](https://github.com/chapmanjacobd/library/blob/main/xklb/mediafiles/process_image.py) for batch resizing/converting. (nb. it will delete source images unless they are smaller than the AVIF output)


palebd

Opus isn't that bad, though. I mean sure it's not flac or ogg, but it's better than mp3 and m4a.


BuonaparteII

Of course! I'm not throwing shade at AV1 or Opus... they are state-of-the-art codecs for compression. That being said I don't think AV1, Opus, and AVIF are good "archival"-quality formats as long as other more "quality balanced" codecs exist--if you care about quality, spend the GBs and use x264, FLAC, and TIFF; but AV1, Opus, and AVIF are valid alternatives to deleting


AbjectKorencek

Or if you don't want to wait that long for it to transcode (although recent av1 encoders have gotten pretty fast) at least remove the unnecessary audio/subtitle tracks which is much faster and can still reduce file size considerably. There's 4k hdr hevc rips (usually between 20 and 30GB) that come with like 3 English audio tracks, dolby atmos, dts and ac3 + tracks in other languages. If you care about audio quality (I don't) keep just the dolby atmos, if you don't care about audio quality keep just the ac3 English track (or whatever language you want/the movie was shot in). And the same rip is now more like 15-20GB (I swear dolby atmos is a huge waste of space as far as I'm concerned). Since there's no transcoding involved it can be done pretty fast (idk, a minute or few minutes depending on your drives), the space saved can be considerable and the video quality will remain the same.


Lumyrn

is avif better than webp? I usually convert all my images to webp and they become one tenth of the original size.


BuonaparteII

Similar to x265 vs AV1: it depends on your encoding settings and the specific type of scenes (action, dark scenes, anime, etc); but yes, on average I would say AVIF is better than WebP in terms of filesize and quality. But I've only tested around ten different types of scenes. Sometimes it was a wash but other times AVIF looked better and it was 20% smaller than WebP AVIF is related to the AV1 codec: https://en.wikipedia.org/wiki/AVIF


Lumyrn

guess I will have to try it myself and see then, thanks for the info!


RuujiHasegawa

Man, I've been on that Opus train the past couple of months and I can't stop myself. You're telling me I can practically double or even triple my music collection by converting it to opus from flac, of course I'm gonna do that. No amount of "storage is cheap" arguments are gonna stop me from this path.


BuonaparteII

You could say the same thing about AV1. It has no right to look as good as it does at 500 kb/s. It's INSANE. But again, you'll get more quality "bang for your buck" with x264. Opus and AV1 create _TONS_ of artifacts but they are audibly/visually pleasant enough that they are difficult to notice unless you are EQ-ing to hell and back or using a LUT or xcalib, etc


TSPhoenix

Are you saying you get *more* artifacts at the same bitrate on the newer codecs? Or are you strictly talking about lossy-to-lossy?


BuonaparteII

> at the same bitrate The artifacts are _different_ from what x264 might produce at the same bitrate, which is why people find newer codecs more acceptable for lower bitrates. But for higher bitrates, yes, I would say that AV1 are not designed to capture the same quality profile as something like H.264 and AV1 would have more noticeable artifacts at the same high bitrates unless you disable [some of the advanced encoding features](https://gitlab.com/AOMediaCodec/SVT-AV1/-/blob/master/Docs/Parameters.md#rate-control-options) to make the algorithm behave a bit more like H.264. AV1, H.264, (and Opus) are all very versatile codecs (at least in theory--it also depends on the specific implementation), but you would want to disable some of the more aggressive [filters](https://hacks.mozilla.org/2018/06/av1-next-generation-video-the-constrained-directional-enhancement-filter/) to have comparable artifacts as H.264 but at that point you may as well use H.264.


creamyhorror

> An alternative to deleting: encode to AV1, AVIF, and Opus. Transcoding to lossy formats degrades quality significantly (plus takes time and energy). The user should have encoded to smaller files to begin with so that only one lossy transcoding step occurred.


HiT3Kvoyivoda

This made me chuckle so loud


Difficult-Gas870

I can relate, hoarding data that I care about is fun, but deleting data that I don't care about is freeing.


Puzzled-Ad-3504

You can get the same satisfaction from using different compression methods. I found a program I will try soon that finds the best compression possible (which may take awhile). Honestly though I only do that if I notice I will run out of space soon ahd still have enough space to make large archives. I have so much stuff sometimes cause anytime a hard drive goes out, I recover the data off it twice to make sure I get almost everything. I would prefer just using RAID, but thats a future plan.


TSPhoenix

> I found a program I will try soon that finds the best compression possible (which may take awhile). And you're not gonna tell us what it is called?!


Puzzled-Ad-3504

I will look it up later when I'm on computer tonight. I didn't name it because I couldn't remember the name🤣 I found it randomly and was like oh I want that. But haven't had time to fiddle around with it yet


Enough_Swordfish_898

You what! I think I'm going to be sick.... Lol keep only what you want, this is the way.


-my_dude

Evil


GeoUsername69

if you delete HDCAMs that you already have 5 copies of you will go to hell when you die


ByteArchivist

The classic 5-10-20 backup scheme must be upheld by all lest thee suffer the wrath of the AllDrive.


GavinBelsonHooliCEO

No shame here. I've been doing a data cleanup to make space for some new files, and going from 500 gigabytes free to 3.8Tb, feels amazing. I'm going to be able to avoid buying another 2 drives for several months to a year Watching that bar next to the drive go from Red to Blue? It's the best.


potato_and_nutella

I tried compactgui on some of my games and it was very satisfying seeing them shrinking


AstroPHX

We are the same… but different. I loved connecting my Synology to OneDrive and removing everything from OD. No longer getting warnings that I was low on space.


PeteTheKid

I’ve got a tiny collection mainly because I’m avoiding biting the bullet and going for a bigger nas, but the thing I like the most is optimising my content eg moving from h264 to h265. Feels like free hard drive space… before anyone mentions yes I know it’s bad to compress already compressed stuff, still looks fine to me…


vfoster

I envy you. If only I were that bold!


Bob4Not

As long as it’s only stuff on your own archive!


tamal4444

Don't come near my hard disk or it will be the last thing you will ever delete.


PacoTaco321

I get you. There's only so much space I personally have, why use it for something I don't care about at all?


jaegan438

Reading that made my skin crawl. Good luck.


miked999b

I had to go a make a coffee, so I could spit it out 😂 720TB 😂 I'm at just under 100TB and I'm thinking I might finally start deleting stuff. So many shows I'm realistically never gonna watch that are taking up tons of space. Or maybe keep the first season, and then I can download the rest if I need it. That sounds far too sensible though, so I probably won't.


Hug_The_NSA

Insanely based post. I had been saving so many movies and stuff that I realized I was never gonna watch again and did a big purge recently too. Not everything needs to be hoarded forever.


Rapportus

A former coworker of mine dropped a pallet of Supermicro drive arrays that he gravity racked on his truck to transfer it between data centers. Deleted something like 800TB usable (this was over 10 years ago), and 1 Sysadmin.


SkywayTechReviews

Sometimes we must delete in order to hoard!


unoriginalpackaging

I have 4.15TB in a temp folder I can’t bring myself to delete until I verify I don’t need it.


ENCORE-Archives

Don't worry, I just like having stuff that is only important to me in my own personal drive, and seeing my space free up is enjoyable to see.


BlossomingPsyche

all good, I have 2-3 copies of everything too. sometimes 2-3 copies oh each different resolution! lol it’s a problem. setting up open media vault today 


TheOneActivehenry

Oh dear. When I clear up space on my PS4 or on my other drives, I make sure it still exists, by copying the videos or things to another drive, and then deleting them from the source drive I copied from. I'm currently in the process of copying over and dumping my PS4 videos onto my Google Drive before I delete them off my PS4 Pro. Rinse and repeat.


cmaxwe

Fuck your couch.


Surfdog2003

Why would someone take the time to post this?


DedZodiak

first time on reddit?


Surfdog2003

Second! Seriously I do wonder why people post things like this. Attention?


LongGoneForgotten

To start a conversation related to the thing they posted, hear about others' experiences and opinions.


darktalos25

I once convinced an annoying kid on a Linux forum that rm / was the best way to ask us less dumb questions and he used it on his data storage. He was pretty mad.