I’ve got a riddle for you. What’s the best way to move 1PB if data from one state to another over a 1gbps MPLS link? I was thinking ZFS send with compression and deduplication, but really I just need anything that reduce the bits on wire as much as possible. Any ideas?



> I’ve got a riddle for you. What’s the best way to move 1PB if data from one state to another

sneekernet, en.wikipedia.org/wiki/Sneakern /a SSD sent via FedEx will honestly probably be the fastest.

If you really need to send it over the wire, then yeah, zfs send seems as good as anything else I can think of.

But I'll boost is case anyone else has brilliant insights.

@codesections @frostygoth Or several boxes of SSDs, but yeah. By my calculations, it will take 3 months to transfer 1 PB over a 1 gbps connection, _assuming_ you can maintain a perfect connection the whole time.

I have a lot of experience with shipping Pelican cases full of hard drives just because FedEx gives you way better bandwidth than most Internet connections.

@codesections @frostygoth Also it depends on your data.

Mostly text? Compression will help a lot.

Images? Compression won't help you much at all. You might be able to shave a couple days off the months of transfer time.

I could be wrong though, IT isn't what I do full time.

@codesections Thanks! I figured shipping and then syncing was the best way to do it, but my buddy says it’s got to be done over the wire and uncompressed it’s gonna take like 93 days to happen.

@codesections @frostygoth I was hoping someone suggested sneakernet.

It's by far the best way for huge amount of data.

@codesections @frostygoth

Fun fact: one of the ways Amazon provides for putting your data into AWS is to send around a semi truck full of hard drives and related servers.

(I don't know whether they'll provide it for getting your data out again, though.)

@suetanvil @codesections @frostygoth same with Backblaze, and I assume it's the same with Azure, Google Cloud and similar parties.

Sign in to participate in the conversation

Fosstodon is an English speaking Mastodon instance that is open to anyone who is interested in technology; particularly free & open source software.