Copying 10 Million Thumbs, Fuck!
 

Here's an unexpected mind fuck... 10 million thumbs, 500 gigs, from one drive to another on the same server... estimated time, 4 fucking days!

would have been much quicker to zip them then transfer the zip.

i tried that first, copied all in just a few hours, but then needed 9 days to extract.. so i deleted the gzip and started over with copy

Yep - You can set with no compression.

my second attempt has been running 13 hours now.. no compression sounds interesting, but not sure if the performance gain will be worth starting over now.. with compression was 9 days to extract.. without, maybe half? which would be the same as copy then...

WTF, one of your thumbs is as big as 50MB? Normally a thumb is below 10kB.

160px × 120px, average 15 KB each.. in this case it's not the size, it's the number.

My calculation: 500GB/10M=50MB

That would be 500TBs - I think.....

I'm sure you're in unix, but on the off chance, I'll leave this here

http://codesector.com/teracopy

(500*10^9)/(10*10^6)=(500*10^3)/(10)=(50*10^3)/(1)=50kb

i'm guesstimating... roughly 500k sets of 20 thumbs each, filled a 500gb drive 92%.. some thumbs are 10kb, some are 35kb.. and roughly 10-million of them..

edit.. was off by 172,000 set.. total is 672k sets for 20

windows or unix?

unix.. freebsd 8

Would try this one also..

You have several options to speed this by an order of magnitude:

Option 1:
If both HDs are the same size and from the same company and are the same model... I mean, both HDs are the same, then you have another option that will be considerably faster.

We use dd here instead of copying.

dd bs=10240k if=/dev/original of=/dev/copy


Option 2:
If on Linux, another option is to remount the original and the target partition with noatime option and then use the usual copy or rsync as normal. This will speed things alot when copying lots of small files. In this case rsync is better, even that it will be a bit slower than cp. The advantage of rsync when copying that much info is visible if the target HD has a sector error and copy process does not work properly. Rsync will warn you while cp will not.


Option 3:
Setup a RAID1 software raid between both HDs and it will automatically mirror the information. It will do practically the same as in option 1, but will do it slower and you will still be able to use that HD to serve thumbs or do whatever you want with them.

DD would be the quickest

That's a big tube

:2 cents:

Must be some solution for such kind of problem, something to finish it in 1 day

It's a flaw in design of file system or copy function of OS. Maybe some program can speed it up that could have raw disk write access and cache the data properly.