|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
02-25-2012, 11:48 PM
|
#1 |
|
Confirmed User
Industry Role:
Join Date: Mar 2005
Location: Los Angeles
Posts: 239
|
Copying 10 Million Thumbs, Fuck!
Here's an unexpected mind fuck... 10 million thumbs, 500 gigs, from one drive to another on the same server... estimated time, 4 fucking days!
|
|
|
|
02-26-2012, 12:09 AM
|
#2 |
|
Totally Borked
Industry Role:
Join Date: Feb 2005
Posts: 6,284
|
would have been much quicker to zip them then transfer the zip.
__________________
 For coding work - hit me up on andy // borkedcoder // com (consider figuring out the email as test #1) All models are wrong, but some are useful. George E.P. Box. p202 |
|
|
|
|
02-26-2012, 12:18 AM
|
#3 |
|
Confirmed User
Industry Role:
Join Date: Mar 2005
Location: Los Angeles
Posts: 239
|
i tried that first, copied all in just a few hours, but then needed 9 days to extract.. so i deleted the gzip and started over with copy
|
|
|
|
|
02-26-2012, 12:18 AM
|
#4 | |
|
Just Doing My Own Thing
Industry Role:
Join Date: Jan 2011
Location: London, Spain, New Zealand, GFY - Not Croydon...
Posts: 25,061
|
Quote:
|
|
|
|
|
02-26-2012, 12:27 AM
|
#5 |
|
Confirmed User
Industry Role:
Join Date: Mar 2005
Location: Los Angeles
Posts: 239
|
my second attempt has been running 13 hours now.. no compression sounds interesting, but not sure if the performance gain will be worth starting over now.. with compression was 9 days to extract.. without, maybe half? which would be the same as copy then...
|
|
|
|
|
02-26-2012, 12:31 AM
|
#6 |
|
Confirmed User
Join Date: Sep 2005
Location: Zurich
Posts: 1,045
|
WTF, one of your thumbs is as big as 50MB? Normally a thumb is below 10kB.
__________________
|
|
|
|
|
02-26-2012, 12:35 AM
|
#7 |
|
Confirmed User
Industry Role:
Join Date: Mar 2005
Location: Los Angeles
Posts: 239
|
160px × 120px, average 15 KB each.. in this case it's not the size, it's the number.
|
|
|
|
|
02-26-2012, 12:40 AM
|
#8 |
|
Confirmed User
Join Date: Sep 2005
Location: Zurich
Posts: 1,045
|
My calculation: 500GB/10M=50MB
|
|
|
|
|
02-26-2012, 01:11 AM
|
#9 | |
|
Just Doing My Own Thing
Industry Role:
Join Date: Jan 2011
Location: London, Spain, New Zealand, GFY - Not Croydon...
Posts: 25,061
|
Quote:
|
|
|
|
|
|
02-26-2012, 01:15 AM
|
#10 |
|
Confirmed User
Industry Role:
Join Date: May 2002
Location: Toronto
Posts: 8,475
|
|
|
|
|
|
02-26-2012, 01:23 AM
|
#11 | |
|
Confirmed User
Industry Role:
Join Date: May 2002
Posts: 8,098
|
Quote:
__________________
https://www.flow.page/savethechildren |
|
|
|
|
|
02-26-2012, 02:34 AM
|
#12 |
|
Confirmed User
Industry Role:
Join Date: Mar 2005
Location: Los Angeles
Posts: 239
|
i'm guesstimating... roughly 500k sets of 20 thumbs each, filled a 500gb drive 92%.. some thumbs are 10kb, some are 35kb.. and roughly 10-million of them..
edit.. was off by 172,000 set.. total is 672k sets for 20 |
|
|
|
|
02-26-2012, 02:40 AM
|
#14 |
|
Confirmed User
Industry Role:
Join Date: Mar 2005
Location: Los Angeles
Posts: 239
|
unix.. freebsd 8
|
|
|
|
|
02-26-2012, 02:47 AM
|
#15 | |
|
Confirmed User
Industry Role:
Join Date: Aug 2005
Location: YUROP
Posts: 8,606
|
Quote:
__________________
 Use coupon 'pauljohn' for a $1 discount at already super cheap NameSilo!Anal Webcams | Kinky Trans Cams Live | Hotwife XXX Tube | Get your Proxies here |
|
|
|
|
|
02-26-2012, 05:06 AM
|
#16 |
|
CjOverkill
Industry Role:
Join Date: Apr 2003
Location: Woldwide
Posts: 1,328
|
You have several options to speed this by an order of magnitude:
Option 1: If both HDs are the same size and from the same company and are the same model... I mean, both HDs are the same, then you have another option that will be considerably faster. We use dd here instead of copying. dd bs=10240k if=/dev/original of=/dev/copy Option 2: If on Linux, another option is to remount the original and the target partition with noatime option and then use the usual copy or rsync as normal. This will speed things alot when copying lots of small files. In this case rsync is better, even that it will be a bit slower than cp. The advantage of rsync when copying that much info is visible if the target HD has a sector error and copy process does not work properly. Rsync will warn you while cp will not. Option 3: Setup a RAID1 software raid between both HDs and it will automatically mirror the information. It will do practically the same as in option 1, but will do it slower and you will still be able to use that HD to serve thumbs or do whatever you want with them.
__________________
CjOverkill Traffic Trading Script Free, secure and fast traffic trading script. Get your copy now 
|
|
|
|
|
02-26-2012, 08:32 AM
|
#18 |
|
No Refunds Issued.
Industry Role:
Join Date: Feb 2001
Location: GFY
Posts: 28,300
|
That's a big tube
 |
|
|
|
|
02-26-2012, 09:38 AM
|
#20 |
|
Confirmed User
Industry Role:
Join Date: Oct 2010
Posts: 264
|
It's a flaw in design of file system or copy function of OS. Maybe some program can speed it up that could have raw disk write access and cache the data properly.
|
|
|
|
The Most extensive list of Paxum affiliate programs -> 