Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums.

You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today!

If you have any problems with the registration process or your account login, please contact us.

Post New Thread Reply

Register GFY Rules Calendar
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >
Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed.

 
Thread Tools
Old 02-13-2015, 03:46 AM   #1
AdultSites
Confirmed User
 
Industry Role:
Join Date: Feb 2015
Posts: 203
How to map / get idea of a large site?

There is a website that I would need to map, get directory structure, and it is around 1,000,000 pages big. I have no access to the server, or cms. Not all the pages show in Google, and other search engines, and even if they would, it would not be too much help.

I probably need a sitemap, but from what I remember, it is kind of hard to make a sitemap for a big page like that. What would be the best way to work on that / get it done? Something free would be good too. I need website structure, and all urls.

Thanks.
AdultSites is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 12:18 PM   #2
clickity click
So Fecking Bummed
 
Industry Role:
Join Date: Aug 2014
Posts: 3,682
Just use something like httrack which will make a directory on your PC with all the files. If you don't need images you can disable scanning for them.
Most sites have a sitemap as well. Usually something like site.com/sitemap.xml

Might be worth checking first.
clickity click is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 12:22 PM   #3
Harmon
( ͡ʘ╭͜ʖ╮͡ʘ)
 
Harmon's Avatar
 
Industry Role:
Join Date: Mar 2004
Posts: 19,987
So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.
Harmon is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 12:27 PM   #4
The Porn Nerd
Living The Dream
 
The Porn Nerd's Avatar
 
Industry Role:
Join Date: Jun 2009
Location: Inside a Monitor
Posts: 19,317
Quote:
Originally Posted by Harmon View Post
So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.
__________________
My Affiliate Programs:
Porn Nerd Cash | Porn Showcase | Aggressive Gold (Coming Soon)

Over 90 paysites to promote!
Skype: peabodymedia
The Porn Nerd is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 01:11 PM   #5
sandman!
Icq: 14420613
 
sandman!'s Avatar
 
Industry Role:
Join Date: Mar 2001
Location: chicago
Posts: 15,421



Quote:
Originally Posted by Harmon View Post
So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.
__________________
Need WebHosting ? Email me for some great deals [email protected]
sandman! is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 03:41 PM   #6
Bladewire
StraightBro
 
Bladewire's Avatar
 
Industry Role:
Join Date: Aug 2003
Location: Monarch Beach, CA USA
Posts: 56,232
Quote:
Originally Posted by Harmon View Post
So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.
Quote:
Originally Posted by The Porn Nerd View Post
Quote:
Originally Posted by sandman! View Post
















__________________


Skype: CallTomNow

Bladewire is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 03:49 PM   #7
AdultKing
Raise Your Weapon
 
AdultKing's Avatar
 
Industry Role:
Join Date: Jun 2003
Location: Outback Australia
Posts: 15,605
Quote:
Originally Posted by Harmon View Post
So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.


This forum only attracts assholes now.
AdultKing is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 05:48 PM   #8
Barry-xlovecam
It's 42
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 18,083
learn to hack :P

Quote:
ssh user@target
user@target password:

omitted


scp user@target:

omitted


ssh again

user@target:~$/path to document route/$ rm remote-tree.csv
user@target:~$ cd~
user@target:~$ nano .bash_history
You want root so you can clean your connections from the system logs
su {or sudo}
??? look it up l4m3r :P
nano pico vi whatever ...
delete your tracks ^x save
^d


And don't use your mother's IP
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 05:53 PM   #9
epitome
So Fucking Lame
 
epitome's Avatar
 
Industry Role:
Join Date: Jun 2009
Location: St. Petersburg, FL
Posts: 12,158
How the fuck was the name AdultSites not registered on GFY until 2015?

Seems like such an obvious choice.
epitome is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-13-2015, 11:34 PM   #10
freecartoonporn
Confirmed User
 
freecartoonporn's Avatar
 
Industry Role:
Join Date: Jan 2012
Location: NC
Posts: 7,683
are you scraping Wikipedia ?

post the url so i can give some input,
freecartoonporn is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-14-2015, 04:28 AM   #11
AdultSites
Confirmed User
 
Industry Role:
Join Date: Feb 2015
Posts: 203
Quote:
Originally Posted by clickity click View Post
Just use something like httrack which will make a directory on your PC with all the files. If you don't need images you can disable scanning for them.
Most sites have a sitemap as well. Usually something like site.com/sitemap.xml

Might be worth checking first.
This program does not seem to be perfect, but I may be doing it wrong too. I will look at the settings, and see what I can make of it. It looks like Teleport Pro, and other download managing programs. They dont seem to be 100% good. They should just work, but there seem to be a lot of bs with that. In the end I end up checking on these programs a lot, instead of getting everything done in the background. Again, I may be doing it wrong, as I did not pay too much attention to this, I was just testing.
AdultSites is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-14-2015, 05:55 AM   #12
clickity click
So Fecking Bummed
 
Industry Role:
Join Date: Aug 2014
Posts: 3,682
what exactly are you trying to do. What it the site. You do know you can't just copy a site and have it running unless it's HTML only.
clickity click is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 02-14-2015, 06:55 AM   #13
Roald
SecretFriends.com
 
Roald's Avatar
 
Industry Role:
Join Date: May 2001
Location: IMC Headquarters
Posts: 27,862
Quote:
Originally Posted by AdultKing View Post


This forum only attracts assholes now.
this forum always attracted assholes. nothing new really.
__________________


WE ARE BUYING PAY SITES! CONTACT ME



ClubSweethearts | ManUpFilms | SinfulXXX | HOT * AdultPrime * HOT


Paying webmasters since 1996! Contact: r.riepen @ sansylgroup.com | skype:roaldr | icq:
Roald is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Post New Thread Reply
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >

Bookmarks

Tags
pages, sitemap, structure, site, idea, map



Advertising inquiries - marketing at gfy dot com

Contact Admin - Advertise - GFY Rules - Top

©2000-, AI Media Network Inc



Powered by vBulletin
Copyright © 2000- Jelsoft Enterprises Limited.