|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
11-08-2018, 08:23 AM
|
#1 |
|
Confirmed User
Join Date: Oct 2002
Posts: 32
|
AWE whitelabel & robots.txt
Hi guys,
is anyone here who is still using whitelabel site by AWE? Now they are blocking googlebot in robots.txt: Code:
User-agent: Googlebot Disallow: / User-agent: * Allow: / Crawl-delay: 1  |
|
|
|
11-08-2018, 08:40 AM
|
#2 |
|
Confirmed Fetishist
Industry Role:
Join Date: Mar 2005
Location: Fetishland
Posts: 11,492
|
it's like that for a long time.
|
|
|
|
11-08-2018, 08:49 AM
|
#3 | |
|
Too lazy to set a custom title
Industry Role:
Join Date: Feb 2005
Posts: 17,168
|
Quote:
 
__________________
 |
|
|
|
|
|
11-08-2018, 10:35 AM
|
#4 |
|
I'll make you famous
Industry Role:
Join Date: Oct 2002
Posts: 13,935
|
Bump for biz
 |
|
|
|
|
11-08-2018, 10:47 AM
|
#5 |
|
Confirmed User
Industry Role:
Join Date: Apr 2001
Posts: 1,738
|
Whitelabels are best used if you have your own source of traffic to send to them. If you want to get SE traffic to a site then your better off using a package that let's you build pages based on the Cam's and then direct traffic into the main sites. User blackmonsters here on gfy makes a WP plugin that pulls from a number of cam shops and let's you create pages for the girls that are online.
EconfirmPro good luck
__________________
TeenFlood.com Online since 1998.
 TFCash KissMeGirl VirginRiches MondoBucks tim at tfcash.com or submit a ticket at our HelpDesk |
|
|
|
|
11-08-2018, 10:53 AM
|
#6 |
|
Confirmed User
Industry Role:
Join Date: Feb 2006
Posts: 2,768
|
I related this issue to them a long time ago, and they replied to me:
They said they tried to give several options in the past to Webmasters so that we can customize our Whitelabels to have unique content on the site so that Google doesn't consider them as having thin content. Despite the efforts, Google started penalizing the Whitelabels, and they even started to penalize their brand sites. So to avoid further penalties for the WhiteLabels and their sites alike, they canonicalized them to Livesex.com. They said that this way they are not indexed, but they also avoid being penalized. Same goes not only for the WhiteLabels but to their sister sites, too (MyCams.com, LivePrivates, etc.). Having more canonical links to Livesex.com might have been the side effect, but it was not the goal. This was what they told me
__________________
List Of Sex Cams |
|
|
|
|
11-08-2018, 01:10 PM
|
#7 |
|
Confirmed User
Industry Role:
Join Date: Dec 2012
Location: Scandinavia
Posts: 276
|
Yeah,
and exactly therefore (the canonical) you should block it via robots.txt so it at least ranks for its domain name. If you block it in robots.txt Google does not does not see the canonical. So it will rank for the domain name (but nothing else). There is a checkbox in the "SEO" section saying "Allow robots" - remove it and it is disallowed via robots.txt. .
__________________
|
|
|
|
|
11-08-2018, 01:27 PM
|
#8 | |
|
Confirmed User
Industry Role:
Join Date: Sep 2009
Posts: 2,302
|
Quote:
Not sure why it's taking so long to remove a line of text but it is.  |
|
|
|
|
|
11-08-2018, 01:45 PM
|
#9 | |
Industry Role:
Join Date: Mar 2003
Location: San Diego
Posts: 32,112
|
Quote:
__________________
|
|
|
|
|
|
|
|||||||
|
|||||||
| Bookmarks |
| Tags |
| awe, googlebot, robots.txt, user-agent, whitelabel |
