Cristian
Level X3
Write the reason you're deleting this FAQ
I'm frequently amazed by the number of webmasters and programmers out there unable to tell the difference between nofollow and noindex tags as well as robots.txt.
Let's start with the nofollow tag.
If you really want to stop Google or any main search engine from crawling a certain section of your website you need to implement a nofollow attribute. A lot of people mistake the nofollow tag with the noindex tag, hoping nofollow will prevent search engines from indexing certain pages. If a page has only the nofollow tag implemented crawlers can still be fetched manually but the spider won't go past the nofollow page.
I personally use the nofollow tag for backlinks I don't want to transfer authority to, but also on pages I don't want crawlers to find. I never used the alone nofollow attribute on internal pages without also using the noindex tag simultaneously.
I usually have two option. The first option is to place a page on noindex but allow robots to crawl it because I probably want some lower level pages to get indexed. The second option would be to place both nofollow and noindex to make sure the page won't get indexed and the links on the page itself don't get crawled.
Second, comes the noindex tag.
The soul purpose of the noindex tag is to prevent search engines from indexing a certain page. If really want to prevent a certain page for turning out in Google search results, implement the following line into the page source code: <META NAME="robots" CONTENT="noindex"> - it worked for me 100% of the time. I had problems with Google my robots.txt but I never saw Google or any other search engine index a page that had robots noindex implemented.
Contrary to popular belief the noindex attribute won't prevent search engines from crawling your website at will. Sure it won't index anything, but it will crawl and index anything they find without a noindex tag.
To ensure crawlers don't index and don't pass a certain page you need to implement both a noindex and a nofollow attribute, like this: <META NAME="robots" CONTENT="noindex,nofollow">
If you have a WordPress blog you can easily make both nofollow and noindex implementation with the help of an SEO plugin like Yoast SEO.
Robots.txt
Robots.txt is the perfect way to prevent certain crawling robots and other spiders from crawling or indexing certain portions or sections of your website. Although robots.txt is usually respected by search engines, I've seen plenty of times Google indexing the homepage of a website I've placed "Disallow: /" which means I don't want any crawler coming and going through my website.
So yeah, placing something at disallowed in robots.txt doesn't necessarily mean it won't get index. Google can overpass that and index it nevertheless. If this happens you might see results that have the following meta description: "A description for this result is not available because of this site's robots.txt – learn more"
If you really want to stop crawlers from finding and indexing a page you need to implement the above robots, noindex and nofollow tags.
Unless I need to tell crawlers no to index and crawl a long list of URL filters or something along the lines of URLs that are generated which Google understand perfectly and doesn't ignore, I usually have a very simple robots.txt that always looks the same.
Example of a simple and proper robots.txt that allows everything:
Are you sure you want to delete this post?
Are you sure you want to delete this post?
| Match | Count | SQL | Script |
| SELECT * FROM members_ledger WHERE ip = ? AND added>=unix_timestamp(NOW())-86400 | 1 | SELECT querystring, added
FROM members_ledger
WHERE
ip = ? AND added>=unix_timestamp(NOW())-86400 | /opt/clerks-staging/docroot/include/functions/includes/security.php 398 fetchMemberLedger() include_once() |
| SELECT * FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='22878' | 1 | SELECT q.*, c.seo as CatSEO, c.name as CatName, c.parentid
FROM `questions` as q
JOIN categories_faq as c ON q.catid=c.CATID
WHERE (q.status=1 OR
(q.status=2 AND userid='')
)
AND q.quesid='22878' | /opt/clerks-staging/docroot/viewfaq.php 496 () () |
| SELECT * FROM seoclerks.members WHERE USERID='134' | 1 | SELECT * FROM seoclerks.members WHERE USERID='134' | /opt/clerks-staging/docroot/include/functions/includes/member.php 445 GetAllUserDetails() () |
| SELECT * FROM seoclerks.members WHERE USERID=134 | 1 | SELECT googleplus_profile FROM seoclerks.members WHERE USERID=134 | /opt/clerks-staging/docroot/include/functions/main.php 16914 GetGooglePlusProfileFromId() () |
| SELECT * FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=22878 LIMIT 5 | 1 | SELECT m.username FROM
ratings_faq as r,
seoclerks.members as m
WHERE r.USERID=m.USERID
AND r.upvote=1
AND r.PID=22878
LIMIT 5 | /opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() () |
| SELECT * FROM answers a, seoclerks.members b WHERE a.quesid='22878' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc | 1 | SELECT a.answer, a.USERID, a.upvotes, a.downvotes, a.ansid, a.parentid, a.combined_votes, a.date_answered,
b.username, b.userlevel, b.profilepicture FROM answers a, seoclerks.members b
WHERE a.quesid='22878' AND a.userid=b.USERID and b.status='1' AND a.status=1
ORDER BY a.combined_votes DESC, a.date_answered asc | /opt/clerks-staging/docroot/viewfaq.php 1245 () () |
| SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=118570 LIMIT 5 | 1 | SELECT m.username FROM
ratings_faqanswers as r,
seoclerks.members as m
WHERE r.USERID=m.USERID
AND r.upvote=1
AND r.PID=118570
LIMIT 5 | /opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() () |
| SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=118581 LIMIT 5 | 1 | SELECT m.username FROM
ratings_faqanswers as r,
seoclerks.members as m
WHERE r.USERID=m.USERID
AND r.upvote=1
AND r.PID=118581
LIMIT 5 | /opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() () |
| SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=170928 LIMIT 5 | 1 | SELECT m.username FROM
ratings_faqanswers as r,
seoclerks.members as m
WHERE r.USERID=m.USERID
AND r.upvote=1
AND r.PID=170928
LIMIT 5 | /opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() () |
| UPDATE questions SET total_views = total_views + 1 WHERE quesid='22878' | 1 | UPDATE questions SET total_views = total_views + 1 WHERE quesid='22878' | /opt/clerks-staging/docroot/include/functions/main.php 1765 update_Faqviewcount() () |
| SELECT * FROM questions WHERE quesid!='22878' AND status='1' AND (question like '%Noindex, Nofollow Robots.txt - difference? %' OR question like '%Noindex,%' OR question like '%Nofollow%' OR question like '%Robots.txt%' OR question like '%difference?%') -- ORDER BY RAND() LIMIT 15 | 1 | SELECT quesid, question, seo, userid
FROM questions
WHERE quesid!='22878' AND status='1'
AND (question like '%Noindex, Nofollow Robots.txt - difference? %' OR question like '%Noindex,%' OR question like '%Nofollow%' OR question like '%Robots.txt%' OR question like '%difference?%')
-- ORDER BY RAND()
LIMIT 15 | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() () |
| SELECT * FROM seoclerks.members WHERE USERID='17457' limit 1 | 1 | SELECT profilepicture FROM seoclerks.members WHERE USERID='17457' limit 1 | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture() |
| SELECT * FROM seoclerks.members WHERE USERID='3080' limit 1 | 1 | SELECT profilepicture FROM seoclerks.members WHERE USERID='3080' limit 1 | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture() |
| SELECT * FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=22878%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID | 1 | SELECT USERID, username FROM members_ledger
WHERE script='/opt/clerks-staging/docroot/viewfaq.php'
AND querystring LIKE '%id=22878%'
AND added>=UNIX_TIMESTAMP(NOW())-1200
GROUP BY USERID
| /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() () |
| SELECT * FROM categories | 1 | SELECT * FROM categories | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() parseRedundantQueriesCache() |
| select * from categories_software order by name asc | 1 | select * from categories_software order by name asc | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetSoftwareCategories() |
| select * from categories_wanttobuy order by name asc | 1 | select * from categories_wanttobuy order by name asc | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_wantcategories() |
| select * from categories_wanttotrade order by name asc | 1 | select * from categories_wanttotrade order by name asc | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_tradecategories() |
| SELECT * FROM seoclerks.members WHERE USERID='584702.jpg' | 1 | SELECT profilepicture FROM seoclerks.members WHERE USERID='584702.jpg' | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage() |
| SELECT * FROM seoclerks.members WHERE USERID='600657.jpg' | 1 | SELECT profilepicture FROM seoclerks.members WHERE USERID='600657.jpg' | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage() |
| select * from categories_faq order by name asc | 1 | select * from categories_faq order by name asc | /opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetFaqCategories() |
| count(*) | sql1 | error_msg |
|---|
| Load | Count | SQL | Max | Min |
Tronia
Noindex = don't access that site (for example Google), so its content won't appear on the search engine,
Nofollow = used for links, self-explanatory name.
There are also other META tags like nosnippet and noarchieve.
Definitly worth reading and learning. Thank you for giving us more detailed information regarding these three things. I can honestly admit that I only knew the very basic functions of each one but I could still tell them apart. Noindex = don't access that site (for example Google), so its content won't appear on the search engine, Nofollow = used for links, self-explanatory name. There are also other META tags like nosnippet and noarchieve. Definitly worth reading and learning.
Are you sure you want to delete this post?