SEOClerks

Clear
Sign In Join

Is it true Robot.txt are no longer used with google

  • seohelper2012 Level 1
    • 5

    Write the reason you're deleting this FAQ

    Is it true Robot.txt are no longer used with google

    I wanted to know if its true robot.txt are no longer needed for google seo purposes

    #google #seo #sitemap #analytics Google
    Share post:

    Comments

    Please login or sign up to leave a comment

    Join
    idealmike

    idealmike

    Robots.txt is still in use today like it has been ever since it was first created for the purpose of telling spiders which pages on your site are where and which ones it can and can't crawl/index. For example, you could tell it where your sitemap is by putting the link to your sitemap in the Robots.txt file but also tell it not to crawl and index certain pages on your site that you wouldn't want to show in the SERPS like admin login pages or directories where your sites files are kept to. It prevents crawling of those directories and thus stops them showing up in the SERPS. They are just one very small part of good website design and on-page SEO and are often overlooked by webmasters even today.
    10 years ago

    Are you sure you want to delete this post?

    Bert

    Bert

    They are still valid if you don't want the search spiders to crawl certain pages. This is what the robot.txt file is for. They didn't really have any seo purpose anyway and most people don't bother with them.
    14 years ago

    Are you sure you want to delete this post?

    SEOMarketer

    SEOMarketer

    I use them to block certain pages that I don't want indexed.
    14 years ago

    Are you sure you want to delete this post?

    riacom

    riacom

    me same used it to block certain pages
    14 years ago

    Are you sure you want to delete this post?

    Corzhens

    Corzhens

    To be honest I do not know about robot.txt because I am not really well versed with SEO yet. I am glad to encounter this discussion to broaden my knowledge with the search engine crawlers. My question now is the robot.txt is a necessity of the site otherwise the site will not be inspected by the crawler as if it is not updated with the contents because dormant sites are not checked by the crawlers. Correct me if I’m wrong.
    7 years ago

    Are you sure you want to delete this post?

    Order Now
    Process Time: 0.053735971450806

    Possible Duplicate queries found!
    MatchCountSQLScript
    SELECT * FROM members_ledger WHERE ip = ? AND added>=unix_timestamp(NOW())-864001SELECT querystring, added FROM members_ledger WHERE ip = ? AND added>=unix_timestamp(NOW())-86400

    		/opt/clerks-staging/docroot/include/functions/includes/security.php 398 fetchMemberLedger() include_once()

    SELECT * FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='578'1SELECT q.*, c.seo as CatSEO, c.name as CatName, c.parentid FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='578'

    		/opt/clerks-staging/docroot/viewfaq.php 496 () ()

    SELECT * FROM seoclerks.members WHERE USERID='4136'1SELECT * FROM seoclerks.members WHERE USERID='4136'

    		/opt/clerks-staging/docroot/include/functions/includes/member.php 445 GetAllUserDetails() ()

    SELECT * FROM categories_faq WHERE CATID='36'1SELECT CATID, seo, name, metatitle, h2header, metakeywords, metadescription, metaheader, parentid, image_name FROM categories_faq WHERE CATID='36'

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() GetCategorySeoFromType()

    SELECT * FROM categories_faq WHERE CATID='30'1SELECT CATID, seo, name, metatitle, h2header, metakeywords, metadescription, metaheader, parentid, image_name FROM categories_faq WHERE CATID='30'

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() GetCategorySeoFromType()

    SELECT * FROM seoclerks.members WHERE USERID=41361SELECT googleplus_profile FROM seoclerks.members WHERE USERID=4136

    		/opt/clerks-staging/docroot/include/functions/main.php 16914 GetGooglePlusProfileFromId() ()

    SELECT * FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=578 LIMIT 51SELECT m.username FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=578 LIMIT 5

    		/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

    SELECT * FROM answers a, seoclerks.members b WHERE a.quesid='578' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc1SELECT a.answer, a.USERID, a.upvotes, a.downvotes, a.ansid, a.parentid, a.combined_votes, a.date_answered, b.username, b.userlevel, b.profilepicture FROM answers a, seoclerks.members b WHERE a.quesid='578' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc

    		/opt/clerks-staging/docroot/viewfaq.php 1245 () ()

    SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=38683 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=38683 LIMIT 5

    		/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

    SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=4441 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=4441 LIMIT 5

    		/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

    SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=4442 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=4442 LIMIT 5

    		/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

    SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=7247 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=7247 LIMIT 5

    		/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

    SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=183301 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=183301 LIMIT 5

    		/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

    UPDATE questions SET total_views = total_views + 1 WHERE quesid='578'1UPDATE questions SET total_views = total_views + 1 WHERE quesid='578'

    		/opt/clerks-staging/docroot/include/functions/main.php 1765 update_Faqviewcount() ()

    SELECT * FROM questions WHERE quesid!='578' AND status='1' AND (question like '%Is true Robot.txt longer google%' OR question like '%true%' OR question like '%Robot.txt%' OR question like '%longer%' OR question like '%google%') -- ORDER BY RAND() LIMIT 151SELECT quesid, question, seo, userid FROM questions WHERE quesid!='578' AND status='1' AND (question like '%Is true Robot.txt longer google%' OR question like '%true%' OR question like '%Robot.txt%' OR question like '%longer%' OR question like '%google%') -- ORDER BY RAND() LIMIT 15

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()

    SELECT * FROM seoclerks.members WHERE USERID='1323' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='1323' limit 1

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

    SELECT * FROM seoclerks.members WHERE USERID='498' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='498' limit 1

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

    SELECT * FROM seoclerks.members WHERE USERID='1129' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='1129' limit 1

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

    SELECT * FROM seoclerks.members WHERE USERID='632' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='632' limit 1

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

    SELECT * FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=578%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID 1SELECT USERID, username FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=578%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()

    SELECT * FROM categories1SELECT * FROM categories

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() parseRedundantQueriesCache()

    select * from categories_software order by name asc1select * from categories_software order by name asc

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetSoftwareCategories()

    select * from categories_wanttobuy order by name asc1select * from categories_wanttobuy order by name asc

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_wantcategories()

    select * from categories_wanttotrade order by name asc1select * from categories_wanttotrade order by name asc

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_tradecategories()

    SELECT * FROM seoclerks.members WHERE USERID='272.jpg'1SELECT profilepicture FROM seoclerks.members WHERE USERID='272.jpg'

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()

    SELECT * FROM seoclerks.members WHERE USERID='337.jpg'1SELECT profilepicture FROM seoclerks.members WHERE USERID='337.jpg'

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()

    SELECT * FROM seoclerks.members WHERE USERID='3524.png'1SELECT profilepicture FROM seoclerks.members WHERE USERID='3524.png'

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()

    SELECT * FROM seoclerks.members WHERE USERID='140.jpg'1SELECT profilepicture FROM seoclerks.members WHERE USERID='140.jpg'

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()

    SELECT * FROM seoclerks.members WHERE USERID='600657.jpg'1SELECT profilepicture FROM seoclerks.members WHERE USERID='600657.jpg'

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()

    select * from categories_faq order by name asc1select * from categories_faq order by name asc

    		/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetFaqCategories()

    Invalid SQL

    count(*)sql1error_msg

    Expensive SQL

    Tuning the following SQL could reduce the server load substantially
    LoadCountSQLMaxMin

    Suspicious SQL

    The following SQL have high average execution times
    Avg TimeCountSQLMaxMin