SEOClerks

What Is The Robots.txt File?



Write the reason you're deleting this FAQ

What Is The Robots.txt File?

What Is The Robots.txt File?

Comments

Please login or sign up to leave a comment

Join
stayonline
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is calledThe Robots Exclusion Protocol.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

  • robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
  • the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.



Are you sure you want to delete this post?

spyindiadanish
Thanks for sharing such a nice memory....



Are you sure you want to delete this post?

jaysh4922
Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.



Are you sure you want to delete this post?

sanfora
it not so important for seo as now google dont care much about html sites authority sites who can rank most of time



Are you sure you want to delete this post?

johnniewalk
Robot .txt file is a option where we can use in webmaster tool if you want to index or not to index a particular image you can use robot .txt file



Are you sure you want to delete this post?

josephinek1
Robot.txt:

It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.



Are you sure you want to delete this post?

patco444
It's a file that can tell which inner pages of your website to be indexed by Google & other SE's What Is The Robots.txt File?



Are you sure you want to delete this post?

Order Now
Process Time: 0.23231482505798

Possible Duplicate queries found!
MatchCountSQLScript
SELECT * FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='15773'1SELECT q.*, c.seo as CatSEO, c.name as CatName, c.parentid FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='15773'

/opt/clerks-staging/docroot/viewfaq.php 496 () ()

SELECT * FROM seoclerks.members WHERE USERID='171679'1SELECT * FROM seoclerks.members WHERE USERID='171679'

/opt/clerks-staging/docroot/include/functions/includes/member.php 445 GetAllUserDetails() ()

SELECT * FROM seoclerks.members WHERE USERID=1716791SELECT googleplus_profile FROM seoclerks.members WHERE USERID=171679

/opt/clerks-staging/docroot/include/functions/main.php 16914 GetGooglePlusProfileFromId() ()

SELECT * FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=15773 LIMIT 51SELECT m.username FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=15773 LIMIT 5

/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

SELECT * FROM answers a, seoclerks.members b WHERE a.quesid='15773' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc1SELECT a.answer, a.USERID, a.upvotes, a.downvotes, a.ansid, a.parentid, a.combined_votes, a.date_answered, b.username, b.userlevel, b.profilepicture FROM answers a, seoclerks.members b WHERE a.quesid='15773' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc

/opt/clerks-staging/docroot/viewfaq.php 1245 () ()

SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84709 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84709 LIMIT 5

/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84710 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84710 LIMIT 5

/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84711 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84711 LIMIT 5

/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84712 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84712 LIMIT 5

/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84713 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84713 LIMIT 5

/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84714 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84714 LIMIT 5

/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84715 LIMIT 51SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=84715 LIMIT 5

/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()

UPDATE questions SET total_views = total_views + 1 WHERE quesid='15773'1UPDATE questions SET total_views = total_views + 1 WHERE quesid='15773'

/opt/clerks-staging/docroot/include/functions/main.php 1765 update_Faqviewcount() ()

SELECT * FROM questions WHERE quesid!='15773' AND status='1' AND (question like '% Robots.txt File?%' OR question like '%Robots.txt%' OR question like '%File?%') -- ORDER BY RAND() LIMIT 151SELECT quesid, question, seo, userid FROM questions WHERE quesid!='15773' AND status='1' AND (question like '% Robots.txt File?%' OR question like '%Robots.txt%' OR question like '%File?%') -- ORDER BY RAND() LIMIT 15

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()

SELECT * FROM seoclerks.members WHERE USERID='31483' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='31483' limit 1

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

SELECT * FROM seoclerks.members WHERE USERID='38865' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='38865' limit 1

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

SELECT * FROM seoclerks.members WHERE USERID='12794' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='12794' limit 1

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

SELECT * FROM seoclerks.members WHERE USERID='4169' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='4169' limit 1

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

SELECT * FROM seoclerks.members WHERE USERID='29659' limit 11SELECT profilepicture FROM seoclerks.members WHERE USERID='29659' limit 1

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()

SELECT * FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=15773%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID 1SELECT USERID, username FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=15773%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()

SELECT * FROM categories1SELECT * FROM categories

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() parseRedundantQueriesCache()

select * from categories_software order by name asc1select * from categories_software order by name asc

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetSoftwareCategories()

select * from categories_wanttobuy order by name asc1select * from categories_wanttobuy order by name asc

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_wantcategories()

select * from categories_wanttotrade order by name asc1select * from categories_wanttotrade order by name asc

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_tradecategories()

SELECT * FROM seoclerks.members WHERE USERID='1607.png'1SELECT profilepicture FROM seoclerks.members WHERE USERID='1607.png'

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()

SELECT * FROM seoclerks.members WHERE USERID='60292.jpg'1SELECT profilepicture FROM seoclerks.members WHERE USERID='60292.jpg'

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()

SELECT * FROM seoclerks.members WHERE USERID='57782.jpg'1SELECT profilepicture FROM seoclerks.members WHERE USERID='57782.jpg'

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()

select * from categories_faq order by name asc1select * from categories_faq order by name asc

/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetFaqCategories()

Invalid SQL

count(*)sql1error_msg

Expensive SQL

Tuning the following SQL could reduce the server load substantially
LoadCountSQLMaxMin

Suspicious SQL

The following SQL have high average execution times
Avg TimeCountSQLMaxMin