Badbots suck

denverdonate Level 3

June 1, 2016, 10:12 pm

Write the reason you're deleting this FAQ

Badbots suck

I have tried the following: .htaccess to try and get them out, used robots.txt and a huge list ban IPs by region, ban seeming active bots that might be scrape content, turned Cpanel supercache, enable heavy duty cache, this would slow them in theory, Cpanel to check no-referrer bots, check AWSTATS and look for bots that declare something that they are not.

People say "ban their IP" ---- this actually is not a good suggestion because....do you think the bot will obey robots.txt if they are bad? And, bots don't use the same IP, a badbot of course will not pay attention. One day I saw in real saw 30+ Bingbots active at once, Bing doesn't love me so much to give me their army of bots over and over, those would be mostly badbots.

Are any of you working with a good strategy to get them away from your site?

Comments

Please login or sign up to leave a comment

Join

aloksof

Sorry i don't have any idea about this.

10 years ago

Are you sure you want to delete this post?

Avg Time	Count	SQL	Max	Min
0.099449	1	SELECT * FROM members WHERE show_freelancer = 1 AND status = 1 AND total_recommendations > 0 AND (skills LIKE '%GoogleTopranking%') ORDER BY total_recommendations desc LIMIT 0,40	0.099449	0.099449

SEOClerks

Badbots suck

Badbots suck

Comments

Please login or sign up to leave a comment

aloksof

Invalid SQL

Expensive SQL

Suspicious SQL

Match	Count	SQL	Script
SELECT * FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='11445'	1	SELECT q.*, c.seo as CatSEO, c.name as CatName, c.parentid FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='11445'	/opt/clerks-staging/docroot/viewfaq.php 496 () ()
SELECT * FROM seoclerks.members WHERE USERID='454497'	1	SELECT * FROM seoclerks.members WHERE USERID='454497'	/opt/clerks-staging/docroot/include/functions/includes/member.php 445 GetAllUserDetails() ()
SELECT * FROM seoclerks.members WHERE USERID=454497	1	SELECT googleplus_profile FROM seoclerks.members WHERE USERID=454497	/opt/clerks-staging/docroot/include/functions/main.php 16914 GetGooglePlusProfileFromId() ()
SELECT * FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=11445 LIMIT 5	1	SELECT m.username FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=11445 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
SELECT * FROM answers a, seoclerks.members b WHERE a.quesid='11445' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc	1	SELECT a.answer, a.USERID, a.upvotes, a.downvotes, a.ansid, a.parentid, a.combined_votes, a.date_answered, b.username, b.userlevel, b.profilepicture FROM answers a, seoclerks.members b WHERE a.quesid='11445' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc	/opt/clerks-staging/docroot/viewfaq.php 1245 () ()
SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=51782 LIMIT 5	1	SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=51782 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
UPDATE questions SET total_views = total_views + 1 WHERE quesid='11445'	1	UPDATE questions SET total_views = total_views + 1 WHERE quesid='11445'	/opt/clerks-staging/docroot/include/functions/main.php 1765 update_Faqviewcount() ()
SELECT * FROM questions WHERE quesid!='11445' AND status='1' AND (question like '%Badbots suck%' OR question like '%Badbots%' OR question like '%suck%') -- ORDER BY RAND() LIMIT 15	1	SELECT quesid, question, seo, userid FROM questions WHERE quesid!='11445' AND status='1' AND (question like '%Badbots suck%' OR question like '%Badbots%' OR question like '%suck%') -- ORDER BY RAND() LIMIT 15	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()
SELECT * FROM seoclerks.members WHERE USERID='19821' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='19821' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=11445%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID	1	SELECT USERID, username FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=11445%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()
SELECT * FROM categories	1	SELECT * FROM categories	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() parseRedundantQueriesCache()
select * from categories_software order by name asc	1	select * from categories_software order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetSoftwareCategories()
select * from categories_wanttobuy order by name asc	1	select * from categories_wanttobuy order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_wantcategories()
select * from categories_wanttotrade order by name asc	1	select * from categories_wanttotrade order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_tradecategories()
SELECT * FROM seoclerks.members WHERE USERID='144798.png'	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='144798.png'	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()
select * from categories_faq order by name asc	1	select * from categories_faq order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetFaqCategories()