Complete website scraping tools (user-data)?

Cristian Level X³

August 27, 2016, 8:09 am

Write the reason you're deleting this FAQ

Complete website scraping tools (user-data)?

I want to gather as much information as possible from one specific website. I'm not going to share the actual website because, well, it doesn't matter and I don't want to promote unnecessary stuff here.

So, my main goal is to figure out which periods of the day that website is getting more traffic, which specific write is driving more clicks to their articles as well as other general traffic related information. But the user's details and weekly/daily traffic periods are the most important.

I don't really want an SEO analysis because I can already do that pretty well, I want more of a user interaction approach analysis and just so you know we are talking about a Wordpress based website here and I have virtually no access to their Google analytics or anything on the inside.

Any good tools I can use? Paid or free, doesn't matter, I just want to scrap some good useful data.

Comments

Please login or sign up to leave a comment

Join

Lynne

Oh I am following this with great interest! I would love to do this with my websites!

Great topic Cristian.

10 years ago

Are you sure you want to delete this post?

TommyCarey

Hmmmm I know of scrapers that do a ton of stuff, but I've never heard of one that will scrape user data to figure out traffic patterns of a persons website.

I wonder if you could just use something like SEMRush to do this. You could even use Spyfu but I'm not 100% sure if they would have what you need. Both of those are paid services and are well known for what they do.

You might have to use a few different services to get exactly what you want since I don't know of any data scrapers to figure out a websites traffic

Complete website scraping tools (user-data)?

I hope that someone here has something they can share with you whether it be a third party scraper or something they custom coded

Also, have you ever through of posting a WTB to get this type of scraper to be designed? I'm sure if you hired the right person they could get this done is a day or two. In the past I've hired people to build custom scrapers for a specific website and it was fairly cheap. I think I paid $60 and it scraped together emails, business names, phone numbers, etc. But that's different from what you need

I wish I could have been more help.

10 years ago

Are you sure you want to delete this post?

Cristian

A WTB may be a good solution though I would prefer a professional tool rather than explain everything to a coder.

I can calculate the organic traffic a website may get with Ahrefs, I can use SimilarWeb to figure out overall potential traffic, but I need specific periods, like a Google Analytics chart that clearly shows me the more popular periods as well as potential patterns.

10 years ago

Are you sure you want to delete this post?

overcast

If you want to crawl the backlinks then scrapy is one tool that runs on python. And it can scrape a large number of links. And it can be adding a lot of value. You can also use this for SEO value. And then it can be then exported into Excel and other tools if possible.

You can also check out import.io. That site seems to be allowing different type of scraping. And it can be good enough for many options. I have learned that it may require a bit of effort on our part to scrape though. Because none of them are not push button scraping solutions.

9 years ago

Are you sure you want to delete this post?

SEOClerks

Complete website scraping tools (user-data)?

Complete website scraping tools (user-data)?

Comments

Please login or sign up to leave a comment

Lynne

TommyCarey

Cristian

overcast

Invalid SQL

Expensive SQL

Suspicious SQL

Match	Count	SQL	Script
SELECT * FROM members_ledger WHERE ip = ? AND added>=unix_timestamp(NOW())-86400	1	SELECT querystring, added FROM members_ledger WHERE ip = ? AND added>=unix_timestamp(NOW())-86400	/opt/clerks-staging/docroot/include/functions/includes/security.php 398 fetchMemberLedger() include_once()
SELECT * FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='19229'	1	SELECT q.*, c.seo as CatSEO, c.name as CatName, c.parentid FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='19229'	/opt/clerks-staging/docroot/viewfaq.php 496 () ()
SELECT * FROM seoclerks.members WHERE USERID='134'	1	SELECT * FROM seoclerks.members WHERE USERID='134'	/opt/clerks-staging/docroot/include/functions/includes/member.php 445 GetAllUserDetails() ()
SELECT * FROM categories_faq WHERE CATID='43'	1	SELECT CATID, seo, name, metatitle, h2header, metakeywords, metadescription, metaheader, parentid, image_name FROM categories_faq WHERE CATID='43'	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() GetCategorySeoFromType()
SELECT * FROM seoclerks.members WHERE USERID=134	1	SELECT googleplus_profile FROM seoclerks.members WHERE USERID=134	/opt/clerks-staging/docroot/include/functions/main.php 16914 GetGooglePlusProfileFromId() ()
SELECT * FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=19229 LIMIT 5	1	SELECT m.username FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=19229 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
SELECT * FROM answers a, seoclerks.members b WHERE a.quesid='19229' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc	1	SELECT a.answer, a.USERID, a.upvotes, a.downvotes, a.ansid, a.parentid, a.combined_votes, a.date_answered, b.username, b.userlevel, b.profilepicture FROM answers a, seoclerks.members b WHERE a.quesid='19229' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc	/opt/clerks-staging/docroot/viewfaq.php 1245 () ()
SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=105793 LIMIT 5	1	SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=105793 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=105814 LIMIT 5	1	SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=105814 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=105830 LIMIT 5	1	SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=105830 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=133573 LIMIT 5	1	SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=133573 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
UPDATE questions SET total_views = total_views + 1 WHERE quesid='19229'	1	UPDATE questions SET total_views = total_views + 1 WHERE quesid='19229'	/opt/clerks-staging/docroot/include/functions/main.php 1765 update_Faqviewcount() ()
SELECT * FROM questions WHERE quesid!='19229' AND status='1' AND (question like '%Complete website scraping tools (user-data)?%' OR question like '%Complete%' OR question like '%website%' OR question like '%scraping%' OR question like '%tools%' OR question like '%(user-data)?%') -- ORDER BY RAND() LIMIT 15	1	SELECT quesid, question, seo, userid FROM questions WHERE quesid!='19229' AND status='1' AND (question like '%Complete website scraping tools (user-data)?%' OR question like '%Complete%' OR question like '%website%' OR question like '%scraping%' OR question like '%tools%' OR question like '%(user-data)?%') -- ORDER BY RAND() LIMIT 15	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()
SELECT * FROM seoclerks.members WHERE USERID='9' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='9' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM seoclerks.members WHERE USERID='895' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='895' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM seoclerks.members WHERE USERID='3160' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='3160' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM seoclerks.members WHERE USERID='147' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='147' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM seoclerks.members WHERE USERID='2077' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='2077' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=19229%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID	1	SELECT USERID, username FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=19229%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()
SELECT * FROM categories	1	SELECT * FROM categories	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() parseRedundantQueriesCache()
select * from categories_software order by name asc	1	select * from categories_software order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetSoftwareCategories()
select * from categories_wanttobuy order by name asc	1	select * from categories_wanttobuy order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_wantcategories()
select * from categories_wanttotrade order by name asc	1	select * from categories_wanttotrade order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_tradecategories()
SELECT * FROM seoclerks.members WHERE USERID='306632.jpg'	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='306632.jpg'	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()
SELECT * FROM seoclerks.members WHERE USERID='19821.png'	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='19821.png'	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()
SELECT * FROM seoclerks.members WHERE USERID='134.png'	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='134.png'	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()
SELECT * FROM seoclerks.members WHERE USERID='589438.jpg'	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='589438.jpg'	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()
select * from categories_faq order by name asc	1	select * from categories_faq order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetFaqCategories()