rsync slow with large number of files, how can I improve it?

Drake83 Level 3

October 12, 2013, 9:43 am

Write the reason you're deleting this FAQ

rsync slow with large number of files, how can I improve it?

I have a huge amount of files I am transferring using rsync. I use it as a backup to transfer from my primary system to an NFS file system that is mounted. This is a great solution, however, it is very slow. It takes hours to complete it seems (I never timed it).

Here is my rsync call:

rsync -zvr --delete /var/www/site/public_html/pics/ /mount/backup/pics/

These are user profile pictures and since I have well over 100,000 users, I have a lot of profile pictures to transfer. They all reside in a single directly. These files are around 1-15kb.

What can I do to speed this up?

Comments

Please login or sign up to leave a comment

Join

Void

rsync is slow and takes a lot of CPU/Memory. If you have a ton of files in a directory, it is probably hanging at the stat() stage. rsync will also make multiple passes per file. Instead of using rsync, you could try tar.

tar -cf foo.tar /var/www/site/public_html/pics/

It should operate with much less resource usage although it may still take a bit on a large amount of files.
You can also explode it to where you want to save it:

tar -cf foo.tar /var/www/clerks/public_html/pics/ | tar -x foo.tar /mount/backup/pics

13 years ago

Are you sure you want to delete this post?

SEOClerks

rsync slow with large number of files, how can I improve it?

rsync slow with large number of files, how can I improve it?

Comments

Please login or sign up to leave a comment

Void

Invalid SQL

Expensive SQL

Suspicious SQL

Match	Count	SQL	Script
SELECT * FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='6724'	1	SELECT q.*, c.seo as CatSEO, c.name as CatName, c.parentid FROM `questions` as q JOIN categories_faq as c ON q.catid=c.CATID WHERE (q.status=1 OR (q.status=2 AND userid='') ) AND q.quesid='6724'	/opt/clerks-staging/docroot/viewfaq.php 496 () ()
SELECT * FROM seoclerks.members WHERE USERID='6289'	1	SELECT * FROM seoclerks.members WHERE USERID='6289'	/opt/clerks-staging/docroot/include/functions/includes/member.php 445 GetAllUserDetails() ()
SELECT * FROM seoclerks.members WHERE USERID=6289	1	SELECT googleplus_profile FROM seoclerks.members WHERE USERID=6289	/opt/clerks-staging/docroot/include/functions/main.php 16914 GetGooglePlusProfileFromId() ()
SELECT * FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=6724 LIMIT 5	1	SELECT m.username FROM ratings_faq as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=6724 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
SELECT * FROM answers a, seoclerks.members b WHERE a.quesid='6724' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc	1	SELECT a.answer, a.USERID, a.upvotes, a.downvotes, a.ansid, a.parentid, a.combined_votes, a.date_answered, b.username, b.userlevel, b.profilepicture FROM answers a, seoclerks.members b WHERE a.quesid='6724' AND a.userid=b.USERID and b.status='1' AND a.status=1 ORDER BY a.combined_votes DESC, a.date_answered asc	/opt/clerks-staging/docroot/viewfaq.php 1245 () ()
SELECT * FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=31286 LIMIT 5	1	SELECT m.username FROM ratings_faqanswers as r, seoclerks.members as m WHERE r.USERID=m.USERID AND r.upvote=1 AND r.PID=31286 LIMIT 5	/opt/clerks-staging/docroot/include/functions/main.php 17159 GetVoters() ()
UPDATE questions SET total_views = total_views + 1 WHERE quesid='6724'	1	UPDATE questions SET total_views = total_views + 1 WHERE quesid='6724'	/opt/clerks-staging/docroot/include/functions/main.php 1765 update_Faqviewcount() ()
SELECT * FROM questions WHERE quesid!='6724' AND status='1' AND (question like '%rsync slow large number files, I improve it?%' OR question like '%rsync%' OR question like '%slow%' OR question like '%large%' OR question like '%number%' OR question like '%files,%' OR question like '%improve%') -- ORDER BY RAND() LIMIT 15	1	SELECT quesid, question, seo, userid FROM questions WHERE quesid!='6724' AND status='1' AND (question like '%rsync slow large number files, I improve it?%' OR question like '%rsync%' OR question like '%slow%' OR question like '%large%' OR question like '%number%' OR question like '%files,%' OR question like '%improve%') -- ORDER BY RAND() LIMIT 15	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()
SELECT * FROM seoclerks.members WHERE USERID='10775' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='10775' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM seoclerks.members WHERE USERID='9874' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='9874' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM seoclerks.members WHERE USERID='17457' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='17457' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM seoclerks.members WHERE USERID='16573' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='16573' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM seoclerks.members WHERE USERID='1323' limit 1	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='1323' limit 1	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_member_profilepicture()
SELECT * FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=6724%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID	1	SELECT USERID, username FROM members_ledger WHERE script='/opt/clerks-staging/docroot/viewfaq.php' AND querystring LIKE '%id=6724%' AND added>=UNIX_TIMESTAMP(NOW())-1200 GROUP BY USERID	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() ()
SELECT * FROM categories	1	SELECT * FROM categories	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() parseRedundantQueriesCache()
select * from categories_software order by name asc	1	select * from categories_software order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetSoftwareCategories()
select * from categories_wanttobuy order by name asc	1	select * from categories_wanttobuy order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_wantcategories()
select * from categories_wanttotrade order by name asc	1	select * from categories_wanttotrade order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_get_tradecategories()
SELECT * FROM seoclerks.members WHERE USERID='4.jpg'	1	SELECT profilepicture FROM seoclerks.members WHERE USERID='4.jpg'	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() getUserProfileImage()
select * from categories_faq order by name asc	1	select * from categories_faq order by name asc	/opt/clerks-staging/docroot/libraries/adodb5/adodb.inc.php 1899 CacheExecute() insert_GetFaqCategories()