Notes

Transfer Mastodon Media to Object Storage with Rclone

Jonah Aragon

Jonah Aragon

— 1 min read 
rclone copy --fast-list --copy-links -P \
  --max-backlog 10000 \
  --order-by size,mixed,75 \
  --transfers 32 \
  --s3-chunk-size 16M --s3-upload-concurrency 8 \
  ./live/public/system/ remote:media-bucket/
  • fast-list is compatible with Garage S3
  • max-backlog queues up to 10,000 files at a time
  • order-by size,mixed,75 takes those 10,000 and sorts them by size so that 75% of the uploads are "small files" and 25% of the uploads are "large files."
  • transfers 32 files simultaneously. Maybe more would work, I was limited by Hetzner's slow block storage device.
  • s3-chunk-size and s3-upload-concurrency apply to multipart uploads of larger files, this is a bit larger than rclone's defaults. Probably could be set larger in other cases, but Mastodon doesn't have many super large files so I'm not sure it would help much for performance here.

Read more