Data migration from on-premise to cloud using rclone tool
Recently we had a requirement to migrate our on-premise data around 70TB from swift storage to aws s3 bucket. We have initially explored various options to migrate, out of which we have chosen rclone for data migration, because it supports a wide range of cloud storage providers, including Google Drive, Dropbox, Amazon S3, and Microsoft OneDrive.
Why RCLONE ?
Rclone is mature, open-source software originally inspired by rsync and written in Go. It is multi threaded, very fast in transfer and allows to control the transfer speed.
Here is a summary of the Rclone equivalents to the Unix commands:
rsync: Rclone sync is a powerful tool for synchronizing files and directories between your local machine and the cloud. It can be used to create backups, mirror directories, or simply keep your local and cloud data in sync.
cp: Rclone copy is a simple tool for copying files and directories from your local machine to the cloud or vice versa.
mv: Rclone move is a simple tool for moving files and directories from one cloud storage provider to another.
mount: Rclone mount allows you to mount a cloud storage bucket as a directory on your local machine. This means that you can access your cloud files as if they were stored on your local hard drive.
ls: Rclone ls lists the contents of a cloud storage bucket.
ncdu: Rclone ncdu displays the disk usage of a cloud storage bucket.
tree: Rclone tree displays a tree view of the contents of a cloud storage bucket.
rm: Rclone rm removes files and directories from the cloud.
cat: Rclone cat displays the contents of a file in the cloud.
How to configure rclone and use?
Here am providing the configuration for amazon s3, we can follow the steps outlined below, also refer https://rclone.org/s3/
Download the rclone software for your respective OS - https://rclone.org/downloads/
Run “rclone config”, it will guide you through an interactive setup process.
Source configuration : https://rclone.org/swift/
Target configuration : https://rclone.org/s3/#configuration
Validate the details in the configuration file (~/.config/rclone/rclone.conf) in your local machine and ensure you are able to connect to the source and target using rclone.
$ cat ~/.config/rclone/rclone.conf
[onprem_swift]
type = swift
env_auth = true
user = my_object_store
key =
auth = https://swift.company.com/auth/v1.0
domain =
tenant = my_object_store
tenant_domain =
region =
storage_url =
auth_version =
endpoint_type = public
[cloud-aws-s3]
type = s3
provider = AWS
access_key_id = <aws s3 access key id>
secret_access_key = < secret key>
region = us-east-2
location_constraint = us-east-2
acl = private
storage_class = STANDARD
Display the directories in swift/s3 buckets
[user@host01 ~]$ rclone lsd cloud-aws-s3:prod
0 2023-10-28 02:33:56 -1 data
0 2023-10-28 02:33:56 -1 images
[user@host01 ~]$ rclone lsd cloud-aws-s3:prod
0 2023-10-28 02:33:56 -1 data
0 2023-10-28 02:33:56 -1 images
Now we are good to transfer the data, let do it
[user@host01 ~]$ rclone sync --progress --transfers=8 --bwlimit=10M onprem_swift:prod cloud-aws-s3:prod
[user@host01 ~] $ ps -ef|grep rclone
user+ 4018633 1 7 00:05 pts/0 00:00:26 rclone sync --progress onprem_swift:prod cloud-aws-s3:prod
user+ 4047166 2812493 0 00:11 pts/0 00:00:00 sh rclone_prod.sh
user+ 4047172 4047166 29 00:11 pts/0 00:00:11 rclone sync --progress onprem_swift:prod cloud-aws-s3:prod
user+ 4050901 2812493 0 00:11 pts/0 00:00:00 grep --color=auto rclone
[user@host01 ~]$
By using the rclone tool, we will have more control in transferring the data in parallel and very easy to use as all the commands are like unix commands.
Best Practices:
We have tried various options to speed up transfer between swift and aws s3, however we did not get the right speed to transfer. Later we analyzed the details thoroughly and decided to create aws s3 bucket in close to the region that we are in i.e us-east2 region and we have used around 5 machines to transfer in parallel, these significantly helped to speed up the migration process with average speed between 40mb/sec and 90mb/sec.
I recommend to use its latest version, it has a great support community which we can make use of for posting our issues to get immediate assistance from the forum.