Migrate your data

How to migrate data from an external source to your UltiHash cluster

This guide provides step-by-step instructions for migrating data, using the example of transferring from Amazon S3 to UltiHash using rclone. (You can visit rclone's GitHub respository here.)

By following these steps, you can efficiently transfer your data to an UltiHash cluster while maintaining security and performance.

Configure rclone

First, add the following configuration to your rclone installation. Modify the endpoint setting to point to your specific UltiHash cluster.

Note: Authentication parameters will be read from the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

[UltiHash]
type = s3
provider = Other
env_auth = true
endpoint = https://ultihash.endpoint.example

Alternative: If you do not want to use environment variables, set env_auth = false and add access_key_id and secret_access_key manually in the rclone config file.

Uploading Data to UltiHash

Step 1: Create a Bucket

Before uploading data, create a bucket in UltiHash:

rclone mkdir UltiHash:/bucket

Step 2: Copy Local Data to UltiHash

To transfer your local data to UltiHash, use the following command:

rclone copy --progress /path/to/local/data UltiHash:/bucket

This command will display the transfer progress in real time.