PyTorch

How to connect UltiHash to PyTorch

PyTorch is an S3 native library. Prior to starting a session that accesses and stores data in and to UltiHash, the user should ensure that their UltiHash credentials are exported in their working terminal and then can proceed as follows:

import s3torchconnector

# Define S3 region (could be any, since the custom S3 endpoint is used)
REGION = "us-east-1"
# Define the endpoint URL for the UltiHash cluster
ENDPOINT_URL = "<https://ultihash.cluster>"

# Enforce the path style URLs
config = s3torchconnector.S3ClientConfig(force_path_style=True)

# EXAMPLE 1: Create a dataset from the data stored on the UltiHash bucket named "test-data"
dataset = s3torchconnector.S3MapDataset.from_prefix(
    "s3://test-data/", 
    endpoint=ENDPOINT_URL, 
    region=REGION, 
    s3client_config=config
)

# EXAMPLE 2: Create a checkpoint
checkpoint = s3torchconnector.S3Checkpoint(
    region=REGION, 
    endpoint=S3_ENDPOINT, 
    s3client_config=config
)