# Get started with UltiHash Lightning-fast, S3-compatible object storage for modern workloads
Test with DockerSee what UltiHash can do with a local test cluster./pages/eaVW5uczhnyznH8lYDRe/files/ldn0BUJqgdT7idq4cDaq
Install on-premisesSet up UltiHash Self-Hosted on local infra/pages/oEcGpN5FzYXMvz7B4WUb/files/z229ScrvvrOtSoeLCvbS
Install on AWSSet up UltiHash Self-Hosted on the cloud with AWS/pages/CIvDCPrbVcJQiXzfeuWa/files/qivi619rRQFm6oFXdpc1
Set up UltiHash ServerlessGet a preconfigured, managed UltiHash cluster/pages/Smu5f2G5FpKTTYexrGEa/files/sIv7UpNBmKGJyylQkvnJ
MigrateEasily transfer your data from other platforms like S3./pages/vM6Vw9Rb92vTOMVfLbSV/files/Qfl3OnBq1vTivUz3hm3b
Use the S3-compatible APIInteract with UltiHash using the S3-compatible API/pages/Hjjh3PJpIW6xsKFqmsB9/files/SD1FLQHybFEXqohVcEYf
## UltiHash in a nutshell UltiHash is the high-performance object storage purpose-built for modern workloads, including AI and advanced analytics.
As businesses grapple with the rapid growth of unstructured data, storing and accessing it efficiently for AI and analytics becomes increasingly challenging. UltiHash addresses this by providing a scalable, Kubernetes-native storage platform with built-in deduplication that eliminates redundant data at the byte level, reducing storage requirements by up to 60%. Its S3-compatible API enables seamless integration with existing data lakes, lakehouses, and workflows across cloud, on-premises, or hybrid environments. This flexibility allows enterprises with frequently accessed datasets to optimise costs, maintain high performance, and avoid the lock-in and limitations of single-cloud architectures. # Get help + troubleshooting How to get help from our expert team of engineers, plus troubleshooting tips If you need a hand setting up UltiHash, or have any other questions, please [send a support ticket to support@ultihash.io](mailto:support@ultihash.io) - we'll get back to you within one business day. In your ticket, please include your Customer ID, which you can find on your [Dashboard](https://www.ultihash.io/user/dashboard).\ \ You can also join our [Discord community](https://ultihash.io/community) and get help from other users as well as UltiHash community managers. ## Support Tiers UltiHash Team provides support for both Freemium and Premium users. The details of the available support levels for each tier are described below: **Freemium Tier**\ For users of the Freemium tier, support is available via email at [support@ultihash.io](https://www.ultihash.io/terms-of-service#). While we aim to respond to all inquiries, no specific response times are guaranteed, and support is provided on a best-effort basis.\ Additionally, community support is available via the UltiHash Discord server, where both users and team members may be present and contribute to resolving questions or issues. **Premium Tier**\ The UltiHash team provides the following level of support for Customers of the Premium tier:\ Our support team is available Monday through Friday, from 9:00 AM to 5:00 PM CET, via dedicated channels (e.g., Slack or Microsoft Teams) and email.\ The email address for support requests is: .\ We ensure timely and efficient support for all inquiries related to the Software, including but not limited to technical issues, configuration assistance, and general usage guidance. *** ## Service Level Agreements * UltiHash Self Hosted (incl. AWS Marketplace customers): * UltiHash Serverless: *** ## Troubleshooting tips for frequent issues
Helm chart install or upgrade failure **Symptoms:** * `helm install` or `helm upgrade` hangs or returns an error * Application pods do not start * Helm status is stuck at `pending-install` or `failed` **Steps to resolve:** * **Inspect the Helm release status:** ```bash helm status -n ``` * **Check for resource creation errors or pending pods:** ```bash kubectl get pods -n ``` * **Describe a failing pod to view events and errors:** ```bash kubectl describe pod -n ``` * **Debug with Helm’s dry run mode:** ```bash helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --dry-run --values values.yaml --debug ``` * After the issue has been found and eliminated, process with install or upgrade further. **Recommendation:** Always use `--dry-run` and `--debug` to validate changes before applying them in production.
Missing or incorrect values in values.yaml **Symptoms:** * Helm fails with a rendering error * Application fails at runtime due to missing config (e.g., secrets, ports, env vars) **Steps to resolve:** * **Compare your values file with the chart defaults:** ``` helm show values oci://registry.ultihash.io/stable/ultihash-cluster ``` * **Test the rendered templates locally:** ``` helm template oci://registry.ultihash.io/stable/ultihash-cluster --values ``` * **Reapply the corrected configuration:** ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --values ``` **Recommendation:** Use a version-controlled values file and validate changes in a staging environment before rolling out to production. #### 3. Application pods stuck in `CrashLoopBackOff` or `ImagePullBackOff` **Purpose:** Diagnose runtime pod failures due to misconfiguration or image issues. **Symptoms:** * Pods keep restarting or cannot pull the container image **Steps to resolve:** * **Inspect the pod state:** ``` kubectl get pods -n ``` * **Check the logs of the failing pod:** ``` kubectl logs -n ``` * **Correct the config causing failure, then upgrade:** ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --values ``` **Recommendation:** Ensure that image repositories are accessible and secrets for private registries are correctly configured in the cluster.
# Test with Docker How to set up a local test environment for UltiHash using Docker {% embed url="" %} This guide will show you how to set up a test environment for UltiHash. You’ll set up a local environment using Docker Compose for container orchestration. {% hint style="info" %} Please note that to test UltiHash, you need to [sign up for a free account](https://www.ultihash.io/signup-login). {% endhint %} {% hint style="info" %} If you want to test UltiHash in a Kubernetes environment, you can do so with [Minikube](/installation/test-with-docker/test-ultihash). {% endhint %} **The main steps are as follows:** {% stepper %} {% step %} Install prerequisite tools {% endstep %} {% step %} Set up UltiHash with Docker Compose {% endstep %} {% step %} Integrate sample data + see space savings {% endstep %} {% endstepper %} {% hint style="danger" %} This setup is intended for local testing - not production use. {% endhint %} {% hint style="info" %} For now, UltiHash is only supported on Linux. This guide provides commands to be run in your terminal, and assumes you're running Ubuntu LTS on an AMD64 (x86\_64) architecture. Other distributions and ARM architectures should work fine, although some commands may need slight adjustment. {% endhint %} ## 1. Install prerequisite tools Before you start setting up the UltiHash cluster, you need some tools installed. If you already have any of these installed, you can simply skip that step. {% stepper %} {% step %} #### Install Docker Engine Docker provides a containerized virtual environment for UltiHash to run on. You can find general instructions for installing Docker Engine at [docs.docker.com/engine/install](https://docs.docker.com/engine/install/). **To quickly install, run:** ```bash # Linux installation: Update package index, install prerequisites, and set up Docker’s GPG key sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the Docker repository to Apt sources and update package index echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update # Install Docker Engine, CLI, and related plugins sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin ``` *** After installing Docker, you may need to add your user to the Docker group. Run: ```bash sudo usermod -aG docker $USER ``` {% hint style="warning" %} **Make sure to restart your computer** at this stage to apply the group changes. {% endhint %} {% endstep %} {% step %} #### Install AWS CLI The AWS CLI is a unified tool to manage AWS services from the command line. You can find general instructions for installing the AWS CLI at [docs.aws.amazon.com/cli/latest/userguide/getting-started-install](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). **To quickly install, run:** ```bash # Download and unzip AWS CLI installer curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip # Install AWS CLI sudo ./aws/install ``` {% endstep %} {% step %} ### Install `boto3` (and `tdqm`) The Amazon Web Services (AWS) SDK for Python (often referred to as `boto3`, allows you to interact with AWS services programatically. **To install, run:** ```bash sudo apt install python3-boto3 ``` `tqdm` is a Python package that provides a progress bar, which will be used in the upload scripts. **To install, run:** ```bash sudo apt install python3-tqdm ``` {% endstep %} {% endstepper %} {% hint style="success" %} **Done!**\ \ You've successfully installed all the prerequisites for testing UltiHash. \ **Next, you'll set up your local cluster using Docker Compose.** {% endhint %} ## 2. Set up UltiHash with Docker Compose Now you’ll set up the local UltiHash environment using Docker Compose. This involves authenticating with the UltiHash registry, downloading the necessary configuration file, and running the UltiHash services locally. {% stepper %} {% step %} **Set up authentication with the registry**\ \ Before you can download and run UltiHash, you need to authenticate with the UltiHash registry. The registry is where the container images (required for running UltiHash) are stored. For this step, you'll need these credentials from your UltiHash [Dashboard](https://ultihash.io/dashboard): * **Registry login** * **Registry password** *** Log in to the UltiHash registry with your credentials: ```bash docker login registry.ultihash.io -u ``` {% hint style="warning" %} Make sure to replace `` with the 'Registry login' from your [Dashboard](https://ultihash.io/dashboard).\ \ When prompted for a password, enter the 'Registry password' from your [Dashboard](https://ultihash.io/dashboard). {% endhint %} {% endstep %} {% step %} **Download `compose.yaml`** The `compose.yaml` file is a Docker Compose configuration file that defines all the services, volumes, and settings needed to run UltiHash.
**Download:** {% file src="/files/raVrRDlSB64ifn5dZSvU" %} {% hint style="info" %} Trouble downloading? Try right-clicking and selecting 'Save link as...' or similar. {% endhint %} {% endstep %} {% step %} **Set up credentials and license** To enable access to UltiHash services, you need to export your credentials and license key. These environment variables will be used for authentication. **Run the following commands:** ```bash export AWS_ACCESS_KEY_ID="TEST-USER" export AWS_SECRET_ACCESS_KEY="SECRET" export UH_CUSTOMER_ID="" export UH_ACCESS_TOKEN="" export UH_MONITORING_TOKEN="" ``` {% hint style="warning" %} Make sure to replace `` , `` , and `` with the 'Customer ID', 'Access token', and 'Monitoring token' from your [Dashboard](https://ultihash.io/dashboard). {% endhint %} {% endstep %} {% step %} **Start UltiHash services**\ \ **Change the working directory** to the folder where you saved `compose.yaml`. For example: ```bash cd ~/Downloads ``` \ **Start the UltiHash cluster**: ```bash docker compose up -d ``` \ If successful, Docker Compose will download the necessary images (if they’re not already cached) and start the UltiHash services. {% endstep %} {% endstepper %} {% hint style="success" %} **Done!** You’ve successfully set up your local UltiHash cluster.\ \ **Next, let's integrate sample data + see space savings.** {% endhint %} ## 3. Integrate sample data + see space savings Now that UltiHash is running on your local cluster, let's integrate some sample data.
{% stepper %} {% step %} ### Prepare dataset If you have a dataset you want to test already, you can skip this step. **Alternatively, you can download one of these datasets from Kaggle:**
DICOM files of brain MRI scans1.51 GBhttps://www.kaggle.com/datasets/amritpal333/adni4dicomnano10514/
JPGs of driving scenarios2.41 GBhttps://www.kaggle.com/datasets/zaynena/selfdriving-car-simulator
PNGs of synthetic textures with defects5.89 GBhttps://www.kaggle.com/datasets/mhskjelvareid/dagm-2007-competition-dataset-optical-inspection
WAVs of human speech for emotion recognition0.33 GBhttps://www.kaggle.com/datasets/barelydedicated/savee-database
TIFF images of climate data16.28 GBhttps://www.kaggle.com/datasets/abireltaief/highresolution-geotiff-images-of-climatic-data
CSV tables of symptoms140 KBhttps://www.kaggle.com/datasets/kaushil268/disease-prediction-using-machine-learning
{% hint style="warning" %} Remember to unzip your test dataset if you download it from Kaggle. {% endhint %} {% hint style="info" %} UltiHash's deduplication can have significantly different results depending on the dataset integrated. For testing, try datasets likely to contain repeated content - like document libraries with shared templates, multimedia collections with common graphics, or code repositories. {% endhint %} {% endstep %} {% step %} ### Create a bucket Object storage systems like UltiHash use a top-level container called a **bucket**. To facilitate scalability, buckets don’t have a traditional hierarchical folder structure: instead, each object in a bucket has a unique key (which can resemble a file path, simulating directories). **To create a bucket, run:** ```bash aws s3api create-bucket --bucket --endpoint-url http://127.0.0.1:8080 ``` {% hint style="warning" %} Make sure to replace `` with your chosen bucket name, e.g. `test-bucket`. {% endhint %} *** **You can see your newly created bucket by running:** ```bash aws s3api list-buckets --endpoint-url http://127.0.0.1:8080 ```
{% endstep %} {% step %} ### Download scripts We've prepared some scripts to make the testing process easier. Download the following scripts for uploading and downloading: {% file src="/files/ZLfMUvRRAfxr9OYWZXMQ" %} {% file src="/files/zcflMM05G31JKqdLbSpR" %} {% file src="/files/sZqR8ejBcOD6QRb9FngD" %} {% file src="/files/FXIUapwSGEPRbLvXIKG3" %} {% hint style="info" %} Trouble downloading these scripts? Try right-clicking and selecting 'Save link as...' or similar. {% endhint %} {% endstep %} {% step %} ### Integrate sample data Now that you have a bucket in which to put objects, let's use the upload script to integrate your sample data. **To integrate your dataset, run:** ```bash python3 --url http://127.0.0.1:8080 --bucket ``` {% hint style="warning" %} Make sure to replace `` with the path to the upload script you downloaded, e.g. `/home/user/Downloads/uh-upload.py`. Also replace `` with your bucket name. Finally, replace `` with the path to the directory for the dataset you prepared or downloaded, e.g. `/home/user/Downloads/test-dataset`. {% endhint %} A bar should display the ongoing progress of your integration. *** Once the integration is complete, you can run the following command to see your objects: ```bash aws s3api list-objects --endpoint-url http://127.0.0.1:8080 --bucket --output text | cat ``` {% hint style="warning" %} Make sure to replace `` with your bucket name. {% endhint %} *** You can also download an entire bucket by running: ```bash python3 --url http://127.0.0.1:8080 --path ``` {% hint style="warning" %} Make sure to replace `` with the path to the upload script you downloaded, e.g. `/home/user/Downloads/uh-download.py`. Also replace `` with the path to the directory you want to download the bucket to, e.g. `/home/user/Downloads`. Finally, replace `` with the name of the bucket to download. {% endhint %} {% endstep %} {% step %} ### See space savings in your cluster You can see the storage space UltiHash is saving across the entire cluster by running the `uh-see-space-savings` script: ```bash python3 --url http://127.0.0.1:8080 ``` {% hint style="warning" %} Make sure to replace `` with the path to the upload script you downloaded, e.g. `/home/user/Downloads/uh-see-space-savings.py`. {% endhint %} {% endstep %} {% endstepper %}
{% hint style="success" %} **Done!**\ \ You’ve successfully integrated a dataset to a local test cluster, and can see the space saved by UltiHash's built-in deduplication. {% endhint %} # Test UltiHash (with Minikube) This guide will show you how to set up a test environment for UltiHash. You'll set up a local Kubernetes environment using Minikube, containerized by Docker. **The main steps are as follows:** {% stepper %} {% step %} Install prerequisite tools {% endstep %} {% step %} Set up a local Kubernetes cluster {% endstep %} {% step %} Deploy UltiHash with Helm {% endstep %} {% step %} Integrate sample data + see space savings {% endstep %} {% endstepper %} {% hint style="danger" %} This setup is intended for local testing - not production use. {% endhint %} {% hint style="info" %} For now, UltiHash is only supported on Linux. This guide provides commands to be run in your terminal, and assumes you're running Ubuntu LTS on an AMD64 (x86\_64) architecture. Other distributions and ARM architectures should work fine, although some commands may need slight adjustment. {% endhint %} ## 1. Install prerequisite tools Before you start setting up the UltiHash cluster, you need some tools installed. If you already have any of these installed, you can simply skip that step. {% stepper %} {% step %} #### Install Docker Engine Docker provides a containerized virtual environment for Minikube to run on. You can find general instructions for installing Docker Engine at [docs.docker.com/engine/install](https://docs.docker.com/engine/install/). **To quickly install, run:** ```bash # Linux installation: Update package index, install prerequisites, and set up Docker’s GPG key sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the Docker repository to Apt sources and update package index echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update # Install Docker Engine, CLI, and related plugins sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin ``` *** After installing Docker, you may need to add your user to the Docker group. Run: ```bash sudo usermod -aG docker $USER ``` {% hint style="warning" %} **Make sure to restart your computer** at this stage to apply the group changes. {% endhint %} {% endstep %} {% step %} #### Install Minikube Minikube lets you run a single-node Kubernetes cluster locally for development and testing. In this case we will use Docker as its container engine. You can find general instructions for installing Minikube at [minikube.sigs.k8s.io/docs/start](https://minikube.sigs.k8s.io/docs/start/). **To quickly install, run:** ```bash # Download latest Minikube for Linux AMD64 curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 # Install Minikube and clean up sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64 ``` {% endstep %} {% step %} #### Install kubectl Kubectl is a tool for interacting with your Kubernetes clusters, allowing you to manage and deploy applications, inspect resources, and troubleshoot issues. You can find general instructions for installing Kubectl at [kubernetes.io/docs/tasks/tools/install-kubectl-linux](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/). **To quickly install, run:** ```bash # Download kubectl for Linux AMD64 curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" # Install kubectl to /usr/local/bin with root permissions sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl ``` {% endstep %} {% step %} #### Install Helm Helm is a package manager for Kubernetes. It makes it easy to manage, install, and update applications (like UltiHash) on your clusters. You can find general instructions for installing Helm at [https://helm.sh/docs/intro/install](https://helm.sh/docs/intro/install/). **To quickly install, run:** ```bash # Add Helm's GPG key, set up repo, and update packages curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null sudo apt-get install apt-transport-https --yes echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list sudo apt-get update # Install Helm sudo apt-get install helm ``` {% endstep %} {% step %} #### Install AWS CLI The AWS CLI is a unified tool to manage AWS services from the command line. You can find general instructions for installing the AWS CLI at [docs.aws.amazon.com/cli/latest/userguide/getting-started-install](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). **To quickly install, run:** ```bash # Download and unzip AWS CLI installer curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip # Install AWS CLI sudo ./aws/install ``` {% endstep %} {% step %} ### Install `boto3` (and `tdqm`) The Amazon Web Services (AWS) SDK for Python (often referred to as `boto3`, allows you to interact with AWS services programatically. **To install, run:** ```bash sudo apt install python3-boto3 ``` `tqdm` is a Python package that provides a progress bar, which will be used in the upload scripts. **To install, run:** ```bash sudo apt install python3-tqdm ``` {% endstep %} {% endstepper %} {% hint style="success" %} **Done!**\ \ You've successfully installed all the prerequisites for testing UltiHash. \ **Next, you'll set up your local Kubernetes cluster using Minikube.** {% endhint %} ## 2. Set up a local Kubernetes cluster Now you’ll set up the local Kubernetes cluster for UltiHash. This involves setting up the Minikube environment, creating a dedicated namespace for UltiHash, and provisioning Kubernetes with necessary credentials using secrets.
{% stepper %} {% step %} #### Set up Minikube environment Create a local Kubernetes cluster: ```bash minikube start --cpus='no-limit' --memory='no-limit' ``` {% hint style="info" %} This command removes the limits on CPU and memory usage to ensure performance with larger uploads; remove these arguments if you prefer. {% endhint %} *** Next, ensure `kubectl` has access and the cluster node has been provisioned: ```bash kubectl get nodes ``` You should see something like: ``` NAME STATUS ROLES AGE VERSION minikube Ready control-plane 32s v1.28.3 ``` *** The [Nginx Ingress Controller](https://github.com/kubernetes/ingress-nginx) manages internal routing within your Minikube cluster, allowing UltiHash services to communicate efficiently. Install it with: ``` minikube addons enable ingress ``` {% endstep %} {% step %} #### Create namespace Choose and create a namespace for the UltiHash installation: ```bash kubectl create ns ``` {% hint style="warning" %} Make sure to replace `` with your chosen namespace, e.g. `uh-test-namespace`. {% endhint %} {% endstep %} {% step %} #### Provision credentials using secrets For this step, you'll need these credentials from your UltiHash [Dashboard](https://ultihash.io/dashboard): * **Registry login** * **Registry password** * **Customer ID** * **Access token** * **Monitoring token** Provision a secret to store the UltiHash registry credentials: ```bash kubectl create secret docker-registry registry-credentials -n --docker-server='registry.ultihash.io' --docker-username='' --docker-password='' ``` {% hint style="warning" %} Make sure to replace \ with your chosen name. Also replace \ and \ with the credentials from your [Dashboard](https://ultihash.io/dashboard). {% endhint %} *** Provision a secret to store the UltiHash credentials and monitoring token: ```bash kubectl create secret generic ultihash -n --from-literal=customer_id='' --from-literal=access_token='' --from-literal=token='' ``` {% hint style="warning" %} Make sure to replace \ with your chosen name. Also replace \, \, and \ with the credentials from your [Dashboard](https://ultihash.io/dashboard). {% endhint %} {% endstep %} {% endstepper %} {% hint style="success" %} **Done!** You’ve successfully set up your local Kubernetes cluster.\ \ **Next, let's configure and deploy UltiHash with Helm.** {% endhint %} ## 3. Configure and deploy Helm chart Now that your Kubernetes environment is ready, it’s time to configure and deploy the UltiHash Helm chart. This will set up the necessary resources and configurations to run UltiHash in your cluster.
{% stepper %} {% step %} #### Create the `values.yaml` configuration file Create a file named `values.yaml` with any text editor. This file will define the settings for your UltiHash deployment. **Copy and paste the following content:** ```yaml etcd: replicaCount: 1 persistence: storageClass: standard database: primary: persistence: storageClass: standard size: 10Gi entrypoint: replicas: 1 ingress: host: storage: groups: - id: 0 type: ROUND_ROBIN storages: 1 size: 10Gi deduplicator: replicas: 1 storageClass: standard storageSize: 10Gi ``` {% hint style="info" %} If you want, you can adjust the number of replicas and storage size for your test. For storage size, you must give the size in gigabytes as `Gi` - not `gb` or similar. {% endhint %} *** **Save `values.yaml` in an easy-to-access place.**
{% endstep %} {% step %} #### Log in to the Helm chart registry For this step, you'll need these credentials from your UltiHash [Dashboard](https://ultihash.io/dashboard): * **Registry login** * **Registry password** **To log in to the registry, run:** ```bash helm registry login registry.ultihash.io -u ``` {% hint style="warning" %} Make sure to replace \ with the credentials from your [Dashboard](https://ultihash.io/dashboard). {% endhint %} *** **Enter your registry password when prompted.** You won't be able to see it when you type it in. {% endstep %} {% step %} #### Deploy Helm chart Deploy the Helm chart to your cluster with a release name of your choosing: ``` helm install oci://registry.ultihash.io/stable/ultihash-cluster -n --values --wait --timeout 10m ``` {% hint style="warning" %} Make sure to replace `` with your chosen release name, e.g. `uh-test-release`.\ \ Also make sure to replace `` with the actual path to your `values.yaml` file, such as `/home/user/values.yaml`. On Ubuntu, copying the file in the file browser automatically copies the path to your clipboard as well. {% endhint %} {% hint style="info" %} This deployment may take up to 10 minutes. {% endhint %}
{% endstep %} {% step %} #### Get access to the UltiHash cluster Run the following command to retrieve the IP address of your Minikube cluster and set it as the `ClusterURL`: ```bash export ClusterURL=http://`minikube ip` ``` {% hint style="info" %} This command saves the URL of your UltiHash cluster in an environment variable called `ClusterURL`, which you’ll use to connect to the cluster locally. {% endhint %} {% endstep %} {% step %} #### Retrieve root user credentials UltiHash requires AWS-style credentials for access. To obtain the root user’s access and secret keys, run: ```bash export AWS_ACCESS_KEY_ID=`kubectl get secret -super-user-credentials -n -o jsonpath="{.data.access-key-id}" | base64 --decode` export AWS_SECRET_ACCESS_KEY=`kubectl get secret -super-user-credentials -n -o jsonpath="{.data.secret-key}" | base64 --decode` ``` {% hint style="warning" %} Make sure to replace `` and `` with your chosen names - both times. {% endhint %} {% endstep %} {% endstepper %} {% hint style="success" %} **Done!**\ \ You’ve successfully deployed UltiHash to a local Kubernetes test environment.\ \ **Next, continue to integrate some sample data.** {% endhint %} ## 4. Integrate sample data + see space savings Now that UltiHash is running on your local Kubernetes cluster, let's integrate some sample data.
{% stepper %} {% step %} ### Prepare dataset If you have a dataset you want to test already, you can skip this step. **Alternatively, you can download one of these datasets from Kaggle:**
Scans of human and animal anatomy1.48 GB, RAWhttps://www.kaggle.com/datasets/imaginar2t/cbctdata
Images from self-driving vehicle simulation2.6 GB, JPGhttps://www.kaggle.com/datasets/zaynena/selfdriving-car-simulator
Images of textures for defect detection6 GB, PNGhttps://www.kaggle.com/datasets/mhskjelvareid/dagm-2007-competition-dataset-optical-inspection
Images of climate data16 GB, TIFFhttps://www.kaggle.com/datasets/abireltaief/highresolution-geotiff-images-of-climatic-data
Logs of symptoms for disease prediction1.4 MB, CSVhttps://www.kaggle.com/datasets/kaushil268/disease-prediction-using-machine-learning
{% hint style="warning" %} Remember to unzip your test dataset if you download it from Kaggle. {% endhint %} {% hint style="info" %} UltiHash's deduplication can have significantly different results depending on the dataset integrated. For testing, try datasets likely to contain repeated content - like document libraries with shared templates, multimedia collections with common graphics, or code repositories. {% endhint %} {% endstep %} {% step %} ### Create a bucket Object storage systems like UltiHash use a top-level container called a **bucket**. To facilitate scalability, buckets don’t have a traditional hierarchical folder structure: instead, each object in a bucket has a unique key (which can resemble a file path, simulating directories). **To create a bucket, run:** ```bash aws s3api create-bucket --bucket --endpoint-url $ClusterURL ``` {% hint style="warning" %} Make sure to replace `` with your chosen bucket name, e.g. `test-bucket`. {% endhint %} *** **You can see your newly created bucket by running:** ```bash aws s3api list-buckets --endpoint-url $ClusterURL ```
{% endstep %} {% step %} ### Download scripts We've prepared some scripts to make the testing process easier. Download the following scripts for uploading and downloading: {% file src="/files/ZLfMUvRRAfxr9OYWZXMQ" %} {% file src="/files/zcflMM05G31JKqdLbSpR" %} {% file src="/files/5R0L9A4yLdgtpn79pbIq" %} {% file src="/files/FXIUapwSGEPRbLvXIKG3" %} {% hint style="info" %} Trouble downloading these scripts? Try right-clicking and selecting 'Save link as...' or similar. {% endhint %} {% endstep %} {% step %} ### Integrate sample data Now that you have a bucket in which to put objects, let's use the upload script to integrate your sample data. **To integrate your dataset, run:** ```bash python3 --url $ClusterURL --bucket ``` {% hint style="warning" %} Make sure to replace `` with the path to the upload script you downloaded, e.g. `/home/user/Downloads/uh-upload.py`. Also replace `` with your bucket name. Finally, replace `` with the path to the directory for the dataset you prepared or downloaded, e.g. `/home/user/Downloads/test-dataset`. {% endhint %} A bar should display the ongoing progress of your integration. *** Once the integration is complete, you can run the following command to see your objects: ```bash aws s3api list-objects --endpoint-url $ClusterURL --bucket --output text | cat ``` {% hint style="warning" %} Make sure to replace `` with your bucket name. {% endhint %} *** You can also download an entire bucket by running: ```bash python3 --url $ClusterURL --path ``` {% hint style="warning" %} Make sure to replace `` with the path to the upload script you downloaded, e.g. `/home/user/Downloads/uh-download.py`. Also replace `` with the path to the directory you want to download the bucket to, e.g. `/home/user/Downloads`. Finally, replace `` with the name of the bucket to download. {% endhint %} {% endstep %} {% step %} ### See space savings in your cluster You can see the storage space UltiHash is saving across the entire cluster by running the `uh-see-space-savings` script: ```bash python3 --url $ClusterURL ``` {% hint style="warning" %} Make sure to replace `` with the path to the upload script you downloaded, e.g. `/home/user/Downloads/uh-see-space-savings.py`. {% endhint %} {% endstep %} {% endstepper %}
{% hint style="success" %} **Done!**\ \ You’ve successfully integrated a dataset to a local test cluster, and can see the space saved by UltiHash's built-in deduplication. {% endhint %} # Install Self-Hosted on-premises How to set up UltiHash Self-Hosted on your local infrastructure with Kubernetes On-premises environments remain vital for many organizations. Through its Kubernetes-native architecture, UltiHash supports easy scaling and load balancing, ensuring that systems can handle fluctuating workloads with minimal reconfiguration. This design ensures that on-premises deployments remain flexible and customizable, allowing businesses to maintain control over their infrastructure. This guide provides a detailed walkthrough for setting up an UltiHash cluster in a Kubernetes environment, whether on-premises or in a cloud environment. The process is divided into four main steps: 1. **Prerequisites**: Gather the necessary credentials, tools, and environment configurations. 2. **Cluster setup**: Configure your Kubernetes cluster, including creating namespaces and provisioning secrets. 3. **Helm installation**: Deploy UltiHash using Helm, customizing the setup for your specific environment. 4. **Post-installation**: Verify the installation.
System hardware requirements * **Storage:** NVMe SSDs are required for optimal disk performance. * **Network:** 10 Gbps interface minimum between nodes. * **Kubernetes:** Version 1.20+ with Nginx Ingress and a CSI Controller installed. * **Containerization:** Docker 19.03+ or Containerd 1.3+. * **Helm:** Version 3.x. * **On-Premises:** Minimum of 1 Kubernetes node with NVMe SSDs. Resource needs will vary depending on the amount of data being stored and managed. For best performance, especially with larger datasets, it’s essential to provision additional resources accordingly.
## Step 1: Prerequisites Before you begin the installation, ensure you have the following: * **Skills:** good knowledge of Kubernetes, kubectl, and helm. * **UltiHash Account**: Sign up at [ultihash.io/signup](https://ultihash.io/signup) and verify your email. * **Credentials**: After signing up on the UltiHash website, you will get the following credentials on [your dashboard](https://www.ultihash.io/user/dashboard): * **Registry login and password** (referred to as `registry_login` and `registry_password`). * **Customer ID** (referred to as `customer_id`). * **Access token** (referred to as `access_token`). * **Monitoring token** (referred to as `monitoring_token`). * **Kubernetes cluster**: * **Version**: Ensure you have a Kubernetes cluster running version 1.20 or higher. * **Controllers**: * **Ingress controller**: Exposes the UltiHash cluster API endpoint outside the Kubernetes cluster. * **CSI controller**: Manages persistent volumes. * Note: You can use any Kubernetes version starting from 1.20, and any CSI controller that dynamically provisions and attaches persistent volumes. For optimal performance, use a CSI controller that imposes the least disk performance degradation. * **Local environment**: * **kubectl**: Ensure Kubernetes command line tool `kubectl` is installed and configured to access the cluster. * **Helm**: Install Kubernetes package manager Helm (version 3.x) to manage Kubernetes packages. ## Step 2: Cluster setup 1. **Namespace creation**: * Create a Kubernetes namespace for the UltiHash installation: ```bash kubectl create ns ``` * Replace `` with your desired namespace name. 2. **Secrets provisioning**: * **Registry credentials**: Provision a secret in Kubernetes to store the UltiHash registry credentials: ```bash kubectl create secret docker-registry registry-credentials -n --docker-server='registry.ultihash.io' --docker-username='' --docker-password='' ``` * Replace `` with the namespace name. Replace ``, and `` with the appropriate values obtained from your [dashboard](https://www.ultihash.io/user/dashboard) on the UltiHash website. * **Ultihash credentials and monitoring token**: Create a secret in Kubernetes for the license key and monitoring token: ```bash kubectl create secret generic ultihash -n --from-literal=customer_id='' --from-literal=access_token='' --from-literal=token='' ``` * Replace `` with the namespace name. Replace `,,` and `` with the corresponding values found on your [UltiHash dashboard](https://www.ultihash.io/user/dashboard). ## Step 3: Helm installation 1. **Helm chart deployment**: * Log into the UltiHash registry with your `registry_login` and `registry_password` : ```bash helm registry login registry.ultihash.io ``` * Deploy the Helm chart with a specific release name and namespace: ```bash helm install oci://registry.ultihash.io/stable/ultihash-cluster -n --values values.yaml --wait ``` * Replace `` and `namespace` with your chosen names. `values.yaml` should be configured as described below. 2. **Component configuration**: * Customize the `values.yaml` file with the necessary configurations: * **Storage class**: Specify the storage class name created by your CSI controller. * **Domain name**: Enter a valid domain name for your UltiHash cluster. * **Service replicas and storage size**: Adjust the number of replicas and storage size for services like `etcd`, `entrypoint`, `storage`, and `deduplicator` based on your requirements. ```yaml global: logLevel: INFO # Default log level for all UltiHash services. Valid values are DEBUG, INFO, WARN, ERROR, or FATAL telemetryExportInterval: 30000 # Export interval for UltiHash services telemetry (in milliseconds). Could be overriden individually for each UltiHash service etcd: replicaCount: persistence: storageClass: database: primary: persistence: storageClass: size: entrypoint: replicas: ingress: host: # FQDN to expose the entrypoint outside the cluster storage: groups: - id: 0 type: ROUND_ROBIN storages: 1 storageClass: size: deduplicator: replicas: storageClass: storageSize: exporter: enabled: true ``` ## Step 4: Post-installation 1. **Verification**: * After deployment, verify that all services are running correctly by checking the Kubernetes namespace: ```bash kubectl get all -n ``` Replace `` with the namespace where UltiHash cluster has been deployed. * Ensure that all pods are either in the `Running` or in the `Completed` state with no errors. 2. **Get access to the UltiHash cluster**: * Obtain the UltiHash root user credentials: ```bash # Obtain credentials for the UltiHash root user aws_access_key_id=`kubectl get secret -super-user-credentials -n -o jsonpath="{.data.access-key-id}" | base64 --decode` aws_secret_access_key=`kubectl get secret -super-user-credentials -n -o jsonpath="{.data.secret-key}" | base64 --decode` # Set the credentials for the UltiHash root user export AWS_ACCESS_KEY_ID=$aws_access_key_id export AWS_SECRET_ACCESS_KEY=$aws_secret_access_key ``` * Replace `` and `` with the Helm release name and namespace name correspondingly. * Use AWS CLI and AWS SDK to interact with the UltiHash cluster: ```bash aws s3api list-buckets --endpoint-url ``` * Replace `` with the appropriate scheme: either `https://` or `http://`, depending on whether your `entrypoint.ingress` object in the Helm values has been configured with or without TLS. The `` corresponds to the domain name chosen for the UltiHash cluster, as set in the `entrypoint.ingress.host` object. *** ### Frequent issues troubleshooting
Helm chart install or upgrade failure **Symptoms:** * `helm install` or `helm upgrade` hangs or returns an error * Application pods do not start * Helm status is stuck at `pending-install` or `failed` **Steps to resolve:** * **Inspect the Helm release status:** ```bash helm status -n ``` * **Check for resource creation errors or pending pods:** ```bash kubectl get pods -n ``` * **Describe a failing pod to view events and errors:** ```bash kubectl describe pod -n ``` * **Debug with Helm’s dry run mode:** ```bash helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --dry-run --values values.yaml --debug ``` * After the issue has been found and eliminated, process with install or upgrade further. **Recommendation:** Always use `--dry-run` and `--debug` to validate changes before applying them in production.
Missing or incorrect values in values.yaml **Symptoms:** * Helm fails with a rendering error * Application fails at runtime due to missing config (e.g., secrets, ports, env vars) **Steps to resolve:** * **Compare your values file with the chart defaults:** ``` helm show values oci://registry.ultihash.io/stable/ultihash-cluster ``` * **Test the rendered templates locally:** ``` helm template oci://registry.ultihash.io/stable/ultihash-cluster --values ``` * **Reapply the corrected configuration:** ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --values ``` **Recommendation:** Use a version-controlled values file and validate changes in a staging environment before rolling out to production. #### 3. Application pods stuck in `CrashLoopBackOff` or `ImagePullBackOff` **Purpose:** Diagnose runtime pod failures due to misconfiguration or image issues. **Symptoms:** * Pods keep restarting or cannot pull the container image **Steps to resolve:** * **Inspect the pod state:** ``` kubectl get pods -n ``` * **Check the logs of the failing pod:** ``` kubectl logs -n ``` * **Correct the config causing failure, then upgrade:** ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --values ``` **Recommendation:** Ensure that image repositories are accessible and secrets for private registries are correctly configured in the cluster.
# Install Self-Hosted on AWS How to set up UltiHash Self-Hosted in the cloud with AWS + Kubernetes For cloud deployments, UltiHash integrates seamlessly with AWS and Elastic Block Storage (EBS). Unlike traditional object storage solutions that charge based on the number of requests, which leads to intransparent and unpredictable costs, UltiHash eliminates these uncertainties. Instead, users can choose from different storage classes based on performance needs, which is especially useful for I/O-intensive and mission-critical workloads. *** This guide describes the full installation process of UltiHash in AWS environment, including: * provision of EKS cluster in a dedicated VPC * deployment of the essential Kubernetes controllers * installation of UltiHash on the EKS cluster This guide outlines the recommended UltiHash setup for managing 10 TB of data. The setup diagram is shown below. UltiHash cluster is deployed on a single EC2 instance of type `r8g.4xlarge` with a network load balancer that routes traffic to it. The cluster uses `gp3` volumes optimized for performance, ensuring efficient storage management. In case you have other storage requirements, you may freely change the volume sizes in the configuration. You are free to select any EC2 instance type and EBS volume configurations for production purposes based on your specific needs. The diagram below depicts the resources to deploy in an AWS account by the Terraform scripts. By default the deployment is done in a single AZ (see the diagram below). However it could be adjusted; see [#eks-cluster-setup](#eks-cluster-setup "mention")

Diagram of the deployed resources

**Expected performance:** * Write throughput: up to 200 MB/s * Read throughput: up to 1000 MB/s **Expected costs:** * Hourly: * EC2 cost: 1.14 USD * EBS cost: 1.58 USD * UltiHash Pay-as-you-go license cost: 0.14 USD * Monthly: * EC2 cost: 829.98 USD * EBS cost: 1152.38 USD * UltiHash 1 month subscription license cost: 92.16 USD UltiHash license is available as pay-as-you-go license with pricing for the number of used GiBs per hour or subscription license with pricing for the number of used GiBs for the subscription duration available in 1 month, 12 month, 24 month and 36 month contract variations. **List of billable AWS services:** * mandatory: EKS, EC2, S3, KMS * optional: SQS, Eventbridge **Estimated amount of time to complete a deployment:** \~45 minutes.
System hardware requirements * **Storage:** NVMe SSDs are required for optimal disk performance. * **Network:** 10 Gbps interface minimum between nodes. * **Kubernetes:** Version 1.20+ with Nginx Ingress and a CSI Controller installed. * **Containerization:** Docker 19.03+ or Containerd 1.3+. * **Helm:** Version 3.x. * **Cloud:** for AWS, EC2 instances with Elastic Block Storage (EBS). GCP/Azure support is in development. Resource needs will vary depending on the amount of data being stored and managed. For best performance, especially with larger datasets, it’s essential to provision additional resources accordingly.
*** {% stepper %} {% step %} ## Prerequisites #### Skills * good knowledge of the following AWS services: IAM, VPC, EKS, EC2 * high-level knowledge of Terraform #### Remote Environment * access to an AWS account * **Warning:** do not use AWS account root to provision and manage the deployed resources! Instead create an IAM user that has sufficient privileges to manage these AWS services: IAM, VPC, EKS, EC2. * IAM permissions required to deploy and manage UltiHash cluster are listed in [#iam-permissions-required-to-deploy-and-manage-an-uh-cluster](#iam-permissions-required-to-deploy-and-manage-an-uh-cluster "mention") * make sure your AWS account has sufficient limits before deploying UltiHash cluster: [#manage-aws-service-limits](#manage-aws-service-limits "mention") * UltiHash cluster could be installed in any region where[ AWS EKS is supported](https://docs.aws.amazon.com/general/latest/gr/eks.html) #### Local Environment * [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-methods) AWS CLI * [installed](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) terraform * [installed](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#v1280) kubectl of version 1.30 * personal credentials found on the [UltiHash dashboard](https://www.ultihash.io/user/dashboard) {% endstep %} {% step %} ## Setup S3 Bucket for Terraform States Since the Terraform state for this setup has to be stored on S3, need to provision a dedicated S3 bucket. Execute the following command, replacing the `` and `` placeholders: ``` aws s3api create-bucket --bucket --create-bucket-configuration LocationConstraint= --region ``` The S3 bucket will be created with the default encryption of type SSE-S3 (AWS managed KMS key) enabled. {% endstep %} {% step %} ## Clone the scripts repository Clone the repository by executing the command below: ``` git clone https://github.com/UltiHash/scripts.git ``` Later its code will be required to setup UltiHash in AWS environment. {% endstep %} {% step %} ## EKS Cluster Setup Since UltiHash has to be deployed on Kubernetes cluster, need to provision EKS cluster on AWS. For this purpose use [this Terraform project](https://github.com/UltiHash/scripts/tree/main/terraform/aws/eks-cluster). The project deploys a dedicated VPC and provisions there an EKS cluster with a single `c5.large` machine to host the essential Kubernetes controllers. **Note:** by default the EKS cluster is provisioned with a public endpoint that is reachable over the Internet. In case the EKS cluster endpoint should be private, change the parameter [cluster\_endpoint\_public\_access](https://github.com/UltiHash/scripts/blob/f1790a97440bc42cb902e188275a63622f3375c0/terraform/aws/eks-cluster/eks.tf#L32C3-L32C40) from *true* to *false*. Once the [scripts](#clone-the-scripts-repository) repository is cloned, perform the following actions to deploy the Terraform project: 1. Update the `bucket name` and its `region` in the [main.tf](https://github.com/UltiHash/scripts/blob/main/terraform/aws/eks-cluster/main.tf) with the onces done at [the previous step](#setup-s3-bucket-for-terraform-states). 2. Update the configuration in [config.tfvars](https://github.com/UltiHash/scripts/blob/main/terraform/aws/eks-cluster/config.tfvars). The only required change is the parameter `cluster_admins` - specify the list of ARNs of IAM users and/or IAM roles that need to have access to the provisioned EKS cluster. Other parameters could be left intact. 3. Initialize and apply the Terraform project ``` cd scripts/terraform/aws/eks-cluster terraform init terraform apply --var-file config.tfvars ``` Wait until the installation is completed. Make sure the access to the EKS cluster has been granted to the required IAM users and roles To check that, download the `kubeconfig` for the EKS cluster, executing the command below. Replace the `` (by default `ultihash-test`) and the `` (by default `eu-central-1`) with the corresponding values defined in [config.tfvars](https://github.com/UltiHash/scripts/blob/main/terraform/aws/eks-cluster/config.tfvars). ``` aws eks update-kubeconfig --name --region ``` Execute the following `kubectl` command to check the available EKS cluster nodes: ``` kubectl get nodes ``` The command has to output a name of a single provisioned EC2 instance. {% endstep %} {% step %} ## Install Controllers on EKS The next step is installation of the essential Kubernetes controllers on the provisioned EKS cluster. For this purpose use [this Terraform project](https://github.com/UltiHash/scripts/tree/main/terraform/aws/eks-cluster-controllers). The project deploys the following Kuberentes controllers on the EKS cluster: * `Nginx Ingress` - exposes UltiHash outside of the EKS cluster with a Network Load Balancer. * `Load Balancer Controller` - provisions a Network Load Balancer for the `Nginx Ingress` controller. * `Karpenter` - provisions EC2 instances on-demand to host UltiHash workloads. * `EBS CSI Driver` - CSI controller that automatically provisions persistent volumes the UltiHash workfloads. The volumes are based on `gp3` storage class and optimised in terms of performance. The default storage class provisions unencrypted EBS volumes. To provision encrypted EBS volumes, create a new storage class like [this](https://gist.github.com/AndrzejKomarnicki/3926bae40060cb07a66a3f193cbbcd7e). Perform the following actions to deploy the Terraform project: 1. Update the `bucket name` and its `region` in the [main.tf](https://github.com/UltiHash/scripts/blob/main/terraform/aws/eks-cluster-controllers/main.tf) with the onces done at [the previous step](#setup-s3-bucket-for-terraform-states). 2. Update the configuration in [config.tfvars](https://github.com/UltiHash/scripts/blob/main/terraform/aws/eks-cluster-controllers/config.tfvars) if required. The helm values for the deployed controlers are found [here](https://github.com/UltiHash/scripts/tree/main/terraform/aws/eks-cluster-controllers/controllers-values). It is not recommended to change any of these configurations, the only parameter that should be selected in advance is the `Network Load Balancer type` (`internal` or `internet-facing`) in this [file](https://github.com/UltiHash/scripts/blob/main/terraform/aws/eks-cluster-controllers/controllers-values/nginx-ingress.yaml). 3. In case it is required to change the instance type for the UltiHash services, update it in the following [Karpenter manifest](https://github.com/UltiHash/scripts/blob/main/terraform/aws/eks-cluster-controllers/karpenter-manifests/10TB.node-pool.yaml). 4. Initialize and apply the Terraform project ``` cd scripts/terraform/aws/eks-cluster-controllers terraform init terraform apply --var-file config.tfvars ``` Wait until the installation is completed. A Network Load balancer should be provisioned in the same region as the EKS cluster. {% endstep %} {% step %} ## UltiHash installation The last step is installation of UltiHash. For this purpose use [this Terraform project](https://github.com/UltiHash/scripts/tree/main/terraform/aws/ultihash). Perform the following actions to deploy the Terraform project: 1. Update the `bucket name` and its `region` in the [main.tf](https://github.com/UltiHash/scripts/blob/main/terraform/aws/ultihash/main.tf) with the ones done at [the previous step](#setup-s3-bucket-for-terraform-states). 2. Update the configuration in [config.tfvars](https://github.com/UltiHash/scripts/blob/main/terraform/aws/ultihash/config.tfvars) with the credentials obtained from your account on [ultihash.io](https://www.ultihash.io/). The credentials in the `config.tfvars` are mocked. The helm values for UltiHash are found [here](https://github.com/UltiHash/scripts/blob/main/terraform/aws/ultihash/ultihash-helm-values.yaml). Adjust the helm values to set your custom storage class if required. 3. Initialize and apply the Terraform project ``` cd scripts/terraform/aws/ultihash terraform init terraform apply --var-file config.tfvars ``` Wait until the installation is completed. The UltiHash cluster is installed in the `default` Kuberentes namespace, you `kubectl` to see the deployed workloads: ``` kubectl get all ``` To get access to the deployed UltiHash cluster, configure your AWS CLI/SDK with the Ultihash root credentials: ```bash # Obtain credentials for the UltiHash root user aws_access_key_id=`kubectl get secret ultihash-super-user-credentials -o jsonpath="{.data.access-key-id}" | base64 --decode` aws_secret_access_key=`kubectl get secret ultihash-super-user-credentials -o jsonpath="{.data.secret-key}" | base64 --decode` # Set the credentials for the UltiHash root user export AWS_ACCESS_KEY_ID=$aws_access_key_id export AWS_SECRET_ACCESS_KEY=$aws_secret_access_key ``` Finally access the UltiHash cluster by using AWS CLI/SDK, use the domain name of the Network Load Balancer provisioned at [the previous step](#install-controllers-on-eks): ```bash aws s3api list-buckets --endpoint-url http://ultihash-test-6a925a272ca1f954.elb.eu-central-1.amazonaws.com/ ``` {% endstep %} {% endstepper %}
How to uninstall UltiHash on AWS To uninstall all previously deployed AWS resources follow the steps below: {% hint style="info" %} Make sure you are in a new Terminal window when uninstalling. {% endhint %} First, uninstall UltiHash by running the following commands: ``` cd scripts/terraform/aws/ultihash terraform destroy --var-file config.tfvars kubectl delete pvc --all ``` Next, uninstall the Kubernetes controllers: ``` cd scripts/terraform/aws/eks-cluster-controllers terraform destroy --var-file config.tfvars ``` Finally, uninstall the EKS cluster: ``` cd scripts/terraform/aws/eks-cluster terraform destroy --var-file config.tfvars ```
*** ### More information {% hint style="info" %} ### Manage AWS service limits When deploying UltiHash cluster on Amazon EKS, it is important to ensure that your AWS account has sufficient **EC2 vCPU-based instance limits** in the selected region. Amazon EKS worker nodes are backed by EC2 instances, and if vCPU quotas are too low, the cluster may fail to scale or provision nodes, causing deployment failures. The relevant quota is: **Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances**\ Default limit: **5 vCPUs per region** If the EKS cluster attempts to launch EC2 instances exceeding your vCPU quota, node provisioning will fail, and workloads may not start or scale properly. In case you need more vCPUs in your region than the quota provides, we recommend increasing quota proactively before scaling out your UltiHash cluster. Check your current **Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances)** quota on [Service Quotas Console for EC2](https://eu-central-1.console.aws.amazon.com/servicequotas/home/services/ec2/quotas/L-1216C47A) and if it is not enough, create a quota increase request by clicking on the button **Request increase at account level** in the top right corner. {% endhint %} {% hint style="info" %} ## Enforce Least Privilege Access Whenever interacting with AWS cloud, we strongly encourage you to **follow the principle of least privilege**. This means permissions should be limited to the **minimum actions and resources** required for each role or service to function. **Why this matters:** * Reduces the **attack surface** and limits the impact of compromised credentials or components. * Prevents **unintentional changes** or access to unauthorized resources. * Aligns with AWS **security best practices** and the **Well-Architected Framework**. * Enables better auditing, control, and compliance with security standards. More information on this topic can be found at [this AWS link.](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) {% endhint %} ## IAM permissions required to deploy and manage an UH cluster The IAM user or the role that is used to provision and manage UH cluster in an AWS account should have the following IAM permissions. The IAM permissions below are applied for all resources, after successful deployment they could be adjusted to match certain resource ARNs for improved security.
S3 permissions (required to manage Terraform states in S3): 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:CreateBucket",
                "s3:ListBucket"
            ],
            "Resource": "*"
        }
    ]
}
EventBridge permissions (required by Karpenter to manage EC2 interruption events): ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "events:TagResource", "events:DeleteRule", "events:PutTargets", "events:DescribeRule", "events:PutRule", "events:ListTagsForResource", "events:RemoveTargets", "events:ListTargetsByRule" ], "Resource": "*" } ] } ```
SQS permissions (required by Karpenter to manage EC2 interruption events): ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sqs:DeleteQueue", "sqs:GetQueueAttributes", "sqs:ListQueueTags", "sqs:CreateQueue", "sqs:SetQueueAttributes" ], "Resource": "*" } ] } ```
KMS permissions (required by EKS cluster to manage Kubernetes secrets): ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:TagResource", "kms:ListAliases", "kms:CreateAlias", "kms:CreateKey", "kms:DeleteAlias" ], "Resource": "*" } ] } ```
EKS permissions (required to manage EKS cluster): 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:DeleteAccessEntry",
                "eks:ListNodegroups",
                "eks:DescribeAddonConfiguration",
                "eks:UpdateAddon",
                "eks:ListAddons",
                "eks:AssociateAccessPolicy",
                "eks:ListAccessEntries",
                "eks:CreateNodegroup",
                "eks:DescribeAccessEntry",
                "eks:DescribeAddon",
                "eks:DeleteCluster",
                "eks:ListAssociatedAccessPolicies",
                "eks:DescribeNodegroup",
                "eks:DeleteAddon",
                "eks:DeleteNodegroup",
                "eks:DisassociateAccessPolicy",
                "eks:TagResource",
                "eks:CreateAddon",
                "eks:CreateAccessEntry",
                "eks:UpdateNodegroupConfig",
                "eks:DescribeCluster",
                "eks:ListAccessPolicies",
                "eks:DescribeAddonVersions",
                "eks:CreateCluster"
            ],
            "Resource": "*"
        }
    ]
}
IAM permissions (required by EKS cluster and EC2 instances): ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:GetRole", "iam:GetPolicyVersion", "iam:GetPolicy", "iam:DeletePolicy", "iam:CreateRole", "iam:DeleteRole", "iam:AttachRolePolicy", "iam:CreateOpenIDConnectProvider", "iam:CreatePolicy", "iam:ListInstanceProfilesForRole", "iam:PassRole", "iam:DetachRolePolicy", "iam:ListPolicyVersions", "iam:ListAttachedRolePolicies", "iam:ListRolePolicies", "iam:GetOpenIDConnectProvider", "iam:DeleteOpenIDConnectProvider", "iam:TagOpenIDConnectProvider" ], "Resource": "*" } ] } ```
EC2 permissions (required to manage EC2 instances): 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:DeleteSubnet",
                "ec2:AttachInternetGateway",
                "ec2:DeleteRouteTable",
                "ec2:AssociateRouteTable",
                "ec2:DescribeInternetGateways",
                "ec2:CreateRoute",
                "ec2:CreateInternetGateway",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:DeleteInternetGateway",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeRouteTables",
                "ec2:DescribeLaunchTemplates",
                "ec2:CreateTags",
                "ec2:CreateRouteTable",
                "ec2:RunInstances",
                "ec2:DetachInternetGateway",
                "ssm:GetParameters",
                "ec2:DisassociateRouteTable",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DescribeSecurityGroupRules",
                "ec2:DeleteNatGateway",
                "ec2:DeleteVpc",
                "ec2:CreateSubnet",
                "ec2:DescribeSubnets",
                "ec2:DeleteNetworkAclEntry",
                "ec2:DisassociateAddress",
                "ec2:DescribeAddresses",
                "ec2:CreateNatGateway",
                "ec2:CreateVpc",
                "ec2:DescribeAddressesAttribute",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeNetworkInterfaces",
                "ec2:CreateSecurityGroup",
                "ec2:ModifyVpcAttribute",
                "ec2:DeleteLaunchTemplateVersions",
                "ec2:ReleaseAddress",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:DeleteLaunchTemplate",
                "ec2:DeleteRoute",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DescribeNatGateways",
                "ec2:AllocateAddress",
                "ec2:DescribeSecurityGroups",
                "ec2:CreateLaunchTemplateVersion",
                "ec2:CreateLaunchTemplate",
                "ec2:DescribeVpcs",
                "ec2:DeleteSecurityGroup",
                "ec2:CreateNetworkAclEntry"
            ],
            "Resource": "*"
        }
    ]
}
*** *** ### Frequent issues troubleshooting
Helm chart install or upgrade failure **Symptoms:** * `helm install` or `helm upgrade` hangs or returns an error * Application pods do not start * Helm status is stuck at `pending-install` or `failed` **Steps to resolve:** * **Inspect the Helm release status:** ```bash helm status -n ``` * **Check for resource creation errors or pending pods:** ```bash kubectl get pods -n ``` * **Describe a failing pod to view events and errors:** ```bash kubectl describe pod -n ``` * **Debug with Helm’s dry run mode:** ```bash helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --dry-run --values values.yaml --debug ``` * After the issue has been found and eliminated, process with install or upgrade further. **Recommendation:** Always use `--dry-run` and `--debug` to validate changes before applying them in production.
Missing or incorrect values in values.yaml **Symptoms:** * Helm fails with a rendering error * Application fails at runtime due to missing config (e.g., secrets, ports, env vars) **Steps to resolve:** * **Compare your values file with the chart defaults:** ``` helm show values oci://registry.ultihash.io/stable/ultihash-cluster ``` * **Test the rendered templates locally:** ``` helm template oci://registry.ultihash.io/stable/ultihash-cluster --values ``` * **Reapply the corrected configuration:** ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --values ``` **Recommendation:** Use a version-controlled values file and validate changes in a staging environment before rolling out to production. #### 3. Application pods stuck in `CrashLoopBackOff` or `ImagePullBackOff` **Purpose:** Diagnose runtime pod failures due to misconfiguration or image issues. **Symptoms:** * Pods keep restarting or cannot pull the container image **Steps to resolve:** * **Inspect the pod state:** ``` kubectl get pods -n ``` * **Check the logs of the failing pod:** ``` kubectl logs -n ``` * **Correct the config causing failure, then upgrade:** ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n --values ``` **Recommendation:** Ensure that image repositories are accessible and secrets for private registries are correctly configured in the cluster.
# Set up UltiHash Serverless How to get started with a preconfigured, managed UltiHash cluster If you don't want to self-host your own UltiHash cluster for any reason, UltiHash Serverless is perfect for you. It gives you the same high-throughput engine behind our self-hosted version, but without the hassle of managing it yourself. Your can choose the cloud provider you prefer (current options are AWS and Hetzner Cloud); and your data stays local, stored securely in Frankfurt am Main, Germany (AWS) or Falkenstein, Germany (Hetzner). UltiHash Serverless offers a number of preconfigured cluster sizes: * 200GB: Ideal for tests and proof-of-concept projects (not available on Hetzner) * 1 TB: Ideal for development and small pipelines * 5 TB: Ideal for production workloads and training * 10 TB: Ideal for high-volume inference, media, or GenAI pipelines Get started with UltiHash Serverless *** ### Read throughput benchmarks Read throughput plays a central role in high-performance data workloads such as model training, fine-tuning, and batch inference. In these scenarios, delays in accessing data can lead to underutilized compute resources (e.g GPUs) and increased costs. UltiHash Serverless is optimized to provide high and consistent read performance, particularly for teams working with large volumes of data stored in the EU. The following benchmarks compare UltiHash to AWS S3 (Standard and Express One Zone) across different configurations. *Please note that if you’re using UltiHash Self-Hosted, the read throughput measured may be different depending on the hardware you’re using. For reference, the following benchmarks have been tested with the AWS EC2 instance type* `c7g.4xlarge` . **UltiHash Serverless (hosted on AWS) vs. S3 Standard (Region `eu-central-1`, Frankfurt)** | Storage System | Read Throughput | | ------------------------- | --------------- | | UltiHash Serverless (AWS) | 1 GB/s | | S3 Standard | 200–250 MB/s | UltiHash Serverless delivers 4–5× higher read throughput than S3 Standard, enabling faster data access and reducing idle time for compute resources, such as GPUs waiting for data loading. \ UltiHash Serverless is designed to deliver high read throughput directly from within the EU. It provides fast, local access to object data, making it well-suited for compute-heavy workloads like model training, inference pipelines, and parallel data processing, while ensuring that all data remains compliant with regional residency requirements. # Migrate your data How to migrate data from an external source to your UltiHash cluster This guide provides step-by-step instructions for migrating data, using the example of transferring from Amazon S3 to UltiHash using `rclone`. ([You can visit rclone's GitHub respository here.](https://github.com/rclone/rclone)) By following these steps, you can efficiently transfer your data to an UltiHash cluster while maintaining security and performance. ### Configure rclone First, add the following configuration to your `rclone` installation. Modify the `endpoint` setting to point to your specific UltiHash cluster. **Note:** Authentication parameters will be read from the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. ```toml [UltiHash] type = s3 provider = Other env_auth = true endpoint = https://ultihash.endpoint.example ``` **Alternative:** If you do not want to use environment variables, set `env_auth = false` and add `access_key_id` and `secret_access_key` manually in the rclone config file. ### Uploading Data to UltiHash #### Step 1: Create a Bucket Before uploading data, create a bucket in UltiHash: ```bash rclone mkdir UltiHash:/bucket ``` #### Step 2: Copy Local Data to UltiHash To transfer your local data to UltiHash, use the following command: ```bash rclone copy --progress /path/to/local/data UltiHash:/bucket ``` This command will display the transfer progress in real time. # Upload + download data How to upload data to UltiHash, and download it directly This page provides simple Python scripts for uploading and downloading data to and from UltiHash using our S3-compatible API. They make use of multithreading for optimal performance. Whether you’re pushing a folder to a bucket or retrieving one locally, these scripts make it easy to get started. For more complex workflows, check out the full-featured SDKs available via [AWS Developer Tools](https://aws.amazon.com/products/developer-tools). ## Uploading Data `uh_upload.py` : ```python # Example: Upload a folder to a bucket on UltiHash python3 uh_upload.py --url --bucket mybucket folder/ ``` ## Downloading Data `uh_download.py` : ```python # Example: Download a bucket from UltiHash python3 uh_download.py --url --path local-folder mybucket ``` These scripts provide a straightforward way to interact with UltiHash storage using the S3-compatible API. For more advanced scenarios, refer to the comprehensive SDKs available at [AWS Developer Tools](https://aws.amazon.com/developer/tools/). # Use the S3-compatible API How to interact with your UltiHash cluster using the S3-compatible API UltiHash offers a powerful, S3-compatible API that allows developers to interact with storage clusters using familiar commands and libraries designed for Amazon S3. The API was implemented in this way to ensure maximum integration flexibility across a wide variety of applications and services in your existing stack - removing the need for complex reconfigurations or middleware solutions.
{% hint style="info" %} ### Achieving S3 compatibility S3 compatibility can be achieved in different ways depending on your environment. For example, Python developers typically use the `boto3` library, which is part of the AWS SDK for Python and provides a straightforward interface for interacting with UltiHash as if it were S3. In contrast, data processing tools like PySpark don’t use `boto3`, instead, they rely on connectors like `s3a`, which is part of the Hadoop ecosystem and optimized for distributed data processing. This distinction is important: `boto3` is great for scripting and general-purpose workloads, while `s3a` is better suited for large-scale data operations in frameworks like Spark. Generally, you can use any S3-compliant SDK to interact with UltiHash. The AWS SDKs offer extensive support across various programming languages like Python, Java, Node.js, and more. As these SDKs are well-documented and maintained, they are the ideal choice for most developers. For developers looking to explore SDKs across various languages and environments, we highly recommend visiting [AWS Developer Tools](https://aws.amazon.com/products/developer-tools), which offers comprehensive support for integrating with S3-compatible APIs, including UltiHash. {% endhint %} ## Essential API operations * [**CreateBucket**](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html): Create new buckets in your UltiHash storage ```bash aws s3api create-bucket --bucket your-bucket-name --endpoint-url https://your-ultihash-endpoint ``` * [**PutObject**](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html): Upload files to your UltiHash buckets. ```bash aws s3 cp local-file.txt s3://your-bucket-name/ --endpoint-url https://your-ultihash-endpoint ``` * [**GetObject**](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html): Retrieve files from your UltiHash storage. ```bash aws s3 cp s3://your-bucket-name/file.txt local-file.txt --endpoint-url https://your-ultihash-endpoint ``` {% hint style="info" %} You can find premade Python scripts for uploading and downloading data [here](/operations/upload-+-download-data). {% endhint %} * [**ListObjectsV2**](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html): List the contents of your buckets. ```bash aws s3api list-objects-v2 --bucket your-bucket-name --endpoint-url https://your-ultihash-endpoint ``` * [**DeleteObject**](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html): Manage your storage by removing one unnecessary object. ```bash aws s3api delete-object --bucket your-bucket-name --key your-object-key --endpoint-url https://your-ultihash-endpoint ``` Facilitate object deletion operations by removing all objects in one bucket ```bash aws s3 rm s3://your-bucket-name/ --recursive --endpoint-url https://your-ultihash-endpoint ``` * [**DeleteBucket**](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html)**:** Manage your storage by removing unnecessary buckets (buckets should be empty before removal). ```bash aws s3api delete-bucket --bucket your-bucket-name --endpoint-url https://your-ultihash-endpoint ``` ## Full list of API operations #### S3 Compatibility Layer * [AbortMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) * [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) * [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) * [CopyObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) * [CreateBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html) * [DeleteBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html) * [DeleteBucketPolicy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketPolicy.html) * [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) * [DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) * [GetBucketPolicy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicy.html) * [GetBucketVersioning](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html) * [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) * [HeadBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html) * [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) * [ListBuckets](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html) * [ListMultipartUploads](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html) * [ListObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html) * [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) * [ListObjectVersions](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html) * [PutBucketPolicy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketPolicy.html) * [PutBucketVersioning](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketVersioning.html) * [PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) * [UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) #### IAM Compatibility Layer * [CreateAccessKey](https://docs.aws.amazon.com/IAM/latest/APIReference/API_CreateAccessKey.html) * [CreateUser](https://docs.aws.amazon.com/IAM/latest/APIReference/API_CreateUser.html) * [DeleteAccessKey](https://docs.aws.amazon.com/IAM/latest/APIReference/API_DeleteAccessKey.html) * [DeleteUser](https://docs.aws.amazon.com/IAM/latest/APIReference/API_DeleteUser.html) * [DeleteUserPolicy](https://docs.aws.amazon.com/IAM/latest/APIReference/API_DeleteUserPolicy.html) * [GetUserPolicy](https://docs.aws.amazon.com/IAM/latest/APIReference/API_GetUserPolicy.html) * [ListUserPolicies](https://docs.aws.amazon.com/IAM/latest/APIReference/API_ListUserPolicies.html) * [PutUserPolicy](https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html) > For more details on available SDKs and language-specific guides, check out the [AWS SDK hub](https://docs.aws.amazon.com/sdkref/latest/guide/overview.html). # Prebuilt connections How to connect UltiHash to the rest of your stack UltiHash offers a powerful S3-compatible API for connecting to a huge range of tools. Below are a selection of tools with [tested custom integrations](https://github.com/UltiHash/scripts/tree/main); many more can be easily connected [via the API](/operations/s3-compatible-api).
AirflowProgrammatically author, schedule and monitor workflows/pages/MQZZS870sdaahF4lWWj1/files/wsPFPRFcLPoLD6CncF4A
AWS GlueEvent-driven, serverless data integration service/pages/eNFt37tW1UnCdmJVa0j9/files/OgfykDZ5VImlgYK6voBX
Delta LakeOpen-source storage framework for building lakehouses/pages/Hjjh3PJpIW6xsKFqmsB9/files/iDL9d0mkkwE8rit8FZaH
IcebergHigh-performance format for huge analytic tables/pages/Hjjh3PJpIW6xsKFqmsB9/files/2DYzxbs0kgkc7WUaoS01
IcechunkStorage engine for tensor / ND-array data/pages/fJo44NAOFe9YrkVRGv1h/files/BeRUupUYAZ5G4JeeMpJM
KafkaDistributed event streaming platform/pages/Hjjh3PJpIW6xsKFqmsB9/files/WtwXxheDgkxQu6dFBfZW
Neo4jConnect a graph database and retrieve raw data/pages/i8uAi71OYA7GUPqlHBI8/files/hmkQX027rOk5tYrbFZhK
PrestoScalable SQL query engine for modern data analytics/pages/mqf6BiTMGdqQ3OYAgELL/files/HsBPTZRyxhjGmJOxTUsF
PySparkOpen-source analytics for large-scale data processing/pages/SfGVWpZGJS1YiKntxB1u/files/nCRP6eBIVdhLpno4nVbE
PyTorchLibrary for deep learning on irregular inputs/pages/4vnMJIsU6ZhC6kUfaXCT/files/BeUIoCf275ydvhXZ3GHh
SuperAnnotateCentralize AI data needs and vendor management/pages/R1p6evzqYunUfKodqeUv/files/E0gXpb4EpRsNvgAonARg
TrinoDistributed SQL query engine for big data analytics/pages/Wpt04XpRz2NZ6KUVhQ4o/files/aR0ErE3TS5OEWYcuCKjc
Vector databasesConnect a vector DB and retrieve raw data based on queries/pages/SlXDBwa6UMdSKuGuzjMm/files/sK5mmb1mQCubSfD8EDM0
# Airflow How to connect UltiHash to Airflow To start an Airflow session and integrate it with UltiHash, users need to ensure that the right pip packages are installed, then create the connection. ```python # Check that these packages are installed pip3 install 'apache-airflow[amazon]' pip3 install apache-airflow-providers-amazon # Create the connection between Airflow and UltiHash airflow connections add 'ultihash' --conn-json '{ "conn_type": "aws", "login": "ACCESS_KEY_ID", "password": "AWS_SECRET_KEY", "extra": { "endpoint_url": "", "verify": "False", "service_config": { "s3": { "endpoint_url": "" } } } }' # The output should be: # Successfully added `conn_id`=ultihash : aws://ACCESS_KEY_ID:******@: ``` Below is a DAG that leveraged the connection with UltiHash cluster created above: ```bash from airflow.models.dag import DAG from airflow.operators.python import PythonOperator from airflow.providers.amazon.aws.operators.s3 import S3CreateBucketOperator, S3DeleteBucketOperator from airflow.utils.dates import days_ago from airflow.providers.amazon.aws.hooks.s3 import S3Hook BUCKET_NAME="bucket-test" AWS_CONN_ID="ultihash" # Specify the connection with UltiHash def bucket_exists(): s3 = S3Hook(AWS_CONN_ID) # Make sure your S3 Hooks leverage the connection with UltiHash. if s3.check_for_bucket(BUCKET_NAME): raise Exception("Bucket %s still exists after removal" % BUCKET_NAME) else: print("Bucket %s has been successfully removed" % BUCKET_NAME) with DAG( dag_id='create_delete_bucket', schedule_interval=None, start_date=days_ago(2), max_active_runs=1, tags=['testing'], ) as dag: # Create a bucket with BUCKET_NAME on Ultihash create_bucket = S3CreateBucketOperator( task_id='create_s3_bucket', bucket_name=BUCKET_NAME, aws_conn_id=AWS_CONN_ID # Make sure your S3 Operators leverage the connection with UltiHash ) # Delete the previously created bucket delete_bucket = S3DeleteBucketOperator( task_id='delete_s3_bucket', bucket_name=BUCKET_NAME, aws_conn_id=AWS_CONN_ID # Make sure your S3 Operators leverage the connection with UltiHash ) # Check if the bucket still exists check_bucket = PythonOperator( task_id='check_if_bucket_exists', python_callable=bucket_exists ) create_bucket >> delete_bucket >> check_bucket ``` > See all information about the integration on GitHub here: # AWS Glue How to connect UltiHash to AWS Glue AWS Glue essentially enables users to use Spark in an AWS environment. To start an AWS Glue session users should be working with a UltiHash cluster deployed on AWS, import AWS Glue and configure the S3A driver as described in the code below: ```python import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job ## @params: [JOB_NAME] args = getResolvedOptions(sys.argv, ['JOB_NAME']) # Define the UltiHash endpoint URL s3_endpoint = "" sc = SparkContext() # AWS access and secret keys could be any, since authentication is not yet supported by UltiHash sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", "mocked") # Replace with the corresponding UltiHash credentials sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", "mocked") # Replace with the corresponding UltiHash credentials # The S3 endpoint is a URL pointing to the deployed UltiHash cluster sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", s3_endpoint) # S3 path style access has to be enabled sc._jsc.hadoopConfiguration().set("fs.s3a.path.style.access", "true") sc._jsc.hadoopConfigration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") sc._jsc.hadoopConfiguration().set("fs.s3a.path.style.access", "true") sc._jsc.hadoopConfiguration().set("fs.s3a.connection.ssl.enabled", "false") glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args['JOB_NAME'], args) ``` > See all information about the integration on GitHub here: # Iceberg How to connect UltiHash to Iceberg A lakehouse architecture is built on two key components: an open table format like Apache Iceberg, and scalable object storage like UltiHash. In this setup, users typically start with structured data (e.g., CSV files) and convert it to the Iceberg format before storing it in UltiHash. This approach enables efficient querying, schema evolution, and ACID guarantees on object storage. Below is a step-by-step guide using PySpark to convert CSV data into an Iceberg table and store it in UltiHash. ### Start a PySpark Session To get started, launch a PySpark session with the required dependencies and configurations: * Make sure your target bucket exists on UltiHash. * Include the necessary Iceberg, Hadoop AWS, and AWS SDK packages. * Configure the S3A driver properly to connect to UltiHash’s endpoint. ``` pyspark \ --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262 \ --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \ --conf spark.sql.catalog.iceberg.type=hadoop \ --conf spark.sql.catalog.iceberg.warehouse=s3a://iceberg \ --conf spark.hadoop.fs.s3a.endpoint=http://127.0.0.1:8080 \ --conf spark.hadoop.fs.s3a.access.key=TEST-USER \ --conf spark.hadoop.fs.s3a.secret.key=SECRET \ --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \ --conf spark.hadoop.fs.s3a.path.style.access=true \ --conf spark.hadoop.fs.s3a.connection.ssl.enabled=false \ --conf spark.driver.bindAddress=127.0.0.1 \ --conf spark.driver.host=127.0.0.1 ``` ### Create an Iceberg Table Once the session is running, start by creating a namespace if it doesn’t already exist. This serves as the logical container for your tables. ``` spark.sql("CREATE NAMESPACE IF NOT EXISTS iceberg.ulti") ``` Now you can define your Iceberg table, specifying the schema and any table-level properties such as format version or metadata retention: ``` spark.sql(""" CREATE TABLE IF NOT EXISTS iceberg.ulti.test_iceberg_table ( id INT, name STRING, price DOUBLE, category STRING, ts TIMESTAMP ) USING iceberg TBLPROPERTIES ( 'format-version'='2', 'write.metadata.previous-versions-max'='5' ) """) ``` To verify the table was created successfully: ``` spark.sql("SHOW TABLES IN iceberg.ulti").show() ``` ### Load data from a structured format (e.g. CSV) Read your CSV data into a DataFrame, enabling schema inference and header detection: ``` df = spark.read.format("csv") \ .option("header", "true") \ .option("inferSchema", "true") \ .load("/Users/ultihash/Downloads/iceberg/test_data.csv") df.show() df.printSchema() ``` ### Write data to UltiHash in Iceberg format With your DataFrame ready, append the records to the Iceberg table stored in your UltiHash bucket: ``` df.write.format("iceberg") \ .mode("append") \ .save("iceberg.ulti.test_iceberg_table") ``` ### Read Iceberg data from UltiHash To confirm the data was written correctly, simply query the table: ``` spark.sql("SELECT * FROM iceberg.ulti.test_iceberg_table").show() ``` # Icechunk How to connect UltiHash to Icechunk Here's a guide to setup UltiHash as IceChunk’s backend storage. Start by importing Icechunk, and configure UltiHash as the storage (including credentials). ```python import icechunk storage = icechunk.s3_storage( bucket="bucket-name", prefix="prefix-name", endpoint_url="", access_key_id="TEST-USER", secret_access_key="SECRET", allow_http=True ) ``` with: 1. The endpoint\_url parameter is set to the URL of the UltiHash cluster. 2. The UltiHash cluster is running over HTTP and not HTTPS, the allow\_http parameter must be set to True. 3. This describes a local machine setup. If the UltiHash cluster is deployed in the cloud, both the credentials (access\_key\_id and secret\_access\_key) and the endpoint\_url parameter must be adjusted in consequence. # Kafka How to connect UltiHash to Kafka This guide walks you through setting up [Kafka S3 sink connector](https://www.confluent.io/hub/confluentinc/kafka-connect-s3) to continuously write messages to UltiHash cluster. You’ll learn how to configure Kafka S3 sink connector, launch a local Kafka setup using Docker, produce Kafka messages, and write them to an UltiHash bucket continuously. #### Download the S3 Sink connector locally To use the S3 Sink connector with Kafka, you’ll need to download it locally. You can either use the command below, or do it manually via this [link](https://www.confluent.io/hub/confluentinc/kafka-connect-s3). Make sure to keep track of the target path file (it needs to reside in the right folder) ```bash mkdir -p ./connect-plugins cd connect-plugins curl -O ``` #### Create a docker-compose.kafka-connect.yml file Create the following yml file. Make sure that the path to the plugins (S3 Sink Connector) is accurate. ```bash version: '3.8' services: zookeeper: image: confluentinc/cp-zookeeper:7.5.0 container_name: zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:7.5.0 container_name: kafka ports: - "9092:9092" depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 connect: image: confluentinc/cp-kafka-connect:7.5.0 container_name: connect ports: - "8083:8083" depends_on: - kafka environment: CONNECT_REST_ADVERTISED_HOST_NAME: connect CONNECT_REST_PORT: 8083 CONNECT_BOOTSTRAP_SERVERS: kafka:9092 CONNECT_GROUP_ID: "connect-cluster" CONNECT_CONFIG_STORAGE_TOPIC: "connect-configs" CONNECT_OFFSET_STORAGE_TOPIC: "connect-offsets" CONNECT_STATUS_STORAGE_TOPIC: "connect-status" CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1 CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1 CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1 CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter" CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.storage.StringConverter" CONNECT_PLUGIN_PATH: "/usr/share/java/plugins" CONNECT_LOG4J_ROOT_LOGLEVEL: INFO CONNECT_RUNNABLE_PRECONDITION_CLASS: CONNECT_DISABLE_PRECHECKS: "true" volumes: - ./plugins:/usr/share/java/plugins ``` #### Start Kafka ```bash docker compose -f docker-compose.kafka-connect.yml up -d docker ps #check that your containers are running ``` #### Configure the S3 Sink Connector Use the following command to configure your S3 Sink Connector. The target bucket in UltiHash needs to exist in your UltiHash Cluster prior to this configuration. * `"connector.class": "io.confluent.connect.s3.S3SinkConnector"` tells Kafka Connect which connector to use. Here, we’re using the S3 sink connector to push data to object storage. * `"tasks.max"` sets the maximum number of parallel tasks the connector can run. Set it to `1` for a simple setup. Increase it if you have multiple partitions or topics and need higher throughput. * `"topics"` tells the connector which Kafka topics to pull messages from. If you're working with more than one topic, just list them separated by commas. * `"s3.bucket.name"` is the name of the bucket in your UltiHash cluster where the messages will be stored. * * `"s3.region"` is required by the connector even if you’re running locally. Can be set to any valid AWS-style region. * `"store.url"` is the URL of your S3-compatible storage (UltiHash). `host.docker.internal` bridges Docker to your local host. * `"s3.part.size"` corresponds to the minimum size (in bytes) before splitting large files into parts. Mostly relevant for big messages. * `"flush.size"` determines how many Kafka messages are grouped together into a single JSON file before being written to UltiHash. * smaller values write more frequently, creating more (but smaller) files, * larger values reduce write frequency and create fewer, larger files — which can be more efficient for batch processing or compression. * `"storage.class"` specifies how data is written to S3. This should always match the connector type. * `"format.class"` defines the format for stored messages. JSON keeps things readable and lightweight. * `"schema.compatibility"` disables schema enforcement. Good if your messages don’t use Avro or schema registries. * `"key.converter"` converts message keys into strings for storage. * `"value.converter"` converts message values into strings for storage. * `"aws.access.key.id"` is your UltiHash access key (works like AWS credentials). * `"aws.secret.access.key"` is your UltiHash secret key. ```bash curl -X PUT \\ -H "Content-Type: application/json" \\ -d '{ "connector.class": "io.confluent.connect.s3.S3SinkConnector", "tasks.max": "1", "topics": "test-topic", "s3.bucket.name": "kafka", "s3.region": "us-east-1", "store.url": "", "s3.part.size": 5242880, "flush.size": 3, "storage.class": "io.confluent.connect.s3.storage.S3Storage", "format.class": "io.confluent.connect.s3.format.json.JsonFormat", "schema.compatibility": "NONE", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.storage.StringConverter", "aws.access.key.id": "", "aws.secret.access.key": "" }' ``` #### Check the S3 Sink Connector status Check the status of the S3 Sink Connector, it should be `“RUNNING”` ```bash curl | jq ``` #### Access the CLI within docker You should enter the Docker container to interact with Kafka directly. ```bash docker exec -it kafka bash ``` #### Write a message to Kafka (which will be communicated to UltiHash automatically) First, access the Kafka console: ```bash kafka-console-producer --broker-list kafka:9092 --topic test-topic ``` Now, you can write your messages: ```bash hello-world this-is-working 🦾💪 #hit enter after each line #Ctrl+C when done exit ``` #### Read your message from UltiHash You can read from UltiHash by seeing the JSON files that were written by Kafka to UltiHash. In that case, you’re seeing a list of JSON files where the amount of JSON files depends on the flush size in the S3 Sink Connector configuration and the amount of messages sent (#messages sent/flush size = total #objects). This list will provide you with the objects’ keys. This will enable you to read the contents of the JSON files, as described below: ```bash #list object in the bucket curl -X GET #read specific JSON files curl ``` # Neo4j How to connect UltiHash to Neo4j Whether you’re setting up a retrieval pipeline or building a knowledge graph, connecting Neo4j to UltiHash lets you efficiently retrieve the raw data behind your graph nodes, like documents, images, or videos. Neo4j stores relationships and metadata, but not the actual file content. That’s where UltiHash comes in. To make this work, you’ll need to setup a middleware app: it listens for graph queries from Neo4j, resolves the node or relationship metadata to actual file references, and pulls the raw data from UltiHash on demand. Your typical pipeline will look like: * Store your raw data (e.g., PDFs, videos) in UltiHash. * Ingest metadata and relationships into Neo4j, including references (e.g., file paths or IDs) to the data stored in UltiHash. * A middleware app handles queries to Neo4j and resolves the relevant nodes or relationships. * Based on the references, the middleware retrieves the raw files from UltiHash. * The raw data is then served to the user or passed to downstream applications. Here’s an example API that handles both the query to Neo4j and the retrieval of raw data from UltiHash, so you can serve the right files directly from your knowledge graph. ### FlaskAPI ```python from neo4j import GraphDatabase from flask import Flask, request, jsonify from io import BytesIO from PIL import Image import boto3 NEO4J_URI = "neo4j+s://your-link.neo4j.io" NEO4J_USER = "user" NEO4J_PASSWORD = "your_password_here" driver = GraphDatabase.driver(NEO4J_URI, auth=(NEO4J_USER, NEO4J_PASSWORD)) ULTIHASH_ENDPOINT = "endpoint_url" ULTIHASH_BUCKET = "movies" #my bucket name ULTIHASH_ACCESS_KEY = "YOUR-ACCESS KEY" ULTIHASH_SECRET_KEY = "YOUR-SECRET-KEY" s3 = boto3.client( "s3", endpoint_url=ULTIHASH_ENDPOINT, aws_access_key_id=ULTIHASH_ACCESS_KEY, aws_secret_access_key=ULTIHASH_SECRET_KEY ) app = Flask(__name__) @app.route("/query", methods=["POST"]) def query(): data = request.get_json() cypher_query = data.get("query") if not cypher_query: return jsonify({"error": "Missing query parameter"}), 400 with driver.session() as session: results = session.run(cypher_query) records = [record.data() for record in results] if not records: return jsonify({"error": "No results found"}), 404 output = [] poster_filenames = [record["poster_filename"] for record in records if "poster_filename" in record] if poster_filenames: fetch_and_open_images(poster_filenames) return jsonify({"results": records}) def fetch_and_open_images(filenames): for filename in filenames: if not filename.lower().endswith(('.jpg', '.jpeg')): #my objects have file extensions filename += ".jpg" try: response = s3.get_object(Bucket=ULTIHASH_BUCKET, Key=filename) file_data = response["Body"].read() image = Image.open(BytesIO(file_data)) image.show() except Exception as e: print(f"Error fetching/opening {filename}: {e}") if __name__ == "__main__": app.run(debug=True) ``` ### API Query Neo4j is built around Cypher, a query language designed specifically for working with graph data. It is optimized for pattern-based queries, making it easy to express relationships and navigate connected data efficiently. For this setup, I wanted to keep the user experience seamless: you can interact with Neo4j just as you normally would using Cypher. The API is built to take in any Cypher query, execute it on the database, and return structured results. That means you don’t have to learn new syntax or change how you query, everything works the same. The only difference? When your query involves the associated files (like movie posters), they’re directly retrieved from UltiHash, so you don’t have to manage raw data retrieval separately (and manually!). ```bash curl -X POST "" \\ -H "Content-Type: application/json" \\ -d '{ "query": "MATCH (a:Actors {actor: \\"Brad Pitt\\"})-[:`Acts in`]->(m:Movies)<-[:`Directed`]-(d:Directors {director: \\"Quentin Tarantino\\"}) RETURN m.movie_title AS movie_title, m.poster_filename AS poster_filename", "fetch_image": true }' ``` # Presto How to connect UltiHash to Presto In order to grant Presto access to UltiHash, users need to create and configure a catalogue called UltiHash, with the ultihash.properties text file as follows: ```python connector.name=hive-hadoop2 hive.metastore.uri=thrift://hive-metastore:9083 hive.non-managed-table-writes-enabled=true hive.s3.endpoint=https://ultihash hive.s3.path-style-access=true hive.s3.aws-access-key=mocked hive.s3.aws-secret-key=mocked ``` In this file, we setup the Hive connector, which is using S3A driver under the hood. This enables Presto to integrate and communicate with UltiHash. Each catalogue points at a UltiHash Cluster, and within a catalogue the user can create SQL tables which point at specific buckets in the UltiHash Cluster. Each catalogue can host several SQL tables. > See all information about the integration on GitHub here: # PySpark How to connect UltiHash to PySpark Before starting a PySpark session with access to data stored in UltiHash, the user should make sure that the correct packages were downloaded and configure the S3A driver. ```python pyspark \\ --packages org.apache.hadoop:hadoop-aws:3.3.4, #Hadoop package check com.amazonaws:aws-java-sdk-bundle:1.12.262 \\ #AWS Java SDK package check --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \\ --conf spark.sql.catalog.iceberg.type=hadoop \\ --conf spark.sql.catalog.iceberg.warehouse=s3a://iceberg \\ --conf spark.hadoop.fs.s3a.endpoint=http://127.0.0.1:8080 \\ #S3A driver config --conf spark.hadoop.fs.s3a.access.key=TEST-USER \\ #S3A driver config --conf spark.hadoop.fs.s3a.secret.key=SECRET \\ #S3A driver config --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \\ #S3A driver config --conf spark.hadoop.fs.s3a.path.style.access=true \\ #S3A driver config --conf spark.hadoop.fs.s3a.connection.ssl.enabled=false \\ #S3A driver config --conf spark.driver.bindAddress=127.0.0.1 \\ --conf spark.driver.host=127.0.0.1 ``` > See all information about the integration on GitHub here: # PyTorch How to connect UltiHash to PyTorch PyTorch is an S3 native library. Prior to starting a session that accesses and stores data in and to UltiHash, the user should ensure that their UltiHash credentials are exported in their working terminal and then can proceed as follows: ```python import s3torchconnector # Define S3 region (could be any, since the custom S3 endpoint is used) REGION = "us-east-1" # Define the endpoint URL for the UltiHash cluster ENDPOINT_URL = "" # Enforce the path style URLs config = s3torchconnector.S3ClientConfig(force_path_style=True) # EXAMPLE 1: Create a dataset from the data stored on the UltiHash bucket named "test-data" dataset = s3torchconnector.S3MapDataset.from_prefix( "s3://test-data/", endpoint=ENDPOINT_URL, region=REGION, s3client_config=config ) # EXAMPLE 2: Create a checkpoint checkpoint = s3torchconnector.S3Checkpoint( region=REGION, endpoint=S3_ENDPOINT, s3client_config=config ) ``` # SuperAnnotate How to connect UltiHash to SuperAnnotate [SuperAnnotate](https://www.superannotate.com/) is a platform for data management. With advanced annotation and QA tools, data curation, automation features, native integrations, and data governance, it enables enterprises to build high-quality datasets and ML pipelines. UltiHash cluster integrates with SuperAnnotate as a storage solution via custom integration. It is strongly encouraged to read [the corresponding article ](https://doc.superannotate.com/docs/integration-custom)before proceeding with this guide. ## Step 1: Prerequisites * UltiHash cluster, deployed in a Kubernetes environment and exposed via a public HTTPs endpoint. To deploy UH cluster on a Kubernetes cluster follow [this instruction](https://docs.ultihash.io/administration/3.-general-installation). A public HTTPs endpoint for UltiHash cluster could be provisioned at the [Ingress object](https://docs.ultihash.io/administration/encryption#configure-tls-on-the-ingress-level) level. * Installed [kubectl](https://kubernetes.io/docs/tasks/tools/). The utility has to be configured to access the Kubernetes cluster where UltiHash cluster is deployed to. * Installed [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). The utility has to be configured with [the credentials](https://docs.ultihash.io/administration/3.-general-installation#step-4-post-installation) of the deployed UltiHash cluster to be able to access it. * Installed [Git](https://git-scm.com/downloads). ## Step 2: Create custom integration on SuperAnnotate On SuperAnnotate find the **Team Setup** page and click there on **Integrations**:
Add a new custom integration:
Provide the integration with the name and URL. The **Request URL** should correspond to the public HTTPs endpoint that belongs to the generator of pre-signed URLs. The following format expected: **.** In the example below, the pre-signed URLs generator is exposed under the domain name **generator.ultihash.example**.
Save the displayed **Secret**, it will be required to pass to onto the generator of pre-signed URLs. Before clicking on the button **Create**, need to deploy the pre-signed URLs Generator alongside the UltiHash cluster. Otherwise you will get the error below:
The generator of pre-signed URLs is the core component of the custom integration. It is a Kubernetes deployment that resides in the same Kubernetes cluster with the UltiHash cluster and generates S3 pre-signed URLs on demand for the requested files residing on the UltiHash cluster. Clone the public Github repository with the scripts and switch to the directory named **superannotate**: ```bash git clone https://github.com/UltiHash/scripts.git && cd superannotate ``` In the directory find the file **presigned-urls-generator.yaml.** Open it in your favorite editor and replace the following placeholders: * **\** - Kubernetes namespace where your UltiHash cluster resides. * **\** - HTTPs endpoint of your UltiHash cluster, e.g. **.** * **\** - secret value from the integration page, e.g. **oAKg5YlvxSqiNF3ni2MUq4uJmnvexRcx67utGa62Gf7zymxYx9Ua2d5q1ZNh6p67.** * **\** - domain name under which the URLs generator is exposed publicly, e.g. **generator.ultihash.example.** All placeholders required to be replaced are marked with the commentary **# REQUIRED** for better navigation. Finally apply the edited Kubernetes definitions: ```bash kubectl apply -f presigned-urls-generator.yaml ``` Once the Kubernetes definitions successfully applied, click the **Create** button on the page of your custom integration. It has to be successfully created. ## Step 3: Upload files on UH cluster Create a bucket on the UltiHash cluster and upload there the data that needs to be accessed by SuperAnnotate. Suppose we have created a bucket named **test** that contains two uploaded 2 images: **image1.jpg** and **image2.jpg.** It is recommended to configure [CORS](https://doc.superannotate.com/docs/cors-authentication) for the bucket containing the data (the bucket **test** in our case). For this create a text file named **cors.json** with the following contents: ```json { "CORSRules": [ { "AllowedOrigins": ["https://app.superannotate.com","https://editor.superannotate.com","https://document.superannotate.com"], "AllowedHeaders": ["*"], "AllowedMethods": ["GET"] } ] } ``` Apply the CORS configuration above for the bucket **test**. In the example below the endpoint URL of the UltiHash cluster is ****. ```bash aws s3api put-bucket-cors --bucket test --endpoint-url https://cluster.ultihash.example --cors-configuration file://cors.json ``` ## Step 4: Make SuperAnnotate read files from UH cluster On SuperAnnotate create a project that correspond to the data type being annotated. Since in the previous step we supposed that the dataset uploaded UltiHash contains images, need create a project of type **Image**.
Inside the project click on the button **Add** -> **Upload Images**. Switch to the **External storage** tab and select the integration created before.
To upload the files from the UltiHash cluster, need to create a CSV file, that lists the files that should be read by SuperAnnotate, and upload this file into the field depicted above. In out case the file will look like this: ``` url,name s3://test/image1.jpg,image1.jpg s3://test/image2.jpg,image2.jpg ``` Each line represents a single file that lies on the UltiHash cluster. The **URL** is the S3 URL that defines path to the file. For example, **s3://test/image1.jpg** means that a file named **image1.jpg** is inside the bucket **test** on the UltiHash cluster. The **name** is the filename that will be displayed on the side of SuperAnnotate. After the CSV file was uploaded, you will see the list of files that need to be uploaded from the UltiHash cluster:
Click **Upload** and after upload is finished, find the files in the root of you project. They are ready to be annotated. # Trino How to connect UltiHash to Trino In order to grant Trino access to UltiHash, users need to create and configure a catalogue called UltiHash, with the ultihash.properties text file as follows: ```python connector.name=hive hive.metastore.uri=thrift://hive-metastore:9083 hive.non-managed-table-writes-enabled=true hive.s3.path-style-access=true hive.s3.endpoint=https://ultihash hive.s3.aws-access-key=mocked hive.s3.aws-secret-key=mocked ``` In this file, we setup the Hive connector, which is using S3A connector under the hood. This enables Trino to integrate and communicated with UltiHash. Each catalogue points at a UltiHash Cluster, and within a catalogue the user can create SQL tables which point at specific buckets in the UltiHash Cluster. Each catalogue can host several SQL tables. > See all information about the integration on GitHub here: # Vector databases How to connect UltiHash to any vector database Whether you're looking to implement semantic search (text, images, multimedia), a recommendation engines or a retrieval-augmented generation (RAG), you'll need to work with a vector database to be able to leverage on vector serach. The vector database is only responsible for storing and indexing embeddings. The actual raw data (such as documents, images, or videos) is stored separately in UltiHash. Your typical pipeline will look like: * Store your raw data (e.g., images) in UltiHash. * Generate embeddings from this raw data and store these embeddings in your chosen vector database. * A middleware application receives a user query, performs a vector search against the database, and obtains references to matching items. * The middleware then retrieves the actual files from UltiHash based on those references. * Finally, the middleware serves the raw data (such as images or documents) to the user. Here's an API that enables vector search through Zilliz (a vector database selected to make this script practical) and the retrieval of the raw data from UltiHash automatically. ### Example FlaskAPI code to query vectorDB and UltiHash ```python #!/usr/bin/env python3 import os # Handles file paths and environment variables import sys # Accesses system functions and command-line arguments import json # Reads/writes JSON data import base64 # Encodes/decodes binary data as text import torch # Loads CLIP for text-to-embedding conversion import argparse # Parses command-line arguments import boto3 # Retrieves images from UltiHash from pathlib import Path # Works with file system paths from flask import Flask, request, jsonify # Framework to handle API requests from pymilvus import connections, Collection # Connects to Zilliz from transformers import CLIPProcessor, CLIPModel # Loads CLIP for text-to-embedding conversion from io import BytesIO # Handles image processing from PIL import Image # Handles image processing # --------------------------- # Connect to Zilliz (Milvus) # --------------------------- connections.connect( alias="default", uri="https://", # Replace with your Zilliz cluster URI token="" # Replace with your Zilliz access token ) print("✅ Connected to Zilliz!") collection_name = "landscapes" collection = Collection(collection_name) # Access the correct collection # --------------------------- # Set up boto3 S3 Client for UltiHash # --------------------------- s3 = boto3.client( 's3', endpoint_url="", # Adjust if necessary aws_access_key_id=os.environ.get("AWS_ACCESS_KEY_ID"), aws_secret_access_key=os.environ.get("AWS_SECRET_ACCESS_KEY") ) # Ensure API calls include the UltiHash license def add_license_header(request, **kwargs): request.headers["UH_LICENSE_STRING"] = "" # Replace with your license key s3.meta.events.register("before-sign.s3", add_license_header) bucket = "landscapes" # Set the correct UltiHash bucket # --------------------------- # Load CLIP text encoder (for query vector generation) # --------------------------- clip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") # --------------------------- # Helper function: Query Zilliz with vector search # --------------------------- def query_landscape(query_text: str, top_k: int = 3): """ Computes the CLIP text embedding for the given query and performs a vector search in Zilliz. Parameters: query_text (str): The text query (e.g. "mountain"). top_k (int): The number of top results to retrieve (default 3). Returns: list: A list of filenames (without extension) from the best matching records. """ # Compute the CLIP text embedding for the query text. inputs = clip_processor(text=[query_text], return_tensors="pt", padding=True) with torch.no_grad(): query_vector = clip_model.get_text_features(**inputs)[0].tolist() # Define search parameters for cosine similarity. search_params = {"metric_type": "COSINE", "params": {"nprobe": 10}} try: search_results = collection.search( data=[query_vector], anns_field="embedding", param=search_params, limit=top_k, # No additional filter here since we're doing a pure vector search. output_fields=["filename"] ) except Exception as e: print(f"Vector search error for query '{query_text}': {e}") return [] filenames = [] if search_results and len(search_results) > 0 and len(search_results[0]) > 0: for hit in search_results[0]: # Access the underlying entity. Depending on your pymilvus version, you might need to use attributes. filenames.append(hit.filename) # Assuming hit.filename holds the stored filename (without extension) print(f"🔍 Retrieved filename from Zilliz: {hit.filename}") else: print(f"No results found for query '{query_text}'.") return filenames # --------------------------- # Initialize Flask Application # --------------------------- app = Flask(__name__) # --------------------------- # Endpoint: /get_landscape_images # --------------------------- @app.route("/get_landscape_images", methods=["POST"]) def get_landscape_images(): """ Expects a JSON payload: { "query": "mountain" } Performs a vector search in Zilliz for the given query text, retrieves the top 3 matching records, fetches the corresponding images from UltiHash (bucket: landscapes) using the filename with '.jpg', and returns the results with base64-encoded images. """ data = request.get_json() if not data or "query" not in data: return jsonify({"error": "Missing 'query' in request."}), 400 query_text = data["query"].strip() results = [] target_dir = Path("/Users/ultihash/test/retrieval-test") target_dir.mkdir(parents=True, exist_ok=True) # Get top 3 matching filenames from Zilliz. filenames = query_landscape(query_text, top_k=3) if not filenames: return jsonify({"error": "No matching records found."}), 404 for filename in filenames: # Append .jpg to build the expected key in UltiHash. file_key = f"{filename}.jpg" try: response = s3.get_object(Bucket=bucket, Key=file_key) file_data = response["Body"].read() except Exception as e: results.append({"filename": filename, "error": f"Failed to fetch image '{file_key}': {str(e)}"}) continue try: # Open the image using Pillow directly from memory. image = Image.open(BytesIO(file_data)) image.show() # This will open the image using your default image viewer. except Exception as e: results.append({"filename": filename, "error": f"Failed to open image: {str(e)}"}) continue results.append({ "filename": filename, "message": "Image fetched and opened successfully." }) print(f"Processed filename {filename} and opened image {file_key}") return jsonify({"results": results}) if __name__ == "__main__": app.run(host="127.0.0.1", port=5000, debug=True) ``` ### API Query ```bash curl -X POST -H "Content-Type: application/json" -d '{"query": "sunset at the beach"}' ``` > For a detailed example of this integration in action, please see our full case study using vector database Zilliz (Milvus) to set up a simple RAG pipeline: [Why does vector search need object storage as its foundation?](https://www.ultihash.io/blog/why-does-vector-search-need-object-storage-as-its-foundation) # Set up pre-signed URLs How to use pre-signed URLs to grant temporary access to objects Pre-signed URLs provide a way to grant temporary access to objects stored in S3-compatible storage systems like UltiHash. They are often used to enable uploads or downloads from users or systems that should not have direct access credentials. This guide walks through the general process of setting up a system that generates pre-signed URLs and using them to interact with your object storage. ### What are pre-signed URLs A pre-signed URL is a time-limited link to an object in storage. It includes all necessary authentication in the query string of the URL itself. Clients can use these URLs to upload or download files without needing access keys or additional permissions. Common use cases include: * Temporary read or write access * Integrating object storage with third-party tools * Allowing front-end applications to upload or download data securely ### Prerequisites To set up a pre-signed URL generator, you will need: * An S3-compatible object storage cluster (e.g. UltiHash), publicly accessible over HTTPS * A Kubernetes cluster to host the generator service * `kubectl` configured to access your Kubernetes cluster * AWS CLI configured to talk to your storage endpoint * An optional domain or public URL under which to expose the generator ### Deploying the URL generator A pre-signed URL generator is typically a lightweight web service running inside your infrastructure. Its job is to accept object path requests and return pre-signed URLs based on your credentials and permissions. To deploy the generator: 1. Clone the repository (this example uses UltiHash’s): ```bash git clone https://github.com/UltiHash/scripts.git && cd superannotate ``` # Save space with deduplication How to use our built-in deduplication - or disable for more throughput As of 2024, an estimated 90% of global data is [estimated to be redundant](https://www.businesswire.com/news/home/20200508005025/en/IDCs-Global-DataSphere-Forecast-Shows-Continued-Steady-Growth-in-the-Creation-and-Consumption-of-Data), emphasizing the need for efficient storage management. UltiHash tackles this challenge with its byte-level deduplication algorithm, designed to minimize storage volumes by identifying and eliminating redundant data across all objects, regardless of format. This method can reduce overall storage needs by up to 60%, enabling organizations to scale their data without proportionally increasing capacity requirements.
> **Want to see how much deduplication you could achieve? Test your data with our 5min terminal demo ↓** > > ``` > curl -fsSL https://ultihash.io/1line | bash > ``` *** ### How does UltiHash's deduplication work? The deduplication process works by splitting objects into fragments of varying sizes depending on the dataset. If a fragment already exists within the system, it isn’t stored again, eliminating unnecessary duplication across datasets. This ongoing comparison ensures that storage resources are utilized efficiently while maintaining data integrity. Unlike traditional compression techniques that often decrease performance, UltiHash’s deduplication runs continuously and is data-type-agnostic, supporting structured, unstructured, and even compressed data. In certain cases, such as with RAW files, tests have shown volume reductions of up to 74%. This makes UltiHash ideal for environments handling large quantities of redundant data, particularly in AI, machine learning, and media-heavy applications. *** ### What kinds of data deduplicate well? UltiHash optimizes data volume out of the box through a built-in deduplication algorithm that eliminates redundancies at a byte level, regardless of data type or format. This results in significant space savings of up to 60% on the entire data volume, depending on various factors including: * compressed vs uncompressed data format: UltiHash generates up to 75% space savings on uncompressed formats (e.g. RAW, TIFF) and up to 51% on compressed formats (e.g. JPG, PNG) * similarity between the objects: the higher the similarity, the more space saved This section documents the space savings generated by UltiHash on different datasets, giving a fair demonstration of UltiHash’s capabilities. The results can be reproduced on any UltiHash cluster.
Dataset + linkDataset sizeSpace savings
DICOM files of brain MRI scans1.51 GB67.13 %
JPGs of driving scenarios2.41 GB52.32 %
PNGs of synthetic textures with defects5.89 GB50 %
WAVs of human speech for emotion recognition0.33 GB50 %
TIFF images of climate data16.28 GB45.84 %
TIFFs of fossil segmentations8.26 GB44.64 %
CSV tables of symptoms0.0014 GB42 %
Models of dinosaur teeth1.87 GB33.09 %
Parquet files with temperature, humidity, wind and land uses1.81 GB21.59 %
NetCDF climatic and atmospheric data0.68 GB18.65 %
LIDAR data of driving scenarios33.26 GB18.31 %
PDFs of Indian supreme court judgements5.57 GB11.95 %
*** ### How to disable deduplication UltiHash includes an integrated deduplication service that is enabled by default. It is recommended for most workloads, especially read-intensive ones, as it does not introduce latency during read operations. However, for write-intensive workloads where the overhead of deduplication is not needed, you can disable the service for the entire storage cluster. To disable deduplication, update the Helm chart values by setting the deduplicator.enabled flag to false and then apply the change to the Helm release: ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n \ --set deduplicator.enabled=false ``` You can also make this change by editing values.yaml directly: ``` deduplicator: enabled: false ``` Then reapply the release: ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n \ --values values.yaml ``` The deduplicator service can be disabled or re-enabled at any time by updating the Helm chart values and upgrading the release. *** ### How to access deduplication metrics UltiHash extends beyond the standard S3 API with features like deduplication metrics, which allow you to query the effective size of your data after deduplication. This unique functionality is crucial for optimizing storage and understanding your actual storage usage. `get_effective_size.py` : ``` # Retrieve deduplicated data size from UltiHash get_effective_size.py --url ``` # Delete stored data How UltiHash provides rapid data deletion at scale Efficient data deletion in a distributed storage system is often overlooked, yet it is inherently complex. In many systems, deleting an object requires coordination across multiple nodes to ensure all associated data fragments are removed safely. This coordination can be time-consuming, and in some architectures, the disk space freed by deletes is only reclaimed during deferred maintenance tasks such as compaction or garbage collection. Both issues can lead to slow deletions, delayed space reclamation, and unpredictable storage availability. UltiHash avoids these drawbacks by maintaining fine-grained reference counting information directly at the level of each storage service instance. When an object is deleted, each instance can immediately determine whether the data fragments it holds are still referenced by any other objects in the cluster. Because this determination is local, there is no need for cluster-wide coordination protocols, which greatly reduces deletion latency. As soon as a data fragment’s reference count reaches zero, the fragment is removed and the disk space it occupied is reclaimed instantly. This ensures that storage capacity is returned to the system without waiting for background compaction or garbage collection cycles. This design provides predictable performance for delete-heavy workloads and ensures that available disk space reflects the true live data set at all times. # Set up object versioning How to enable object versioning on UltiHash UltiHash supports S3-compatible object versioning, a critical feature for teams working with evolving datasets, model artifacts, and analytic outputs. Versioning allows you to track every change to an object over time and retrieve or roll back to any previous version on demand, all without altering your workflows. Unlike traditional storage systems where versioning often results in costly storage bloat, UltiHash’s deduplication engine ensures redundant data across versions is only stored once. You get traceability without the cost of storage overheads When multiple versions of the same object share overlapping binary content (e.g., large datasets with only partial changes, or models with identical weights in early layers), UltiHash detects duplicated parts and matches these redundancies of files. This dramatically reduces incremental storage growth: instead of duplicating whole objects through versioning, only the changes are saved. You can keep detailed version histories without drastically growing your storage footprint. ### Enabling Versioning UltiHash’s versioning is modeled on S3 semantics and available via the standard S3-compatible API. By default, versioning is not enabled, and you need to explicitly turn it on through UltiHash S3-compatible API before it starts tracking versions. Versioning is bucket-scoped and must be explicitly enabled first. Once turned on, every update or overwrite to an object is saved as a new, immutable version rather than replacing the old one. Each version is automatically assigned a unique `versionId`, which is returned after each write ( `PUT` ) operation: this lets you retrieve, compare, or roll back to any specific version whenever needed.`‍` ```javascript aws s3api put-bucket-versioning \ --bucket your-bucket-name \ --versioning-configuration Status=Enabled ``` Once versioning is enabled: * Every `PUT` or overwrite generates a new immutable version with its own `versionId`, so older versions are always preserved. * By default, `GET` requests return the latest version unless you explicitly request a specific `versionId`. * Overwriting an object doesn’t delete previous data anymore, every change is saved as a new version. If you need to permanently remove an object (including all its versions), you must explicitly delete each version using its `versionId`. #### Example: Tracking a Model Artifact ```javascript aws s3api put-bucket-versioning \ --bucket models \ --versioning-configuration Status=Enabled aws s3api put-object --bucket models --key resnet50.pt --body ./resnet50.pt # returns versionId: "3a4b6..." # Upload a new version aws s3api put-object --bucket models --key resnet50.pt --body ./resnet50_v2.pt # returns versionId: "5f1c2..." ``` In this example, we first upload an initial model file (`resnet50.pt`) to the `models` bucket. Later, we upload an updated file (`resnet50_v2.pt`) to the same object key, which is the unique name (or "path") used to reference that object in the bucket. While the key stays the same, the actual object content changes, and with versioning enabled, each change automatically creates a new, immutable version. Each `PUT` command returns a unique `versionId`, allowing you to precisely track, retrieve, or roll back to any specific version at any time without overwriting earlier data. Each version is accessible for audit, rollback, or comparison. When retrieving an object, if you don’t specify a `versionId`, you’ll always receive the latest version by default. To download a specific version instead, you should include the `--version-id` parameter: ```javascript aws s3api get-object --bucket models --key resnet50.pt --version-id 3a4b6... ./resnet50_v1.pt ``` The part `./resnet50_v1.pt` is manually chosen by you as the local filename to save the downloaded file, UltiHash does not add this automatically. ### Namespace Behavior UltiHash treats each object as unique based on its full namespace that is, the combination of bucket name and object key.`‍` ```javascript bucket-1/data/file.csv bucket-1/data-v2/file.csv ``` In this example, even if the content of both files is identical, they’re considered two separate objects with their own independent version histories because the namespaces are different. Even if versioning is not enabled in this case, UltiHash’s deduplication engine will still detect and reduce storage for shared content, but versioning always operates per object key. # Customize your deployment How to further customise your UltiHash Self-Hosted deployment via Kubernetes and Helm The Helm chart used during installation is flexible and allows for various configurations to fine-tune the UltiHash setup according to your specific needs. Below are key areas where you might want to make changes: ### Ingress configuration * **Purpose**: Configure how the UltiHash cluster is accessed externally. * **Example**: Set up Ingress with specific annotations and TLS configuration. ```yaml entrypoint: ingress: host: annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/proxy-body-size: "0" tls: - hosts: - secretName: ``` * **Recommendation**: Ensure the ingress controller is configured for your environment (e.g., Nginx) and that TLS is used for secure communication. ### Resource allocation * **Purpose**: Adjust the resource allocations for service replicas. * **Example**: Customize resource requests and limits for critical services. ```yaml etcd: resources: limits: memory: "2Gi" cpu: "500m" entrypoint: resources: limits: memory: "16Gi" cpu: "8" database: primary: resources: limits: memory: "16Gi" cpu: "8" deduplicator: resources: limits: memory: "64Gi" cpu: "16" storage: resources: limits: memory: "32Gi" cpu: "16" ``` * **Recommendation**: Adjust resources to balance performance with cost. *** You also might need to adjust Kubernetes-specific settings to optimize the UltiHash deployment: ### Node affinity and tolerations * **Purpose**: Control where pods are scheduled within your Kubernetes cluster. * **Example**: Use node affinity to keep storage pods on different nodes. ```yaml storage: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: role operator: In values: - storage topologyKey: kubernetes.io/hostname ``` * **Recommendation**: Use affinity rules to optimize performance and ensure critical services run on appropriate nodes. *** ### Secret management * **Purpose**: Securely manage and rotate Kubernetes secrets used by the UltiHash cluster. * **Example**: Rotate a Kubernetes secret without causing downtime: ```bash kubectl delete secret docker-registry registry-credential kubectl create secret docker-registry registry-credentials \\\\ --docker-server='registry.ultihash.io' \\\\ --docker-username='' \\\\ --docker-password='' # Gracefully restart UltiHash services to apply the changes helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster -n --values.yaml --force ``` * **Recommendation**: Rotate secrets periodically and implement alerts for expired or compromised secrets. # Monitor your cluster UltiHash can be integrated with various monitoring, logging, and analytics tools. Below are key integrations you may want to configure: ### 1. Telemetry and monitoring * **Purpose**: Export metrics and logs to external systems like Prometheus and Loki. * **Example**: Configure the OpenTelemetry Collector to export data to Prometheus. Please refer to the [Open Telemetry documentation](https://opentelemetry.io/docs/collector/). ```yaml collector: config: exporters: prometheus/mycompany: endpoint: "1.2.3.4:1234" service: pipelines: metrics: receivers: - otlp - prometheus/mycompany ``` * **Recommendation**: Set up monitoring early to ensure you can track system performance and diagnose issues as they arise. The metrics exported by the UltiHash cluster are listed below, categorized into multiple groups. #### Storage service requests * `storage_read_fragment_req`: number of requests received for reading a fragment * `storage_read_address_req`: number of requests received for reading an address * `storage_write_req`: number of requests received for writing data * `storage_sync_req`: number of requests received to sync data to persistent storage * `storage_remove_fragment_req`: number of requests received to remove a fragment from storage * `storage_used_req`: number of requests received to get the used space #### Deduplicator service requests * `deduplicator_req`: number of requests received to deduplicate uploaded data #### Entrypoint service requests * `entrypoint_abort_multipart_req`: number of [`AbortMultipartUpload`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) requests received * `entrypoint_complete_multipart_req`: number of [`CompleteMultipartUpload`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) requests received * `entrypoint_create_bucket_req`: number of [`CreateBucket`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateBucket.html) requests received * `entrypoint_delete_bucket_req`: number of [`DeleteBucket`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_DeleteBucket.html) requests received * `entrypoint_delete_object_req`: number of [`DeleteObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) requests received * `entrypoint_delete_objects_req`: number of [`DeleteObjects`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) requests received * `entrypoint_get_bucket_req`: number of [`GetBucket`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_GetBucket.html) requests received * `entrypoint_get_object_req`: number of [`GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) requests received * `entrypoint_head_object_req`: number of [`HeadObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) requests received * `entrypoint_init_multipart_req`: number of [`CreateMultipartUpload`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) requests received * `entrypoint_list_buckets_req`: number of [`ListBuckets`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html) requests received * `entrypoint_list_multipart_req`: number of [`ListMultipartUploads`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html) requests received * `entrypoint_list_objects_req`: number of [`ListObjects`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html) requests received * `entrypoint_list_objects_v2_req`: number of [`ListObjectsV2`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) requests received * `entrypoint_multipart_req`: number of [`UploadPart`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) requests received * `entrypoint_put_object_req`: number of [`PutObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) requests received #### Utilization Metrics * `gdv_l1_cache_hit_counter`: Hit count of the L1 cache in the `global_data_view` * `gdv_l1_cache_miss_counter`: Miss count of the L1 cache in the `global_data_view` * `gdv_l2_cache_hit_counter`: Hit count of the L2 cache in the `global_data_view` * `gdv_l2_cache_miss_counter`: Miss count of the L2 cache in the `global_data_view` * `deduplicator_set_fragment_counter`: The number of fragments pointed in the deduplicator set maintained by the `deduplicator service` * `deduplicator_set_fragment_size_counter`: The aggregated size of fragments pointed in the deduplicator set maintained by the `deduplicator service` * `entrypoint_ingested_data_counter`: The total data volume ingested by a `entrypoint service` * `entrypoint_egressed_data_counter`: The total data volume egressed by a `entrypoint service` * `entrypoint_original_data_volume_gauge`: The original/raw data volume in the storage cluster, maintained by the `entrypoint service` * `active_connections`: Number of currently handled connections * `storage_available_space_gauge`: Storage space available to a `storage service` instance * `storage_used_space_gauge`: Storage space used by a `storage service` instance # Scale your cluster How to scale your UltiHash cluster based on your needs Scaling the UltiHash cluster allows you to adjust the number of replicas for core services, such as the storage and deduplicator components, based on workload demand. You may need to scale when workloads increase, resource utilisation remains high, or to improve redundancy and high availability. By increasing the number of replicas, you can handle more concurrent requests, improve throughput, and reduce the impact of node or pod failures. UltiHash is deployed via a Helm chart, which lets you update replica counts for each service and roll out changes across the cluster. *** ### How to scale To scale the storage and deduplicator services: 1. Update the replica counts using either your values.yaml file or the --set flags in the Helm command. 2. Run the following command: ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n \ --set storage.replicas=3,deduplicator.replicas=3 ``` * Replace \ with your Helm release name. * Replace \ with the Kubernetes namespace where UltiHash is deployed. * Adjust the replica counts (3 in the example above) as needed. To verify that the replicas have been updated, run: ``` kubectl get pods -n ``` This will display the current state of all pods in the namespace, showing the updated replica counts for each service. # Update your cluster How to apply update your UltiHash cluster's software and license Updating your UltiHash cluster ensures that you are running the latest version of the software and that any license changes are properly applied. Updates are performed using Helm, which allows you to replace pods gradually (rolling updates) or force an update when required. *** ### Rolling updates Rolling updates allow you to safely apply updates to UltiHash components without causing downtime. This process replaces pods incrementally so that workloads continue to run while updated pods come online. Use rolling updates whenever you upgrade to a new version or make configuration changes. #### How to perform a rolling update 1. Confirm that the desired UltiHash version is available in the registry and ensure you have a backup of your configuration and data. 2. Run the following Helm command: ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n \ --version 1.3.0 \ --values values.yaml ``` * Replace \ with your Helm release name. * Replace \ with the Kubernetes namespace where UltiHash is deployed. * Update 1.3.0 to the version you want to deploy. * If using a custom configuration, adjust values.yaml accordingly.
You can track the progress with: ``` kubectl get pods -n ``` This will display pods terminating and new pods starting as the update rolls out. *** ### License updates When you update your UltiHash license through the user dashboard, the new license is automatically propagated to your cluster within one hour. #### Apply a license update immediately If you need the license change to take effect right away, you can force an update with Helm: ``` helm upgrade oci://registry.ultihash.io/stable/ultihash-cluster \ -n \ --version 1.3.0 \ --values values.yaml \ --force ``` * Replace \ with your Helm release name. * Replace \ with the Kubernetes namespace where UltiHash is deployed. * Update 1.3.0 to match the version currently running. The --force flag re-deploys all resources and applies the updated license immediately. # Backup + restore your cluster How to backup your UltiHash cluster and restore from backup Backing up your UltiHash cluster ensures that you can recover from data loss or corruption. Backups allow you to capture the full state of the cluster, including configuration and stored data, so you can restore it in-place or in a separate environment if needed. *** ### Backup the cluster You should back up your UltiHash cluster regularly to protect against accidental data loss, corruption, or infrastructure failures. The backup process captures the entire Kubernetes namespace where the cluster is deployed. #### How to create a backup Use [Velero](https://velero.io/) to create a backup of the Kubernetes namespace containing your UltiHash cluster: ``` velero backup create --include-namespaces --wait ``` * Replace \ with a unique name for the backup. * Replace \ with the Kubernetes namespace where UltiHash is deployed. * The --wait flag ensures the command waits until the backup is complete before exiting.
Perform backups frequently enough that you can avoid substantial data loss if a disaster occurs. *** ### Restore the cluster If data is lost or corrupted, you can restore your UltiHash cluster from a previously created backup. Restores can be performed in-place (into the same cluster) or in a separate environment, which is useful for migrations or testing recovery procedures. #### How to restore from a backup Run the following Velero command, specifying the name of the backup you want to restore: ``` velero restore create --from-backup --wait ``` * Replace \ with a name for the restore operation. * Replace \ with the name of the backup you are restoring from. * The --wait flag ensures the command waits until the restore is complete before exiting. Restoring to a separate environment is often useful when migrating a cluster or testing your disaster recovery plan. # Manage users + access policies How to manage users and configure policies to restrict access UltiHash incorporates policy-based access management, enabling users to configure access restrictions at a granular level across datasets. This ensures that sensitive data is only accessible to authorized users, allowing organizations to meet regulatory requirements while protecting critical information. *** ## User Authentication By default, all requests to the cluster are rejected. You need to explicitly enable access to buckets and objects for users. UltiHash implements a subset of AWS's IAM calls to enable user management and allows you to assign policies to buckets and users. Any access to UltiHash cluster is authenticated using *AWS Signature Version 4*. This requires you to set an *access key* and a *secret key* for request authentication. Depending on the S3 client the way how you pass these credentials may vary. We will use AWS CLI in the following examples. You can install it using the following commands: ``` # Create a python virtual environment and activate it python3 -m venv venv $> . venv/bin/activate # Install AWS cli using pip (venv) $> pip install awscli ``` You can configure access to UltiHash cluster in your `$HOME/.aws/config` file. We will create a profile `uh` to be used with UltiHash: ``` [profile uh] endpoint_url = https://my-uh-cluster.my-company.io region=my-region ``` We can now run `aws` commands using the profile parameters: ``` $> aws --profile=uh ... ``` By adding our credentials to `$HOME/.aws.credentials` we can also authenticate ourselfs to the cluster: ``` [uh] aws_secret_key_id = FANCY-ROOT-KEY aws_secret_access_key = SECRET ``` *** ## Root User Account During the first deployment UltiHash creates a root user account in the cluster that can be used to implement further user management. The credentials for the root user are stored as a Kubernetes secret. You can find it out the following way: ``` # Obtain access key ID kubectl get secret -super-user-credentials -n -o jsonpath="{.data.access-key-id}" | base64 --decode; echo # Obtain secret key value kubectl get secret -super-user-credentials -n -o jsonpath="{.data.secret-key}" | base64 --decode; echo ``` Replace the \ and \ with the Helm chart release name and Kubernetes namespace correspondingly. In order to prevent you from locking yourself out of your cluster, requests issued by the root user are not checked against any policy but executed right away. You should not use this account for anything else than cluster administration. *** ## Managing User Accounts ### User Creation and Deletion To create a new user, use IAM calls: ``` $> aws --profile=uh iam create-user --user-name 'foo' { "User": { "Path": "/", "UserName": "foo", "UserId": "e55d4d45-9848-4778-a5b7-2b148cf3b850", "Arn": "arn:uh:iam::da2ce577214c40019c7bfd6397dea8ca:user/foo" } } ``` You can remove users using the `delete-user` call: ``` $> aws --profile=uh iam delete-user --user-name 'foo' ``` ### Access Keys To create a new access key, use `create-access-key` action: ``` $> aws --profile=uh iam create-access-key --user-name 'foo' { "AccessKey": { "UserName": "foo", "AccessKeyId": "atdyycpemrypcgjmynaq", "Status": "Active", "SecretAccessKey": "edkeqpdhkenuqylgwdxbapvfmvgaweda" } } ``` To delete the access key: ``` $> aws --profile=uh iam delete-access-key --access-key-id ACCESS-KEY-ID ``` *** ## Policies UltiHash implements a subset of the IAM policy framework used in AWS. UltiHash allows assigning policies to buckets and to user accounts. Policy documents must be crafted according to the AWS policy grammar (see ) ### User Policies User policies are policies that are assigned to a user. The policies are evaluated only for requests that are issued by that user. They can be used to grant special permissions based on user level. You can assign policies to each user account using IAM actions: To add an action to a user account: ``` # Set a policy allowing global access to any action for user foo $> aws --profile=uh iam put-user-policy --user-name 'foo' --policy-name 'allow-all' \ --policy-document '{ "Version": "2012-10-17", "Statement": { "Sid": "AllowAllForAnybody", "Effect": "Allow", "Action": "*", "Principal": "*", "Resource": "*" } }' ``` To list all assigned actions of a user account: ``` $> aws --profile=uh iam list-user-policies --user-name foo { "PolicyNames": [ "allow-all" ] } ``` To remove an action from a user account: ``` $> aws --profile=uh iam delete-user-policy --user-name foo --policy-name allow-all ``` ### Bucket Policies Bucket policies are policies assigned to a bucket. They are evaluated for all actions that target that bucket. To assign a policy to a bucket: ``` $> aws --profile=uh s3api put-bucket-policy --bucket test --policy '{ "Version": "2012-10-17", "Statement": { "Sid": "AllowAllForAnybody", "Effect": "Allow", "Action": "*", "Principal": "*", "Resource": "*" } }' ``` To read policies assigned to a bucket: ``` $> aws --profile=uh s3api get-bucket-policy --bucket test { "Policy": "{ \"Version\": \"2012-10-17\", \"Statement\": { \"Sid\": \"AllowAllForAnybody\", \"Effect\": \"Allow\", \"Action\": \"*\", \"Principal\": \"*\", \"Resource\": \"*\" } }" } ``` To delete a policy from a bucket: ``` $> aws --profile=uh s3api delete-bucket-policy --bucket test ```
Differences to IAM policy framework UltiHash supports only the following condition operators: * StringEquals * StringNotEquals * StringEqualsIgnoreCase * StringNotEqualsIgnoreCase * StringLike * StringNotLike * NumericEquals * NumericNotEquals * NumericLessThan * NumericLessThanEquals * NumericGreaterThan * NumericGreaterThanEquals Compare to AWS IAM policy operators [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html).
# Erasure coding for data resiliency How to increase the resilience of your cluster with UltiHash's parity-based storage Ensuring data resiliency is a critical concern for distributed storage systems, particularly in mitigating the risks posed by outages or hardware failures. UltiHash employs Reed-Solomon erasure coding to improve service availability and to also enhance data durability. This method organizes data in units of data called stripes, where each stripe gets split up into `k` shards, and an additional `m` additional parity shards are computed and stored for redundancy. With this Reed-Solomon-based erasure coding scheme, an UltiHash storage cluster comprised of `k + m` data nodes offers a usable capacity equal to `k` nodes while being able to tolerate failures of up to `m` nodes without losing data or interrupting service availability. In the event of failing data nodes, the parity shards allow the cluster to reconstruct the original data while keeping the service available, not requiring additional maintenance downtime. This approach provides a robust and efficient mechanism for improving service uptime and for mitigating the risk of data loss. *** ### How to configure storage groups with erasure coding UltiHash supports storage groups can be configured with or without erasure coding enabled, depending on user requirements concerning availability, resilience, and performance. To configure erasure coding in the UltiHash storage cluster, you can define storage groups using Reed-Solomon coding parameters. Each group is characterized by `k` data shards and `m` parity shards, allowing the system to tolerate up to `m` node failures without data loss. Storage groups with erasure coding enabled are specified in the configuration file as follows: ```yaml storage: groups: - id: 0 type: ERASURE_CODING storages: 6 # number of storage instances in the group data_shards: 4 parity_shards: 2 stripe_size_kib: 256 storageClass: local-path size: 70Gi # volume size allocated for each storage instance ``` In this example: * `4` data shards and `2` parity shards, allowing the system to tolerate up to two storage instance failures without data loss. * A total of `6` storage instances is used, matching the required total of `data_shards + parity_shards`. * A `stripe_size_kib` value of `256`, meaning data is handled in stripes of 256 KiB, which are then split up across the data shards. * Each storage instance uses the `local-path` storage class with `70Gi` of allocated volume. Implications: * Fault Tolerance: The group can lose any 2 of its 6 nodes and still recover the original data. * Storage Overhead: The parity overhead is 2 out of 6 shards, or approximately 33%. * Efficiency: This configuration balances improved fault tolerance with relatively efficient storage usage compared to full replication. Recommendations and Limitations: * strip sizes must be evenly divisible by the number of data shards * stripe sizes beyond 4096 KiB rarely make sense and lead to degraded performance * currently, only a single storage group is supported By carefully selecting `k` and `m` values, you can balance storage efficiency and fault tolerance to meet your specific requirements. ### How to configure storage groups without erasure coding In use cases where data resilience is not a major concern, e.g. in cases where temporary data or intermediate results are stored, UltiHash supports setting up storage groups exposing the raw storage resources with no erasure coding applied: ```yaml storage: groups: - id: 0 type: ROUND_ROBIN storages: 6 storageClass: local-path size: 70Gi ``` In this example: * `6` storage instances are defined, and data is distributed sequentially across them. * Each storage instance uses the `local-path` storage class with `70Gi` of allocated volume. Implications: * Data Distribution: Objects are written to storage backends in round-robin order, without redundancy or parity. This ensures even distribution of storage utilization across the devices. * Fault Tolerance: There is no inherent fault tolerance—if a storage instance fails, the data it holds may be permanently lost unless external redundancy mechanisms (e.g., backups or replication) are used. * Storage Efficiency: Full storage capacity is available for use. No additional space is reserved for parity or replication, making this setup 100% efficient in terms of raw capacity. * Performance: This method provides low overhead and potentially better write throughput, as there is no encoding or parity computation involved. This configuration is best suited for scenarios where performance and storage efficiency are more critical than fault tolerance, or where durability is managed at a higher layer in the stack. # Set up encryption How to enable in-flight and at-rest encryption on your UltiHash cluster ### In-Flight (TLS) Encryption In-flight encryption for UltiHash can be configured on two different levels: * Internal Ingress object - part of the UltiHash deployment on Kubernetes * External load balancer - a proxy that fronts Kubernetes cluster and distributes traffic to its nodes #### Configure TLS on the Ingress level The Helm chart deploys UltiHash with a dedicated [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/#what-is-ingress) object by default. The object is located in the same namespace as the deployed UltiHash cluster and can be found by executing the command below (replace `` with the Kubernetes namespace where your UltiHash cluster is deployed): ```bash kubectl get ingress -n ``` The Ingress object can be configured in a custom way via Helm values: ```yaml entrypoint: ingress: # Make sure Ingress is enabled to expose UltiHash cluster outside your Kubernetes cluster enabled: true # Specify a domain name under which the UltiHash cluster will be accessible outside the Kubernetes cluster host: example.domain.name # Add annotations specific for your Ingress controller if required annotations: {} # Configure in-flight encryption by using TLS tls: [] ``` To enable TLS encryption at Ingress level perform the following actions: * register a domain name for UltiHash cluster in your private or public DNS server (for example `example.domain.name`) * generate TLS private key and certificate bound to your registered domain name * create a Kubernetes secret in your UltiHash namespace with TLS credentials as shown [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls) (assume the secret's name is `secret-tls`) * enable the Ingress object using your domain name and the corresponding TLS secret in the helm values: ```yaml entrypoint: ingress: enabled: true host: example.domain.name annotations: {} tls: - hosts: - example.domain.name secretName: secret-tls ``` #### Configure TLS encryption on the external load balancer level The TLS configuration for an external load balancer heavily depends on the type of the used load balancer. Please refer to the documentation of your load balancer. [This guide](https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/) shows the TLS configuration for a [network load balancer](https://aws.amazon.com/elasticloadbalancing/network-load-balancer/) on AWS (the recommended load balancer type to use on AWS for higher performance). In the case of [Nginx Inress controller](https://docs.nginx.com/nginx-ingress-controller/) on AWS EKS cluster, here is an example of a network load balancer configuration with TLS. To provision a network load balancer automatically, the [AWS load balancer controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/) has to be installed in advance. Modify the helm values of the [Nginx Ingress controller's helm chart](https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx) in the following way: ```yaml controller: service: type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-name: nlb-name # Specify the name for the load balancer service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing # Specify the scheme for the load balancer (internal or internet-facing) service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /healthz service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: 10254 service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xxxx, mySubnet # Specify the subnet IDs or name in which the load balancer has to be provisioned service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:3223213123233:certificate/c6a3ff73-3eb8-4e72-9e68-2dsa4cce549c # Specify ARN of the ACM certificate (has to be provisioned in advance) service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http ``` ### Encryption at Rest The configuration of at-rest encryption for UltiHash cluster depends on the CSI driver used on your Kubernetes cluster. Please refer to the documentation of your CSI driver. The example below shows how to enable encryption at rest for data stored in UltiHash in the case of [AWS EBS CSI driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver?tab=readme-ov-file). The driver has to be installed in advance before deploying UltiHash cluster, please follow the [official guide](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md). After the EBS CSI driver is installed, provision a storage class on your Kubernetes cluster with the configuration as shown below: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: example-storage # Specify a name for the storage class parameters: encrypted: "true" # Enable the CSI driver to encrypt the EBS volumes it provisions type: gp3 # Select the required type of EBS volumes to provision ('gp2', 'gp3', 'io1', or other) # kmsKeyID: "" # (optional): specify the KMS key to encrypt the provisioned EBS volumes. If omitted, the AWS-managed KMS key will be used provisioner: ebs.csi.aws.com reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true ``` To enable UltiHash cluster using the storage class above, specify it in the helm values of the UltiHash helm chart: ```yaml etcd: persistence: storageClass: example-storage database: primary: persistence: storageClass: example-storage storage: storageClass: example-storage deduplicator: storageClass: example-storage ``` # Changelog All the latest changes to UltiHash's core image and Helm chart {% content-ref url="/pages/aBN5QtSo1xfYMa4pct9m" %} [Core image](/reference/changelog/core-image) {% endcontent-ref %} {% content-ref url="/pages/vtKZGTv2UaCFQrjqJy3b" %} [Helm chart](/reference/changelog/helm-chart) {% endcontent-ref %} # Core image ## \[1.6.0] 2025-07-16 **Enhancements** * Add support for S3-compatible object versioning **Bug Fixes** * Fixes an issue when operating erasure-coded storage groups with disabled deduplication * Fixes an issue where storage service instances might wrongfully acknowledge write operations of data that has not been flushed to persistent storage yet ## \[1.5.0] 2025-07-08 **Enhancements** * Improved write performance of erasure-coded storage groups * Enabled deletion support for erasure-coded storage groups **Bug Fixes** * Fix issue with GetObject requests which could lead to stale downloads with high CPU utilization ## \[1.4.0] 2025-06-04 **Enhancements** * Add support for erasure-coded storage groups based on Reed-Solomon coding techniques, improving the availability and resilience of UltiHash storage clusters. **Other** * Upgraded employed libraries to their latest available releases. ## \[1.3.1] 2025-04-04 **Other** * Added new OTEL metrics to track license-related attributes. ## \[1.3.0] 2025-02-24 **Enhancements** * **Implemented support for pay-as-you-go license:** Enabled charges based solely on actual used storage capacity. This prevents overpayments for unused disk space. * **Implemented support for per-bucket CORS configuration:** Enabled using `s3:PutBucketCors`, `s3:GetBucketCors` and `s3:DeleteBucketCors` to configure cross-origin interaction. * **Implemented support for pre-signed URL:** Enabled writing and reading S3 objects by using pre-signed URL bypassing IAM policies. **Other** * **Optionally disable deduplication:** The option allows disabling deduplicator service completely. ## \[1.2.1] 2025-02-21 **Bug Fixes** * **S3 Requests Validation Improved:** Now AWS SDK for NodeJS can communicate with UH cluster without producing validation-related errors. ## \[1.2.0] 2025-01-21 **Enhancements** * **Implemented OTEL Traces Support:** Enabled generating OTEL traces to improve debugging capabilities. * **Improved Cluster Startup:** Enabled the cluster to not fail if etcd gets unavailable. **Bug Fixes** * **Concurrency Issue Resolved:** Cluster became more reliable by preventing concurrency problems. * **Log Level Option Respected:** Export of cluster logs respects the log level set by user instead of exporting all existing logs regardless of the requested log level. * **Storage Reclaiming Issue Resolved:** Fixed the issue that would prevent completely reclaiming storage of deleted objects. ## \[1.1.2] 2024-12-19 **Enhancements** * **Optimized Write Operations:** Refined the `STORAGE_WRITE_REQ` message for more efficient coordination between the deduplication engine and the storage layer, boosting overall write performance. * **Improved Reference Counting:** Enhanced the reference counting infrastructure within the storage service to better handle large-scale workloads, ensuring higher efficiency and stability. **Bug Fixes** * **Content-Range Header Issue Resolved:** Fixed an issue where incorrect return values were provided in the `Content-Range` header during ranged GET requests, ensuring accurate and standards-compliant responses. ## \[1.1.1] 2024-12-09 * Request downstream error to be treated as an internal error to allow the client to retry * Synchronized access to database directory ## \[1.1.0] 2024-11-11 * Re-connect when losing connection to etcd * Support authentication against etcd * Correctly report original size in responses to PutObject and CompleteMultipartUpload * Fix request ID for `100 Continue` responses * Add support for policy variables: `uh:ActionId`, `uh:ResourceArn`, `aws:username`, `aws:userid`, `aws:PrincipalArn`, `aws:SourceIp`, `aws:referer`, `aws:UserAgent`, `s3:x-amz-content-sha256`, `s3:x-amz-copy-source`, `s3:delimiter`, `s3:prefix`, `aws:CurrentTime` * Add default content-type for objects * Fix a bug in parsing HTTP range requests * Remove output to log.log * Add support for date condition operators: `DateEquals`, `DateNotEquals`, `DateLessThan`, `DateLessThanEquals`, `DateGreaterThan`, `DateGreaterThanEquals` * Conditions should be less than `2261-01-01T00:00:00Z` * Reduced size of object metadata * Improved read performance * Fix a bug in storage, leading to failed deduplication requests * Improved robustness for successive write-delete-write command sequences and small object sizes * Fix bug on `NotAction`, `NotPrincipal`, `NotResource`: implement NOR for multiple context values * Implement special characters: `${*}`, `${?}`, `${$}` * Support variable on resource policy statement * Prevent services from being destroyed while etcd is not booted up * Restrictions on naming buckets have been resolved (e.g. cannot start with a number) ## \[1.0.0] 2024-10-15 * Added basic policy and authentication support * Finished deletion support ## \[0.6.0] 2024-09-18 * The storage service can now perform delete operations and reclaim disk space * Deduplicator and Entrypoint services are now making use of storage-level delete operations * Fix potential data corruption issue caused by deleting and re-uploading data * Partial implementation of erasure coding / storage groups * Partial implementation of authentication / policy support ## \[0.5.4] 2024-08-08 * Support for object mime types * Improved handling of multipart uploads * Added support for using multiple working directories/storage devices to storage service * Added link/unlink interface to storage service, enabling deletion of unused data ## \[0.5.3] 2024-07-01 * Several S3A compatibility fixes ## \[0.5.2] 2024-06-25 * Increased waiting time for etcd to 5 minutes ## \[0.5.1] 2024-06-24 * Improved compatibility level with S3 API * Fixed handling of XML-reserved characters in object keys * Added REST API for querying raw and effective size of data in the cluster * Fixed incorrect XML in S3 API responses * Fixed occasional application hangs, resulting in bad performance * Improved deduplication algorithm * More efficient storage usage by truncating unused spaces in files ## \[0.5.0] 2024-06-06 * Removed support for S3 Outpost's GetBucket * Deduplication statistic collection * Improve API support for PySpark (copy object, head object, etc.) * Replace directory service with PostgreSQL * Stablize memory requirements for deduplication * Performance improvements for deduplication ## \[0.4.2] 2024-05-22 * Fixed two bugs that result in poor deduplication * Support running as monolith * A few bug fixes in S3 API * Performance improvements ## \[0.4.1] 2024-04-26 * Fixed a bug where data corruption could occur when using multiple deduplicator instances * Implemented asynchronous write support in storage service * Improved write throughput of storage service * Improved handling of command line arguments * Reduced memory footprint of entrypoint service * Fixed timeout issues when uploading large objects * Improved throughput of deduplicator service * Fixed formatting of timestamps in HTTP response * Fixed computation of ETAG ## \[0.4.0] 2024-03-26 * Refactored data store, mostly lockless writes and completely lockless reads * Fixed a bug in the free spot manager * Fixed concurrency issue in directory service * Fix hanging server when requesting empty files * Use buffered file I/O to improve performance of deduplicator WARNING: this breaks compatibility with instances * Caching fragments before writing them to storage ## \[0.3.4] - 2024-03-15 * Fixed delete objects raising segmentation fault in entrypoint ## \[0.3.3] - 2024-03-15 * Fixed entity too small error in entrypoint ## \[0.3.2] - 2024-03-14 * Fixed cache usage * Object Metadata in bucket list response * More unit tests and fixes ## \[0.3.1] - 2024-03-14 * Implemented pagination for better download behaviour * Fixed potential locking issue in the worker pool * Reduced log messages to single line entries * Fixed computation of space savings for empty objects ## \[0.3.0] - 2024-03-13 * Fixed potential log corruption issue in deduplicator * Increase maximum storage service size * Logging of service requests * Add counter counting active connections ## \[0.2.1] - 2024-03-06 * Fix access violations in entrypoint and deduplicator * Fix memory leak in license validation * Fix double-free in MD5 checksum computation * Fix use-after-free in directory bucket deletion * Finalized multipart upload and pending deletion of multipart uploads * Propagate errors about missing downstream services in entrypoint ## \[0.2.0] - 2024-02-29 * Implement etcd-based service coordination * Use open-telemetry for metrics and log reporting * Implement basic licensing * Fix several stability and performance issues # Helm chart ## \[1.6.0] 2025-07-17 * Update core Docker image tag to 1.6.0 * Bump etcd to chart version 12.0.10 / app version 3.6.2 * Bump postgresl to chart version 16.7.19 / app version 17.5.0 * Bump opentelemetry-collector to chart version 0.129.0 / app version 0.130.0 * Bump prometheus-node-exporter to chart version 4.47.1 / app version 1.9.1 ## \[1.5.0] 2025-07-09 * Update core Docker image to 1.5.0 * Upgraded employed libraries to their latest available releases ## \[1.4.0] 2025-06-05 * Introduce support for erasure coding (EC) storage groups * Remove support for multiple storage volumes per storage instance * Update core Docker image tag to 1.4.0 ## \[1.3.1] 2025-04-07 * Update core Docker image tag to 1.3.1 ## \[1.3.0] - 2025-03-04 * Rename recovery service to coordinator * Enable optional deactivation of deduplicator and OTEL collector * Update core Docker image tag to 1.3.0 ## \[1.2.0] - 2025-01-22 * Update core Docker image tag to 1.2.0 ## \[1.1.2] - 2024-12-19 * Update core Docker image tag to 1.1.2 ## \[1.1.1] - 2024-12-09 * Update the subcharts to the recent versions * Update core Docker image tag to 1.1.1 ## \[1.1.0] - 2024-11-13 * Implemented support for password-based authentication against etcd * Implemented S3A tests support * Implemented PySpark test support * Made the `database-init` job name unique to prevent `helm upgrade` failing * Added support for extra volumes and volume mounts * Updated core Docker image tag to 1.1.0 ## \[1.0.0] - 2024-10-16 * Implemented support for global policies * Implemented automated credentials generation for the UH root user * Update core Docker image tag to 1.0.0 ## \[0.6.0] - 2024-09-19 * Implemented flexible database schema management with Flyway * Introduced the readiness probe for the Entrypoint service * Introduced EC recovery service * Update core Docker image tag to 0.6.0 ## \[0.5.5] - 2024-08-08 * Added support for using multiple working directories/storage devices to storage service * Update core Docker image tag to 0.5.4 ## \[0.5.4] - 2024-07-05 * Provide ability to define custom ENV for the stability test ## \[0.5.3] - 2024-07-02 * Update core Docker image tag to 0.5.3 ## \[0.5.2] - 2024-06-27 * Remove initContainers from the UH services * Update core Docker image tag to 0.5.2 ## \[0.5.1] - 2024-06-24 * Implement custom environment variables support for UH services * Implement proxy URL support for the init containers * Update core Docker image tag to 0.5.1 ## \[0.5.0] - 2024-06-07 * Update core Docker image tag to 0.5.0 * Enable changing parameters of the stability test * Implemenent PostgreSQL database instead Directory ## \[0.4.3] - 2024-05-22 * Update core Docker image tag to 0.4.2 * Enable passing aditional parameters to the stability test ## \[0.4.2] - 2024-04-29 * Update core Docker image tag to 0.4.1 * Implement fixes for the release 0.4.1 ## \[0.4.1] - 2024-04-18 * Introduce the stability test * Make the init-container script multiplatform * Implement tolerations support ## \[0.4.0] - 2024-03-27 * Update core Docker image tag to 0.4.0 ## \[0.3.3] - 2024-03-25 * Implement image pull policy support * Implement feature testing support * Introduce consistent Helm labels * Implement security context fixes ## \[0.3.2] - 2024-03-19 * Implement security context for the UH services ## \[0.3.1] - 2024-03-15 * Update core Docker image tag to 0.3.4 * Enable collecting pod metrics from Kubelet ## \[0.3.0] - 2024-03-14 * Update core Docker image tag to 0.3.2 * Implement telemetry export support for the entrypoint service * Enable only required metric collectors for Prometheus Node Exporter ## \[0.2.0] - 2024-03-06 * Update core Docker image tag to 0.2.1 * Implement Kubernetes metadata collection * Implement support for changing log level and telemetry collection interval * Implement resource budgets * Implement placement policies