Hep.gg

Examples

Client configs for aws-cli, AWS SDK v3, boto3, and rclone.

Examples

Copy-pasteable configs for the common S3 clients. All examples target the primary instance (https://s3.hep.gg, region ca-central-hydra-1). For a bucket on the secondary instance, swap the endpoint to https://ca-east-hydra-1.s3.hep.gg and the region to ca-east-hydra-1. Replace the credentials with your bucket's key from the dashboard.

AWS CLI

The CLI uses path-style automatically when you pass --endpoint-url. You still need the region.

aws-cli
# Credentials for the bucket
export AWS_ACCESS_KEY_ID=<your-bucket-access-key>
export AWS_SECRET_ACCESS_KEY=<your-bucket-secret-key>
 
# List objects
aws --endpoint-url https://s3.hep.gg --region ca-central-hydra-1 \
    s3 ls s3://my-bucket/
 
# Upload
aws --endpoint-url https://s3.hep.gg --region ca-central-hydra-1 \
    s3 cp ./file.zip s3://my-bucket/
 
# Download
aws --endpoint-url https://s3.hep.gg --region ca-central-hydra-1 \
    s3 cp s3://my-bucket/file.zip ./
 
# Delete
aws --endpoint-url https://s3.hep.gg --region ca-central-hydra-1 \
    s3 rm s3://my-bucket/file.zip

AWS SDK for JavaScript (v3)

forcePathStyle: true is required.

@aws-sdk/client-s3
import {
  S3Client,
  PutObjectCommand,
  GetObjectCommand,
  ListObjectsV2Command,
} from "@aws-sdk/client-s3";
 
const s3 = new S3Client({
  endpoint: "https://s3.hep.gg",
  region: "ca-central-hydra-1",
  forcePathStyle: true,
  credentials: {
    accessKeyId: "<your-bucket-access-key>",
    secretAccessKey: "<your-bucket-secret-key>",
  },
});
 
// Upload
await s3.send(
  new PutObjectCommand({
    Bucket: "my-bucket",
    Key: "file.zip",
    Body: fileBytes,
    ContentType: "application/zip",
  }),
);
 
// List
const list = await s3.send(new ListObjectsV2Command({ Bucket: "my-bucket" }));
console.log(list.Contents);
 
// Download
const obj = await s3.send(
  new GetObjectCommand({ Bucket: "my-bucket", Key: "file.zip" }),
);
const bytes = await obj.Body.transformToByteArray();

boto3 (Python)

Set addressing_style = "path" via the client Config.

boto3
import boto3
from botocore.config import Config
 
s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.hep.gg",
    region_name="ca-central-hydra-1",
    aws_access_key_id="<your-bucket-access-key>",
    aws_secret_access_key="<your-bucket-secret-key>",
    config=Config(s3={"addressing_style": "path"}),
)
 
# Upload
s3.upload_file("file.zip", "my-bucket", "file.zip")
 
# List
resp = s3.list_objects_v2(Bucket="my-bucket")
for obj in resp.get("Contents", []):
    print(obj["Key"], obj["Size"])
 
# Download
s3.download_file("my-bucket", "file.zip", "file.zip")

rclone

Good for backups and incremental syncs.

rclone.conf
[hepgg]
type = s3
provider = Other
access_key_id = <your-bucket-access-key>
secret_access_key = <your-bucket-secret-key>
endpoint = https://s3.hep.gg
region = ca-central-hydra-1
force_path_style = true
acl = private
rclone sync
# Sync a local folder to a bucket
rclone sync ./photos hepgg:my-bucket/photos --progress