Label studio and Minio Local Setup

Label Studio Support
2

Hello,

Thank you for providing detailed information about your setup and the issue you’re encountering with connecting Label Studio to your MinIO instance.

Based on the error message:

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://minio:9001/images"

It seems that the Label Studio application container cannot resolve or connect to the MinIO service at the hostname minio on port 9001.

Problem Analysis

  • Separate Docker Networks: Since you are running Label Studio and MinIO in separate Docker Compose files, they are likely on different Docker networks. This means that the app service in your Label Studio setup cannot resolve the hostname minio to connect to the MinIO service.

Solution

To resolve this issue, you need to ensure that both Label Studio and MinIO services are on the same Docker network so they can communicate with each other. Here’s how you can do it:

1. Combine Docker Compose Files

Combine your docker-compose.yml and minio-compose.yml into a single docker-compose.yml file. This ensures that both services are on the same network and can resolve each other’s hostnames.

Updated docker-compose.yml:

version: '3.9'
services:
  nginx:
    build: .
    image: heartexlabs/label-studio:latest
    restart: unless-stopped
    ports:
      - '6060:80'  # Map to port 80 inside the container
      - '6061:443' # Map to port 443 inside the container
    depends_on:
      - app
    environment:
      - LABEL_STUDIO_HOST=${LABEL_STUDIO_HOST:-192.168.1.29}
    volumes:
      - ./mydata:/label-studio/data:rw
      - ./deploy/nginx/certs:/certs:ro
      - ./deploy/default.conf:/etc/nginx/nginx.conf  # Mount the custom nginx.conf
    command: nginx

  app:
    stdin_open: true
    tty: true
    build: .
    image: heartexlabs/label-studio:latest
    restart: unless-stopped
    expose:
      - '6000'
    depends_on:
      - db
      - minio
    environment:
      - DJANGO_DB=default
      - POSTGRE_NAME=postgres
      - POSTGRE_USER=postgres
      - POSTGRE_PASSWORD=
      - POSTGRE_PORT=5432
      - POSTGRE_HOST=db
      - LABEL_STUDIO_HOST=${LABEL_STUDIO_HOST:-192.168.1.29}
      - JSON_LOG=1
      - LABEL_STUDIO_LOCAL_FILES_SERVING_ENABLED=true
      - LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT=/home/cypher
      - MINIO_ENDPOINT=http://minio:9000
      - MINIO_ACCESS_KEY=your_username
      - MINIO_SECRET_KEY=your_pasword
      - MINIO_BUCKET=images
    volumes:
      - ./mydata:/label-studio/data:rw
    command: label-studio-uwsgi

  db:
    image: pgautoupgrade/pgautoupgrade:13-alpine
    hostname: db
    restart: unless-stopped
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
    volumes:
      - ${POSTGRES_DATA_DIR:-./postgres-data}:/var/lib/postgresql/data
      - ./deploy/pgsql/certs:/var/lib/postgresql/certs:ro

  minio:
    image: bitnami/minio:2022
    restart: unless-stopped
    ports:
      - '9000:9000'
      - '9001:9001'
    volumes:
      - minio_data:/data
    environment:
      - MINIO_ROOT_USER=your_username
      - MINIO_ROOT_PASSWORD=your_pasword
    command: minio server /data --console-address ":9001"

volumes:
  minio_data:
    driver: local

Explanation:

  • Services:
    • minio: Added the MinIO service to the same Docker Compose file.
    • app: Updated depends_on to include minio.
  • Environment Variables:
    • MINIO_ENDPOINT: Set to http://minio:9000. Since both services are on the same network, the minio hostname can be resolved by the app service.
    • MINIO_ACCESS_KEY and MINIO_SECRET_KEY: Set to match with MinIO credentials.
    • MINIO_BUCKET: Set to images, which is the bucket you created.
  • Volumes:
    • Defined a minio_data volume for MinIO to persist data.

2. Update MinIO Configuration

  • Bucket Creation: Ensure that the images bucket exists in MinIO.
  • User and Policy:
    • Create a user and assign the appropriate policy to allow Label Studio to access the bucket.
    • Use the policy you’ve provided, ensuring it is correctly applied to the user.

Policy Example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::images"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:DeleteObject", "s3:GetObject", "s3:PutObject"],
      "Resource": ["arn:aws:s3:::images/*"]
    }
  ]
}

3. Rebuild and Start Services

Run the following command to build and start the updated Docker Compose configuration:

docker-compose up -d --build

4. Configure Label Studio Cloud Storage

In Label Studio, set up the S3 storage connection:

  • Go to: Project Settings > Cloud Storage
  • Add Source Storage:
    • Type: AWS S3
    • Bucket Name: images
    • Region Name: (Leave empty or set if required)
    • S3 Endpoint: http://minio:9000
    • Access Key ID: your_username (Same as MINIO_ROOT_USER)
    • Secret Access Key: your_pasword (Same as MINIO_ROOT_PASSWORD)
    • Use SSL: Unchecked (since MinIO is running without SSL)
    • Use Presigned URLs: Unchecked
    • Treat every bucket object as a source file: Check if you want to import all objects
  • Check Connection: Click to verify the connection.
  • Add Storage: Save the configuration.

5. Troubleshooting

If you still encounter issues:

  • Hostnames and Networking:
    • Ensure that the minio hostname is resolvable from the app container. You can test this by entering the app container shell and pinging minio:
docker exec -it <app_container_name> bash
ping minio
  • Alternate Endpoint:
    • If the above does not work, try using host.docker.internal:
      • Update MINIO_ENDPOINT in the app service to:
- MINIO_ENDPOINT=http://host.docker.internal:9000

* In Label Studio Cloud Storage settings, set the S3 Endpoint to `http://host.docker.internal:9000`.
* Note: `host.docker.internal` allows Docker containers to access services running on the host machine.
  • Ports and Firewall:
    • Ensure that ports 9000 and 9001 are not blocked by any firewall settings.

6. Verify Access Keys and Secrets

  • Double-check that the MINIO_ACCESS_KEY and MINIO_SECRET_KEY in your app service match the MinIO credentials.

7. Ensure Correct MinIO Command

  • In your minio service, use the following command to start MinIO with the console address:
command: minio server /data --console-address ":9001"

Additional Information

  • Context from Documentation:
    • According to the Label Studio and MinIO integration guides, using http://minio:9000 as the S3 Endpoint within the same Docker network is standard practice.
    • It is important to disable Use Presigned URLs when MinIO is accessible directly within the Docker network.
  • Common Issues:
    • Hostname Resolution: When services are in different Docker Compose files, they cannot resolve each other’s hostnames because they are on separate networks.
    • Separate Networks: Combining the services into one Docker Compose file places them on the same network, allowing hostname resolution.

Summary

  • Combine Docker Compose Files: Ensure all services are in a single docker-compose.yml to allow networking between them.
  • Configure Environment Variables: Set the correct MINIO_ENDPOINT, MINIO_ACCESS_KEY, and MINIO_SECRET_KEY.
  • Update Label Studio Settings: Configure the S3 storage in Label Studio with the correct endpoint and credentials.
  • Test Connectivity: Verify that the app container can reach the minio service.
  • Check MinIO Policies: Ensure that the policies are correctly set to allow necessary S3 actions.

References

1 Like