Ṣeto: a small orchestrator for Docker

I have been using Docker Compose for a long time. It works very well when applications run on a single machine, but things become more interesting when containers have to run across several machines.

A small cluster is rarely made of identical machines. One machine may have a GPU, another may have a Zigbee controller, while another may simply provide more storage or CPU resources. Some services can run anywhere, while others are tied to a particular machine.

I created Ṣeto to handle these situations without replacing Docker Compose or Docker Swarm.

The name comes from Yoruba. Ṣeto means "to arrange", "to organise", or "to put in order". This is essentially what the project tries to do: keep the existing Docker tools and automate the infrastructure around them.

When Docker Compose is not enough

Docker Compose is a great way to describe an application:

services:
  web:
    image: nginx

Running it is straightforward:

docker compose up -d

The problem starts when the application is no longer tied to one machine. A container may need to run on a specific node because it requires hardware that is not available elsewhere.

For example, one machine may have an NVIDIA GPU:

node-1
└── NVIDIA GPU

while another may have a Zigbee controller:

node-2
└── /dev/ttyUSB0

A service using the GPU obviously needs to run on node-1, while a service using the Zigbee controller needs to stay on node-2.

Docker Swarm already provides scheduling and placement constraints, but I wanted to describe these requirements in terms of the physical machines themselves.

Hardware-aware placement

Ṣeto adds placement information to the Compose configuration. A service can declare that it needs to run on a machine with a particular device:

x-mode: compose

x-placement:
  - host.device.gpu == true

services:
  inference:
    image: my-inference

Ṣeto discovers the available nodes and evaluates the placement constraints before deploying the application.

This is useful for workloads that need GPUs, cameras, Bluetooth, Zigbee, Z-Wave, Google Coral devices, or other physical hardware.

The important part is that the hardware can be detected on the target machine instead of relying only on a manually configured label. This makes the placement information closer to the actual state of the infrastructure.

Swarm and Compose

This also means that not every workload has to use the same deployment model.

Services that can run on any suitable node can still use Docker Swarm, while services that need direct access to physical hardware can use Docker Compose on a selected node.

The resulting cluster can therefore contain both kinds of workloads:

                         Ṣeto
                           |
              +------------+------------+
              |                         |
            Swarm                    Compose
              |                         |
       general services          hardware workloads
              |                         |
        +-----+-----+             +-----+-----+
        |           |             |           |
      node-1      node-2        node-3      node-4
                                GPU         Zigbee

I prefer this approach to trying to force every workload into the same orchestration model. Docker already provides both mechanisms, so Ṣeto can use the one that fits the workload.

Persistent storage

The next problem is persistent storage.

A local bind mount is simple:

services:
  app:
    volumes:
      - ./data:/app/data

This works as long as the application stays on the same machine. Once the application can move between nodes, the data needs to be available from those nodes as well.

For small infrastructures, NFS is often enough. Ṣeto therefore treats NFS as another infrastructure primitive and allows volumes to be declared directly in the application configuration:

services:
  app:
    image: my-app
    volumes-nfs:
      - ./data:/app/data

Ṣeto takes care of configuring the NFS server and clients and creating the Docker volumes required by the deployment.

The storage model is intentionally simple. There is a storage node and the other machines consume the exported directories:

                    NFS server
                        |
             +----------+----------+
             |          |          |
             v          v          v
           node-1     node-2     node-3

Ṣeto also supports storage replicas, but it does not try to turn NFS into a distributed filesystem such as Ceph. For the kind of small clusters I use, keeping this part simple is often more useful than introducing another complex storage platform.

Shared NFS volumes

Some applications need several services to access the same data. Ṣeto supports this through named NFS volumes:

services:
  app:
    volumes-nfs:
      - "@shared-data:/app/data:rw"

  worker:
    volumes-nfs:
      - "@shared-data:/app/data:ro"

The @ prefix identifies a shared volume. The same volume can then be mounted by multiple services with different access modes.

This is useful for applications where one service produces data and another service consumes it.

Immutable data

Not everything needs to live on NFS, though.

Some directories contain application assets that do not change while the service is running. Static web content is a good example.

For this use case, Ṣeto supports volumes-image:

services:
  web:
    image: nginx
    volumes-image:
      - ./static:/usr/share/nginx/html

The contents can then be included in the application image rather than being mounted from a shared filesystem.

This gives a simple distinction between mutable application state, which belongs in persistent storage, and immutable application data, which can be packaged with the image.

Multiple architectures

The machines in a small cluster may also use different CPU architectures. For example, an infrastructure can contain both traditional amd64 servers and ARM machines.

Docker already supports multi-platform images through manifests. Ṣeto can mirror these images to a private registry using Docker Buildx.

The idea is to keep the registry close to the infrastructure while retaining the different platform variants:

             Public Registry
                    |
                    v
             multi-platform
                  image
                    |
                    v
             Private Registry
               /    |    \
           amd64  arm64  arm/v7

The appropriate image can then be pulled by each node according to its architecture.

Traefik

The hybrid Swarm and Compose model also creates a service discovery problem.

Traefik can discover services from Docker Swarm, but a Compose application running on another machine is not part of the same Docker API.

Ṣeto provides a small provider that exposes the information from remote Compose deployments to Traefik.

The service can still use normal Traefik labels:

          remote Compose
                |
              labels
                |
                v
          Ṣeto provider
                |
                | HTTP
                v
             Traefik

This allows services running through Swarm and services running directly through Compose to use the same reverse proxy.

A Compose configuration with Ṣeto

The result is still recognisable as a Compose file. Ṣeto adds a few properties for the parts that Compose does not normally describe:

x-mode: compose

x-placement:
  - host.device.gpu == true

services:
  app:
    image: my-app
    volumes-nfs:
      - "@data:/app/data:rw"

The application configuration describes both the container and some of the infrastructure requirements around it.

Ṣeto resolves this configuration and performs the required operations on the target machines.

This is probably the simplest way to describe what the project does: it adds a small declarative layer around Docker.

Why Ṣeto?

I did not want to build another container runtime or another Kubernetes-like platform. Docker already provides the container runtime, Docker Compose provides a simple way to describe applications, Docker Swarm provides scheduling, NFS provides shared storage, and Traefik provides reverse proxying.

All of these tools already do their respective jobs well. The problem is what happens when they have to work together on a small infrastructure where machines are not identical. Some workloads need to run on a particular node because they depend on physical hardware, while others can be scheduled anywhere in the Swarm cluster. Some data needs to be persistent and shared, while other data is better packaged directly into an image.

Ṣeto is the small layer I use to connect these pieces. It keeps Docker and Compose at the centre of the system while adding the missing parts around placement, storage and deployment.

A small tool for a real infrastructure

Ṣeto started from these practical problems rather than from a desire to build a new orchestration platform.

I wanted to keep using Docker, Compose, Swarm, NFS and Traefik while being able to describe the differences between the machines in my infrastructure.

The result is a small tool that sits between the application configuration and the machines that run it. It handles the parts that become repetitive when deploying the same infrastructure across several heterogeneous nodes, while leaving the underlying Docker tools visible and usable.

The source code is available on GitLab.