How to pool your computers into one compute cluster

Turn the workstations, laptops and GPU boxes you already own into one job queue for simulations, renders and data processing — without a sysadmin or cloud bill.

Most small labs, studios and home offices have more computing power than they use. One machine runs a long simulation overnight while two others sit idle. Pooling them into a single job queue fixes that: you submit work once, and it runs wherever there is room.

What a small cluster needs

• A coordinator. One computer keeps the queue and decides where each job runs.
• Node agents. Every other computer runs a small agent that reports its free CPU cores, memory and GPUs, runs the jobs it is given and streams their output back.
• A shared way to describe jobs. A job is just a command — python simulate.py --n 4000 — plus how many cores, how much memory and whether it needs a GPU.
• Files every node can read. Put input data on a network share or copy it to each machine, so a job can run on any node.

Step by step

1. Pick the computer that is on most of the time as the coordinator. It does very little work itself.
2. Give the machines fixed addresses on your local network, or note their host names.
3. Install the cluster software on every machine and join each node to the coordinator with its address and a shared token.
4. Allow the cluster's ports through the firewall on the coordinator — local network only.
5. Make each job a command that runs from a terminal without asking questions. Write results to files, not to the screen.
6. Submit a parameter sweep as an array of jobs (for example 240 render frames or 50 simulation sizes) instead of one giant script.

Scheduling: why "first come, first served" wastes time

A simple queue starts jobs in the order they arrive on the first machine that fits. That is fine for a handful of jobs, but it causes three problems as the queue grows: long jobs block short ones, jobs that need a lot of memory crash machines that were already busy, and long jobs land on the slowest laptop while the fast workstation finishes early and sits idle.

A better scheduler predicts how long each job will take and how much memory it will need. Scientific and rendering commands are very regular: if --n 2000 took 2.5 minutes and --n 4000 took 10, the runtime grows with the square of n, and --n 8000 will take about 40. With those estimates the scheduler can run short jobs first, reserve the right amount of memory and send long jobs to the fastest machine.

Security on a home or lab network

• Anyone who can submit jobs can run commands on every node. Treat the cluster token like a password.
• Keep the cluster on a trusted local network or a VPN; do not expose its ports to the internet.
• Run node agents as a normal user, not as administrator or root.

Trying it

Andromeda Cluster does all of the above for Windows and Linux machines. The free Community edition pools up to four computers. Pro adds an AI scheduler that learns your commands' runtimes and memory use from the jobs you have already run, and places each job on the node that will finish it first.