add inference tutorial

| [Bash Basics](/tutorials/bash/) | 30 min | Navigate the filesystem, manage files, write scripts |

| [Slurm Basics](/tutorials/slurm/) | 45 min | Submit jobs, request GPUs, monitor your work |

| [Apptainer](/tutorials/apptainer/) | 45 min | Package your environment in containers |

| [Running LLMs](/tutorials/running-llms/) | 20 min | Run LLM inference on DAIC with Ollama and Slurm |

| [Vim](/tutorials/vim/) | 30 min | Edit files efficiently on the cluster |

## Which tutorial do I need?

**My code needs specific packages/versions**

→ Read [Apptainer](/tutorials/apptainer/) to containerize your environment

**I want to run an LLM on DAIC**

→ Read [Slurm Basics](/tutorials/slurm/) and [Apptainer](/tutorials/apptainer/) first, then follow [Running LLMs](/tutorials/running-llms/).

**I need to edit files on the cluster**

→ Learn [Vim](/tutorials/vim/) for efficient editing over SSH

6. Run parameter sweeps with job arrays

7. Package complex environments in containers

8. Edit files directly on the cluster

9. Run LLM inference on a DAIC GPU node with Ollama

## Getting help

---

title: "Tutorial: Running LLMs on DAIC"

weight: 4

description: "Guide to inference on DAIC with Ollama models."

---

This guide shows how to run inference with LLMs on DAIC using [Ollama](https://ollama.com/). It uses the [REIT LLM Serving Template](https://gitlab.ewi.tudelft.nl/reit/reit-llm-serving-template), which submits the Slurm jobs needed to start an Ollama server on a GPU node and run a sample inference request.

The template starts two Slurm jobs for one inference workflow:

1. A **server job** that runs Ollama on a GPU node.

2. A **client job** that waits for the server, pulls a model, and sends a sample inference request.

{{% alert title="Note" color="info" %}}

[NVIDIA Dynamo](https://docs.nvidia.com/dynamo/) support also exists in the template repository for advanced REIT/TULIP evaluation work. It is not the recommended starting point for normal DAIC users. This tutorial focuses on the stable Ollama workflow.

{{% /alert %}}

## 1. Clone the Template Repository

Clone the template repository wherever you normally keep code. The cloned repository can live in your home directory, a Git workspace, or project storage.

```bash

git clone https://gitlab.ewi.tudelft.nl/reit/reit-llm-serving-template.git

cd reit-llm-serving-template

```

## 2. Run the Standard Batch Workflow

For most users, the easiest path is the launcher script:

```bash

bash start-serve-client.sh \

  --backend ollama \

  --project </path/to/your/project/in/umbrella/or/bulk/storage>

```

This starts the Ollama server first, then submits the client job with a Slurm dependency. The client job waits until Ollama is reachable before running the sample request.

The runtime directory passed with `--project` should be on umbrella or bulk storage. That directory stores generated containers, model files, and host/port files, so it can become too large for your home directory. It does not need to be the same directory as the cloned repository.

If you run the launcher from outside the cloned repository, pass `--template` so the Slurm jobs can find the template files:

```bash

bash /path/to/reit-llm-serving-template/start-serve-client.sh \

  --backend ollama \

  --project </path/to/your/project/in/umbrella/or/bulk/storage> \

  --template /path/to/reit-llm-serving-template

```

To use a different Ollama model, set `MODEL_NAME` before the launcher command:

```bash

MODEL_NAME=llama3.1:8b \

  bash start-serve-client.sh \

    --backend ollama \

    --project </path/to/your/project/in/umbrella/or/bulk/storage>

```

The default model in the template is `qwen3.5:2b`. You can use any model tag available in the [Ollama library](https://ollama.com/library).

{{% alert title="Tip" color="success" %}}

- Use a small model for the first test. Larger models may need more GPU memory, more job time, and more disk space for model downloads.

- The sample `test.py` disables reasoning by default for supported reasoning models, so the first test behaves like a normal short inference request.

{{% /alert %}}

## 3. Check Job Progress

The launcher prints the submitted server and client job IDs. You can monitor them with:

```bash

squeue -j <server-job-id>,<client-job-id>

```

The Slurm output files are written in the directory where you submit the launcher command:

```bash

cat log-ollama-server-<server-job-id>.out

cat log-ollama-client-<client-job-id>.out

```

The client log should show that it waited for the server, pulled the configured model, ran `test.py`, and printed a model response.

## 4. What the Template Does

The launcher submits scripts from the Ollama backend directory:

```text

start-serve-client.sh

  ├── backends/ollama/server.sbatch

  └── backends/ollama/client.sbatch

```

The server job:

- allocates a GPU node through Slurm

- builds or reuses an Ollama Apptainer image

- starts `ollama serve` on a random high port

- writes connection details under `${PROJECT_DIR}/ollama/`

- cleans up `host.txt` and `port.txt` when Ollama exits

The client job:

- reads `${PROJECT_DIR}/ollama/host.txt`

- reads `${PROJECT_DIR}/ollama/port.txt`

- waits until the Ollama server is reachable

- pulls `MODEL_NAME`

- runs the sample OpenAI-compatible client in `test.py`

Using the launcher also keeps the model cache under `${PROJECT_DIR}/ollama/models/`, prevents accidental serving from a login node, builds missing containers automatically, and cleans up the discovery files when the server exits.

## 5. Common Customizations

Most users only need to change a few settings.

| Setting | Where | Purpose |

| ------- | ----- | ------- |

| `MODEL_NAME` | environment or `backends/ollama/client.sbatch` | Model to pull and test. |

| `PROJECT_DIR` | launcher `--project` argument | Runtime directory for containers, model cache, and host/port files. Use umbrella or bulk storage. |

| `TEMPLATE_ROOT_DIR` | launcher `--template` argument | Repository directory containing `backends/`, `test.py`, and `client-container.def`. Usually detected automatically. |

| Slurm time/memory/GPU settings | `backends/ollama/server.sbatch` and `backends/ollama/client.sbatch` | Resource requests for your workload. |

| Sample request logic | `test.py` | Prompt, request parameters, or custom client code. |

For example, to test another model:

```bash

MODEL_NAME=llama3.1:8b \

  bash start-serve-client.sh \

    --backend ollama \

    --project </path/to/your/project/in/umbrella/or/bulk/storage>

```

## 6. What You Should Normally Not Edit

These values are usually set by the launcher or by the backend scripts:

| Setting or file | Why |

| --------------- | --- |

| `BACKEND_DIR` | The launcher sets this based on `--backend ollama`. |

| `${PROJECT_DIR}/ollama/host.txt` | Written by the server job and read by the client job. |

| `${PROJECT_DIR}/ollama/port.txt` | Written by the server job and read by the client job. |

| `backends/ollama/ollama-function.sh` | Contains the helper functions for image setup, serving, and client forwarding. |

Only edit backend helper functions if you understand the server/client workflow and need to change the template behavior.

## 7. Advanced: Interactive Debugging Workflow

For debugging, you can start an interactive GPU allocation and run the Ollama server manually.

```bash

srun --cpus-per-task=2 --mem=8G --time=00:30:00 --gres=gpu:1 --pty bash -il

```

Inside the allocation:

```bash

export PROJECT_DIR=</path/to/your/project/in/umbrella/or/bulk/storage>

export TEMPLATE_ROOT_DIR=/path/to/reit-llm-serving-template

source "${TEMPLATE_ROOT_DIR}/backends/ollama/ollama-function.sh"

ollama serve

```

Keep that terminal open. In a second terminal, connect to DAIC and run:

```bash

export PROJECT_DIR=</path/to/your/project/in/umbrella/or/bulk/storage>

export TEMPLATE_ROOT_DIR=/path/to/reit-llm-serving-template

source "${TEMPLATE_ROOT_DIR}/backends/ollama/ollama-function.sh"

ollama run qwen3.5:2b

```

You can check the Ollama endpoint directly with:

```bash

curl "http://$(cat "${PROJECT_DIR}/ollama/host.txt"):$(cat "${PROJECT_DIR}/ollama/port.txt")"

```

Stop the server with `Ctrl-C` in the first terminal. The helper removes `host.txt` and `port.txt` when the server exits.

## 8. Troubleshooting

| Symptom | What to check |

| ------- | ------------- |

| `sbatch: command not found` | You are not on a Slurm login node, or Slurm is not available in the current environment. |

| Server job starts but client cannot connect | Check `squeue -j <server-job-id>` and inspect `log-ollama-server-<job-id>.out`. |

| `host.txt` or `port.txt` is missing | The server may not have started yet, may have exited, or may have cleaned up after exit. |

| Client connects to an old or wrong Ollama endpoint | Check whether `${PROJECT_DIR}/ollama/host.txt` and `${PROJECT_DIR}/ollama/port.txt` are stale. Remove them only after confirming no related Ollama server job is running. |

| Port already in use | Restart the server job. The template normally chooses a random high port, so repeated conflicts should be rare. |

| Model download or cache issues | Check available storage under `PROJECT_DIR`; Ollama model files are stored under `${PROJECT_DIR}/ollama/models/`. |

| GPU memory errors | Request a GPU with more memory, reduce model size, or use a quantized/smaller model. |

| Reasoning models appear slow | Some reasoning models spend time generating reasoning tokens. The sample `test.py` disables reasoning by default where supported; check the client code before benchmarking speed. |

| Client cannot find `test.py` | Ensure `TEMPLATE_ROOT_DIR` points to the repository root. When using `start-serve-client.sh`, this is set automatically. |

| Job exits before client connects | Increase `#SBATCH --time` in `backends/ollama/server.sbatch` and check the server log for startup failures. |

| Apptainer build fails with error 137 | The build was likely killed for using too much memory. Increase the server job memory request. |

## 9. When to Contact REIT

Contact REIT if:

- the template scripts fail after you have verified the Slurm job is running

- the Ollama server starts but the generated host/port files are wrong

- the client can reach the server but `test.py` consistently fails

- you need help adapting the workflow for a larger evaluation or a production-facing service

For ordinary model choice, prompt design, or GPU memory sizing, first try a smaller model and inspect the server/client logs.

## Acknowledgment

This tutorial was inspired by the Stanford [`ollama_helper`](https://github.com/gsbdarc/ollama_helper) project. The DAIC template adapts similar ideas to TU Delft's Slurm environment.