Serve a .tic

Port 8000 by default. Model output is unchanged: no quantization, no approximation.

Usage

isiro serve <bundle> --target vllm [--host HOST] [--port PORT] [--max-model-len N]

<bundle>: a compiled -TIC folder or path to model.tic inside it. Config and tokenizer files sit beside the .tic. The same path form works for isiro verify and isiro info.

--target: required; vllm in v0.1.0.

--host / --port / --max-model-len override serve.yaml when passed. Prometheus /metrics is on the same port.

Serve settings

The TIC bundle includes serve.yaml. You can edit it for defaults such as host, port, max_model_len, gpu_memory_utilization, max_num_seqs, prefix cache, and the target image tag under targets.vllm.

CLI --host, --port, and --max-model-len override the file when you pass them. For max_num_seqs, gpu_memory_utilization, and prefix cache, copy serve.yaml next to the .tic and edit that copy for the GPU. The shipped defaults (8192 context, 0.90 util, 32 seqs, prefix cache off) are a start-safe load, not a long-context or single-request latency tune. Do not republish the Hub bundle just to change those knobs. For tested image tags, run isiro status --compat. For flag details, run isiro serve --help.

The bundle must already be on the machine.

An example TIC bundle can be downloaded from Hugging Face e.g.

# Install Hugging Face Hub
python -m venv .venv
source .venv/bin/activate
pip install --upgrade huggingface_hub
 
# Download
hf download isiroai/Qwen2.5-7B-Instruct-TIC --local-dir Qwen2.5-7B-Instruct-TIC

(Linux / macOS sample. More install options: Hugging Face installation)

isiro serve Qwen2.5-7B-Instruct-TIC --target vllm

Stop

docker stop isiro-serve-8000

Use the port you passed (default 8000). Container name is isiro-serve-<port>.

Ctrl+C in the serve terminal may leave the container holding the GPU or port; if that happens, run the same docker stop command.

NVIDIA GPU supported today.

Run isiro serve --help for host, port, metrics, and other options.

See API for OpenAI-compatible endpoints. To chat from a browser UI, see Chat with your model.