Triton AI Inferenceby Inseyab connecting…
Range

Pipelines

load / unload here — each drives its underlying model(s)
–
Pipeline State VRAM Inferences Capacity Req/s Avg ms Feeds Action VRAM MB Instances Batch Max feeds Info
waiting for data…

Hardware live GPU telemetry via NVML

Pipelines Loaded
– / –
Requests served
– req/s now
across every loaded pipeline
Disk — physical volumes
–
▶

Add a model to the pool

HuggingFace · git · YOLO .pt · ONNX URL

Model from HuggingFace or git

Paste a repo id — the server reads its config.json and works out what kind of model it is before downloading anything. Chat/vision, embeddings, speech-to-text and text-to-speech all come from here.

Type a repo id to inspect it.

YOLO model (.pt upload)

Upload an ultralytics .pt; it is exported to ONNX on the server (streamed, large files OK).

Export at the size the model was trained/validated at. A mismatch gives plausible boxes with near-zero confidence, not an error.
Serving profile — this pipeline only; leave empty for the defaults
Canvas is the shape every frame is served at. Match the camera's aspect and nothing is spent on padding: a 704×576 feed halves to exactly 288×352. TensorRT is tuned for that shape and nothing else. FP16 converts the graph once, keeping inputs and outputs FP32. Camera frame and streams are what the VRAM dry run measures with — certify the cap at the load it will really see. There is no decoding choice here: every pipeline takes both on every request — Triton-side decoding (send the JPEG/PNG) and origin-based decoding (send the frame you already decoded). The request picks; the pipeline's Integrate dialog shows both.

RF-DETR model (.pth upload)

Upload a trained Roboflow RF-DETR checkpoint (checkpoint_best_*.pth); it is exported to ONNX on the server. Detection and segmentation, any size.

Serving profile — this pipeline only; leave empty for the defaults
The network size, task and class names come from the checkpoint. Frames are stretched to that square size — the way RF-DETR was trained — not letterboxed. Served in FP32: converting RF-DETR to FP16 measurably changed which objects it found. TensorRT, Triton-side or origin-based decoding and a Script are chosen on the row, as for YOLO.

ONNX model by URL

Direct link to a .onnx file (InsightFace, custom detectors, …).

Model bundle — several ONNX models as one pipeline

Choose the folder that holds one sub-folder per model, each with its config.pbtxt and 1/model.onnx (or upload it as one .tar.gz/.zip). The models run in order inside Triton and a client makes one call.

Each model keeps its own config (inputs, outputs, batch, instances). PaddleOCR text models (detection → angle → recognition) are recognised from their graphs and wired exactly as RapidOCR runs them; any other set runs model 1 on the frame and passes each output to the next model — put a pipeline.json in the folder to set the order or the pre-processing. The bundle is checked and run once on the server's CPU before it is added. Decoding can be changed on the row later.
Pick the folder (left) or an archive of it (right).
⚠GPU memory pressure