Technical Overview

AI inference cost is exploding.

In the cloud the cost shows up in GPU bills, power, and cooling. On-prem it shows up in utilization, hardware planning, and the cost of scaling out. At the edge it shows up as battery drain, latency limits, and tighter device memory.

Modern accelerators are extremely fast at arithmetic. However, many workloads still struggle with cost, latency, and energy. The bottleneck is often not compute. It is moving data fast enough to keep those compute engines busy.

Every inference step pulls model data through memory, layer after layer, token after token. That traffic adds up as models grow.

Modern compute is already incredibly fast. The hard part is feeding it data fast enough.

This is the memory wall problem.

On typical accelerators, moving data from DRAM or HBM costs orders of magnitude more energy than movement in registers or on-chip SRAM. Depending on the memory hierarchy and hardware generation, an off-chip access can land in the hundreds to thousands of picojoules, while register operations are often fractions of a picojoule.

In memory-bound workloads, arithmetic units sit idle while waiting for those transfers.

Lossless compression

For many teams, approximation is not just a technical tradeoff. It is a risk. Finance, healthcare, defense, and regulated enterprise workloads often need efficiency without changing model behavior.

What if the model did not need to change? What if efficiency came from how the trained model is represented and executed, not from altering its weights?

Trained weights are not random. Model training and optimization leave structure in the coefficients. Lossless compression can exploit that structure, move fewer bytes, and expand back to the exact original checkpoint bit for bit. The hard part is doing it without materializing a full dense model in memory again, in a representation that decodes efficiently on the hardware for real-time inference.

ISIRO Runtime is built for that problem on the production inference path. It preserves bit-exact weights with no approximation, retraining, or model changes. Model accuracy stays at 100%.

A smaller weight footprint leaves more GPU memory for KV cache. The KV cache is further compressed (available for pilots) for additional savings. Together that is more capacity on the hardware you already run. BF16 is in general release. FP8 and FP4 are available for pilots.

Where ISIRO Runtime sits in your stack

At Isiro, we offer ISIRO Runtime, an AI inference efficiency layer powered by TIC. Models are compiled once from formats such as .onnx, .safetensors, and .pt into a compact .tic execution-native representation. ISIRO Runtime serves from that compressed state during inference, reducing memory traffic and associated cost and energy while preserving model accuracy.

ISIRO Runtime sits between models and existing inference stack as an efficiency layer. It serves the .tic file with the existing inference frameworks as targets.

During inference

During inference, weights stay compact in memory instead of being materialized as a full dense tensor. Packed bytes flow toward on-chip SRAM and registers. Fused decode runs right next to the matmul units that consume them.

ISIRO Runtime handles the on-device path for the .tic file such as load-time verification, compressed residency, fused decode, and kernel execution on the hot path. Adapters connect ISIRO Runtime to the inference frameworks you already use as orchestration targets, preserving the same request handling, batching, and scheduling.

Representative results

.tic footprint

On evaluated BF16 models, compiled .tic weights are about 29% smaller than the raw checkpoint, with bit-exact weights. That figure is the weight footprint only. Some models and their savings are on the Models page.

Serve performance

Results vary by GPU and by model. Measured baseline vs .tic serve A/B numbers, methodology, and how to reproduce them are in the ISIRO repository.

---