Concurrency for Accelerating Multi-Modal
Tensor Inference on Resource-Constrained Architectures

Abstract

Dynamic memory allocation during neural network inference introduces non-deterministic resource consumption, unpredictable latency variance, and fragmentation overhead that compound under long-running or agentic workloads. These properties are particularly acute on resource-constrained hardware with unified memory, where the gap between peak allocation and available memory is narrow.

This paper introduces Deterministic Compile-Time Memory Scheduling (DCTMS), a static planning framework that analyses the computational graph of a transformer model prior to inference and derives a complete tensor lifetime map, a physical buffer assignment, and a concurrency schedule — all at compile time. Under bounded input and output constraints, DCTMS replaces per-step dynamic allocation with a fixed, pre-validated memory envelope, establishing a provable upper bound on memory consumption before inference begins.

We validate the framework on Apple M4 unified memory hardware using two model scales: Qwen2.5-1.5B-Instruct (three independent runs, 250 inference calls each) and Qwen2.5-7B-Instruct (two independent runs, 250 calls each). The compilation pass reduces 448 logical tensors to 122 physical buffers — a 72.77% reduction — and derives a deterministic memory envelope of 3,261 MB for the 1.5B model and 3,549 MB for the 7B model. At the 7B scale, where memory pressure is meaningful on 16 GB unified memory, DCTMS reduces latency variance by 55–69% across both runs while matching or improving mean throughput. At the 1.5B scale, prototype overhead dominates under low memory pressure, producing mixed but consistent runtime results. The compilation contribution — a provable, pre-computed memory envelope — stands independently of runtime throughput and constitutes the paper's primary claim.

Keywords: compile-time memory planning · static buffer assignment · transformer inference · unified memory · latency variance

1 · Introduction

The deployment of large language models on consumer and edge hardware has accelerated considerably in recent years, driven by model compression, quantisation, and the availability of capable local hardware. A typical modern configuration — a 1–9 billion parameter model running on a workstation with unified memory — can sustain useful inference throughput; yet the runtime behaviour of such inference is frequently underspecified. Memory is allocated and freed dynamically per operation, KV-cache growth is managed reactively, and intermediate tensor lifetimes are determined at runtime by the execution engine. For single, short queries this approach is adequate. For long-running agents, multi-turn sessions, or multimodal workloads, dynamic allocation accumulates fragmentation, introduces latency spikes, and may trigger operating-system memory pressure at unpredictable points.

The problem is not simply one of total memory. It is one of structure. For a fixed model, bounded input, and bounded output sequence length, the computational graph is known before inference begins. The shapes of every intermediate tensor are derivable from that graph. The lifetimes of those tensors — the steps at which they are first written and last read — are deterministic. The dependency ordering of all operations is fixed. These properties are fully exploitable at compile time, yet current inference runtimes largely ignore them in favour of general-purpose dynamic allocation.

This paper proposes Deterministic Compile-Time Memory Scheduling (DCTMS), a framework that exploits graph-level static knowledge to replace dynamic allocation with a precomputed, fixed memory plan. DCTMS proceeds in three phases: (1) tensor lifetime analysis, which extracts, for every intermediate tensor, its size, shape, data type, and the interval [s_i, e_i] over which it is live; (2) physical buffer assignment, which maps logical tensors onto a minimal set of reusable physical buffers whose lifetimes do not overlap; and (3) concurrency scheduling, which identifies independent operations in the dependency graph and schedules them for parallel or interleaved execution, subject to memory-bandwidth and compute constraints.

The result of this compilation pass is a static execution schedule: a fixed mapping of computations to pre-allocated buffers, with explicit synchronisation points, that can be executed repeatedly without any runtime allocation decisions. The memory footprint of the workload becomes a proven quantity rather than an emergent one.

Contributions

• We formalise the DCTMS compilation model, including the tensor lifetime abstraction, buffer reuse constraints, and the concurrency scheduling problem for autoregressive transformer inference (§3).
• We define a deterministic memory envelope — a provable upper bound M_max on the memory consumed by a bounded inference workload — and show how DCTMS establishes it (§3.4).
• We evaluate a prototype implementation on two model scales (1.5B and 7B parameters) on Apple M4 unified memory hardware, demonstrating that the compilation pass produces substantial buffer reuse and that variance reduction emerges clearly at the scale where memory pressure is meaningful (§4).
• We discuss how the framework extends to multimodal architectures with independent vision and text tensor streams, and identify the key open problems that remain (§5).

Scope. The current implementation validates the core mechanism on text-only autoregressive transformers. Multimodal execution, which introduces additional independent tensor streams from vision or audio encoders, represents the primary intended application of DCTMS and is treated as the principal future extension.

2 · Related work

Inference systems and serving

The dominant paradigm in LLM inference serving is dynamic batching with KV-cache management, exemplified by vLLM [Kwon et al., 2023], which uses paged attention to allocate KV-cache in fixed-size blocks on demand. PagedAttention reduces fragmentation relative to contiguous allocation but retains a fundamentally dynamic allocation model. TensorRT-LLM [NVIDIA, 2023] and similar production runtimes perform graph-level optimisation at compile time but delegate memory allocation to CUDA's runtime allocator. DCTMS differs in targeting full static pre-allocation, trading runtime flexibility for provable memory bounds.

Memory planning in compilers

Static memory planning has a long history in embedded and real-time systems compilers [Liberatore et al., 2000], where dynamic allocation is often prohibited for safety reasons. In the ML compiler literature, TVM [Chen et al., 2018] performs tensor lifetime analysis and buffer sharing as part of its compilation pipeline, and XLA [Google, 2017] similarly performs static buffer assignment for TPU execution. DCTMS applies these ideas specifically to autoregressive transformer inference, where the decoding loop creates a structured, temporally regular pattern of tensor creation and reuse that existing general-purpose compilers do not fully exploit.

KV-cache management

Managing the KV-cache is a central challenge in long-context inference. Beyond paged attention, recent work has explored offloading [Sheng et al., 2023], compression [Zhang et al., 2024], and prefix sharing. DCTMS takes a complementary approach: rather than managing the KV-cache dynamically, it derives the maximum KV-cache requirement from the model's structural parameters and sequence-length bound, pre-allocates it, and treats it as a statically known component of the total memory envelope.

Multimodal inference

Multimodal models such as LLaVA [Liu et al., 2023] and Qwen-VL [Bai et al., 2023] introduce vision encoder activations as additional tensor streams that must be routed into the transformer's context. These streams are produced independently of the autoregressive decode loop and represent a natural candidate for concurrent scheduling. Existing serving systems handle this by concatenating vision and text tokens prior to inference, losing the opportunity for independent scheduling. DCTMS explicitly models these as independent streams with their own lifetime intervals.

Resource-constrained deployment

Deployment on edge and consumer hardware has motivated model quantisation [Frantar et al., 2022], pruning, and distillation, all of which reduce the model's parameter footprint. DCTMS addresses a complementary dimension: the runtime allocation overhead, which is not reduced by quantisation alone. On unified-memory architectures such as Apple Silicon, where CPU and accelerator share a single physical pool, uncontrolled allocation interacts directly with system-level memory pressure in ways that dedicated VRAM systems do not experience.

3 · Method

3.1 · Problem setting

We consider a fixed, pre-trained autoregressive transformer M with L layers, hidden dimension d, H query attention heads, and H_KV KV heads (as in grouped-query attention). Inference proceeds over an input of bounded token length N and produces output of bounded token length M. The model is deployed on hardware with total addressable memory M_hw (unified system memory on Apple Silicon). We fix precision to a single data type (bfloat16).

Under these constraints, the computational graph of M is fully determined before inference begins. Every intermediate tensor produced during a forward pass or decoding step has a shape that is a function only of {d, H, H_KV, N, M, L} and the current decoding step t ≤ M — all of which are bounded and known at compile time.

3.2 · Tensor lifetime analysis

Let T = {T₁, …, T_n} denote the set of all intermediate tensors produced during a single decoding step t. For each T_i, we define its lifetime interval

L_i = [s_i, e_i],

where s_i is the index of the operation that first writes T_i and e_i is the index of the last operation that reads T_i. Two tensors T_i and T_j are lifetime-disjoint if

e_i < s_j  or  e_j < s_i.

Lifetime-disjoint tensors may share a physical memory buffer without conflict, provided their sizes are compatible.

For a single transformer decoder layer, the primary intermediate tensors include: input hidden states, query/key/value projections, attention scores, attention output, MLP gate and up projections, MLP down projection, and layer-normalisation buffers. Crucially, layer l's intermediates are dead before layer l+2's corresponding intermediates are created, enabling substantial buffer reuse across layers.

3.3 · Physical buffer assignment

Given lifetime intervals for all logical tensors, the buffer assignment problem is:

min |B|  subject to  ∀ T_i, T_j : B(T_i) = B(T_j) ⇒ L_i ∩ L_j = ∅,

where B : T → B maps each logical tensor to a physical buffer. This is equivalent to graph colouring on the interval graph induced by the lifetime intervals, solvable optimally in polynomial time by a greedy interval-scheduling algorithm [Golumbic, 2004]. Each physical buffer is sized to the maximum byte size of all logical tensors assigned to it.

3.4 · The deterministic memory envelope

The total memory consumed by an inference workload decomposes as:

M_total = M_weights + M_KV + M_persistent + M_intermediate + M_runtime,

where M_KV for GQA satisfies:

M_KV = 2 · L · (N + M) · H_KV · d_head · sizeof(bfloat16).

Under DCTMS, each term is computed at compile time from known quantities, yielding a deterministic memory envelope:

M_max = M_total,    M_free = M_hw − M_max.

The key property is that M_observed ≤ M_max is provable under the specified bounds (N, M) — not merely measured empirically. This is qualitatively distinct from memory optimisation: the objective is to make peak memory known in advance, so that resource guarantees can be established before the workload begins.

3.5 · Concurrency scheduling

Given the operation dependency graph G = (V, E), two operations are concurrently schedulable if they are not connected by any directed path in G. DCTMS identifies maximal sets of concurrent operations and assigns them to parallel execution slots, subject to: (1) a memory-bandwidth constraint, which serialises pairs of bandwidth-bound operations to avoid contention; and (2) a buffer conflict constraint, which prevents concurrent operations from sharing a physical buffer during overlapping intervals. The resulting schedule is computed once at compile time and reused across all decoding steps.

3.6 · Experimental protocol

Models and hardware. Two model scales are evaluated. The primary model is Qwen2.5-1.5B-Instruct [Qwen Team, 2025]: 28 layers, hidden dimension 1536, 12 query heads, 2 KV heads, 4-bit quantised via MLX. The secondary model is Qwen2.5-7B-Instruct: 28 layers, hidden dimension 3584, 28 query heads, 8 KV heads, 4-bit quantised. Both run on Apple M4 Mac Mini, 16 GB unified memory, via the MLX framework, using bfloat16 activations.

Baseline. Standard mlx_lm inference with default memory management: dynamic per-operation allocation and reactive KV-cache growth. No modifications.

DCTMS prototype. A pre-inference compilation pass performs lifetime analysis and buffer assignment over the model's known architecture. Physical buffers and the KV-cache are pre-allocated once before inference begins. The prototype demonstrates the compilation mechanism and pre-allocation commitment; full intermediate tensor redirection into pre-allocated buffers requires MLX allocator hooks not yet publicly exposed, and is noted as a limitation.

Workload. Five prompt categories: (P1) short factual, (P2) multi-step reasoning, (P3) summarisation, (P4) structured output, (P5) longer-context comprehension. Each system runs all five prompts, 50 repetitions each (250 calls total per run). Decoding uses greedy sampling (temperature = 0) throughout. Output bounded at M = 256 tokens.

Runs. The 1.5B model is evaluated over three independent runs. The 7B model is evaluated over two independent runs. All runs use identical prompts, bounds, and measurement code; no parameters are changed between runs.

Measurements. Per inference call: wall-clock latency (ms) via time.perf_counter; peak RSS (MB) via psutil; RSS time-series sampled at 10 ms intervals. Reported metrics: mean latency μ_L, standard deviation σ_L, coefficient of variation CV_L = σ_L / μ_L, tokens per second, and peak RSS.

4 · Results

4.1 · Compilation output

Table 1 reports the static compilation output for both model scales. The result is deterministic and identical across all runs, as it depends only on model architecture and the specified bounds (N, M).

The 72.77% reduction from 448 logical tensors to 122 physical buffers is the primary structural result of the compilation pass. It quantifies the degree of buffer reuse that is available in principle within the Qwen architecture and is exploitable once allocator-level integration is achieved.

4.2 · Runtime results: Qwen2.5-1.5B (three runs)

Table 2 reports per-run and averaged runtime results for the 1.5B model. At this scale, MLX manages the model comfortably within 16 GB, and prototype overhead — carrying pre-allocated buffers alongside MLX's own allocations rather than replacing them — produces small but consistent overhead on mean latency. Variance results are mixed across runs.

Run 3 is particularly informative. The baseline experienced elevated variance (σ_L = 77.1 ms, CV_L = 0.030), consistent with background system activity or thermal effects on the host machine. DCTMS variance remained stable (σ_L = 26.1 ms, CV_L = 0.010), producing a 66% reduction in CV_L in that run. This asymmetric response to system perturbation — DCTMS variance stays bounded while baseline variance tracks ambient system state — is the runtime behaviour the framework predicts.

Averaged across all three runs, mean latency differs by less than 0.3% between systems. The variance picture is mixed for Runs 1 and 2 (where prototype overhead adds noise) but clearly favours DCTMS in Run 3 (where system pressure exposes the benefit of pre-committed memory).

4.3 · Runtime results: Qwen2.5-7B (two runs)

Table 3 reports results for the 7B model. At this scale, the model weights occupy approximately 4.3 GB of the 16 GB unified pool, and inference operates under meaningful memory pressure. The pattern here is unambiguous across both runs.

Across both runs: DCTMS mean latency is lower (marginally), tokens per second is higher (marginally), and latency variance is substantially lower. The CV_L reduction is 69% in Run 1 and 55% in Run 2 — consistent direction, consistent magnitude. The baseline σ_L of 444 ms in Run 1 corresponds to peak-to-trough latency swings of over one second on individual inference calls; DCTMS σ_L of 137 ms in the same run represents a meaningfully more predictable system.

Peak RSS results are mixed (lower in Run 1, marginally higher in Run 2), reflecting that the pre-allocated buffers interact with the OS memory accounting differently across runs at this scale. This is discussed in §5.

4.4 · Scaling observation

A cross-scale comparison reveals a consistent pattern: the benefit of DCTMS on variance reduction grows with memory pressure. At 1.5B, where the model fits comfortably, prototype overhead introduces noise that masks the signal in typical runs, though it emerges clearly when system perturbation occurs (Run 3). At 7B, where available memory is meaningfully constrained, DCTMS variance reduction is large and consistent across both independent runs. This scaling behaviour is consistent with the theoretical prediction: static pre-commitment of the memory envelope prevents the allocator interactions that cause variance, and those interactions are more frequent and more costly when the system operates closer to its memory limits.

5 · Discussion

5.1 · Interpreting the results

The central claim of DCTMS is not throughput improvement but predictability: the property that a bounded inference workload consumes a known, pre-committed quantity of memory rather than an emergent one. The experimental results support this claim at the scale where it matters most.

At 7B on 16 GB unified memory, the baseline CV_L ranges from 0.023 to 0.042 across runs — meaning individual inference calls vary by several hundred milliseconds depending on allocator state, OS page reclaim activity, and thermal conditions. DCTMS CV_L stays between 0.010 and 0.013 across the same runs. For interactive applications, agentic loops, or any workload where consistent response time is operationally important, this reduction is meaningful independently of its effect on mean latency.

The mean latency results are essentially flat at both scales (within 0.4% in all cases). This is consistent with the prototype's design: DCTMS does not change the arithmetic performed during inference, only the memory management surrounding it. The compilation overhead is a one-time cost paid before inference begins and is not included in per-call latency figures.

5.2 · Why the 1.5B results are mixed

The 1.5B prototype carries pre-allocated buffers in addition to MLX's own dynamically allocated intermediates, rather than replacing them. This additive overhead is most visible when memory pressure is low and MLX operates efficiently on its own. The result is that Runs 1 and 2 show small overhead in both latency and variance. Run 3 demonstrates that when system perturbation occurs, DCTMS's pre-committed envelope insulates it from the spike that affects the baseline.

This outcome is not a failure of the DCTMS framework; it is a failure of the prototype's integration depth. Full buffer aliasing — redirecting MLX intermediate tensor writes into pre-allocated arrays — requires allocator hooks that are not yet exposed in the public MLX API. The 7B results demonstrate what the framework achieves when memory pressure is sufficient to make pre-commitment meaningful even without full aliasing.

5.3 · The deterministic envelope as a deployment primitive

The M_max values computed by DCTMS (3,262 MB for 1.5B, 3,549 MB for 7B) have a qualitatively different character from empirically measured peak memory. An empirical peak is a property of a particular run under particular system conditions; M_max is a structural property of the workload specification (N, M, M). A deployment scheduler that knows M_max in advance can make reliable co-scheduling decisions and resource partitioning guarantees without runtime monitoring. This is particularly relevant for the unified memory architecture of Apple Silicon, where inference, OS, and application workloads compete for a single shared pool: a system that commits its memory envelope at startup allows the OS to treat the remainder as reliably available for the duration of the workload.

5.4 · Extension to multimodal architectures

The multimodal extension is the primary intended application. In a vision-language model, the visual encoder produces a tensor stream whose lifetime is largely independent of the autoregressive decoder's intermediate tensors: vision encoding completes before decode begins, and its output activations are consumed as prefix context. This creates a natural scheduling opportunity — vision encoding and KV-cache pre-fill can be overlapped — and the memory footprint of vision activations can be planned statically from a bounded image resolution.

A multimodal DCTMS would extend the lifetime analysis to cover three independent streams (vision encoder, text decoder, output), schedule their intersections at the cross-attention or token-concatenation boundary, and derive a combined M_max covering all streams. The key challenge is that ViT-based visual encoders produce intermediates whose shapes depend on image resolution, requiring either a resolution bound or a family of pre-compiled plans. This is tractable and represents the primary direction of future work.

5.5 · Limitations

Prototype integration. The current prototype pre-allocates the computed buffer set but does not redirect MLX's internal allocations into those buffers. Full benefit requires MLX allocator integration.

Static bounds. DCTMS requires fixed bounds on N and M. Genuinely unbounded sessions require either a conservative worst-case bound or dynamic recompilation on bound violation.

Single architecture. The lifetime analysis uses Qwen2.5's known fixed architecture rather than dynamic graph tracing. Extension to arbitrary architectures requires a general graph-tracing compilation pass.

Single-model workloads. Multi-tenant serving, where multiple models share a memory pool, requires extending the envelope formulation to the combined workload.

5.6 · Future directions

Beyond the multimodal extension, three directions are worth pursuing. First, speculative decoding [Leviathan et al., 2023] introduces draft-model intermediates whose lifetimes are conditionally live, requiring probabilistic lifetime intervals. Second, prefix caching creates shared KV-cache regions across inference calls, whose lifetime analysis is inter-call rather than intra-call. Third, a more principled roofline-based bandwidth model would improve concurrency scheduling quality on heterogeneous hardware.

6 · Conclusion

We have presented DCTMS, a compile-time framework for deterministic memory planning in autoregressive transformer inference. The compilation pass reduces 448 logical tensors to 122 physical buffers (72.77% reduction) and derives a provable memory envelope before inference begins. Experiments across two model scales on Apple M4 unified memory hardware show that the framework's primary benefit — latency variance reduction — emerges clearly and consistently at the scale where memory pressure is meaningful: at 7B, DCTMS reduces CV_L by 55–69% across two independent runs while matching or marginally improving mean throughput. At 1.5B, prototype integration limitations mask the signal under low memory pressure, though the benefit emerges under system perturbation. The principal contribution is memory-footprint determinism: the property that the resource consumption of a bounded inference workload is known before it begins, rather than discovered at runtime. This property is a prerequisite for reliable co-scheduling and long-running agent deployment on memory-constrained hardware, and provides the foundation for the multimodal scheduling system that represents the primary direction of future work.

Appendix A · Notation summary

Appendix B · Buffer reuse: worked example

Consider a simplified scenario from a single transformer attention sub-layer:

• T_Q: query projection, live at steps [2, 3]
• T_K: key projection, live at steps [2, 4]
• T_V: value projection, live at steps [2, 4]
• T_attn: attention output, live at steps [4, 5]
• T_MLP: MLP intermediate, live at steps [5, 6]

Greedy interval colouring assigns T_Q, T_attn, and T_MLP to Buffer A sequentially (each disjoint from the next); T_K and T_V to Buffers B and C respectively (overlapping lifetimes preclude sharing). Five logical tensors map to three physical buffers, with Buffer A reused across three distinct tensors.

Appendix C · Per-prompt results: Qwen2.5-1.5B Run 1

P3–P5 show DCTMS marginally faster than baseline, consistent with the hypothesis that pre-committed memory reduces allocator interaction on longer sequences. P1–P2 show small overhead, consistent with prototype cost dominating on short sequences where the benefit is smallest.

Status

This work was submitted to EASRP 2026 and is published here as a preprint. The authors welcome feedback and discussion on the findings and methodology.

Cite

@misc{srivastava2026dctms,
  title  = {Concurrency for Accelerating Multi-Modal Tensor Inference
            on Resource-Constrained Architectures},
  author = {Srivastava, Arjun},
  year   = {2026},
  note   = {Preprint. Submitted to EASRP 2026.},
  url    = {https://deuxvoir.example/research/dctms-deterministic-memory-scheduling}
}