Struct CudaGraphModel 

pub struct CudaGraphModel { /* private fields */ }

A CUDA graph compiled from a LoadedModel for fixed-batch inference.

Created via LoadedModel::capture_graph. All device buffers are pre-allocated; the kernel sequence is captured once and replayed on each [run] call with a single cuGraphLaunch + cuStreamSynchronize.

Implementations

impl CudaGraphModel

pub fn output_shapes(&self) -> &[Vec<usize>]

Concrete shapes of the output tensors, in the order of output_node_indices.

pub fn run_timed(&self, inputs: &[&[f32]]) -> Result<(Vec<Vec<f32>>, f32)>

Like [run] but also returns GPU execution time in milliseconds.

The GPU time is measured with CUDA events bracketing only cuGraphLaunch (pure kernel execution, excluding host↔device copies). The returned f32 is milliseconds of GPU time for the whole batch.

pub fn run(&self, inputs: &[&[f32]]) -> Result<Vec<Vec<f32>>>

Copy f32 inputs to device, replay the CUDA graph, copy f32 outputs to host.

Returns one Vec<f32> per requested output node (same order as output_node_indices passed to LoadedModel::capture_graph).

pub fn input_slice_mut(&mut self, i: usize) -> &mut [f32]

Mutable slice into the i-th pinned (page-locked) input staging buffer.

Write your input data here before calling [run_inplace] / [run_timed_inplace] to avoid the intermediate CPU copy that [run] / [run_timed] perform when given a pageable &[f32].

Safety

The slice is valid until this CudaGraphModel is dropped.

pub fn output_slice(&self, i: usize) -> &[f32]

Immutable slice into the i-th pinned (page-locked) output staging buffer. Valid after [run_inplace] / [run_timed_inplace] returns.

pub fn run_inplace(&self) -> Result<()>

Copy pinned inputs → device, launch graph, copy device → pinned outputs.

Callers must fill [input_slice_mut] before calling and read [output_slice] afterwards. No heap allocations are performed.

pub fn run_timed_inplace(&self) -> Result<f32>

Like [run_inplace] but also returns GPU execution time in milliseconds.

Trait Implementations

impl Drop for CudaGraphModel

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for CudaGraphModel