Edit this page

openvino.runtime.AsyncInferQueue

class openvino.runtime.AsyncInferQueue

Bases: openvino.pyopenvino.AsyncInferQueue

AsyncInferQueue with pool of asynchronous requests.

AsyncInferQueue represents helper that creates a pool of asynchronous InferRequests and provides synchronization functions to control flow of a simple pipeline.

__init__(self: openvino.pyopenvino.AsyncInferQueue, model: openvino.pyopenvino.CompiledModel, jobs: int = 0) → None

Creates AsyncInferQueue.

Parameters

  • model (openvino.runtime.CompiledModel) – Model to be used to create InferRequests in a pool.

  • jobs – Number of InferRequests objects in a pool. If 0, jobs number

will be set automatically to the optimal number. Default: 0 :type jobs: int :rtype: openvino.runtime.AsyncInferQueue

Methods

__delattr__(name, /)

Implement delattr(self, name).

__dir__()

Default dir() implementation.

__eq__(value, /)

Return self==value.

__format__(format_spec, /)

Default object formatter.

__ge__(value, /)

Return self>=value.

__getattribute__(name, /)

Return getattr(self, name).

__getitem__(i)

Gets InferRequest from the pool with given i id.

__gt__(value, /)

Return self>value.

__hash__()

Return hash(self).

__init__(self, model[, jobs])

Creates AsyncInferQueue.

__init_subclass__

This method is called when a class is subclassed.

__iter__(self)

__le__(value, /)

Return self<=value.

__len__(self)

Number of InferRequests in the pool.

__lt__(value, /)

Return self<value.

__ne__(value, /)

Return self!=value.

__new__(**kwargs)

__reduce__()

Helper for pickle.

__reduce_ex__(protocol, /)

Helper for pickle.

__repr__(self)

__setattr__(name, value, /)

Implement setattr(self, name, value).

__sizeof__()

Size of object in memory, in bytes.

__str__()

Return str(self).

__subclasshook__

Abstract classes can override this to customize issubclass().

get_idle_request_id(self)

Returns next free id of InferRequest from queue's pool.

is_ready(self)

One of 'flow control' functions.

set_callback(self, arg0)

Sets unified callback on all InferRequests from queue's pool.

start_async([inputs, userdata])

Run asynchronous inference using the next available InferRequest from the pool.

wait_all(self)

One of 'flow control' functions.

Attributes

userdata

return

List of all passed userdata. None if the data wasn't passed yet.
__class__

alias of pybind11_builtins.pybind11_type

__delattr__(name, /)

Implement delattr(self, name).

__dir__()

Default dir() implementation.

__eq__(value, /)

Return self==value.

__format__(format_spec, /)

Default object formatter.

__ge__(value, /)

Return self>=value.

__getattribute__(name, /)

Return getattr(self, name).

__getitem__(i: int) → openvino.runtime.ie_api.InferRequest

Gets InferRequest from the pool with given i id.

Parameters

i (int) – InferRequest id.

Returns

InferRequests from the pool with given id.

Return type

openvino.runtime.InferRequest

__gt__(value, /)

Return self>value.

__hash__()

Return hash(self).

__init__(self: openvino.pyopenvino.AsyncInferQueue, model: openvino.pyopenvino.CompiledModel, jobs: int = 0) → None

Creates AsyncInferQueue.

Parameters

  • model (openvino.runtime.CompiledModel) – Model to be used to create InferRequests in a pool.

  • jobs – Number of InferRequests objects in a pool. If 0, jobs number

will be set automatically to the optimal number. Default: 0 :type jobs: int :rtype: openvino.runtime.AsyncInferQueue

__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__iter__(self: openvino.pyopenvino.AsyncInferQueue) → Iterator
__le__(value, /)

Return self<=value.

__len__(self: openvino.pyopenvino.AsyncInferQueue) → int

Number of InferRequests in the pool.

Return type

int

__lt__(value, /)

Return self<value.

__ne__(value, /)

Return self!=value.

__new__(**kwargs)
__reduce__()

Helper for pickle.

__reduce_ex__(protocol, /)

Helper for pickle.

__repr__(self: openvino.pyopenvino.AsyncInferQueue) → str
__setattr__(name, value, /)

Implement setattr(self, name, value).

__sizeof__()

Size of object in memory, in bytes.

__str__()

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

get_idle_request_id(self: openvino.pyopenvino.AsyncInferQueue) → int

Returns next free id of InferRequest from queue’s pool. Function waits for any request to complete and then returns this request’s id.

Function releases GIL, other threads can work while this function waits.

Return type

int

is_ready(self: openvino.pyopenvino.AsyncInferQueue) → bool

One of ‘flow control’ functions. Returns True if any free request in the pool, otherwise False.

Function releases GIL, other threads can work while this function waits.

Returns

If there is at least one free InferRequest in a pool, returns True.

Return type

bool

set_callback(self: openvino.pyopenvino.AsyncInferQueue, arg0: function) → None

Sets unified callback on all InferRequests from queue’s pool. The signature of such function should have two arguments, where the first one is InferRequest object and the second one is userdata connected to InferRequest from the AsyncInferQueue’s pool.

def f(request, userdata):
    result = request.output_tensors[0]
    print(result + userdata)

async_infer_queue.set_callback(f)
Parameters

callback (function) – Any Python defined function that matches callback’s requirements.

start_async(inputs: Optional[Union[dict, list]] = None, userdata: Optional[Any] = None) → None

Run asynchronous inference using the next available InferRequest from the pool.

The allowed types of keys in the inputs dictionary are:

  1. int

  2. str

  3. openvino.runtime.ConstOutput

The allowed types of values in the inputs are:

  1. numpy.array

  2. openvino.runtime.Tensor

Parameters

  • inputs (Union[Dict[keys, values], List[values]], optional) – Data to be set on input tensors of the next available InferRequest.

  • userdata (Any, optional) – Any data that will be passed to a callback.

property userdata
Returns

List of all passed userdata. None if the data wasn’t passed yet.

Return type

List[Any]

wait_all(self: openvino.pyopenvino.AsyncInferQueue) → None

One of ‘flow control’ functions. Blocking call. Waits for all InferRequests in a pool to finish scheduled work.

Function releases GIL, other threads can work while this function waits.

Prev Next