Edit this page

interface InferenceEngine::IInferRequestInternal

Overview

An internal API of synchronous inference request to be implemented by plugin, which is used in InferRequestBase forwarding mechanism. More…

#include <ie_iinfer_request_internal.hpp>

template IInferRequestInternal: public std::enable_shared_from_this< IInferRequestInternal >
{
    // typedefs

    typedef std::shared_ptr<IInferRequestInternal> Ptr;
    typedef std::function<void(std::exception_ptr)> Callback;

    // construction

    IInferRequestInternal();

    IInferRequestInternal(
        const InputsDataMap& networkInputs,
        const OutputsDataMap& networkOutputs
        );

    IInferRequestInternal(
        const std::vector<std::shared_ptr<const ov::Node>>& networkInputs,
        const std::vector<std::shared_ptr<const ov::Node>>& networkOutputs
        );

    // methods

    virtual void Infer();
    virtual void InferImpl();
    virtual void Cancel();
    virtual std::map<std::string, InferenceEngineProfileInfo> GetPerformanceCounts() const;
    virtual void SetBlob(const std::string& name, const Blob::Ptr& data);

    virtual void SetBlobs(
        const std::string& name,
        const std::vector<Blob::Ptr>& blobs
        );

    virtual void SetBlobsImpl(
        const std::string& name,
        const BatchedBlob::Ptr& batched_blob
        );

    virtual Blob::Ptr GetBlob(const std::string& name);
    virtual BatchedBlob::Ptr GetBlobs(const std::string& name);

    virtual void SetBlob(
        const std::string& name,
        const Blob::Ptr& data,
        const PreProcessInfo& info
        );

    virtual const PreProcessInfo& GetPreProcess(const std::string& name) const;
    virtual void SetBatch(int batch);
    virtual std::vector<std::shared_ptr<IVariableStateInternal>> QueryState();
    virtual void StartAsync();
    virtual void StartAsyncImpl();
    virtual StatusCode Wait(int64_t millis_timeout);
    virtual void SetCallback(Callback callback);

    void checkBlob(
        const Blob::Ptr& blob,
        const std::string& name,
        bool isInput,
        const SizeVector& refDims = {}
        ) const;

    virtual void checkBlobs();
    void setPointerToExecutableNetworkInternal(const std::shared_ptr<IExecutableNetworkInternal>& exeNetwork);
    std::shared_ptr<IExecutableNetworkInternal> getPointerToExecutableNetworkInternal() const;
    void setPointerToSo(const std::shared_ptr<void>& so);
    std::shared_ptr<void> getPointerToSo() const;
    void \* GetUserData();
    void SetUserData(void \* userData);
    const std::vector<std::shared_ptr<const ov::Node>>& GetInputs() const;
    const std::vector<std::shared_ptr<const ov::Node>>& GetOutputs() const;

    virtual void setModelInputsOutputs(
        const std::vector<std::shared_ptr<const ov::Node>>& inputs,
        const std::vector<std::shared_ptr<const ov::Node>>& outputs
        );

protected:
};

// direct descendants

class AsyncInferRequestThreadSafeDefault;

Detailed Documentation

An internal API of synchronous inference request to be implemented by plugin, which is used in InferRequestBase forwarding mechanism.

Typedefs

typedef std::shared_ptr<IInferRequestInternal> Ptr

A shared pointer to a IInferRequestInternal interface.

typedef std::function<void(std::exception_ptr)> Callback

Alias for callback type.

Construction

IInferRequestInternal(
    const InputsDataMap& networkInputs,
    const OutputsDataMap& networkOutputs
    )

Constructs a new instance.

Parameters:

networkInputs	The network inputs info
networkOutputs	The network outputs data

IInferRequestInternal(
    const std::vector<std::shared_ptr<const ov::Node>>& networkInputs,
    const std::vector<std::shared_ptr<const ov::Node>>& networkOutputs
    )

Constructs a new instance.

Parameters:

inputs	The network inputs
outputs	The network outputs

Methods

virtual void Infer()

Infers specified input(s) in synchronous mode.

blocks all method of InferRequest while request is ongoing (running or waiting in queue)

virtual void InferImpl()

The minimal infer function to be implemented by plugins. It infers specified input(s) in synchronous mode.

• This method is used in IInferRequestInternal::Infer, which calls the common code first and after uses this plugin dependent implementation.

• Blocks all method of InferRequest while request is ongoing (running or waiting in queue)

virtual void Cancel()

Cancel current inference request execution.

virtual std::map<std::string, InferenceEngineProfileInfo> GetPerformanceCounts() const

Queries performance measures per layer to get feedback of what is the most time consuming layer. Note: not all plugins may provide meaningful data.

Returns:

• a map of layer names to profiling information for that layer.

virtual void SetBlob(const std::string& name, const Blob::Ptr& data)

Set input/output data to infer.

Memory allocation doesn’t happen

Parameters:

name	a name of input or output blob.
data	a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

virtual void SetBlobs(
    const std::string& name,
    const std::vector<Blob::Ptr>& blobs
    )

Set batch of input data to infer. Default implementation performs basic validation and checks that all tensors are not remote. Plugin-specific implementations may override this behavior to handle remote tensors case. If plugin expects only memory blobs (not remote blobs), consider to override only SetBlobsImpl and reuse basic existing implementation.

Parameters:

name	an operation name of input or output blob.
blobs	input blobs. The type of Blob must correspond to the model’s input precision and size.

virtual void SetBlobsImpl(
    const std::string& name,
    const BatchedBlob::Ptr& batched_blob
    )

Set batch of input data to infer. Default implementation throws “Not implemented” exception To support ‘set_input_tensors’/’set_tensors’ plugin-specific implementations shall:

• Inside SetBlobsImpl: update ‘InferenceEngine::IInferRequestInternal::batched_inputs’ map

• Inside ‘SetBlob’: erase appropriate ‘ InferenceEngine::IInferRequestInternal::_batched_inputs [name]’ item

• Inside ‘InferImpl’: call ‘convertBatchedInputBlobs’ on the beginning to convert many user blobs into single one

• If needed, override ‘convertBatchedInputBlob’ to perform custom concatenation and data copy to input blob

Parameters:

name	an operation name of input or output blob.
batched_blob	input blobs combined in batched blob. Called only if number of blobs > 1 precision and size.

virtual Blob::Ptr GetBlob(const std::string& name)

Get input/output data to infer.

Memory allocation doesn’t happen

Parameters:

name	a name of input or output blob.
data	a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

virtual BatchedBlob::Ptr GetBlobs(const std::string& name)

Get input/output data to infer.

Memory allocation doesn’t happen

Parameters:

name	a name of input or output blob.

Returns:

data - a reference to input batched blob.

virtual void SetBlob(
    const std::string& name,
    const Blob::Ptr& data,
    const PreProcessInfo& info
    )

Sets pre-process for input data.

Parameters:

name	Name of input blob.
data	a reference to input or output blob. The type of Blob must correspond to the network input precision and size.
info	Preprocess info for blob.

virtual const PreProcessInfo& GetPreProcess(const std::string& name) const

Gets pre-process for input data.

Parameters:

name	Name of input blob.
info	pointer to a pointer to PreProcessInfo structure

virtual void SetBatch(int batch)

Sets new batch size when dynamic batching is enabled in executable network that created this request.

Parameters:

batch

new batch size to be used by all the following inference calls for this request.

virtual std::vector<std::shared_ptr<IVariableStateInternal>> QueryState()

Queries memory states.

Returns:

Returns memory states

virtual void StartAsync()

Start inference of specified input(s) in asynchronous mode.

The method returns immediately. Inference starts also immediately.

virtual void StartAsyncImpl()

The minimal asynchronous inference function to be implemented by plugins. It starts inference of specified input(s) in asynchronous mode.

• The methos is used in AsyncInferRequestInternal::StartAsync which performs common steps first and calls plugin dependent implementation of this method after.

• It returns immediately. Inference starts also immediately.

virtual StatusCode Wait(int64_t millis_timeout)

Waits for the result to become available. Blocks until specified millis_timeout has elapsed or the result becomes available, whichever comes first.

There are special cases when millis_timeout is equal some value of WaitMode enum:

• STATUS_ONLY - immediately returns request status (InferRequest::StatusCode). It doesn’t block or interrupt current thread.

• RESULT_READY - waits until inference result becomes available

Parameters:

millis_timeout

maximum duration in milliseconds to block for

Returns:

A status code

virtual void SetCallback(Callback callback)

Set callback function which will be called on success or failure of asynchronous request.

Parameters:

callback

function to be called with the following description:

void checkBlob(
    const Blob::Ptr& blob,
    const std::string& name,
    bool isInput,
    const SizeVector& refDims = {}
    ) const

Check that blob is valid. Throws an exception if it’s not.

Parameters:

blob	The blob to check
name	The name of input or output depending of if the blob is input or output
isInput	Indicates if is input
refDims	The reference dims, empty if not specified

virtual void checkBlobs()

Check that all of the blobs is valid. Throws an exception if it’s not.

void setPointerToExecutableNetworkInternal(const std::shared_ptr<IExecutableNetworkInternal>& exeNetwork)

Sets the pointer to executable network internal.

Needed to correctly handle ownership between objects.

Parameters:

exeNetwork

The executable network

std::shared_ptr<IExecutableNetworkInternal> getPointerToExecutableNetworkInternal() const

Returns the pointer to executable network internal.

Returns:

The executable network

void setPointerToSo(const std::shared_ptr<void>& so)

Sets the pointer to so when needed.

Needed to correctly handle ownership between objects.

Parameters:

so	The library so

std::shared_ptr<void> getPointerToSo() const

Returns the pointer to so.

Returns:

The library

void \* GetUserData()

Gets the pointer to userData.

Returns:

Pointer to user data

void SetUserData(void \* userData)

Sets the pointer to userData.

Parameters:

Pointer

to user data

virtual void setModelInputsOutputs(
    const std::vector<std::shared_ptr<const ov::Node>>& inputs,
    const std::vector<std::shared_ptr<const ov::Node>>& outputs
    )

Sets inputs/outputs from ov::Model.

Prev Next