Model Representation in OpenVINO™ Runtime¶

In OpenVINO™ Runtime a model is represented by the ov::Model class.

The ov::Model object stores shared pointers to ov::op::v0::Parameter, ov::op::v0::Result and ov::op::Sink operations that are inputs, outputs and sinks of the graph. Sinks of the graph have no consumers and are not included in the results vector. All other operations hold each other via shared pointers: child operation holds its parent (hard link). If an operation has no consumers and it’s not the Result or Sink operation (shared pointer counter is zero), then it will be destructed and won’t be accessible anymore.

Each operation in ov::Model has the std::shared_ptr<ov::Node> type.

For details on how to build a model in OpenVINO™ Runtime, see the Build a Model in OpenVINO™ Runtime section.

OpenVINO™ Runtime allows to use different approaches to work with model inputs/outputs:

ov::Model::inputs() / ov::Model::outputs() methods allow to get vector of all input/output ports.
For a model which has only one input or output you can use methods ov::Model::input() or ov::Model::output() without arguments to get input or output port respectively.
Methods ov::Model::input() and ov::Model::output() can be used with index of input or output from the framework model to get specific port by index.
You can use tensor name of input or output from the original framework model together with methods ov::Model::input() or ov::Model::output() to get specific port. It means that you don’t need to have any additional mapping of names from framework to OpenVINO, as it was before, OpenVINO™ Runtime allows using of native framework tensor names.

/\* Take information about all topology inputs \*/
auto inputs = model->inputs();
/\* Take information about all topology outputs \*/
auto outputs = model->outputs();

inputs = model.inputs
outputs = model.outputs

OpenVINO™ Runtime model representation uses special classes to work with model data types and shapes. For data types the ov::element::Type is used.

Shapes Representation¶

OpenVINO™ Runtime provides two types for shape representation:

ov::Shape - Represents static (fully defined) shapes.
ov::PartialShape - Represents dynamic shapes. That means that the rank or some of dimensions are dynamic (dimension defines an interval or undefined). ov::PartialShape can be converted to ov::Shape using the get_shape() method if all dimensions are static; otherwise the conversion raises an exception.

ov::Shape static_shape;
ov::PartialShape partial_shape = node->output(0).get_partial_shape(); // get zero output partial shape
if (!partial_shape.is_dynamic() /\* or partial_shape.is_static() \*/) {
    static_shape = partial_shape.get_shape();
}

partial_shape = node.output(0).get_partial_shape() # get zero output partial shape
if not partial_shape.is_dynamic: # or partial_shape.is_static
    static_shape = partial_shape.get_shape()

But in most cases before getting static shape using get_shape() method, you need to check that shape is static.

Operations¶

The ov::Op class represents any abstract operation in the model representation. Use this class to create custom operations.

Operation Sets¶

Operation set (opset) is a collection of operations that can be used to construct a model. The ov::OpSet class provides a functionality to work with operation sets. For each operation set, OpenVINO™ Runtime provides a separate namespace, for example opset8. Each OpenVINO™ Release introduces new operations and add these operations to a new operation set. New operation sets help to introduce a new version of operations that change behavior of previous operations. Using operation sets allows you to avoid changes in your application if new operations have been introduced. For a complete list of operation sets supported in OpenVINO™ toolkit, see Available Operations Sets. To add support of custom operations, see the Add Custom OpenVINO Operations document.

Build a Model in OpenVINO™ Runtime¶

You can create a model from source. This section illustrates how to construct a model composed of operations from an available operation set.

Operation set opsetX integrates a list of pre-compiled operations that work for this purpose. In other words, opsetX defines a set of operations for building a graph.

To build an ov::Model instance from opset8 operations, include the following files:

#include <openvino/core/model.hpp>
#include <openvino/opsets/opset8.hpp>

import openvino.runtime as ov

The following code demonstrates how to create a simple model:

std::shared_ptr<ov::Model> create_simple_model() {
    // This example shows how to create ov::Model
    //
    // Parameter--->Multiply--->Add--->Result
    //    Constant---'          /
    //              Constant---'

    // Create opset8::Parameter operation with static shape
    auto data = std::make_shared<ov::opset8::Parameter>(ov::element::f32, ov::Shape{3, 1, 2});

    auto mul_constant = ov::opset8::Constant::create(ov::element::f32, ov::Shape{1}, {1.5});
    auto mul = std::make_shared<ov::opset8::Multiply>(data, mul_constant);

    auto add_constant = ov::opset8::Constant::create(ov::element::f32, ov::Shape{1}, {0.5});
    auto add = std::make_shared<ov::opset8::Add>(mul, add_constant);

    // Create opset8::Result operation
    auto res = std::make_shared<ov::opset8::Result>(mul);

    // Create nGraph function
    return std::make_shared<ov::Model>(ov::ResultVector{res}, ov::ParameterVector{data});
}

def create_simple_model():
    # This example shows how to create ov::Function
    #
    # Parameter--->Multiply--->Add--->Result
    #    Constant---'          /
    #              Constant---'
    data = ov.opset8.parameter([3, 1, 2], ov.Type.f32)
    mul_constant = ov.opset8.constant([1.5], ov.Type.f32)
    mul = ov.opset8.multiply(data, mul_constant)
    add_constant = ov.opset8.constant([0.5], ov.Type.f32)
    add = ov.opset8.add(mul, add_constant)
    res = ov.opset8.result(add)
    return ov.Model([res], [data], "model")

The following code creates a model with several outputs:

std::shared_ptr<ov::Model> create_advanced_model() {
    // Advanced example with multi output operation
    //
    // Parameter->Split---0-->Result
    //               | `--1-->Relu-->Result
    //               `----2-->Result

    auto data = std::make_shared<ov::opset8::Parameter>(ov::element::f32, ov::Shape{1, 3, 64, 64});

    // Create Constant for axis value
    auto axis_const = ov::opset8::Constant::create(ov::element::i64, ov::Shape{} /\*scalar shape\*/, {1});

    // Create opset8::Split operation that splits input to three slices across 1st dimension
    auto split = std::make_shared<ov::opset8::Split>(data, axis_const, 3);

    // Create opset8::Relu operation that takes 1st Split output as input
    auto relu = std::make_shared<ov::opset8::Relu>(split->output(1) /\*specify explicit output\*/);

    // Results operations will be created automatically based on provided OutputVector
    return std::make_shared<ov::Model>(ov::OutputVector{split->output(0), relu, split->output(2)},
                                       ov::ParameterVector{data});
}

def create_advanced_model():
    # Advanced example with multi output operation
    #
    # Parameter->Split---0-->Result
    #               | `--1-->Relu-->Result
    #               `----2-->Result
    data = ov.opset8.parameter(ov.Shape([1, 3, 64, 64]), ov.Type.f32)
    # Create Constant for axis value
    axis_const = ov.opset8.constant(ov.Type.i64, ov.Shape({}), [1])

    # Create opset8::Split operation that splits input to three slices across 1st dimension
    split = ov.opset8.split(data, axis_const, 3)

    # Create opset8::Relu operation that takes 1st Split output as input
    relu = ov.opset8.relu(split.output(1))

    # Results operations will be created automatically based on provided OutputVector
    return ov.Model([split.output(0), relu, split.output[2]], [data], "model")

Model debug capabilities¶

OpenVINO™ provides several debug capabilities:

To receive additional messages about applied model modifications, rebuild the OpenVINO™ Runtime library with the -DENABLE_OPENVINO_DEBUG=ON option.

Model can be visualized to image from the xDot format:

void visualize_example(const std::shared_ptr<ov::Model>& m) {
    // Need include:
    // \* openvino/pass/manager.hpp
    // \* openvino/pass/visualize_tree.hpp
    ov::pass::Manager manager;

    // Serialize ov::Model to before.svg file before transformation
    manager.register_pass<ov::pass::VisualizeTree>("image.svg");

    manager.run_passes(m);
}

def visualize_example(m : ov.Model):
    # Need import:
    # \* import openvino.runtime.passes as passes
    pass_manager = passes.Manager()
    pass_manager.register_pass(pass_name="VisualTree", file_name='image.svg')
    pass_manager.run_passes(m)

`ov::pass::VisualizeTree` can be parametrized via environment variables:

    OV_VISUALIZE_TREE_OUTPUT_SHAPES=1       - visualize shapes
    OV_VISUALIZE_TREE_OUTPUT_TYPES=1        - visualize types
    OV_VISUALIZE_TREE_MIN_MAX_DENORMAL=1    - pretty denormal values
    OV_VISUALIZE_TREE_RUNTIME_INFO=1        - print runtime information
    OV_VISUALIZE_TREE_IO=1                  - print I/O ports
    OV_VISUALIZE_TREE_MEMBERS_NAME=1        - print member names

Also model can be serialized to IR:

void serialize_example(const std::shared_ptr<ov::Model>& f) {
    // Need include:
    // \* openvino/pass/manager.hpp
    // \* openvino/pass/serialize.hpp
    ov::pass::Manager manager;

    // Serialize ov::Model to IR
    manager.register_pass<ov::pass::Serialize>("/path/to/file/model.xml", "/path/to/file/model.bin");

    manager.run_passes(f);
}

def serialize_example(m : ov.Model):
    # Need import:
    # \* import openvino.runtime.passes as passes
    pass_manager = passes.Manager()
    pass_manager.register_pass(pass_name="Serialize", xml_path='model.xml', bin_path='model.bin')
    pass_manager.run_passes(m)

Available Operation Sets

Prev Next