Cross Check Tool¶
Cross Check Tool is a console application that enables comparing accuracy and performance metrics for two successive model inferences that are performed on two different supported Intel devices or with different precisions. The Cross Check Tool can compare the metrics per layer or all over the model.
Running the Cross Check Tool¶
Cross Check Tool is distributed as a Python module and there is no need to build it. To run the Cross Check Tool, execute the cross_check_tool.py file with necessary parameters. Please note that the Inference Engine assumes that weights are in the same folder as the .xml file.
You can get the list of all available options using the -h option:
$python3 cross_check_tool.py -h
Cross Check Tool is a console application that enables comparing accuracy and
provides performance metrics
optional arguments:
-h, --help show this help message and exit
Model specific arguments:
--input INPUT, -i INPUT
Path to an input image file or multi-input file to
infer. Generates input(s) from normal distribution if
empty
--model MODEL, -m MODEL
Path to an .xml file that represents the first IR of
the trained model to infer.
--reference_model REFERENCE_MODEL, -ref_m REFERENCE_MODEL
Path to an .xml file that represents the second IR in
different precision to compare the metrics.
--layers LAYERS, -layers LAYERS
Defines layers to check. Options: all, None - for
output layers check, list of comma-separated layer
names to check. Default value is None.
--ref_layers REFERENCE_LAYERS, -reference_layers REFERENCE_LAYERS
Defines layers to check in reference model. Options: all, None - for
output layers check, list of comma-separated layer
names to check. If not specified the same layers will
be processed as in --layers parameter.
--num_of_iterations NUM_OF_ITERATIONS, -ni NUM_OF_ITERATIONS
Number of iterations to collect all over the net
performance
Plugin specific arguments:
--plugin_path PLUGIN_PATH, -pp PLUGIN_PATH
Path to a plugin folder.
--device DEVICE, -d DEVICE
The first target device to infer the model specified
with the -m or --model option. CPU, GPU, HDDL or
MYRIAD are acceptable.
--config CONFIG, -conf CONFIG
Path to config file for -d or -device device plugin
--reference_device REFERENCE_DEVICE, -ref_d REFERENCE_DEVICE
The second target device to infer the model and
compare the metrics. CPU, GPU, HDDL or MYRIAD are
acceptable.
--reference_config REFERENCE_CONFIG, -ref_conf REFERENCE_CONFIG
Path to config file for -ref_d or -reference_device
device plugin
-l L Required for (CPU)-targeted custom layers.
Comma separated paths to a shared libraries with the
kernels implementation.
CCT mode arguments:
--dump Enables blobs statistics dumping
--load LOAD Path to a file to load blobs fromCross Check Tool can also be installed via:
$python3 -m pip install <openvino_repo>/tools/cross_check_toolIn this case, to run the tool, call cross_check_tool on the command line with necessary parameters.
Examples¶
To check per-layer accuracy and performance of inference in FP32 precision on the CPU against the GPU, run:
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \ -m <path_to_FP32_xml> \ -d GPU \ -ref_d CPU \ --layers all
The output looks as follows:
[ INFO ] Cross check with one IR was enabled [ INFO ] GPU:FP32 vs CPU:FP32 [ INFO ] The same IR on both devices: <path_to_IR> [ INFO ] Statistics will be dumped for X layers: <layer_1_name>, <layer_2_name>, ... , <layer_X_name> [ INFO ] Layer <layer_1_name> statistics Max absolute difference : 1.15204E-03 Min absolute difference : 0.0 Max relative difference : 1.15204E+17 Min relative difference : 0.0 Min reference value : -1.69513E+03 Min absolute reference value : 2.71080E-06 Max reference value : 1.17132E+03 Max absolute reference value : 1.69513E+03 Min actual value : -1.69513E+03 Min absolute actual value : 8.66465E-05 Max actual value : 1.17132E+03 Max absolute actual value : 1.69513E+03 Device: -d GPU -ref_d CPU Status: OPTIMIZED_OUT OPTIMIZED_OUT Layer type: Convolution Convolution Real time, microsec: 0 120 Number of NAN: 0 0 Number of INF: 0 0 Number of ZERO: 0 0 ... <list_of_layer_statistics> ... [ INFO ] Overall max absolute difference = 0.00115203857421875 [ INFO ] Overall min absolute difference = 0.0 [ INFO ] Overall max relative difference = 1.1520386483093504e+17 [ INFO ] Overall min relative difference = 0.0 [ INFO ] Execution successful
To check the overall accuracy and performance of inference on the CPU in FP32 precision against the Intel Movidius Myriad device in FP16 precision, run:
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \ -m <path_to_FP16_xml> \ -d MYRIAD \ -ref_m <path_to_FP32_xml> \ -ref_d CPU
The output looks as follows:
[ INFO ] Cross check with two IRs was enabled [ INFO ] GPU:FP16 vs CPU:FP32 [ INFO ] IR for MYRIAD : <path_to_FP16_xml> [ INFO ] IR for CPU : <path_to_FP32_xml> [ INFO ] Statistics will be dumped for 1 layer: <output_layer_name(s)> [ INFO ] Layer <output_layer_name> statistics Max absolute difference : 2.32944E-02 Min absolute difference : 3.63002E-13 Max relative difference : 6.41717E+10 Min relative difference : 1.0 Min reference value : 3.63002E-13 Min absolute reference value : 3.63002E-13 Max reference value : 7.38138E-01 Max absolute reference value : 7.38138E-01 Min actual value : 0.0 Min absolute actual value : 0.0 Max actual value : 7.14844E-01 Max absolute actual value : 7.14844E-01 Device: -d MYRIAD -ref_d CPU Status: OPTIMIZED_OUT OPTIMIZED_OUT Layer type: Reshape Reshape Real time, microsec: 0 0 Number of NAN: 0 0 Number of INF: 0 0 Number of ZERO: 0 0 ---------------------------------------------------------------------- Overall performance, microseconds: 2.79943E+05 6.24670E+04 ---------------------------------------------------------------------- [ INFO ] Overall max absolute difference = 0.023294448852539062 [ INFO ] Overall min absolute difference = 3.630019191052519e-13 [ INFO ] Overall max relative difference = 64171696128.0 [ INFO ] Overall min relative difference = 1.0 [ INFO ] Execution successful
To dump layer statistics from a specific list of layers, run:
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \ -m <path_to_FP16_xml> \ -d MYRIAD \ --dump \ --layers <comma_separated_list_of_layers>
The output looks as follows:
[ INFO ] Dump mode was enabled [ INFO ] <layer_1_name> layer processing ... [ INFO ] <layer_X_name> layer processing [ INFO ] Dump file path: <path_where_dump_will_be_saved> [ INFO ] Execution successful
If you do not provide the
-ikey, the Cross Check Tool generates an input from normal distributed noise and saves it in a multi-input file format with the filename<path_to_xml>_input_layers_dump.txtin the same folder as the Intermediate Representation (IR).To check the overall accuracy and performance of inference on the CPU in FP32 precision against dumped results, run:
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \ -m <path_to_FP32_xml> \ -d CPU \ --load <path_to_dump> \ --layers all
The output looks as follows:
[ INFO ] Load mode was enabled [ INFO ] IR for CPU : <path_to_FP32_xml> [ INFO ] Loading blob from /localdisk/models/FP16/icv_squeezenet_v1.0.xml_GPU_dump.npz [ INFO ] Statistics will be dumped for X layers: <layer_1_name>, <layer_2_name>, ... , <layer_X_name> [ INFO ] Layer <layer_1_name> statistics Max absolute difference : 0.0 Min absolute difference : 0.0 Max relative difference : 0.0 Min relative difference : 0.0 Min reference value : 0.0 Min absolute reference value : 0.0 Max reference value : 7.14844E-01 Max absolute reference value : 7.14844E-01 Min actual value : 0.0 Min absolute actual value : 0.0 Max actual value : 7.14844E-01 Max absolute actual value : 7.14844E-01 Device: -d CPU -load GPU Status: OPTIMIZED_OUT OPTIMIZED_OUT Layer type: Reshape Reshape Real time, microsec: 0 0 Number of NAN: 0 0 Number of INF: 0 0 Number of ZERO: 609 699 ... <list_of_layer_statistics> ... [ INFO ] Overall max absolute difference = 0.0 [ INFO ] Overall min absolute difference = 0.0 [ INFO ] Overall max relative difference = 0.0 [ INFO ] Overall min relative difference = 0.0 [ INFO ] Execution successful
Multi-input and dump file format¶
Multi-input and dump file is a numpy compressed .npz file with hierarchy:
{
‘layer_name’: {
‘blob’: np.array([…])
‘pc’: {
‘device’: ‘device_name’,
‘real_time’: int_real_time_in_microseconds_from_plugin,
‘exec_type’: ‘exec_type_from_plugin’,
‘layer_type’: ‘layer_type_from_plugin’,
‘status’: ‘status_from_plugin’
}
},
‘another_layer_name’: {
‘blob’: np.array([…])
‘pc’: {
‘device’: ‘device_name’,
‘real_time’: int_real_time_in_microseconds_from_plugin,
‘exec_type’: ‘exec_type_from_plugin’,
‘layer_type’: ‘layer_type_from_plugin’,
‘status’: ‘status_from_plugin’
}
},
...
}Configuration file¶
There is an option to pass configuration file to plugin by providing --config and/or --reference_config keys.
Configuration file is a text file with content of pairs of keys and values.
Structure of configuration file:
KEY VALUE
ANOTHER_KEY ANOTHER_VALUE,VALUE_1