Intel® Distribution of OpenVINO™ toolkit Benchmark Results¶
This benchmark setup includes a single machine on which both the benchmark application and the OpenVINO™ installation reside.
The benchmark application loads the OpenVINO™ Runtime and executes inferences on the specified hardware (CPU, GPU or VPU). The benchmark application measures the time spent on actual inferencing (excluding any pre or post processing) and then reports on the inferences per second (or Frames Per Second). For more information on the benchmark application, please also refer to the entry 5 of the FAQ section.
Measuring inference performance involves many variables and is extremely use-case and application dependent. We use the below four parameters for measurements, which are key elements to consider for a successful deep learning inference application:
Throughput - Measures the number of inferences delivered within a latency threshold. (for example, number of Frames Per Second - FPS). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.
Value - While throughput is important, what is more critical in edge AI deployments is the performance efficiency or performance-per-cost. Application performance in throughput per dollar of system cost is the best measure of value.
Efficiency - System power is a key consideration from the edge to the data center. When selecting deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider. Intel designs provide excellent power efficiency for running deep learning workloads.
Latency - This measures the synchronous execution of inference requests and is reported in milliseconds. Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before the next is started. This performance metric is relevant in usage scenarios where a single image input needs to be acted upon as soon as possible. An example would be the healthcare sector where medical personnel only request analysis of a single ultra sound scanning image or in real-time or near real-time applications for example an industrial robot’s response to actions in its environment or obstacle avoidance for autonomous vehicles.
bert-base-cased [124]¶
bert-large-uncased-whole-word-masking-squad-int8-0001 [384]¶
deeplabv3-TF [513x513]¶
densenet-121-TF [224x224]¶
efficientdet-d0 [512x512]¶
faster-rcnn-resnet50-coco-TF [600x1024]¶
inception-v4-TF [299x299]¶
mobilenet-ssd-CF [300x300]¶
mobilenet-v2-pytorch [224x224]¶
resnet-18-pytorch [224x224]¶
resnet_50_TF [224x224]¶
ssd-resnet34-1200-onnx [1200x1200]¶
unet-camvid-onnx-0001 [368x480]¶
yolo-v3-tiny-tf [416x416]¶
yolo_v4-tf [608x608]¶
Platform Configurations¶
Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are based on release 2022.1.
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of March 17, 2022 and may not reflect all publicly available updates. See configuration disclosure for details. No product can be absolutely secure.
Performance varies by use, configuration and other factors. Learn more at www.intel.com/PerformanceIndex.
Your costs and results may vary.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.
Testing by Intel done on: see test date for each HW platform below.
CPU Inference Engines
Configuration |
Intel® Xeon® E-2124G |
Intel® Xeon® W1290P |
|---|---|---|
Motherboard |
ASUS* WS C246 PRO |
ASUS* WS W480-ACE |
CPU |
Intel® Xeon® E-2124G CPU @ 3.40GHz |
Intel® Xeon® W-1290P CPU @ 3.70GHz |
Hyper Threading |
OFF |
ON |
Turbo Setting |
ON |
ON |
Memory |
2 x 16 GB DDR4 2666MHz |
4 x 16 GB DDR4 @ 2666MHz |
Operating System |
Ubuntu* 20.04.3 LTS |
Ubuntu* 20.04.3 LTS |
Kernel Version |
5.4.0-42-generic |
5.4.0-42-generic |
BIOS Vendor |
American Megatrends Inc.* |
American Megatrends Inc. |
BIOS Version |
1901 |
2301 |
BIOS Release |
September 24, 2021 |
July 8, 2021 |
BIOS Settings |
Select optimized default settings, save & exit |
Select optimized default settings, save & exit |
Batch size |
1 |
1 |
Precision |
INT8 |
INT8 |
Number of concurrent inference requests |
4 |
5 |
Test Date |
March 17, 2022 |
March 17, 2022 |
Rated maximum TDP/socket in Watt |
||
CPU Price/socket on Feb 14, 2022, USD Prices may vary |
CPU Inference Engines (continue)
Configuration |
Intel® Xeon® Silver 4216R |
Intel® Xeon® Silver 4316 |
|---|---|---|
Motherboard |
Intel® Server Board S2600STB |
Intel Corporation / WilsonCity |
CPU |
Intel® Xeon® Silver 4216R CPU @ 2.20GHz |
Intel® Xeon® Silver 4316 CPU @ 2.30GHz |
Hyper Threading |
ON |
ON |
Turbo Setting |
ON |
ON |
Memory |
12 x 32 GB DDR4 2666MHz |
16 x 32 GB DDR4 @ 2666MHz |
Operating System |
Ubuntu* 20.04.3 LTS |
Ubuntu* 20.04.3 LTS |
Kernel Version |
5.3.0-24-generic |
5.4.0-100-generic |
BIOS Vendor |
Intel Corporation |
Intel Corporation |
BIOS Version |
SE5C620.86B.02.01. 0013.121520200651 |
WLYDCRB1.SYS.0021. P41.2109200451 |
BIOS Release |
December 15, 2020 |
September 20, 2021 |
BIOS Settings |
Select optimized default settings, change power policy to “performance”, save & exit |
Select optimized default settings, save & exit |
Batch size |
1 |
1 |
Precision |
INT8 |
INT8 |
Number of concurrent inference requests |
32 |
10 |
Test Date |
March 17, 2022 |
March 17, 2022 |
Rated maximum TDP/socket in Watt |
||
CPU Price/socket on June 21, 2021, USD Prices may vary |
CPU Inference Engines (continue)
Configuration |
Intel® Xeon® Gold 5218T |
Intel® Xeon® Platinum 8270 |
Intel® Xeon® Platinum 8380 |
|---|---|---|---|
Motherboard |
Intel® Server Board S2600STB |
Intel® Server Board S2600STB |
Intel Corporation / WilsonCity |
CPU |
Intel® Xeon® Gold 5218T CPU @ 2.10GHz |
Intel® Xeon® Platinum 8270 CPU @ 2.70GHz |
Intel® Xeon® Platinum 8380 CPU @ 2.30GHz |
Hyper Threading |
ON |
ON |
ON |
Turbo Setting |
ON |
ON |
ON |
Memory |
12 x 32 GB DDR4 2666MHz |
12 x 32 GB DDR4 2933MHz |
16 x 16 GB DDR4 3200MHz |
Operating System |
Ubuntu* 20.04.3 LTS |
Ubuntu* 20.04.3 LTS |
Ubuntu* 20.04.1 LTS |
Kernel Version |
5.3.0-24-generic |
5.3.0-24-generic |
5.4.0-64-generic |
BIOS Vendor |
Intel Corporation |
Intel Corporation |
Intel Corporation |
BIOS Version |
SE5C620.86B.02.01. 0013.121520200651 |
SE5C620.86B.02.01. 0013.121520200651 |
WLYDCRB1.SYS.0020. P86.2103050636 |
BIOS Release |
December 15, 2020 |
December 15, 2020 |
March 5, 2021 |
BIOS Settings |
Select optimized default settings, change power policy to “performance”, save & exit |
Select optimized default settings, change power policy to “performance”, save & exit |
Select optimized default settings, change power policy to “performance”, save & exit |
Batch size |
1 |
1 |
1 |
Precision |
INT8 |
INT8 |
INT8 |
Number of concurrent inference requests |
32 |
52 |
80 |
Test Date |
March 17, 2022 |
March 17, 2022 |
March 17, 2022 |
Rated maximum TDP/socket in Watt |
|||
CPU Price/socket on Feb 14, 2022, USD Prices may vary |
CPU Inference Engines (continue)
Configuration |
Intel® Core™ i9-10920X |
Intel® Core™ i9-10900TE |
Intel® Core™ i9-12900 |
|---|---|---|---|
Motherboard |
ASUS* PRIME X299-A II |
B595 |
Intel Corporation internal/Reference Validation Platform |
CPU |
Intel® Core™ i9-10920X CPU @ 3.50GHz |
Intel® Core™ i9-10900TE CPU @ 1.80GHz |
12th Gen Intel® Core™ i9-12900 |
Hyper Threading |
ON |
ON |
OFF |
Turbo Setting |
ON |
ON |
|
Memory |
4 x 16 GB DDR4 2666MHz |
2 x 8 GB DDR4 @ 2400 MHz |
4 x 8 GB DDR4 4800MHz |
Operating System |
Ubuntu 20.04.3 LTS |
Ubuntu 20.04.3 LTS |
Microsoft Windows 10 Pro |
Kernel Version |
5.4.0-42-generic |
5.4.0-42-generic |
10.0.19043 N/A Build 19043 |
BIOS Vendor |
American Megatrends Inc.* |
American Megatrends Inc.* |
Intel Corporation |
BIOS Version |
1004 |
Z667AR10.BIN |
ADLSFWI1.R00.2303. B00.2107210432 |
BIOS Release |
March 19, 2021 |
July 15, 2020 |
July 21, 2021 |
BIOS Settings |
Default Settings |
Default Settings |
Default Settings |
Batch size |
1 |
1 |
1 |
Precision |
INT8 |
INT8 |
INT8 |
Number of concurrent inference requests |
24 |
5 |
4 |
Test Date |
March 17, 2022 |
March 17, 2022 |
March 17, 2022 |
Rated maximum TDP/socket in Watt |
|||
CPU Price/socket on Feb 14, 2022, USD Prices may vary |
CPU Inference Engines (continue)
Configuration |
Intel® Core™ i7-8700T |
Intel® Core™ i7-1185G7 |
|---|---|---|
Motherboard |
GIGABYTE* Z370M DS3H-CF |
Intel Corporation internal/Reference Validation Platform |
CPU |
Intel® Core™ i7-8700T CPU @ 2.40GHz |
Intel® Core™ i7-1185G7 @ 3.00GHz |
Hyper Threading |
ON |
ON |
Turbo Setting |
ON |
ON |
Memory |
4 x 16 GB DDR4 2400MHz |
2 x 8 GB DDR4 3200MHz |
Operating System |
Ubuntu 20.04.3 LTS |
Ubuntu 20.04.3 LTS |
Kernel Version |
5.4.0-42-generic |
5.8.0-050800-generic |
BIOS Vendor |
American Megatrends Inc.* |
Intel Corporation |
BIOS Version |
F14c |
TGLSFWI1.R00.4391. A00.2109201819 |
BIOS Release |
March 23, 2021 |
September 20, 2021 |
BIOS Settings |
Select optimized default settings, set OS type to “other”, save & exit |
Default Settings |
Batch size |
1 |
1 |
Precision |
INT8 |
INT8 |
Number of concurrent inference requests |
4 |
4 |
Test Date |
March 17, 2022 |
March 17, 2022 |
Rated maximum TDP/socket in Watt |
||
CPU Price/socket on Feb 14, 2022, USD Prices may vary |
CPU Inference Engines (continue)
Configuration |
Intel® Core™ i3-8100 |
Intel® Core™ i5-8500 |
Intel® Core™ i5-10500TE |
|---|---|---|---|
Motherboard |
GIGABYTE* Z390 UD |
ASUS* PRIME Z370-A |
GIGABYTE* Z490 AORUS PRO AX |
CPU |
Intel® Core™ i3-8100 CPU @ 3.60GHz |
Intel® Core™ i5-8500 CPU @ 3.00GHz |
Intel® Core™ i5-10500TE CPU @ 2.30GHz |
Hyper Threading |
OFF |
OFF |
ON |
Turbo Setting |
OFF |
ON |
ON |
Memory |
4 x 8 GB DDR4 2400MHz |
2 x 16 GB DDR4 2666MHz |
2 x 16 GB DDR4 @ 2666MHz |
Operating System |
Ubuntu* 20.04.3 LTS |
Ubuntu* 20.04.3 LTS |
Ubuntu* 20.04.3 LTS |
Kernel Version |
5.3.0-24-generic |
5.4.0-42-generic |
5.4.0-42-generic |
BIOS Vendor |
American Megatrends Inc.* |
American Megatrends Inc.* |
American Megatrends Inc.* |
BIOS Version |
F8 |
3004 |
F21 |
BIOS Release |
May 24, 2019 |
July 12, 2021 |
November 23, 2021 |
BIOS Settings |
Select optimized default settings, set OS type to “other”, save & exit |
Select optimized default settings, save & exit |
Select optimized default settings, set OS type to “other”, save & exit |
Batch size |
1 |
1 |
1 |
Precision |
INT8 |
INT8 |
INT8 |
Number of concurrent inference requests |
4 |
3 |
4 |
Test Date |
March 17, 2022 |
March 17, 2022 |
March 17, 2022 |
Rated maximum TDP/socket in Watt |
|||
CPU Price/socket on Feb 14, 2022, USD Prices may vary |
CPU Inference Engines (continue)
Configuration |
Intel Atom® x5-E3940 |
Intel Atom® x6425RE |
Intel® Celeron® 6305E |
|---|---|---|---|
Motherboard |
Intel Corporation internal/Reference Validation Platform |
Intel Corporation internal/Reference Validation Platform |
Intel Corporation internal/Reference Validation Platform |
CPU |
Intel Atom® Processor E3940 @ 1.60GHz |
Intel Atom® x6425RE Processor @ 1.90GHz |
Intel® Celeron® 6305E @ 1.80GHz |
Hyper Threading |
OFF |
OFF |
OFF |
Turbo Setting |
ON |
ON |
ON |
Memory |
1 x 8 GB DDR3 1600MHz |
2 x 4GB DDR4 3200MHz |
2 x 8 GB DDR4 3200MHz |
Operating System |
Ubuntu* 20.04.3 LTS |
Ubuntu* 20.04.3 LTS |
Ubuntu 20.04.3 LTS |
Kernel Version |
5.4.0-42-generic |
5.13.0-27-generic |
5.13.0-1008-intel |
BIOS Vendor |
American Megatrends Inc.* |
Intel Corporation |
Intel Corporation |
BIOS Version |
5.12 |
EHLSFWI1.R00.3273. A01.2106300759 |
TGLIFUI1.R00.4064.A02.2102260133 |
BIOS Release |
September 6, 2017 |
June 30, 2021 |
February 26, 2021 |
BIOS Settings |
Default settings |
Default settings |
Default settings |
Batch size |
1 |
1 |
1 |
Precision |
INT8 |
INT8 |
INT8 |
Number of concurrent inference requests |
4 |
4 |
4 |
Test Date |
March 17, 2022 |
March 17, 2022 |
March 17, 2022 |
Rated maximum TDP/socket in Watt |
|||
CPU Price/socket on Feb 14, 2022, USD Prices may vary |
Accelerator Inference Engines
Configuration |
Intel® Neural Compute Stick 2 |
Intel® Vision Accelerator Design with Intel® Movidius™ VPUs (Mustang-V100-MX8) |
|---|---|---|
VPU |
1 X Intel® Movidius™ Myriad™ X MA2485 |
8 X Intel® Movidius™ Myriad™ X MA2485 |
Connection |
USB 2.0/3.0 |
PCIe X4 |
Batch size |
1 |
1 |
Precision |
FP16 |
FP16 |
Number of concurrent inference requests |
4 |
32 |
Rated maximum TDP/socket in Watt |
2.5 |
|
CPU Price/socket on Feb 14, 2022, USD Prices may vary |
||
Host Computer |
Intel® Core™ i7 |
Intel® Core™ i5 |
Motherboard |
ASUS* Z370-A II |
Uzelinfo* / US-E1300 |
CPU |
Intel® Core™ i7-8700 CPU @ 3.20GHz |
Intel® Core™ i5-6600 CPU @ 3.30GHz |
Hyper Threading |
ON |
OFF |
Turbo Setting |
ON |
ON |
Memory |
4 x 16 GB DDR4 2666MHz |
2 x 16 GB DDR4 2400MHz |
Operating System |
Ubuntu* 20.04.3 LTS |
Ubuntu* 20.04.3 LTS |
Kernel Version |
5.0.0-23-generic |
5.0.0-23-generic |
BIOS Vendor |
American Megatrends Inc.* |
American Megatrends Inc.* |
BIOS Version |
411 |
5.12 |
BIOS Release |
September 21, 2018 |
September 21, 2018 |
Test Date |
March 17, 2022 |
March 17, 2022 |
For more detailed configuration descriptions, see Configuration Details.