Deblur Photos with DeblurGAN-v2 and OpenVINO¶

This tutorial demonstrates Single Image Motion Deblurring with DeblurGAN-v2 in OpenVINO, by first converting the VITA-Group/DeblurGANv2 model to OpenVINO’s Intermediate Representation (IR) format. Model information can be found here.

What is deblurring?¶

Deblurring is the task of removing motion blurs that usually occur in photos shot with hand-held cameras when there are moving objects in the scene. Blurs not only reduce the human perception about the quality of the image, but also complicate computer vision analyses.

Kupyn, O., Martyniuk, T., Wu, J., & Wang, Z. (2019). Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8878-8887).

Preparations¶

Imports¶

import json
import shutil
import sys
from pathlib import Path

import cv2
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
from IPython.display import Markdown, display
from openvino.runtime import Core

sys.path.append("../utils")
from notebook_utils import load_image

Settings¶

# Device to use for inference. For example, "CPU", or "GPU"
DEVICE = "CPU"

# Directory to save the model in
model_dir = Path("model")
model_dir.mkdir(exist_ok=True)

# Output directory
output_dir = Path("output")
output_dir.mkdir(exist_ok=True)

# The name of the Open Model Zoo model
model_name = "deblurgan-v2"
model_xml_path = model_dir / Path(f"{model_name}.xml")

precision = "FP16"
base_model_dir = Path("~/open_model_zoo_models").expanduser()
omz_cache_dir = Path("~/open_model_zoo_cache").expanduser()

Download DeblurGAN-v2 Model from Open Model Zoo¶

download_command = (
    f"omz_downloader --name {model_name} --output_dir"
    f" {base_model_dir} --cache_dir {omz_cache_dir}"
)
display(Markdown(f"Download command: `{download_command}`"))
display(Markdown(f"Downloading {model_name}..."))
! $download_command

Download command: omz_downloader --name deblurgan-v2 --output_dir /home/runner/open_model_zoo_models --cache_dir /home/runner/open_model_zoo_cache

Downloading deblurgan-v2…

################|| Downloading deblurgan-v2 ||################

========== Downloading /home/runner/open_model_zoo_models/public/deblurgan-v2/models/__init__.py

========== Downloading /home/runner/open_model_zoo_models/public/deblurgan-v2/models/fpn_mobilenet.py

========== Downloading /home/runner/open_model_zoo_models/public/deblurgan-v2/models/mobilenet_v2.py

========== Downloading /home/runner/open_model_zoo_models/public/deblurgan-v2/models/networks.py

========== Downloading /home/runner/open_model_zoo_models/public/deblurgan-v2/ckpt/fpn_mobilenet.h5

========== Replacing text in /home/runner/open_model_zoo_models/public/deblurgan-v2/models/networks.py
========== Replacing text in /home/runner/open_model_zoo_models/public/deblurgan-v2/models/fpn_mobilenet.py
========== Replacing text in /home/runner/open_model_zoo_models/public/deblurgan-v2/models/fpn_mobilenet.py

Convert DeblurGAN-v2 Model to OpenVINO IR format¶

Model Conversion may take a while. Conversion succeeded if the last lines of the output include [ SUCCESS ] Generated IR version 11 model.

convert_command = (
    f"omz_converter --name {model_name} --precisions {precision} "
    f"--download_dir {base_model_dir} --output_dir {base_model_dir}"
)
display(Markdown(f"Convert command: `{convert_command}`"))
display(Markdown(f"Converting {model_name}..."))

! $convert_command

Convert command: omz_converter --name deblurgan-v2 --precisions FP16 --download_dir /home/runner/open_model_zoo_models --output_dir /home/runner/open_model_zoo_models

Converting deblurgan-v2…

========== Converting deblurgan-v2 to ONNX
Conversion to ONNX command: /opt/hostedtoolcache/Python/3.8.12/x64/bin/python -- /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-path=/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/openvino/model_zoo/models/public/deblurgan-v2 --model-path=/home/runner/open_model_zoo_models/public/deblurgan-v2 --model-name=DeblurV2 --import-module=model --input-shape=1,3,736,1312 --output-file=/home/runner/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx '--model-param=weights=r"/home/runner/open_model_zoo_models/public/deblurgan-v2/ckpt/fpn_mobilenet.h5"' '--model-param=model_name="fpn_mobilenet"' --input-names=blur_image --output-names=deblur_image

ONNX check passed successfully.

========== Converting deblurgan-v2 to IR (FP16)
Conversion command: /opt/hostedtoolcache/Python/3.8.12/x64/bin/python -- /opt/hostedtoolcache/Python/3.8.12/x64/bin/mo --framework=onnx --data_type=FP16 --output_dir=/home/runner/open_model_zoo_models/public/deblurgan-v2/FP16 --model_name=deblurgan-v2 --input=blur_image --model_name=deblurgan-v2 --reverse_input_channels '--mean_values=blur_image[127.5]' '--scale_values=blur_image[127.5]' --output=deblur_image --input_model=/home/runner/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx '--layout=blur_image(NCHW)' '--input_shape=[1, 3, 736, 1312]'

Model Optimizer arguments:
Common parameters:
    - Path to the Input Model:  /home/runner/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx
    - Path for generated IR:    /home/runner/open_model_zoo_models/public/deblurgan-v2/FP16
    - IR output name:   deblurgan-v2
    - Log level:    ERROR
    - Batch:    Not specified, inherited from the model
    - Input layers:     blur_image
    - Output layers:    deblur_image
    - Input shapes:     [1, 3, 736, 1312]
    - Source layout:    Not specified
    - Target layout:    Not specified
    - Layout:   blur_image(NCHW)
    - Mean values:  blur_image[127.5]
    - Scale values:     blur_image[127.5]
    - Scale factor:     Not specified
    - Precision of IR:  FP16
    - Enable fusing:    True
    - User transformations:     Not specified
    - Reverse input channels:   True
    - Enable IR generation for fixed input shape:   False
    - Use the transformations config file:  None
Advanced parameters:
    - Force the usage of legacy Frontend of Model Optimizer for model conversion into IR:   False
    - Force the usage of new Frontend of Model Optimizer for model conversion into IR:  False
OpenVINO runtime found in:  /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/openvino
OpenVINO runtime version:   2022.1.0-7019-cdb9bec7210-releases/2022/1
Model Optimizer version:    2022.1.0-7019-cdb9bec7210-releases/2022/1
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /home/runner/open_model_zoo_models/public/deblurgan-v2/FP16/deblurgan-v2.xml
[ SUCCESS ] BIN file: /home/runner/open_model_zoo_models/public/deblurgan-v2/FP16/deblurgan-v2.bin
[ SUCCESS ] Total execution time: 1.35 seconds.
[ SUCCESS ] Memory consumed: 129 MB.
It's been a while, check for a new version of Intel(R) Distribution of OpenVINO(TM) toolkit here https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit/download.html?cid=other&source=prod&campid=ww_2022_bu_IOTG_OpenVINO-2022-1&content=upg_all&medium=organic or on the GitHub*
[ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11.
Find more information about API v2.0 and IR v11 at https://docs.openvino.ai

Copy Model¶

Copy the DeblurGAN-v2 Model to the specified model_dir directory in the settings.

model_info = %sx omz_info_dumper --name $model_name

model_info_json = json.loads(model_info.get_nlstr())[0]

model_downloaded_dir = (
    base_model_dir / Path(model_info_json["subdirectory"]) / Path(precision)
)

for model_file in model_downloaded_dir.iterdir():
    try:
        shutil.copyfile(model_file, model_dir / model_file.name)
    except FileExistsError:
        pass

Load the Model¶

Load and compile the DeblurGAN-v2 model in the Inference Engine with ie.read_model and compile it for the specified device with ie.compile_model. Get input and output keys and the expected input shape for the model.

ie = Core()
model = ie.read_model(model=model_xml_path)
compiled_model = ie.compile_model(model=model, device_name=DEVICE)

model_input_layer = compiled_model.input(0)
model_output_layer = compiled_model.output(0)

model_input_layer

<ConstOutput: names[blur_image] shape{1,3,736,1312} type: f32>

model_output_layer

<ConstOutput: names[deblur_image] shape{1,3,736,1312} type: f32>

Deblur Image¶

Load, resize and reshape input image¶

The input image is read using the default load_image function from notebooks.utils, resized to meet the network’s expected input sizes, and reshaped to (N, C, H, W), where N=number of images in the batch, C=number of channels, H=height, and W=width.

# Image filename (local path or URL)
filename = "https://user-images.githubusercontent.com/41332813/166901955-5d813e4c-a895-4da2-a36c-a96b3f627ebb.png"

# Load input image
image = load_image(filename)

# In case immage is stored in RGBA format
image = np.asarray(Image.fromarray(image).convert('RGB'))

# N,C,H,W = batch size, number of channels, height, width
N, C, H, W = model_input_layer.shape

# Resize image to meet network expected input sizes
resized_image = cv2.resize(image, (W, H))

# Reshape to network input shape
input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0)

plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

<matplotlib.image.AxesImage at 0x7fa102a79df0>

../_images/217-vision-deblur-with-output_24_1.png

Do inference on input image¶

Do the inference, convert the result to an image shape and resize it to the original image size.

# Inference
result = compiled_model([input_image])[model_output_layer]

# Convert result to an image shape
result_image = result[0].transpose((1, 2, 0))

# Resize to original image size
resized_result_image = cv2.resize(result_image, image.shape[:2][::-1])

plt.imshow(cv2.cvtColor(result_image, cv2.COLOR_BGR2RGB))

<matplotlib.image.AxesImage at 0x7fa10096e310>

../_images/217-vision-deblur-with-output_27_1.png

Load groundtruth image¶

Load the groundtruth image for comparison.

# Load grountruth image
groundtruth_image = load_image("https://user-images.githubusercontent.com/41332813/166900611-83be8ae5-23ed-4426-b535-fad960795687.png")

Display results¶

Images are in BGR format, so they will be converted to the RGB format in order to be properly displayed by matplotlib.

# Create subplot(r,c) by providing the no. of rows (r),
# number of columns (c) and figure size
f, ax = plt.subplots(1, 3, figsize=(20, 15))

# Use the created array and display the images horizontally.
ax[0].set_title("Blurred")
ax[0].imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

ax[1].set_title("DeblurGAN-v2")
ax[1].imshow(cv2.cvtColor(resized_result_image, cv2.COLOR_BGR2RGB))

ax[2].set_title("Groundtruth")
ax[2].imshow(cv2.cvtColor(groundtruth_image, cv2.COLOR_BGR2RGB))

<matplotlib.image.AxesImage at 0x7fa10089e610>

../_images/217-vision-deblur-with-output_31_1.png

Save the deblurred image¶

Save the output image of the DeblurGAN-v2 model in the output_dir directory.

savename = "deblurred.png"
plt.imsave(
    output_dir / Path(savename),
    cv2.cvtColor(resized_result_image, cv2.COLOR_BGR2RGB),
)

Prev Next