Controlling Lens blur

[harisreedhar]

Is it possible to control lens blur with a depth map?

[NatLee]

Do you mean a feature can process the input image with a depth map into a blur image?

[harisreedhar]

Yes. For example depth map generated using MIDAS to control blur intensity.

[NatLee]

It sounds like we need a new filter for generating blur on images.
Here I found a repo that use depth map to mix and control blur with original input image.
Maybe I can embed it into this tool after discussing with its author.

[harisreedhar]

Hi thanks for the repo. It worked with gaussian_blur. For some reason lens_blur not working it outputs black image, may be i did something wrong.

import os
import sys
import cv2
import numpy as np
from blurgenerator import lens_blur, gaussian_blur

def map_range(value, inMin, inMax, outMin, outMax):
    return outMin + (((value - inMin) / (inMax - inMin)) * (outMax - outMin))

def blur_with_depth(img, depth, num_layers=10, min_blur=1, max_blur=100):
    min_depth = np.min(np.unique(depth))
    max_depth = np.max(np.unique(depth))
    step = (max_depth - min_depth) // num_layers
    layers = np.array(range(min_depth, max_depth, step))
    out = np.zeros(img.shape)
    
    for i, value in enumerate(layers):
        dm = cv2.cvtColor(depth, cv2.COLOR_BGR2GRAY)
        m = np.zeros(dm.shape)
        m[dm > value] = 255
        m[dm > (value + step)] = 0
        
        l_mask = depth.copy()
        l_mask[:,:,0] = m[:,:]
        l_mask[:,:,1] = m[:,:]
        l_mask[:,:,2] = m[:,:]
        
        blur_amount = int(map_range(value, 0, 255, min_blur, max_blur))
        slice = gaussian_blur(img, blur_amount)
        #slice = lens_blur(img, radius=blur_amount)
        
        _, mask = cv2.threshold(l_mask, 100, 255, cv2.THRESH_BINARY)
        mask_inv = cv2.bitwise_not(mask)
        layer = cv2.bitwise_and(slice, slice, mask = mask[:,:,0])
        out = cv2.add(out, layer, dtype=0)
    
    h,w,c = out.shape
    ha = h*2 // 3
    wa = w*2 // 3
    out = cv2.resize(out, (wa,ha))

img = cv2.imread("input.jpg")
depth = cv2.imread("depth.png")
output = blur_with_depth(img, depth, num_layers=20, min_blur=1, max_blur=50) 
cv2.imwrite('output.jpg', output)

[NatLee]

Hi thank you for writing this script.
Can you share 1 pair of example images for testing?

[harisreedhar]

I tested with these images:

[NatLee]

Hi thanks for the repo. It worked with gaussian_blur. For some reason lens_blur not working it outputs black image, may be i did something wrong.

I tried the script and found that output was actually not black.

Cause the lens blur did preprocessing made img get be normalized from 255 to 1.

output *= 255 will get this below:

And when using script here:

import numpy as np
import cv2

from blurgenerator import lens_blur, gaussian_blur

def map_range(value, inMin, inMax, outMin, outMax):
    return outMin + (((value - inMin) / (inMax - inMin)) * (outMax - outMin))

def blur_with_depth(img, depth, num_layers=10, min_blur=1, max_blur=100):
    min_depth = np.min(np.unique(depth))
    max_depth = np.max(np.unique(depth))
    step = (max_depth - min_depth) // num_layers
    layers = np.array(range(min_depth, max_depth, step))
    out = np.zeros(img.shape)

    for idx, value in enumerate(layers):
        dm = cv2.cvtColor(depth, cv2.COLOR_BGR2GRAY)
        m = np.zeros(dm.shape)
        m[dm > value] = 255
        m[dm > (value + step)] = 0

        l_mask = depth.copy()
        l_mask[:,:,0] = m[:,:]
        l_mask[:,:,1] = m[:,:]
        l_mask[:,:,2] = m[:,:]

        blur_amount = int(map_range(value, 0, 255, min_blur, max_blur))
        #slice = gaussian_blur(img, blur_amount)
        slice = lens_blur(img/255., radius=blur_amount)

        _, mask = cv2.threshold(l_mask, 100, 255, cv2.THRESH_BINARY)
        mask_inv = cv2.bitwise_not(mask)
        layer = cv2.bitwise_and(slice, slice, mask = mask[:,:,0])
        out = cv2.add(out, layer, dtype=0)

    h,w,c = out.shape
    ha = h*2 // 3
    wa = w*2 // 3
    out = cv2.resize(out, (wa,ha))
    return out

img = cv2.imread("input.jpg")
depth = cv2.imread("depth.png")
output = blur_with_depth(img, depth, num_layers=2, min_blur=1, max_blur=50)
cv2.imwrite('output.jpg', output)

We'll generate blur on original image.

But the speed generating blur is very slowly.

I think the method of lens blur need to be optimized. -> https://github.com/NatLee/Blur-Generator/blob/main/src/blurgenerator/lens_blur.py#L118

Thank you for the script and images.

[harisreedhar]

So normalizing will solve that issue thanks.
You can improve lens blur speed quite a bit by using cv2.filter2D instead of scipy.signal.convolve2d. See gimp lens blur for reference.

[NatLee]

Okay, let me check it

[NatLee]

@harisreedhar I have already added this feature in the lastest release version. Please check it.
Thank you for your ideas. 😊