DEAL WITH IT in Python with Face Detection

DEAL WITH IT is a meme where glasses fly in from off the screen, and on to a user's face. The best instances of this meme do so in a unique way.

Today we'll write an automatic meme generator, using any static image with faces as our input. This code makes a great starting point for a meme API, or building your own animated version using video input.

I got the idea for this post from Erik Taheri, and his in browser Javscript version of this effect.

[embedded content]

At the end of the article as a bonus, I've also included a version where the effect can be done in real time from a webcam, using OpenCV.

The Tools of Face Detection and Gif Creation

Tools of Face Detection and Gif Creation

We'll use Dlib's get_frontal_face_detector, along with the 68 point shape prediction model we used in the Snapchat Lens article.

Our program will take in a command line argument, the input image. It will then use the face detection algorithm in Dlib to see if there are any faces. If there are, it will create an end position for each face, where the glasses should end up.

We'll need to then scale and rotate our glasses to fit each person's face. We'll use the set of points returned from Dlib's 68 point model to find a center of mass for the eyes, and a rotation for the space between them.

After we've found a final position and rotation for the glasses, we can then animate a gif, with the glasses coming in from the top of the screen. We'll draw this using MoviePy and a make_frame function.

Architecture of Automatic Gifs

Automatic Gif Architecture

The architecture of the application is fairly straightforward. We first take in an image, and then convert it to a grayscale NumPy array. Once we've got that, we can then pass our detected faces into our face orientation prediction model.

With the returned orientation of the face, we can then select out the eyes, and scale and rotate our glasses frames to fit the person's face.

We can accumulate a set of faces and their final positions, appending them to a list.

Finally, with this list we can then create a draw routine using MoviePy that then generates our animated gif.

Writing the Code

With our code architecture planned out, the next thing we need to do is build our code step by step.

We'll first import all our tools, and take in an image from the command line:

import dlib from PIL import Image import argparse from imutils import face_utils import numpy as np import moviepy.editor as mpy parser = argparse.ArgumentParser() parser.add_argument("-image", required=True, help="path to input image") args = parser.parse_args()

With this in place, we can then resize our images to fit a smaller width, so our gifs don't turn out massive, and import our face detector and shape predictors.

We can also open the glasses and text we'll be pasting on to our image.

At this point, we should also detect if there are even any faces detected in the image. If not, we should exit immediately.

detector = dlib.get_frontal_face_detector() predictor = dlib.shape_predictor('shape_predictor_68.dat') # resize to a max_width to keep gif size small max_width = 500 # open our image, convert to rgba img = Image.open(args.image).convert('RGBA') # two images we'll need, glasses and deal with it text deal = Image.open("deals.png") text = Image.open('text.png') if img.size[0] > max_width: scaled_height = int(max_width * img.size[1] / img.size[0]) img.thumbnail((max_width, scaled_height)) img_gray = np.array(img.convert('L')) # need grayscale for dlib face detection rects = detector(img_gray, 0) if len(rects) == 0: print("No faces found, exiting.") exit() print("%i faces found in source image. processing into gif now." % len(rects))

Great! Now we can loop over each of the faces detected, and build up a list of scaled and rotated glasses, along with their final positions.

faces = [] for rect in rects: face = {} print(rect.top(), rect.right(), rect.bottom(), rect.left()) shades_width = rect.right() - rect.left() # predictor used to detect orientation in place where current face is shape = predictor(img_gray, rect) shape = face_utils.shape_to_np(shape) # grab the outlines of each eye from the input image leftEye = shape[36:42] rightEye = shape[42:48] # compute the center of mass for each eye leftEyeCenter = leftEye.mean(axis=0).astype("int") rightEyeCenter = rightEye.mean(axis=0).astype("int") # compute the angle between the eye centroids dY = leftEyeCenter[1] - rightEyeCenter[1] dX = leftEyeCenter[0] - rightEyeCenter[0] angle = np.rad2deg(np.arctan2(dY, dX)) # resize glasses to fit face width current_deal = deal.resize((shades_width, int(shades_width * deal.size[1] / deal.size[0])), resample=Image.LANCZOS) # rotate and flip to fit eye centers current_deal = current_deal.rotate(angle, expand=True) current_deal = current_deal.transpose(Image.FLIP_TOP_BOTTOM) # add the scaled image to a list, shift the final position to the # left of the leftmost eye face['glasses_image'] = current_deal left_eye_x = leftEye[0,0] - shades_width // 4 left_eye_y = leftEye[0,1] - shades_width // 6 face['final_pos'] = (left_eye_x, left_eye_y) faces.append(face)

With our final positions in place along with our scaled and rotated glasses, we can then put together our movie. We'll set a duration for the whole gif, along with a time to stop the glasses coming down, so we can put the deal with it text on the screen.

# how long our gif should be duration = 4 def make_frame(t): draw_img = img.convert('RGBA') # returns copy of original image if t == 0: # no glasses first image return np.asarray(draw_img) for face in faces: if t <= duration - 2: # leave 2 seconds for text current_x = int(face['final_pos'][0]) # start from proper x current_y = int(face['final_pos'][1] * t / (duration - 2)) # move to position w/ 2 secs to spare draw_img.paste(face['glasses_image'], (current_x, current_y) , face['glasses_image']) else: # draw the text for last 2 seconds draw_img.paste(face['glasses_image'], face['final_pos'], face['glasses_image']) draw_img.paste(text, (75, draw_img.height // 2 - 32), text) return np.asarray(draw_img)

You'll notice I used an image for the text overlay, and not Pillow's built in Text drawing functions. I did this because Pillow doesn't have a built in Stroke function for text. Without the stroke, the text becomes illegible on brighter images.

Finally, we need to create a VideoClip object in MoviePy, and pass in our animation generating frame along with a fps.

animation = mpy.VideoClip(make_frame, duration=duration) animation.write_gif("deal.gif", fps=4)

With this, we're done!

Animating Memes In Real Time

Automatic Dealing with It

Now that we've got a base set for generating our gifs, it isn't too difficult to adapt our code to work with the webcam in real time.

Instead of loading our source image from the command line, we can use OpenCV as our source images, and keep track of the animations with a counter. The new code to do this is pretty straightforward, the real meat being:

# I got lazy, didn't want to bother with transparent pngs in opencv # this is probably slower than it should be if dealing: if current_animation < glasses_on: current_y = int(current_animation / glasses_on * left_eye_y) img.paste(current_deal, (left_eye_x, current_y), current_deal) else: img.paste(current_deal, (left_eye_x, left_eye_y), current_deal) img.paste(text, (75, img.height // 2 - 32), text) if dealing: current_animation += 1 if current_animation > animation_length: dealing = False current_animation = 0 else: frame = cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR) cv2.imshow("deal generator", frame) key = cv2.waitKey(1) & 0xFF if key == ord("q"): break if key == ord("d"): dealing = not dealing

This creates a counter, and steps through how many frames have progressed to keep track of time. Using this, we can then animate our glasses to their final position. We see if the user presses the d key, to deal with it, and when they do, we begin our animation.

You can read the rest of the code for this at Github.

Where To Go From Here

We've successfully built the very first part in a program that could be used as an API to generate memes automatically.

By hooking up our program to something like Flask, we could display a web page allowing for users to upload their own images, and get back fully complete memes.

With something like youtube-dl, we could have users paste in YouTube url videos to automatically generate memes.

In case you missed it, the code is available at Github.

If you're interested in more projects like these, please sign up an account on Make Art with Python. You'll get the first three chapters of my new book free when you do, and you'll help me continue making projects like these.

Let's block ads! (Why?)