Pixel-level Image and Video
Demo see below
When I searched the internet trying to find some inspirations for assignment this week, I suddenly saw the famous Andy-Warhol-style painting — the Marilyn Monroe portrait. So I thought, why not creating a Andy-Warhol-style video?
So I started to decompose how Andy Warhol created his portrait. His portrait had several features:
- Bottom layer: single color for background, single color for the person;
- Middle: bright color for hairs, eyes, and mouth;
- Top layer: black facial contour
Immediately I thought about resolving the first two layers using bodypix and face detection models. As for the third layer, I believed that I could use the pixel drawing I learnt in the class.
Firstly, I tried to painting the black contour. Inspired by a Photoshop tutorial to process a photo to Andy-Wharhol style, I used brightness as the threshold draw the contour. Brightness (averaged RGB): if (brightness < 100) fill(0) and draw rect. The contour code was as follows.
video.loadPixels();
for(var i = 0; i < cols; i++){
for (var j = 0; j < rows; j++){
var loc = ((cols - i - 1) + j * cols) * 4;
var r = video.pixels[loc ];
var g = video.pixels[loc + 1];
var b = video.pixels[loc + 2];
var sz = (r+g+b)/3;
if (sz < 80){ // set threshold
fill(0);
noStroke();
var x = i*videoScale;
var y = j*videoScale;
rect(width-x, y, videoScale, videoScale);
}
}
}
Secondly, I tried to add the two-tone background color under the contour. After trying several different ways, I still could not find a way to replace the palette of the maskBackground color from bodypix function in ml5.js. By default, the mask of bodypix was black and didn’t open options for us to change the color of mask, which made me upset. So I turned to create the face layer first. Using face api in ml5.js, I was quickly draw the two-layer portrait (contourFace.js).
Secondly, I tried to add the two-tone background color under the contour. After trying several different ways, I still could not find a way to replace the palette of the maskBackground color from bodypix function in ml5.js. By default, the mask of bodypix was black and didn’t open options for us to change the color of mask, which made me upset. So I turned to create the face layer first. Using face api in ml5.js, I was quickly draw the two-layer portrait (contourFace.js).
Next, I continued to find the solutions to replace the color fo mask in bodypix. I searched Joey’s library and further the original library of bodypix. Unfortunately, the segmentWithPart function did allow us to change the color of parts of the body BUT the background. I tried to use this, the effect, however, was not satisfying. The original bodypix git repo didn’t open the id == -1 part for us. And I couldn’t modify the source code in terms of my ability.
I began to think about the solution in another way. Since the bodypix gave a image output, can I used the pixel-level judgement to it? I set a threshold to distinguish the background and person. And it worked!
Lastly, I need to combine all the algos together. I tried two structures but both of them had drawbacks. At first, I put faceapi and bodypix in draw function and it would flicker frame by frame. Then I changed to start call back bodypix, and then call face.api in bodypix and in face api call bodypix again, which resulted in a recurrent function. The problem was that it always stuck and I don’t know why. So I turned to Joey for help. Joey was sooooooo nice! He wrote a structure for me and I realized that I shouldn’t draw parts of the bodypix layer in bodypix function and draw face in face.api function. Instead, I should restore the results and draw them all together. Thanks to Joey’s help, the code worked, although it ran super super slow because of its heavy computation !
p.s. I tried to use style-GAN to transfer the Andy-Warhol style portrait and failed. Here is the demo.