Use a Green Screen in Javascript With Vonage Video

Published June 24, 2020 by Kevin Lewis

When creating a Vonage Video publisher, the stream can be sourced directly from a user camera, from a <video> element, or a HTML <canvas> element. Once pixels get drawn to the canvas, they can be easily manipulated before being used in a Video API session.

In this tutorial, you’ll learn how to remove a green screen and replace it with a new, custom image that you can include in your video calls.

Overview of the project components

Several components are required to make the project work. Firstly, a <video> element will take a stream from the user’s camera. Each frame, the video element content will be drawn on a canvas, where we will loop through pixels to remove those which are green. On a second canvas, we will draw the replacement background image and then layer the first canvas’ non-green pixels on top.

With our desired output on a canvas, we can use the canvas as a source for a Vonage Video API publisher, which we can use in our video sessions with friends.

If you want to look at the finished code, you can find it at https://github.com/nexmo-community/video-green-screen

Scaffold Markup

Create a new project folder followed by a new file index.html, populating this file with the following code:

You’ll also need the image that you want to replace your green screen within the project folder. This tutorial will use one of the Vonage brand gradients. After you get the canvas contexts, load the image:

Get Webcam Video

Set the <video> element’s source to the stream from the user’s webcam. This snippet will pick the default camera:

Create an empty process() function. Once the user’s video device is ready and ‘playing’, run the function every frame:

Draw Video Stream to the Canvas

Update process():

Refresh your page, and you should see your <video> element, your first <canvas> with a duplicate image, and your second <canvas> with the new background image. The goal is to get any non-green pixels on top of the background in the second canvas.

Loop Through Pixels

In a canvas, the entire image represented in a single long array of pixels. While you may initially believe that our 320×240 image will have 76,800 entries in the array, you’d be mistaken.

Each visible pixel is made up of four array items – one for its red value, one for green, one for blue, and the final to set its opacity. These values are important as we build and use the loop.

Get this frame’s pixels array inside of the process() function, and build the loop:

Notice that the counter is set to increment by 4. Each time this loop runs, i will be the array index of the next visible pixel’s red value.

Understanding HSL

Before green pixels can be removed, I’d like to introduce you to the Hue Saturation Lightness (HSL) color format.

Hue is a color wheel, sauturation is the amount of grey, lightness is a scale of black to white

You can think of hue as a color wheel – and use the position on the wheel to specify a color, from 0 to 360. The green ‘range’ might be different for each person, but 90 to 200 works well for me.

However, when reading and writing pixels to a <canvas> you must use the Red Green Blue (RGB) color format. At the very bottom of your <script>, add this RGBToHSL() function provided on CSS Tricks:

Making Green Pixels Transparent

Inside the process() loop, get the RGB and HSL values for each pixel, and set pixels which are green to be transparent:

After the loop, update the canvas image:

You may find that 90 and 200 needs updating, given the color of your screen and lighting.

The first canvas has no background - appearing white

Draw Remaining Pixels on Replacement Background

For each pixel that is remaining in the first canvas, draw it on the second. After the if statement in the process() loop, add an else condition:

The x and y values are the visual pixels, so the i value should be divided by 4.

The second canvas now has the non-removed pixels

Include Canvas in Video API Session

Create a new project in your Vonage Video Dashboard. Once created, scroll down to Project Tools and create a new Routed session. Take the Session ID and create a new token.

At the top of your <script>, create three new variables with data from the project dashboard:

Next, copy the <script> tag from the Vonage Video API Client SDK page and put it above your existing <script> tag.

At the bottom of your <script> tag, get your basic Vonage Video API session initialized and publish from the second canvas:

Hide Elements

The <video> and <canvas> elements are required to make this work, but you probably don’t want them visible in your webpage. In your <head>, add the following CSS to hide them:

What Will Your Background Be?

Hopefully, you found this blog post useful and can now create custom backgrounds to your heart’s content. While we focused on greenscreens, any pixel-level manipulation can be done with the same approach.

To take this further, you may choose to provide users with controls that alter the HSL values which are ‘in range’ to be replaced, or a file selector to change the image.

You can find the final project at https://github.com/nexmo-community/video-green-screen

As ever, if you need any support feel free to reach out in the Vonage Developer Community Slack. We hope to see you there.

Leave a Reply

Your email address will not be published.

Get the latest posts from Nexmo’s next-generation communications blog delivered to your inbox.

By signing up to our communications blog, you accept our privacy policy , which sets out how we use your data and the rights you have in respect of your data. You can opt out of receiving our updates by clicking the unsubscribe link in the email or by emailing us at [email protected].