I See What You’re Saying: Sentiment Analysis With Azure Face API and Vonage

Published January 27, 2020 by Michael Jolley

You know that person. It could be your significant other, a child, a co-worker, or a friend. The person that says one thing, but you can tell by their face, they mean something completely different. You probably just pictured them in your head. Maybe you remember the exact conversation. Perhaps it went like this:

You: Okay?

Them: Fine.

Spoiler Alert: It wasn’t fine.

Wouldn’t it be great if you could know the sentiment behind what they were saying? With Vonage Video API and Azure’s Face API you can!

In this tutorial, we will build a multi-party video conference that allows us to analyze the sentiment of each participant based on their facial expression. Then we’ll display that sentiment as an emoji over their video.

Prerequisites

Before getting started, you’ll need a few things:

Want to skip to the end? You can find all the source code for this tutorial on GitHub.

Getting Started

We’ll use JavaScript to do the heavy lifting, so let’s get the HTML & CSS out of the way.

In the root of the video-sentiment folder, create an index.html file and copy the following to it.

Next, create a css directory and add an app.css file to it. Copy the CSS below into that file.

Let’s Get Configured

Great! Now we can make that pretty HTML & CSS do something. Create a js folder and add a config.js file.

The config.js file contains configuration parameters that we’ll get from our Vonage Video API and Azure accounts. Copy the following into the config.js file.

OpenTok Settings

We’ll get the OPENTOK_API_KEY, OPENTOK_SESSION_ID and OPENTOK_TOKEN variables from our Vonage Video API Account.

In your Vonage Video API Account, click the ‘Projects’ menu and ‘Create New Project.` Then click the ‘Create Custom Project’ button. Give your new project a name and press the ‘Create’ button. You can leave the preferred codec as ‘VP8’.

Screenshot of the "project created" dialog within a Vonage Video API account.

You can then copy your API Key and paste it as the value for the OPENTOK_API_KEY setting.

Next, click on “View Project”. At the bottom of the project detail page, you’ll find the Project Tools where you can create a Session ID and Token. Choose “Routed” for your session’s media mode and press the “Create Session ID” button. Then, copy the generated Session ID and paste it as the value of the OPENTOK_SESSION_ID setting.

Finally, paste the generated session ID into the Session ID field of the Generate Token form and press the “Generate Token” button. Copy the generated Token as the value of the OPENTOK_TOKEN setting.

Project Tools area of a specific project in a Vonage Video API account.

Azure Face API Settings

Log into your Azure account and create a new Face API Cognitive Service. Once created, click on the service and go to the “Quick start” blade. There you’ll find your Key and Endpoint. Copy these two values to the AZURE_FACE_API_SUBSCRIPTION_KEY and AZURE_FACE_API_ENDPOINT settings, respectively.

Screenshot of the Quick start blade within Azure for the Face API service.

I Feel Seen

With our configuration ready, let’s add some JavaScript to connect to an OpenTok session. Add an app.js file to the js folder and copy the following to it.

Four things are going on here:

  1. We load variables based on those we specified in the config.js file
  2. We create a handleError method that we’ll use throughout when an error occurs
  3. We add a dataURItoBlob method that we’ll use to convert a base64/URLEncoded image to a blob for sending to Azure Face API
  4. We added two arrays named streams and emotions

The streams array will hold all active participant streams so we can access them to capture images to send to the Azure Face API.

The emotions array will hold strings that represent any emotions returned by Azure Face API. This will be used to display a legend of emojis to the user dynamically.

Initializing the OpenTok Session

Add the initializeSession method below to the bottom of the app.js file.

The initializeSession method initializes our Vonage Video API client with the session we specified with the Session ID. It then adds event handlers for the streamCreated and streamDestroyed events to manage adding and removing streams from our streams array. Finally, it connects to the session using the Token we set in our config.js file.

You can now open the index.html in Chrome or Firefox. When you load the page, you may need to allow the browser to access your webcam and microphone. After that, you should see a video stream of yourself (or whatever your webcam is looking at) displaying on the page.

If that worked, mute your audio then open another tab (keeping the original open) and load the same file. You should now be able to see a second video.

Troubleshooting tip: If there’s no video showing up on the page, open the “console” tab in your browser tools (command+option+i on Mac, CTRL+i on Windows) and check for errors. The most likely issue is that your Vonage Video API key, session ID, or token is not set up properly. Since you hardcoded your credentials, it’s also possible that your token has expired.

I Know That Look

Now we can see and hear participants, but what is their face telling us that their mouth isn’t? Let’s add a button that allows us to analyze each participant.

In the index.html file, replace the comment that says <!-- Footer will go here --> with the following:

This adds a footer at the bottom of the page with an “Analyze” button and an unordered list that we’ll use as a legend between emojis and sentiments.

Now let’s add the JavaScript to handle our sentiment analysis. Add the following to the bottom of the app.js file.

Let's review what this code does.

The assignEmoji method takes in a CSS class associated with the emotion for a specific video stream and the index of that stream in our UI. It does the following:

  1. Adds the provided class to our emotions array
  2. Adds a div over the appropriate video panel with the class for the emoji to display
  3. Adds a used class to the li in our footer for that emoji so that it will display in the legend

The processEmotion method receives the payload of face data from the Azure Face API and identifies the emotion with the highest ranking. It then calls assignEmoji with the appropriate CSS class for that emotion and the index of the video it is processing.

The sendToAzure method receives an HTML video element and the index of that video object on our page. It gets the stream associated with that video element and then creates an HTML canvas in the same dimensions as the stream. Next, it draws a capture of the stream to the new canvas and sends an XMLHttpRequest to the Azure Face API with the image it created. The Azure Face API will return a JSON object that we will then send to the processEmotion method.

Lastly, the processImages method clears any existing emojis from the UI and gets all HTML video tags in the DOM and sends them to the sendToAzure method to be processed. This method is called by our "Analyze" button in the footer.

What Are You Really Thinking?

Now when we open the index.html page in our browsers we can press the "Analyze" button to see what emotion Azure's Face API has identified. There are a few limitations at the moment. For instance, if Azure Face API recognizes two faces in the frame it will return data for both, but our code currently only adds an emoji for the first.

Also, I'm not certain, but it may not work for teenagers. I made my teenage daughter test it dozens of times but it only returned "disgust" and "contempt" as the emotions. Maybe this wasn't such a good idea. Maybe it's better to not know what they really think. 😂

Four video frames of Michael showing different sentiments returned from Azure's Face API

Further Reading

Want to learn more about using sentiment analysis with Nexmo? Check out the following blog posts:

Leave a Reply

Your email address will not be published.

Get the latest posts from Nexmo’s next-generation communications blog delivered to your inbox.

By signing up to our communications blog, you accept our privacy policy , which sets out how we use your data and the rights you have in respect of your data. You can opt out of receiving our updates by clicking the unsubscribe link in the email or by emailing us at [email protected].