<div class="Objective_card card-content-body"><p>You will be creating a web application which will perform real time video processing of a webcam video stream consisting of a green screen and replace it with a background video or an image of your choice.</p></div>

Objective

<div class="Project_Context_card card-content-body"><p>Have you ever seen a movie in which a superhero flies through the sky? You can see all the skyscrapers whizzing by in the background, and it looks so cool. Have you ever WONDERed how movie makers film such shots?
<br/>
<br/></p><p>The special effects created during weather forecasts and many, many television shows and movies utilize a special tool called a green screen. Why is it called a green screen? Mainly because it consists of a large screen that is green! The green screen is an integral part of the special effects process known formally as chromakey.
<br/>
<br/></p><p>Chromakeying, sometimes known as color keying, is the process of singling out a particular color in an electronic image and then using computer software to make that color transparent. This allows another image, which can be just about anything you can imagine, to show through.
<br/>
<br/></p><p>This project is a good start for beginners, a new idea for intermediates and a refreshing hobby project for professionals. It involves the basic use of all of HTML, CSS and JS languages. Reason for following this tech-stack is that these languages are easy to use and are also fast in terms of execution time.
<br/></p></div>

Project Context

<div class="Project_Stages_card card-content-body"><p>The project consists of the following stages:</p><p><img alt="project_stages" src="https://storage.googleapis.com/crio-content-container-assets/ME_ME_PROJECT_CHROMAKEY_MODULE_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_flowchart.png" style="max-width:750px;"/></p></div>

Project Stages

<div class="High-Level_Approach_card card-content-body"><ul>
<li>Capture input video stream with green background.</li>
<li>Refine the captured stream.</li>
<li>Merge the processed input stream with another background video or image of your choice.</li>
</ul><p>The desired end result of this project is like this:
<br/></p><p><div class="video"><iframe allowfullscreen="allowfullscreen" class="iframe-video" height="350px" mozallowfullscreen="mozallowfullscreen" msallowfullscreen="msallowfullscreen" oallowfullscreen="oallowfullscreen" src="https://www.youtube.com/embed/wif1a3FcSdQ" webkitallowfullscreen="webkitallowfullscreen" width="600px"></iframe></div></p></div>

High-Level Approach

<div class="Applications_card card-content-body"><p>Some of the applications of Green Screen Video Processing are:-</p><ul>
<li>Personalized background for video meeting applications like Zoom</li>
<li>An important technique utilized for heavy computer graphics in film industry.</li>
</ul></div>

Applications

<div class="Credits_card card-content-body"><p><a href="https://redstapler.co/" target="_blank">Red Stapler</a></p></div>

Credits

<div class="Getting_Started_card card-content-body"><p>We'll start of by initializing the project structure.</p></div>

Getting Started

<div class="Requirements_card card-content-body"><ul>
<li>Install Node.js.</li>
<li>Create a project folder with appropriate name and inside this directory run <code>npm init</code> and fill all the require details in the prompt.</li>
<li>Create the essential files required as mentioned below.</li>
</ul><p><img alt="required_file_structure" src="https://storage.googleapis.com/crio-content-container-assets/ME_ME_PROJECT_CHROMAKEY_MODULE_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_file-structure.png" style="max-width:750px;"/></p><div class="References"><h3>References</h3><ul>
<li>Check out Crio’s HTML and CSS <a href="https://learn.crio.do/home/me/ME_HTML_CSS" target="_blank">byte</a> to get yourself fully equipped with HTML/CSS.</li>
<li>Don’t know what the <code>.gitignore</code> file is? No problem, just refer to <a href="https://www.pluralsight.com/guides/how-to-use-gitignore-file" target="_blank">this</a>.</li>
</ul></div><div class="Tip"><h3>Tip</h3><ul>
<li>Since this is a direct JavaScript based application, running the HTML file will suffice to view the webpage. But while developing the project you will have to reload each time you make a change in your code to view the corresponding changes.</li>
<li>Download <a href="https://marketplace.visualstudio.com/items?itemName=ritwickdey.LiveServer" target="_blank">Live Server</a> extension for <a href="https://code.visualstudio.com/download" target="_blank">VSCode</a> users, <a href="https://atom.io/packages/atom-live-server" target="_blank">Atom-live-server</a> for atom users. For others the best ways are either using npm (<a href="https://medium.com/@svinkle/start-a-local-live-reload-web-server-with-one-command-72f99bc6e855" target="_blank">article</a> and <a href="https://www.npmjs.com/package/live-server" target="_blank">package</a>) or use <a href="https://threejs.org/docs/#manual/en/introduction/How-to-run-things-locally" target="_blank">python server</a>. Other ways do exist. Feel free to explore them and implement if needed.</li>
</ul></div></div>

Requirements

<div class="Update_the_required_Skeleton_Project_files_card card-content-body"><p>Now we can implement the various components of our project.</p><p>In this milestone, a basic template of some code will be required in the project files.</p></div>

Update the required Skeleton Project files

<div class="Requirements_card card-content-body"><ul>
<li>Let’s start simple, with a bare-bones HTML5 code template in a <code>index.html</code> file with all the CSS and JS files imported  present in the project folder.</li>
<li>The HTML document must contain two <code>&lt;canvas&gt;</code> elements with the IDs <code>c1</code> and <code>c2</code> and one <code>&lt;video&gt;</code> element with ID <code>v1</code>. The given IDs for above three HTML elements will be used by JavaScript methods to access and interact with them in the further stages.</li>
<li>The <code>index.js</code> file must contain references to the video and canvas elements along with <code>Canvas 2D Context</code> for the canvas <code>c1</code> and <code>c2</code>.</li>
<li>A background image or video of your choice must be programmatically created using JavaScript which will be required further to replace the green pixels in the input video stream.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://www.w3schools.com/css/css_templates.asp" target="_blank">HTML CSS Basic Template</a></li>
<li><a href="https://www.w3schools.com/html/html5_canvas.asp" target="_blank">HTML Canvas</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/getContext" target="_blank">HTMLCanvasElement.getContext()</a></li>
<li><a href="https://www.geeksforgeeks.org/how-to-create-an-image-element-dynamically-using-javascript/" target="_blank">How to create an image element dynamically using JavaScript ?</a></li>
</ul></div></div>

<div class="Expected_Outcome_card card-content-body"><p>A required skeleton document with HTML elements is created to display three different video streams present at each different stages in the video processing pipeline.
<br/></p></div>

Expected Outcome

<div class="Capture_Webcam_Video_Stream_and_Draw_it_to_Canvas_card card-content-body"><p>In this milestone, we will capture the input webcam video stream and display it on the <code>&lt;video&gt;</code> and 1st <code>&lt;canvas&gt;</code>.</p></div>

Capture Webcam Video Stream and Draw it to Canvas

<div class="Requirements_card card-content-body"><ul>
<li>Capture the <code>&lt;video&gt;</code> element's Webcam stream using <code>MediaDevices.getUserMedia()</code> which is available as a Web API under <code>MediaDevices</code> interface.</li>
<li>Create an empty function <code>computeFrame()</code> which will be used for frame by frame video processing.</li>
<li>Attach event listener with event type as "play" to <code>&lt;video&gt;</code> element with ID <code>v1</code> and pass second parameter as anonymous function enclosed of <code>setInterval()</code> method attached to <code>computeFrame()</code> function.</li>
<li>Add source code for drawing Video stream to Canvas Context <code>c1</code> and programmatically created background image or video of your choice to Canvas <code>c2</code> in <code>computeFrame</code> function.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia" target="_blank">MediaDevices.getUserMedia()</a></li>
<li><a href="https://www.w3schools.com/graphics/canvas_images.asp" target="_blank">Draw an image on a canvas</a></li>
</ul></div></div>

<div class="Expected_Outcome_card card-content-body"><p>After implementing the above requirements, video is displayed in the <code>&lt;video&gt;</code> element and canvas1, selected background image is displayed in the canvas2.</p><p><img alt="task_3_expected_outcome" src="https://storage.googleapis.com/crio-content-container-assets/ME_ME_PROJECT_CHROMAKEY_MODULE_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_task-3.png" style="max-width:750px;"/>
<br/></p></div>

<div class="Display_video_stream_with_removed_green_pixels_card card-content-body"><p>In this milestone, we will set the green pixels alpha value to transparent ( i.e zero ) and display in the 1st Canvas.</p></div>

Display video stream with removed green pixels

<div class="Requirements_card card-content-body"><p>In the computeFrame() function,</p><ul>
<li>Get frame from the Canvas Context <code>c1</code> and extract pixels from the <code>data</code> attribute of the frame.</li>
<li>Loop through the pixels where each pixel contains an array of <code>(r,g,b,a)</code> values and convert each pixel from RGB to HSL format.</li>
<li>Set the Alpha Value <code>a</code> of each pixel to zero if the Hue value lies in the range of Green color.</li>
<li>Re-assign the modified pixels to the original frame and display it in the Canvas Context <code>c1</code> at the end of the function outside the pixel manipulation loop.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://css-tricks.com/converting-color-spaces-in-javascript/#rgb-to-hsl" target="_blank">Convert RGB to HSL </a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Pixel_manipulation_with_canvas" target="_blank">Canvas Pixel Manipulation</a></li>
</ul></div></div>

<div class="Expected_Outcome_card card-content-body"><p>After implementing the above requirements, processed video with transparent pixels is displayed in the canvas1.</p><p><img alt="task_4_expected_outcome" src="https://storage.googleapis.com/crio-content-container-assets/ME_ME_PROJECT_CHROMAKEY_MODULE_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_task-4.png" style="max-width:750px;"/>
<br/></p></div>

<div class="Merge_Processed_Video_Stream_with_Personalized_Background_card card-content-body"><p>In this milestone, we will merge the processed video stream with a personalized background image or video and display it in the 2nd Canvas.</p></div>

Merge Processed Video Stream with Personalized Background

<div class="Requirements_card card-content-body"><p>In the Pixel Manipulation loop of computeFrame() function,</p><ul>
<li>Set the original color values of <code>rgba</code> for pixels from the frame using <code>fillStyle</code> attribute of the Canvas Context <code>c2</code> if the Hue value doesn't lies in the range of Green color.</li>
<li>Calculate the X co-ordinates of the pixel by the formula <code>((Current Pixel iteration value/ length of pixel array) % Width of Canvas c2)</code>.</li>
<li>Calculate the Y co-ordinates of the pixel by the formula <code>((Current Pixel iteration value / length of pixel array) / Width of Canvas c2)</code>.</li>
<li>Fill the original pixel located at the above calculated co-ordinates using <code>fillRect</code> method of Canvas Context.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Pixel_manipulation_with_canvas" target="_blank">Canvas Pixel Manipulation</a></li>
</ul></div></div>

<div class="Expected_Outcome_card card-content-body"><p>After implementing the above requirements, below is the processed video with personalized background.
<img alt="Output" src="https://storage.googleapis.com/crio-content-container-assets/ME_ME_PROJECT_CHROMAKEY_MODULE_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_output.png" style="max-width:750px;"/>
<br/></p></div>

<div class="Spice_it_up!_card card-content-body"><p>As the basic implementation is all done, for all the curious cats out there, these are some of the line items which can be implemented to spice up the existing functionality.
<br/>
[Note: This is not a mandatory milestone]
<br/></p></div>

Spice it up!

<div class="Requirements_card card-content-body"><ul>
<li>Try to do the following:
<ul>
<li>Can you add a toggle option to switch between different personalized backgrounds?</li>
<li>Can you add color filter functionality like grayscale,brighten,etc?</li>
<li>Can you add a functionality to support dynamic Background Screens other than Green ( eg. Blue Screen ) ?</li>
</ul>
</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://www.html5rocks.com/en/tutorials/canvas/imagefilters/" target="_blank">Image Filters with Canvas</a></li>
</ul></div></div>

<div class="Expected_Outcome_card card-content-body"><p>You should be able to add more features to your application.
<br/></p></div>

CONTAINER_REGULAR_TASK_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_TASK_1

Getting started

CONTAINER_REGULAR_TASK_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_TASK_2

Scaffold Project Files

CONTAINER_REGULAR_TASK_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_TASK_3

Capture and Draw WebCam Video

CONTAINER_REGULAR_TASK_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_TASK_4

CONTAINER_REGULAR_TASK_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_TASK_5

CONTAINER_REGULAR_TASK_ME_PROJECT_CHROMAKEY_MODULE_CHROMAKEY_TASK_6

Realtime Video Processing using ChromaKey (GreenScreen) Effect

Companion App: A mental health tracker built using Flutter and Firebase.

Exploratory Analysis of Geolocational Data

Learn to create apps on Kubernetes - for beginners.

OurApp - a social media web app in NodeJS

Visualising and forecasting stocks using Dash