Introduction
It's no secret that many of us began to rely heavily on video conferencing during the first half of 2020. As a result, we saw an explosion of growth in webcam use that today has become an integral part of our society. In December 2020, Zoom reported more than 350 million daily users of their platform and around 3.3 trillion minutes in 2021. While we expect this high usage to drop off as we get further away from the pandemic, it is clear that video conferencing is here to stay.
What quickly became apparent during the growth was a lack of webcam systems that could consistently perform with high image quality. According to a 2019 study by Lifesize, 77% of video calls were made from a laptop or desktop computer. However, there was clearly a gap between image quality standards and test methods for webcams compared to other camera systems, e.g., mobile phones.
Over a decade ago, Skype began to develop an international standard for image quality performance. While they made some progress, the development was mostly halted once Microsoft bought them. As a result, there was little international agreement on webcam KPIs and test methods until the VCX-Forum began developing VCX-WebCam 2023. Since its inception, Image Engineering has contributed to the development of all the standards published by VCX.
VCX-WebCam 2023
VCX-WebCam 2023 introduces the first internationally recognized benchmark for video image quality. The VCX-Forum was born out of a desire to have mobile phone cameras be independently and objectively scored so that consumers could find the best quality camera for their needs. Since 2018, the VCX-PhoneCam standard has become internationally recognized for mobile phone performance and quality. As a result of the success of the mobile phone standard, VCX was approached with the idea of creating a video camera quality standard based on the principles of VCX.
There are many examples of camera systems being graded using basic pass/fail test methods. However, these types of tests do not observe trends across current device markets or consider the recent performance expectations for these market segments in correlation to subjective image quality publications.
A perfect example of this problem is webcam cameras. Before their exponential growth during the pandemic, basic functionality was seen as acceptable, and thus, pass/fail tests were adequate for quality control. As growth accelerated, the VCX-Forum began developing a different approach for grading webcam systems. This test method is based on current devices and performance expectations from the segment in question (webcams) and then correlated with objective image quality characteristics.
To accomplish this goal, VCX developed a series of relevant focus areas for webcam testing.
Focus areas
The group identified focus areas to form the basis of how a webcam "should" function. These areas are a product of the main "pain points" experienced by the industry and are what contribute to the overall score valuation of each webcam.
Low light
Ensuring a camera system continues to demonstrate acceptable levels of image quality in low-light scenarios is often the most challenging issue for developers. Webcam systems are no exception and are often used in low-light situations and must be able to perform accordingly. The VCX test measures whether the image quality of a webcam is sharp, has minimal noise, and has accurate color reproduction even at a light level of 10 lux.
Mixed light and light changes
A webcam should always present the user accurately, including in mixed or changing light scenarios. The VCX test analyzes how a webcam system reacts to an evolving light situation, including changes to the intensity and light spectra (e.g., sunlight versus indoor LED illumination). The system should not introduce color shading issues affecting skin tones, even if the light changes.
Face-priority
Face detection and an accurate portrayal of the user are paramount for a webcam system. VCX evaluates how well a webcam accomplishes this task regardless of the user's face size, features, or skin tones. A VCX study during development showed that dark skin tones received lower results almost universally across the board. Thus, prioritizing the face is a crucial focus area for developers.
Challenging scenes
As with other camera systems, webcams will also be used in challenging scenes. VCX tests a variety of camera responses to changing settings, such as whether the video is stable and can still prioritize the face even in a high dynamic range scene, e.g., in front of a window. Another example is testing the auto white balance performance in changing color scenes, again particularly important with diverse skin tones.
Motion
We are all aware of how movement by the user can significantly affect the overall quality of the image during a video conference. The VCX test thus analyzes if the 3A convergence algorithms converge accurately and repeatably during movement without oscillation, sharp step functions, ghosting, or other motion artifacts.
Webcam KPIs
Once the focus areas were clear, VCX established a series of KPIs to test a webcam's overall image quality performance. These KPIs are adopted from industry standards where possible, but innovation was encouraged where gaps existed concerning webcams. For each KPI, VCX reviewed what the industry was doing "today" versus updated approaches.
As a result, certain KPIs, such as noise, contrast, and spatial frequency response, were modernized to fit with webcam expectations. Others, such as SNR or simple SFR measurements via slanted edges only, were removed from consideration as they have not kept pace with image signal processor (ISP) algorithm development.
The final KPIs include:
- Visual noise
- Dynamic range
- Resolution and SFR
- Sharpening
- White balance and convergence
- Frame rate
- Contrast response
- Exposure accuracy and convergence
- Texture loss
- Color accuracy
- Image uniformity
KPI testing equipment
VCX-Webcam 2023 requires specific equipment that meets the test specifications outlined in the VCX whitepaper*. This equipment includes test charts, illumination and measurement devices, and analysis software. Each device is used to measure the specific KPIs described in the VCX test.
Test charts
The VCX-Forum created a unique test chart (TE295) with multiple test targets built around a realistic mannequin head with numerous skin tone options. VCX partnered with a prop house to develop these heads after real people to ensure proper skin tones and human facial characteristics. The heads can also be attached to a separate head rotator to simulate the typical head movements of a webcam user.
Other charts to complete the VCX setup include a transmissive 20-patch OECF chart (IE recommends the TE270X or the TE264) to analyze dynamic range and a uniform field diffuser (TE255) for an integrating sphere to evaluate shading and image uniformity.
Illumination devices
Webcams must function in a variety of light scenarios. As a result, VCX requires that devices are evaluated for their performance at multiple light levels, including bright, mid, and low light. The most efficient way to create the necessary light is using a programmable light source like the iQ-Flatlights. The iQ-Flatlights use iQ-LED technology to generate custom spectra, including those outlined by VCX. Without a programmable light source, a tester would be required to have multiple light sources to simulate the necessary testing scenarios, thus increasing the possibility of human error, and convergence tests would be impossible without a programmable light source which is a test that VCX requires.
An integrating sphere is also required to illuminate transparent charts. IE recommends using the iQ-LED-powered LE7 device. Once again, iQ-LED technology provides a programmable light source to recreate the illumination testing requirements of VCX easily.
Measurement devices
As with other camera systems, webcam systems must be able to meet various timing parameters. For these tests, we recommend the LED-Panel, which can also be used in combination with the TE295 test chart. VCX also requires a 75" or larger TV to be placed as the backdrop of the testing scene. Using a TV allows the tester to simulate a variety of backgrounds that a webcam might encounter to see how the image quality reacts.
Analysis software
To analyze test results, VCX specifies an image quality evaluation software capable of analyzing all image quality KPIs and video files. We recommend using the iQ-Analyzer-X, built specifically for evaluating a camera system's image quality and video performance.
Setting up a test lab
With the right equipment, any company can set up a test lab for testing their webcam systems, according to VCX. VCX also offers the ability to become a certified and trusted lab to contribute to the overall score of a webcam (published on the VCX website). These labs must be certified by the VCX-Forum group before they can contribute to the scores, but any company can test their cameras according to the VCX-Webcam 2023 standard as long as they have the right equipment and setup. A typical configuration uses a combination of the VCX equipment discussed above properly spaced in a lab. See the image below for an example.
Example of a VCX test
Once the test setup is complete, testing can begin. The VCX whitepaper outlines the minimum benchmarks a webcam should achieve to be considered "acceptable" by the standard. The metrics used for the benchmarks were calculated based on the extensive testing of various webcam systems by the VCX-Forum.
As previously mentioned, each metric/KPI must be tested in three light levels (bright, mid, and low) and with three different skin tones.
The VCX score
Once testing is complete, the KPI measurement results are transformed into numerical values. Each value is then combined into a subgroup, and the weighted sum of these groups produces the final VCX score for the camera. These scores are always objective, and only certified VCX labs can contribute to the score. For more information on the scoring system, please visit the VCX website.
Video conferences today are no longer just a luxury; they are a necessity, and people want to demonstrate themselves at their best. The webcam systems need to project a high level of image quality to do that. Still, even with the explosion of webcam use, there are relatively few systems that we would consider having high video image quality.
As a result, the VCX-Forum recently published the VCX-WebCam 2023 standard to address the issues within the industry. The standard establishes performance benchmarks based on relevant KPIs and focus areas for webcam systems. As with the VCX-PhoneCam standard, each webcam is evaluated for various metrics under varying light conditions and given an overall score that consumers can use to help them make more educated buying decisions.
Become a VCX member
The VCX-Forum is an open, non-profit organization dedicated to establishing industry standards. Any industry expert is welcome to join the organization and contribute to standard publications. For more information, please visit the VCX website.
Contact us
Contact us at for more information on VCX test equipment.
*Contact the VCX-Forum at www.vcx-forum.org/contact for access to the white paper.
References
Lifesize. (2019). 2019 Impact of Video Conferencing Report. TMCnet.com. https://blog.tmcnet.com/blog/rich-tehrani/wp-content/uploads/2019/09/2019-Impact-of-Video-Conferencing-Report-Lifesize_FINAL.pdf
Molla, R. (2020 December 4). The pandemic was great for Zoom. What happens when there’s a vaccine? Vox. https://www.vox.com/recode/21726260/zoom-microsoft-teams-video-conferencing-post-pandemic-coronavirus