The image quality of a camera is one of the most important quality factors of contemporary smartphones – for users just as well as for manufacturers. Therefore we developed a unique test program for cell phone cameras which gives a representative and quick overview of all essential camera characteristics.
This test procedure is performed by the iQ lab regularly, and the numerical results of it are amongst others used by Vodafone to generate the VCX-score.
The test procedure “iQ lab - Mobile Test” covers measurements of resolution, dynamic range, visual noise, distortion, sharpening, texture loss, color reproduction and shading under different light conditions as well as image stabilization functions and time-related aspects (e.g., shooting time lag).
It is possible to order measurement results of existing tests as well as order a new test of a mobile device of your choice or combine each with another. You also can choose just to take the results of one camera (back or front). We offer two different presentations of the results: A PDF-overview of all numerical results combined with the test images or a test report of all numerical results which additionally includes an assessment of the results, done by the Image Engineering experts.
If you have further questions or need information on pricing, please contact us.
Image quality measurement
Acting on the assumption, that a typical user takes pictures with the default settings, all analyses are performed using these settings. The parameters are only modified if the test requires it (e.g., image stabilization).
The central part of the iQ lab - Mobile Test is done by using the TE42, at which, depending on the default aspect ratio of the device under test, a 4:3 or 16:9 version of the chart comes into operation. This chart ensures the measurements of the resolution in the image corners are comparable.
The multipurpose test chart enables us to gain an incredible amount of information at once, and we also make sure, that all characteristics of the image quality are captured under the same conditions.
Smartphone cameras are used in any environment – while most devices perform quite well under bright light conditions, most differences in the performance are observed in low light. For that reason we additionally test the image quality under different lighting conditions:
- Default settings of the camera ~1000 lux
- Default settings of the camera, 4x-Zoom ~1000 lux
- Default settings of the camera -2 EV (equates ~250 lux)
- Default settings of the camera -4 EV (equates ~63 lux)
- Default settings of the camera, flash activated -4 EV (equates ~63 lux)
For each of these light conditions and settings following characteristics are measured (the iQ-Analyzer is used to gain the information) and reported.
Resolution measurement
Resolution is measured in the center and the image corners according to ISO12233:2014 using the sinusoidal Siemens stars of the chart. The stars are divided into segments (eight for the center star and three for the corner stars) and the resolution is measured for each of them. The obtained spatial frequency response (SFR) is calculated per segment and per star.
The SFR contains a lot of information about sharpness and limiting resolution at the various positions and orientations. The obtained data can be used to check the optical centering of the system, to analyze optical errors like astigmatism, to verify orientation-specific or frequency-specific image processing and to determine the limiting resolution. The acutance as a representation of the sharpness is also identified and reported.
The multipurpose test chart enables us to gain an incredible amount of information at once, and we also make sure, that all characteristics of the image quality are captured under the same conditions.
Corresponding patches
Texture loss measurement
The so-called texture loss is a critical parameter in the objective image quality assessment of today’s cameras. Especially cameras build in mobile phones show significant loss of low contrast and fine details which are hard to describe using standard resolution measurement procedures (e.g., as defined in ISO12233:2014).
The combination of very small form factor and high pixel count leads to high demand for noise reduction in the signal-processing pipeline of these devices. The Dead Leaves pattern is used for quite a while in this context. For an extensive assessment, the TE42 has two Dead Leaves patches, one high contrast and one with low contrast.
The evaluation method of the dead leaves pattern is called “DeadLeaves_cross” and is described in detail in this paper
Corresponding patches
Sharpening measurement
The TE42 contains four slanted edges, whereby two different contrasts are used, and each is available in horizontal and vertical orientation. The edges are used to evaluate the resolution according to ISO12233:2014. While the Siemens star is more robust against sharpening, the slanted edges are used to evaluate the sharpening behavior of the camera in detail.
Different contrasts are used as the sharpening algorithms in the image signal processor of today’s cameras may detect the edge contrast and adjust the sharpening according to this. The different contrasts of the edges are a low contrast edge (60% edge modulation) and a high contrast edge (80% edge modulation contrast).
Corresponding patches
Dynamic range and noise measurement
The circular gray scale of the chart is used to determine the optoelectronic conversion function (OECF). It describes how the digital camera transfers luminance into digital values. The function is specified for all three color channels red, green and blue in color images.
For a description of the noise, the signal to noise ratio (SNR) and the visual noise (VN) are calculated according to ISO15739. While the SNR is more common measurement of noise, visual noise is also evaluated as output referred noise. It takes into account that the visual perception of noise can be different for human observers compared to a mere SNR approach.
The dynamic range (DR) is calculated as well from the OECF. It describes the maximum scene contrast the digital camera can reproduce. We report it in f-stops.
Corresponding patches
Color reproduction measurement
The color patches around the OECF patches are created with respect to the Calibrite ColorChecker SG and are used for evaluation of the color reproduction. The color of these patches is well known and has been evaluated using a spectrometer.
The image data and the reference data is converted into color coordinates in the CIE L*a*b* color space that represents the color reception of the human visual system. From these values, the color distance ?E (Delta E) that describes the color reproduction quality is calculated.
Corresponding patches
Distortion measurement
The registration marks are used to determine the distortion of the device under test. The positions of all markers in the image are detected, based on that the so-called TV Distortion is calculated. The distance between the center of R to G and B to G (chromatic aberration) is also calculated for each mark. This enables us to report the TV-Distortion in % and the chromatic aberration in sub-pixel accuracy.
Corresponding patches
Shading measurement
All kinds of effects that change the intensity or color over the field in the image are measured as shading. For this measurement, the 24 ROI's evenly distributed over the images of the TE42 are evaluated. Reported values are:
- Intensity shading in f-stops
- Color shading in Eab
These values are respectively gained for all illuminations and settings.
Corresponding patches
Timing measurement
Timing measurements are performed according to ISO 15781 using a LED-Panel in the iQ-AF Box. It consists of 10 rows of ten LEDs that light up at a selectable frequency. Shutter lag, shooting time lag, start-up-time and frame sequence are measured.
To measure shooting time lag, the LED-Panel is started with a microswitch connected with a cable. The microswitch is mounted on display, and both “switches” are pressed and released simultaneously. Beforehand the device is defocused so the focus time is included. In the pictures, the elapsed time can be determined by counting the LEDs that have already been lit up.
This measurement is executed with two illuminations:
- 800 lux
- 30 lux
For the determination of the shutter time lag the device under test is prefocused on the target and then released. The difference between shooting time lag and shutter time lag is the time the device under test needs to focus (focus time).
Additionally, the time the camera of the smartphone needs to take ten images (frame sequence) is measured. Therefore the continuous-mode is activated if possible.
The time needed to get ready to shoot after the camera-app was started (start-up time) is a critical value and is also determined by using the LED-Panel. The start-up time includes the shooting time lag.
These three tests only are executed with illumination of 800 lux.
Image stabilization
As smartphones are generally hand-held, while all image quality tests are performed using a tripod, we also check the performance under a simulated tremor. We use an iQ-AF Box (16:9) and the so-called STEVE (Stabilization Evaluation Equipment). The TE261 chart within the iQ-AF Box is positioned in front of STEVE so that always the whole chart height is captured.
First of all reference images are generated, which means that STEVE isn’t moving and if possible the image stabilization of the mobile phone is switched off. After this STEVE is switched on and a test series without enabled image stabilization (if selectable) is done. If the device doesn’t offer an arbitrary image-stabilization, then the measurement is finished. Otherwise, an additional test series is generated with a moving STEVE and enabled image stabilization.
This procedure is done for two illuminations:
- 800 lux
- 30 lux
The generated images are analyzed with the iQ-Analyzer to gain information about the slanted edge in the center of the chart. The contrast, edge width and visual noise for all images are stated. The performance of the image stabilization is expressed as the delta between the reference and the images with an activated STEVE.
| What we measure | What you get | ||
|---|---|---|---|
| tested with | settings | measurement data for | |
| Image quality measurement |
 |
|
resolution center/corners (Siemens stars) |
| texture loss (dead leaves) | |||
| sharpening (slanted edges) | |||
| dynamic range | |||
| visual noise | |||
| signal to noise ratio | |||
| color reproduction | |||
| distortion wide- and tele-angle | |||
| shading wide- and tele-angle | |||
| Timing measurement |
 |
ISOauto, wide-angle and tele-angle if available, 30 lx and 300 lx | shutter delay |
| shooting delay wide-angle | |||
| shooting delay tele-angle | |||
| Image stabilization |
 |
Two settings: 800 lx and 40 lx; STEVE - movement is a sine-shaped function with 4 Hz and an amplitude of 0,14° | |
Products used in this test procedure
Software
Solutions
Test charts
Measurement devices
