Dual camera algorithms

Image fusion

Creating Superior Image Quality

Play Video

high quality standards

Recent flagship smartphones are integrating dual cameras, enhancing the camera user experience in various ways. To create improved image quality, the two images generated by these dual cameras need to be combined into a single image, providing higher resolution, better low-light performance, lower noise levels and other photography features not available from a single image. The process of combining these two images into a single image is known as image fusion.

In smartphones, dual camera image fusion comes into play in several ways. The first employs a dual camera with one color sensor and another monochromatic sensor (with the Bayer filter removed). The monochromatic sensor captures 2.5 times more light and thus reaches better resolution and signal-to-noise ratio (SNR). By fusing the images coming from both cameras, the output image has better resolution and SNR, especially in low light conditions.

2x zoom dual camera

The second approach is a zoom dual camera – A wide field of view camera coupled with a telephoto narrow field of view camera. In this case, image fusion improves the SNR and resolution from no zoom up to the point the telephoto camera field-of-view is the dominant one. In this low zoom factor range, the fusion utilizes the fact that few Tele pixels are mapped into the fusion image. This means that the output fused image is comprised from 1 pixel from the wide pixel and number of pixel from the Tele, resulting in better SNR and higher resolution.

Performing image fusion presents several algorithmic challenges:

  • Occlusions: Since the two images are taken from a different point of view, there could be areas in one image which are occluded from the other image.
  • Optical distortion: a form of aberration in which the lens creates a deviation in the output image. This type of aberration of the lens makes the process of determining the parallax for each pixel more difficult and requires the algorithm to compensate for the distortion.
  • Differences in depth of field: areas that may be in focus in one camera and out of focus in the other camera, in which case the registration becomes much trickier.
  • Transition between overlapped area to non-overlapped area: The differences in the field of view and the resolution create an inevitable inclusion of one image inside the other (usually it is the wide image that includes the tele image).
  • ISP differences: since each of the 2 images is processed by a different ISP, the input images could suffer from different sharpness, noise level, contrast and colors.
  • Wide/Tele resolution differences: as part of the optical design, the Tele image results in higher resolution than the Wide image. Due to the resolution differences between the 2 cameras, small objects which are detected by the tele camera might not be detected by the wide camera.

Corephotonics’ image fusion algorithm deals with the above challenges, while requiring minimal processing load, fast run time and optimal resource utilization of the application processor. Having a mature, well tested algorithm significantly reduces the chance of image artifacts, which are unacceptable in today’s uncompromising mobile imaging world.

Depth mapping

Stereoscopic 3D Depth Map Creation

depth algorithm

Play Video

Corephotonics’ depth algorithm is based on stereoscopic vision, similar to that of the human eye. We use our dual cameras to produce a dense, detailed and accurate depth map of the scene.

In stereo vision, objects close to the camera are located in different regions of each image. As the object moves further away from the cameras, the disparity reduces until, at infinity, the object will seem to be at the same place in both images. A stereoscopic depth map is based on such camera disparities.

The Corephotonics depth map algorithm is uniquely designed to deal with various challenges, while requiring minimal load from the application processor. Such challenges include:

  1. Optical aberrations: Optical characteristics of one lens versus the other can make the registration process more challenging.
  2. Depth bleeding: A phenomenon in which the transition between a close object and a far object is smooth and gradual instead of sharp and clear.
  3. Occlusions: Depending on the scene, there could be areas within the field of view of one camera that are occluded from the other camera’s field of view. In order to build a depth map from two images, the algorithm has to find matching pixels between the left and right images, and occlusions impede this matching.
  4. Texture-less planes: In order to distinguish between different depths within the image, registration is performed. Regions with no texture information create a challenge in distinguishing different depths.

The algorithm

The algorithm is highly optimized not only for accuracy but also for real-time execution on smartphone application processors, up to 30fps, while maintaining a low memory footprint.

In turn, such depth maps can be used for various applications, including:

  • Applying a Bokeh effect (also known as Portrait Mode), whereby the main object (usually a person) remains in clear focus, whereas the rest of the scene is blurred, giving the image a shallow depth-of-field effect that is typically found in DSLRs.
  • Augmented Reality (AR) and Virtual Reality (VR) applications, enabling such camera-based devices to accurately sense their environment for the most immersive user experience.
  • Refocusing the scene on different objects at different distances from the camera. Such an effect can be applied post-capture, enhancing the camera user experience “after the fact”.
  • Accelerating the auto-focus camera function by augmenting existing focus technologies with depth-based focusing.
  • Scene analysis and object recognition based on depth maps.
 

Smooth camera transition

Continuous and Smooth Zoom for Images and Videos

We are all used to zooming

We are all used to zooming in and out before snapping a picture in order to get the framing right, and to zooming in when recording a video in order to highlight the most important part of the scene. When we pinch-to-zoom using our camera app, we are used to a natural, continuous zooming experience, similar to a true optical zoom experience on a DSLR.

With Corephotonics’ dual camera optical zoom, the user experience is the same, but dramatically better: zooming in with Corephotonics’ zoom dual camera gives users the same smooth, fluid experience they are used to, but instead of digitally zooming in, both cameras are used to provide optical zoom quality.

Corephotonics’ software library smoothly transitions from one camera to the other, avoiding a noticeable “switch” or “jump”, which is evident in competing solutions. The software analyzes the frames from both cameras, corrects for assembly inaccuracies and for color and luminance mismatches, accounts for imperfect optics and takes care of parallax. It also supports optical and electronic image stabilization (EIS).

For every frame in the video stream, the output frame from the library, On (x,y) , is calculated as a combination of the wide frame, Wn (x,y) , and the tele frame, Tn (x,y) , according to the following formula:

On (x,y)=αn (n,x,y,zf…) Pn {Wn (x,y)}+(1-α(n,x,y,zf…)) Qn {Tn (x,y)}

The transformations Pn {…}  and  Qn {…}  warp the input frames according to a projective transformation, and function  α(…) c controls how the frames are combined. Together, they guarantee smooth transition when zooming in.

The Corephotonics libraries optimize image quality for video recording and for preview, while keeping the camera native high frame rate (from 30fps to 120fps) and maintaining lowest power consumption.

Image quality

Image quality testing
UI/UX testing
Testing benchmark design and integration

Camera hardware

Compact Lens design
Micro electro-mechanical systems for zoom, auto-focus, optical image stabilization
Diverse actuator technologies and control systems
Environmental and reliability testing in preparation for ultra high volume Manufacturing

Computer Vision

Deep Computer Vision models for: scene understanding, object detection and recognition and tracking; classification; depth analysis
Stereo vision and depth mapping
Image fusion
Dynamic multi aperture calibration
Heterogeneous computing (MT CPU, GPU, DSP, unified-memory architecture)
Mobile camera software architecture
UI/UX design for camera applications