Light field coding and processing for view sequences

Light field cameras offer to capture extensive scene information, which can be computationally manipulated to render views at different viewpoints with desired optical configuration. However, light field data is a huge burden on storage devices and transmission channels. Therefore, encoders typically use computationally complex schemes to curtail data redundancy to achieve high compression gains.

Thus, existing light field compression methods are unsuitable for lightweight and fast encoding. In this PhD thesis, novel light field coding systems will be proposed to address this challenge for reconstructing light field data with different fidelity settings at the decoder. Precisely, we propose a novel efficient light field compression method for lossless/near-lossless reconstruction by extending an existing lossless image compression scheme, Context-based Adaptive Lossless Image Coding (CALIC), and eventually achieving a 6% compression gain over HEVC, with highly efficient encoding.

The research is further extended to restore a high-quality light field from a near-lossless hard-decoded counterpart leveraging a highly rigid light field structure, reported providing 5.61dB PSNR gain over the hard-decoded input. Moreover, a distributed light field coding scheme is proposed to significantly reduce the computational burden at the encoder by incorporating a high-quality light field view synthesis method at the decoder.

Quantitatively, it achieved 59% reduction in bitrate against HEVC-Intra. A light field can be synthesized using a sparse set of views. Thus encoding these views only and reconstructing the remaining light field using a view synthesis process at the decoder can be a feasible way of reducing the complexity of the encoder.

Furthermore, the view synthesis process can facilitate a traditional light field compression system by providing high-quality view predictions at the encoder to minimize the channel bandwidth consumption for the transmission of residual signals. A high-quality light field view synthesis network is proposed based on these motivations, which achieved 1dB gain over the state-of-the-art view synthesis method.

On the same line, research is conducted to enable efficient rendering of panoramic views by processing multi-cylinder image representation. Specifically, a pipeline is designed based on multi-view geometry concepts to obtain this representation using three multiplane images extracted from an existing learning-based approach.