A new rearch paper authored by Dr. Zheheng Jiang was presented at the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) in Vancouver, Canada.
Titled A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Image, this paper describes a groundbreaking method of constructing a 3-dimensional model of a human hand from one photograph.
It is important to be able to compare structures in 3D to compensate for changes in orientation etc when comparing images
The full paper can be found here.
Abstract:
Recently, deep learning based approaches have shown promising results in 3D hand reconstruction from a single RGB image. These approaches can be roughly divided into model-based approaches, which are heavily dependent on the model’s parameter space, and model-free approaches, which require large numbers of 3D ground truths to reduce depth ambiguity and struggle in weakly-supervised scenar- ios. To overcome these issues, we propose a novel proba- bilistic model to achieve the robustness of model-based ap- proaches and reduced dependence on the model’s param- eter space of model-free approaches. The proposed prob- abilistic model incorporates a model-based network as a prior-net to estimate the prior probability distribution of joints and vertices. An Attention-based Mesh Vertices Un- certainty Regression (AMVUR) model is proposed to cap- ture dependencies among vertices and the correlation be- tween joints and mesh vertices to improve their feature rep- resentation. We further propose a learning based occlusion- aware Hand Texture Regression model to achieve high- fidelity texture reconstruction. We demonstrate the flexibil- ity of the proposed probabilistic model to be trained in both supervised and weakly-supervised scenarios. The experi- mental results demonstrate our probabilistic model’s state- of-the-art accuracy in 3D hand and texture reconstruction from a single image in both training schemes, including in the presence of severe occlusions.