ToTem NRSfM: Object-wise Non-Rigid Structure-from-Motion with a Topological Template

We present a Non-Rigid Structure-from-Motion (NRSfM) method to reconstruct an object whose topology is known. We represent the topology by a 3D shape that weakly resembles the object, which we call a Topological Template (ToTem).

The ToTem has two main differences with the template used in Shape-from-Template (SfT). First, the shape in the ToTem is not necessarily feasible for the object, whereas it must be in the SfT's template. Second, the ToTem only models shape, excluding the classical texture map representing color in the SfT's template. These two differences greatly alleviate the practical difficulty of constructing a template. However, they make the reconstruction problem challenging, as they preclude the use of strong deformation constraints between the template shape and the reconstruction and the possibility of directly establishing correspondences between the template and the images.

Our method uses an isometric deformation prior and proceeds in four steps:

1. It reconstructs point clouds from the images
2. It aligns the ToTem to the point clouds
3. It creates a coherent surface parameterization
4. It performs a global refinement, posed as a Non-Rigid Bundle Adjustment (NRBA)

We show experimentally that our method outperforms the existing methods for its isolated steps and NRSfM methods overall, in terms of 3D accuracy, ability to reconstruct the object's visible surface, and ability to approximate the object's invisible surface.

Pre-print: shorturl.at/zDUV0 (also provided in the 'Docs' folder)

---

Dependencies

The following are the dependencies for the code (included in the 'Dependencies' folder):

• CrustOpen: based on [Bernardini et al, 1999]
• Generalized Procrustes Analysis (GPA): based on [Bai et al., 2022]
• Gloptipoly on SeDuMi

---

Scripts

The following scripts are included in this repository:

• IsometricNRSfM.m: replicates section 4 of our article
• Parameterisation.m: replicates section 6 of our article
• SurfaceBasedNonRigidBundleAdjustment.m: replicates section 7 of our article

The code provided in this repository has been tested on the following platforms:

• Matlab 2022a on Windows 11
• Matlab 2022b on Ubuntu 20.04 LTS (Focal Fossa)

(Please submit an 'issue' if you face any problems with installing the code or executing the scripts. We will try to address them as soon as possible.)

Data format

All data have been provided in standard NRSfM/SfT format, we explain it below:

Say the data contains n images tracking up to m feature correspondences across images.

The data is a MATLAB struct, say Data. It contains the following subfields:

• Data.Pgth(i).P is a [3 x m] matrix, containing 3D groundtruth points, for all i in [1, n]

• Data.p(i).p is a [2 x m] or [3 x m] matrix, containing tracked point correspondences (image coordinates), for all i in [1, n]. If the matrix is of size [3 x m], the last row is necessarily all ones

• Data.v is a [n x m] matrix which is zero if a point (indexed by column) is invisible at that particular image corresponding to the row number, one otherwise.

Citation

Our article was accepted at the International Journal of Computer Vision (IJCV) in 2024. If you find this code or the associated paper useful, you may cite the article with:

Bibtex:

@article{sengupta2024totem,
  title={ToTem NRSfM: Object-wise Non-Rigid Structure-from-Motion with a Topological Template},
  author={Sengupta, Agniva and Bartoli, Adrien},
  journal={International Journal of Computer Vision},
  pages={1--42},
  year={2024},
  publisher={Springer}
}

APA:

Sengupta, A., & Bartoli, A. (2024). ToTem NRSfM: Object-wise Non-Rigid Structure-from-Motion with a Topological Template. International Journal of Computer Vision, 1-42.

Pre-print and additional datasets are also available at the EnCoV research group and Prof. Adrien Bartoli's website: https://encov.ip.uca.fr/ab/

---

References

[Bernardini et al, 1999]: Bernardini, F., Mittleman, J., Rushmeier, H., Silva, C., & Taubin, G. (1999). The ball-pivoting algorithm for surface reconstruction. IEEE transactions on visualization and computer graphics, 5(4), 349-359.

[Bai et al., 2022]: Bai, Fang, and Adrien Bartoli. "Procrustes analysis with deformations: A closed-form solution by eigenvalue decomposition." International Journal of Computer Vision (2022): 1-27.