A system for generating a three-dimensional model of an object from a two-dimensional image sequence. According to one embodiment, the system includes an image sensor for capturing a sequence of two-dimensional images of a scene, the scene including the object, a two-dimensional motion filter module in communication with the image sensor for determining from the sequence of images a plurality of two-dimensional motion parameters for the object, and a three-dimensional structure recovery module in communication with the two-dimensional motion filter module for estimating a set of three-dimensional shape parameters and a set of three-dimensional motion parameters from the set of two-dimensional motion parameters using a rank 1 factorization of a matrix.