Punchcard: Videotrace

VideoTrace is a system for interactively generating realistic 3D models of objects from video—models that might be inserted into a video game, a simulation environment, or another video sequence. The user interacts with VideoTrace by tracing the shape of the object to be modelled over one or more frames of the video. By interpreting the sketch drawn by the user in light of 3D information obtained from computer vision techniques, a small number of simple 2D interactions can be used to generate a realistic 3D model. Each of the sketching operations in VideoTrace provides an intuitive and powerful means of modelling shape from video, and executes quickly enough to be used interactively. Immediate feedback allows the user to model rapidly those parts of the scene which are of interest and to the level of detail required. The combination of automated and manual reconstruction allows VideoTrace to model parts of the scene not visible, and to succeed in cases where purely automated approaches would fail.

See the modelling examples list for more demonstrations, or the applications page, for ways to use the software.

Architectural applications

VideoTrace allows a user to create models of object from video sequences. There are a number of applications which require models of buildings, including those in architecture and defence. VideoTrace thus enables a user to generate a model of a building or a set of buildings from a video sequence. These sequences can be captured from street level, or from above.VideoTrace uses image data to interpret the user interactions, but exploits 3D point clouds when available, to guide the fitting process. It is thus well suited to the case where a laser scanner or similar has been used to generate a set of depth samples in addition to the imagery.

The building / urban models generated by VideoTrace can be used in simulation environments, for the visualisation of proposed building designs in term of their surroundings, and to put models into visualisation environments like Google Earth.

There are examples of modelling the Sydney Opera House, a (more normal) house, and the archway on this site.

Modelling a significant part of a city can be a very labour intensive process.  Videotrace simplifies the job of generating the geometry, as this video of the modelling of part of Copenhagen shows

Generating a model using Videotrace assumes that you have video of the object which you want to mode, but makes the process of generating an accurate model, and the associated textures, a relatively simple process.

See the modelling of the Sydney Opera House, and the house modelling posts for more examples.

We’re currently working with the Australian Centre for Visual Technologies on  a version of an Augmented Reality version of Videotrace that you can use while walking around.  The idea is that you’d use a camera as a modelling tool.  You would then walk around a city, modelling it as you go. Good progress is being made towards this goal, with a prototype system in the making.

Image editing programs have become very sophisticated, allowing even inexperienced uses to change the content of images.  Cutting and psting objects between images, or just deleting objects from images, are relatively easy to achieve. Personal video editing software, however, is far more primitive in the range of operations supported.  It allows clips to be shortened, or combined, but only the content of the video remains largely unchanged.  Copying an object from one video sequence to another, for example, requires a film post-production (special effects) company.  The reason for this difference between image editing and video editing software is that most video content editing opeation require 3D information.  This 3D information includes the path of the camera through the scene, but also the shape of the objects visible.

This video shows what happens if you perform the equivalent of image editing on video.  All we have done here is copy pixels from one video sequence into another.

Videotrace provides a easy way of recovering this required 3D information, and possibly a way of making 3D special effect available to all.  The following is an example of what is possible

The creation of video editing software which allows ordinary users to insert 3D special effects into video is one of the longer term goals of Punchcard.  Work towards this goal is being carried out with the Australian Centre for Visual Technologies, the team that developed the original Videotrace technology

One application of the VideoTrace technology is in generating models of the objects in the real world in order to insert them into virtual environments. Examples might include modelling your car so that you can drive it in a game, or modelling an area for use in a simulation.

Virtual worlds such as Second Life and Home are limited by the object models they have available. The modelling process used to insert objects into these environments is typically either an arduous process requiring a great deal of time and skill, or so facile as to lack the power to create models of interest. The result is a virtual world full of objects made up of cubes and cylinders.

VideoTrace offers the opportunity to generate detailed texture-mapped models of real world objects simply and quickly. Inserting your couch into your virtual world requires only that you take a video of it and trace over its structure in VideoTrace.

The process of producing the truck model is described here.