Planar Tracker Node

The Planar Tracker node is designed to solve a match-moving problem that commonly comes up during post-production. As an example, live-action footage can often contain a planar surface such as a license plate or a road sign that needs new numbers in the license plate or a new city’s name on the road sign. Often, the problem is that the camera is moving in the shot, so the license plate or road sign is continuously changing perspective. You cannot just merge a new license plate over the existing one without accounting for the perspective distortions. A time-intensive way to solve this problem would be to use a Corner Pin node and manually keyframe the four corners. The Planar Tracker automates this keyframing process and tracks the perspective distortions of a planar surface over time. That tracking data is then applied with those same perspective distortions to a different foreground.

What the Planar Tracker Saves

While the Planar Tracker does save the resulting final track in the composition on disk, it does not save temporary tracking information such as the individual point trackers (compared with the Camera Tracker, which does save the individual point trackers). Some consequences of this include:

The point trackers no longer appear in the viewer when a comp containing a Planar Tracker node is saved and reloaded.
Tracking may not be resumed after a comp containing a Planar Tracker node has been saved and reloaded. In particular, this also applies to auto saves. For this reason, it is good to complete all planar tracking within one session.
The size of composition files is kept reasonable (in some situations, a Planar Tracker can produce hundreds of megabytes of temporary tracking data).
Saving and loading of compositions is faster and more interactive.

Planar Tracker Node Inputs

The Planar Tracker has four inputs:

Background: The orange background image input accepts a 2D image with the planar surface to be tracked.
Corner Pin 1: The green corner pin 1 input accepts a 2D image to be pinned on top of the background. There may be multiple corner pin inputs, named Corner Pin 1, Corner Pin 2,…etc.
Occlusion Mask: The white occlusion mask input is used to mask out regions that do not need to be tracked. Regions where this mask is white will not be tracked. For example, a person moving in front of and occluding bits of the pattern may be confusing the tracker, and a quickly-created rough rotomask around the person can be used to tell the tracker to ignore the masked-out bits.
Effect Mask: The blue input is for a mask shape created by polylines, basic primitive shapes, paint strokes, or bitmaps from other tools. Connecting a mask to this input limits the output of the Planar Tracker to certain areas.

Planar Tracker Node Setup

A basic Planar Tracker setup consists of just two nodes: a MediaIn connects to the background input and the Planar Tracker can be used as a separate branch from the rest of the node tree. Once the tracking is completed, a Planar Transform node should be generated to use the planar tracking data.

A Typical Planar Tracker Workflow

The following steps outline the workflow with the Planar Tracker:

Remove lens distortion: The more lens distortion in the footage, the more the resulting track will slide and wobble.
Connect footage: Connect a Loader or MediaIn node that contains a planar surface to the orange background input and view the Planar Tracker node in a viewer.
Select a reference frame: Move to a frame where the planar surface to be tracked is not occluded and click the Set button to set this as a reference frame.
Choose the pattern: In the viewer, make sure the onscreen controls are visible, and draw a polygon around the planar surface you want to track. This is called the “pattern.” In most cases, this will probably be a rectangle, but an arbitrary closed polygon can be used. The pixels enclosed by this region will serve as the pattern that will be searched for on other frames. Note that it is important that the pattern is drawn on the reference frame. Do not confuse the pattern with the region to corner pin (which always has four corners and is separately specified in Corner Pin mode).
Adjust render range: In the Keyframes Editor, adjust the render range to match the range of frames where the planar surface is visible.
Adjust track options: Frequently changed options include Tracker, Motion Type, and Track Channel.
Mask out occluders: If moving objects partially cover up the planar surface, you may wish to connect an occlusion mask to the Planar Tracker. When using the Hybrid tracker, providing a mask to deal with occluding objects is strongly recommended, while with the Point tracker it is recommended to try tracking without a mask.
Track: Click the Go button to return to the reference frame. Press the Track To End button and wait for the track to complete. Click the Go button to return to the reference frame again. Press the Track To Start button and wait for the track to complete. Note that the tracks in the viewer are not selectable or deletable as they are in a Camera Tracker.
Check track quality: Visually inspect the track to see how accurate it is. Does it stick to the surface? Switching to Steady mode can help here.
Use the track: At this point, in most cases you will create a Planar Transform node from the Inspector and use it to apply the tracked perspective distortion onto masked images. If the image you are applying the tracking data to is full frame unmasked clip, you can use the Steady, Corner Pin, and Stabilize operation modes in in the Planar Tracker.

Planar Tracker Node Controls Tab

The Controls tab contains controls for determining how the Planar Tracker will be used, setting the reference frame and initiating the track.

Operation Mode
The Operation Mode menu selects the purpose of the Planar Tracker node. The Planar Tracker has four modes of operation:

Track: Used to isolate a planar surface and track its movement over time. Then, you can create a Planar Transform node that uses this data to match move another clip in various ways..
Steady: After analyzing a planar surface, this mode removes all motion and distortions from the planar surface, usually in preparation for some kind of paint or roto task, prior to “unsteadying” the clip to restore the motion.
Corner Pin: After analyzing a planar surface, this mode computes and applies a matching perspective distortion to a foreground image you connect to the foreground input of the Planar Tracker node, and merges it on top of the tracked footage.
Stabilize: After analyzing a planar surface, this mode allows smoothing of a clip’s translation, rotation, and scale over time. This is good for getting unwanted vibrations out of a clip while retaining the overall camera motion that was intended.

The last three modes (Steady, Corner Pin, and Stabilize) use the tracking data produced in Track mode.

Reference Time
The Reference Time determines the frame where the pattern is outlined. It is also the time from which tracking begins. The reference frame cannot be changed once it has been set without destroying all pre-existing tracking information, so scrub through the footage to be tracked and choose carefully. The reference frame must be chosen carefully to give the best possible quality track.

You choose a reference frame by moving the playhead to an appropriate frame and then clicking the Set button to choose that frame.

Pattern Polygon
You specify which region of the image you want to track by drawing a polygon on the reference frame. Typically, when you first add a Planar Tracker node, you are immediately ready to start drawing a polygon in the viewer, so it’s best to do this right away. When choosing where to draw a polygon, make sure the region selected belongs to a physically planar surface in the shot. In a pinch, a region that is only approximately planar can be used, but the less planar the surface, the poorer the quality of the resulting track.

As a rule of thumb, the more pixels in the pattern, the better the quality of the track. In particular, this means the reference frame pattern should be:

As large as possible.
As much in frame as possible.
As unoccluded as possible by any moving foreground objects.
At its maximal size (e.g., when tracking an approaching road sign, it is good to pick a later frame where it is 400 x 200 pixels big rather than 80 x 40 pixels).
Relatively undistorted (e.g., when the camera orbits around a flat stop sign, it is better to pick a frame where the sign is face-on parallel to the camera rather than a frame where it is at a highly oblique angle).

If the pattern contains too few pixels or not enough trackable features, this can cause problems with the resulting track, such as jitter, wobble, and slippage. Sometimes dropping down to a simpler motion type can help in this situation.

After you’ve drawn a pattern, a set of Pattern parameters lets you transform and invert the resulting polygon, if necessary.

Track Mode
Track mode is unlike the other three options in the Operation menu in that is the only option that initiates the planar tracking. The other modes use the tracking data generated by the Track mode.

Tracker
There are two available trackers to pick from:

Point: Tracks points from frame to frame. Internally, this tracker does not actually track pointsper-se but rather small patterns like Fusion’s Tracker node. The point tracker possesses the ability to automatically create its internal occlusion mask to detect and reject outlier tracks that do not belong to the dominant motion. Tracks are colored green or red in the viewer, depending on whether the point tracker thinks they belong to the dominant motion or they have been rejected. The user can optionally supply an external occlusion mask to further guide the Point tracker.
Hybrid Point/Area: Uses an Area tracker to track all the pixels in the pattern. Unlike the Point tracker, the Area tracker does not possess the ability to automatically reject parts of the pattern that do not belong to the dominant motion, so you must manually provide it with an occlusion mask. Note that for performance reasons, the Hybrid tracker internally first runs the Point tracker, which is why the point tracks can still be seen in the viewer. There is no best tracker. They each have their advantages and disadvantages:
- Artist Effort (occlusion masks): The Point tracker will automatically create its internal occlusion mask. However, with the Hybrid tracker, you need to spend more time manually creating occlusion masks.
- Accuracy: The Hybrid tracker is more accurate and less prone to wobble, jitter, and drift since it tracks all the pixels in the pattern rather than a few salient feature points.
- Speed: The Hybrid tracker is slower than the Point tracker.

In general, it is recommended to first quickly track the shot with the Point tracker and examine the results. If the results are not good enough, then try the Hybrid tracker.

Motion Type
Determines how the Planar Tracker internally models the distortion of the planar surface being tracked. The five distortion models are:

Translation.
Translation, Rotation (rigid motions).
Translation, Rotation, Scale (takes squares to squares, scale is uniform in x and y).
Affine includes translation, rotation, scale, skew (maps squares to parallelograms).
Perspective (maps squares to generic quadrilaterals).

Each successive model is more general and includes all previous models as a special case.

When in doubt, choose Perspective for the initial track attempt. If the footage being tracked has perspective distortions in it, and the Planar Tracker is forced to work with a simpler motion type, this can end up causing the track to slide and wobble.

Sometimes with troublesome shots, it can help to drop down to a simpler motion model—for example, when many track points are clustered on one side of the tracked region or when tracking a small region where there are not many trackable pixels.

Output
Controls what is output from the Planar Tracker node while in the Track operation mode.

Background: Outputs the input image unchanged.
Background – Preprocessed: The Planar Tracker does various types of preprocessing on the input image (e.g., converting it to luma) before tracking. It can be useful to see this when deciding which track channel to choose.
Mask: Outputs the pattern as a black and white mask.
Mask Over Background: Outputs the pattern mask merged over the background.

Track Channel
Determines which image channel in the background image is tracked. It is good to pick a channel with high contrast, lots of trackable features, and low noise. Allowed values are red, green, blue, and luminance.

Tracking Controls
These controls are used to control the Tracker. Note that while tracking, only track to a new frame if the current frame is already tracked or it is the reference frame.

Track to start: Tracks from the current frame backward in time to the start (as determined by the current render range).
Step tracker to previous frame: Tracks from the current frame to the previous frame.
Stop tracking: Stops any ongoing tracking operations.
Step tracker to next frame: Tracks from the current frame to the next frame.
Track to end: Tracks from the current frame forward in time to the end (as determined by the current render range).
Trim to start: Removes all tracking data before the current frame.
Delete: Deletes all tracking data at all times. Use this to destroy all current results and start tracking from scratch.
Trim to end: Removes all tracking data after the current frame. This can be useful, for example, to trim the end of a track that has become inaccurate when the pattern starts to move off frame.

Show Splines
This button to the right of the “Trim to end” button opens the Spline Editor and shows the splines associated with the Planar Tracker node. This can be useful for manually deleting points from the Track and Stable Track splines.

Right-Click Here for Track Spline
While tracking, a spline containing 4 x 4 matrices at each keypoint is created. This is known as the “Track spline” or just “Track” for short. These matrices completely describe the distortions of the tracked pattern.

Create Planar Transform
After tracking footage, this button can be pressed to create a Planar Transform node on the Node Editor. The current tracking data is embedded in the Planar Transform node so that it can replicate the planar distortions tracked by the Planar Tracker node. Unless you are compositing a full frame foreground that matches the same dimensions as the raster, it is best to create a Planar Transform and use it to apply motion to the foreground.

Steady Mode
In Steady mode, the Planar Tracker transforms the background plate to keep the pattern as motionless as possible. Any leftover motion is because the Planar Tracker failed to follow the pattern accurately or because the pattern did not belong to a physically planar surface.

Steady mode is not very useful for actual stabilization, but is useful for checking the quality of a track. If the track is good, during playback the pattern should not move at all while the rest of the background plate distorts around it. It can be helpful to zoom in on parts of the pattern and place the mouse cursor over a feature and see how far that feature drifts away from the mouse cursor over time.

Steady Time
This is the time where the pattern’s position is snapshotted and frozen in place. It is most common to set this to the reference frame.

Invert Steady Transform
This causes the Planar Tracker node to reverse the effects of the steady transform. This means two Planar Tracker nodes connected back to back with the second set to invert should give back the original image. If you place an effects node in between the two, then the effect will be locked in place. This should only be used to accomplish effects that cannot be done through corner pinning, since it involves two resamplings, causing softening of the background image.

Clipping Mode
Determines what happens to the parts of the background image that get moved off frame by the steady transform:

Domain: The off frame parts are kept.
Frame: The off frames parts are thrown away.

Domain mode is useful when Steady mode is being used to “lock” an effect to the pattern. As an example, consider painting on the license plate of a moving car. One way to do this is to use a Planar Tracker node to steady the license plate, then a Paint node to paint on the license plate, and then a second Planar Tracker to undo the steady transform. If the Clipping mode is set to Domain, the off frame parts generated by the first Planar Tracker are preserved so that the second Planar Tracker can, in turn, map them back into the frame.

Corner Pin Mode
In Corner Pin mode, one or more textures can be attached to a previously tracked planar surface and undergo the same perspective distortions as the surface.

The corner pin workflow with Planar Tracker is:

Track: select a planar surface in the shot that you wish to attach a texture to or replace the texture on. Track the shot (see the tracking workflow in the Track section).
Switch the Operation Mode to Corner Pin: When Corner Pin mode is entered from Track mode, the pattern polygon is hidden and a corner pin control is shown in the viewer.
Connect in the texture: In the Node Editor, connect the output of the MediaIn node containing the texture to the Corner Pin 1 input on the Planar Tracker node.
Adjust corner pin: Drag the corners of the corner pin in the viewer until the texture is positioned correctly. Sometimes the Show Grid option is useful when positioning the texture. Additionally, if it helps to position it more accurately, scrub to other times and make adjustments to the corner pin.
Review: Play back the footage and make sure the texture “sticks” to the planar surface.

Merge Mode
Controls how the foreground (the corner pinned texture) is merged over the background (the tracked footage). If there are multiple corner pins, this option is shared by all of them. There are four options to pick from:

BG only
FG only
FG over BG
BG over FG

Number of Corner Pins
Use the + and – buttons to increase or decrease the number of corner pins. Each time an additional corner pin is created, a corresponding input appears on the node in the Node Editor.

Corner Pin 1 Input Group
Each corner pin has a group of related inputs:

Enable: Controls the visibility of the corner pin in the viewer.
Show Grid: Shows a grid over the corner pin. This can be useful when positioning the corners.
Merge Options: Controls merging of corner pin texture over the background. (See the documentation for the Merge node.)
Reference Time Positions: The positions of the four corners at the reference time. If the track was not perfect, these positions can be animated to make adjustments on top of the track.

Stabilize Mode
Stabilize mode is used to smooth out shakiness in the camera by applying a transform that partially counteracts the camera shake. This stabilizing transform (contained in the Stable Track spline) is computed by comparing neighboring frames.

Be aware that the Planar Tracker stabilizes based on the motion of the pattern, so it is essential to choose the pattern carefully. If the motion of the pattern does not represent the motion of the camera, then there may be unexpected results. For example, if tracking the side of a moving truck and the camera is moving alongside it, the Planar Tracker smooths the combined motion of both the truck and the mounted camera. In some cases, this is not the desired effect. It may be better to choose the pattern to be on some fixed object like the road or the side of a building, which would result in smoothing only the motion of the camera.

One unavoidable side effect of the stabilization process is that transparent edges appear along the edges of the image. These edges appear because the stabilizer does not have any information about what lies off frame, so it cannot fill in the missing bits. The Planar Tracker node offers the option to either crop or zoom away these edges. When filming, if the need for post-production stabilization is anticipated, it can sometimes be useful to film at a higher resolution (or lower zoom).

The Planar Tracker Stabilization Workflow

Track: Select a roughly planar region that represents the motion that you want to stabilize. Track the shot (see the tracking workflow in the Track section).
Switch the Operation Mode to Stabilize: Until a stabilization is computed, the Planar Tracker will just output the input footage.
Adjust stabilization options: Frequently changed options are Parameters to Smooth and Smoothing Radius.
Compute stabilization: Press the Compute Stabilization button and wait for the stabilization computations to finish. Play back the output of the Planar Tracker node to see the effects of the stabilization. Notice that transparent edges have been introduced around the edges of the image by the stabilization transform.
Refine: Adjust the stabilization options and recompute the stabilization as many times as desired.
Handle transparent edges (optional): Set the Frame Mode to either Zoom or Crop as desired and then click the Auto Zoom or Auto Crop button. Play back the footage to observe the effects. If there is too much zoom or the image has been cropped too small, try reducing the amount of smoothing.

Parameters to Smooth
Specify which of the following parameters to smooth:

X Translation
Y Translation
Rotation
Scale

Smoothing Window
When stabilizing a particular frame, this determines how the contributions of neighboring frames are weighted. Available choices are Box and Gaussian.

Smoothing Radius (Frames)
Determines the number of frames whose transforms are averaged together to compute the stabilization. A larger Smoothing Radius results in more smoothing but introduces more transparent edges.

Compute Stabilization
Clicking this button runs the stabilizer, overwriting the results of any previous stabilization. As soon as the stabilization is finished, the output of the Planar Tracker node will be immediately updated with the stabilization applied.

Clipping Mode
Determines what happens to the parts of the background image that get moved off frame by the stabilization:

Domain: The off frame parts are kept.
Frame: The off frames parts are thrown away.

Frame Mode
This controls how transparent edges are handled. The available options include:

Full: Do nothing. Leaves the transparent edges as is.
Crop: Crops away the transparent edges. When this option is selected, the size of Planar Tracker’s output image is smaller than the input image. No image resamplings occur. In Crop mode, use the Auto Crop button or manually adjust the crop window by changing the X Offset, Y Offset, and Scale sliders.
- Auto Crop Button: When this button is clicked, the Planar Tracker will examine all the frames and pick the largest possible crop window that removes all the transparent edges. The computed crop window will always be centered in frame and pixel aligned. When clicked, Auto Crop updates the X/Y Offset and Scale sliders.
Zoom: Scales the image bigger until the transparent edges are off frame. Choosing this option causes an image resampling to occur. The downside of this approach is that it reduces the quality (slightly softens) of the output image. In Zoom mode, use the Auto Zoom button or manually adjust the zoom window by changing the X Offset, Y Offset, and Scale sliders.
- Auto Zoom: When this button is clicked, the Planar Tracker will examine all the frames and pick the smallest possible zoom factor that removes all the transparent edges. The computed zoom window will always be centered in frame. When clicked, Auto Zoom updates the X/Y Offset and Scale sliders.

Right-Click Here for Stable Track Spline
Right-clicking over this label provides access to a spline whose keyframes contain 4 x 4 matrices which in turn represent the stabilization transforms. This is mostly here for completeness and for advanced users.

Planar Tracker Node Options Tab

These controls affect the look of onscreen controls in the viewer

Darken Image
Darkens the image while in Track mode in order to better see the controls and tracks in the viewer. The Shift+D keyboard shortcut toggles this.

Show Track Markers
Toggles the display of the dots marking the location of trackers at the current time.

Show Trails
Toggles the display of the trails following the location of trackers.

Trail Length
Allows changing the length of tracker trails. If the pattern is moving very slowly, increasing the length can sometimes make the trails easier to follow in the viewer. If the pattern is moving very fast, the tracks can look like spaghetti in the viewer. Decreasing the length can help.

Inlier/Outlier Colors
When tracking, the tracker analyzes the frame and detects which of the multitudinous tracks belong to the dominant motion and which ones represent anomalous, unexplainable motion. By default, tracks belonging to the dominant motion are colored green (and are called inliers) and those that do not belong are colored red (and are called outliers). Only the inlier tracks are used when computing the final resulting track.

Planar Tracker Node Settings Tab

The Settings tab in the Inspector is also duplicated in other Tracking nodes. These common controls are described in detail HERE.