Rotoscope in Nuke
StyleFrame Team7 min read
To rotoscope in Nuke, add a Roto node (press O), draw Bezier shapes around your subject, and keyframe those shapes through the shot so the Roto node's alpha channel becomes a matte. Refine the edges with feathering, combine shapes with Merge operations such as stencil and mask, and use tracking to cut down the number of hand-set keyframes. The matte then feeds keying, color correction, cleanup, or any other compositing step downstream.
Roto vs. RotoPaint: Which Node to Use
Nuke has two main nodes for this work, and picking the right one avoids rework later.
- Roto (shortcut O) is built for shapes and mattes.
- RotoPaint includes the same shape tools plus paint tools. Use it when you need to paint out wires or clean up plates as well as cut mattes.
For pure matte work, Roto is usually the better choice. If a shot needs both a matte and cleanup, RotoPaint saves you from chaining nodes.
Setting Up Your First Roto
Connect the node and check the output
Select your Read node, press O, and open the Roto properties. By default the output is the alpha channel, so you will see nothing in the RGB view after drawing a shape. Press A over the Viewer to see the alpha.
Draw the shape
Pick the Bezier tool and click around the subject. A plain click gives a sharp corner. Click and drag to pull out smooth tangent handles. Close the shape by clicking the first point.
A few habits that save time later:
- Use fewer points than you think you need. Fewer points mean fewer things to keep aligned from frame to frame.
- Break complex subjects into parts. A character might become head, torso, upper arm, forearm, hand, thigh, and shin, each as its own shape that moves independently.
- Name every shape. "L_forearm" is easier to find at 3 a.m. than "Bezier14."
- Smooth, don't over-fiddle. Select points and use Nuke's smoothing options to even out curves before adjusting handles by hand.
Understand "Clip To"
The Roto node clips its output to the project format by default. If you draw outside the frame, or want to push a shape off-screen and transform it later, that clipping will cut your alpha at the format edge. Change the clip setting to no clip when the matte needs to extend beyond the frame, such as when you plan to reposition or reformat the result downstream.
Keyframing: Where the Time Goes
Once the first shape is drawn, you animate it. In Nuke's Roto, drawing or moving a shape on a frame sets a keyframe for that shape at that frame, so you always need to know which frame you are on before you touch a point.
Work in a sensible order
- Set keys at the extremes first. Go to the first and last frames of the range, then jump to the middle and adjust. Keep bisecting the gaps and fix only where the shape drifts.
- Use the shape's transform for broad motion. Select an entire shape and move, rotate, or scale it with the transform jack. This is often faster than moving points one at a time, and it keeps the shape's form intact.
- Adjust individual points only where needed. Fingers, cloth edges, and deforming limbs usually need point-level work. Rigid objects often don't.
- Scrub, don't just play. Playback at speed hides edge jitter. Step through frames or scrub slowly and look at the outline against the plate.
Use tracking to carry the motion
Nuke lets you track points and link shapes to the result, which reduces how many keys you set by hand. A common approach is to track a stable feature on the subject, apply that motion to the shape or a group of shapes, and then correct only the deformation on top. If the camera is moving but the subject is mostly static, a tracked transform can do most of the work.
Tracking is rarely a one-click fix. Expect to review the result, correct drift, and sometimes split a shot into sections with different tracking references.
Edges: Feather, Falloff, and Motion Blur
A hard-edged matte almost always looks wrong in a composite, because real footage has soft edges from lens blur, motion, and atmosphere.
- Feather softens the entire shape edge. Use it sparingly for global softness.
- Feather falloff controls how that softness falls off across the edge.
- Per-point feather lets you pull out the feather handle on individual points. This is the right tool for areas like hair edges that are crisp in one spot and soft in another.
- Motion blur: check with your supervisor or brief whether mattes should be delivered with or without motion blur, since it changes how the matte behaves downstream.
If you only need a quick softening, a Blur node after the Roto works, but it blurs the whole matte equally, so it is less precise than shape-level feathering.
Combining Shapes and Mattes
Real shots rarely need a single shape. Here are the standard ways to combine them.
Subtracting one matte from another
If a hand passes in front of an object, create separate Roto nodes for the object and the hand. Combine them with a Merge node (shortcut M) set to stencil. The B input is the object's alpha, the A input is the hand, and the result removes the hand's area from the object's matte.
Applying the matte to footage
To isolate the subject from the plate, merge the plate with the matte using the mask operation. Everything outside the alpha is removed.
Premultiplication
Premultiplication trips up many new compositors. In short:
- A matte on its own is just an alpha.
- A premultiplied image has RGB values multiplied by alpha, which is what Merge expects for standard "over" operations.
- If you see dark or bright fringes around your edges, check whether you premultiplied once, not twice, and whether you unpremultiplied before color corrections.
Garbage mattes and keying
Roto is often used to support a key rather than replace it. A loose garbage matte removes unwanted areas such as rigging, light stands, or screen edges, so the keyer only has to deal with the area around your subject. These mattes don't need pixel-perfect edges, so a few well-placed keyframes are enough.
Machine Learning with CopyCat
Nuke's CopyCat node trains a neural network on your own examples. You hand-roto a handful of representative frames, feed the plate and those mattes into CopyCat, train, and then apply the resulting model to the rest of the shot through an Inference node.
What to keep in mind:
- Training frames matter more than anything else. Choose frames that show the subject in different poses and lighting, and make the mattes accurate.
- Start small. Run a short training pass to see whether the approach works on the shot before committing to a long one.
- Expect to review and fix. The output is a starting point. Fast motion, thin hair, and occlusions can still need hand cleanup.
- It works per shot (or per sequence of similar shots). Models trained on one plate may not generalize to a different one.
CopyCat is most helpful on long shots where the subject keeps a consistent look and hand-keying every few frames would be tedious.
Best Practices from Production
Roto is repetitive work, so good habits compound.
- Read the brief before drawing anything. Confirm the frame range, which scan version to use, whether the plate is retimed or reframed, how elements should be split across channels, what the matte naming should be, and whether motion blur is expected.
- Keep shapes organized. Group related shapes, name them clearly, and color-code where it helps.
- Render mattes into channels. If the brief asks for several elements in one file, route them into separate channels and name your Write nodes according to the convention.
- Check mattes over a flat color. Merge your matte over a bright solid to expose gaps and edge issues the plate hides.
- Save versions. A new script version before a big revision is cheap insurance.
Quick Reference
Final Thoughts
Good roto in Nuke comes down to clean, simple shapes, smart keyframing, careful edge treatment, and a solid understanding of how mattes behave in a merge tree. Learn the Roto node thoroughly, add tracking and CopyCat where they save time, and keep the brief's technical requirements in view. The faster your matte is accurate, the sooner you can get to the creative work in the composite.