Step 1. Prepare your file
Use Photoshop to open an image you’d like to edit; for this example, I’ll use the photo of a coffee mug. Once open, locate the Layers panel (Window -> Layers). You should see only one layer, called Background. Control-click (or right-click) this layer and, from the contextual menu, choose Layer from Background. You’ll be asked to name the layer; to keep it simple, just call it Photo. From the main menu, choose Layer -> New Fill Layer -> Solid Color. Stick with the default name (Color Fill 1) and click OK. Choose a color you want to use as a background (this will help illustrate the mask later on), and once again, click OK. You’ll notice that your picture disappears behind the solid color, but we'll fix that now. Return to the Layers panel and drag the Color Fill 1 layer below the Photo layer. This will keep the color fill layer hidden for the time being.
Create a color fill layer and place it below your photo.
Step 2. Outline your item
Click the Paths tab, which usually grouped with Layers and Channels. (If Paths is not visible, choose Window -> Paths.) Once you’ve located the Paths panel, click the menu icon in the top right and choose New Path. Give the path a name—Item Outline, perhaps—and click OK. Now choose the Pen Tool, which you’ll use to create a vector shape that outlines the item (in this case, a coffee mug). Place your first anchor point in a concave corner of the item, and work from there, placing anchor points along the perimeter until you close the shape. Stay as close to the edge of the item as you can; or, if you’re planning to knock out the background, try insetting the path just a touch.
Tips for working with the Pen Tool
1. A single click of the mouse extends a straight line between anchor points.
2. You can create a smooth point (a point with two directional handles) by clicking and dragging the mouse. Direction lines will extend from the anchor point, and while still holding down the mouse button, you can control the shape of the curve.
3. You can create a corner point (any point that introduces a hard angle into your path) by option-clicking the most recent anchor point you created. This reduces the anchor point to a single direction line while the preceding path segment retains its curve. This is very helpful in dealing with concave indents in your item.
Once you’ve fully outlined the item and closed the path, you can tweak it using the Direct Selection Tool. Click and drag any anchor point to reposition it, or tug on the direction lines to tweak the curvature of the path.
Tips for working with the Direct Selection Tool
1. Eliminate direction lines by holding option-command as you click an anchor point.
2. Add (or replace) direction lines by holding option-command as you click the anchor point and drag the mouse.
3. Convert a smooth point to a corner point with independent direction lines by option clicking either handle. The direction lines can then be moved separately.
Use the Pen Tool to create a vector shape that encircles the item you want to mask.
Step 3. Multiple shapes
Certain items will call for more than one vector shape to properly outline. Our coffee mug, for example, requires one shape around its outside edge and another along the inside edge of the handle. Grab the Pen Tool again and use it to create new vector shapes along any inside edges of your item.
Here’s where things get a bit tricky. You’ll need to tell each of these new shapes how you expect them to function. Choose the Path Selection Tool (from the same part of the toolbar as the Direct Selection Tool) and shift-click all interior shapes.
In the options bar, you’ll see four icons (overlapping squares) that let you change how the shapes interact. Click the second icon—Subtract from Shape Area—to tell the interior shapes to punch through the first shape you created. A quick look at the path thumbnail will tell you if you’ve done it right: the item itself should be white, with both interior and exterior transparent areas showing as grey. Assuming all is well, you can go ahead and combine the shapes into a single compound shape. Choose the Path Selection tool and then click the path thumbnail; then, within the options bar, click Combine.
You’ll need a number of shapes to properly outline items such as this mug. Interior shapes must be set to “Subtract from the shape area.”
Step 4. Adding your mask
You’re all set to add the the mask to the photo. First, click the path thumbnail to highlight your new path. Note that the path lines are quite faint and might be difficult to see against some backgrounds; however, as long as you've clicked the path thumbnail and the path itself is highlighted (in the Paths panel), you can move forward. You've spent most of this tutorial working in the Paths panel, but to change the path into a vector mask, you'll have to return to the Layers panel. Choose Layers (Window ->Layers) and click the Photo layer. Call up the Masks panel by choosing Window -> Masks, and in the top right, you’ll see a small pen icon.
Click this icon once to add the vector mask to the layer. The background of the photo will then disappear, leaving you with just the item and the color fill layer you added at the start of this process. The Masks panel also allows you to feather the vector mask, which softens its edges and makes the transition between masked and non-masked areas less harsh. I recommend feathering at least one pixel, but you should experiment for best results.
Once added to the layer, the mask knocks out the background, allowing you to see through to the color fill.
Vector masks in Photoshop CS5 are a bit labor-intensive to build, but they’re also much cleaner and more accurate than those created with pixel selection tools. And remember, you can edit the mask at any time—just click the vector mask thumbnail in the layer, choose the Direct Selection Tool, and then use it to tweak the vector shape.