Vector Mask Painting
By MissNysha
899 Favourites80 Comments28K Views
This is a guide for those who want some more in depth clarification about my fav digital coloring technique: vector mask painting, which combines vectoring and tablet painting in a single layer. This time, you don't have to hunt through my other tutorials to try and understand the concept! 
Intermediate level, as it assumes some photoshop vectoring knowledge. See my previous Ultimate vectoring guide for further details: [link]
Also check out my Hair Coloring Tutorial to see me using vector mask painting in full! [link]
Made in photoshop cs2, but the technique still applies to other versions.
Enjoy!
All parts made by me - No stealing or ripping any of its content.
Intermediate level, as it assumes some photoshop vectoring knowledge. See my previous Ultimate vectoring guide for further details: [link]
Also check out my Hair Coloring Tutorial to see me using vector mask painting in full! [link]
Made in photoshop cs2, but the technique still applies to other versions.
Enjoy!
All parts made by me - No stealing or ripping any of its content.
Image details
Image size
1000x6229px 1.35 MB
Published:
© 2011 - 2021 MissNysha
Comments82
Join the community to add your comment. Already a deviant? Log In
This is awesome! Thank you!
Glad you like it!
Omg you are perfect! *-*
Hey there, sorry for bothering again. I have a question, I've just tried to group all the layers and add a group vector mask to it but when I was about to apply the gradient tool, it said "Could not use the gradient tool because the pixels in a group cannot be edited without merging the group"
Could you please help me with that?
Thank you.
Could you please help me with that?
Thank you.
Please PM me a note with a screenshot please.
Thank you!! This tutorial is really helpful, but still don't know how to make the colors so soft like yours. I have a question, when you paint a character, do you color the skin in a single layer (or vector mask) or make separated layers?
Thanks, again
Thanks, again
I make a new layer for each color so they can blend into each other softly (I mess around with the opacities and blending modes). I also use layer masks to erase around edges to "mold" colors into a nice soft shape too (with a large airbrush). I may merge these layers together later if the file size gets too big. All layers are grouped within a vector mask to keep me organized and their edges nice and clean.
I see. Oh the problem is, I don't know how to erase the edges, if you don't mind, [link] please take a look at the picture I'm working. May I know how to erase that part?
I'm sorry for my English.
Thank you very much.
![[link]](https://www.deviantart.com/users/outgoing?http://i1179.photobucket.com/albums/x382/bluelagoon12/part_zps6a878b63.png){kind=link}
I'm sorry for my English.
Thank you very much.
Hmm let's see...
This could be caused by several things. Depends on how your layers are stacked too. If you have the same vector shape stacked in several layers, the outside edge develops a pixelly outline that can't be erased unless you group all the similar layers under a single mask (example of the bow at the bottom of the tutorial).
Also, if you used the paint bucket to fill in a pen tool path, and rasterized it, then the edges may become pixellated like this. I'd have to know what you did to make this, or see your layers to really give you a good answer.
This could be caused by several things. Depends on how your layers are stacked too. If you have the same vector shape stacked in several layers, the outside edge develops a pixelly outline that can't be erased unless you group all the similar layers under a single mask (example of the bow at the bottom of the tutorial).
Also, if you used the paint bucket to fill in a pen tool path, and rasterized it, then the edges may become pixellated like this. I'd have to know what you did to make this, or see your layers to really give you a good answer.
I've figured it out, it's because I didn't erase the layer under the vector mask. Thanks for answering my question.
Oh I see, I'm new at vector mask so I still don't know how to use it. I want to make soft shading for the skin, clothes, and hair, but I don't know how to make them without making them pixellated.
Should I use brush or gradient tool? And is it okay to use smudge tool? I'm afraid the edges may become pixellated if I use it.
Ah I'm so sorry I don't have the file anymore, I was only practicing and I didn't save it.
Oh I see, I'm new at vector mask so I still don't know how to use it. I want to make soft shading for the skin, clothes, and hair, but I don't know how to make them without making them pixellated.
Should I use brush or gradient tool? And is it okay to use smudge tool? I'm afraid the edges may become pixellated if I use it.Ah I'm so sorry I don't have the file anymore, I was only practicing and I didn't save it.
Good! Glad you figured it out.
I use both a large airbrush and the gradient tool. I use the gradient tool to apply lighting effects mostly though. I do not use the smudge tool on skin - only the airbrush, because the smudge tool can make some weird effects that do not make skin smooth. I use the smudge tool on hair though to paint the details.
I use both a large airbrush and the gradient tool. I use the gradient tool to apply lighting effects mostly though. I do not use the smudge tool on skin - only the airbrush, because the smudge tool can make some weird effects that do not make skin smooth. I use the smudge tool on hair though to paint the details.
So you only use airbrush and apply gradients for shading? If you don't mind, may I know how many layers do you usually use for skin color? And do you use tablet to paint?
That's right. Maybe about 10 layers or so, maybe more sometimes. And yes I use a Wacom tablet.
View all replies
Very interesting technique. It reminds me of what I do in Illustrator. In CS6 you get better painting brushes and vector masks really help a lot there, but I never thought about trying them in Photoshp..I'll definitely have to try that out some time! Thank you for taking the time to write this tutorial!
Glad to help! I do not have CS6 yet, but I am definitely intrigued by what the new Illustrator can do! Especially if the upgrade does not have a RAM limit, like how Photoshop CS5 currently runs (one of my favorite new features).
I definitely have to check that out! I switched directly from CS4 to CS6, so I don't know much about the detailed changes. But I'm sure you'll enjoy the new features of CS6.. it's a blast! ^^
This is quite cool.
I make a lot of vector graphics in Flash, and would like to try out painting within those vector shapes, but I suppose then I'll have to settle for the magic wand tool, I hope that works fairly well without getting weird looking edges.
I make a lot of vector graphics in Flash, and would like to try out painting within those vector shapes, but I suppose then I'll have to settle for the magic wand tool, I hope that works fairly well without getting weird looking edges.
Actually I do not recommend the magic wand tool, because it selects a certain color base, with a lot of random fuzziness that results in pixellated edges. (unless you're only selecting 1 out of 2 colored areas) Using the pen tool to create vector masks is the solution to make the cleanest edges possible. It shouldn't be too much different from vectoring shapes in flash.
To rate the tools available, my selection/edges hierarchy goes like this:
Group vector mask > painted group layer mask > lock transparent pixels (when working on a single layer only) > polygon lasso tool > magic wand.
So you can see I only use the wand tool as a last resort. Although it may be the fastest, the randomness you can't control on its edges really brings the overall image quality down, which other viewers will easily notice. Plus once you make a mask, you only need to create it once, as opposed to having to reselect the area you want over and over, so in the long run it saves time too.
Is there a reason why you must settle on the wand tool?
To rate the tools available, my selection/edges hierarchy goes like this:
Group vector mask > painted group layer mask > lock transparent pixels (when working on a single layer only) > polygon lasso tool > magic wand.
So you can see I only use the wand tool as a last resort. Although it may be the fastest, the randomness you can't control on its edges really brings the overall image quality down, which other viewers will easily notice. Plus once you make a mask, you only need to create it once, as opposed to having to reselect the area you want over and over, so in the long run it saves time too.
Is there a reason why you must settle on the wand tool?
Thanks for the response!
I've found that if you use the magic wand tool to select an area, you can then click on "Refine Edge" and then "Decontaminate Colors" and it will create a layer mask for your selection so that you won't have to re-select anything later. But you're right in that it's a bit messy.
Well the way vectors work in Flash is quite different from Photoshop or Illustrator and I find it more intuitive and flexible. It took a long time but I managed to figure out how to get the vectors from Flash to Photoshop. I had to export the Flash graphic as an fxg file, open it in Illustrator and save as an ai file, then open that in Fireworks and save as a psd file that when opened in Photoshop has every shape as a color fill and a vector mask.
The only issue with this is that if you have two adjacent vector shapes that share the same border edge, which you often have in Flash, you end up with a thin transparent line in between the two shapes in Photoshop, as if they're not quite touching. But I suppose I can just make sure shapes are always overlapping each other.
I've found that if you use the magic wand tool to select an area, you can then click on "Refine Edge" and then "Decontaminate Colors" and it will create a layer mask for your selection so that you won't have to re-select anything later. But you're right in that it's a bit messy.
Well the way vectors work in Flash is quite different from Photoshop or Illustrator and I find it more intuitive and flexible. It took a long time but I managed to figure out how to get the vectors from Flash to Photoshop. I had to export the Flash graphic as an fxg file, open it in Illustrator and save as an ai file, then open that in Fireworks and save as a psd file that when opened in Photoshop has every shape as a color fill and a vector mask.
The only issue with this is that if you have two adjacent vector shapes that share the same border edge, which you often have in Flash, you end up with a thin transparent line in between the two shapes in Photoshop, as if they're not quite touching. But I suppose I can just make sure shapes are always overlapping each other.
very nice
i have one question though.
Why do you have a layer mask on all of your layers or more specifically what do you use it for?
i have one question though.
Why do you have a layer mask on all of your layers or more specifically what do you use it for?
I use layer masks separately from vector masks for non-destructive erasing. Basically I can soften and fade parts without permanently erasing them - and if I make any mistakes I can just paint back with white to bring parts back.
interesting, I may just try that.
Thanks!
Thanks!
Hi What my question is if i did some work @ photoshop and drag my work into illustrato and vise versa is this the efficient way in terms of getting good quarlity and making working easier? Or do i need to save in some sort of format like like jpeg. or something to make the process smooth and correct- to one software into the other.
What's the difference betwee eps and ai. ?
Sorry about the lame question but im a complete beginner in the technical size of things.
What's the difference betwee eps and ai. ?
Sorry about the lame question but im a complete beginner in the technical size of things.
Thank you for the tips
Join the community to add your comment. Already a deviant? Log In