I just recently finished my DIY screen, and her is how I handled masking.
1st of all the sceen is surounded by black velvet. I read that felt works better, but the velvet looks much better when the room is lit. Masking is accomplished with the aid of YXY. It's a wonderful computer program. It allowed me to setup different aspect ratio's for windvd, with blanking and everything. Now your probably asking why is he doing this? Just show the dvd full screen and mask tht top and bottom. Uh uh, to much work. So what I do is use YXY to move the dvd image to the TOP of the screen. This allows me to mask only the bottom.
For the physical masking I hung peg board on each side the screen. I then have a rod that has velvet hanging/draped from it, which is placed in hooks on the peg board. I have hooks at the proper intervals for both widescreen aspect ratio's and full screen. Any other formats can be accomodated this way, by simply adding hooks. The velvet has a pocket sewn in it so the rod slides through and is covered with velvet. Giving a nice straight edge for masking. Then curtains on the side of the screen cover the peg board board and rod ends, hiding it perfectly.
Hope this helps someone, and is fairly clear in explaing what I have done, as I just woke up.