Not sure if this is your problem or not. If you are trying to use device control on the VHS side of the deck, it will automatically switch to DV mode because the VHS side has no device control. The only way to capture from the VHS side is to:
1. Load your tape and place the deck in VHS mode.
2. Manually start the tape using the play button on the front of the deck or the remote control.
3. Record the DV stream "on the fly" with whatever software you are using to capture.
I've used this method in Adobe Premiere and it works. However, the audio and video record at different rates, so they lose sync over time. There may be a way to correct this, but I haven't looked into it. Also, since I can't accurately record in or out points, there is no way to archive the project without saving the video file as well.