Right-click on any video file. Select 'Properties', then 'Open with' and 'Choose Program'. Make sure that there's a tick in the box marked 'Always use this program to open these files'. Select 'Windows Media Player' and click 'OK'.
If you've got files in several different formats (e.g. mpeg, avi, wmv, etc), you'll need to do this for one file of each type.
Alternatively, you can do the same thing from within the Control Panel. (I use Windows Millennium so these instructions might not exactly match what you're using but you should be able to work it out):
Access the control Panel (Start > Settings > Control Panel on my PC)
Double-click 'Folder options'.
Click on the 'File types' tab.
Select a relevant file type (e.g. AVI)
Click 'Change'
Choose Windows Media Player
Click 'OK'
Repeat for all other required file types.
Chris