Windows doesn't download anything into the Picture and Fax Viewer, nor does the Picture and Fax Viewer store anything. If you open a picture from within an e-mail, first the picture must be decoded to make it viewable.
Windows has to store the decoded picture somewhere, so it stores it in your temporary files location. The picture and Fax Viewer is the default application for viewing pictures, so that's what's used to display the picture.
Once you open the viewer, clicking the forward and back buttons simply displays any other pictures that happen to be in the same folder as the picture that you chose to view.
If you don't want to see these other files, you can either empty your temporary files folder, or save the picture to a specific location