You don't really need to clear the temporary internet files unless they are starting to impact the amount of space on your hard drive, as long as you have plenty of free hard drive space then regularly clearing them is going to slow down your browsing (your computer will have to download all aspects of every page you visit from the internet again, rather than just getting them from it's local cache)
Additionally clearing the temporary internet files too often is likely to cause huge amounts of file fragmentation over time which will also slow the computer down.
The "you must clear your temporary internet files" that you often hear is a left over from years ago when computer hard drives were much smaller (and more expensive) when freeing up the space was more of a concern than it is these days, with the much larger hard drives now it's just not a problem any more.
Although none of the above is anything to do with your browsing history (which is what the OP is asking about) the only reason to clear that is if you don't want people to be able to check your history to see what you've been looking at.