- did you give it a full charge when you first got it? it really needs to gave a good long first charge
- apple's quoted times assume you just press play on the thing. if you do this, the battery life will be better than if you're constantly pressing skip.
if you're just listening normally to it, and have been fully-charging it, then the battery is probably faulty. my friend had a faulty iPod mini with the same symptoms as you; they changed it with no problem.