to enter the special diagnostic mode, you need to force a hard reset, then after the Apple logo appears you press another special button combination.
here's a quick graphical representation on the special button presses to get your iPod into the hidden diagnostic mode (along with the buttons to get it into disk mode or to reset your iPod)
The buttons you need to press vary depending on the model of iPod.
some further info on the various tests within diagnostic mode can be found here
basically once you press the required buttons, you should hear a beep and see a couple menu choices for auto or manual tests.
NOTE: some of the tests require you to connect a USB and FireWire cable, and the test may seem to be stuck until you do so - therefore I recommend you use the manual mode and only select the appropriate tests.
You can use the diagnostics to check for basic function of I/O, Flash and SDRAM memory, power circuits, scroll wheel and buttons, etc. There's also an LCD display test that fills the screen with solid colors, and you can confirm logic board version number and other system config info.
some models of iPod will let you do a hard disk scan test. But even models lacking the full HD scan test will let you check the SMART status of the drive (of course, assuming it's a hard drive based iPod)
Check out the links I mentioned above for more info.