I've just started to have this problem come and go too.
After playing around for a while, I discovered that this only happens when making calls when the N80 has been slid 'open'. When the handset is 'closed', the speaker works fine.
I tried making a call with the handset closed and then slid the slider up whilst the call was active - lo and behold as I slid the phone open, the sound disappeard. Makes me feel like this is a dodgy connection rather than a firmware issue. If this is the case, and it's not an isolated case, it doesn't bode well for the durability of the N80

(mine's about 5 months old)
You can always tell when you're not going to have a sucessful call as the keypad tones do not work either when you press any buttons with the slider open (they presumably use the same speaker).
I'll be off to the Orange shop for my second replacement N80... (on the first one the front camera refused to work).