Right there witih ya.
Five stars from several clients - then BAM - one client rates 4-stars across the board, and my reputation is tarnished.
Have you posted rebuttals to those clients' ratings? I just posted my first rebuttal - though I don't think it shows up on the oDesk site for others looking over my profile to see.
Or have you asked thos low rating clients to provide you feedback as to THEIR dissatisfaction - so you can make sure you never provide less than 5-star service again (as if you did)? Worth a try anyway. Gotta do it within a certain amount of time after contract end though.
I hope you gain enough clients that it brings your rep back up, friend!