I wrote a fairly long article about the rating system on ArmorGames, looking in to how it influences the rating of those people who haven't rated a game.
I can't be bother to re-write it and edit the parts that don't make sense without the picture, so I'll just link it. If you're interested, go here:
Let me know any thought's you may have as perhaps I'm wrong. But I doubt it. Basically I think there should be an intelligence test that people should complete to have their vote counted or perhaps hide the average rating until after the user has cast their vote.