I think it depends on what you are looking for. Rock Band is more fun if you have a lot of people playing, and not everyone want to play the guitar. Plus, it has some excellent songs.
Guitar Hero 3 is also excellent, but it is limited in what you can do. However, it is set up more like a game, with bosses and such. So that makes it more fun to play as an individual. It also has some excellent songs.
Soo... I would say I like both equally, in different ways.