You can't compare these two games. I mean both of them have their cons and pros. GH3: Overall Longer game, more songs, good for solo gaming or guitar battles... RockBand: Awesome Party game, player customization, etc...
Both of them are really good, but you can't compare them really, they are similar, yet very different.