The reasons higher grade gems cost more mana in this case is because when higher grade gems are tossed, the number of monsters per wave increases. A level 2 grade gem enrage increases the number of monsters more than a level 1 grade gem enrage. The score per monster is also affected, but in a more subtle way.
My advice is to find a nice balance between finger aches, experience gain, net mana gain (You'll want to get more mana out of the mobs than spending on enraging), and ability to defeat the waves themselves. There are enraging formulae out there if you really want to get specific, but I've never been such a hardcore Gemcraft player myself.
You'll have better luck posting in the bigger Gemcraft threads in the Flash Game/Walkthrough sections where those players frequent.
