Well you're the Hawke scion and the Champion of Kirkwall and not to mention killer of thousands of thugs crazy enough to attack you. I guess people hate you less, because you bleed magic less.
In all truth it was probably made like that so you won't stray from the main story too much. I mean how can you save Kirkwall if you're locked up in the Circle? You're the hero everyone looks up to or the brutal villain that everyone fears.
Also Carver ****es me off. Every time I try to do something nice for the guy he starts ranting about "being in my shadow". I wish he'd just grow up.