I prefer Handbrake for encoding. One of the extremely few that will encode to an MKV container and gives you some extreme fine tuning controls over the quality. Very simple output choices will showing the full potential of what H264 can really do.
But yeah, encoding videos in higher quality usually means having to learn what containers are and how codecs work.
Video editing/capturing/encoding the one of the most CPU intensive things out there (beats gaming by a long shot). My CPU goes up to 70 degrees on full load but I have an aftermarket heatsink installed (the original allowed it to go up to a dangerous 95). You should think about investing in a better CPU heatsink if you're worried about your CPU cooking.