Do you really think nothing effects your body physiologically when you're asleep?
Exposing infants to loud noises is a form of child abuse, but far more insidious than just hitting them. You are destroying their very means of interacting with the world.
When my cousin was a baby, I was quiet while she slept. Now she's 3 and we're still quiet while she sleeps.
Aside from doing damage, have you any idea how irritable a tired child is?