I was literally just having this discussion with someone, and specifically mentioned that Thor's hammer would have enough gravitational effect that if he brought it to Earth, the entire solar system would be thrown out of whack by its presence, and the Earth would essentially collapse. Also, him hitting someone with it would atomize them instantly. Which, from that perspective, he doesn't "fly" using the hammer, he just holds it up and the universe moves closer to him.
Neil deGrasse Tyson claimed that Thor's hammer weighed the same as 300 billion elephants, and people have spent a lot of time refuting that calculation since. No matter how you cut it, that hammer is bad news.
Via The Mary Sue.