The Navy is actually capable of sailing through the Calm Belt using ships whose hulls were plated in Sea Prism Stone, which (usually) prevents the Sea Kings that live there from noticing them. When Koby meets up with the Straw Hats in Water 7, he reveals that this is how he got into the Grand Line on Garp's ship, though the Navy keeps this ability under wraps for the strategic advantage it provides them. There are a few other people who have been shown crossing through the Calm Belts too: the island of Amazon Lily is located within one of them, but Boa Hancock and the Kuja tribe are able to move through it freely, probably because their ship is pulled by those two giant sea serpents. And Rayleigh was able to swim freely to Amazon Lily because he has balls the size of planets. Given that, there doesn't seem to be any reason why someone couldn't just sail straight to Laugh Tale from the outside if it was simply a matter of geography. The fact that no one has up to this point strongly suggests that there's something else at work, but we don't know what that is yet.
...well, maybe I already know what it is and you don't. Or maybe I'm in the dark too. Suspense!