Start-up outside the border?

Anyone have advice on starting up a business outside the US? I've traveled to the caribbean a few times, and feel this part of the world would work better for what I have been working on. However, being in NY, I'm a bit hesitant with making the move. Anyone have any experiences they can share? Advice, pointers, etc?