I think misperception of anywhere outside the US being "unsafe" is a major factor. I remember for my first trip to Asia my parents were both trying to convince me to go someplace within the US or down to the Caribbean.
My mother is still nagging me about worries of bird flu and malaria each time I travel to Thailand. My father goes on about the terrorism in the south of the country.
I feel quite safe when I travel. The only thing I worry about is being ripped off by a tuk tuk driver or having to barter too much with a t-shirt vendor.
Also lets not forget how damn stingy these American companies are with vacation time. I was ready to do a song and dance when I got my 3 weeks this year. Most people only have 2 weeks.