Current theories hold that all chile peppers originated in central Bolivia, South America. Some species drifted to the Andes mountains of Chile and Peru. Others moved south into southern Bolivia and southern South America. Other species migrated into the Amazon basin and into Central America and Mexico.
All chile peppers come from the Americas. Columbus "discovered" them in the West Indies and brought them back to Europe where they eventually spread to the rest of the world.
Tidbits
Other than culinary and medicinal, chile peppers have some uses that are quite interesting. For example, capsaicin is being added to paint for the bottoms of ships to keep them clear of barnacles. Some home uses for chile powder include putting some cayenne in with the bird seed to keep out the squirrels. Birds have an incredible tolerance for pepper heat and the vitamin A improves their plumage.