Current theories hold that all chile peppers originated in central Bolivia, South America. Some species drifted to the Andes mountains of Chile and Peru. Others moved south into southern Bolivia and southern South America. Other species migrated into the Amazon basin and into Central America and Mexico.
All chile peppers come from the Americas. Columbus "discovered" them in the West Indies and brought them back to Europe where they eventually spread to the rest of the world.
Tidbits
The burning sensation from eating chile peppers is caused by a group of compounds called capsaicinoids. It is often simply called capsaicin. These compounds are concentrated in the white veins in the pepper that hold the seeds. The capsaicinoids in their pure form are a white, crystalline powder highly irritating to skin