Functional neuroimaging studies have implicated the fusiform gyri (FG) in structural encoding of faces, while event-related potential (ERP) and magnetoencephalography studies have shown that such encoding occurs approximately 170 ms poststimulus. Behavioral and functional neuroimaging studies suggest that processes involved in face recognition may be strongly modulated by socially relevant information conveyed by faces. To test the hypothesis that affective information indeed modulates early stages of face processing, ERPs were recorded to individually assessed liked, neutral, and disliked faces and checkerboard-reversal stimuli. At the N170 latency, the cortical three-dimensional distribution of current density was computed in stereotactic space using a tomographic source localization technique. Mean activity was extracted from the FG, defined by structure-probability maps, and a meta-cluster delineated by the coordinates of the voxel with the strongest face-sensitive response from five published functional magnetic resonance imaging studies. In the FG, approximately 160 ms poststimulus, liked faces elicited stronger activation than disliked and neutral faces and checkerboard-reversal stimuli. Further, confirming recent results, affect-modulated brain electrical activity started very early in the human brain (approximately 112 ms). These findings suggest that affective features conveyed by faces modulate structural face encoding. Behavioral results from an independent study revealed that the stimuli were not biased toward particular facial expressions and confirmed that liked faces were rated as more attractive. Increased FG activation for liked faces may thus be interpreted as reflecting enhanced attention due to their saliency.