The results presented here indicate that selection in relation to sex is an underappreciated force in primate brain evolution. Social selection on females and sexual selection on males accounts for significant variation in primate brain architecture. Whereas female sociality is tied to increased cognitive abilities, male sociality and sexual selection on males is not. Instead, sexual selection on males has favored brain structures involved in aggression, sensory motor functions and autonomic functions. This is important because selection pressures acting on the brain have previously almost exclusively been treated as uniform in males and females. Given available data, however, it is impossible to know whether these effects lead to dimorphism or correlated effects in both sexes. Development of new datasets that make use of sexed brains will be needed to address this fundamental question.