Human serum albumin (HSA) is an abundant plasma protein that binds a remarkably wide range of drugs, thereby restricting their free, active concentrations. The problem of overcoming the binding affinity of lead compounds for HSA represents a major challenge in drug development. Crystallographic analysis of 17 different complexes of HSA with a wide variety of drugs and small-molecule toxins reveals the precise architecture of the two primary drug-binding sites on the protein, identifying residues that are key determinants of binding specificity and illuminating the capacity of both pockets for flexible accommodation. Numerous secondary binding sites for drugs distributed across the protein have also been identified. The binding of fatty acids, the primary physiological ligand for the protein, is shown to alter the polarity and increase the volume of drug site 1. These results clarify the interpretation of accumulated drug binding data and provide a valuable template for design efforts to modulate the interaction with HSA.