The profile of (S)-nicotine metabolism in human liver microsomes was examined at concentrations approaching in vivo conditions (10 microM). At such concentrations, no (S)-nicotine N-1'-oxygenation was seen, and thus C-oxidation to the (S)-nicotine delta 1',5'-iminium ion was the sole product observed in the metabolic profile in the presence of the human liver microsomes. For simplicity of analysis, the (S)-nicotine delta 1',5'-iminium ion formed was converted to (S)-cotinine in the presence of exogenously added aldehyde oxidase. To explain the lack of (S)-nicotine N-1'-oxygenation at low (S)-nicotine concentrations, inhibition of flavin-containing monooxygenase (FMO) activity by (S)-cotinine was examined. Although (S)-cotinine was observed to inhibit pig FMO1 (Ki = 675 microM), partially purified cDNA-expressed adult human liver FMO3 was not inhibited by (S)-cotinine. We therefore concluded that the kinetic properties of the nicotine N'- and C-oxidases were responsible for the metabolic product profile observed. Kinetic constants were determined for individual human liver microsomal preparations from low (10 microM) and high (500 microM) (S)-nicotine concentrations by monitoring (S)-cotinine formation with HPLC. The mean Kmapp and Vmax for formation of (S)-cotinine by the microsomes examined were 39.6 microM and 444.3 pmol.min-1.(mg protein)-1, respectively. The formation of (S)-cotinine was strongly dependent on the previous drug administration history of each subject, and among the highest rates for (S)-cotinine formation were those of the barbiturate-pretreated subjects. The rate of (S)-cotinine formation at low (10 microM) concentration correlated well with immunoreactivity for cytochrome P450 2A6 (r = 0.89). In vitro-in vivo correlation of the results suggests that the low amount of (S)-nicotine N-1'-oxygenation and the large amount of (S)-cotinine formed in human smokers (i.e. 4 and 30% of a typical dose, respectively) are determined primarily by the kinetic properties of the human monooxygenase enzyme systems. However, additional non-hepatic monooxygenase(s) contributes to (S)-nicotine metabolism.