Principal component analysis is a key machine learning technique for reducing the number of dimensions in a dataset, but new research shows that its output can be inconsistent and unreliable.
What’s new: Eran Elhaik at Lund University assessed the use of principal component analysis (PCA) in population genetics, the study of patterns in DNA among large groups of people. Working with synthetic and real-world datasets, he showed that using PCA on substantially similar datasets can produce contradictory results.
Key insight: PCA has characteristics that prior research proposed as risk factors for unreproducible scientific research. For instance, it tends to be used to generate hypotheses, accommodates flexible experimental designs that can lead to bias, and is used so frequently — in population genetics, at least — that many conclusions are likely to be invalid on a statistical basis alone. Studies of population genetics use PCA to reduce the dimensions of raw genetic data and cluster the reduced data to find patterns. For example, some studies assume that the closer different populations are clustered, the more likely they share a common geographical origin. If PCA alters the clusters in response to minor changes in the input, then the analysis doesn’t necessarily reflect genetic relationships.
How it Works: The author tested the consistency of PCA-based analyses using a synthetic dataset and three real-world human genotype datasets.
- To create the synthetic dataset, the author modeled a simplified scenario in which people expressed one of three genes (signified by the colors red, green, and blue) or none (black). He assigned the vector [1,0,0] to each red individual, [0,1,0] to green, [0,0,1] to blue, and [0,0,0] to black. He used PCA to reduce the vectors into two dimensions and plotted the results on a 2D graph, so each group formed a cluster.
- He used the real-world datasets to analyze 12 common tasks in population genetics, such as determining the geographical origin of population groups.
- He ran several experiments on the synthetic and real-world data, manipulating the proportions of different populations, processing the data via PCA, and plotting the results.
Results: Clustering a dataset that included 10 red, green, and blue examples and 200 black ones, the black cluster was roughly equidistant from the red, green, and blue clusters. However, with five fewer blue individuals, the black cluster was much closer to the blue cluster, showing that PCA can process similar data into significantly different cluster patterns. Using real-world data, the author replicated a 2009 study that used PCA to conclude that Indians were genetically distinct from European, Asian, and African populations. However, when he manipulated the proportion of non-Indian populations, the results suggested that Indians descend from Europeans, East Asians, or Africans. Overall, PCA-based analysis of the real-world datasets fluctuated arbitrarily enough to cast doubt on earlier research conclusions.
Why it matters: This study demonstrates that PCA-based analyses can be irreproducible. This conclusion calls into question an estimated 32,000 to 216,000 genetic studies that used PCA as well as PCA-based analyses in other fields.
We’re thinking: PCA remains a useful tool for exploring data, but drawing firm conclusions from the resulting low-dimensional visualizations is often scientifically inappropriate. Proceed with caution.