Description:

From 20GB to 100TB: a journey on the metagenomic road

Dr. Zhong Wang, Computational Biologist at Lawrence Berkeley Lab, program head for genome R&D analysis at DOE Joint Genome Institute (JGI), and adjunct professor at University of California, Merced

Abstract: Metagenomics has revolutionized our understanding of microbial functions, ecology, and evolution. Unraveling the complexity of environmental microbial communities demands numerous gigabases, or even terabases, of sequence data, thereby posing extraordinary computational challenges associated with data analysis. In this talk, I will chronicle our journey, navigating through the challenges presented by initially modest 20GB datasets to our current capability of handling substantial 100TB experiments. This journey entailed more than just augmenting storage or computational power; it also involved innovative thinking, experimentation with hardware scaling solutions, and the development of scalable software tools designed for immense datasets. By sharing our experiences, successes, and failures, this presentation aims to offer insights and strategies to fellow biologists and bioinformaticians navigating the rapidly expanding sea of metagenomic data.