Benchmarking bioinformatic virus identification tools using real-world metagenomic data across biomes

Benchmarking overview

Image: Wu, LY., Wijesekara, Y., Piedade, G.J. et al.

The challenge is to find a bioinformatic tool that detects many viral sequences (high sensitivity) while not detecting false positives, i.e., sequences from cellular organisms (specificity). Previous benchmarking studies used testing datasets based on sequences from known viruses, e.g. from RefSeq, yet this leads to overlap between testing and training/reference datasets since most tools are developed based on known viruses. However, the complexity and diversity of naturally occurring viruses is much higher than that of known viruses. To overcome these limitations, we introduced a novel evaluation method using paired real-world virome and total metagenome datasets from the same environment. In this approach, the viral particle-enriched datasets serve as ground truth positives, while the total metagenomic datasets represent ground truth negatives. We removed overlapping sequences from both viral and microbial datasets, as they could represent cellular sequences in the viral fraction that escaped DNase treatment or viral sequences in the cellular fraction (prophages, active infections). This method provides a new perspective of assessment of the tools' performance, guiding researchers in choosing the most appropriate tools for their specific needs.

The quality of our ground truth datasets relies on the purity of the selected metagenomic data. We selected studies with robust procedures to separate viruses from microbes, including physical size fractionation through 0.22 μm filters and DNase treatment of the virome to remove free DNA. Finding these datasets was not easy. In the end, we used a soil datasetExternal link with adequate procedures for virus-microbe separation and an Antarctic marine datasetExternal link from an understudied region in collaboration with Gonçalo Piedade and Corina Brussaard from NIOZ.External link

Our study revealed that all tools detect high-quality viral sequences more easily than short fragments and that machine-learning tools were much better than homology-based tools. Thus, we are currently developing a new CNN tool, Jaeger, to hunt those viruses with even higher precision. For updates, visit https://github.com/Yasas1994/JaegerExternal link. Stay tuned!

Two affiliated PhD researchers led this study: