Funcotator / Clinical Pipeline should move from ExAC to gnomAD

[jonn-smith]

Feature request

Tool(s) or class(es) involved

Funcotator

Description

Currently the data sources for the clinical pipeline work contain ExAC. This must be updated to use gnomAD.

The change will require a new release of the data sources which must be connected to the data source downloader tool.

Additionally, these new data sources must be validated in four ways:

• By visually inspecting the gnomAD source file for correctness.
• By verifying that the source file for gnomAD does not contain special characters.
• By validating that the source file for gnomAD is a valid VCF (assuming it is used VCF format).
• By running a large file and spot checking at least 10 variants for correctness.

---

[jonn-smith]

This will seemingly make the clinical pipeline data sources so large they are completely unusable. Gnomad data for the whole genome is apparently 106Gb, exome data is 16Gb (this would be OK, but is at the upper limit) (http://gnomad.broadinstitute.org/downloads).

@LeeTL1220 - what are your thoughts?

[jonn-smith]

Sent an email to Niall and Alyssa asking about subsetting the gnomAD files.

[jonn-smith]

Also, the gnomAD files are made from B37 data. A liftover to HG38 set exists as well.

[jonn-smith]

One potential plan:

• filter out any variants that do not PASS
• convert to TSV
• only keep the AF/AF_* fields
• use Exome data

[jonn-smith]

We will just read from gnomAD directly using the NIO updates for Funcotator data sources.

Should fix this at the same time as #5428 and #5429.

[jonn-smith]

To facilitate the download of gnomAD records, I removed all info annotations that had nothing to do with allele frequency, leaving behind only the fields that we would want to use for our use case.

The WDLs/jsons are now in the funcotator/scripts/data_sources directory (separate PR).

All variants were kept, even those that were filtered.

The network IO slows down Funcotator significantly, but not enough to make it unusable. For this reason, and partly because gnomAD requires an internet connection, the gnomAD data sources are disabled by default.