Abstract
| Original language | English |
|---|---|
| Title of host publication | BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies |
| Publisher | ACM |
| Pages | 219-226 |
| ISBN (Electronic) | 9781450355490 |
| DOIs | |
| Publication status | Published - 2017 |
| Event | Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies - Austin, United States Duration: 5 Dec 2017 → 8 Dec 2017 Conference number: 4 |
Conference
| Conference | Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies |
|---|---|
| Country | United States |
| City | Austin |
| Period | 5/12/17 → 8/12/17 |
Fingerprint
Keywords
- Big Data
- bioinformatics
- variant calling
- Hadoop
- HDFS
- Apache Spark
- Apache Parquet
Cite this
}
Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet. / Boufea, Aikaterini; Finkers, H.J.; van Kaauwen, M.P.W.; Kramer, M.R.; Athanasiadis, I.N.
BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies. ACM, 2017. p. 219-226.Research output: Chapter in Book/Report/Conference proceeding › Chapter › Academic › peer-review
TY - CHAP
T1 - Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet
AU - Boufea, Aikaterini
AU - Finkers, H.J.
AU - van Kaauwen, M.P.W.
AU - Kramer, M.R.
AU - Athanasiadis, I.N.
PY - 2017
Y1 - 2017
N2 - Big Data has been seen as a remedy for the efficient management of the ever-increasing genomic data. In this paper, we investigate the use of Apache Spark to store and process Variant Calling Files (VCF) on a Hadoop cluster. We demonstrate Tomatula, a software tool for converting VCF files to Apache Parquet storage format, and an application to query variant calling datasets. We evaluate how the wall time (i.e. time until the query answer is returned to the user) scales out on a Hadoop cluster storing VCF files, either in the original flat-file format, or using the Apache Parquet columnar storage format. Apache Parquet can compress the VCF data by around a factor of 10, and supports easier querying of VCF files as it exposes the field structure. We discuss advantages and disadvantages in terms of storage capacity and querying performance with both flat VCF files and Apache Parquet using an open plant breeding dataset. We conclude that Apache Parquet offers benefits for reducing storage size and wall time, and scales out with larger datasets.
AB - Big Data has been seen as a remedy for the efficient management of the ever-increasing genomic data. In this paper, we investigate the use of Apache Spark to store and process Variant Calling Files (VCF) on a Hadoop cluster. We demonstrate Tomatula, a software tool for converting VCF files to Apache Parquet storage format, and an application to query variant calling datasets. We evaluate how the wall time (i.e. time until the query answer is returned to the user) scales out on a Hadoop cluster storing VCF files, either in the original flat-file format, or using the Apache Parquet columnar storage format. Apache Parquet can compress the VCF data by around a factor of 10, and supports easier querying of VCF files as it exposes the field structure. We discuss advantages and disadvantages in terms of storage capacity and querying performance with both flat VCF files and Apache Parquet using an open plant breeding dataset. We conclude that Apache Parquet offers benefits for reducing storage size and wall time, and scales out with larger datasets.
KW - Big Data
KW - bioinformatics
KW - variant calling
KW - Hadoop
KW - HDFS
KW - Apache Spark
KW - Apache Parquet
U2 - 10.1145/3148055.3148060
DO - 10.1145/3148055.3148060
M3 - Chapter
SP - 219
EP - 226
BT - BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies
PB - ACM
ER -