Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet

Aikaterini Boufea; H.J. Finkers; M.P.W. van Kaauwen; M.R. Kramer; I.N. Athanasiadis

doi:10.1145/3148055.3148060

Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet

Aikaterini Boufea
, H.J. Finkers
, M.P.W. van Kaauwen
, M.R. Kramer
, I.N. Athanasiadis

Research output: Chapter in Book/Report/Conference proceeding › Chapter › Academic › peer-review

5 Citations (Scopus)

Abstract

Big Data has been seen as a remedy for the efficient management of the ever-increasing genomic data. In this paper, we investigate the use of Apache Spark to store and process Variant Calling Files (VCF) on a Hadoop cluster. We demonstrate Tomatula, a software tool for converting VCF files to Apache Parquet storage format, and an application to query variant calling datasets. We evaluate how the wall time (i.e. time until the query answer is returned to the user) scales out on a Hadoop cluster storing VCF files, either in the original flat-file format, or using the Apache Parquet columnar storage format. Apache Parquet can compress the VCF data by around a factor of 10, and supports easier querying of VCF files as it exposes the field structure. We discuss advantages and disadvantages in terms of storage capacity and querying performance with both flat VCF files and Apache Parquet using an open plant breeding dataset. We conclude that Apache Parquet offers benefits for reducing storage size and wall time, and scales out with larger datasets.

Original language	English
Title of host publication	BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies
Publisher	Association for Computing Machinery (ACM)
Pages	219-226
ISBN (Electronic)	9781450355490
DOIs	https://doi.org/10.1145/3148055.3148060
Publication status	Published - 2017
Event	Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies - Austin, United States Duration: 5 Dec 2017 → 8 Dec 2017 Conference number: 4

Conference/symposium

Conference/symposium	Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies
Country/Territory	United States
City	Austin
Period	5/12/17 → 8/12/17

Keywords

Big Data
bioinformatics
variant calling
Hadoop
HDFS
Apache Spark
Apache Parquet

Access to Document

10.1145/3148055.3148060

Tomatula
Boufea, K. (Developer) & Athanasiadis, I. N. (Developer), 2017
Research output: Non-textual form › Software

Open Access

Experimental results of "Managing variant calling datasets the big data way"
Boufea, K. (Creator) & Athanasiadis, I. (Creator), Zenodo, 22 May 2017
DOI: 10.5281/zenodo.582145
Dataset

Cite this

Boufea, A., Finkers, H. J., van Kaauwen, M. P. W., Kramer, M. R., & Athanasiadis, I. N. (2017). Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet. In BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (pp. 219-226). Association for Computing Machinery (ACM). https://doi.org/10.1145/3148055.3148060

@inbook{b9fb90d9894244c9a49dc5343aa8ae23,

title = "Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet",

abstract = "Big Data has been seen as a remedy for the efficient management of the ever-increasing genomic data. In this paper, we investigate the use of Apache Spark to store and process Variant Calling Files (VCF) on a Hadoop cluster. We demonstrate Tomatula, a software tool for converting VCF files to Apache Parquet storage format, and an application to query variant calling datasets. We evaluate how the wall time (i.e. time until the query answer is returned to the user) scales out on a Hadoop cluster storing VCF files, either in the original flat-file format, or using the Apache Parquet columnar storage format. Apache Parquet can compress the VCF data by around a factor of 10, and supports easier querying of VCF files as it exposes the field structure. We discuss advantages and disadvantages in terms of storage capacity and querying performance with both flat VCF files and Apache Parquet using an open plant breeding dataset. We conclude that Apache Parquet offers benefits for reducing storage size and wall time, and scales out with larger datasets.",

keywords = "Big Data, bioinformatics, variant calling, Hadoop, HDFS, Apache Spark, Apache Parquet",

author = "Aikaterini Boufea and H.J. Finkers and \{van Kaauwen\}, M.P.W. and M.R. Kramer and I.N. Athanasiadis",

year = "2017",

doi = "10.1145/3148055.3148060",

language = "English",

pages = "219--226",

booktitle = "BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies",

publisher = "Association for Computing Machinery (ACM)",

address = "United States",

note = "Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies ; Conference date: 05-12-2017 Through 08-12-2017",

}

Boufea, A, Finkers, HJ, van Kaauwen, MPW, Kramer, MR & Athanasiadis, IN 2017, Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet. in BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies. Association for Computing Machinery (ACM), pp. 219-226, Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, Austin, United States, 5/12/17. https://doi.org/10.1145/3148055.3148060

Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet. / Boufea, Aikaterini; Finkers, H.J.; van Kaauwen, M.P.W. et al.
BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies. Association for Computing Machinery (ACM), 2017. p. 219-226.

Research output: Chapter in Book/Report/Conference proceeding › Chapter › Academic › peer-review

TY - CHAP

T1 - Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet

AU - Boufea, Aikaterini

AU - Finkers, H.J.

AU - van Kaauwen, M.P.W.

AU - Kramer, M.R.

AU - Athanasiadis, I.N.

N1 - Conference code: 4

PY - 2017

Y1 - 2017

N2 - Big Data has been seen as a remedy for the efficient management of the ever-increasing genomic data. In this paper, we investigate the use of Apache Spark to store and process Variant Calling Files (VCF) on a Hadoop cluster. We demonstrate Tomatula, a software tool for converting VCF files to Apache Parquet storage format, and an application to query variant calling datasets. We evaluate how the wall time (i.e. time until the query answer is returned to the user) scales out on a Hadoop cluster storing VCF files, either in the original flat-file format, or using the Apache Parquet columnar storage format. Apache Parquet can compress the VCF data by around a factor of 10, and supports easier querying of VCF files as it exposes the field structure. We discuss advantages and disadvantages in terms of storage capacity and querying performance with both flat VCF files and Apache Parquet using an open plant breeding dataset. We conclude that Apache Parquet offers benefits for reducing storage size and wall time, and scales out with larger datasets.

AB - Big Data has been seen as a remedy for the efficient management of the ever-increasing genomic data. In this paper, we investigate the use of Apache Spark to store and process Variant Calling Files (VCF) on a Hadoop cluster. We demonstrate Tomatula, a software tool for converting VCF files to Apache Parquet storage format, and an application to query variant calling datasets. We evaluate how the wall time (i.e. time until the query answer is returned to the user) scales out on a Hadoop cluster storing VCF files, either in the original flat-file format, or using the Apache Parquet columnar storage format. Apache Parquet can compress the VCF data by around a factor of 10, and supports easier querying of VCF files as it exposes the field structure. We discuss advantages and disadvantages in terms of storage capacity and querying performance with both flat VCF files and Apache Parquet using an open plant breeding dataset. We conclude that Apache Parquet offers benefits for reducing storage size and wall time, and scales out with larger datasets.

KW - Big Data

KW - bioinformatics

KW - variant calling

KW - Hadoop

KW - HDFS

KW - Apache Spark

KW - Apache Parquet

U2 - 10.1145/3148055.3148060

DO - 10.1145/3148055.3148060

M3 - Chapter

SP - 219

EP - 226

BT - BDCAT '17 Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies

PB - Association for Computing Machinery (ACM)

T2 - Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies

Y2 - 5 December 2017 through 8 December 2017

ER -

Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet

Abstract

Conference/symposium

Keywords

Access to Document

Fingerprint

Research output

Tomatula

Datasets

Experimental results of "Managing variant calling datasets the big data way"

Cite this