A new data compression device allows reducing the size of the Gaia mission release by 15 %
The Gaia group of the University of Barcelona (ICCUB-IEEC), in collaboration with Dapcom Data Services, spin-off with the participation of the UB and UPC, has published an alternative copy of the files of the second release for the Gaia mission (Gaia DR2), which gathers data from 1,700 million stars. The copy has been carried out in FAPEC Archiver 19.0, a new format that allows reducing the size of files by 15 % compared to common compressors.
The Gaia group of the University of Barcelona (ICCUB-IEEC), in collaboration with Dapcom Data Services, spin-off with the participation of the UB and UPC, has published an alternative copy of the files of the second release for the Gaia mission (Gaia DR2), which gathers data from 1,700 million stars. The copy has been carried out in FAPEC Archiver 19.0, a new format that allows reducing the size of files by 15 % compared to common compressors.
FAPEC Archiver 19.0 is a professional software offering a fast and high ability of data compression. One of the options it provides is to compress tabula text data (CSV), with the thickness of Gaia DR2 files. To show the functioning of FAPEC, Dapcom converted Gaia DR2ʼs CSV files to this format, which reduced their total size: from the original 554 GB (GZIP format) to 471 GB, that is, FAPEC compresses these CSV files by 15 % compared to GZIP. Other data compressors such as Bzip2, RAR, Zstandard and 7-Zip do not reach this reduction.
Gaia DR2 release, apart from the position of 1,700 million stars, includes data on distance, movement and color for more than 1,300 million stars of the Milky Way and nearby galaxies. Apart from the online documentation -published in open-access on April 25, 2018-, the scientific community was provided with a version of CSV files to be downloaded. This option allows conducting more comprehensive analysis of the data.
Speed, efficiency and adaptability
This new software offers an improvement of data compression that can vary between 10 and 10 % regarding other de facto standards such as ZIP. In several cases, the improvement can be about 50 % or 100%, that is, it can double other systemsʼ compression. The compression factor depends on the kind of data. FAPEC works with binary and text files (such as CSV), temporary series, multi-dimensional data, multi-band pictures, or specific formats like the ones used in genomics.
Another of the main benefits is the speed of the compression procedure, as well as the ability to adapt to the specific case of a specific type of data, such as a scientific experiment, a sensor or a camera. FAPEC, for instance, is currently used to compress GNSS radio occultation data in satellites.
Big amounts of data
FAPEC is a compressor aimed at professionals who work with big amounts of data: scientific experiments, high performance instruments, research environment, and even professional photography. In the field of research, apart from physics and astronomy, it can be applied to marine probes in geosciences, in the field of genomics, or in medicine (magnetic resonances, for instance).