HDF5 as the SwissFEL Data File Format
- Jocelyne Zofia
- Mar 1, 2022
- 1 min read
Developments of the Software Infrastructure for High-Level Beam Dynamics Applications at SwissFEL, an X-ray free-electron laser
Paul Scherrer Institut - Internship, October 2018
About the SwissFEL
What is the SwissFEL:
https://www.psi.ch/de/swissfel/about-swissfel
Module for easier handling of SwissFEL hdf5 data files: https://github.com/paulscherrerinstitute/sf_datafile
“A user interface is like a joke. If you have to explain it, it’s not that good”. — Martin Leblanc
What is HDF5 :
Whenever we want to handle big data, namely large numerical data in Python such as datasets (which are array-like objects), hierarchically organised groups (that is containers storing datasets and other groups), (user-defined) metadata “attributes” (which are attachable to datasets and groups), the 'HDF5' mechanism allows a structured and self-describing format and storage allocation.
The HDF5 software library was developed at the National Centre for Supercomputing applications (NCSA).
HDF5 supports n-dimensional datasets, where each dataset element may be a complex object, and besides allowing the storage of heterogeneous data, large numerical arrays of homogenous type and data models that can be organized hierarchically, it runs on different computational platforms taking into account cross-platform issues. Furthermore, it provides high-level API with C, C++, Fortran 90, and Java interfaces.
Organisation of HDF5: users can build specific “application formats” that organise data, to fix a ”format-within-a-format” to specify what arrangements to use for information storage.
HDF5 in Python: the Pythonic interface to HDF5 binary data format is given by the h5py package, and it maps HDF5 type system onto NumPy dtypes (making it easy to interchange data with NumPy); furthermore it contains high-level wrappers for HDF5 object files, groups, datasets, attributes. the h5py package is basically used for talking to HDF5.
HDF5 is simply the version n. 5 of HDF*.
More on the topic of HDF5 in Python:
Comments