Working with YOPPSiteMIP data

Detailed information on the YOPPSiteMIP activity is available in the central YOPP website. In this page, information on how the data are served through the YOPP Data Portal is provided. For information on how to contribute data to this experiment please check out How to Contribute data to YOPPSiteMIP.

Both observations and model output are encoded in NetCDF following the Climate and Forecast conventions. Observations are encoded as Merged Observatory Data File (MODF) while model output is encoded as Merged Model Data File (MMDF). Links to description of the file formats are provided in the central YOPPSiteMIP page mentioned above.

Currently the portal holds many MMDF, but few MODF. There is an ongoing effort to provide updates to the MODF's.

Within the YOPP Data Portal, MODF and MMDF are indexed in a catalogue. This catalogue is available in the menu above using the menu item marked "Browse Available Datasets". In order to filter only YOPPSiteMIP data, please add YOPPSiteMIP to the full text search field. This will give you a listing of the available datasets. However this listing will give you the parents. Most of the datasets have multiple children (e.g. temporal instances of the dataset). If the parent tick box is unchecked more than 33000 datasets will pop up as compared to about 135 dataset as of early 2023. This is important to remember especially if these results are compared with the results in the machine interface which can be used for programmatically accessing the records using e.g. Python (more on this below). That interface does not directly differentiate between parent and child.

All YOPPSiteMIP data are served through a THREDDS Data Server, meaning data can be downloaded as files, but also remotely accessed using the OPeNDAP protocol. In practise this implies that instead of opening a file location on disk, a URL is used to open and access the data. If direct access to the THREDDS Data Server is needed please contact the YOPP Data Portal Team and beware that locations of data within that server can change over time as part of the process of enriching the metadata information.

In order to work programmatically with the YOPP Data Catalogue, machine interfaces are available. Currently OAI-PMH (for downloading the full catalogue) and OGC CSW is recommended. Other interfaces will come. The machine interface to the catalogue is provided through the Arctic Data Centre which provides the long term data preservation for the YOPPSiteMIP data. The machine end påoint is under active development and is being adapted to future technologies so addresses may change over time. Currently it is available at https://adc.csw.met.no/ and some example GET queries for catalogue information is provided below. These can be used in Python, R or other programming languages. Jupyter Notebooks will be added.