Domain: Energy & Environment
Detecting anomalies in large datasets of weather forescast data.
UBIMET is the globally leading provider of meteorological forecast systems, information and tailor-made services. With all its services, UBIMET places the highest emphasis on quality and meeting the customers’ demands. Around the clock, 365 days a year, data, weather models, radar and satellite images are analysed and interpreted by high-performance computing systems and specially trained experts. The precise meteorological information provided by UBIMET enables companies to work faster, in a more efficient and cost-conscious manner, and more successfully in their sector than the competition.
Anomaly detection is of interest in many industry areas. Thus, novel solutions on how to detect changes in large datasets is a challenge that requires efficient methods. While the problem can easily be transferred into other domains, this specific challenge focuses on weather forecast data.
DataThe challenge has the following sample datasets available for download
The sample data provided in the data catalogue contains 2 forecast model runs, each containing 320 parameters for 66 forecast hours to allow you to get familiar with the data format and parameters. The full data set will contain 2 historic runs per day with a forecast horizon of 66 hours for up to 2.5 years (~13 TB/year).
- Identify: (1) datasets, (2) affected parameters and (3) time-step from which a deviation in their parameter characteristics from previous datasets (anomalies or step-changes) can be observed.
- The system needs to be able to detect changes in near-real-time to inform subsequent systems of issues to prevent uncontrolled propagation. A live system will need to be able to process at least 150 MB/min with 320 parameters.
- UBIMET, therefore encourages solutions that make use of scalable parallel algorithms and infrastructure.