Need help understanding the daily statistic in preprocessor

[rswamina]

I am trying to compute means for every day across a couple of decades for a given climate model. My recipe looks like this :

# ESMValTool
# recipe_test_daily_projections.yml
---
documentation:
  description: |
    Recipe that calculates daily means.

  title: Recipe to calculate daily means.

  authors:
    - swaminathan_ranjini


  maintainer:
    - swaminathan_ranjini

  projects:
    - ukesm

preprocessors:

  preproc_daily_data_means:
    daily_statistics:
      operator: mean

datasets:

  - {dataset: UKESM1-0-LL, project: CMIP6, exp: [historical,ssp585], ensemble: r1i1p1f2, start_year: 2063, end_year: 2083, grid: gn} 


diagnostics:

  daily_means:
    description: Daily mean calculations
    themes:
      - phys
    realms:
      - atmos
    variables:
      tas:
        mip: day
        preprocessor: preproc_daily_data_means

    scripts: null

I want to get 360 values with the mean for day 1 across the 21 years and 1 mean for day 2 and so on until day 360? What should I expect if I add another 21 years worth of data from a different ensemble member? What is a good solution for this in a way to maximize preprocessor use? How does the daily_statistics preprocessor function work here?

[rswamina]

I think I can self-answer the above question now. The daily mean statistic aggregates by day and year so I need to have two datasets spanning the same years for averaging across them.

[zklaus]

I'm not completely sure, but I think there may be confusion here.

Daily statistics will aggregate sub-daily data to daily data, i.e. it will take, for example, 6-hourly data at hours 0, 6, 12, and 18 and calculate the mean as their sum divided by 4. This is in line with the other statistics preprocessors for time (hourly, monthly, seasonal, annual, and decadal).

It sounds like what you are trying to do is take daily data across many years and turn this data into daily data that is representative of the data in the entire timespan, i.e. average all first of January across all the years etc. This is called a climatology and can be calculated with the climate_statistics preprocessor.

[rswamina]

Thanks @zklaus , this is useful to know. I hadn't used this climate statistic before. Will this apply if I have 10 year periods with different start and end years for each data set?

[zklaus]

Good question.

Without having it tried, I think in some sense yes: If you use the climate_statistics preprocessor with period: day on several datasets that have different timespans, you should get one result per dataset that is a daily climatology, i.e. for each dataset you will get a statistic for every day of the year that is representative of the entire timespan.

They may have different coordinate bounds and I am not sure how that will interact with subsequent preprocessors (for example a multi-model mean), but I guess that will be a different question.

Side note: In these Github discussions, you can write "answers" in the big field below, or you can write "replies" in the little field below each existing answer. In this case, I think it would have been better to write your followup as a reply to my earlier answer. That way, all the relevant information that clarifies the answer but doesn't stand as an independent answer will stay together and no context is lost.

[rswamina]

@zklaus - I verified that the climate_statistics preprocessor with period: day calculates daily climatologies. For the recipe in this thread above, I end up with 360 values per grid cell. I think there might be some issues if I added data from other models like you say but am not sure what form that would take (based on different bounds etc.), so will raise that separately if I need to use it. Also, I wa a little puzzled by the climate_statistics preprocessor also having an option for period:daily. Effectively day and daily did the same thing for me. I wondered if day could also be used for sub-daily measures or if anything else was difefrent between the two.

[zklaus]

The periods day and daily, as well as month, monthly, and mon, and also season and seasonal, are equivalent, respectively. As far as I understand, they all can be applied to sub-daily data, or indeed any kind of data of at least the nominal time resolution or any high resolution.

PS: If you feel that the new answer better helps with your original question, feel free to change the "accepted" mark so that in future visitors to this question get the best help they can.