-
Notifications
You must be signed in to change notification settings - Fork 10
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Version 2.0. Rewrite the structure of how requests are made. Closes 1…
…. Closes 2.
- Loading branch information
Showing
40 changed files
with
804 additions
and
123 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,6 @@ | |
| *.mltbx | ||
| doc/html/*.html | ||
| **/*.nc | ||
| **/*.csv | ||
| **/*.zip | ||
| **/*.grib | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,82 +1,59 @@ | ||
| function [filePaths, citation] = climateDataStoreDownload(name,options) | ||
| function [filePaths, citation] = climateDataStoreDownload(datasetName,datasetOptions,options) | ||
| %climateDataStoreDownload Get data from Copernicus Climate Data Store | ||
| % Download a data set from the Copernicus Climate Data Store. | ||
| % (https://cds.climate.copernicus.eu/cdsapp) | ||
| % Download a data set from the Copernicus Climate Data Store. (https://cds.climate.copernicus.eu/cdsapp) | ||
| % | ||
| % Find your dataset at Climate Data Store and click on the "download data" | ||
| % tab. Make your selections for the subset of data you want. Click "show | ||
| % API request" at the bottom. | ||
| % Find your dataset at Climate Data Store and click on the "download data" tab. Make your selections for the subset of data you want. Click "show | ||
| % API request" at the bottom. | ||
| % | ||
| % NAME is the name of the data set to retrieve, and will also be used as | ||
| % the name of the directory to put the downloaded files in. OPTIONS is a | ||
| % MATLAB structure matching the python structure shown when you choose | ||
| % "Show API request". The data files are downloaded, and the function | ||
| % returns FILEPATHS, a list of files downloaded. In addition, the function | ||
| % provides CITATION, which is the correct citiation to use with the data | ||
| % retrieved. | ||
| % [filePaths, citation] = climateDataStoreDownload(datasetName,datasetOptions) retrieves the data set with the name datasetName, and will also be used | ||
| % as the name of the directory or file downloaded files, with a date/time stamp added. datasetOptions is a MATLAB structure matching the python | ||
| % structure shown when you choose "Show API request". filePaths is string array of the files downloaded. The function returns FILEPATHS, a list of | ||
| % files downloaded. The function also returns citation, which is the correct citiation to use with the data retrieved. | ||
| % | ||
| % climateDataStoreDownload(...,Timeout Sets the maximum time in seconds to wait for a response. | ||
| % climateDataStoreDownload(...,DontExpandZIP=true Results that are ZIP files are not automatically expanded. | ||
| % climateDataStoreDownload(...,DontPromptForCredentials=true If no credentials are present, don't request them (intended for tests) | ||
| % | ||
| % Downloading the files can take some time, depending on how large they are. | ||
| % Downloading the files can take some time, depending on how large they are (I've had it take 30 minutes!). You can check on the status of your | ||
| % request by visiting https://cds.climate.copernicus.eu/cdsapp#!/yourrequests. For large requests, you may want to consider using | ||
| % climateDataStoreDownloadAsync, which queues a request and returns, allowing you to keep working. | ||
| % | ||
| % Notes: | ||
| % * You must have: | ||
| % * python 3.8 installed | ||
| % (https://www.python.org/ftp/python/3.8.10/python-3.8.10-amd64.exe) | ||
| % You must have: | ||
| % * python 3.8 installed (https://www.python.org/ftp/python/3.8.10/python-3.8.10-amd64.exe) | ||
| % * the cdsapi python package (pip3 install cdsapi) | ||
| % * Your credentials need to be in a .cdsapirc file in your user | ||
| % directory. See https://cds.climate.copernicus.eu/api-how-to for more info | ||
| % * This function relies on the Python API to access the Copernicus Climate | ||
| % Data Store (CDS) (https://github.com/ecmwf/cdsapi) by the European Centre | ||
| % for Medium-Range Weather Forecasts (ECMWF) | ||
| % * Your credentials need to be in a .cdsapirc file in your user directory. See https://cds.climate.copernicus.eu/api-how-to for more info | ||
| % * This function relies on the Python API to access the Copernicus Climate Data Store (CDS) (https://github.com/ecmwf/cdsapi) by the European | ||
| % Centre for Medium-Range Weather Forecasts (ECMWF) | ||
| % | ||
| % Example: Sea Ice thickness | ||
| % (https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-sea-ice-thickness) | ||
| % options.version = "1_0"; | ||
| % options.variable = "all"; | ||
| % options.satellite = "cryosat_2"; | ||
| % options.cdr_type = ["cdr","icdr"]; | ||
| % options.year = ["2011","2021"]; | ||
| % options.month = "03"; | ||
| % downloadedFilePaths = climateDataStoreDownload('satellite-sea-ice-thickness',options); | ||
|
|
||
| % Copyright 2021 The MathWorks, Inc. | ||
| % Example: Sea Ice thickness (https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-sea-ice-thickness) | ||
| % datasetOptions.version = "1_0"; | ||
| % datasetOptions.variable = "all"; | ||
| % datasetOptions.satellite = "cryosat_2"; | ||
| % datasetOptions.cdr_type = ["cdr","icdr"]; | ||
| % datasetOptions.year = ["2011","2021"]; | ||
| % datasetOptions.month = "03"; | ||
| % downloadedFilePaths = climateDataStoreDownload('satellite-sea-ice-thickness',datasetOptions); | ||
| % | ||
| % See Also: climateDataStoreDownloadAsync | ||
|
|
||
| validateattributes(name,{'string','char'},{'scalartext'}); | ||
| validateattributes(options,'struct',{'scalar'}); | ||
| % Copyright 2022 The MathWorks, Inc. | ||
|
|
||
| % Diagnostics | ||
| setupPythonIfNeeded(); | ||
| setupCDSAPIIfNeeded(); | ||
| arguments | ||
| datasetName (1,1) string {mustBeNonzeroLengthText} | ||
| datasetOptions (1,1) struct | ||
| options.DontExpandZIP (1,1) logical = false; | ||
| options.DontPromptForCredentials (1,1) logical = false; | ||
| options.Timeout (1,1) double {mustBePositive} = Inf; | ||
| end | ||
|
|
||
|
|
||
|
|
||
| % Convert the structure from strings to char and cellstr. | ||
| options = makeStringsChars(options); | ||
|
|
||
| % Force download option to ZIP | ||
| options.format = 'zip'; | ||
|
|
||
| zipfilePath = string(tempname) + ".zip"; | ||
|
|
||
| retrieveFromCDS(name,options,zipfilePath); | ||
|
|
||
| filePaths = string(unzip(zipfilePath,name)'); | ||
| citation = "Generated using Copernicus Climate Change Service information " + string(datetime('now','Format','y')); | ||
|
|
||
| % Delete the temporary ZIP file | ||
| delete(zipfilePath); | ||
| end | ||
|
|
||
| function theStruct = makeStringsChars(theStruct) | ||
| fields = string(fieldnames(theStruct)); | ||
| for iField = 1:numel(fields) | ||
| if isstring(theStruct.(fields(iField))) | ||
| if isscalar(theStruct.(fields(iField))) | ||
| % Convert scalar strings to char | ||
| theStruct.(fields(iField)) = char(theStruct.(fields(iField))); | ||
| else | ||
| % Convert string arrays to cell array of chars | ||
| theStruct.(fields(iField)) = cellstr(theStruct.(fields(iField))); | ||
| end | ||
| end | ||
| f = climateDataStoreDownloadFuture(datasetName, datasetOptions,options); | ||
| f.wait(options.Timeout); | ||
| if f.State == "failed" | ||
| throwAsCaller(f.Error) | ||
| elseif f.State ~= "completed" | ||
| throwAsCaller(MException("climateDataStore:UnexpectedState","cdsRequestState should be complete, it's %s",f.State)) | ||
| end | ||
|
|
||
| filePaths = f.OutputArguments{1}; | ||
| citation = f.OutputArguments{2}; | ||
| end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| function F = climateDataStoreDownloadAsync(datasetName,datasetOptions,options) | ||
| %climateDataStoreDownload Queue a data request from Copernicus Climate Data Store | ||
| % Queue a data set request with the Copernicus Climate Data Store. (https://cds.climate.copernicus.eu/cdsapp) | ||
| % | ||
| % Find your dataset at Climate Data Store and click on the "download data" tab. Make your selections for the subset of data you want. Click "show | ||
| % API request" at the bottom. | ||
| % | ||
| % F = climateDataStoreDownloadAsync(datasetName,datasetOptions) queues a request for the data set with the name datasetName, and will also be used | ||
| % as the name of the directory or file downloaded files, with a date/time stamp added. datasetOptions is a MATLAB structure matching the python | ||
| % structure shown when you choose "Show API request". F is an object you can use to query the state of your request, and cancel the request. Note | ||
| % that if you delete F, or allow it to be cleared, your request with CDS will be cancelled. Note that the operation does not take place on a | ||
| % seperate thread -- download of files will not take place until you interact with F, either through checking Status or using wait(). | ||
| % | ||
| % climateDataStoreDownloadAsync(...,DontExpandZIP=true Results that are ZIP files are not automatically expanded. | ||
| % climateDataStoreDownloadAsync(...,DontPromptForCredentials=true If no credentials are present, don't request them (intended for tests) | ||
| % | ||
| % Downloading the files can take some time, depending on how large they are (I've had it take 30 minutes!). You can check on the status of your | ||
| % request by visiting https://cds.climate.copernicus.eu/cdsapp#!/yourrequests. For simple/quick requests, you may want to consider using | ||
| % climateDataStoreDownload, which is easier to use, but blocks MATLAB. | ||
| % | ||
| % Note that many requests return immediately. You should check Status property right away. Results may be available. | ||
| % | ||
| % Notes: | ||
| % You must have: | ||
| % * python 3.8 installed (https://www.python.org/ftp/python/3.8.10/python-3.8.10-amd64.exe) | ||
| % * the cdsapi python package (pip3 install cdsapi) | ||
| % * Your credentials need to be in a .cdsapirc file in your user directory. See https://cds.climate.copernicus.eu/api-how-to for more info | ||
| % * This function relies on the Python API to access the Copernicus Climate Data Store (CDS) (https://github.com/ecmwf/cdsapi) by the European | ||
| % Centre for Medium-Range Weather Forecasts (ECMWF) | ||
| % | ||
| % Example: ERA5 hourly data on pressure levels (https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels) | ||
| % datasetName ="reanalysis-era5-pressure-levels"; | ||
| % datasetOptions.product_type = "reanalysis"; | ||
| % datasetOptions.format = "grib"; | ||
| % datasetOptions.year = "2020"; | ||
| % datasetOptions.month = "01"; | ||
| % datasetOptions.day = "01"; | ||
| % datasetOptions.pressure_level = "1"; | ||
| % datasetOptions.variable = "divergence"; | ||
| % datasetOptions.time = "06:00"; | ||
| % F = climateDataStoreDownloadAsync(datasetName, datasetOptions); | ||
| % % Run whatever MATLAB code you want in here. | ||
| % F.wait(); | ||
| % if F.state == "completed" | ||
| % downloadedFilePaths = OutputArguments{1}; | ||
| % citation = OutputArguments{2}; | ||
| % end | ||
| % | ||
| % See Also: climateDataStoreDownload, climateDataStoreDownloadFuture | ||
|
|
||
| % Copyright 2022 The MathWorks, Inc. | ||
|
|
||
| arguments | ||
| datasetName (1,1) string {mustBeNonzeroLengthText} | ||
| datasetOptions (1,1) struct | ||
| options.DontExpandZIP (1,1) logical = false; | ||
| options.DontPromptForCredentials (1,1) logical = false; | ||
| end | ||
|
|
||
| F = climateDataStoreDownloadFuture(datasetName,datasetOptions, options); | ||
| end |
Oops, something went wrong.