* test refactor

* added tests for MITgcm#59

* fixes MITgcm#59

* added calendar

* caught a little bug

* added reading functionality of PTRtave files

* changed trname to travename to avoid duplication

* Added the units back as [kg m-3]

* Extract version number from git tag

This keeps the version number in the documentation consistent with the code release. Code taken from MITgcm/MITgcm@96334be Closes MITgcm#71

* Fixing style errors.

* Move import statement to conform with code linter

* Make mds_store test work with xarray >=0.10.1

As noted in MITgcm#72 an upstream change in xarray broke one of the tests. This commit also modifies dependencies in setup.py to xarray >= 0.10.1

…gcm#77)

* Generalizes _get_all_iternums in order to handle compressed data

* fix excessive line length

* Fixing style errors.

* cast `prefix` to list if it isn't already one

* Check for unicode strings too

* fix python2/3 compatibilty issue for string test

* no need for extra line

* Fixing style errors.

* extend capabilities of read_raw_data

* possibility to read part of the file, with offset and partial_read
* choice of row/column major order

* testing + better error handling

* function will warn user if trying to pass inconsistent args
* function checks byte offset < file size

* Fixing style errors.

* get it shorter

* completing tests for codecov

* Fixing style errors.

* fix line length

* fix line length

* try to improve coverage

* finalize request

* remove blank lines
* iteration on dtype cleaner

* replace assert with raise error

* try to fool code cov

* fix indentation

* codecov didn't bite the bait

* fix error in testing

fix for RTD ? fix MITgcm#75

* refactoring of low level functions

* Fixing style errors.

* run flake8

* Fixing style errors.

* unit test

* fix style

* changes requested pt1

* improvements on read_xy_chunk

* padding in separate function
* return numpy array or memmap

* Fixing style errors.

* fix line length

* add more testing

* stupid bug

* more testing

* Fixing style errors.

* test for read_generic_data

* Fixing style errors.

* improve coverage

* Fixing style errors.

* unit test for read_all_variables

* Fixing style errors.

* corrections requested

* eager mode for read_generic_data

* read_3d_chunk fct

* testing for read_3d_chunk

* Fixing style errors.

* read big chunks + testing

* missing conflict

* Fixing style errors.

* Refactor read_all + test

* Fixing style errors.

* better docstrings

* working on tests, pb memmap

* Fixing style errors.

* revisions

* read_3d_chunk -> read_xyz_chunk
* correction docstring

* Fixing style errors.

* correct name function in test

* more docstrings

* one case of memmap evaluation figured out

+ wrong test in read_xyz_chunk

* Fixing style errors.

* not strickly enforce memmap test

* Fixing style errors.

Update documentation with conda instructions

* add native output description for thsice and seaice

* maintaining the tradition of not getting it right the first time

* try to make standard_names more or less CF compliant

* Fixing style errors.

* Fixing style errors.

* Fixing style errors.

* Fixing style errors.

* Fixing style errors.

* Fixing style errors.

* Fixing style errors.

* Fixing style errors.

* insert line breaks in four long lines

* clean refactored read_mds

* test extra_metadata

* Fixing style errors.

* corrections requested

* Fixing style errors.

* fix strickler

* replaced dask_delayed by use_dask

* fix cosmetic errors

* Fixing style errors.

* fix cosmetic errors

* get extra metadata

* Fixing style errors.

* fix assert

* use get_extra_metadata in read_mds

* refactor testing, small/big to 2D/3D

* Fixing style errors.

…ITgcm#97)

* clean PR

* clean up

* Fixing style errors.

* add test for extra_metadata

* Fixing style errors.

* fix line length

* fix extra_metadata variables

* Fixing style errors.

* fix line length

* wip: almost done

* Fixing style errors.

* fix bug

* Fixing style errors.

* fix test

* added angles to llc grid coords

* Fixing style errors.

* correcting a bug, fixes MITgcm#102

* more comments

* testing for bad nx,ny combination

* Fixing style errors.

* cosmetics

* Fixing style errors.

* extra variables from MITgcm/model/src/write_grid.F

* to do: add sigma coordinate variables

* only grabs variables when they exist

* look for variables in data_dir not grid_dir

…xtra-grid-vars

* look for extra grid prefixes in dir, only add these to all_grid_vars

* look for extra grid variables in grid_dir, not data_dir

Extra grid vars

* transfer test data to figshare

* python2 compat

* Fixing style errors.

* make strickler happier

* Fixing style errors.

* md5+cache+doc

* Fixing style errors.

* fix

* make it py2 compat

* start working on reading grid

* read ext grid

* works for llc

* works with aste+docstring

* Fixing style errors.

* fix cosmetics

* fix cosmetics

* fixes

* Fixing style errors.

* testing

* Fixing style errors.

* fix strickler

* add link/md5

* fix strickler

* more testing

* Fixing style errors.

* fix strickler

* fix strickler

* make dask lazy

* Fixing style errors.

* fix from strickler

* silly bug

* more testing

* Fixing style errors.

* fix line length

* fix striclker

* remove debug print

* fix numerous warnings

* fix numerous warnings

* fix line length

* fix line length

* Fixing style errors.

* Fixing style errors.

* fix

* Fixing style errors.

* more testing

* Fixing style errors.

* remove duplicate

* needs unbroken line

* coverage

* work on write function

* split in various functions

* able to rebuild facets

* needs new func

* bugfix + add testing

* funcs to compact

* Fixing style errors.

* tests in

* Fixing style errors.

* coverage

* fix test

* cleaner solution

* precision->dtype

* missing items

* Meta data for mask(C/W/S) variables

* Add function to compute mask variables from hFac

* Compute masks via binary op to save memory

* Test mask values by ensuring mask*hFac == hFac

* Fixing style errors.

* test mask values with xr.testing.assert_equal rather than .all()

* test new xarray

* add environment

* fix typo

* add conda forge channel

* test against xarray master

* more testing env

* add new testing

* dependency not supported

* update travis

* try fix rtd build

* fix conda?

* try fix rtd build

* fix conda?

* try fix rtd build

* try fix rtd build

* fix conda?

* try fix rtd build

* install xmitgcm

* install xmitgcm

* doc writing binary file

* update index

* fix format

* fix typo

* fix for 2d diags issue

* remove comment

* remove old numpy env from ci

* also search in grid_dir for avail_diags.log

* added test

* updated whats new

* versioneer stuff

* remove old version string

* exclude versioneer from coverage

* exclude more versioneer stuff from coverage

* added llcreader module

* before refactor

* wip: optimizing

* wip: more performance improvements

* wrapping up

* major refactor; 1000x faster :)

* working on docs

* new models and tests

* update ci stuff

* verbose travis

* debugging tests

* modify imports

* set conda-forge channel

* back to importorskip

* refactor of llcmodel internals

* remove unnecessary class

* docstring update

* trying to fix tests again

* add missing module

…ring the release of 0.4.0

* wip

* increased fsspec dependency to 0.4.4 and simplified code

* remove looseversion

* wrote file_utils

* wrote file_utils

* almost passing tests

* add cachetools to requirements

* move cachetools install to pip

* changed cache mechanism

* fix messed up tar files

* make tests work

* Added parse_namelist

* Add some options for levels

* Working implementation of levels subsetting

* Add warning and options to namelist parser

* Merged upstream/master changes

* Small fix | Added in docstring | Added 2 tests

* add back fixture

* wrap tmpdir in str

https://stackoverflow.com/questions/27034001/os-path-join-fails-with-typeerror-object-of-type-localpath-has-no-len

* fix MITgcm#174

* add cftime to ci env

* serialize to zarr not netcdf

* add zarr to py2.7 env

* Python 3.8 compatibility

* files for Travis CI

* Added py27 compatibility

Co-authored-by: Uchida Takaya <[email protected]>

Solves MITgcm#193

Co-authored-by: Antonio Quintana <[email protected]>

* access to llc2160 grid is a go, with a couple caveats for now

* use grid var names not filenames, get vertical grid vars working

* lil typo

* fixes loading grid dir issues

* it works for llc2160

* add directory to some LLC4320 grid vars

* separate grid metadata function, add shrunk_grid flag

* add grid_varnames to paramter list

* test that grid variables show up in portal datasets

* prefixes->variables

* try Zl...

* extra logic to create Zl, Zu, and Zp1

* make reading grid optional

* wow, not copying dicts really messes things up

* only build Zu, Zl if RF is there

* shouldn't kp1 have nz+1 levels

* only read grid if grid_varnames has entries

* also test selected k_p1 levels

* typo...

* ... more typos...

* rhoref, extra dx/dy metrics

* add new metrics to test

* typos, and add rhoRef to vgrid prefixes

* load vgrid as one big chunk, all at once. test it too!

* oops, fixed test

* fix for global vgrid vars... what about layers.. tbd

* cleanup vgrid check

* cleanup grid var dict situation

* also test latlon vgrid coord chunking

* faces->ll

* update docs

* tiny typo

* added more 4320 grid files

* add xg and yg to llcreader known_models

* add other grid variables

* init gha files

* add aiohttp

* use cache v2 and goanpeca->conda-incubator

* trying to recreate GHA error...

* try sorted lists?

* try without aiohttp on py27

* no need for travis yml

* no need for coveragerc

* attempt to fix py27 with old zarr

* alternate soln with setup

* didn't work, back to first soln

* empty commit to trigger CI

* add test for k_levels=[1]

* add explicit test for MITgcm#233

* test for bug in llc90

* fixes MITgcm#224

* resolve final bugs

* very rough draft .... almost have all tiles correct...

* bering strait working

* all faces accounted for ... now to code it better...

* quick prefix fix

* look for meta file for data prec, otherwise use class default

* debugged ... was looking in grid dir

* to aws

* prefix bug

* give standard dimensions some attrs for xgcm

* temporary aws changes

* look for extra grid var filename prefixes

* semi general padding for non shrunk fields

* slightly better

* lots of variables

* added KPP nonlocal salt/temp flux

* dtype from meta prop to dask array creation

* some small bugs...

* provide dtype apriori, reading on the fly too slow. revert utils

* semi better handling of index key

* no latlon for aste

* specify iters for nonuniform step

* aws nested store, specify iters, add tests

* tests actually pass locally...

* python 3 prints

* ASTE documentation

* pretty it up

* hopefully clearer about iters vs iter params

* change some wording

* update ref names and order

* fixed up padding logic

* update that aste ref ... again

* get my face back!

* _pad_facet sufficiently documented...with pics

* feat: IMPLEMENTS CHAECKING FOR VARIABLE MATES

On a latlon grid some variables have mates. This commit will produce
a meaningful error if attempting to use `get_dataset` or
`faces_dataset_to_latlon` to load a variable without its mate.

See Github issue MITgcm#234

* test: ADDED TEST FOR THE MATE ERROR

* test: REMOVED TEST FOR VECTOR MATE ERROR

removed test for vector mate error to determine whether CI problems are
coming from the test or an xarray-master commit

* fix: ADDED TEST BACK AND CHANGED ARGUMENTS

…ITgcm#236)

convention here seems to be

    XC=dict(dims=["j", "i"], attrs=dict(
        standard_name="longitude", long_name="longitude",
        units="degrees_east", coordinate="YC XC")),

some instances fixed where "coordinate" was spelled "coordinates".  May have no effect.

…gcm#240)

* feat: CHECKS ITERATION PARAMETERS WHEN GETTING LLCMODEL DATASETS

When loading datasets using the llcreader the user is able to specify
the iteration start, stop and step. These can also be specified as
attributes in the `BaseLLCModel`. This commit introduces a check that
compares the `iter_start` and `iter_step` fed into `get_dataset` with
the `BaseLLCModel` attributes. If they are found to be incompatible
a warning is triggered.

A warning over an Error was chosen as subsuquent code may still execute
successfully even if there is a mismatch between the user specified and
`BaseLLCModel` derived values for `iter_start` and `iter_stop`.

* refactor: CHANGED WARNING TO RUNTIMEWARNING

* test: ADDED TESTS FOR ITERATION CHECKS

* feat: ADDED CHECK ITERS FUNCTION

* feat/refac: REORDERS LOADING AND CHECKING OF ITERATION PARAMS

Currently if one tries to load an ECCO portal model by specifying iterations, an error is raised. This is due to the model having attributes `iter_start`, `iter_step` and `iter_stop`. If the user tries to call `get_dataset` with a list `iters` the function
first reads in the `iter_params` from the model's attributes. The function then goes on to check that none of the `iter_params` are defined. Upon finding they have been defined the function raises an error.

The refactor does the following:
(1) Check to see if the user has set any `iter_params` in the function call.
    (i) If they have, check the user hasn't set `iters`. If the user has set iters there's a problem, so raise an error.
    (ii) If no `iters` are set then use the user set `iter_params`

(2) If no user set `iter_params` try reading in a user set `iters` and use this

(3) If still no luck, try reading the model `iter_params`. If any of them are `None` keep going...

(4) Finally try reading the model `iters`.

(5) If nothing has been found then raise an error about the lack of iteration instructions.

(6) Check the `iter_params` or `iters` for consistency with the models attributes.

* test: UPDATED TESTS

- Refactored the ecco portal tests
- Added an aste portal test

* fix miniconda action

Co-authored-by: Ryan Abernathey <[email protected]>