Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-tile-output #289

Closed
wants to merge 116 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
0d8cfbf
test refactor (#54)
rabernat Nov 8, 2017
06405fa
fixes #56 (#57)
rabernat Nov 13, 2017
15da54b
Fix llc chunking (#60)
rabernat Dec 14, 2017
5954626
fix time encoding (#61)
rabernat Dec 14, 2017
45d02b5
Update examples.rst (#65)
ifenty Jan 17, 2018
d81f7dc
Added functionality to read PTRtave files (#63)
Feb 5, 2018
42aa2be
Adding .stickler.yml (#70)
stickler-ci Feb 21, 2018
4a2bc59
Extract version number from git tag (#72)
edoddridge Mar 5, 2018
298adcb
Generalizes _get_all_iternums in order to handle compressed data (#77)
apatlpo Apr 6, 2018
d8127ce
Cast `prefix` to list if it isn't already one (#79)
edoddridge Jun 8, 2018
46f3232
fix the problem with testing type of prefix (#83)
raphaeldussin Jun 22, 2018
bb1e63c
extend capabilities of read_raw_data (#84)
raphaeldussin Jul 16, 2018
8f17b81
update dev history
raphaeldussin Jul 17, 2018
a3edf9e
Bump version: 0.2.1 → 0.2.2
raphaeldussin Jul 17, 2018
727552e
fix for RTD ?
raphaeldussin Jul 30, 2018
21e8a89
Merge pull request #86 from raphaeldussin/tag_v0.2.2
raphaeldussin Jul 30, 2018
72feddc
bug fix get llc shape (#88)
raphaeldussin Aug 7, 2018
8a5488d
try to fix doc build
rabernat Aug 10, 2018
4da34ff
WIP refactoring of low level functions (#87)
raphaeldussin Aug 23, 2018
39af349
add conda instructions
raphaeldussin Sep 4, 2018
64c588a
Merge pull request #94 from raphaeldussin/docedits
raphaeldussin Sep 4, 2018
1cbf6d3
Add seaice and thsice native output (#96)
mjlosch Sep 10, 2018
04223e2
clean refactored read_mds (#93)
raphaeldussin Sep 14, 2018
86e5003
refactoring mds_store to include the option to pass extra metadata (#97)
raphaeldussin Sep 17, 2018
300739e
fix token bug (#101)
rabernat Sep 22, 2018
67bf9bb
added angles to llc grid coords (#92)
rabernat Sep 22, 2018
410af6e
Fix bug llc, fixes #102 (#104)
raphaeldussin Nov 9, 2018
71cb8d5
added extra grid variables to variables.py
timothyas Oct 24, 2018
35073cf
subroutines to read extra grid vars
timothyas Oct 24, 2018
589b347
extra grid vars loaded no matter user prefix
timothyas Oct 25, 2018
a9c402b
Small rearrangement, skipping hybrid sigma coordinate output for now.
timothyas Nov 10, 2018
f32ad02
Fixing style errors.
stickler-ci Nov 10, 2018
d4b3c3b
Fixed lines > 79 char long
timothyas Nov 10, 2018
e5d0ba1
Merge branch 'extra-grid-vars' of github.com:timothyas/xmitgcm into e…
timothyas Nov 10, 2018
0cac299
Remove excess extra_grid_vars functions
timothyas Nov 25, 2018
a2ab37f
Fixing style errors.
stickler-ci Nov 25, 2018
73fae38
Merge pull request #110 from timothyas/extra-grid-vars
raphaeldussin Dec 17, 2018
4bf2a40
transfer test data to figshare (#115)
raphaeldussin Jan 14, 2019
6aa98ca
Read mitgrid files (#114)
raphaeldussin Jan 15, 2019
274f811
silly bug read grid (#117)
raphaeldussin Jan 22, 2019
008cbc3
Fix warnings (#113)
raphaeldussin Jan 22, 2019
ddcb898
Write binary files from xmitgcm (#118)
raphaeldussin Feb 18, 2019
3cac02a
Calculate binary masks from hFac variables (#120)
timothyas Mar 18, 2019
6464531
Add try/except for OrderedDict (#123)
malmans2 Mar 26, 2019
4155cda
add new CI config (#125)
raphaeldussin Mar 28, 2019
8e949f2
add read grid demo (#130)
raphaeldussin Apr 30, 2019
bc48c88
try fix rtd build (#132)
raphaeldussin Apr 30, 2019
e85a2b8
Fix rtd (#133)
raphaeldussin Apr 30, 2019
eb44671
Fix rtd (#134)
raphaeldussin May 1, 2019
170645a
Fix rtd (#135)
raphaeldussin May 1, 2019
d453ce4
New docs to write binary files (#136)
raphaeldussin May 2, 2019
d951bc7
fix docstring (#137)
raphaeldussin May 2, 2019
0e330f2
Fix docstrings (#138)
raphaeldussin May 2, 2019
140888a
missing white line (#139)
raphaeldussin May 2, 2019
ff30a93
update links to https (#129)
lpmi-13 May 18, 2019
fa46d1d
fix for 2d diagnostics llc bug (#146)
rabernat May 20, 2019
2cac4bf
also search in grid_dir for avail_diags.log (#147)
rabernat May 20, 2019
264f9c6
Cleanup for release (#150)
rabernat May 20, 2019
98ea1e7
add llcreader module (#152)
rabernat Jun 19, 2019
5d47c04
lighten build requirements
rabernat Jun 19, 2019
b5934f1
more doc env tweaks
rabernat Jun 19, 2019
c8b2d05
add zarr and fsspec
rabernat Jun 19, 2019
5cc6f69
Release v0.4.0
rabernat Jul 11, 2019
d3ecf98
re-added bugfix from #155 which was accidentally overwritten during t…
rabernat Jul 11, 2019
0dce402
Release v0.4.1
rabernat Jul 11, 2019
4712763
add API doc (#164)
rabernat Aug 6, 2019
e7243d1
dask and mmap mutually exclusive (#165)
malmans2 Sep 5, 2019
20d8d4f
Change deprecated call of py.test.Raises (#167)
aulemahal Sep 5, 2019
31cbe08
Fix fsspec keyword arg bug (#170)
rabernat Sep 11, 2019
c276286
rename repo to MITgcm (#172)
rabernat Sep 27, 2019
f865eab
Use cache for listing directories (#168)
rabernat Oct 25, 2019
95b2cb5
remove old ci stuff (#176)
rabernat Oct 25, 2019
cfe8287
Read data with level subsets, read namelists. (#149)
aulemahal Oct 25, 2019
53686ca
Modify datetime decoding (#177)
rabernat Oct 25, 2019
55555c3
hopefully fix path join bug (#181)
rabernat Nov 13, 2019
923bbf3
remove extra print statement from `open_mdsdataset` (#183)
cspencerjones Nov 22, 2019
cc84ee3
add mask_override option to llcreader (#192)
rabernat Feb 26, 2020
48c72a9
Python 3.8 compatibility (#194)
Mar 6, 2020
172ab1b
new combine_by_coords option combine_attrs (#200)
timothyas Apr 14, 2020
4841b4b
iter_stop matches last model date (#203)
antonimmo Jun 3, 2020
c65bc73
llcreader: get grid info from portal (#197)
timothyas Jun 4, 2020
79baec6
add xg and yg to llcreader known_models (#208)
rabernat Jun 18, 2020
5aba259
Travis to gha (#223)
timothyas Sep 30, 2020
929fb05
python 2.7 env with old zarr (#226)
timothyas Oct 12, 2020
7968355
Skip conflict checks with recent xarray in open_mdsdataset (#230)
aulemahal Nov 3, 2020
1a2990f
add metadata to llcreader dimensions (#239)
rabernat Nov 12, 2020
e68b3e5
Fix llcreader klevels bug (#225)
rabernat Nov 13, 2020
f9e1b57
llcreader + ASTE Release 1 (#231)
timothyas Nov 16, 2020
46546d1
feat: IMPLEMENTS CHECKING FOR VARIABLE MATES (#237)
fraserwg Nov 16, 2020
3c88d45
fixed probable typo for 'coordinate' entry in dimensions dictionary (…
ifenty Nov 16, 2020
fba360b
feat: CHECKS ITERATION PARAMETERS WHEN GETTING LLCMODEL DATASETS (#240)
fraserwg Nov 20, 2020
dfab5be
added publish workflow (#241)
rabernat Nov 20, 2020
6b664ce
Update ci.yaml
rabernat Nov 20, 2020
72f327b
Update development.rst (#242)
cspencerjones Nov 24, 2020
843a3ce
fix tiny typo (#245)
brian-rose Dec 3, 2020
ec8cacf
add a standard set of ADJ variables (#247)
mjlosch Jan 12, 2021
95c708f
fixed mistake with naming of Zl/Zu and k_l/k_u. Now l='lower', u='up…
ifenty Jan 12, 2021
fb87e5c
Update xarray master ci (#248)
timothyas Jan 19, 2021
e4d02ef
fix variable metadata typos (#249)
rabernat Jan 21, 2021
e6ee132
Add pleiades grid path & better grid_path error message (#255)
timothyas Feb 11, 2021
26fa060
Cube Sphere support (#98)
raphaeldussin Mar 2, 2021
1324448
Update usage.rst (#252)
rabernat Mar 2, 2021
2dea547
Remove unnecessary parts (#243)
rabernat Mar 2, 2021
41c10bc
try py 3.7?
timothyas Apr 14, 2021
3673f59
Merge pull request #260 from timothyas/xr_master_ci
timothyas May 5, 2021
e9b2aaa
modified open_mdsdataset to add extra_variables (#205)
cspencerjones May 17, 2021
403039b
fix combine_by_coords (#267)
AaronDavidSchneider May 28, 2021
f5bcb5b
modifying read_mds ready for merging with approach 2 code
fraserwg Aug 5, 2021
331ae94
should now be happily working with tiled 3D data
fraserwg Aug 5, 2021
9ea819f
almost working with tiled data. Missing prefixes when using the _MDSD…
fraserwg Aug 5, 2021
4b5eec3
fixed changes which caused breaking of tests. Can now load a tiled md…
fraserwg Aug 5, 2021
56a4f61
added documentation about tiling to the code
fraserwg Aug 5, 2021
cc2a5af
updated documentation
fraserwg Aug 5, 2021
cb012dd
some bug fixes
fraserwg Nov 23, 2021
57009ef
added a tiled dataset fixture
fraserwg Nov 23, 2021
2a69f97
updated the tests to deal with tiled datasets
fraserwg Nov 23, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.2.1
current_version = 0.2.2
commit = True
tag = True

Expand Down
12 changes: 0 additions & 12 deletions .coveragerc

This file was deleted.

1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
xmitgcm/_version.py export-subst
98 changes: 98 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# This uses actios:
# checkout: https://github.com/actions/checkout
# cache: https://github.com/actions/cache
# codecov-action: https://github.com/codecov/codecov-action

name: CI
on:
push:
branches:
- "*"
pull_request:
branches:
- "*"
# Nothing scheduled, just PRs and pushes
# schedule:
# - cron: "0 0 * * *" # Daily at midnight

jobs:
build:
name: Build (${{ matrix.python-version }} | ${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: ["ubuntu-latest"]
python-version: ["3.7", "3.8", "3.9"]
steps:
- uses: actions/checkout@v2
- name: Cache conda
uses: actions/cache@v2
env:
# Increase this value to reset cache if ci/environment.yml has not changed
CACHE_NUMBER: 0
with:
path: ~/conda_pkgs_dir
key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{ hashFiles('ci/environment.yml') }}
- uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: test_env # Defined in ci/environment*.yml
auto-update-conda: false
python-version: ${{ matrix.python-version }}
environment-file: ci/environment-py${{ matrix.python-version }}.yml
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
- name: Set up conda environment
shell: bash -l {0}
run: |
python -m pip install -e .
conda list
- name: Run Tests
shell: bash -l {0}
run: |
py.test --cov=./ --cov-report=xml
- name: Upload code coverage to Codecov
uses: codecov/codecov-action@v1
with:
file: ./coverage.xml
flags: unittests
env_vars: OS,PYTHON
name: codecov-umbrella
fail_ci_if_error: false

xarray-master:
name: Build xarray-master
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Cache conda
uses: actions/cache@v2
env:
# Increase this value to reset cache if ci/environment-upstream-dev.yml has not changed
CACHE_NUMBER: 0
with:
path: ~/conda_pkgs_dir
key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{ hashFiles('ci/environment-upstream-dev.yml') }}
- uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: test_env # Defined in ci/environment-*.yml
auto-update-conda: false
python-version: 3.7
environment-file: ci/environment-py37-xarraymaster.yml
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
- name: Set up conda environment
shell: bash -l {0}
run: |
python -m pip install -e .
conda list
- name: Run Tests
shell: bash -l {0}
run: |
py.test --cov=./ --cov-report=xml
- name: Upload code coverage to Codecov
uses: codecov/codecov-action@v1
with:
file: ./coverage.xml
flags: unittests
env_vars: OS,PYTHON
name: codecov-umbrella
fail_ci_if_error: false
26 changes: 26 additions & 0 deletions .github/workflows/pythonpublish.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: Upload Python Package

on:
release:
types: [created]

jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: '3.x'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools setuptools-scm wheel twine
- name: Build and publish
env:
TWINE_USERNAME: "__token__"
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
run: |
python setup.py sdist bdist_wheel
twine upload dist/*
5 changes: 5 additions & 0 deletions .stickler.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
linters:
flake8:
fixer: true
fixers:
enable: true
41 changes: 0 additions & 41 deletions .travis.yml

This file was deleted.

3 changes: 3 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
include LICENSE
include versioneer.py
include xmitgcm/_version.py
42 changes: 21 additions & 21 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,20 @@ parallel, out-of-core_ analysis of MITgcm output data.
Links
-----

- HTML documentation: http://xmitgcm.readthedocs.org
- Issue tracker: http://github.com/xgcm/xmitgcm/issues
- Source code: http://github.com/xgcm/xmitgcm
- HTML documentation: https://xmitgcm.readthedocs.org
- Issue tracker: https://github.com/MITgcm/xmitgcm/issues
- Source code: https://github.com/MITgcm/xmitgcm

Installation
------------

Requirements
^^^^^^^^^^^^

xmitgcm is compatible with python 3 and python 2.7. It requires xarray_
(>= version 0.8.2) and dask_ (>= version 0.11.2).
xmitgcm is compatible with python >=3.7. It requires xarray_
(>= version 0.11.0) and dask_ (>= version 1.0).
These packages are most reliably installed via the
`conda <http://conda.pydata.org/docs/>`_ environment management
`conda <https://conda.pydata.org/docs/>`_ environment management
system, which is part of the Anaconda_ python distribution. Assuming you have
conda available on your system, the dependencies can be installed with the
command::
Expand All @@ -47,10 +47,10 @@ Installation from github
^^^^^^^^^^^^^^^^^^^^^^^^

xmitgcm is under active development. To obtain the latest development version,
you may clone the `source repository <https://github.com/xgcm/xmitgcm>`_
you may clone the `source repository <https://github.com/MITgcm/xmitgcm>`_
and install it::

git clone https://github.com/xgcm/xmitgcm.git
git clone https://github.com/MITgcm/xmitgcm.git
cd xmitgcm
python setup.py install

Expand All @@ -73,7 +73,7 @@ This will create a directory called ``global_oce_latlon`` which we will use
for the rest of these examples. If you have your own data, replace this with
the path to your mitgcm files.

To opean MITgcm MDS data as an xarray.Dataset, do the following in python::
To open MITgcm MDS data as an xarray.Dataset, do the following in python::

from xmitgcm import open_mdsdataset
data_dir = './global_oce_latlon'
Expand All @@ -85,29 +85,29 @@ try to determine the file prefixes and iteration numbers to read. In some
configurations, the ``open_mdsdataset`` function may work without further
keyword arguments. In most cases, you will have to specify further details.

Consult the `online documentation <http://xmitgcm.readthedocs.org>`_ for
Consult the `online documentation <https://xmitgcm.readthedocs.org>`_ for
more details.

.. |DOI| image:: https://zenodo.org/badge/70649781.svg
:target: https://zenodo.org/badge/latestdoi/70649781
.. |Build Status| image:: https://travis-ci.org/xgcm/xmitgcm.svg?branch=master
:target: https://travis-ci.org/xgcm/xmitgcm
.. |Build Status| image:: https://travis-ci.org/MITgcm/xmitgcm.svg?branch=master
:target: https://travis-ci.org/MITgcm/xmitgcm
:alt: travis-ci build status
.. |codecov| image:: https://codecov.io/github/xgcm/xmitgcm/coverage.svg?branch=master
:target: https://codecov.io/github/xgcm/xmitgcm?branch=master
.. |codecov| image:: https://codecov.io/github/MITgcm/xmitgcm/coverage.svg?branch=master
:target: https://codecov.io/github/MITgcm/xmitgcm?branch=master
:alt: code coverage
.. |pypi| image:: https://badge.fury.io/py/xmitgcm.svg
:target: https://badge.fury.io/py/xmitgcm
:alt: pypi package
.. |docs| image:: http://readthedocs.org/projects/xmitgcm/badge/?version=stable
:target: http://xmitgcm.readthedocs.org/en/stable/?badge=stable
.. |docs| image:: https://readthedocs.org/projects/xmitgcm/badge/?version=stable
:target: https://xmitgcm.readthedocs.org/en/stable/?badge=stable
:alt: documentation status

.. _dask: http://dask.pydata.org
.. _xarray: http://xarray.pydata.org
.. _Comodo: http://pycomodo.forge.imag.fr/norm.html
.. _issues: https://github.com/xgcm/xmitgcm/issues
.. _`pull requests`: https://github.com/xgcm/xmitgcm/pulls
.. _dask: https://dask.pydata.org
.. _xarray: https://xarray.pydata.org
.. _Comodo: https://pycomodo.forge.imag.fr/norm.html
.. _issues: https://github.com/MITgcm/xmitgcm/issues
.. _`pull requests`: https://github.com/MITgcm/xmitgcm/pulls
.. _MITgcm: http://mitgcm.org/public/r2_manual/latest/online_documents/node277.html
.. _out-of-core: https://en.wikipedia.org/wiki/Out-of-core_algorithm
.. _Anaconda: https://www.continuum.io/downloads
11 changes: 0 additions & 11 deletions ci/environment-py27-numpy1.11.1.yml

This file was deleted.

11 changes: 0 additions & 11 deletions ci/environment-py27.yml

This file was deleted.

9 changes: 8 additions & 1 deletion ci/environment-py36.yml → ci/environment-py3.7.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,18 @@
name: test_env
channels:
- conda-forge
dependencies:
- python=3.6
- python=3.7
- xarray
- dask
- numpy
- pytest
- future
- zarr
- cftime
- aiohttp
- pip:
- cachetools
- fsspec
- codecov
- pytest-cov
17 changes: 17 additions & 0 deletions ci/environment-py3.8.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: test_env
channels:
- conda-forge
dependencies:
- python=3.8
- xarray
- dask
- numpy
- pytest
- future
- zarr
- cftime
- aiohttp
- pytest-cov
- codecov
- fsspec
- cachetools
17 changes: 17 additions & 0 deletions ci/environment-py3.9.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: test_env
channels:
- conda-forge
dependencies:
- python=3.9
- xarray
- dask
- numpy
- pytest
- future
- zarr
- cftime
- aiohttp
- pytest-cov
- codecov
- fsspec
- cachetools
11 changes: 0 additions & 11 deletions ci/environment-py35.yml

This file was deleted.

Loading