Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bnb/dh refactor #220

Open
wants to merge 383 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
383 commits
Select commit Hold shift + click to select a range
303da0e
qdm bc tests updated and passing
bnb32 Jun 17, 2024
b3651ed
era dl tests added.
bnb32 Jun 18, 2024
2d84896
stats calculation updates to enable dim kwargs. lon range test added
bnb32 Jun 18, 2024
4c39c68
Sup3rQa refactor with deriver acting on bbias corrected data.
bnb32 Jun 19, 2024
8a4d730
added mask for regridder to use single site if within min distance. u…
bnb32 Jun 20, 2024
0db46cd
remaining tests updated. all passing
bnb32 Jun 21, 2024
3ccb237
moved DataRetrievalBase to bias.base. Added new class __repr__ in fac…
bnb32 Jun 21, 2024
30c763f
SamplerCollection removed and method moved to AbstractBatchQueue
bnb32 Jun 21, 2024
7fd0607
test added for solar multi step gan with exo data. this covers the `E…
bnb32 Jun 22, 2024
6a51cbb
interpolate_na added to accessor as a dask compatible nn_fill_array a…
bnb32 Jun 22, 2024
7289b40
add log interp to `Interpolator` so era data can be loaded with a han…
bnb32 Jun 22, 2024
b2cdfa9
updated log interp test in utilities
bnb32 Jun 22, 2024
c96dc79
_compute_if_dask and _compute_chunks_if_dask wrapper methods to handl…
bnb32 Jun 22, 2024
e5265a2
linting
bnb32 Jun 22, 2024
4fd32de
added time step consistency check to dual extracter
bnb32 Jun 22, 2024
c38c3e3
test train dual test updates with separate validation queue. removed …
bnb32 Jun 23, 2024
abc6614
linting
bnb32 Jun 23, 2024
babe2dd
moved cdsapi module check to `get_cds_client` method.
bnb32 Jun 23, 2024
fd86c9b
`typing.Self` not available in < 3.11
bnb32 Jun 23, 2024
e732105
< 3.11 doesnt like my fancy map calls
bnb32 Jun 23, 2024
ca15e3f
appeasing < 3.11
bnb32 Jun 23, 2024
4757e1e
< 3.11 not a fan of * unpacking, it seems
bnb32 Jun 23, 2024
48363d8
Union instead of | for < 3.10
bnb32 Jun 23, 2024
b039236
`.time_step` added to `Sup3rX`. other mods for backwards compat.
bnb32 Jun 23, 2024
1b76299
some adjustments for loader dimension naming
bnb32 Jun 23, 2024
02e76c4
sample_counter test fix
bnb32 Jun 24, 2024
e98d10a
moved some kwargs to input_handler_kwargs for ExoExtracter and ExoDat…
bnb32 Jun 24, 2024
a21a42b
rename exo_handler to exo_handler_name
bnb32 Jun 24, 2024
35a065f
additional stats test
bnb32 Jun 24, 2024
4355475
dc training test updates
bnb32 Jun 24, 2024
a417a06
linting
bnb32 Jun 24, 2024
f65c3e9
broke up file_handling in writer directory. direct data handler which…
bnb32 Jun 24, 2024
66ca67a
type general base class for type general data handler and extracter
bnb32 Jun 24, 2024
a1a0034
custom signatures added to factory classes so that kwargs can be parsed.
bnb32 Jun 24, 2024
65619e1
hadnt added `batch_handler.stop()` in `Sup3rCondMom.train().` Could b…
bnb32 Jun 25, 2024
0111f79
linting: implicit string concat
bnb32 Jun 25, 2024
368d2bd
no .compute() call for eager batcher
bnb32 Jun 25, 2024
01f46b2
duplicate logger get removed. moved `get_exo_chunk` to `ExoData.get_c…
bnb32 Jun 25, 2024
d7309c9
type agnostic data handlers, loaders, and topo extracters. can remove…
bnb32 Jun 25, 2024
662b742
dc training tests updated. bin count record fixed.
bnb32 Jun 25, 2024
1746967
additional checks on SolarCC model loss.
bnb32 Jun 26, 2024
f37d992
pylint not finding very clear class members
bnb32 Jun 26, 2024
69ec57e
spellchecks
bnb32 Jun 26, 2024
021b8f4
tf.config run functions eaglerly for accurate pytest-cov result.
bnb32 Jun 26, 2024
73dbbc3
pytest-env test dep for global env vars during testing. global pytest…
bnb32 Jun 26, 2024
b2f8cc3
fix: wrong error catch for `test_feature_errors`
bnb32 Jun 26, 2024
a68193d
batch handler tester factory in helpers
bnb32 Jun 26, 2024
dd09592
deterministic sampling enforced by eager vs lazy testing
bnb32 Jun 26, 2024
0178d32
github tests hanging bc of cov env?
bnb32 Jun 27, 2024
431a93f
removed tf debug options. cause for hanging?
bnb32 Jun 27, 2024
bc6dc6b
some upper -> lower case changes to just reduce some local warnings. …
bnb32 Jun 27, 2024
d8b8f9f
fix: deriver caching test
bnb32 Jun 27, 2024
32ef4de
removing some np.array_equal vs dask warnings
bnb32 Jun 27, 2024
7e53679
tf run eagerly to get accurate pytest-cov report on solar model
bnb32 Jun 27, 2024
6532057
removed duplicate arg
bnb32 Jul 1, 2024
605b6d6
fix: missing doc string
bnb32 Jul 1, 2024
9657848
\b escape for nicer click help message formatting
bnb32 Jul 1, 2024
37d98e9
linting
bnb32 Jul 1, 2024
af3f6f3
removed funky sample counter test
bnb32 Jul 1, 2024
9033b47
safe_serialize method for handling objects like datetimeindex in json…
bnb32 Jul 10, 2024
d56a0ed
need to pass time_index to bias_transforms as list bc it gets convert…
bnb32 Jul 10, 2024
4660b5e
lower casing in docs
bnb32 Jul 10, 2024
f2986f1
fixed n_chunk calc in tests
bnb32 Jul 10, 2024
dea1b60
redudant code: moved cmd log to add_cmd_status.
bnb32 Jul 10, 2024
eb7dc4a
swapping out time_index arg for date_range_kwargs. Time indices can b…
bnb32 Jul 10, 2024
22ebb83
need string rep of timestamp
bnb32 Jul 10, 2024
8d33713
some fwp strat init cleaning
bnb32 Jul 11, 2024
7c5cb36
rename: exo_kwargs -> exo_handler_kwargs.
bnb32 Jul 12, 2024
cc67308
yml linting?
bnb32 Jul 12, 2024
b8e1f92
refact: removed getattr -> fwp strat in fwp, removed output handler a…
bnb32 Jul 12, 2024
e5c7221
refact: split collector classes into collectors dir with nc, base, h5…
bnb32 Jul 12, 2024
2df1c0e
todo flag added for generalizing pool kick off
bnb32 Jul 12, 2024
d27dca4
fix: circular imports from splitting collectors
bnb32 Jul 12, 2024
c6503ae
refact: removed uneeded attrs in ForwardPass
bnb32 Jul 12, 2024
ac3f176
feat: added compute call if dask before fwp generator call
bnb32 Jul 12, 2024
3721cde
feat: skip preflight for fwp strat on head nodes.
bnb32 Jul 12, 2024
4d0fc4d
refact: cleaned up fwp strategy init and setup skip of some init rout…
bnb32 Jul 13, 2024
5b4ba1f
fix: missing time index in input handler when features = []
bnb32 Jul 13, 2024
2651f55
refact: _parse_time_slice
bnb32 Jul 15, 2024
6ab0873
fix: len(strat.node_chunks) -> n_chunks
bnb32 Jul 15, 2024
3f7d7c3
adds: repeat topo over time to be consistently 3d across features.
bnb32 Jul 15, 2024
28ee2f0
refact: simplifying stats compute methods.
bnb32 Jul 15, 2024
d435804
add: __rmul__ for multipliying Sup3rX objects by weights
bnb32 Jul 16, 2024
45383f6
fix: lower exo feature names
bnb32 Jul 16, 2024
10dc3af
add: writing to tmp file first for cacher
bnb32 Jul 16, 2024
e6132b7
fix: pr test actions
bnb32 Jul 16, 2024
31ea00f
stupid gh actions
bnb32 Jul 16, 2024
1d0bada
added `sample` method to quickly access data for batching. updated fr…
bnb32 Jul 17, 2024
eb68749
missed imports for np.random update
bnb32 Jul 17, 2024
60e48f9
removed unused code in accessor and sup3rdataset
bnb32 Jul 17, 2024
3d4a027
fix: height interp mask fails when multiple values match min distance.
bnb32 Jul 18, 2024
dd6a0ee
ensuring np.float32 for height interp
bnb32 Jul 18, 2024
41b9cb1
removed default device from model load (now included in params). some…
bnb32 Jul 19, 2024
ffb4d5c
some linting clean up
bnb32 Jul 19, 2024
53069e4
cyclic import fix
bnb32 Jul 19, 2024
ef76b6e
rebasing on time dependent qdm merge
bnb32 Jul 19, 2024
8fc0600
linting
bnb32 Jul 19, 2024
fbe8e86
fix: using set() put features in the wrong order.
bnb32 Jul 19, 2024
6be9677
trying "synchronize" to trigger tests
bnb32 Jul 19, 2024
7c9b991
default should trigger tests. so heres a check
bnb32 Jul 19, 2024
a58d720
dtype = np.float32 checks for height interp. qdm test fixes after rebase
bnb32 Jul 20, 2024
b1b5bf5
fix: ill defined gen config fixture
bnb32 Jul 20, 2024
d3dd59b
pickle doesnt like un-computed mask from argmin. added compute call. …
bnb32 Jul 21, 2024
43fb3b3
enforcing same rng state across all tests. some logging added to era …
bnb32 Jul 22, 2024
b9e65f4
dont need legacy np.random call
bnb32 Jul 22, 2024
bf21ea6
additional test for dual feature sets
bnb32 Jul 22, 2024
a1624b6
moved normalization to outside of batch iteration. more performant.
bnb32 Jul 23, 2024
76b95c4
stats args removed from batch queue. this is now handled just by stat…
bnb32 Jul 23, 2024
88819d6
normalization fixes
bnb32 Jul 23, 2024
744939c
rebase fixes on presrat
bnb32 Jul 23, 2024
6c7e3cd
methods in bias transforms currently need eager loading for some arra…
bnb32 Jul 24, 2024
e48cc9d
enabled loading of bc factor files with Extracter class. currently co…
bnb32 Jul 24, 2024
9fc1e15
get source type hdf -> h5
bnb32 Jul 24, 2024
9b250fb
unused fixture in qdm tests. date_range_kwargs in presat tests. datas…
bnb32 Jul 24, 2024
f38133c
check for time independence during data extraction
bnb32 Jul 24, 2024
0fa79aa
test fixes
bnb32 Jul 24, 2024
51802a1
moved bias correction to `init_chunk` instead of performing on full d…
bnb32 Jul 25, 2024
68358c8
changed samplers to sample batches using adjacent time slices which a…
bnb32 Jul 27, 2024
54f1828
sampler updates for tests. rex version req bump
bnb32 Jul 27, 2024
6df02bf
reducing loading mem by casting elevation to dask array. moved duplic…
bnb32 Jul 27, 2024
a499420
slight change with cdapi update
bnb32 Jul 27, 2024
3256c64
moved loader name maps and Dimension class to names.py. accounted for…
bnb32 Jul 28, 2024
0dc8928
more efficient era downloader writing. incremental stats - will load …
bnb32 Jul 28, 2024
6120a31
Bump `rex` req
ppinchuk Jul 29, 2024
6fcd406
Deprecate `Regridder` implementation
ppinchuk Jul 29, 2024
f552ddb
Drop regridding test (now in `rex`)
ppinchuk Jul 29, 2024
0d0dda2
Update import
ppinchuk Jul 29, 2024
e1b8f87
Remove unused import
ppinchuk Jul 29, 2024
c72c817
Merge pull request #228 from NREL/pp/rex_regridder
ppinchuk Jul 29, 2024
43e6242
np.asarray replacement for _compute_if_dask in most places. added fwp…
bnb32 Jul 30, 2024
d51d887
Loaders and loader utils used in era_downloader to remove duplicate c…
bnb32 Jul 30, 2024
80dbb5c
-rename: extracter -> rasterizer
bnb32 Jul 31, 2024
de031fe
cleaned up signature and doc parsing utils and tests. gaps version bump
bnb32 Jul 31, 2024
20b4fa2
gaps version bump.
bnb32 Aug 1, 2024
1b18ed4
log_args decorator in era downloader. removed manual logging
bnb32 Aug 1, 2024
53f6968
simplified auto gen of composite signatures and docs. moved some logi…
bnb32 Aug 1, 2024
725c880
sup3rmeta subclasscheck update
bnb32 Aug 1, 2024
bb2c0c1
type hint fix
bnb32 Aug 1, 2024
27f1eae
test fixes
bnb32 Aug 1, 2024
749633b
nc for cc shouldnt actually have base loader arg. fixed order of comp…
bnb32 Aug 1, 2024
250a0fe
test on units persisting through load and temp change to celsius. rem…
bnb32 Aug 3, 2024
8bd542d
rh look up tests for nc cc handler
bnb32 Aug 3, 2024
c045b81
general exo rasterizers for nc and h5 and any feature, with Sup3rX me…
bnb32 Aug 4, 2024
a27110e
added recursive derivation. simplified exo handling: get highest res …
bnb32 Aug 5, 2024
fbde72e
self reference in feature registry leads to infinite recursion - adde…
bnb32 Aug 6, 2024
3c9c72c
sup3rwind citation update
bnb32 Aug 6, 2024
a707019
good error from 3.8 - bad dims in nc writer
bnb32 Aug 6, 2024
33cc102
readme updates and test fixes
bnb32 Aug 6, 2024
9b1f29b
missed parse time slice in fwp strat
bnb32 Aug 6, 2024
a6acb90
fix: broken rename
bnb32 Aug 6, 2024
40e5021
cleaned up `__getitem__` and added `.values` for @grantbuster :) also…
bnb32 Aug 7, 2024
211976f
need to return dataarray when ellipsis in `__getitem__` keys
bnb32 Aug 7, 2024
fe47f3e
more __getitem__ simplifying
bnb32 Aug 7, 2024
71073d2
keys parsing fix in sup3rx getitem
bnb32 Aug 7, 2024
d04e6e7
older xarray doesnt have to_dataarray()
bnb32 Aug 7, 2024
3d4114e
edge case in getitem. spatial only feature caching / regridding
bnb32 Aug 7, 2024
47d1fe7
custom types removed.
bnb32 Aug 7, 2024
d8e0fe2
simplified args for `Sup3rDataset`. Single check for tuples in `Conta…
bnb32 Aug 8, 2024
acea804
bad type check fix
bnb32 Aug 8, 2024
0e32ff5
xr.Dataset.load() method conflict with Loader. change latter to _load()
bnb32 Aug 9, 2024
ca52250
sza derived feature added. wrapped some function from rex class with …
bnb32 Aug 11, 2024
a708443
use dask solar zenith in sza exo rasterizer
bnb32 Aug 11, 2024
b3f8260
out of core writing to netcdf with parallel delayed dask tasks
bnb32 Aug 11, 2024
90ecb90
dont need threadpool for cacher with dask delayed tasks
bnb32 Aug 11, 2024
05166f9
netcdf version req
bnb32 Aug 11, 2024
0b26b69
netcdf4 req added
bnb32 Aug 12, 2024
c5f0dec
with netcd4f4 installed engine='h5netcdf' needs to be explicit. befor…
bnb32 Aug 12, 2024
5805bd4
compute needed in qa for export with RexOutput
bnb32 Aug 12, 2024
bde79ed
test fixes, netcd4 version req cap, and xr open mfdataset wrapper wit…
bnb32 Aug 12, 2024
009edd5
one annoying test remaining
bnb32 Aug 12, 2024
2ae0cbc
delayed dask and schedulers used to replace threadpoolexecutor logic.…
bnb32 Aug 12, 2024
77c36ca
added initial meta / time_index building from only unique spatial / t…
bnb32 Aug 15, 2024
e6ed04a
collection speed up fix. attrs copy update for rex update. rex versio…
bnb32 Aug 16, 2024
a8a7e28
caching h5 with lat / lon instead of /meta/lat and /meta/lon.
bnb32 Aug 16, 2024
6bbdf98
more epochs for training test
bnb32 Aug 16, 2024
9957800
not caching exo data with time if time independent. this can be a hug…
bnb32 Aug 18, 2024
0f34e35
bottleneck dependency for fast ffill operation, and others.
bnb32 Aug 18, 2024
3117212
added method to automatically determine exogenous data steps from lr_…
bnb32 Aug 18, 2024
78e69ce
test fixes
bnb32 Aug 18, 2024
a00fdaf
test fix
bnb32 Aug 19, 2024
ca3b03e
changed masking logic in level interpolation to enable delayed comput…
bnb32 Aug 20, 2024
f619328
auto check on exo features needs model list from solar multi step gan.
bnb32 Aug 21, 2024
d719c59
run caching tests with max_workers=1
bnb32 Aug 21, 2024
dacd1d5
fixes for solar multi step gan + exo data
bnb32 Aug 22, 2024
9453b25
limit on padding removed since data is loaded as ndarray before paddi…
bnb32 Aug 22, 2024
9058109
type checking additions
bnb32 Aug 22, 2024
bafcb8d
test fixes
bnb32 Aug 22, 2024
9e71ed6
solar multistep fix and max_workers -> 1 for tests
bnb32 Aug 22, 2024
fccea44
default max_workers=1 for cacher
bnb32 Aug 22, 2024
7bac634
adjustment for ensemble downloads
bnb32 Aug 27, 2024
79a4602
h5_attrs -> output_attrs. used in enforce_limits in both netcdf and h…
bnb32 Aug 27, 2024
41f82dc
rsds added to output_attrs. other tests will pass when rex qdm_lim br…
bnb32 Aug 27, 2024
9430b02
moved calculation of k factor and tau into separate static methods fo…
grantbuster Jul 27, 2024
0c5f9b9
moved presrat transform into separate method for modular call directl…
grantbuster Jul 27, 2024
d56f70f
bug fix: wrong time index passed to run single and default relative s…
grantbuster Jul 28, 2024
3edbcef
more explicit enforcement of a zero threshold value and passing to QD…
grantbuster Jul 28, 2024
8ee6b2b
protect against divide by zero in calculation of k
grantbuster Jul 29, 2024
e413982
moved ntimesteps and window size to kwargs so easier to modify. Found…
grantbuster Jul 31, 2024
5657774
moved presrat class to its own module, merged new fixes with dh_refac…
grantbuster Aug 1, 2024
6596038
fixed presrat test - zero rate threshold can manipulate values now
grantbuster Aug 1, 2024
ca862eb
added outrange feature to qdm and presrat
grantbuster Aug 5, 2024
4314053
delta denom min should fix the low precip value issues and no need to…
grantbuster Aug 16, 2024
8d8474c
bug fix for daily cs ghi == 0 causing nan values in clearsky ratio
grantbuster Aug 16, 2024
1c1056d
added tmin/tmax to h5 attrs
grantbuster Aug 28, 2024
1fdf415
dont need to derive clearsky_ghi if needed feature is already in load…
bnb32 Aug 28, 2024
a7c4d1d
modfication to SolarCC model to enable running as stand alone forward…
bnb32 Aug 28, 2024
0d198b6
dont need separate status fle write. this is handled in get_node_cmd
bnb32 Aug 30, 2024
661e650
clean up docstrings
grantbuster Aug 30, 2024
83d6475
bump rex version requirement for QDM kwargs
grantbuster Aug 30, 2024
ce76db8
split up attribute collection for meta and time index. significantly …
bnb32 Aug 31, 2024
af2336b
redistribution of unfinished chunks needs some refactoring. reverting…
bnb32 Aug 31, 2024
7e90cb0
added collection test with more chunks
bnb32 Aug 31, 2024
c21ed3c
added time_shift argument to deriver and data handler so that daily d…
bnb32 Sep 4, 2024
a5f42b8
moved output attrs to json file and included in setup package data
grantbuster Sep 12, 2024
ae2324b
Merge branch 'bnb/dh_refactor' into gb/presrat_updates_2
bnb32 Sep 12, 2024
84d3be5
linting fix
bnb32 Sep 12, 2024
c692000
Merge pull request #230 from NREL/gb/presrat_updates_2
bnb32 Sep 12, 2024
7232eb1
regex edit for correctly getting chunk indices from solar module outp…
bnb32 Sep 13, 2024
1920fcd
increase padding for failing test
bnb32 Sep 13, 2024
3808db1
time_shift arg added to run_temporal_chunk so this can be provided in…
bnb32 Sep 14, 2024
e4f009f
time shift added to solar module, so that daily gcm data can be shift…
bnb32 Sep 12, 2024
9042364
12 default shift only meant for current solar module runs.
bnb32 Sep 16, 2024
2aa353b
default time shift as 0.
bnb32 Sep 16, 2024
7c8fe53
era downloader test fixes
bnb32 Sep 16, 2024
711c471
missed era downloader updates from other branch
bnb32 Sep 17, 2024
dd6250a
xr open mfdataset wrapper added
bnb32 Sep 17, 2024
8ae857d
dont need time shift in the solar module. can just shift this in a se…
bnb32 Sep 17, 2024
e4428fd
better node distribution for solar module
bnb32 Sep 19, 2024
7ec885a
removing todo note for better solar module node distributuon
bnb32 Sep 20, 2024
69e8344
removed expanded time dimension in exo rasterizer for time independen…
bnb32 Sep 21, 2024
cf5e466
exo test fix with removal of time dimension expansion for time indepe…
bnb32 Sep 21, 2024
e9b42e9
only use temporal slices to get exo data chunk if exo data is time de…
bnb32 Sep 21, 2024
2ea6206
need to check coords also for pressure derivation. fixed this in base…
bnb32 Sep 24, 2024
e7e44b1
missed default xr kwargs for combining monthly files
bnb32 Sep 24, 2024
92ee0f7
typo in xr_open_mfdataset logging
bnb32 Sep 24, 2024
c3a3309
added height interpolation for the case of just single level data. e.…
bnb32 Sep 26, 2024
b900d0d
wrong conditions on combining lev and var arrays in interpolation rou…
bnb32 Sep 27, 2024
cc9eaae
``.values`` method fix - returns ``as_array`` but loaded into memory.…
bnb32 Oct 14, 2024
71d4000
lowered bottleneck verrsion req
bnb32 Oct 14, 2024
94975f9
removed bottlenexck requirement
bnb32 Oct 15, 2024
9d5f23e
bias calc cli needs to be able to reference QDM and presrat
grantbuster Oct 15, 2024
9ea18ec
try preloading dh data - parallel compute is prohibitivly slow with l…
grantbuster Oct 16, 2024
df9ee4e
docs
grantbuster Oct 16, 2024
2d01ae9
preload kwarg and method for bias calc
grantbuster Oct 16, 2024
79a1e7a
attempted fix on numpy array indexing with preloaded data handler
grantbuster Oct 16, 2024
31a2f38
added test for various slicing operations on sup3rdataset with and wi…
grantbuster Oct 16, 2024
96fc946
check datahandler objects for compute method so rex classes dont quer…
grantbuster Oct 16, 2024
532eb4d
if chunks=None load into memory. This mimics the behavior of xr.open_…
bnb32 Oct 17, 2024
e6f7dc5
Merge pull request #233 from NREL/bnb/compute_on_none_chunks
grantbuster Oct 21, 2024
f864740
Merge pull request #232 from NREL/gb/bc_debug
grantbuster Oct 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .flake8

This file was deleted.

2 changes: 1 addition & 1 deletion .github/linters/.flake8
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[flake8]
ignore = E731,E402,F,W503
ignore = E731,E402,F,W503,E203
exclude = .git,__pycache__,docs/source/conf.py,old,build,dist,bin/tmp/*
max-complexity = 12
4 changes: 3 additions & 1 deletion .github/linters/.python-lint
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ confidence=
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
disable=
no-member,
attribute-defined-outside-init,
arguments-renamed,
unspecified-encoding,
consider-using-f-string,
Expand Down Expand Up @@ -498,4 +500,4 @@ known-third-party=enchant

# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=Exception
overgeneral-exceptions=builtin.Exception
1 change: 1 addition & 0 deletions .github/workflows/linter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ jobs:
VALIDATE_PYTHON_ISORT: false
VALIDATE_PYTHON_MYPY: false
VALIDATE_DOCKERFILE_HADOLINT: false
VALIDATE_GITHUB_ACTIONS: false
VALIDATE_JSCPD: false
VALIDATE_JSON: false
VALIDATE_MARKDOWN: false
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pull_request_tests.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: Pytests

on:
on:
pull_request:
workflow_dispatch:

Expand Down Expand Up @@ -37,4 +37,4 @@ jobs:
python -m pip install .[test]
- name: Run pytest
run: |
python -m pytest -v --disable-warnings
python -m pytest -v --disable-warnings
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ nosetests.xml
coverage.xml
*.cover
.hypothesis/
coverage/lcov.info
lcov.info
*.png

# Translations
Expand Down Expand Up @@ -119,3 +121,6 @@ tags
# pixi environments
.pixi
*.egg-info

# test dirs
exo_cache
33 changes: 20 additions & 13 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,25 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-json
- id: check-merge-conflict
- id: check-symlinks
- id: check-toml
- id: flake8
- id: mixed-line-ending
- repo: https://github.com/PyCQA/pylint
- id: check-json
- id: check-yaml
- id: check-merge-conflict
- id: check-symlinks
- id: check-toml
- id: flake8
args: [--config, .github/linters/.flake8]
- id: mixed-line-ending
- repo: https://github.com/PyCQA/pylint
rev: v3.1.0
hooks:
- id: pylint
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.4.10
hooks:
- id: ruff
- id: pylint
args:
[
--rcfile,
.github/linters/.python-lint,
]
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.5.3
hooks:
- id: ruff
73 changes: 8 additions & 65 deletions .pylintrc
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[MASTER]
[MAIN]

# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
Expand Down Expand Up @@ -55,76 +55,20 @@ confidence=
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
disable=
no-name-in-module,
no-member,
attribute-defined-outside-init,
arguments-renamed,
unspecified-encoding,
consider-using-f-string,
# Defaults
print-statement,
parameter-unpacking,
unpacking-in-except,
old-raise-syntax,
backtick,
import-star-module-level,
raw-checker-failed,
bad-inline-option,
locally-disabled,
file-ignored,
suppressed-message,
useless-suppression,
deprecated-pragma,
apply-builtin,
basestring-builtin,
buffer-builtin,
cmp-builtin,
coerce-builtin,
execfile-builtin,
file-builtin,
long-builtin,
raw_input-builtin,
reduce-builtin,
standarderror-builtin,
unicode-builtin,
xrange-builtin,
coerce-method,
delslice-method,
getslice-method,
setslice-method,
no-absolute-import,
old-division,
dict-iter-method,
dict-view-method,
next-method-called,
metaclass-assignment,
indexing-exception,
raising-string,
reload-builtin,
oct-method,
hex-method,
nonzero-method,
cmp-method,
input-builtin,
round-builtin,
intern-builtin,
unichr-builtin,
map-builtin-not-iterating,
zip-builtin-not-iterating,
range-builtin-not-iterating,
filter-builtin-not-iterating,
using-cmp-argument,
div-method,
idiv-method,
rdiv-method,
exception-message-attribute,
invalid-str-codec,
sys-max-int,
bad-python3-import,
deprecated-string-function,
deprecated-str-translate-call,
deprecated-itertools-function,
deprecated-types-field,
next-method-defined,
dict-items-not-iterating,
dict-keys-not-iterating,
dict-values-not-iterating,
consider-using-f-string,
unspecified-encoding,
# Custom
protected-access,
fixme,
Expand All @@ -145,7 +89,6 @@ disable=
too-many-nested-blocks,
invalid-name,
import-error,
bad-continuation,
try-except-raise,
no-else-raise,
no-else-return,
Expand Down Expand Up @@ -558,4 +501,4 @@ known-third-party=enchant

# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=Exception
overgeneral-exceptions=builtin.Exception
4 changes: 2 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#################
*****************
Welcome to SUP3R!
#################
*****************

|Docs| |Tests| |Linter| |PyPi| |PythonV| |Codecov| |Zenodo|

Expand Down
7 changes: 3 additions & 4 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# -*- coding: utf-8 -*-
"""
Documentation config file
"""
Expand All @@ -16,8 +15,8 @@
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sphinx_rtd_theme
import sys

sys.path.insert(0, os.path.abspath('../../'))

# -- Project information -----------------------------------------------------
Expand All @@ -31,6 +30,7 @@
sys.path.append(pkg)

from sup3r import __version__ as v

# The short X.Y version
version = v
# The full version, including alpha/beta/rc tags
Expand Down Expand Up @@ -104,8 +104,7 @@
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
html_theme = 'pydata_sphinx_theme'

# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
Expand Down
2 changes: 1 addition & 1 deletion examples/sup3rcc/run_configs/solar/config_fwp.json
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
"input_handler_kwargs": {
"target": [23.2, -129],
"shape": [26, 59],
"temporal_slice": [null, null, null],
"time_slice": [null, null, null],
"nsrdb_source_fp": "/scratch/gbuster/sup3r/source_gcm_data/nsrdb_clearsky.h5",
"nsrdb_agg": 625,
"nsrdb_smoothing": 0,
Expand Down
2 changes: 1 addition & 1 deletion examples/sup3rcc/run_configs/trh/config_fwp.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"input_handler_kwargs": {
"target": [23.2, -129],
"shape": [26, 59],
"temporal_slice": [null, null, null],
"time_slice": [null, null, null],
"worker_kwargs": {
"max_workers": 1
}
Expand Down
2 changes: 1 addition & 1 deletion examples/sup3rcc/run_configs/wind/config_fwp.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"input_handler_kwargs": {
"target": [23.2, -129],
"shape": [26, 59],
"temporal_slice": [null, null, null],
"time_slice": [null, null, null],
"worker_kwargs": {
"max_workers": 1
}
Expand Down
4 changes: 2 additions & 2 deletions examples/sup3rwind/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@ The Sup3rWind data is also loaded into `HSDS <https://www.hdfgroup.org/solutions
Example Sup3rWind Data Usage
-----------------------------

Sup3rWind data can be used in generally the same way as Sup3rCC data, with the condition that Sup3rWind includes only wind data and ancillary variables for modeling wind energy generation. Refer to the Sup3rCC example notebook `here <https://github.com/NREL/sup3r/tree/main/examples/sup3rcc/using_the_data.ipynb>`_ for usage patterns.
Sup3rWind data can be used in generally the same way as `Sup3rCC <https://nrel.github.io/sup3r/examples/sup3rcc.html>`_ data, with the condition that Sup3rWind includes only wind data and ancillary variables for modeling wind energy generation. Refer to the Sup3rCC example notebook `here <https://github.com/NREL/sup3r/tree/main/examples/sup3rcc/using_the_data.ipynb>`_ for usage patterns.

Running Sup3rWind Models
-------------------------

The process for running the Sup3rWind models is much the same as for Sup3rCC (``sup3r/examples/sup3rcc/README.rst``).
The process for running the Sup3rWind models is much the same as for `Sup3rCC <https://nrel.github.io/sup3r/examples/sup3rcc.html>`_.

#. Download the Sup3rWind models to your hardware using the AWS CLI: ``$ aws s3 cp s3://nrel-pds-wtk/sup3rwind/models/``
#. Download the ERA5 data that you want to downscale from `ERA5-single-levels <https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview/>`_ and/or `ERA5-pressure-levels <https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=overview/>`_.
Expand Down
29 changes: 20 additions & 9 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ classifiers=[
"Programming Language :: Python :: 3.11",
]
dependencies = [
"NREL-rex>=0.2.84",
"NREL-rex>=0.2.87",
"NREL-phygnn>=0.0.23",
"NREL-gaps>=0.6.0",
"NREL-gaps>=0.6.13",
"NREL-farms>=1.0.4",
"dask>=2022.0",
"h5netcdf>=1.1.0",
Expand All @@ -54,11 +54,12 @@ dev = [
]
doc = [
"sphinx>=7.0",
"sphinx_rtd_theme>=2.0",
"sphinx-click>=4.0",
"pydata-sphinx-theme>=15.4"
]
test = [
"pytest>=5.2",
"pytest-env"
]

[project.urls]
Expand Down Expand Up @@ -224,10 +225,10 @@ max-complexity = 12
[tool.ruff.lint.per-file-ignores]
"__init__.py" = [
"F401", # unused-import
]
#"docs/source/conf.py" = [
# "E402", # unused-import
# ]
]
"docs/source/conf.py" = [
"E402", # unused-import
]

[tool.ruff.lint.pylint]
max-args = 5 # (PLR0913) Maximum number of arguments for function / method
Expand Down Expand Up @@ -274,16 +275,17 @@ xarray = ">=2023.0"

[tool.pixi.pypi-dependencies]
NREL-sup3r = { path = ".", editable = true }
NREL-rex = { version = ">=0.2.84" }
NREL-rex = { version = ">=0.2.87" }
NREL-phygnn = { version = ">=0.0.23" }
NREL-gaps = { version = ">=0.6.0" }
NREL-gaps = { version = ">=0.6.13" }
NREL-farms = { version = ">=1.0.4" }

[tool.pixi.environments]
default = { solve-group = "default" }
dev = { features = ["dev", "doc", "test"], solve-group = "default" }
doc = { features = ["doc"], solve-group = "default" }
test = { features = ["test"], solve-group = "default" }
viz = { features = ["viz"], solve-group = "default" }
bnb32 marked this conversation as resolved.
Show resolved Hide resolved

[tool.pixi.tasks]
test = "pytest --pdb --durations=10 tests"
Expand All @@ -294,10 +296,19 @@ sphinx_rtd_theme = ">=2.0"

[tool.pixi.feature.test.dependencies]
pytest = ">=5.2"
pytest-cov = ">=5.0.0"

[tool.pixi.feature.dev.dependencies]
build = ">=0.6"
twine = ">=5.0"
ruff = ">=0.4"
ipython = ">=8.0"
pytest-xdist = ">=3.0"

[tool.pixi.feature.viz.dependencies]
jupyter = ">1.0.0"
bnb32 marked this conversation as resolved.
Show resolved Hide resolved
hvplot = ">0.10.0"
bnb32 marked this conversation as resolved.
Show resolved Hide resolved

[tool.pytest_env]
CUDA_VISIBLE_DEVICES = "-1"
TF_ENABLE_ONEDNN_OPTS = "0"
Loading
Loading