Can't create new cluster after modifying environment

[rsignell-usgs]

Qhub gurus, I guess I'm being punished for bragging about qhub: this morning after adding a package to my environment, I couldn't start a cluster. I then tried increasing the size of my worker instance and increasing the size of my "Pangeo worker". Neither worked. Still can't create a cluster.

The error says:

GatewayClusterError: Cluster 'ebeca6a0ac634498869cf9ee4367a444' failed to start, see logs for more information

How do I see the logs in question?

[dharhas]

dask/dask-gateway#355

Seems to indicate that you should use kubectl logs

[rsignell-usgs]

@tylerpotts, the above commands worked, and I can see in k9s that the gateway-dask-gateway pod started up again, but we still can't create clusters with our custom environments.

Could I try reverting to earlier images by simply logging into the conda-store pod and deleting the most recent environment images?

I think this must be related to the new environment images because I can still start a cluster if I use the "default" environment image.

[rsignell-usgs]

I'm getting 404s from the gateway-scheduler on some image. But that shouldn't matter, right?

Here's what our pods look like, in case you seen anything suspicious:

[jkellndorfer]

Same error for me as @rsignell-usgs ! Can't start a cluster anymore.
GatewayClusterError: Cluster 'b9de302b5dc5432bb4af20a7ffe18d11' failed to start, see logs for more information

[rsignell-usgs]

@costrouc or @tylerpotts , did something change on your end recently that would cause the environment builds via github actions to build differently?
If you create/modify an environment on one of your qhubs, can you then start a cluster with that environment?
I'm betting no.

[tylerpotts]

@rsignell-usgs @jkellndorfer I'm escalating this internally and will get back to you

[brl0]

I would be curious to see what packages and versions are in the problematic environments. Could you run a conda list on the environment and post the info here? You can make it collapsible by wrapping it in <details> and </details>.

[rsignell-usgs]

@brl0, thanks for helping us out here. Here's the conda list:

rsignell-usgs@jupyter-rsignell-2dusgs:~$ conda list
# packages in environment at /home/conda/environments/pangeo:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
abseil-cpp                20200923.3           h9c3ff4c_0    conda-forge
adal                      1.2.6              pyh44b312d_0    conda-forge
adlfs                     0.6.3              pyhd8ed1ab_0    conda-forge
affine                    2.3.0                      py_0    conda-forge
aiobotocore               1.2.1              pyhd8ed1ab_0    conda-forge
aiohttp                   3.7.4            py38h497a2fe_0    conda-forge
aioitertools              0.7.1              pyhd8ed1ab_0    conda-forge
anyio                     2.2.0            py38h578d9bd_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py38h497a2fe_2    conda-forge
arrow-cpp                 3.0.0           py38h14434a2_7_cpu    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
async_generator           1.10                       py_0    conda-forge
atk-1.0                   2.36.0               h3371d22_4    conda-forge
attrs                     20.3.0             pyhd3deb0d_0    conda-forge
aws-c-cal                 0.4.5                hb3cee0b_7    conda-forge
aws-c-common              0.5.1                h7f98852_0    conda-forge
aws-c-event-stream        0.2.7                hf6e4101_0    conda-forge
aws-c-io                  0.9.0                hd4bc808_1    conda-forge
aws-checksums             0.1.11               hbd38b91_2    conda-forge
aws-sdk-cpp               1.8.138              hfa2f2a4_3    conda-forge
awscli                    1.18.212         py38h578d9bd_0    conda-forge
azure-core                1.10.0             pyhd8ed1ab_0    conda-forge
azure-datalake-store      0.0.51             pyh9f0ad1d_0    conda-forge
azure-identity            1.5.0              pyhd8ed1ab_0    conda-forge
azure-storage-blob        12.6.0             pyhd3deb0d_0    conda-forge
babel                     2.9.0              pyhd3deb0d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.1                      py_0    conda-forge
beautifulsoup4            4.9.3              pyhb0f4dca_0    conda-forge
bleach                    3.3.0              pyh44b312d_0    conda-forge
blinker                   1.4                        py_1    conda-forge
blosc                     1.21.0               h9c3ff4c_0    conda-forge
bokeh                     2.3.0            py38h578d9bd_0    conda-forge
boost-cpp                 1.74.0               hc6e9bd1_2    conda-forge
boto3                     1.16.52            pyhd8ed1ab_0    conda-forge
botocore                  1.19.52            pyhd8ed1ab_0    conda-forge
branca                    0.4.2              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h9c3ff4c_4    conda-forge
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h36c2ea0_0    conda-forge
ca-certificates           2020.12.5            ha878542_0    conda-forge
cached-property           1.5.1                      py_0    conda-forge
cachetools                4.2.1              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            h7979940_1007    conda-forge
cartopy                   0.18.0          py38hab71064_13    conda-forge
certifi                   2020.12.5        py38h578d9bd_1    conda-forge
cf_xarray                 0.5.1              pyh44b312d_0    conda-forge
cffi                      1.14.5           py38ha65f79e_0    conda-forge
cfgrib                    0.9.8.5            pyhd8ed1ab_0    conda-forge
cfitsio                   3.470                hb418390_7    conda-forge
cftime                    1.4.1            py38h5c078b8_0    conda-forge
chardet                   4.0.0            py38h578d9bd_1    conda-forge
charls                    2.2.0                h9c3ff4c_0    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.1              pyhd8ed1ab_0    conda-forge
cloudpickle               1.6.0                      py_0    conda-forge
colorama                  0.4.3                      py_0    conda-forge
colorcet                  2.0.6              pyhd8ed1ab_0    conda-forge
cryptography              3.4.4            py38h3e25421_0    conda-forge
csa                       1.22.0            h14c3975_2002    conda-forge
curl                      7.71.1               he644dc0_8    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cytoolz                   0.11.0           py38h497a2fe_3    conda-forge
dask                      2.14.0                     py_0    conda-forge
dask-core                 2.14.0                     py_0    conda-forge
dask-gateway              0.6.1                    py38_0    conda-forge
dask-labextension         5.0.1              pyhd8ed1ab_1    conda-forge
datashader                0.12.0             pyhd3deb0d_0    conda-forge
datashape                 0.5.4                      py_1    conda-forge
decorator                 4.4.2                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
descartes                 1.1.0                      py_4    conda-forge
distributed               2.14.0           py38h32f6830_0    conda-forge
docopt                    0.6.2                      py_1    conda-forge
docutils                  0.15.2           py38h578d9bd_1    conda-forge
eccodes                   2.20.0               ha0e6eb6_0    conda-forge
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
entwine                   2.1.0                hc169ed7_8    conda-forge
erddapy                   0.9.0              pyhd8ed1ab_0    conda-forge
esmf                      8.0.1           nompi_he31a43a_2    conda-forge
esmpy                     8.0.1           nompi_py38h5410a82_2    conda-forge
expat                     2.2.10               h9c3ff4c_0    conda-forge
fasteners                 0.14.1                     py_3    conda-forge
fastparquet               0.5.0            py38h5c078b8_0    conda-forge
fiona                     1.8.18           py38h37fbd03_0    conda-forge
folium                    0.12.0             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      2.001                hab24e00_0    conda-forge
font-ttf-source-code-pro  2.030                hab24e00_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.13.1            hba837de_1004    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
freexl                    1.0.5             h516909a_1002    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
fsspec                    0.8.7              pyhd8ed1ab_0    conda-forge
g2clib                    1.6.0                hf3f1b0b_9    conda-forge
gcsfs                     0.7.2              pyhd8ed1ab_0    conda-forge
gdal                      3.1.4            py38h25844d8_8    conda-forge
gdk-pixbuf                2.42.2               h0c95a7a_2    conda-forge
geolinks                  0.2.0                      py_1    conda-forge
geopandas                 0.9.0              pyhd8ed1ab_0    conda-forge
geos                      3.9.1                h9c3ff4c_2    conda-forge
geotiff                   1.6.0                h2b14fbe_4    conda-forge
geoviews                  1.8.2                      py_0    conda-forge
geoviews-core             1.8.2                      py_0    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
glog                      0.4.0                h49b9bf7_3    conda-forge
google-auth               1.24.0             pyhd3deb0d_0    conda-forge
google-auth-oauthlib      0.4.2              pyhd8ed1ab_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
graphviz                  2.46.1               h93c640b_4    conda-forge
gridgen                   1.51.5            h14c3975_1000    conda-forge
gridutils                 1.0               h14c3975_1003    conda-forge
grpc-cpp                  1.36.1               h7919d58_0    conda-forge
gtk2                      2.24.33              hab0c2f8_0    conda-forge
gts                       0.7.6                h64030ff_2    conda-forge
h5netcdf                  0.10.0             pyhd8ed1ab_0    conda-forge
h5py                      3.1.0           nompi_py38hafa665b_100    conda-forge
harfbuzz                  2.7.4                h5cf4720_0    conda-forge
hdf4                      4.2.13            h10796ff_1004    conda-forge
hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
hologridgen               1.0.2              pyhd8ed1ab_0    conda-forge
holoviews                 1.14.2             pyhd8ed1ab_0    conda-forge
hvplot                    0.7.1              pyh44b312d_0    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
imagecodecs               2021.1.28        py38h5b4e65a_0    conda-forge
imageio                   2.9.0                      py_0    conda-forge
importlib-metadata        3.7.0            py38h578d9bd_0    conda-forge
importlib_metadata        3.7.0                hd8ed1ab_0    conda-forge
importlib_resources       5.1.2            py38h578d9bd_0    conda-forge
intake                    0.6.1              pyhd8ed1ab_0    conda-forge
intake-geopandas          0.2.4              pyhd8ed1ab_0    conda-forge
intake-stac               0.3.0                      py_0    conda-forge
intake-xarray             0.4.1              pyhd8ed1ab_0    conda-forge
ipykernel                 5.5.0            py38h81c977d_1    conda-forge
ipyleaflet                0.13.6             pyhd8ed1ab_1    conda-forge
ipyspin                   0.1.5              pyhd8ed1ab_0    conda-forge
ipython                   7.21.0           py38h81c977d_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.6.3              pyhd3deb0d_0    conda-forge
isodate                   0.6.0                      py_1    conda-forge
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jedi                      0.18.0           py38h578d9bd_2    conda-forge
jinja2                    2.11.3             pyh44b312d_0    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
joblib                    1.0.1              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
jq                        1.6               h36c2ea0_1000    conda-forge
json-c                    0.15                 h98cffda_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsoncpp                   1.9.4                h4bd325d_1    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter-packaging         0.7.12             pyhd8ed1ab_0    conda-forge
jupyter-panel-proxy       0.1.0                      py_0    conda-forge
jupyter-server-proxy      1.6.0              pyhd8ed1ab_0    conda-forge
jupyter_client            6.1.11             pyhd8ed1ab_1    conda-forge
jupyter_core              4.7.1            py38h578d9bd_0    conda-forge
jupyter_server            1.4.1            py38h578d9bd_0    conda-forge
jupyterlab                3.0.9              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.3.0              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        1.0.0              pyhd8ed1ab_1    conda-forge
jxrlib                    1.1                  h7f98852_2    conda-forge
kealib                    1.4.14               he4dc956_1    conda-forge
kiwisolver                1.3.1            py38h1fd1430_1    conda-forge
krb5                      1.17.2               h926e7f8_0    conda-forge
laszip                    3.4.3                he1b5a44_1    conda-forge
laz-perf                  1.5.0                he1b5a44_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
lerc                      2.2.1                h9c3ff4c_0    conda-forge
libaec                    1.0.4                h9c3ff4c_1    conda-forge
libblas                   3.9.0                8_openblas    conda-forge
libcblas                  3.9.0                8_openblas    conda-forge
libcurl                   7.71.1               hcdd3856_8    conda-forge
libdap4                   3.20.6               hd7c4107_1    conda-forge
libdeflate                1.7                  h7f98852_5    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_18    conda-forge
libgd                     2.3.0                h47910db_1    conda-forge
libgdal                   3.1.4                h38ff51b_8    conda-forge
libgfortran-ng            9.3.0               hff62375_18    conda-forge
libgfortran5              9.3.0               hff62375_18    conda-forge
libglib                   2.66.7               h3e27bee_1    conda-forge
libgomp                   9.3.0               h2828fa1_18    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libkml                    1.3.0             h02e6976_1012    conda-forge
liblapack                 3.9.0                8_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libnetcdf                 4.7.4           nompi_h56d31a8_107    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libopenblas               0.3.12          pthreads_h4812303_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     13.1                 hfd2b0eb_2    conda-forge
libprotobuf               3.15.4               h780b84a_0    conda-forge
librsvg                   2.50.3               hfa39831_1    conda-forge
librttopo                 1.1.0                h1185371_6    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libspatialindex           1.9.3                h9c3ff4c_3    conda-forge
libspatialite             5.0.1                he52d314_3    conda-forge
libssh2                   1.9.0                hab1572f_5    conda-forge
libstdcxx-ng              9.3.0               h6de172a_18    conda-forge
libthrift                 0.13.0               h5aa387f_6    conda-forge
libtiff                   4.2.0                hdc55705_0    conda-forge
libtool                   2.4.6             h58526e2_1007    conda-forge
libutf8proc               2.6.1                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp                   1.2.0                h3452ae3_0    conda-forge
libwebp-base              1.2.0                h7f98852_0    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxml2                   2.9.10               h72842e0_3    conda-forge
libxslt                   1.1.33               h15afd5d_2    conda-forge
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
llvmlite                  0.35.0           py38h4630a5e_1    conda-forge
locket                    0.2.0                      py_2    conda-forge
lxml                      4.6.2            py38hf1fe3a4_1    conda-forge
lz4                       3.1.3            py38hebdc3cf_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
markdown                  3.3.4              pyhd8ed1ab_0    conda-forge
markupsafe                1.1.1            py38h497a2fe_3    conda-forge
matplotlib-base           3.3.4            py38h0efea84_0    conda-forge
mercantile                1.1.6              pyh9f0ad1d_0    conda-forge
metpy                     1.0                pyhd8ed1ab_0    conda-forge
mistune                   0.8.4           py38h497a2fe_1003    conda-forge
monotonic                 1.5                        py_0    conda-forge
msal                      1.9.0              pyh44b312d_0    conda-forge
msal_extensions           0.3.0              pyh9f0ad1d_0    conda-forge
msgpack-python            1.0.2            py38h1fd1430_1    conda-forge
msrest                    0.6.21             pyh44b312d_0    conda-forge
msrestazure               0.6.4              pyhd8ed1ab_0    conda-forge
multidict                 5.1.0            py38h497a2fe_1    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
munch                     2.5.0                      py_0    conda-forge
mysql-connector-c         6.1.11            h6eb9d5d_1007    conda-forge
nbclassic                 0.2.6              pyhd8ed1ab_0    conda-forge
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
nbconvert                 6.0.7            py38h578d9bd_3    conda-forge
nbformat                  5.1.2              pyhd8ed1ab_1    conda-forge
nbgitpuller               0.9.0                      py_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nest-asyncio              1.4.3              pyhd8ed1ab_0    conda-forge
netcdf-fortran            4.5.3           nompi_h1a0d97b_101    conda-forge
netcdf4                   1.5.6           nompi_py38h1cdf482_100    conda-forge
networkx                  2.5                        py_0    conda-forge
nitro                     2.7.dev6             he1b5a44_3    conda-forge
nn                        1.86.0            h14c3975_2003    conda-forge
notebook                  6.2.0            py38h578d9bd_0    conda-forge
numba                     0.52.0           py38h51da96c_0    conda-forge
numcodecs                 0.7.3            py38h709712a_0    conda-forge
numpy                     1.20.1           py38h18fd61f_0    conda-forge
oauthlib                  3.0.1                      py_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
oniguruma                 6.9.3                h36c2ea0_0    conda-forge
openjpeg                  2.4.0                hf7af979_0    conda-forge
openssl                   1.1.1j               h7f98852_0    conda-forge
orc                       1.6.7                heec2584_1    conda-forge
osmnx                     1.0.1              pyhd3deb0d_0    conda-forge
owslib                    0.23.0             pyhd8ed1ab_0    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pandas                    1.2.3            py38h51da96c_0    conda-forge
pandoc                    2.11.4               h7f98852_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
panel                     0.11.0             pyhd8ed1ab_0    conda-forge
pango                     1.42.4               h69149e4_5    conda-forge
param                     1.10.1             pyhd3deb0d_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.8.1              pyhd8ed1ab_0    conda-forge
partd                     1.1.0                      py_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pdal                      2.2.0                h73bc3e3_6    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.1.1            py38ha0e1e83_0    conda-forge
pint                      0.16.1                     py_0    conda-forge
pip                       21.0.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pooch                     1.3.0              pyhd8ed1ab_0    conda-forge
poppler                   0.89.0               h2de54a5_5    conda-forge
poppler-data              0.4.10                        0    conda-forge
portalocker               1.7.0            py38h578d9bd_1    conda-forge
postgresql                13.1                 h6303168_2    conda-forge
proj                      7.2.0                h277dcde_2    conda-forge
prometheus_client         0.9.0              pyhd3deb0d_0    conda-forge
prompt-toolkit            3.0.16             pyha770c72_0    conda-forge
protobuf                  3.15.4           py38h709712a_0    conda-forge
psutil                    5.8.0            py38h497a2fe_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pyarrow                   3.0.0           py38h92f0514_7_cpu    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pyepsg                    0.4.0                      py_0    conda-forge
pygments                  2.8.0              pyhd8ed1ab_0    conda-forge
pygridgen                 0.2.1            py38hab2c0dc_4    conda-forge
pyjwt                     1.7.1                      py_0    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyproj                    3.0.0.post1      py38h8df5ac0_0    conda-forge
pyrsistent                0.17.3           py38h497a2fe_2    conda-forge
pyshp                     2.1.3              pyh44b312d_0    conda-forge
pysocks                   1.7.1            py38h578d9bd_3    conda-forge
python                    3.8.8           hffdb5ce_0_cpython    conda-forge
python-dateutil           2.7.5                      py_0    conda-forge
python-gist               0.9.2              pyh9f0ad1d_1    conda-forge
python-gnupg              0.4.6              pyh9f0ad1d_0    conda-forge
python-snappy             0.5.4            py38hd7b2518_3    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyviz_comms               2.0.1              pyhd3deb0d_0    conda-forge
pywavelets                1.1.1            py38h5c078b8_3    conda-forge
pyyaml                    5.3.1            py38h497a2fe_2    conda-forge
pyzmq                     22.0.3           py38h2035c66_1    conda-forge
rasterio                  1.2.1            py38h033aa8a_0    conda-forge
re2                       2020.11.01           h58526e2_0    conda-forge
readline                  8.0                  he28a2e2_2    conda-forge
rechunker                 0.3.3              pyhd8ed1ab_0    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge
retrying                  1.3.3                      py_2    conda-forge
rio-cogeo                 2.0.1                      py_0    conda-forge
rioxarray                 0.3.1              pyhd8ed1ab_0    conda-forge
rsa                       4.4.1              pyh9f0ad1d_0    conda-forge
rtree                     0.9.7            py38h02d302b_1    conda-forge
s2n                       1.0.0                h9b69904_0    conda-forge
s3fs                      0.5.2              pyhd8ed1ab_0    conda-forge
s3transfer                0.3.4              pyhd8ed1ab_0    conda-forge
sat-search                0.3.0                      py_0    conda-forge
sat-stac                  0.4.1              pyh44b312d_0    conda-forge
scikit-image              0.18.1           py38h51da96c_0    conda-forge
scikit-learn              0.24.1           py38h658cfdd_0    conda-forge
scipy                     1.6.0            py38hb2138dd_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                49.6.0           py38h578d9bd_3    conda-forge
shapely                   1.7.1            py38h4fc1155_4    conda-forge
simpervisor               0.4                pyhd8ed1ab_0    conda-forge
simplejson                3.17.2           py38h497a2fe_2    conda-forge
siphon                    0.8.0           py38h578d9bd_1004    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
snappy                    1.1.8                he1b5a44_3    conda-forge
sniffio                   1.2.0            py38h578d9bd_1    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.0.1                      py_1    conda-forge
spatialpandas             0.3.6              pyhd8ed1ab_0    conda-forge
sqlite                    3.34.0               h74cdb3f_0    conda-forge
supermercado              0.2.0              pyh9f0ad1d_0    conda-forge
tblib                     1.6.0                      py_0    conda-forge
terminado                 0.9.2            py38h578d9bd_0    conda-forge
testpath                  0.4.4                      py_0    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge
thrift                    0.11.0          py38he1b5a44_1001    conda-forge
tifffile                  2021.2.26          pyhd8ed1ab_0    conda-forge
tiledb                    2.2.4                hb9a9e87_1    conda-forge
tk                        8.6.10               h21135ba_1    conda-forge
toolz                     0.11.1                     py_0    conda-forge
tornado                   6.1              py38h497a2fe_1    conda-forge
tqdm                      4.58.0             pyhd8ed1ab_0    conda-forge
traitlets                 5.0.5                      py_0    conda-forge
traittypes                0.2.1              pyh9f0ad1d_2    conda-forge
typing-extensions         3.7.4.3                       0    conda-forge
typing_extensions         3.7.4.3                    py_0    conda-forge
tzcode                    2021a                h7f98852_1    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
unzip                     6.0                  h7f98852_2    conda-forge
urllib3                   1.26.3             pyhd8ed1ab_0    conda-forge
utide                     0.2.5                      py_0    conda-forge
vim                       8.2.2569         py38h526f742_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wgrib2                    2.0.8                h5a10b45_1    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
widgetsnbextension        3.5.1            py38h578d9bd_4    conda-forge
wrapt                     1.12.1           py38h497a2fe_3    conda-forge
xarray                    0.16.2             pyhd8ed1ab_0    conda-forge
xarray-spatial            0.1.2              pyhd3deb0d_0    conda-forge
xarray_leaflet            0.1.13             pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.3                h9d8b166_2    conda-forge
xesmf                     0.5.2              pyhd8ed1ab_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.0                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xrviz                     0.1.4                      py_1    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
yarl                      1.6.3            py38h497a2fe_1    conda-forge
zarr                      2.6.1              pyhd8ed1ab_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_0    conda-forge
zfp                       0.5.5                h9c3ff4c_4    conda-forge
zict                      2.0.0                      py_0    conda-forge
zip                       3.0                  h14c3975_1    conda-forge
zipp                      3.4.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge

[rsignell-usgs]

now gateway.cluster_options() is not even working.

[rsignell-usgs]

Did you guys change anything on the github action side lately?

[tylerpotts]

@rsignell-usgs @jkellndorfer Thanks for making us aware of this problem. @aktech Is actively looking into this right now. I created #382 for tracking

[tylerpotts]

The resolution to this is in #382

[Adam-D-Lewis]

I'm closing old discussions in a repo cleanup effort. Feel free to re-open if needed.