Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remotetest migration & removal of blended products #144

Merged
merged 4 commits into from
May 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,11 @@ Configuration files
- kept in git repo TdsConfig

- each machine has their own subdirectory:
- idd : common to all (4.6) idd servers
- idd : common to all idd servers
- rdavm : at CISL/RDA
- remotetest : where? opendap/netcdf C test server
- startup: this should be the minimal configuration for running a thredds server
- thredds : atm-nwsc (/opt/tds) (idd + casestudies)
- threddsTest : atm-nwsc (/opt/tds-test) (idd + casestudies + preprocessed)
- threddsTest : atm-nwsc (/opt/tds-test) (idd + casestudies + preprocessed + remotetest)
- threddsDev : lead (/opt/tds-dev)
- awsL2 : jetstream machine serving AWS Level2 radar archive data

Expand All @@ -30,7 +29,7 @@ Configuration files
The build and upload of config.zip files is automatically performed by GitHub Actions for
all changes committed to the repository.

- config.zip hosted on [Unidata Nexus](https://artifacts.unidata.ucar.edu/#browse/browse:downloads-tds-config)
- config.zip hosted on the [Unidata artifacts server](https://artifacts.unidata.ucar.edu/#browse/browse:downloads-tds-config)

- to make changes to config files:
- Edit files, e.g. `thredds/*`
Expand Down
4 changes: 3 additions & 1 deletion idd/idd/satellite.xml
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@
<addDatasetSize/>
</datasetScan>
</dataset><!-- End of NESDIS NOAA20 data -->

<!--
<dataset name="Hyrdology Products">
<metadata inherited="true">
<serviceName>nongrid</serviceName>
Expand All @@ -211,7 +213,7 @@
</filter>
<addDatasetSize/>
</datasetScan>
</dataset><!-- End of NESDIS Blended data -->
</dataset> End of NESDIS Blended data -->
</dataset><!-- End of Satellite Data -->
</catalog>

98 changes: 7 additions & 91 deletions remotetest/README.md
Original file line number Diff line number Diff line change
@@ -1,96 +1,12 @@
Setup Instructions For The remotetest.unidata.ucar.edu Tomcat Server
remotetest.unidata.ucar.edu host now combined with thredds-test server
====
** This method of updating the remotetest server is defunct and should not be used! **

When installing a new thredds server on
_remotetest.unidata.ucar.edu_, the process
is as follows. This overrides any other instructions.
The remotetest.unidata.ucar.edu is no longer a standalone host.

For purposes of this process, it is assumed that the content and logs
directories are in _/opt/remotetest_
on the machine _remotetest.unidata.ucar.edu_.
The `dts.war` and `d4ts.war` files are hosted on the same tomcat instance as the thredds-test server.

1. Enter the directory _TdsConfig/remotetest_.
2. Edit as needed the following files:
* _thredds/threddsConfig.xml_
* _thredds/catalog.xml_
3. Run the shell script 'content.sh' to construct the file
'content.tar'
4. Copy _content.tar_ to the _remotetest.unidata.ucar.edu_ machine.
5. Stop the tomcat server.
6. Move to the directory _/opt/remotetest/content_.
7. Untar content.tar into _/opt/remotetest/content_.
8. (optional) clear logs: _rm -f /opt/remotetest/logs/*_
9. Build and install server side keystore and truststore (see below)
10. (optional) modify *$CATALINA_HOME/conf/server.xml* (see below)
11. (optional) modify *$CATALINA_HOME/conf/tomcat-users.xml* (see below)
12. Restart the tomcat server.
* The `defunct` directory contains the old files and documentation for updating this host.
* The `thredds` directory contains the data used by this host.

Optional Changes to $CATALINA_HOME/conf/server.xml
----
On a one time basis, you will need to modify *$CATALINA_HOME/conf/server.xml*
to support SSL and also to support a password protected dataset.

1. Insert the following &lt;Connector&gt; into conf/server.xml
<pre>
&lt;Connector
port="8443"
maxThreads="150"
SSLEnabled="true"
scheme="https"
secure="true"
clientAuth="want"
sslProtocol="TLS"
keyAlias="1"
keystoreFile="conf/ServerKeystore.jks"
keystorePass="password"
keyPass="password"
/&gt;
</pre>
Note the 'clientAuth="want"' line. This tells tomcat to attempt
to use client-side keys but if not successful, ignore and continue.
See below with respect to the 'keystoreFile' line.

Optional Changes to $CATALINA_HOME/conf/tomcat-users.xml
----
Since access to a restricted dataset is used as a test,
the tomcat-users.xml file must be modified to include the following lines.
<pre>
<role rolename="restrictedDatasetUser"/>
<role rolename="tiggeData"/>
<user username="tiggeUser"
password="tigge"
roles="restrictedDatasetUser,tiggeData"/>
</pre>
The roles defined here must be consistent with _content/thredds/catalog.xml_.

Depending on your tomcat installation, the tiggeUser password may
need to be encrypted. You will also need to ensure that 'https:'
access is enabled either in tomcat (see above) or apache httpd.

Key/Trust Store Construction and Installation:
----
If you are running a standalone Tomcat (typically using Intellij),
and in order to support server keys and (optionally) client side keys
the shell script certs.sh must executed. This will construct
JKS format key and trust stores.

1. Run certs.sh to produce
* ServerKeystore.jks
* ClientKeystore.jks
* ClientTruststore.jks
2. Put ServerKeystore.jks and ClientTruststore.jks into ${CATALINA_HOME}/conf

Notes:
-----
1. Verify that all the datasets referenced in catalog.xml
are in place in /opt/remotetest/content/thredds/public.
Note that the "locations" in catalog.xml are not actual paths:
effectively the "content/" part refers to
/opt/remotetest/content/thredds/public

2. If you plan to modify /opt/remotetest/content, make sure
you keep a backup copy.

3. Make sure that the following environment variable in setenv.sh is set:
CONTENT_ROOT="-Dtds.content.root.path=/opt/remotetest/content"

Please reference the `threddstest/catalog.xml` file for `threddstest/remotetest` directory for more information.
Empty file removed remotetest/build.info
Empty file.
9 changes: 9 additions & 0 deletions remotetest/defunct/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
remotetest.unidata.ucar.edu host now combined with thredds-test server
====
** This method of updating the remotetest server is defunct and should not be used! **

The remotetest.unidata.ucar.edu is no longer a standalone host.

The `dts.war` and `d4ts.war` files are hosted on the same tomcat instance as the thredds-test server.

Please reference the `threddstest/catalog.xml` file for `threddstest/remotetest` directory for more information.
96 changes: 96 additions & 0 deletions remotetest/defunct/README.md.old
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
Setup Instructions For The remotetest.unidata.ucar.edu Tomcat Server
====

When installing a new thredds server on
_remotetest.unidata.ucar.edu_, the process
is as follows. This overrides any other instructions.

For purposes of this process, it is assumed that the content and logs
directories are in _/opt/remotetest_
on the machine _remotetest.unidata.ucar.edu_.

1. Enter the directory _TdsConfig/remotetest_.
2. Edit as needed the following files:
* _thredds/threddsConfig.xml_
* _thredds/catalog.xml_
3. Run the shell script 'content.sh' to construct the file
'content.tar'
4. Copy _content.tar_ to the _remotetest.unidata.ucar.edu_ machine.
5. Stop the tomcat server.
6. Move to the directory _/opt/remotetest/content_.
7. Untar content.tar into _/opt/remotetest/content_.
8. (optional) clear logs: _rm -f /opt/remotetest/logs/*_
9. Build and install server side keystore and truststore (see below)
10. (optional) modify *$CATALINA_HOME/conf/server.xml* (see below)
11. (optional) modify *$CATALINA_HOME/conf/tomcat-users.xml* (see below)
12. Restart the tomcat server.

Optional Changes to $CATALINA_HOME/conf/server.xml
----
On a one time basis, you will need to modify *$CATALINA_HOME/conf/server.xml*
to support SSL and also to support a password protected dataset.

1. Insert the following &lt;Connector&gt; into conf/server.xml
<pre>
&lt;Connector
port="8443"
maxThreads="150"
SSLEnabled="true"
scheme="https"
secure="true"
clientAuth="want"
sslProtocol="TLS"
keyAlias="1"
keystoreFile="conf/ServerKeystore.jks"
keystorePass="password"
keyPass="password"
/&gt;
</pre>
Note the 'clientAuth="want"' line. This tells tomcat to attempt
to use client-side keys but if not successful, ignore and continue.
See below with respect to the 'keystoreFile' line.

Optional Changes to $CATALINA_HOME/conf/tomcat-users.xml
----
Since access to a restricted dataset is used as a test,
the tomcat-users.xml file must be modified to include the following lines.
<pre>
<role rolename="restrictedDatasetUser"/>
<role rolename="tiggeData"/>
<user username="tiggeUser"
password="tigge"
roles="restrictedDatasetUser,tiggeData"/>
</pre>
The roles defined here must be consistent with _content/thredds/catalog.xml_.

Depending on your tomcat installation, the tiggeUser password may
need to be encrypted. You will also need to ensure that 'https:'
access is enabled either in tomcat (see above) or apache httpd.

Key/Trust Store Construction and Installation:
----
If you are running a standalone Tomcat (typically using Intellij),
and in order to support server keys and (optionally) client side keys
the shell script certs.sh must executed. This will construct
JKS format key and trust stores.

1. Run certs.sh to produce
* ServerKeystore.jks
* ClientKeystore.jks
* ClientTruststore.jks
2. Put ServerKeystore.jks and ClientTruststore.jks into ${CATALINA_HOME}/conf

Notes:
-----
1. Verify that all the datasets referenced in catalog.xml
are in place in /opt/remotetest/content/thredds/public.
Note that the "locations" in catalog.xml are not actual paths:
effectively the "content/" part refers to
/opt/remotetest/content/thredds/public

2. If you plan to modify /opt/remotetest/content, make sure
you keep a backup copy.

3. Make sure that the following environment variable in setenv.sh is set:
CONTENT_ROOT="-Dtds.content.root.path=/opt/remotetest/content"

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
4 changes: 4 additions & 0 deletions threddsTest/catalog.xml
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,8 @@
<catalogRef xlink:href="casestudies/catalog.xml" xlink:title="Unidata case studies" name=""/>
<catalogRef xlink:href="preprocessed/catalog.xml" xlink:title="Preprocessed data" name=""/>
</dataset>

<dataset name="DAP and DAP4 Tests">
<catalogRef xlink:href="remotetest/catalog.xml" xlink:title="Test Data" name=""/>
</dataset>
</catalog>
43 changes: 43 additions & 0 deletions threddsTest/remotetest/catalog.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:ncml="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" version="1.2">

<service name="all" base="" serviceType="compound">
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
<service name="dap4" serviceType="DAP4" base="/thredds/dap4/" />
<service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
<!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
<!--service name="wms" serviceType="WMS" base="/thredds/wms/" /-->
<!--service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" /-->
</service>

<service name="dap" base="" serviceType="compound">
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
<service name="dap4" serviceType="DAP4" base="/thredds/dap4/" />
</service>

<datasetScan name="Standard Test Data" ID="testDatasetScan" path="testdata" location="content/testdata/">
<metadata inherited="true">
<serviceName>all</serviceName>
<dataType>Grid</dataType>
</metadata>
</datasetScan>

<datasetScan name="OPeNDAP Test Data" ID="testDodsScan" path="testdods" location="content/testdods/">
<metadata inherited="true">
<serviceName>all</serviceName>
<dataType>Grid</dataType>
</metadata>
</datasetScan>

<!--
<datasetScan name="Restricted Test Data" ID="testrestricted" path="restrict" location="content/restrict/" restrictAccess="tiggeData">
<metadata inherited="true">
<serviceName>all</serviceName>
<dataType>Grid</dataType>
</metadata>
</datasetScan>
-->

</catalog>
Loading