Packaging of Zowe on server

Packaging of Zowe on z/OS

Current z/OS packaging

Main Zowe PAX file for z/OS is a regular PAX archive that is unpacked to a "build" directory (for example /u/users/plape03/zowe/builds/1.0.1).

This PAX file is exploded to:

├── files
│   ├── ZLUX.pax
│   ├── api-mediation-package-0.8.4.pax
│   ├── assets
│   │   └── api-catalog.png
│   ├── data-sets-api-server-0.1.1-boot.jar
│   ├── explorer-jes-0.0.15.pax
│   ├── explorer-mvs-0.0.10.pax
│   ├── explorer-uss-0.0.9.pax
│   ├── jobs-api-server-0.1.1-boot.jar
│   ├── scripts
│   │   ├── data-sets-api-server-start.sh
│   │   └── jobs-api-server-start.sh
│   ├── templates
│   │   └── ZOWESVR.jcl
│   └── zss.pax
├── install
│   ├── zowe-check-prereqs.sh
│   ├── zowe-install-apf-server.sh
│   ├── zowe-install-apf-server.yaml
│   ├── zowe-install.sh
│   └── zowe-install.yaml
├── licenses
│   └── zowe_licenses_full.zip
├── manifest.json
└── scripts
    ├── ocopyshr.clist
    ├── ocopyshr.sh
    ├── opercmd
    ├── run-zowe.sh
    ├── zlux-install-script.sh
    ├── zowe-api-mediation-configure.sh
    ├── zowe-api-mediation-install.sh
    ├── zowe-config-stc.sh
    ├── zowe-copy-proc.sh
    ├── zowe-explorer-api-configure.sh
    ├── zowe-explorer-api-install.sh
    ├── zowe-explorer-ui-configure.sh
    ├── zowe-explorer-ui-install.sh
    ├── zowe-init.sh
    ├── zowe-install-existing-plugin.sh
    ├── zowe-install-iframe-plugin.sh
    ├── zowe-parse-yaml.sh
    ├── zowe-prepare-runtime.sh
    ├── zowe-runtime-authorize.sh
    ├── zowe-start.sh
    ├── zowe-stop.sh
    ├── zowe-verify.sh
    ├── zowe-zlux-configure-certificates.sh
    ├── zowe-zlux-configure-ports.sh
    └── zss
        ├── ocopyshr.rexx
        ├── zowe-xmem-apf.sh
        ├── zowe-xmem-check-access.sh
        ├── zowe-xmem-check-if-pdse.sh
        ├── zowe-xmem-check-if-sms.sh
        ├── zowe-xmem-check-profile.sh
        ├── zowe-xmem-check-stc-profile.sh
        ├── zowe-xmem-check-user.sh
        ├── zowe-xmem-dataset-exists.sh
        ├── zowe-xmem-define-stc-profile.sh
        ├── zowe-xmem-define-stc-user.sh
        ├── zowe-xmem-define-xmem-profile.sh
        ├── zowe-xmem-deploy-loadmodule.sh
        ├── zowe-xmem-deploy-parmlib.sh
        ├── zowe-xmem-deploy-proclib.sh
        ├── zowe-xmem-parse-yaml.sh
        ├── zowe-xmem-permit.sh
        └── zowe-xmem-ppt.sh

The install/zowe-install.sh script executes scripts in the scripts directory. These scripts:

• Unpack or copy files in the files directory to the Zowe instance root directory
• Read values of some configuration parameters from install/zowe-install.yaml or the users profile ~/.zowe_profile
• Detect values of some configuration parameters
• Modify files in the Zowe instance directory
• Copy a member to PROCLIB

How does a file get to the Zowe PAX file for z/OS?

1. The file is stored in GitHub of a Zowe project (for example https://github.com/zowe/api-layer/blob/master/scripts/apiml_cm.sh) or is used by the build process of a Zowe project and ends up in an artifact (for example: gateway-service/build/libs/gateway-service.jar)
2. The project has its own process how the artifacts for Zowe are prepared (for example https://github.com/zowe/api-layer/blob/master/zowe-install/build.gradle)
3. Artifacts for Zowe are published to Zowe Artifactory (https://gizaartifactory.jfrog.io/gizaartifactory/libs-release-local/com/ca/mfaas/sdk/mfaas-zowe-install/)
4. The zowe-install-packaging pipeline that can run only of Zowe Jenkins (https://github.com/zowe/zowe-install-packaging/blob/master/Jenkinsfile) gets the artifacts using specification at https://github.com/zowe/zowe-install-packaging/blob/master/artifactory-download-spec.json.template to download all the necessary artifacts
5. These artifacts together with other files are added to the Zowe PAX file for z/OS.

One possible approach to SMP/E installation is to take the files that are installed by Zowe installation and package them as SMP/E elements (e.g. PAX file for each directory that then explored by pax during SMP/E APPLY phase). Another approach is described in the proposal.

Proposal

The main idea is to use SMP/E only for one thing - trusted distribution of "packages" to z/OS. By "package", we mean a package in a format that is platform independent, for example, a PAX or NPM package.

The first goal is to base the packaging on tools that are proven in more complex use cases.

The second goal is to make installation in the development environment, on and off z/OS, on a workstation. Running all components on PC requires a lot of understanding of how each component works and is configured and is the layout of files is different from Zowe installation on z/OS. Replacing a single component on z/OS without reinstalling full Zowe requires a lot of knowledge too and differs for each component. Using standard packaging will help address these complexities and inconsistencies.

• Each component of Zowe is packaged using an existing package manager (for example Yarn - https://yarnpkg.com/ or Npm - https://www.npmjs.com/package/npm) that can run on z/OS and on other platforms (it limits us to Java or Node.js, possibly Bash). We assume that Npm is chosen in examples of the document
• The Zowe on z/OS is a set of Zowe packages and their dependencies
• The installation (laying the files down a file system) is done by the "package manager" which checks dependencies and installs packages from a repository of the packages (npmjs.com, Zowe Artifactory)
  • The command for Yarn to install all dependencies yarn install --offline --production --cache-folder <package-repository>
• Any installation can be technically updated by new package(s) that can come from any source
  • Using npm install package-path-or-specification (see https://docs.npmjs.com/cli/install)
• Zowe packages that do not require z/OS can be running on any platform that has the Java and Node.js prereqs (for example Windows, macOS, Linux, Docker)
• There are two options how this can be used with SMP/E:
  1. The SMP/E distributes PAX archives that packages the contents (files, data sets) installed by the package manager
     • No package manager is required on z/OS
     • The result is same as it would be installed by the package manager
     • It is technically updatable by the package manager (for development purposes) but SMP/E installation should be updated only using PTFs
     • This is option will be used since it does not have any additional dependencies and keeps all the benefits for development use cases
  2. The SMP/E distributes the Zowe packages (archives) as SMP/E elements together with an "installation wrapper" that installs the packages using the package manager
     • The wrapper checks the prereqs (Node.js), bootstraps the package manager, extracts the package specifications package.json and package-lock.json and starts the installation by running npm install
• The z/OS specific packages (e.g. zowe-zss) will have the os set in package JSON to be the one for z/OS
• The configuration scripts are just another package that can be updated via the package manager (they will be in zowe-config package which will be dependent on other packages - e.g. zowe-api-mediation or zowe-api-mediation-config)

Use cases

Non-SMP/E PAX installation

1. The user extracts the PAX in the same way as for 1.0.x version
2. The user executes zowe-install.sh that calls Npm to extract the packages to the runtime directory (no configuration is done), the runtime directory is a parameter of the installation script
3. The user creates zowe-config.yaml in a user directory or has an existing configuration file
4. The user executes zowe-config.sh from <zowe-runtime>/install that will configure new Zowe instance in a different zFS than the runtime

SMP/E PAX installation

1. The user creates new SMP/E environment by following the process or submitting GIMUNZIP JCL in the PAX and then running SMP/E JCLs that are in one dataset extracted from the GIMZIP files, or by using z/OSMF portable package, or CA MSM
2. The target-zone zFS file contains the same files as the PAX file from method #1
   • SMP/E will also call zowe-install.sh so the target zFS will contain not just the source package but also the runtime
3. The rest are the same configuration steps

Online installation

Steps 1 and 2 can be replaced by npm install zowe-zos. The packages will be downloaded from npmjs.com.

Update

The update is started by updating the packages via the package manager. SMP/E can be used to update the package repository and eventually running the package manager. Non-SMP/E PAX file can be used to update an existing package repository.

The update process will need call configuration in some cases. The packages should have additional action config that needs to support configuration in cases that the instance is already configured from the previous installation and update the configuration and migrate the data.

Development version installation/update

Any instance of Zowe can be updated by new packages from any source including package built by a developer on the workstation or a branch build. Only a single package can be used without.

Installation on other platforms

The package manager can be used to install components that do not require z/OS on any platform that supports the package manager and the required platform (Java, Node.js). The installation is the same, distribute the packages somehow and then use the package manager.

Configuration

• Additional actions (config and zos-config for configuration actions) will be added as scripts in bin directory of each package
• They will be called by zowe-config.json that will obtain the list of packages from the package manager and call these actions if present
• The input to these actions will be the instance directory that contains the configuration parameters
• The config action updates metadata in /config directory to contain correct version information for the configuration (for purposes of migrations)
• The configuration and configuration updates are running as a transaction (if one action fails then the original state is preserved)

Benefits of using an existing package manager

• Does the work of extracting well
• Defines common format
• Supported by existing tools (Artifactory, Npmjs.com)
• Support dependencies
  • We can easily replace a big package by smaller packages (e.g. API ML by a package for each service)
• Provide standardized information about what is exactly installed
• Checks consistency (hashes)
• Support upgrades
• Decouples the source repository from the installation process
• Can be used on other platforms than z/OS for non z/OS only components
  • For development and testing outside of z/OS
  • For running in production on different platforms - Linux (on System z), Docker (on z/OS)
• Forces to do packaging in a structured way
• Additional actions (e.g. install-zos for z/OS specific actions during installation) can be defined in the package on predefined place

Disadvantages

• Not easy to do quick hacks (e.g. update files installed by another component)

Zowe Server Directory Structure

The modular approach requires more rigid directory structure.

The runtime (read-only binaries, scripts, and resources) is:

• bin - scripts of the server to configure, start, and other actions. Scripts that the instance directory as the argument
• modules (symlink to node_modules) - installed components
  • internal structure of modules is not prescribed
  • the required scripts are bin/start.sh and bin/config.sh that are called by zowe-server configuration and startup scripts
  • it is recommended to have bin for scripts and JARs, and config for the application.yml file that ties the Zowe environment variables to the Spring Configuration in case of Java Spring Boot applications
• resources - non-executable and read-only data (e.g. templates)

The instance:

• config - the configuration parameters
• a directory for each component that requires some data
  • components are expected to manage a zFS file-system for large data
• additional directories for shared data (e.g. keystore)

Predefined Actions

The package provides actions that can be invoked by the Zowe installation, update, configuration, and start-up scripts.

The actions are created as z/OS compatible Bash scripts in bin/ directory:

• bin/config.sh - executed during configuration process. This script needs to configure the component in Zowe new instance. All dependencies are already configured before this script is executed.
• bin/start.sh - executed during the Zowe server start up. All dependencies are are already starting.

The shell scripts can call other programs but they have to make sure that the environment is initialized for them. We can expect that JAVA_HOME is configured provided by the user in the configuration file and expect that Java will be there. Node.js can be used by components that require Node.js.

Prototype

The zowe/sample-spring-boot-api-service GitHub repository contains a prototype with few components:

1. zowe-server - depends on all server components
2. zowe-fake-cm - fake certificate management
3. zowe-sample-spring-boot-api-service - a service (depends on zowe-fake-cm)

Commands to install, configure, and start the prototype:

git clone https://github.com/zowe/sample-spring-boot-api-service.git
cd sample-spring-boot-api-service/
git checkout compoments-prototype
cd zowe-server
npm install  
bin/zowe-config.sh instance
bin/zowe-start.sh instance

This installs the prototype server with fake CM and a sample service. Then it is configured and started.

Feedback

• The scripts can be grouped to one (zowe-server.sh) with subcommmands (init, config, start)
• The parameter is not the instance directory but a configuration parameter file or member (files need to be supported for other platforms than z/OS)
• The actions provided by components will be in z/OS shell (Bash compatible) and they can call Java for more complex logic
• Start command can use pre-configured script to avoid complex resolution
• Configuration parameters are distinct from user data and preferences. They are for system programmer to make Zowe Server running not to configure features
• init command creates configuration file with default values and necessary interactive questions

Examples (not implemented)

Developer wants to run Zowe Server on workstation:

npm install -g @zowe/zowe-server
zowe-server init  
zowe-server config configuration_file.env
zowe-server start configuration_file.env

Developer wants to install a subset of Zowe components:

git clone https://github.com/zowe/zowe-server.git
cd zowe-server
code package.json  # edit and update the dependencies
npm install  
bin/zowe-server.sh init
bin/zowe-server.sh config configuration_file.env
bin/zowe-server.sh start configuration_file.env