Skip to content

Test Zowe in Sysplex

Jump to bottom
Jack (T.) Jia edited this page Nov 19, 2021 · 14 revisions

This documentation is manual test plan for deploying Zowe in Sysplex High Availability mode. To deploy Zowe in Sysplex, check documentation here.

A - Test Configuration and Starting Zowe

Test Case A1 - zowe.yaml creation

Steps

  • After normal installation, run command <instance-dir>/bin/utils/convert-to-zowe-yaml.sh <instance-dir>/zowe.yaml.

Exit Criteria

  • <instance-dir>/zowe.yaml should be untagged and should be able to read from vi or cat utilities.
  • <instance-dir>/zowe.yaml should contains all customized values defined in original instance.env

Test Case A2 - zowe.yaml adoption

Steps

  • Generate <instance-dir>/zowe.yaml,
  • Delete or rename <instance-dir>/instance.env,
  • Start Zowe with ZWESVSTC on tivlp13 by running <instance-dir>/bin/zowe-start.sh.

Exit Criteria

  • Zowe should be started normally with job name ZWE1SV,
  • All values defined in <instance-dir>/zowe.yaml should be converted back to <instance-dir>/.env/.instance-tivlp13.env.
  • Zowe job log should show correct instance environment variables.

Test Case A3 - customize zowe.yaml adoption with ZWESVSTC

Steps

  • Delete or rename <instance-dir>/instance.env,
  • Put attachment zowe.yaml version 1 as <instance-dir>/zowe.yaml,
  • Start Zowe with ZWESVSTC on tivlp13 by running <instance-dir>/bin/zowe-start.sh.

Exit Criteria

  • Zowe should be started normally with job name ZWE1SV,
  • All customized values defined in <instance-dir>/zowe.yaml should be converted back to <instance-dir>/.env/.instance-tivlp13.env.
    • all Zowe services should be started normally on default ports.
  • Zowe job log should show correct instance environment variables.

Test Case A4 - customize zowe.yaml adoption with ZWESLSTC

Steps

  • Delete or rename <instance-dir>/instance.env,
  • Put attachment zowe.yaml version 1 as <instance-dir>/zowe.yaml,
  • Start Zowe on tivlp13 with S ZWESLSTC,HAINST=TIVLP13,JOBNAME=ZWESL13.

Exit Criteria

  • Zowe should be started normally with job name ZWESL13,
  • All customized values defined in <instance-dir>/zowe.yaml should be converted back to <instance-dir>/.env/.instance-tivlp13.env.
    • all Zowe services should be started normally on default ports.
  • Zowe job log should show correct instance environment variables.

Test Case A5 - zowe.yaml allows customization per instance

Steps

  • Delete or rename <instance-dir>/instance.env,
  • Put attachment zowe.yaml version 1 as <instance-dir>/zowe.yaml,
  • Start Zowe on tivlp14 with S ZWESLSTC,HAINST=TIVLP14A,JOBNAME=ZWESL14A.

Exit Criteria

  • Zowe should be started normally with job name ZWESL14A on tivlp14,
  • All customized values defined in <instance-dir>/zowe.yaml should be converted back to <instance-dir>/.env/.instance-tivlp14a.env.
    • CATALOG_PORT should have value of 17552.
    • ZWE_LAUNCH_COMPONENTS shouldn't have app-server and zss in it.
  • Zowe job log should show information of starting app-server and zss.

Test Case A6 - zowe.yaml allows deeper granular level of configuration

Steps

  • Delete or rename <instance-dir>/instance.env,
  • Put attachment zowe.yaml version 1 as <instance-dir>/zowe.yaml,
  • Modify <instance-dir>/zowe.yaml and add haInstances.tivlp13.components.gateway.debug entry with value true,
  • Start Zowe on tivlp13 with S ZWESLSTC,HAINST=TIVLP13,JOBNAME=ZWESL13,
  • Start Zowe on tivlp14 with S ZWESLSTC,HAINST=TIVLP14A,JOBNAME=ZWESL14A.

Exit Criteria

  • zowe.yaml should look like: 
    haInstances:
  # HA instance ID
  tivlp13:
    # hostname where this instance will be started
    hostname: tivlp13.svl.ibm.com
    ip: 9.30.238.14
    # for this HA instance, we didn't customize "components", so it will use default value.
    components:
      gateway:
        debug: true
  • Zowe should be started normally with job name ZWESL13 and ZWESL14A,
  • All customized values defined in <instance-dir>/zowe.yaml should be converted back to <instance-dir>/.env/.instance-tivlp13.env and <instance-dir>/.env/.instance-tivlp14a.env.
    • all Zowe services should be started normally on default ports.
  • Only <instance-dir>/.env/gateway/.instance-tivlp13.env should have entry APIML_DEBUG_MODE_ENABLED="true", but all other services from both Zowe instances should have APIML_DEBUG_MODE_ENABLED="false".
  • Zowe job log should show correct instance environment variables and regular level of logging. Only gateway service running on TIVLP13 shows debug level logging.

B - Test Zowe Launcher Enhancement

Test Case B1 - Component Restart

Steps

  • Follow all steps defined for test case A4,
  • On tivlp13, use SDSF.DA panel, find ZWE1EF address space can cancel it. This is Explorer Files API process.

Exit Criteria

  • We should see a new ZWE1EF address space started in few seconds.
  • We should see logs like this in ZWESLSTC job log: 
    <ZWELNCH:67174462> ZWESVUSR INFO component files-api(65662) terminated, status = 0        
<ZWELNCH:67174462> ZWESVUSR INFO next attempt to restart component files-api in 1 seconds 
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0001I component files-api started                    
<ZWELS:16842878>  INFO (start-component.sh:95) starting component files-api ...

Test Case B2 - Regular Shutdown

Steps

  • Follow all steps defined for test case A4,
  • On tivlp13, stop Zowe with P ZWESLSTC.

Exit Criteria

  • We should see a new ZWE1EF address space started in few seconds.
  • We should see logs like this in ZWESLSTC job log: 
    <ZWELNCH:67174462> ZWESVUSR INFO termination command received
<ZWELNCH:67174462> ZWESVUSR INFO console listener stopped    
<ZWELNCH:67174462> ZWESVUSR INFO console thread stopped      
<ZWELNCH:67174462> ZWESVUSR INFO stopping components         
<ZWELNCH:67174462> ZWESVUSR INFO component gateway(83951810) terminated, status = 36608
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component gateway stopped                   
<ZWELNCH:67174462> ZWESVUSR INFO component discovery(33620024) terminated, status = 36608            
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component discovery stopped                               
<ZWELNCH:67174462> ZWESVUSR INFO component api-catalog(16842934) terminated, status = 36608          
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component api-catalog stopped                             
<ZWELNCH:67174462> ZWESVUSR INFO component caching-service(16842935) terminated, status = 36608      
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component caching-service stopped                         
<ZWELNCH:67174462> ZWESVUSR INFO component app-server(50397239) terminated, status = 36608           
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component app-server stopped                              
<ZWELNCH:67174462> ZWESVUSR INFO component zss(67174468) terminated, status = 36608                  
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component zss stopped                                     
<ZWELNCH:67174462> ZWESVUSR INFO component jobs-api(50397373) terminated, status = 36608             
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component jobs-api stopped                                
<ZWELNCH:67174462> ZWESVUSR INFO component files-api(16842878) terminated, status = 36608            
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component files-api stopped                               
<ZWELNCH:67174462> ZWESVUSR INFO component explorer-jes(16842864) terminated, status = 36608         
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component explorer-jes stopped                            
<ZWELNCH:67174462> ZWESVUSR INFO component explorer-mvs(65698) terminated, status = 36608            
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component explorer-mvs stopped                            
<ZWELNCH:67174462> ZWESVUSR INFO component explorer-uss(65708) terminated, status = 36608            
<ZWELNCH:67174462> ZWESVUSR INFO ZWEL0002I component explorer-uss stopped                            
<ZWELNCH:67174462> ZWESVUSR INFO components stopped                                                  
<ZWELNCH:67174462> ZWESVUSR INFO Zowe Launcher stopped

C - Test Component Instance Registration

Test Case C1 - expected quantity of services showing on registration

Steps

  • Delete or rename <instance-dir>/instance.env,
  • Put attachment zowe.yaml version 1 as <instance-dir>/zowe.yaml,
  • Start Zowe on tivlp13 with S ZWESLSTC,HAINST=TIVLP13,JOBNAME=ZWESL13,
  • Start Zowe on tivlp14 with S ZWESLSTC,HAINST=TIVLP14A,JOBNAME=ZWESL14A,
  • Start Zowe on tivlp14 with S ZWESLSTC,HAINST=TIVLP14B,JOBNAME=ZWESL14B.

Exit Criteria

  • Zowe should function as normal.
  • Visit and login to https://tivlp13.svl.ibm.com:7553 or https://tivlp14.svl.ibm.com:17553.
    • There should no warnings showing on the page.
    • Instances currently registered with Eureka section should show as below:
      • APICATALOG: 2 instances on tivlp14.svl.ibm.com:apicatalog:17552, tivlp13.svl.ibm.com:apicatalog:7552
      • CACHINGSERVICE: 2 instances on tivlp14.svl.ibm.com:cachingservice:17555, tivlp13.svl.ibm.com:cachingservice:7555
      • DATASETS: 2 instances on STATIC-tivlp13.svl.ibm.com:datasets:8547, STATIC-tivlp14.svl.ibm.com:datasets:18547
      • DISCOVERY: 2 instances on tivlp14.svl.ibm.com:discovery:17553, tivlp13.svl.ibm.com:discovery:7553
      • EXPLORER-JES: 2 instances on STATIC-tivlp13.svl.ibm.com:explorer-jes:8546, STATIC-tivlp14.svl.ibm.com:explorer-jes:18546
      • EXPLORER-MVS: 2 instances on STATIC-tivlp13.svl.ibm.com:explorer-mvs:8548, STATIC-tivlp14.svl.ibm.com:explorer-mvs:18548
      • EXPLORER-USS: 2 instances on STATIC-tivlp13.svl.ibm.com:explorer-uss:8550, STATIC-tivlp14.svl.ibm.com:explorer-uss:18550
      • GATEWAY: 2 instances on tivlp14.svl.ibm.com:gateway:7554, tivlp13.svl.ibm.com:gateway:7554
      • IBMZOSMF: 1 instance on STATIC-zhaplex.svl.ibm.com:ibmzosmf:443
      • JOBS: 3 instances on STATIC-tivlp14.svl.ibm.com:jobs:18545, STATIC-tivlp14.svl.ibm.com:jobs:8545, STATIC-tivlp13.svl.ibm.com:jobs:8545
      • UNIXFILES: 2 instances on STATIC-tivlp14.svl.ibm.com:unixfiles:18547, STATIC-tivlp13.svl.ibm.com:unixfiles:8547
      • ZLUX: 2 instances on tivlp14.svl.ibm.com:zlux:28544, tivlp13.svl.ibm.com:zlux:8544
      • ZOSMF: 1 instance on STATIC-zhaplex.svl.ibm.com:zosmf:443
      • ZSS: 2 instances on STATIC-tivlp13.svl.ibm.com:zss:8542, STATIC-tivlp14.svl.ibm.com:zss:28542
    • DS Replicas should show both tivlp13.svl.ibm.com and tivlp14.svl.ibm.com.
    • registered-replicas and available-replicas should both show https://tivlp13.svl.ibm.com:7553/eureka/ and https://tivlp14.svl.ibm.com:17553/eureka/.
  • Visit and login to https://tivlp13.svl.ibm.com:7554/gateway/services or https://tivlp14.svl.ibm.com:7554/gateway/services.
    • We should see same amount of instances like above for each of the services.
    • For each instance, hostname and ipAddr should be correct.

D - Test Component Availability

Test Case D1 - Component Availability

Steps

  • Follow all steps defined for test case C1,
  • On tivlp13, stop Zowe with P ZWESLSTC.

Exit Criteria

Test Case D2 - Component Availability Recovery

Steps

  • Follow all steps defined for test case D1,
  • On tivlp13, start Zowe again with S ZWESLSTC,HAINST=TIVLP13,JOBNAME=ZWESL13.

Exit Criteria

Test Case D3 - LPAR Availability

Steps

  • Follow all steps defined for test case C1,
  • Shutdown tivlp13 LPAR.

Exit Criteria

Test Case D4 - LPAR Recovery

Steps

  • Follow all steps defined for test case D3,
  • Recover LPAR tivlp13 and start Zowe again with S ZWESLSTC,HAINST=TIVLP13,JOBNAME=ZWESL13.

Exit Criteria

Attachments

zowe.yaml version 1

zowe:
  runtimeDirectory: /global/zowe/runtime122
  jobPrefix: ZWE
  identifier: "1"
  externalDomains:
    - zhaplex.svl.ibm.com
    # - 9.30.238.45
  environments:
    ZOWE_IP_ADDRESS: 9.30.238.45
    KEYSTORE_DIRECTORY: /global/zowe/keystore122
    ZOWE_ZLUX_SSH_PORT: "22"
    ZOWE_ZLUX_TELNET_PORT: "992"
    ZOWE_ZLUX_SECURITY_TYPE: "tls"
    EXTERNAL_ROOT_CA: ""
    SSO_FALLBACK_TO_NATIVE_AUTH: "true"
    RESOLVER_CONFIG: "//'USER.PARMLIB(TCPDATA)'"
  externalPort: "7554"
  launchScript:
    logLevel: ""
  externalCertificate:
    keystore:
      alias: localhost
      password: password
      file: /global/zowe/keystore122/localhost/localhost.keystore.p12
      type: PKCS12
    trustStore:
      file: /global/zowe/keystore122/localhost/localhost.truststore.p12
      certificateAuthorities: ""
    pem:
      key: /global/zowe/keystore122/localhost/localhost.keystore.key
      certificate: /global/zowe/keystore122/localhost/localhost.keystore.cer-ebcdic
      certificateAuthority: /global/zowe/keystore122/local_ca/localca.cer-ebcdic
  internalCertificate:
    keystore:
      alias: localhost
      password: password
      file: /global/zowe/keystore122/localhost/localhost.keystore.p12
      type: PKCS12
    trustStore:
      file: /global/zowe/keystore122/localhost/localhost.truststore.p12
      certificateAuthorities: ""
    pem:
      key: /global/zowe/keystore122/localhost/localhost.keystore.key
      certificate: /global/zowe/keystore122/localhost/localhost.keystore.cer-ebcdic
      certificateAuthority: /global/zowe/keystore122/local_ca/localca.cer-ebcdic
  sso:
    token:
      name: ""
      label: ""
components:
  gateway:
    enabled: true
    port: "7554"
    apiml:
      service:
        allowEncodedSlashes: true
        corsEnabled: false
        preferIpAddress: false
      gateway:
        timeoutMillis: "600000"
      security:
        x509:
          enabled: false
          externalMapperUrl: https://zhaplex.svl.ibm.com:7554/zss/api/v1/certificate/x509/map
        zosmf:
          applid: IZUDFLT
        auth:
          provider: zosmf
        authorization:
          endpoint:
            url: https://zhaplex.svl.ibm.com:7554/zss/api/v1/saf-auth
        ssl:
          verifySslCertificatesOfServices: false
          nonStrictVerifySslCertificatesOfServices: true
    debug: false
    server:
      maxConnectionsPerRoute: "100"
      maxTotalConnections: "1000"
  discovery:
    enabled: true
    port: "7553"
    apiml:
      service:
        preferIpAddress: false
      security:
        ssl:
          verifySslCertificatesOfServices: false
          nonStrictVerifySslCertificatesOfServices: true
    debug: false
    alternativeStaticApiDefinitionsDirectories: ""
  api-catalog:
    enabled: true
    port: "7552"
    environment:
      preferIpAddress: false
    debug: false
  caching-service:
    enabled: true
    environment:
      preferIpAddress: false
    debug: false
    port: "7555"
    storage:
      size: "10000"
      evictionStrategy: reject
      mode: VSAM
      vsam:
        name: "IBMUSER.ZWE122.CACHE1"
    apiml:
      security:
        ssl:
          verifySslCertificatesOfServices: false
          nonStrictVerifySslCertificatesOfServices: true
  app-server:
    enabled: true
    port: "8544"
  zss:
    enabled: true
    port: "8542"
    crossMemoryServerName: ZWESIS_STD
    tls: true
  jobs-api:
    enabled: true
    port: "8545"
  files-api:
    enabled: true
    port: "8547"
  explorer-jes:
    enabled: true
    port: "8546"
    frameAncestors: zhaplex.svl.ibm.com:*,9.30.238.45:*
  explorer-mvs:
    enabled: true
    port: "8548"
    frameAncestors: zhaplex.svl.ibm.com:*,9.30.238.45:*
  explorer-uss:
    enabled: true
    port: "8550"
    frameAncestors: zhaplex.svl.ibm.com:*,9.30.238.45:*
java:
  home: /usr/lpp/java/J8.0_64
node:
  home: /global/nodejs/v12/node-v12.18.0-os390-s390x
zOSMF:
  port: "443"
  host: zhaplex.svl.ibm.com
  applId: IZUDFLT

# we can customize configurations for each HA instance
haInstances:
  # HA instance ID
  tivlp13:
    # hostname where this instance will be started
    hostname: tivlp13.svl.ibm.com
    ip: 9.30.238.14
    # for this HA instance, we didn't customize "components", so it will use default value.
  
  # HA instance ID, we will start 2 instances on tivlp14
  # Please note, we can only start one gateway in same LPAR
  tivlp14a:
    # hostname where this instance will be started
    hostname: tivlp14.svl.ibm.com
    ip: 9.30.238.15

    # these configurations will overwrite highest level "components" definitions
    components:
      discovery:
        # use customized port on this instance
        port: "17553"
      api-catalog:
        port: "17552"
      app-server:
        # no app-server in this instance
        enabled: false
      zss:
        # no app-server in this instance
        enabled: false
      jobs-api:
        port: "18545"
      files-api:
        port: "18547"
      explorer-jes:
        port: "18546"
      explorer-mvs:
        port: "18548"
      explorer-uss:
        port: "18550"
      caching-service:
        port: "17555"
  tivlp14b:
    hostname: tivlp14.svl.ibm.com
    ip: 9.30.238.15

    # these configurations will overwrite highest level "components" definitions
    components:
      gateway:
        enabled: false
      discovery:
        enabled: false
      api-catalog:
        enabled: false
      app-server:
        enabled: true
        port: "28544"
      zss:
        enabled: true
        port: "28542"
      jobs-api:
        enabled: true
      files-api:
        enabled: false
      explorer-jes:
        enabled: false
      explorer-mvs:
        enabled: false
      explorer-uss:
        enabled: false
      caching-service:
        enabled: false

System Squad

Zowe Builds

Bill of Materials

SMPE

PSWI

Major Features

v2

High available and Sysplex

Containerization

CI/CD and Automation

Clone this wiki locally