Skip to content

CSD Developer Tricks and Tools

Jump to bottom
herman edited this page Mar 27, 2015 · 10 revisions

Below are a few tools and tricks to help with developing a new Custom Service Descriptor (CSD).

Validating the CSD

Cloudera Manager uses a validator to check if the CSD is valid before loading it into its repository. In CM, validation issues are exposed in two places:

  1. The server log - /var/log/cloudera-scm-server/cloudera-scm-server.log
  2. In the list REST API. For example:
{         
 "invalidCsds": {
   "/opt/cloudera/csd/ECHO-1.0.jar": [
     "service.label must be present and not blank",
     "service.description must be present and not blank"
   ]
}

During development, it is also convenient to run the same validator from the command line. The validator can be invoked by the following steps:

  1. cd validator
  2. mvn clean install
  3. java -jar target/validator.jar -s <path_to_sdl>
$ java -jar target/validator.jar -s ECHO/src/descriptor/service.sdl 
Validating: ECHO/src/descriptor/service.sdl
==> service.label must be present and not blank
==> service.description must be present and not blank

Using a CSD directory

During development, it is convenient to avoid rebuilding a jar file every time you make a change to the service.sdl file or scripts. For this reason, along with reading CSDs from jar files, Cloudera Manager can also read CSDs directly from directories. The directory needs to be named in the same manner as the jar file. Here is an example with the ECHO-1.0 CSD we created in the primer.

$ tree /opt/cloudera/csd/
/opt/cloudera/csd/
├── ECHO-1.0
│   ├── descriptor
│   │   └── service.sdl
│   └── scripts
│       └── control.sh
└── ECHO-1.0.jar

When Cloudera Manager loads the CSDs from disk, directories take precedence over jar CSDs. In the above case, the ECHO-1.0 directory will win. This can also be seen by using the list REST API. The response will indicate that a CSD is "shadowed" and that the available CSD is the directory:

{
    "message": "OK",
    "data": {
        "externalRootPath": "/opt/cloudera/csd",
        "internalRootPath": "/usr/share/cmf/csd",
        "availableCsds": [
            {
                "csdName": "ECHO-1.0",
                "serviceType": "ECHO",
                "source": "/opt/cloudera/csd/ECHO-1.0",
                "isInstalled": true,
                "hasPlaceholder": false
            }
        ],
        "shadowedCsds": [
            {
                "csdName": "ECHO-1.0.jar",
                "serviceType": "ECHO",
                "source": "/opt/cloudera/csd/ECHO-1.0.jar",
                "isInstalled": true,
                "hasPlaceholder": false
            }
        ],
        "invalidCsds": {},
        "repoEnabled": true
    }
}
  • Changed in Cloudera Manager 5.4.0, added 'serviceType' and changed 'csdName' to be file/jar name

Partial Installation (Development Mode Only)

WARNING WARNING, DANGER DANGER: These APIs only exist to aid development of CSDs and could disappear with the next release. There is no guarantee that there will be backwards support for them. Use at own risk.

During development, it is helpful to have a dynamic way of reinstalling your CSD so that you can see your changes without having to restart Cloudera Manager. In Cloudera Manager 5 there are a few management APIs to help developers while they write their CSD.

List

Reveals the internal state of the CSD repository since the last refresh (or restart).

Request:

GET /cmf/csd/list

Sample Response:

{
    "message": "OK",
    "data": {
        "externalRootPath": "/opt/cloudera/csd",
        "internalRootPath": "/usr/share/cmf/csd",
        "availableCsds": [
            {
                "csdName": "SPARK-1.0-20140220.014146-19.jar",
                "serviceType": "SPARK",
                "source": "/usr/share/cmf/csd/SPARK-1.0-20140220.014146-19.jar",
                "isInstalled": true,
                "hasPlaceholder": false
            },
            {
                "csdName": "ECHO-1.0.jar",
                "serviceType": "ECHO",
                "source": "/opt/cloudera/csd/ECHO-1.0.jar",
                "isInstalled": true,
                "hasPlaceholder": false
            }
        ],
        "shadowedCsds": [],
        "invalidCsds": {},
        "repoEnabled": true
    }
}
Key Description
externalRootPath The CSD repository location for third party CSDs.
internalRootPath Cloudera Manager has an internal repository for CSDs that come bundled by default. This is the path to that repository. This repository is internal and should not modified.
availableCsds The list of valid and loaded CSDs. Additional information is specified for each CSD such as: the name, where it is located, if it is installed, or has a placeholder.
shadowedCsds If any CSDs are being shadowed by others. This can happen if a directory is being used - see Using a directory. Or if an external CSD conflicts with an internal CSD.
invalidCsds Lists all the invalid CSDs and their validation errors.
repoEnabled True if the repo is enabled.

##Refresh

Refreshes the CSD repository and lists all the CSDs found with any validation errors. Refreshing does not install any CSDs.

Request:

GET /cmf/csd/refresh

Response

Same as List.

##Install

Once the CSD is listed in the repository without any errors, it can be installed. Installing a CSD registers the CSD service type with Cloudera Manager and makes it available in various wizards.

Request:

GET /cmf/csd/install?csdName=<name>

Response:

{
    "message": "OK",
    "data": null
}

##Uninstall

Uninstalling a CSD will remove the service type from the "Add Services" wizard.

Request:

GET /cmf/csd/uninstall?csdName=<name>

Response:

{
    "message": "OK",
    "data": null
}

If a service of this service type exists on the cluster, this command will fail with the following error:

{
    "message": "CSD [<name>] cannot be uninstalled since services [<service_name1>,<service_name2>] are still using the CSD",
    "data": null
}

To uninstall the CSD, the services will first need to be removed from the cluster. Only then can the CSD be uninstalled.

There is an option to force the uninstallation of the CSD. You can pass force=true and the CSD will be uninstalled and a placeholder service handler will be added for any services of that type that exist on the cluster. This operation is only recommended if the CSD will be added again.

##Reinstall

Reinstalls all the installed CSDs - available CSDs that are not installed won't get touched.

Request:

GET /cmf/csd/reinstall

Response:

{
    "message": "OK",
    "data": null
}

You may also reinstall a specific CSD:

Request:

GET /cmf/csd/reinstall?csdName=<name>

Response:

{
    "message": "OK",
    "data": null
}

During normal development when you make a change to a CSD, you will need to do the following things:

  1. Refresh the repo: GET <hostname:port>/cmf/csd/refresh.
  2. Check that your CSD doesn't have any validation errors.
  3. Uninstall your CSD, probably with force=true: GET <hostname:port>/cmf/csd/uninstall?csdName=<csd_name>&force=true.
  4. Install your CSD GET <hostname:port>/cmf/csd/install?csdName=<csd_name>.

The reinstall command will do all of these operations on the installed CSDs. If there is a validation error on any of the installed CSDs after refresh, the operation will not proceed. This allows the developer to fix any validation error that surface before proceeding again.

Limitations

Partial Installation of CSDs is not recommended for customers as there can be some unpredictable behavior. Known issues/limitations include:

  1. Monitoring does not work on a partially installed CSD
  2. Graphs might show errors because of unknown entity types.
Clone this wiki locally