Add BYO catalog connector example for MLX

README.md

@@ -36,6 +36,12 @@ Tutorials to get started with generic pipelines in Elyra:
 - [Run generic pipelines on Apache Airflow](pipelines/run-generic-pipelines-on-apache-airflow)
 - [Run runtime-specific pipelines on Kubeflow Pipelines](pipelines/run-pipelines-on-kubeflow-pipelines)
 
+### Pipeline component catalog connectors
+
+Elyra loads [custom components](https://elyra.readthedocs.io/en/stable/user_guide/pipeline-components.html) from component catalogs:
+- [Find catalog connectors](component-catalog-connectors/connector-directory.md)
+- [Build a custom catalog connector](component-catalog-connectors/build-a-custom-connector.md)
+
 ### Custom pipeline component examples
 
 Pipeline nodes are implemented using

component-catalog-connectors/README.md

@@ -0,0 +1,3 @@
+## Component catalog connectors
+
+This content is under development.

component-catalog-connectors/build-a-custom-connector.md

@@ -0,0 +1,3 @@
+## How to build a component catalog connector
+
+This content is under development.

component-catalog-connectors/connector-directory.md

@@ -0,0 +1,11 @@
+## Component catalog connectors
+
+The following third-party catalog connectors should work with Elyra. Connectors are provided as is and, unless specified otherwise, are not maintained by the Elyra core committers. 
+
+- To add your connector to the list [create a pull request](https://github.com/elyra-ai/examples/pulls). 
+- Learn [how to build your own catalog connector](build-a-custom-connector.md).
+
+| Connector      | Description |
+| ----------- | ----------- |
+| [Machine Learning Exchange](mlx-connector/)      |    Provides access to [Machine Learning Exchange](https://github.com/machine-learning-exchange) catalogs   |
+

component-catalog-connectors/mlx-connector/MANIFEST.in

@@ -0,0 +1,37 @@
+#
+# Copyright 2018-2021 Elyra Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# exclude from ANY directory
+global-exclude *.ipynb
+global-exclude *.py[cod]
+global-exclude __pycache__
+global-exclude .git
+global-exclude .ipynb_checkpoints
+global-exclude .DS_Store
+global-exclude *.sh
+global-exclude docs
+global-exclude tests
+
+# explicit includes
+include CONTRIBUTING.md
+include README.md
+include LICENSE
+include dist/*.tgz
+
+recursive-exclude * __pycache__
+recursive-exclude * *.py[co]
+
+recursive-include mlx_catalog_connector mlx-catalog.json

component-catalog-connectors/mlx-connector/Makefile

@@ -0,0 +1,43 @@
+#
+# Copyright 2021-2021 Elyra Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+.PHONY: help clean lint test-dependencies source-install install dist
+
+SHELL:=/bin/bash
+
+help:
+# http://marmelab.com/blog/2016/02/29/auto-documented-makefile.html
+	@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
+
+clean:
+	rm -rf dist/
+	rm -rf build/
+	rm -rf *.egg-info
+
+test-dependencies:
+	@pip install -q -r test_requirements.txt
+
+lint: test-dependencies
+	flake8 mlx_catalog_connector
+
+dist: clean lint  ## Build distribution
+	python setup.py bdist_wheel sdist	
+
+source-install: dist  ## Install MLX component connector package from source 
+	pip install .
+
+install:  ## Install MLX component connector package from PyPI
+	pip install mlx-component-catalog-connector

component-catalog-connectors/mlx-connector/README.md

@@ -0,0 +1,63 @@
+## Machine Learning Exchange catalog connector
+
+This catalog connector enables Elyra to load Kubeflow Pipelines components from [Machine Learning Exchange](https://github.com/machine-learning-exchange) (MLX) deployments.
+
+### Install the connector
+
+You can install the MLX catalog connector from PyPI or source code. Note that a **rebuild of JupyterLab is not required**.
+
+**Prerequisites**
+
+- [Install Elyra](https://elyra.readthedocs.io/en/stable/getting_started/installation.html) (version 3.3 and above).
+- [Machine Learning Exchange deployment](https://github.com/machine-learning-exchange/mlx) ([quickstart guide](https://github.com/machine-learning-exchange/mlx/tree/main/quickstart))
+
+**Install from PyPI**
+
+  ```
+  $ pip install mlx-component-catalog-connector
+  ```
+
+**Install from source code**
+
+   ```
+   $ git clone https://github.com/elyra-ai/examples.git
+   $ cd examples/component-catalog-connectors/mlx-connector/
+   $ make source-install
+   ```
+
+### Use the connector
+
+1. Launch JupyterLab.
+1. [Open the `Manage Components` panel](
+https://elyra.readthedocs.io/en/stable/user_guide/pipeline-components.html#managing-custom-components-using-the-jupyterlab-ui).
+1. Add a new MLX component catalog ('`+`' > '`New Machine Learning Exchange Component Catalog`').
+1. Specify a catalog name, e.g. '`MLX dev catalog`'.
+1. Select Kubeflow Pipelines as runtime.
+1. (Optional) Specify a category under which the catalog's component will be organized in the palette.
+1. Configure the `MLX API URL`, e.g. '`http://my-mlx-server.mydomain:8080/`'. 
+   > Note: The Machine Learning Exchange API URL is different from the GUI URL!
+1. Apply an optional filter expression to the component names. The `*` (zero or more characters) and `?` (zero or one character) wildcards are supported.
+1. Save the catalog entry.
+1. Open the Kubeflow Pipelines Visual Pipeline Editor and expand the palette. The components that were loaded from the specified MLX URL are displayed.
+
+### Uninstall the connector
+
+1. Remove all MLX catalog entries from the '`Manage Components`' panel.
+1. Stop JupyterLab.
+1. Uninstall the `mlx-component-catalog-connector` package.
+   ```
+   $ pip uninstall -y mlx-component-catalog-connector
+   ```
+
+### Troubleshooting
+
+**Problem: The palette does not diplay any components from the configured catalog.**
+
+**Solution:** If the the Elyra GUI does not display any error message indicating that a problem was encountered, inspect the JupyterLab log file.
+
+Example error message (The specified MLX URL is invalid):
+
+```
+Error fetching component list from MLX catalog http://localhost:8080: ... Failed to establish a new connection: [Errno 61] Connection refused'
+```
+

component-catalog-connectors/mlx-connector/mlx_catalog_connector/__init__.py

@@ -0,0 +1,15 @@
+#
+# Copyright 2018-2021 Elyra Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#

component-catalog-connectors/mlx-connector/mlx_catalog_connector/_version.py

@@ -0,0 +1,16 @@
+#
+# Copyright 2018-2021 Elyra Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+__version__ = '0.0.1'

component-catalog-connectors/mlx-connector/mlx_catalog_connector/mlx-catalog.json

@@ -0,0 +1,89 @@
+{
+  "$schema": "https://raw.githubusercontent.com/elyra-ai/elyra/master/elyra/metadata/schemas/meta-schema.json",
+  "$id": "https://raw.githubusercontent.com/elyra-ai/examples/master/component-catalog-connectors/mlx-connector/mlx_catalog_connector/mlx-catalog.json",
+  "title": "Machine Learning Exchange Component Catalog",
+  "name": "mlx-catalog",
+  "display_name": "Machine Learning Exchange Component Catalog",
+  "schemaspace": "component-registries",
+  "schemaspace_id": "ae79159a-489d-4656-83a6-1adfbc567c70",
+  "metadata_class_name": "elyra.pipeline.component_metadata.ComponentCatalogMetadata",
+  "uihints": {
+    "title": "Machine Learning Exchange Component Catalog",
+    "icon": "",
+    "reference_url": "https://elyra.readthedocs.io/en/stable/user_guide/pipeline-components.html"
+  },
+  "properties": {
+    "schema_name": {
+      "title": "Schema Name",
+      "description": "The schema associated with this instance",
+      "type": "string",
+      "const": "mlx-catalog"
+    },
+    "display_name": {
+      "title": "Display Name",
+      "description": "Display name of this Component Catalog",
+      "type": "string",
+      "minLength": 1
+    },
+    "version": {
+      "title": "Version",
+      "description": "The version associated with this instance",
+      "type": "integer",
+      "const": 1
+    },
+    "metadata": {
+      "description": "Additional data specific to this metadata",
+      "type": "object",
+      "properties": {
+        "description": {
+          "title": "Description",
+          "description": "Description of this Component Catalog",
+          "type": "string"
+        },
+        "runtime": {
+          "title": "Runtime",
+          "description": "The runtime associated with this Component Catalog",
+          "type": "string",
+          "$comment": "This enum is dynamically generated to contain the available runtime values.",
+          "enum": ["kfp"],
+          "default": "kfp",
+          "uihints": {
+            "field_type": "dropdown"
+          }
+        },
+        "categories": {
+          "title": "Category Names",
+          "description": "Category names associated with this Component Catalog (the components defined in this registry will be organized in the component palette according to these categories)",
+          "type": "array",
+          "items": {
+            "type": "string",
+            "maxLength": 18
+          },
+          "uihints": {
+            "field_type": "array",
+            "category": "Component Categories"
+          }
+        },
+        "mlx_api_url": {
+          "title": "MLX API URL",
+          "description": "API endpoint URL for the Machine Learning Exchange server",
+          "type": "string",
+          "format": "uri",
+          "uihints": {
+            "category": "Source"
+          }
+        },
+        "filter": {
+          "title": "Component name filter",
+          "description": "Only return components that match the specified name filter expression. * and ? are valid wildcards.",
+          "type": "string",
+          "uihints": {
+            "category": "Source"
+          }
+        }        
+      },
+      "required": ["runtime", "mlx_api_url"]
+    }
+  },
+  "required": ["schema_name", "display_name", "version", "metadata"]
+}