Skip to content

Commit

Permalink
edit: custom resources documentation (#1696)
Browse files Browse the repository at this point in the history
* update: custom resources doc with reference to marketplace docs

* add: some use cases

* add: more use cases

* move use cases to separate page; apply suggestions

* fix: broken links

* add: notes from code review

* improve marketplace documentation

* add: deployment sequence of template-based cr

* improve examples and distinguish CR type

* restructure use cases page and add some CR examples (wip)

* add lambda use case; apply some suggestions

* Update docs/marketplace/add_to_marketplace/add_item_by_type/add_custom_resource.md
  • Loading branch information
GiovannaMonti authored Oct 4, 2024
1 parent 7b01a65 commit 2051d28
Show file tree
Hide file tree
Showing 14 changed files with 348 additions and 46 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,22 @@ title: Create Custom Resources
sidebar_label: Create Custom Resources
---

## How to use the Custom Resources
A Custom Resource allows you to define custom objects that are not part of the standard Console resources and can be used to extend the Console capabilities. It provides a way to write configurations that can be translated into different configuration-as-code elements.

A Custom Resource allows you to define custom objects that are not part of the standard Console supported resources.
## How to use the Custom Resource

For example, it is possible to:
With Custom Resources it is possible to:

- configure Kubernetes Custom Resource that are managed by the cluster (e.g. the Traefik `IngressRoute`);
- generate manifests for different runtimes using the [External Configuration Generator](/console/company-configuration/providers/extensions/orchestrator-generator/overview.mdx).
- configure resources for virtually any runtime using templates that are populated with custom values at deployment time
- configure Kubernetes Custom Resources that are managed by the cluster (e.g. the Traefik `IngressRoute`)

## How to manage a Custom Resource
To learn more about possible use cases, you can go to the [dedicated section](/console/design-your-projects/custom-resources/use-cases.md)

:::tip
You can also use Custom Resources together with the [External Configuration Generator](/console/company-configuration/providers/extensions/orchestrator-generator/overview.mdx) to generate manifests for different runtimes.
:::

## How to manage a Custom Resource in Console

It is possible to manage the Custom Resource from inside the Design section of the Console, in the dedicated area called *Custom Resources*.

Expand All @@ -31,7 +37,7 @@ To create a resource from scratch, you need to provide the following information

In creation, you will see the preview of the generated manifest.

![Create from scratch](./img/custom-resources/create-from-scratch.png)
![Create from scratch](./img/create-from-scratch.png)

#### Create a Custom Resource from Marketplace

Expand All @@ -41,11 +47,11 @@ To allow users to add a Custom Resource to their project from marketplace, you n

To create a resource from Marketplace, you need to select the Custom Resource you want to create.

![Create from Marketplace](./img/custom-resources/create-from-marketplace.png)
![Create from Marketplace](./img/create-from-marketplace.png)

The Marketplace could contain *versioned* Custom Resources. In that case, when selecting the Custom Resource to create, you will see the available versions and you will be able to select the one you prefer.

![Create from Marketplace a versioned Custom Resource](./img/custom-resources/create-from-marketplace-versioned.png)
![Create from Marketplace a versioned Custom Resource](./img/create-from-marketplace-versioned.png)

In this case, you can only modify the *name* of the Custom Resource: the *kind* and the *apiVersion* fields are managed by the versioned marketplace item, so you cannot modify them manually, neither during the creation nor the update.

Expand All @@ -63,15 +69,15 @@ The Custom Resource has some supported fields, other fields will be ignored. The
- **annotations**: the annotations of the Custom Resource, it can be any key/value pair;
- **spec** (*required*): the spec of the Custom Resource, it can be any object.

![Update](./img/custom-resources/update-gateway-custom-resource.png)
![Update](./img/update-gateway-custom-resource.png)

Custom Resources created from Marketplace items can not have the `apiVersion` and the `kind` fields modified. Attempting to do so will result in an error badge shown in the UI, and the updates on the manifest will be ignored.

Each version of the Custom Resource defines specific values for these fields.
You can select a new version by clicking on the icon at the top right corner of the badge, near the version name, and selecting the *Change version* option from the pop-up menu.
A modal window will open, where you can see the available versions of the Custom Resource and select the one you want to use.

![Change version](./img/custom-resources/change-custom-resource-version.png)
![Change version](./img/change-custom-resource-version.png)

Inside the modal you can also see the `apiVersion` and the `kind` of the Custom Resource of the selected version, to give you a better idea of the configuration you are selecting.

Expand All @@ -91,12 +97,48 @@ To delete a Custom Resource, you have to click on the delete button at the botto

You need to insert the Custom Resource name and click on the delete button.

![Delete Custom Resource](./img/custom-resources/delete.png)
![Delete Custom Resource](./img/delete.png)

## How to manage a Custom Resource in the Marketplace

Users with appropriate permissions can manage the lifecycle of a Custom Resource on the Marketplace, being able to publish their own Custom Resources and update them.

Custom Resources are published to the Marketplace using the `custom-resource` type in the marketplace item schema. Versioning is supported, allowing users to make updates without overwriting previous versions of the Custom Resource.

To learn more on how to add or manage a Custom Resource in the Marketplace, see the [Add Custom Resource to the Marketplace](/marketplace/add_to_marketplace/add_item_by_type/add_custom_resource.md) section.

### Template-based Custom Resource management

:::info
Please note that template-based Custom Resources are only supported in Projects using the [Enhanced Workflow](/development_suite/set-up-infrastructure/overview.md).
:::

By default, Custom Resources available in the Marketplace are built using templates. The custom resource item in the Marketplace defines a `generator` field that specifies the template and base folder for generated files. The `apiVersion` field is also set to the special value `custom-generator.console.mia-platform.eu/v1`, necessary to dynamically generate the resource from the Marketplace.

Templates are stored as strings and interpolated at deployment time using the [mustache template system](https://mustache.github.io/). After creating the resource, the user can proceed with its deployment. The template outlines the configuration files that will be generated during deployment: during this process the template is populated with user-provided values, and the resulting files are stored in a dedicated folder in the Project repository, as defined in the Marketplace item specification.

Here is a sequence of what happens when a template-based Custom Resource is deployed:

```mermaid
sequenceDiagram
participant User
participant Console
User->>Console: Save configuration with template-based Custom Resource<br/>generated from Marketplace
Console-->>User: Configuration saved
User->>Console: Deploy the configuration
Console->>Console: Get custom resource from the Marketplace
Console->>Console: Interpolate the template with user-defined values
Console->>Console: Generate configuration files and store them in a dedicated folder
Console-->>User: Deployment successful
```

:::info
Deploying template-based Custom Resources for different runtimes than Kubernetes will require additional configuration on the deployment pipeline, in order to handle the additional files generated from the template.
:::

## Future Improvements

In the future, we plan to add more features to the Resources, such as:

- the ability to add a custom template to be interpolated to generate the manifest;
- see the Custom Resources runtime inside the Runtime Section of the Console;
- manage Custom Resource with a specific dynamically generated UI.
167 changes: 167 additions & 0 deletions docs/console/design-your-projects/custom-resources/use-cases.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
---
id: custom-resources-use-cases
title: Use Cases
sidebar_label: Use Cases
---

The following examples showcase how users can leverage **Custom Resources** in different scenarios.

:::tip
Find out all there is to know about Custom Resources in the [dedicated section](/console/design-your-projects/custom-resources/custom-resources.md).
:::

## Template-based Custom Resources

### AWS Lambda

AWS Lambda functions can be managed in different ways, depending on the user’s needs. One option allows users to create a Kubernetes custom resource for Lambda directly from the Marketplace. The Lambda function is managed by an operator within the Kubernetes cluster, and users can write the Lambda code directly from the Console.

Another option is for the user to create a custom resource from the Marketplace that generates a code repository for the Lambda function. In this case, users can develop the Lambda code separately and then package it into a Docker image.

A third approach is to manage AWS Lambda functions via CloudFormation. This method is ideal for users who prefer working with AWS-native tools and infrastructure-as-code patterns.

These three options provide different levels of flexibility for users who require integration with AWS infrastructure.

Here is an example of a template-based Custom Resource that creates a Lambda function:

<details>
<summary>AWS Lambda</summary>
<p>

```yaml
name: my-lambda-function
meta:
kind: LambdaTemplateGenerator
apiVersion: custom-generator.console.mia-platform.eu/v1
spec:
targetRuntime: nodejs20.x
zipCode: 'exports.handler = async (event) => ({ statusCode: 200, body: JSON.stringify("Hello from Lambda!") })'
memorySize: 1024
timeout: 60
id: MyFunction
generator:
type: template
configurationBaseFolder: aws-cloudformation
templates:
- name: lambda
template: |
AWSTemplateFormatVersion: '2010-09-09'
Resources:
Fn%spec.id%:
Type: AWS::Lambda::Function
Properties:
%#spec.role%
Role: %spec.role%
%/spec.role%
%^spec.role%
Role: arn:aws:iam::694348909644:role/lambda-role
%/spec.role%
FunctionName: %metadata.name%
Handler: index.handler
Runtime: %spec.targetRuntime%
Code:
%#spec.zipCode%
ZipFile: |
%spec.zipCode%
%/spec.zipCode%
%#spec.s3Code%
S3Bucket: %spec.s3Code.bucketName%
S3Key: %metadata.name%.zip
%/spec.s3Code%
MemorySize: %spec.memorySize%
Timeout: %spec.timeout%
FunctionUrl%spec.id%:
Type: AWS::Lambda::Url
Properties:
TargetFunctionArn: !Ref Fn%spec.id%
AuthType: NONE
Outputs:
PublicUrl:
Value: !GetAtt FunctionUrl%spec.id%.FunctionUrl
Description: The public URL to access the Lambda function
```
</p>
</details>
### Amazon EC2
This use case is similar to the AWS Lambda example but focuses on the creation of EC2 instances. Users can either generate a Kubernetes CRD to manage EC2 instances through an operator in the cluster or generate Terraform files to manage infrastructure outside of Kubernetes.
The first approach, using Kubernetes operators, allows users to create and manage EC2 instances directly through their Kubernetes environment. The second approach, which involves generating Terraform files, offers a more advanced option for users needing to manage complex cloud infrastructure across multiple environments.
This flexibility allows users to choose between simple, Kubernetes-native management or more comprehensive infrastructure-as-code practices using Terraform, depending on their project needs.
### MongoDB Atlas Creation
This use case involves managing MongoDB Atlas databases with Kubernetes custom resources. Users can manage a MongoDB Atlas database by leveraging a Kubernetes operator, defining essential credentials like username, password, and the database name via Custom Resources in the Console.
For more advanced use cases, users can write a custom resource that generates a Terraform file, which can be used to manage MongoDB Atlas with greater control, including configurations for scaling, backups, and security settings.
This approach provides a straightforward method for managing small-scale database setups via the Kubernetes operator, while advanced users can opt for full Terraform-based management to integrate MongoDB with broader infrastructure.
## Kubernetes-specific Custom Resources
### Traefik IngressRoute
In this use case, the user manages Kubernetes Ingress routing for Traefik. They can either copy an existing Traefik IngressRoute Custom Resource Definition (CRD) directly into the editor in Console, or create a new one from scratch. This approach provides users with fine-tuned control over how Ingress routes are configured.
Alternatively, users can opt to create the resource from the Marketplace. This version offers simplified management of key configuration fields, such as the host and service name, making the process more accessible for users who may not need the full flexibility of direct CRD management.
This use case is ideal for users who require granular control over routing rules and protocols, or for those who prefer a simplified way to handle basic Ingress configurations.
Here is an example of a Traefik IngressRoute Custom Resource:
<details>
<summary>Traefik IngressRoute</summary>
<p>
```yaml
name: traefik-ingressroute
meta:
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
labels:
- name: app.kubernetes.io/instance
value: ingress-controller
spec:
entryPoints:
- websecure
routes:
- match: Host(`{{PROJECT_HOST}}`)
kind: Rule
services:
- name: api-gateway
port: 8080
```
</p>
</details>
### kube-green SleepInfo
This use case involves using [kube-green](https://kube-green.dev/) to automate sleep schedules for Kubernetes clusters in order to optimize resource usage. Like the Traefik IngressRoute use case, users can manage the SleepInfo CRD directly within the Console. This resource can be used to define when a cluster should "sleep", reducing its resource usage, during non-peak hours.
However, unlike the previous example, this resource is typically activated only in non-production environments, such as development or staging, to prevent disruptions in production-critical services.
This use case is particularly beneficial for organizations that want to reduce the CO2 footprint of their clusters and optimize resource costs in environments that don't require constant uptime.
Here is an example of a kube-green SleepInfo Custom Resource:
<details>
<summary>kube-green SleepInfo</summary>
<p>
```yaml
name: sleepInfo
meta:
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
spec:
sleepAt: "20:00"
timeZone: Europe/Rome
weekdays: "1-5"
```
</p>
</details>
2 changes: 1 addition & 1 deletion docs/data_catalog/database_setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ every time a new version of `data-fabric/fabric-admin` image is available, which
Data Catalog application.

:::tip
In case in your Console Project it is possible to enable [Custom Resources](/console/design-your-projects/custom-resources.md), then instead of
In case in your Console Project it is possible to enable [Custom Resources](/console/design-your-projects/custom-resources/custom-resources.md), then instead of
creating a `cronjob` manifest it would be possible to define a `job` one, which does start as soon as released and does not need to be
manually launched.

Expand Down
2 changes: 1 addition & 1 deletion docs/fast_data/runtime_management/database_setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ every time a new version of `data-fabric/fabric-admin` image is available, which
Fast Data Runtime Management application.

:::tip
In case in your Console Project it is possible to enable [Custom Resources](/console/design-your-projects/custom-resources.md), then instead of
In case in your Console Project it is possible to enable [Custom Resources](/console/design-your-projects/custom-resources/custom-resources.md), then instead of
creating a `cronjob` manifest it would be possible to define a `job` one, which does start as soon as released and does not need to be
manually launched.

Expand Down
Loading

0 comments on commit 2051d28

Please sign in to comment.