Skip to content

Commit

Permalink
fix: Improve BYOB Readme (#302)
Browse files Browse the repository at this point in the history 
* improved the Readmes to point to our public documentations

* improved the Readmes to point to our public documentations
  • Loading branch information
@mkaesz-wandb
mkaesz-wandb authored Oct 16, 2024
1 parent 7254b61 commit 47975f1
Show file tree
Hide file tree
Showing 3 changed files with 29 additions and 166 deletions.
62 changes: 4 additions & 58 deletions examples/byob-sse-s3/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,14 @@

## About

This example does not deploy an instance of Weights & Biases. Instead it is an
example of the resources that need to be created to deploy use with an S3 bucket
for.
Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector].

This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket and permissions) in the customer's account and grants the W&B AWS account access to them.

This module uses AE256 Encryption to protect the object store.

---

When using bring your own bucket you will need to grant our account
(`830241207209`) access to an S3 Bucket and KMS Key for encryption and decryption.
decryption

## Using Terraform

Terraform is the preferred method for deploying BYOB.
Expand Down Expand Up @@ -59,54 +55,4 @@ Do not configure a KMS key on the object store. Your configuration should look l

### Creating S3 Bucket

Lastly, you'll need to create the S3 bucket. Make sure to enable CORS access. Your CORS configuration should look like the following:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<ExposeHeader>ETag</ExposeHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>
```

As stated above, server side encryption will be handled via SSE-S3 encryption with AE256.

Finally, grant the Weights & Biases Deployment account access to this S3 bucket:

```json
{
"Version": "2012-10-17",
"Id": "WandBAccess",
"Statement": [
{
"Sid": "WAndBAccountAccess",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::830241207209:root" },
"Action": [
"s3:GetObject*",
"s3:GetEncryptionConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:PutObject",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:GetBucketVersioning"
],
"Resource": [
"arn:aws:s3:::<WANDB_BUCKET>",
"arn:aws:s3:::<WANDB_BUCKET>/*"
]
}
]
}
```
Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually.
102 changes: 4 additions & 98 deletions examples/byob/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,11 @@

## About

This example does not deploy an instance of Weights & Biases. Instead it is an
example of the resources that need to be created to deploy use with an S3 bucket
for.
Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector].

---
This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket, KMS key and permissions) in the customer's account and grants the W&B AWS account access to them.

When using bring your own bucket you will need to grant our account
(`830241207209`) access to an S3 Bucket and KMS Key for encryption and decryption.
decryption
---

## Using Terraform

Expand Down Expand Up @@ -40,94 +36,4 @@ can version, reuse, and share.

## Using AWS Console

### Creating KMS Key

We require you to provision a KMS Key which will be used to encrypt and decrypt
your S3 bucket. Make sure to enable key usage type for `ENCRYPT_DECRYPT`
purposes. It will require to have the following policy:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid" : "Internal",
"Effect" : "Allow",
"Principal" : { "AWS" : "<you account id>" },
"Action" : "kms:*",
"Resource" : "<aws_kms_key.key.arn>"
},
{
"Sid" : "External",
"Effect" : "Allow",
"Principal" : { "AWS" : "arn:aws:iam::830241207209:root" },
"Action" : [
"kms:Decrypt",
"kms:Describe*",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*"
],
"Resource" : "<aws_kms_key.key.arn>"
}
]
}
```

This policy gives access to your internal account, a swell while also providing
our service account with the requires permissions. Please keep a record of the
KMS ARN as we will need that during the deployment.

### Creating S3 Bucket

Lastly, you'll need to create the S3 bucket. Make sure to enable CORS access. Your CORS configuration should look like the following:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<ExposeHeader>ETag</ExposeHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>
```

Also, enable server side encryption and use the KMS key you just generated.

Finally, grant the Weights & Biases Deployment account access to this S3 bucket:

```json
{
"Version": "2012-10-17",
"Id": "WandBAccess",
"Statement": [
{
"Sid": "WAndBAccountAccess",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::830241207209:root" },
"Action" : [
"s3:GetObject*",
"s3:GetEncryptionConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:PutObject",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:GetBucketVersioning"
],
"Resource": [
"arn:aws:s3:::<WANDB_BUCKET>",
"arn:aws:s3:::<WANDB_BUCKET>/*"
]
}
]
}
```
Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually.
31 changes: 21 additions & 10 deletions modules/secure_storage_connector/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
# Weights & Biases Secure Storage Connector Module

This is a Terraform module for provisioning an s3 bucket to be used with Weights and Biases.
A KMS key used to encrypt S3 objects will also be created by default. To use this bucket with Weights and Biases
multi-tenant cloud, pass `arn:aws:iam::725579432336:role/WandbIntegration` for the `aws_principal_arn` variable.
## About

## AWS Services Used
Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector].

This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket, KMS key and permissions) in the customer's account and grants the W&B AWS account access to them.

---

## Using Terraform

### AWS Services Used

- AWS Identity & Access Management (IAM)
- AWS Key Management System (KMS)
- Amazon S3

## How to Use This Module
### How to Use This Module

- Ensure account meets module pre-requisites from above.
- Create a Terraform configuration that pulls in this module and specifies
Expand All @@ -35,20 +41,20 @@ module "secure_storage_connector" {

<!-- BEGIN_TF_DOCS -->

## Requirements
### Requirements

| Name | Version |
| --------------------------------------------------------------------------- | ------- |
| <a name="requirement_terraform"></a> [terraform](#requirement_terraform) | ~> 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement_aws) | ~> 3.60 |

## Providers
### Providers

| Name | Version |
| ------------------------------------------------ | ------- |
| <a name="provider_aws"></a> [aws](#provider_aws) | 3.61.0 |

## Inputs
### Inputs

| Name | Description | Type | Default | Required |
|--------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|----------|-----------|:--------:|
Expand All @@ -58,12 +64,17 @@ module "secure_storage_connector" {
| <a name="input_sse_algorithm"></a> [sse_algorithm](#input_sse_algorithm) | The server-side encryption algorithm to use. Valid values are `AES256` and `aws:kms` | `string` | `aws:kms` | no |
| <a name="input_aws_principal_arn"></a> [aws_principal_arn](#input_aws_principal_arn) | AWS principal that can access the bucket | `string` | `null` | yes |

## Outputs
### Outputs

| Name | Description |
|-----------------------------------------------------------------------------|-------------------------------------------------------------------------|
| <a name="bucket_name"></a> [bucket_name](#bucket_name) | The name of the bucket created |
| <a name="bucket_arn"></a> [bucket_arn](#output_bucket_arn) | The arn of the bucket created |
| <a name="bucket_kms_key_arn"></a> [bucket_kms_key_arn](#bucket_kms_key_arn) | The arn of the kms key created |

<!-- END_TF_DOCS -->
<!-- END_TF_DOCS -->


## Using AWS Console

Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually.

0 comments on commit 47975f1

Please sign in to comment.