Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Impl Open-VE Micro #20

Merged
merged 9 commits into from
Aug 23, 2024
Merged

feat: Impl Open-VE Micro #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ebdf2be
impl
Aug 23, 2024
dccb558
fix
Aug 23, 2024
57b3368
fix store
Aug 23, 2024
db3aa53
fix address
Aug 23, 2024
29dd802
impl
Aug 23, 2024
e8089ce
fix
Aug 23, 2024
ec82657
make operation parallel
Aug 23, 2024
01648d5
add readme
Aug 23, 2024
80c1cc0
fix
Aug 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test-runn.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
go install github.com/k1LoW/runn/cmd/runn@latest
- name: "Start Server"
run: |
go run go/cmd/open-ve/main.go run --http-addr=:8080 &
go run go/cmd/open-ve/main.go run --http-port=8080 &
- name: "Run runn"
run: |
export ENDPOINT=http://0.0.0.0:8080
Expand Down
203 changes: 44 additions & 159 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,24 +18,29 @@ Open-VE offers an HTTP API and a gRPC API. We will provide a client SDK in the f

#### 1. CLI Flags or Environment Variables

| CLI Args | Env | Default | Desc |
| ----------------------------- | ----------------------------------- | ------------ | --------------------------- |
| `--http-addr` | `OPEN-VE_HTTP_ADDR` | `:8080` | HTTP server address |
| `--http-cors-allowed-origins` | `OPEN-VE_HTTP_CORS_ALLOWED_ORIGINS` | `["*"]` | CORS allowed origins |
| `--http-cors-allowed-headers` | `OPEN-VE_HTTP_CORS_ALLOWED_HEADERS` | `["*"]` | CORS allowed headers |
| `--http-tls-enabled` | `OPEN-VE_HTTP_TLS_ENABLED` | `false` | HTTP server TLS enabled |
| `--http-tls-cert-path` | `OPEN-VE_HTTP_TLS_CERT_PATH` | `""` | HTTP server TLS cert path |
| `--http-tls-key-path` | `OPEN-VE_HTTP_TLS_KEY_PATH` | `""` | HTTP server TLS key path |
| `--grpc-addr` | `OPEN-VE_GRPC_ADDR` | `:9000` | gRPC server address |
| `--grpc-tls-enabled` | `OPEN-VE_GRPC_TLS_ENABLED` | `false` | gRPC server TLS enabled |
| `--grpc-tls-cert-path` | `OPEN-VE_GRPC_TLS_CERT_PATH` | `""` | gRPC server TLS cert path |
| `--grpc-tls-key-path` | `OPEN-VE_GRPC_TLS_KEY_PATH` | `""` | gRPC server TLS key path |
| `--store-engine` | `OPEN-VE_STORE_ENGINE` | `memory` | store engine (redis/memory) |
| `--store-redis-addr` | `OPEN-VE_STORE_REDIS_ADDR` | `redis:6379` | Redis address |
| `--store-redis-password` | `OPEN-VE_STORE_REDIS_PASSWORD` | `""` | Redis password |
| `--store-redis-db` | `OPEN-VE_STORE_REDIS_DB` | `0` | Redis DB |
| `--store-redis-pool-size` | `OPEN-VE_STORE_REDIS_POOL_SIZE` | `1000` | Redis pool size |
| `--log-level` | `OPEN-VE_LOG_LEVEL` | `info` | Log level |
| CLI Args | Env | Default | Desc |
| ----------------------------- | ----------------------------------- | ------------ | ---------------------------------------------------------------------- |
| `--mode` | `OPEN-VE_MODE` | `master` | master or slave |
| `--slave-id` | `OPEN-VE_SLAVE_ID` | | Unique slave ID (if mode is slave, this is required) |
| `--slave-slave-http-addr` | `OPEN-VE_SLAVE_SLAVE_HTTP_ADDR` | | HTTP server address (if mode is slave, this is required) |
| `--slave-master-http-addr` | `OPEN-VE_SLAVE_MASTER_HTTP_ADDR` | | Master HTTP server address (if mode is slave, this is required) |
| `--slave-master-tsl-enabled` | `OPEN-VE_SLAVE_MASTER_TLS_ENABLED` | `false` | Connect to master server with TLS (if mode is slave, this is required) |
| `--http-port` | `OPEN-VE_HTTP_PORT` | `8080` | HTTP server port number |
| `--http-cors-allowed-origins` | `OPEN-VE_HTTP_CORS_ALLOWED_ORIGINS` | `["*"]` | CORS allowed origins |
| `--http-cors-allowed-headers` | `OPEN-VE_HTTP_CORS_ALLOWED_HEADERS` | `["*"]` | CORS allowed headers |
| `--http-tls-enabled` | `OPEN-VE_HTTP_TLS_ENABLED` | `false` | HTTP server TLS enabled |
| `--http-tls-cert-path` | `OPEN-VE_HTTP_TLS_CERT_PATH` | `""` | HTTP server TLS cert path |
| `--http-tls-key-path` | `OPEN-VE_HTTP_TLS_KEY_PATH` | `""` | HTTP server TLS key path |
| `--grpc-port` | `OPEN-VE_GRPC_ADDR` | `:9000` | gRPC server port number |
| `--grpc-tls-enabled` | `OPEN-VE_GRPC_TLS_ENABLED` | `false` | gRPC server TLS enabled |
| `--grpc-tls-cert-path` | `OPEN-VE_GRPC_TLS_CERT_PATH` | `""` | gRPC server TLS cert path |
| `--grpc-tls-key-path` | `OPEN-VE_GRPC_TLS_KEY_PATH` | `""` | gRPC server TLS key path |
| `--store-engine` | `OPEN-VE_STORE_ENGINE` | `memory` | store engine (redis/memory) |
| `--store-redis-addr` | `OPEN-VE_STORE_REDIS_ADDR` | `redis:6379` | Redis address |
| `--store-redis-password` | `OPEN-VE_STORE_REDIS_PASSWORD` | `""` | Redis password |
| `--store-redis-db` | `OPEN-VE_STORE_REDIS_DB` | `0` | Redis DB |
| `--store-redis-pool-size` | `OPEN-VE_STORE_REDIS_POOL_SIZE` | `1000` | Redis pool size |
| `--log-level` | `OPEN-VE_LOG_LEVEL` | `info` | Log level |

#### 2. Config File

Expand All @@ -44,16 +49,17 @@ You can also use a config file in YAML format.
Place the `config.yaml` in the same directory or `$HOME/.open-ve/config.yaml`.

```yaml
mode: "master"
http:
addr: ":8080"
port: "8080"
corsAllowedOrigins: ["*"]
corsAllowedHeaders: ["*"]
tls:
enabled: false
certPath: ""
keyPath: ""
grpc:
addr: ":9000"
poer: "9000"
tls:
enabled: false
certPath: ""
Expand Down Expand Up @@ -91,7 +97,21 @@ go build -o open-ve go/cmd/open-ve/main.go
docker-compose up
```

## CEL
## System Design

### Master-Slave Architecture

Open-VE supports a master-slave architecture designed for scalability and compatibility with microservice environments.

In slave mode, Open-VE connects to the master server and syncs validation rules every 30 seconds.

When a validation check request is made to the master server, it distributes the request across the connected slave servers.

Additionally, you can directly request validation checks from the slave servers.

![micro-validator](https://github.com/user-attachments/assets/e248d40c-bcc7-4219-a65a-5b243e101000)

### CEL

We use [CEL](https://github.com/google/cel-spec/blob/master/doc/langdef.md) as the expression language for validation rules.

Expand All @@ -111,143 +131,8 @@ Supported types:
| message names | | ❓ |
| `type` | | ❓ |

## Example (HTTP API)

### Register Validation Rules

Request:

```bash
curl --request POST \
--url http://localhost:8080/v1/dsl \
--header 'Content-Type: application/json' \
--data '{
"validations": [
{
"id": "item",
"cels": [
"price > 0", # price must be greater than 0
"size(image) < 360" # image size must be less than 360 bytes
],
"variables": [
{
"name": "price",
"type": "int"
},
{
"name": "image",
"type": "bytes"
}
]
},
{
"id": "user",
"cels": [
"size(name) < 20" # name length must be less than 20
],
"variables": [
{
"name": "name",
"type": "string"
}
]
}
]
}'
```

Response:

```json
{}
```

### Get Current Validation Rules

Request:

```bash
curl --request GET \
--url http://localhost:8080/v1/dsl \
--header 'Content-Type: application/json'
```

Response:

```json
{
"validations": [
{
"id": "item",
"cels": ["price > 0", "size(image) < 360"],
"variables": [
{
"name": "price",
"type": "int"
},
{
"name": "image",
"type": "bytes"
}
]
},
{
"id": "user",
"cels": ["size(name) < 20"],
"variables": [
{
"name": "name",
"type": "string"
}
]
}
]
}
```

### Validate
## Examples

Request:

```bash
curl --request POST \
--url 'http://localhost:8080/v1/check' \
--header 'Content-Type: application/json' \
--data '{
"validations": [
{
"id": "item",
"variables": {
"price": -100,
"image": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGO4unY2AAR4Ah51j5XwAAAAAElFTkSuQmCC" # send base64 encoded image
}
},
{
"id": "user",
"variables": {
"name": "longlonglonglongname"
}
}
]
}'
- [Example of Master Slave Architecture](docs/Master-Slave-Example.md)

```

Response:

```json
{
"results": [
{
"id": "item",
"isValid": false,
"message": "failed validations: price > 0"
},
{
"id": "user",
"isValid": false,
"message": "failed validations: size(name) < 20"
}
]
}
```
- [Example of Monolithic Architecture](docs/Monolithic-Example.md)
45 changes: 40 additions & 5 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ services:
networks:
- default
restart: unless-stopped
server:
master-node:
build:
context: .
container_name: server
container_name: master-node
ports:
- "8080:8080"
- "9000:9000"
Expand All @@ -21,17 +21,52 @@ services:
depends_on:
- redis
environment:
- OPEN-VE_HTTP_ADDR=
- OPEN-VE_MODE=master
- OPEN-VE_HTTP_PORT=
- OPEN-VE_HTTP_CORS_ALLOWED_ORIGINS=
- OPEN-VE_HTTP_CORS_ALLOWED_HEADERS=
- OPEN-VE_HTTP_TLS_ENABLED=
- OPEN-VE_HTTP_TLS_CERT_PATH=
- OPEN-VE_HTTP_TLS_KEY_PATH=
- OPEN-VE_GRPC_ADDR=
- OPEN-VE_GRPC_PORT=
- OPEN-VE_GRPC_TLS_ENABLED=
- OPEN-VE_GRPC_TLS_CERT_PATH=
- OPEN-VE_GRPC_TLS_KEY_PATH=
- OPEN-VE_STORE_ENGINE=
- OPEN-VE_STORE_ENGINE=redis
- OPEN-VE_STORE_REDIS_ADDR=
- OPEN-VE_STORE_REDIS_PASSWORD=
- OPEN-VE_STORE_REDIS_DB=
- OPEN-VE_STORE_REDIS_POOL_SIZE=
- OPEN-VE_LOG_LEVEL=
slave-node:
build:
context: .
container_name: slave-node
ports:
- "8090:8080"
- "9010:9000"
networks:
- default
depends_on:
- redis
- master-node
environment:
- OPEN-VE_MODE=slave
- OPEN-VE_SLAVE_ID=slave-node
- OPEN-VE_SLAVE_SLAVE_HTTP_ADDR=http://slave-node:8080
- OPEN-VE_SLAVE_MASTER_HTTP_ADDR=http://master-node:8080
- OPEN-VE_SLAVE_MASTER_TLS_ENABLED=
- OPEN-VE_HTTP_PORT=
- OPEN-VE_HTTP_CORS_ALLOWED_ORIGINS=
- OPEN-VE_HTTP_CORS_ALLOWED_HEADERS=
- OPEN-VE_HTTP_TLS_ENABLED=
- OPEN-VE_HTTP_TLS_CERT_PATH=
- OPEN-VE_HTTP_TLS_KEY_PATH=
- OPEN-VE_GRPC_PORT=
- OPEN-VE_GRPC_TLS_ENABLED=
- OPEN-VE_GRPC_TLS_CERT_PATH=
- OPEN-VE_GRPC_TLS_KEY_PATH=
- OPEN-VE_STORE_ENGINE=redis
- OPEN-VE_STORE_REDIS_ADDR=
- OPEN-VE_STORE_REDIS_PASSWORD=
- OPEN-VE_STORE_REDIS_DB=
Expand Down
Loading
Loading