forked from cloudfoundry/docs-bosh
-
Notifications
You must be signed in to change notification settings - Fork 0
/
agent-cpi-interactions.html.md.erb
240 lines (190 loc) · 8.96 KB
/
agent-cpi-interactions.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
---
title: Agent-CPI Interactions
---
Here is an overview of the interactions between the CPI and the Agent on CloudStack as an example:
* The CPI drives the IaaS and the Agent.
* The Agent is a versatile process which configures the OS to leverage IaaS-provisionned resources (network interfaces, disks, etc.), and perform other BOSH tasks (jobs compilation, jobs instanciation, etc.)
* The CPI asks the IaaS to instanciate VM template, VMs, volumes and possibly other constructs (floating IPs, security groups, connect LBs, etc.)
* The Agent is initially driven by the CPI through the bosh-registry, and then by the Director through NATS-based messaging. The registry provides Director-side meta data to the Agent.
<div class="gray"><%= image_tag("images/cpi-interactions-overview.png") %></div>
<%#
Image source in Saas (no need to own a graffle license and run macos)
https://www.gliffy.com/go/html5/9205487?toke=&app=1b5094b0-6042-11e2-bcfd-0800200c9a66&dev=false
%>
The following sequence diagram illustrates in more details the CPI, agent, and registry interactions,
in the case of the [CloudStack CPI](https://github.com/cloudfoundry-community/bosh-cloudstack-cpi-release)
<div class="gray"><%= image_tag("images/cpi-sequence-diag.png") %></div>
<%#
Image source in plantuml format.
syntax at http://plantuml.com/sequence.html
Render it online http://plantuml.com/plantuml/ or from a private plantuml instance, see http://plantuml.com/running.html
@startuml
box "DIRECTOR"
participant director
end box
box "CPI" #LightBlue
participant cpi
participant bosh_registry
end box
box "CLOUDSTACK"
participant cloudstack_api
participant vrouter
end box
box "VM" #LightBlue
participant vm
participant bosh_agent
end box
director -> cpi : create_vm;
cpi -> cloudstack_api : create vm \nand set user-data;
cpi -> bosh_registry : feed bosh registry: \n(networking, root+eph disks, \nbosh nats, blobstore
cloudstack_api -> vrouter : set user data\nin metadata\n service;
cloudstack_api -> vm : provision vm;
activate vm
vm -> bosh_agent : vm boostraps in dhcp,\nbosh-agent starts \n (reading agent.json);
bosh_agent -> vrouter : query meta-data server:\nget bosh_registry address from user data;
bosh_agent -> bosh_registry : gets bootstrap info, ip adress and disks;
bosh_agent -> vm : reconfigure network static ip;
bosh_agent -> vm : mount and partition ephemeral disk;
hide footbox
@enduml
%>
---
## <a id="agent-config"></a> Agent config file formats
This section details the configuration and protocols supported by the Agent.
The [VM Configuration Locations](vm-config.html#agent) provides a list of the configuration files that the Agent and their roles.
----
### <a id="agent-json"></a> `agent.json` file
`/var/vcap/bosh/agent.json`: Start up settings for the Agent that describe how to find bootstrap settings, and disable certain Agent functionality.
The loading agent.json file is described into [config_test.go](https://github.com/cloudfoundry/bosh-agent/blob/master/app/config_test.go)
The Platform part of the file is documented into [LinuxOptions](https://godoc.org/github.com/cloudfoundry/bosh-agent/platform#LinuxOptions)
The Infrastructure part of the file is documented into [Options](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#Options) and in particular:
* the sources of configuration with [SettingsOptions](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#SettingsOptions):
* [CDROM](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#CDROMSourceOptions)
* [ConfigDrive](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#ConfigDriveSourceOptions)
* [HTTP](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#HTTPSourceOptions)
Sample `agent.json` which configures the agent to read from an HTTP meta data service at a custom URL.
```json
{
"Platform": {
"Linux": {
"CreatePartitionIfNoEphemeralDisk": true,
"DevicePathResolutionType": "virtio"
}
},
"Infrastructure": {
"NetworkingType": "static",
"Settings": {
"Sources": [
{
"Type": "HTTP",
"URI": "http://10.234.228.142"
}
],
"UseServerName": true,
"UseRegistry": true
}
}
}
```
Sample `agent.json` which configures the agent ot read from a config drive. See [reference sample config-drive json](https://github.com/cloudfoundry/bosh-agent/blob/cecf354d90f6cedf540799fb50395adb034b16d2/integration/assets/config-drive-agent.json). See [config drive integration test](https://github.com/cloudfoundry/bosh-agent/blob/cecf354d90f6cedf540799fb50395adb034b16d2/integration/config_drive_test.go)
```json
{
"Platform": {
"Linux": {
"UseDefaultTmpDir": true,
"UsePreformattedPersistentDisk": true,
"BindMountPersistentDisk": false,
"DevicePathResolutionType": "virtio"
}
},
"Infrastructure": {
"Settings": {
"Sources": [
{
"Type": "ConfigDrive",
"DiskPaths": [
"/dev/disk/by-label/CONFIG-2",
"/dev/disk/by-label/config-2"
],
"MetaDataPath": "ec2/latest/meta-data.json",
"UserDataPath": "ec2/latest/user-data.json"
}
],
"UseServerName": true,
"UseRegistry": true
}
}
}
```
----
### <a id="meta-data"></a> Meta-data server
The meta-data server initial serves the registry URL and DNS server to be able to resolve it.
Following is a sample content od the user-data part of the HTTP meta-data
```json
{
"server": {"name": "vm-384sd4-r7re9e"},
"registry": {"endpoint": "http://192.168.0.255:8080/client/api"},
"dns": {"nameserver": ["10.234.50.180"]}
}
```
The supported format of the meta-data server by the bosh-agent are documented into [UserDataContentsType](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#UserDataContentsType) and [http\_metadata\_service_test.go](https://github.com/cloudfoundry/bosh-agent/blob/1dca3244702c18bf2c36483c529d4e7b3fb92b2e/infrastructure/http_metadata_service_test.go)
along with the expected behavior of the bosh agent when reading this config.
----
### <a id="registry"></a> Registry
The registry provides bosh-side meta data to the bosh agent.
From [Warden CPI doc](https://github.com/cppforlife/bosh-warden-cpi-release/blob/be1869737c0bfba96662dde3499c9181863f91a7/docs/bosh-micro-usage.md):
* Registry is used by CPI to pass data to the Agent. Registry is started on a server specified by registry properties.
* SSH tunnel will be started if SSH tunnel options are provided. SSH tunnel will start reverse ssh tunnel from the micro BOSH VM to the registry port. That makes registry available to the agent on remote machine.
#### <a id="registry-protocol"></a> Registry HTTP protocol
The Agent expects to communicate with the bosh registry with a REST API documented into [api\_controller\_spec.rb](https://github.com/cloudfoundry/bosh/blob/2f73281f1a2a155ee807e7c0c9b8187c5a742f78/bosh-registry/spec/unit/bosh/registry/api_controller_spec.rb)
Reference registry client and servers implementations are available in:
* go-lang through [frodenas/bosh-registry](https://github.com/frodenas/bosh-registry)
* java through [cloudfoundry-community/bosh-cloudstack-cpi-core](https://github.com/cloudfoundry-community/bosh-cloudstack-cpi-core/tree/44d14a5f184d2d5e8f1f2fcd6344e734d3344673/src/main/java/com/orange/oss/cloudfoundry/cscpi/boshregistry)
#### <a id="registry-format"></a> Registry settings format
The JSON payload of the settings stored in the registry are called settings, and their format supported by the Agent is documented in [settings_test.go](https://github.com/cloudfoundry/bosh-agent/blob/ebf9a36dfddf783ffae5ae3748787d259a29374a/settings/settings_test.go)
Following is a sample content of the registry
```json
{
"agent_id": "agent-xxxxxx",
"blobstore": {
"provider": "local",
"options": {
"endpoint": "http://xx.xx.xx.xx:25250",
"password": "password",
"blobstore_path": "/var/vcap/micro_bosh/data/cache",
"user": "agent"
}
},
"disks": {
"system": "/dev/xvda",
"ephemeral": "/dev/sdb",
"persistent": {}
},
"env": {},
"networks": {
"default": {
"type": "manual",
"ip": "10.234.228.158",
"netmask": "255.255.255.192",
"cloud_properties": {"name": "3112 - preprod - back"},
"dns": [
"10.234.50.180",
"10.234.71.124"
],
"gateway": "10.234.228.129",
"mac": null
}
},
"ntp": [],
"mbus": "nats://nats:[email protected]:4222",
"vm": {"name": "vm-yyyy"},
"trusted_certs": null
}
```
----
### <a id="settings_json"></a> `settings.json` file
`/var/vcap/bosh/settings.json`: Local copy of the bootstrap settings used by the Agent to configure network, system properties for the VM. They are refreshed every time Agent is restarted.
The `settings.json` payload format is the same as the settings format initially returned by the registry.
---
Next: [Building a Stemcell](build-stemcell.html)
Previous: [CPI API v1](cpi-api-v1.html)