Author: Claudio Bisegni
Company: SLAC National Accelerator Laboratory
A c++ implementation of a two way gateway from kafka and EPICS Control System, that aim to be a central processing unit for specific daq needs for high level alghoritms.
It reveice command from a specific kafka's topic and permit to inject EPICS data torwards other topics.
Actual implementation receive command and interpret command and uses only JSON serializaiton (described later). It's lack of cluster management and other advanced feature
### Command Task List
- [x] Get Command
- [x] Monitor Command
- [X] Put Command
- Scalar
- ScalarArray
### Functional Task List
- [x] JSON Serialization
- [X] MSGPack Binary serialization
- [X] MSGPack compact serialization
- [X] Multithreading EPICS Monitor
- [ ] Cluster implementation
For the serialization and message format see documentation here
The application architecture logic is designed following the scheme below. The dotted line boxes represent element not yet developed.
There are two principal layer:
- Controller:
- Node Controller
- Command Controller
- Cluster Controller (to design)
- Service:
- Publisher and Subscriber:
- Data storage
- Log
- EPICS
- Cluster Services (to design)
This project aim to realize an EPICS gateway for interact with epics IOCs using kafka. It uses a input topic from a kafka cluster for receive json encoded commands that permit to execute IO operation on the IOCs.
k2eg rely upon boost program options to manage the startup option configuration. Below the complete list of the parameter:
k2eg --help
Epics k2eg:
--help Produce help information
--version Print the application version
--conf-file arg (=0) Specify if we need to load
configuration from file
--conf-file-name arg Specify the configuration file
--log-level arg (=info) Specify the log level[trace, debug,
info, error, fatal]
--log-on-console Specify when the logger print in
console
--log-on-file Specify when the logger print in file
--log-file-name arg Specify the log file path
--log-file-max-size arg (=1) Specify the maximum log file size in
mbyte
--log-on-syslog Specify when the logger print in syslog
server
--syslog-server arg Specify syslog hotsname
--syslog-port arg (=514) Specify syslog server port
--cmd-input-topic arg Specify the messages bus queue where
the k2eg receive the configuration
command
--cmd-max-fecth-element arg (=10) The max number of command fetched per
consume operation
--cmd-max-fecth-time-out arg (=250) Specify the timeout for waith the
command in microseconds
--nc-monitor-expiration-timeout arg (=3600)
Specify the amount of time with no
consumer on a queue after which monitor
can be stopped
--nc-purge-queue-on-exp-timeout arg (=1)
Specify when the a queue purged when
the monitor that push data onto is
stopped
--nc-monitor-consumer-filterout-regex arg
Specify regular expression to used to
filter out consumer group from those
used to calculate the number of active
consumer of monitor queue
--pub-server-address arg Publisher server address
--pub-impl-kv arg The key:value list for publisher
implementation driver
--sub-server-address arg Subscriber server address
--sub-group-id arg (=k2eg-default-group)
Subscriber group id
--sub-impl-kv arg The key:value list for subscriber
implementation driver
--storage-path arg (=/opt/app/k2eg.sqlite)
The path where the storage files are
saved
--monitor-worker-cron-schedule arg The cron string forconfigure the
monitor checking scheduler
--scheduler-check-delay-seconds arg (=60)
The number of second for which the
scheduler thread are going to sleep
--scheduler-thread-number arg (=1) The number of the scheduler worker
--metric-enable arg (=0) Enable metric management
--metric-server-http-port arg (=8080) The port used for publish the http
metric serverThere are two ohter different way to configure the application other to the common command line option and are:
The uses of the configuration file is activated using the command line option:
k2eg --conf-file --conf-file-name <path/to/configuration/file>the content of the file need to folow the rules:
...
log-file-max-size=1234
log-on-syslog=true
syslog-server=syslog-server
syslog-port=5678
sub-server-address=sub-address
sub-group-id=sub-group-id
...
The uses of the environment variable si automatically managed, each variable with the prefix EPICS_k2eg_ is valuated, for example the enable to use the config file can be done via ENV variable with:
export EPICS_k2eg_conf-file
export EPICS_k2eg_conf-file-name=<path/to/configuration/file>This implemets the base caget|pvaget fucntion of epics command, if possible will use a client from a monitor thread, otherwhise a new client is allcoated to perform a get operation. The reply_id field is nto mandatory and it is forwarded, into the reply message.
{
"command": "get",
"serialization": "json|msgpack",
"pv_name": "(pva|ca)://PV_NAME",
"reply_topic": "reply-destination-topic",
"reply_id": "reply id"
}This implements the base camonitor|pvamonitor function of epics client, create a monitor thread into the gateway that send over the destination topic the received values. The field monitor_destination_topic shuld be used to specify the monitor event topic, if not speified the event will be forwarded to the reply topic.
Monitor Activation
{
"command": "monitor",
"serialization": "json|msgpack",
"pv_name": "(pva|ca)://PV_NAME",
"reply_topic": "reply-destination-topic",
"reply_id":"reply id",
"monitor_destination_topic":"alternate-destination-topic"
}Multi Monitor Activation
{
"command": "monitor",
"serialization": "json|msgpack",
"pv_name": ["(pva|ca)://PV_NAME", ....],
"reply_topic": "reply-destination-topic",
"reply_id":"reply id",
"monitor_destination_topic":"alternate-destination-topic"
}The Monitor deactivation is managed by k2eg itself
Put comamdn permit to change the value field of a PV. The value need to be string that will be automatically converted to the type of the PV. The put command can send a reply with the result of the put operation, in this case the dest_topic field si mandatory and the reply_id field is nto mandatory and it is forwarded, into the reply message.
{
"command": "monitor",
"pv_name": "(pva|ca)://PV_NAME.attribute",
"value": "pv new value",
"reply_topic": "reply-destination-topic",
"reply_id":"reply id"
}