Skip to content

Protobuf ‐ ROS 2 interoperability

Jump to bottom
Michel Hidalgo edited this page Jan 29, 2024 · 5 revisions

When exposing Protobuf interfaces to ROS 2, proto2ros streamlines message translation and generation, so as to trivially augment rosidl pipeline invocations to take Protobuf definitions and output ROS 2 message definitions. Additionally, conversion APIs are generated to simplify bridging Protobuf <-> ROS 2 equivalences.

Table of contents

Features

Message generation

Protobuf enumeration and message definitions are translated to equivalent ROS 2 message definitions. Protobuf definitions, including comments, are extracted from Protobuf descriptor sets, as generated by protoc. More than one ROS 2 message definition may be necessary to represent a given Protobuf definition, as explained in the map types and one-of fields subsections.

Package mapping

All Protobuf packages to which processed definitions belong are implicitly mapped to the ROS 2 package that will host their ROS 2 message equivalents. As for the rest, the user may specify a package mapping in proto2ros configuration. Whether that is necessary or not depends on how message types are mapped.

Type mapping

Scalar types

The mapping between Protobuf and ROS 2 scalar types is shown below.

Protobuf scalar type name ROS 2 scalar type name
bool bool
double float64
fixed32 uint32
fixed64 uint64
float float32
int32 int32
int64 int64
sfixed32 int32
sfixed64 int64
sint32 int32
sint64 int64
uint32 uint32
uint64 uint64
string string
bytes uint8[]

Note that, unlike all others, the Protobuf bytes scalar type maps to a sequence type in ROS 2. This is convenient but forces special handling of repeated bytes fields.

Messages types

Every Protobuf message maps to a ROS 2 message. For statically typed message (as opposed to dynamically typed google.protobuf.Any messages), this mapping is derived from the sequential application of the following rules, on first match wins basis:

  • A user-defined message mapping in proto2ros configuration matches. Fully qualified message names are used verbatim.
  • A user-defined or implicit package mapping in proto2ros configuration matches. If multiple matches are found, the longest match applies. Fully qualified message names are camel-cased.
  • Passing through unknown Protobuf messages is allowed in proto2ros configuration. Then proto2ros/AnyProto is used.

For google.protobuf.Any messages, the rules for any types apply.

It is through message mappings that ad-hoc equivalences can be established e.g. some Protobuf message may be equivalent to some standard ROS 2 message (and this is already the case for several core Protobuf messages, see default proto2ros configuration). It is through package mappings that equivalences generated by proto2ros in full, for a given ROS 2 package or one of its dependencies, interact with each other.

For example, given the following configuration overlay:

message_mapping:
  third_party.data.Text: std_msgs/String
  google.protobuf.Any: custom_msgs/Any
package_mapping:
  third_party.data: data_msgs
  third_party.data.legacy: data_legacy_msgs

google.protobuf.Any would map to proto2ros/Any, third_party.data.Text would map to std_msgs/String, third_party.data.Blob would map to data_msgs/Blob, third_party.data.legacy.Image would map to data_legacy_msgs/Image, and some_package.Data would map to proto2ros/AnyProto if passthrough_unknown is enabled, raising an error otherwise.

Enum types

As ROS 2 messages lack the notion of enumerated types entirely, a ROS 2 message is generated for each Protobuf enumeration. This ROS 2 message defines a homonymous integer constant for each enum value and a single value integer field to bear it. A sample equivalence is shown below.

Protobuf .proto definition ROS 2 .msg definition 
# some.proto

enum Status {
  STATUS_UNKNOWN = 0;
  STATUS_OK = 1;
  STATUS_FAILURE = 2;
}
 
# Status.msg

int32 STATUS_UNKNOWN=0
int32 STATUS_OK=1
int32 STATUS_FAILURE=2

int32 value
Map types

Over the wire, Protobuf map types are bound to be equivalent to a sequence of key-value pairs (or map entry messages). For ROS 2, the exact same convention is observed. A ROS 2 message is thus generated for each map entry message. A sample equivalence is shown below.

Protobuf .proto definition Protobuf equivalent syntax ROS 2 .msg definition 
# some.proto

message Device {
  map<string, string> attributes = 1;
}
 
# some.proto

message Device {
  message AttributesEntry {
    string key = 1;
    string value = 2;
  }

  repeated AttributesEntry attributes = 1;
}
 
# DeviceAttributesEntry.msg

string key
string value
 
# Device.msg

<ros_package_name>/DeviceAttributesEntry[] attributes
Any types

Dynamically typed (i.e. google.protobuf.Any) Protobuf messages are mapped to ROS 2 messages as specified by any expansions. An any expansion is a type set $T$ that fully characterizes what to expect of a google.protobuf.Any message field. These are indexed after Protobuf message name and field name. Cardinality $|T|$ always satisfies $|T| &gt; 0$.

Given an applicable any expansion is found:

  • if $|T| == 1$ and allow_any_casts is enabled, the corresponding equivalent ROS 2 message type for that sole Protobuf message type, as dictated by message type mapping rules, will be used (as if that statically typed Protobuf message type had been found in place of google.protobuf.Any);
  • otherwise, a dynamically typed (i.e. proto2ros/Any) ROS 2 message is used as the equivalent ROS 2 message type, which will bear the equivalent ROS 2 type set (with $|T| &gt; 0$) as dictated by message type mapping rules.

Else, a proto2ros/AnyProto ROS 2 message type is used as the equivalent ROS 2 message type, bearing the unmodified, serialized Protobuf message.

For example, given the following configuration overlay:

any_expansions:
  third_party.data.Storage.params: third_party.data.StorageParams
  third_party.data.StorageParams.implementation_specific: [third_party.data.S3Params, third_party.data.PGParams]
allow_any_casts: true

google.protobuf.Any for the params field in the third_party.data.Storage Protobuf message would map to the ROS 2 equivalent, as per message type mapping rules, of the third_party.data.StorageParams Protobuf message, whereas google.protobuf.Any for the implementation_specific field in the third_party.data.StorageParams Protobuf message would map to proto2ros/Any. Conversion APIs, however, can use the information provided by these any expansions to perform the necessary casting in runtime.

Recursive types

Protobuf supports recursive message definitions but ROS 2 does not. To workaround this limitation, a message dependency graph reflecting the composition relationships between known messages is built and analyzed for cycles. Once a cycle has been identified, it is broken by the weakest link (i.e. the minimal change set) using proto2ros/Any, functionally type erasing one or more fields.

Field mapping

Optional fields

Optional fields in Protobuf messages, and fields with explicit presence tracking in general, are conventionally implemented using a bit mask field in ROS 2 messages. As ROS 2 messages lack the notion of optional fields entirely, an unsigned integer has_field field explicitly conveys which message fields bear meaningful information. For each optional field f, an unsigned integer constant F_FIELD_SET bit mask is defined. Bitwise binary operations can then be used to explicitly indicate and check for field presence. A sample equivalence is shown below.

Protobuf .proto definition ROS 2 .msg definition 
# some.proto

message Option {
  optional string value = 1;
}
 
# Option.msg

uint8 VALUE_FIELD_SET=1

string value

uint8 has_field 255

Note that, to match ROS 2 message semantics, the bit mask is fully set by default. That is, all fields are assumed to be present by default.

Implementation note: bit masks can be 8, 16, 32, or 64 bit long, depending on the number of optional fields. Protobuf messages with more than 64 optional fields are therefore not supported.

Repeated fields

Repeated fields in Protobuf messages are mapped to array fields in ROS 2 messages. This applies to all field types except to bytes fields. This exception is necessary as scalar bytes fields are already mapped to array fields in ROS 2. In this case, scalar type mapping rules are overridden and repeated bytes fields are mapped to array fields of proto2ros/Bytes ROS 2 message type. A sample equivalence is shown below.

Protobuf .proto definition ROS 2 .msg definition 
# some.proto

message Payload {
  repeated int32 keys = 1;
  repeated bytes blobs = 2;
  bytes checksum = 3;
}
 
# Payload.msg

int32[] keys
proto2ros/Bytes[] blobs
uint8[] checksum
One-of fields

As ROS 2 messages lack the notion of one-of fields entirely, a ROS 2 message is generated for each one-of construct in a Protobuf message, bearing all one-of fields, as well as an integer which field. This ROS 2 message is functionally equivalent to a tagged union. For each field f in the one-of construct o, an integer constant O_F_SET tag is defined. Assigning the which field to a given tag thus conveys presence of the corresponding field. In place for each one-of construct, a message field of the corresponding type is defined. A sample equivalence is shown below.

Protobuf .proto definition ROS 2 .msg definition 
# some.proto

message Timestamp {
  oneof value {
    uint64 seconds_since_epoch = 1;
    string datestring = 2; 
  }
}
 
# Timestamp.msg

<ros_package_name>/TimestampOneOfValue value
 
# TimestampSecondsSinceEpoch.msg

uint64 seconds_since_epoch
 
# TimestampDatestring.msg

string datestring
 
# TimestampOneOfValue.msg

int8 VALUE_NOT_SET=0
int8 VALUE_SECOND_SINCE_EPOCH_SET=1
int8 VALUE_DATESTRING_SET=2

<ros_package_name>/TimestampSecondsSinceEpoch seconds_since_epoch
<ros_package_name>/TimestampDatestring datestring
int8 value_choice  # deprecated
int8 which

Implementation note: 8 bit tags are used for one-of constructs. Protobuf messages with more than 256 one-of fields are therefore not supported.

Deprecated fields

Deprecated fields are kept, unless drop_deprecated is enabled. If kept, these fields are annotated with a comment in the corresponding ROS 2 message definition. A sample equivalence is shown below.

Protobuf .proto definition ROS 2 .msg definition (drop_deprecated disabled) ROS 2 .msg definition (drop_deprecated enabled) 
# some.proto

message Duration {
  int64 seconds = 1;
  int64 nanosec = 2 [deprecated = true];
  int64 nanoseconds = 3;
}
 
# Duration.msg

int64 seconds
int64 nanosec  # deprecated
int64 nanoseconds
 
# Duration.msg

int64 seconds
int64 nanoseconds
Reserved fields

Reserved fields are ignored.

Protobuf .proto definition ROS 2 .msg definition 
# some.proto

message Goal {
  string location = 1;
  reserved "time_budget";
}
 
# Goal.msg

string location

Code generation

To simplify conversion from Protobuf messages to equivalent ROS 2 messages and back, proto2ros generates conversion code, nicely wrapped around convert(from, to) function overloads (i.e. type dispatched). Note, however, that conversion code is only generated for message equivalences that proto2ros itself generated in full. For ad-hoc equivalences, as specified using message mappings, the user must implement the corresponding overloads. For auxiliary messages underpinning enums, map types, one-of fields, and the like, no overloads are generated at all (as there is no Protobuf message to convert to/from).

Python APIs

Conversion APIs are exposed on a per Python package basis, as {ros_package_name}.conversions.convert. While convenient, the mechanisms that enable these overloads do not play along with static analyzers such as mypy. To workaround this limitation, each overload is also made available, fully type annotated, under a unique name. This name is derived from argument type names as follows:

  • convert_{ros_package_name}_{ros_message_name}_message_to_{proto_package_name}_{proto_message_name}_proto for ROS 2 message to Protobuf message conversion API overloads
  • convert_{proto_package_name}_{proto_message_name}_proto_to_{ros_package_name}_{ros_message_name}_message for Protobuf message to ROS 2 message conversion API overloads

All message names above are snake-cased. Note that user-defined overloads for ad-hoc equivalences must follow this naming pattern.

Implementation note: all explicit and implicit _pb2 (i.e. Protobuf) Python imports must be available at generation time. This requirement allows proto2ros to cope with an omission in Protobuf descriptor sets: these do not specify the mapping between fully qualified Protobuf message names and their Python counterparts. To workaround this limitation, known _pb2 modules are traversed to reconstruct this mapping.

Configuration

Both message and code generation are configured by a number of settings, listed below.

Name Description Default value
drop_deprecated Whether to drop deprecated fields on conversion or not. If not dropped, deprecated fields are annotated with a comment. False
passthrough_unknown Whether to forward Protobuf messages for which no equivalent ROS message is known as a serialized binary blob in a proto2ros/AnyProto field or not. True
message_mapping A mapping from fully qualified Protobuf message names to fully qualified ROS message names. This mapping comes first during composite type translation. {google.protobuf.Any: proto2ros/AnyProto, google.protobuf.Timestamp: builtin_interfaces/Time, google.protobuf.Duration: builtin_interfaces/Duration, google.protobuf.DoubleValue: std_msgs/Float64, google.protobuf.FloatValue: std_msgs/Float32, google.protobuf.Int64Value: std_msgs/Int64, google.protobuf.UInt64Value: std_msgs/UInt64, google.protobuf.Int32Value: std_msgs/Int32, google.protobuf.UInt32Value: std_msgs/UInt32, google.protobuf.BoolValue: std_msgs/Bool, google.protobuf.StringValue: std_msgs/String, google.protobuf.BytesValue: proto2ros/Bytes, google.protobuf.ListValue: proto2ros/List, google.protobuf.Value: proto2ros/Value, google.protobuf.Struct: proto2ros/Struct}
package_mapping A mapping from Protobuf package names to ROS package names, to tell where a ROS equivalent for a Protobuf construct will be found. Note that no checks for package existence are performed. This mapping comes second during composite type translation (i.e. when direct message mapping fails). {}
any_expansions A mapping from fully qualified Protobuf field names (i.e. a fully qualified Protobuf message name followed by a dot "." followed by the field name) of google.protobuf.Any type to Protobuf message type sets that these fields are expected to pack. A single Protobuf message type may also be specified in lieu of a single element set. All Protobuf message types must be fully qualified. {}
allow_any_casts When a single Protobuf message type is specified in an any expansion, allowing any casts means to allow using the equivalent ROS message type instead of a dynamically typed, proto2ros/Any field. For further reference on any expansions, see Any types section below. True
known_message_specifications A mapping from ROS message names to known message specifications. Necessary to cascade message generation for interdependent packages. {}
python_imports Set of Python modules to be imported (as import <module-name>) in generated conversion modules. Typically, Protobuf and ROS message Python modules. [std_msgs.msg, proto2ros.msg, builtin_interfaces.msg, google.protobuf.any_pb2, google.protobuf.duration_pb2, google.protobuf.struct_pb2, google.protobuf.timestamp_pb2, google.protobuf.wrappers_pb2]
inline_python_imports Set of Python modules to be imported into module scope (as from <module-name> import *) in generated conversion modules. Typically, conversion Python modules. [proto2ros.conversions.basic]
skip_implicit_imports Whether to skip importing Python modules for known Protobuf and ROS packages in generated conversion modules or not. These known modules are those derived from .proto source file names and the one homonymous to the ROS 2 package that hosts the generated interfaces. False

These defaults can be replaced entirely via configuration file or overridden one by one via configuration overlays. Configuration overlays are configuration files that update the baseline configuration, default or user-defined. Scalar values are replaced, lists are extended, dictionaries are updated (i.e. shallow merged).

Use cases

Dual Protobuf / ROS 2 package

A package may provide both Protobuf and ROS 2 messages, all generated from Protobuf definitions.

cmake_minimum_required(VERSION 3.12)
project(proto2ros_tests)

find_package(ament_cmake REQUIRED)
find_package(builtin_interfaces REQUIRED)
find_package(rosidl_default_generators REQUIRED)
find_package(proto2ros REQUIRED)

find_package(Protobuf REQUIRED)

# Generate Python code for some.proto
protobuf_generate(
  LANGUAGE python
  OUT_VAR proto_py_sources
  PROTOS some.proto
  IMPORT_DIRS proto
)
# Add dependable target for generated _pb2 Python code
add_custom_target(
  ${PROJECT_NAME}_proto_gen ALL
  DEPENDS ${proto_py_sources}
)

# Generate equivalent ROS 2 messages and conversion Python code
proto2ros_generate(
  ${PROJECT_NAME}_messages_gen
  PROTOS proto/some.proto
  INTERFACES_OUT_VAR ros_messages
  PYTHON_OUT_VAR ros_py_sources
  APPEND_PYTHONPATH "${PROTO_OUT_DIR}"
)
# Make it depend on generated _pb2 Python code (needed at configure time)
add_dependencies(
  ${PROJECT_NAME}_messages_gen 
  ${PROJECT_NAME}_proto_gen
)

# Generate ROS 2 message code.
rosidl_generate_interfaces(
  ${PROJECT_NAME} ${ros_messages}
  DEPENDENCIES builtin_interfaces proto2ros
)
add_dependencies(${PROJECT_NAME} ${PROJECT_NAME}_messages_gen)

# Add generated Python _pb2 and conversion code to the
# Python package implicitly defined and installed by the 
# rosidl pipeline
rosidl_generated_python_package_add(
  ${PROJECT_NAME}_additional_modules
  MODULES ${proto_py_sources} ${ros_py_sources}
  PACKAGES ${PROJECT_NAME}
  DESTINATION ${PROJECT_NAME}
)

ament_package()

proto2ros_tests is a good example of this.

ROS 2 vendored Protobuf messages

Protobuf messages may already be provided by some third-party package, in which case, it is only the equivalent ROS 2 messages that are relevant. For a third-party package and .proto files that are hosted on public repositories, the FetchContent module and the proto2ros_vendor_package CMake macro fully address this use case:

cmake_minimum_required(VERSION 3.8)
project(vendored_third_party)

find_package(ament_cmake REQUIRED)
find_package(proto2ros REQUIRED)

# Fetch third party package sources (incl. .proto files)
include(FetchContent)
FetchContent_Declare(
  third_party
  GIT_REPOSITORY ...
  GIT_TAG ..._
)
FetchContent_Populate(third_party)

# Collect third party .proto files
set(${PROJECT_NAME}_PROTO_DIR "${third_party_SOURCE_DIR}/protos")
file(GLOB ${PROJECT_NAME}_PROTOS "${${PROJECT_NAME}_PROTO_DIR}/*.proto")

# Generate ROS 2 messages and code (wraps rosidl)
proto2ros_vendor_package(${PROJECT_NAME}
  PROTOS ${${PROJECT_NAME}_PROTOS}
  IMPORT_DIRS ${${PROJECT_NAME}_PROTO_DIR}
)

ament_package()

bosdyn_msgs is a good example of this.

Clone this wiki locally