Service to Task Config Mapper

[kkasravi]

GearPump Service to Task Config Mapper

Goals

GearPump Tasks often require input parameters described by a UserConfig object that is provided to the Processor[Task]. The Processor API typically takes:

• a parallelism parameter
• a description
• a UserConfig object

The UserConfig object is passed to the specific Task when it is constructed on the server.

Any GearPump application that submits a DAG may require a way to convert a JSON string that describes services to a specific UserConfig that is Task specific. For example, particular service bindings available on a particular target platform that describes HBASE or HDFS related information may need to be converted to a UserConfig format appropriate for a specific type of Task.

1. There should be a way to convert an arbitrary JSON object that holds Task specific information to a UserConfig object appropriate to that Task.
2. The JSON object that holds Task specific information may not be known or may describe configuration information for a new type of Task that is being defined in the GearPump application.
3. These JSON objects will have a type and optionally a name that will uniquely that JSON type.
4. The Service to Task Config Mapper is a utility that takes a type or (type, name) pair and a String holding the JSON data and returns a UserConfig object that can be passed to the Processor[Task].
5. There should be a way to provide or register individual type mappers that knows how to deserialize the JSON data into a UserConfig object.
6. Within the GearPump UI that submits an application there should also be a way to convert JSON data to a UserConfig object for a given Task that can be added to the GearPump manifest that describes the services the application requires. See Application manifest definition and associated functionality #1455.
7. Within the GearPump UI we will need a javascript or scala.js object that implements the conversion described in 6.

Approve (quorum 2+)

• @clockfly
• @kkasravi
• @TomaszT
• @karol-brejna-i

[kkasravi]

This has been moved to gearpump_tap_integration