Github connector

[srenault-meeds]

Rationale

Meeds aims to provide to end-users connection to other tools so path can be done to access apps, or to collect actions done in order to gamify it in Meeds software

Currently, it is not fully provided. Only a github connector is proposed but it needs to be studied again for consistency sake
Thus, we propose to firstly review this connector in order to provide other connectors following the same path (UX/UI and Tech)

1. Functional Requirements

1. Admin can manage connectors

• From the administration menu, admin (admin or reward admin) can access connectors page (/gamification/connectors)

• From that page, all default connectors will be proposed. For now, we will find the Github one

• For each default connector type, it is possible to:
  a. Disable it, even default ones

b. Edit characteristics among a range of default ones related to the connector. For Github, we would have 3 items:

• Organization
• Repository
• Triggers

To add a repository, it is mandatory to identify the organization.
To add triggers, there are no relation with other parameters. The following items will be provided by default and admin can decide to enable/disable it if needed. Some are new and need to be studied, as for example: Create PR, Comment PR, Review PR, Validate PR (reviewer), PR validated (sender), Create issue, Comment issue, Close issue

These parameters will be different among the connector to be set. Indeed, for example and for information:
-> "Name": For Github, the name of the organization is meeds-io, for twitter, the name is IoMeeds
-> "Domain": For Github, a repository is a domain, for discord, the category is a domain
-> "Subdomain": For discord, a channel is a subdomain
-> "Triggers": For github, for example, PR request is a trigger (AKA event)

2. Program owners can add action related to connectors

• When adding an action in a program, the program owner can decide to choose an event to trigger with the current UX for now. An improvement will be done once we add other connector: meeds, twitter, for example

• Any action undone will be considered as cancelled in the achievements and thus in the reward computing

3. End-users can set their connector from multiple places

a. From its personal settings, a user will be able to connect his gamified third-app account (github here) thanks to oAuth connection

b. As a user, account can be managed the following way:

• disconnect my account
• only one account can be added to each connector

2. Technical Requirements

Expected Volume & Performance

Terminology

• Trigger: this designates the actions that happens on Meeds Server or on External Software (Github, Twitter, LinkedIn...). An example of trigger is what github sends via WebHook to external system, see the full list here: https://docs.github.com/en/webhooks-and-events/webhooks/webhook-events-and-payloads
• Event: this designates a Trigger interception in Meeds Server and that will result to trigger gamification events internally. In fact, a "Trigger" actionned on external system can result to generate more than one "event" that will be broadcasted to Gamification. The "event" an be configured by Rewarding administrator to define its eligibility, like : restricting permissions on it, designate some specific repositories/organizations.

Thus, the "events" are handled by Rewarding admin and are alterable, but supported triggers by a connector addon is embedded by code and has to be defined through configuration.

Security

• The Oauth Tokens doesn't have to be stored in Meeds database, but only used to validate user account on external provider
• The API key (secret) used by Github to transmit triggered events into Meeds server has to be generated by the rewarding manager using a specific UI in Gamification administration UI inside Github connector settings. This Meeds API Key has to be encrypted using CodecInitializer before storing it in database.
• A Github oAuth Client & Secret keys has to be provided by rewarding admin in a dedicated UI (General Settings Tab, by example). Those oAuth Keys has to be encrypted using CodecInitializer before storing it in database. Those Keys will be used to validate oAuth Token provided by user from Meeds Server to Github Server (Server-To-Server Flow, see schema below).
• The generation of those API Keys has to be made manually by Rewarding Administrator for now (even that we can make a UI to simplify its generation). In case, we want to simplify the generation of those oAuth keys, a "Browser-To-Github" flow has to be used without implying Meeds server in the flow (using oAuth JS client specific library).

Extensibility

• An admin UI has to be provided to manage Gamification connectors and its settings. Each connector has to be able to add specific settings inside this UI.
• The Trigger definitions can be made using configuration by plugin. Each connector will be able to add its own default events definitions that will use those triggers (See above for difference between Trigger & Event). When a connector addon is installed, it should be able to store its events the first time it's installed. When the addon is removed, the events definition has to be disabled/deleted automatically from events proposed events list, and the Trigger definition will be automatically removed as well. (Trigger definitions has to be IMemory and not stored in database, in contrary to events)
• The event title has to be I18N using a Translation Service to let a Program Owner use it when adding actions/rules.

Configurability

• All allowed UI settings for events management, has to be possible to be initialized through configuration. By example, for events instantiation:

  <external-component-plugins>
    <target-component>io.meeds.gamification.service.EventRegistry</target-component>
    <component-plugin>
      <name>EventConfigPlugin</name>
      <set-method>addPlugin</set-method>
      <type>io.meeds.gamification.plugin.EventConfigPlugin</type>
      <init-params>
        <object-param>
          <name>instance</name>
          <object type="io.meeds.gamification.model.EventDTO">
            <field name="trigger">
              <string>joinSpace</string>
            </field>
            <field name="title">
              <string>def_joinSpace</string>
            </field>
            <field name="enabled">
              <boolean>true</boolean>
            </field>
            <!--
              The trigger that cancels this event has to be defined in Trigger definition.
              If set to true, when the `leaveSpace` is triggered, all `joinSpace` actions has
              to be canceled, using the same objectId (=spaceId in this case) and objectType ('space' in this case)
            -->
            <field name="canCancel">
              <boolean>true</boolean>
            </field>
            <field name="permissions">
              <collection type="java.util.ArrayList" item-type="java.lang.String">
                <value>
                  <string>*:/platform/users</string>
                </value>
                <value>
                  <string>*:/platform/externals</string>
                </value>
              </collection>
            </field>
            <field name="settings">
              <map type="java.util.HashMap">
                <entry>
                  <key>
                    <string>A.Specific.Setting.NAME....</string>
                  </key>
                  <value>
                    <string>20</string>
                  </value>
                </entry>
              </map>
            </field>
          </object>
        </object-param>
      </init-params>
    </component-plugin>
  </external-component-plugins>

Upgradability

• The current events definition that is made through hidden rules has to be deleted. The default event definitions (injected by configuration, by each addon) has to replace the existing ones.

Existing Features

• Existing actions and programs has to remain working without any manual change to do by rewarding admins. The upgrade has to be transparent.
• The existing Action drawer has to be improved to help categorizing events list for a program owner who will manage actions

Feature Flags

No feature flag is needed.

4. Software Architecture

Security

• The Github Webhook will have to connect to the REST using an authentication Bearer:

Authorization: Bearer <meeds_api_key>

Note: meeds_api_key: Key generated through UI that has to be personal and non-unique. This key will be encrypted before storing it inside Database, using CodecInitializer.

• oAuth flow to allow user to add a github account has to be as follows :

Warning: The Access Token of user must not be stored in internal database.

• The Github oAuth Keys has to be encypted using CodecInitialize before storing it in database using SettingService with Global context and Github Application Scope.
• The Meeds API Key has to be encypted using CodecInitialize before storing it in database using SettingService with User context and GithubApiKey Application Scope.

REST endpoints

Some REST endpoints has to be added to serve new assets:

• /gamification/triggers : this will allow to retrieve configured triggers switch installed addons. (Readonly listing)
• /gamification/events : this will allow to manage triggers by a rewarding administrator and will allow to a program owner to list the automatic events that it will be able to add in actions/rules.
• /gamification/github/webhook : this endoint will have to replace existing REST endpoint /gamification/connectors/github/webhooks

Access UI

All administration UIs has to be made extensible (injected using extensionRegistry) so that it prepare integration with future connectors.

Services & processing

• The external triggers handeling has to be made asynchronously through a queued executor. In fact, when the WebHook sends events, it has to be queued and then asynchronously processed by gamification by invoking io.meeds.gamification.service.RealizationService.createRealizationsAsync. The queue has to hold the objects sent by WebHook without parsing data, example of code:

• REST layer

    @POST
    @Consumes(MediaType.APPLICATION_JSON)
    public Response trigger(@HeaderParam("x-github-event")
                            String event,
                            @HeaderParam("x-hub-signature")
                            String signature,
                            String payload) {
      gihubConnectorService.addToQueue(event, signature, payload);
    }

• Service layer

    @Override
    public void start() {
      ...
      queueProcessorExecutor.schedule(this::processQueue, githubQueueProcessorPeriodicity, TimeUnit.SECONDS);
    }
  
    @Override
    public void stop() {
      queueProcessorExecutor.shutdown();
    }
  
    ....
  
    public void addToQueue(String event,
                           String signature,
                           String payload) {
      queue.add(new GithubPayload(event, signature, payload));
    }

• A framework has to be developed based on social-component-oauth-auth APIs to make an integration with github for User AccessToken validation (Server-To-Server flow). This framework has to provide REST endpoint & Service layer to process Token and save github/twitter/linkedIn... user name inside internal Meeds User Profile attribute that has to be protected against any modification made through "/users" REST endpoint (even by a super administrator).

Data and persistence

A new Table has to be added in gamification module to allow managing events:

We will need Service + Storage (width a dedicated cache for listing & CRUD) + DAO layers defined for this new entity.