Migrate Notifications to the new API

.github/workflows/deploy.yml

@@ -26,6 +26,10 @@ on:
         type: string
         required: false
         default: ""
+      site-url:
+        description: The URL of the site to deploy
+        type: string
+        required: true
 
 jobs:
   netlify-deploy:
@@ -119,6 +123,7 @@ jobs:
           REALM_API_PRIVATE_KEY: ${{ secrets.REALM_API_PRIVATE_KEY }}
           ROLLBAR_POST_SERVER_ITEM_ACCESS_TOKEN: ${{ secrets.GATSBY_ROLLBAR_TOKEN }}
           API_MONGODB_CONNECTION_STRING: ${{ secrets.API_MONGODB_CONNECTION_STRING }}
+          SITE_URL: ${{ inputs.site-url }}
 
       - name: Build size
         run: |

.github/workflows/preview.yml

@@ -47,7 +47,7 @@ jobs:
     with:
       sha: ${{ github.event.pull_request.head.sha }}
       environment: staging
-      netlify-context: deploy-preview
+      netlify-context: branch-deploy
       runner-label: ${{ vars.PREVIEW_RUNNER_LABEL }}
 
   call-test:
@@ -90,6 +90,7 @@ jobs:
     with:
       sha: ${{ github.event.pull_request.head.sha }}
       environment: staging
-      netlify-context: deploy-preview
+      netlify-context: branch-deploy
       netlify-alias: pr-${{ github.event.pull_request.number }}
       runner-label: ${{ vars.PREVIEW_RUNNER_LABEL }}
+      site-url: https://pr-${{ github.event.pull_request.number }}--staging-aiid.netlify.app

.github/workflows/production.yml

@@ -95,3 +95,4 @@ jobs:
       netlify-alias:
       runner-label: ${{ vars.PRODUCTION_RUNNER_LABEL }}
       cache-modifier: ${{ needs.cache-modifier.outputs.cache-modifier  }}
+      site-url: ${{ vars.SITE_URL }}

.github/workflows/staging.yml

@@ -94,3 +94,4 @@ jobs:
       netlify-alias:
       runner-label: ${{ vars.STAGING_RUNNER_LABEL }}
       cache-modifier: ${{ needs.cache-modifier.outputs.cache-modifier  }}
+      site-url: ${{ vars.SITE_URL }}

.github/workflows/test-playwright-full.yml

@@ -138,7 +138,7 @@ jobs:
           NOTIFICATIONS_SENDER_NAME: Test Preview
           NOTIFICATIONS_SENDER: [email protected]
           PROCESS_NOTIFICATIONS_SECRET: sarasa
-          SITE_URL: https://incidentdatabase.test
+          SITE_URL: http://localhost:8000
 
       - name: Upload Playwright traces
         if: failure()

.github/workflows/test-playwright.yml

@@ -93,7 +93,7 @@ jobs:
           NOTIFICATIONS_SENDER_NAME: Test Preview
           NOTIFICATIONS_SENDER: [email protected]
           PROCESS_NOTIFICATIONS_SECRET: sarasa 
-          SITE_URL: https://incidentdatabase.test
+          SITE_URL: http://localhost:8000
 
       - uses: actions/upload-artifact@v4
         if: ${{ !cancelled() }}

README.md

@@ -823,121 +823,6 @@ On Facebook Authentication settings, set the "Client ID" with the Facebook App I
 
 About Facebook Authentication instructions: https://www.mongodb.com/docs/realm/web/authenticate/#facebook-authentication
 
-## Email subscriptions
-
-### Subscription types
-
-- **All**: This subscription type is not defined yet.
-    ```
-    {
-        "userId": "63320ce63ec803072c9f529c"
-        "type": "all",
-    }
-    ```
-- **Incident**: Users with this subscription type will be notified when the incident associated is updated. This subscription type needs an incident_id value associated.
-    ```
-    {
-        "userId": "63320ce63ec803072c9f529c"
-        "type": "incident",
-        "incident_id": 10,
-    }
-    ```
-- **New Incident**: Users with this subscription type will be notified when a new Incident is created. The notification will be sent after finish the next site build when the Incident page is actually created.
-    ```
-    {
-        "userId": "63320ce63ec803072c9f529c"
-        "type": "new-incidents",
-    }
-    ```
-- **Entities**: Users can subscribe to a specific Entity. The user with this subscription type will be notified when a new Incident associated with an specific Entity is created or when an existing Incident is updated to be associated with that Entity.
-    ```
-    {
-        "userId": "63320ce63ec803072c9f529c",
-        "type": "entity"
-        "entityId": "openai",
-    }
-    ```
-- **Submission Promoted**: Users that submit a new Incident Report are automatically subscribed to its promotion. Once the submission has been approved by an editor, the user will recieve an email informing that the submission is now an incident/issue/report.
-    ```
-    {
-        "userId": "63320ce63ec803072c9f529c",
-        "type": "submission-promoted"
-    }
-    ```
-
-These subscription types are also documented in [subscriptions.js](site/gatsby-site/src/utils/subscriptions.js) file.
-
-### Email notifications
-
-[Sendgrid](https://sendgrid.com/) is used to send email notifications.
-
-To allow email notifications add these secret values to your Atlas App Service following the instructions in the [Atlas App Services documentation](https://www.mongodb.com/docs/atlas/app-services/values-and-secrets/define-and-manage-secrets/).
-
-To disable all email noticications, fill the following `SendGridApiKey` secret value with an empty string. If this secret value is an empty string, the emails will be ignored.
-
-```
-SendGridApiKey = [SendGrid API key from https://app.sendgrid.com/settings/api_keys]
-publicApiKey = [Public API key from the Atlas Organization. See comment below for more information]
-privateApiKey = [Private API key from the Atlas Organization. See comment below for more information]
-groupId = [Atlas Service App group ID, eg: "62cc90978bc4600cafdcf16e"]
-appId = [Atlas Service App ID, eg: "62cc98647e6a26c53d5b4b53"]
-```
-
-To get your Public and Private API Key, follow these [instructions](https://www.mongodb.com/docs/atlas/configure-api-access/#std-label-create-org-api-key).
-
-To get the group ID and the app ID, the easiest way is to navigate to your Atlas Service App dashboard and copy from the URL.
-The URL format is https://services.cloud.mongodb.com/groups/[groupId]/apps/[appId]/dashboard
-
-Email notifications to New Incidents (subscription type **New Incident**), Incident updates (subscription type **Incident**) and Submission Promoted (subscription type **Submission Promoted**) are sent when the next build finishes. This is because we have to wait until the new Incident page is generated and accessible.
-When a new Incident is created or updates, a pending notification item is saved into the `notifications` DB collection with `processed=false` field.
-And finally, as part of the site build process, we processed all pending notifications (`processed=false`), send the emails to all recipients, and update the items with `processed=true` and `sentDate=[now]`.
-
-### Notifications collection definition
-
-- **Incident Updated**
-    ```
-    {
-        "type": "incident-updated",
-        "incident_id": 374,
-        "processed": false
-    }
-    ```
-- **New Incident Report**
-    ```
-    {
-        "type": "new-report-incident",
-        "incident_id": 374,
-        "report_number": 2172,
-        "processed": false
-    }
-    ```
-- **New Incident**
-    ```
-    {
-        "type": 'new-incidents',
-        "incident_id": incidentId,
-        "processed": false,
-    }
-    ```
-- **Entities**
-    ```
-    {
-        "type": "entity",
-        "incident_id": 374,
-        "entity_id": "openai",
-        "isUpdate": true,
-        "processed": false
-    }
-    ```
-- **Submission Promoted**
-    ```
-    {
-        "type": "submission-promoted",
-        "incident_id": 374,
-        "processed": false
-    }
-    ```
-
 ### Error logging
 
 This project uses [Rollbar](https://rollbar.com) for error logging for the whole site, including background processes.

site/gatsby-site/.gitignore

@@ -13,4 +13,4 @@ node_modules
 /blob-report/
 /playwright/.cache/
 
-/netlify/functions/lookupIndex.json
+/netlify/functions/lookupIndex.json

site/gatsby-site/README.md

@@ -1,7 +1,6 @@
+## Setting Up a Local Development Environment
 
-### Setting Up a Local Development Environment
-
-To set up a local development environment for the AIID project, follow these steps:
+Once you have cloned the repository, to set up a local development environment for the AIID project, follow these steps:
 
 1. **Navigate to the Gatsby Site Directory**
 
@@ -48,6 +47,10 @@ To set up a local development environment for the AIID project, follow these ste
    CLOUDFLARE_R2_SECRET_ACCESS_KEY=  # Secret access key for Cloudflare R2 storage
    CLOUDFLARE_R2_BUCKET_NAME=  # Name of the Cloudflare R2 bucket for storage
    GATSBY_CLOUDFLARE_R2_PUBLIC_BUCKET_URL=  # Public URL for accessing the Cloudflare R2 bucket from the Gatsby app
+   MAILERSEND_API_KEY= # API key for MailerSend email service or dummy value if you don't plan to send emails
+   NOTIFICATIONS_SENDER_NAME=AIID Notifications # Name of the sender for email notifications
+   [email protected] # Email address of the sender for email notifications
+   SITE_URL=http://localhost:8000
    ```
 
    Ensure that each variable is set correctly to match your development environment's requirements.
@@ -69,3 +72,180 @@ To set up a local development environment for the AIID project, follow these ste
    ```
 
 Follow these steps to get your local environment up and running for development with the AIID project. Make sure to replace the placeholder values in the `.env` file with your actual credentials to ensure proper functionality.
+
+## AIID API 
+
+### Overview
+
+The AIID API is built to facilitate interactions with the AI Incident Database. It is implemented as a collection of serverless functions that are composed ("stitched") into a singular GraphQL endpoint.
+
+1. **Access the Apollo Explorer**
+
+   Navigate to `http://localhost:8000/graphql` in your web browser. The Apollo Explorer instance should be displayed, allowing you to introspect and run queries against the API.
+
+### Performing Queries
+
+Here is an example query you can run in the Apollo Explorer to retrieve incidents:
+
+```graphql
+query {
+  incident {
+    incident_id
+    title
+  }
+}
+```
+
+### Expected Response
+
+The query should return a response similar to this:
+
+```json
+{
+  "data": {
+    "incident": {
+      "incident_id": 1,
+      "title": "Incident 1"
+    }
+  }
+}
+```
+
+### Project Structure
+
+#### Folders
+
+The API is contained within the `server` directory. The following folders are present in the project:
+
+- **fields/**: Contains the field definitions for the GraphQL root fields.
+- **generated/**: Holds the generated GraphQL types derived from the schema using the GraphQL code generator CLI.
+- **tests/**: Contains the test cases written using Jest.
+- **types/**: Contains the base GraphQL types.
+
+#### Important Files
+
+- **`remote.ts`**: Handles the auto-generated MongoDB Atlas schema, ignoring fields that have migrated to the local GraphQL schema.  
+- **`local.ts`**: Handles the local GraphQL schema, where migrated fields from the remote schema are added. These fields are ignored in `remote.ts`.  
+- **`schema.ts`**: Combines the remote and local schemas into the final schema using **schema stitching** from GraphQL Tools.
+- **`netlify/functions/graphql.ts`**: Sets up the **GraphQL server** and exposes it as a **Netlify function**, loading the schema from `schema.ts`.
+- 
+### Running Tests
+
+To run Jest tests locally:
+
+```sh
+npm run test:api
+```
+
+It is recommended to install the Jest extension for VS Code to enhance the testing experience.
+
+### Running Code Generation
+
+To run the GraphQL code generation CLI to generate TypeScript typings from the schema, allowing statically typed Apollo client mutations and queries, and statically typed server code:
+
+```sh
+npm run codegen
+```
+
+### Schema and API Stitching
+
+> [!IMPORTANT]
+The API previously relied on the MongoDB Atlas Realm GraphQL API to fetch data, [but now that it has been deprecated](https://www.mongodb.com/developer/products/atlas/deprecating-mongodb-atlas-graphql-hosting-services/), we have migrated to a new API that we implement in this codebase. During the migration from Realm to our API, we stitch two GraphQL schemas: the auto-generated one from Atlas and the one defined in this codebase. These schemas can be found in the `remote.ts` and `local.ts` files respectively. The migration process involves progressively transferring GraphQL fields and functionality from the remote schema to the local one. The final combined schema is found in `schema.ts`.
+
+### Email notifications
+
+#### Subscription types
+
+- **All**: This subscription type is not defined yet.
+    ```
+    {
+        "userId": "63320ce63ec803072c9f529c"
+        "type": "all",
+    }
+    ```
+- **Incident**: Users with this subscription type will be notified when the incident associated is updated. This subscription type needs an incident_id value associated.
+    ```
+    {
+        "userId": "63320ce63ec803072c9f529c"
+        "type": "incident",
+        "incident_id": 10,
+    }
+    ```
+- **New Incident**: Users with this subscription type will be notified when a new Incident is created. The notification will be sent after finish the next site build when the Incident page is actually created.
+    ```
+    {
+        "userId": "63320ce63ec803072c9f529c"
+        "type": "new-incidents",
+    }
+    ```
+- **Entities**: Users can subscribe to a specific Entity. The user with this subscription type will be notified when a new Incident associated with an specific Entity is created or when an existing Incident is updated to be associated with that Entity.
+    ```
+    {
+        "userId": "63320ce63ec803072c9f529c",
+        "type": "entity"
+        "entityId": "openai",
+    }
+    ```
+- **Submission Promoted**: Users that submit a new Incident Report are automatically subscribed to its promotion. Once the submission has been approved by an editor, the user will receive an email informing that the submission is now an incident/issue/report.
+    ```
+    {
+        "userId": "63320ce63ec803072c9f529c",
+        "type": "submission-promoted"
+    }
+    ```
+
+These subscription types are also documented in [subscriptions.js](site/gatsby-site/src/utils/subscriptions.js) file.
+
+#### Sending Email Notifications
+
+[MailerSend](https://www.mailersend.com/) is used to send email notifications.
+
+Email notifications to New Incidents (subscription type **New Incident**), Incident updates (subscription type **Incident**) and Submission Promoted (subscription type **Submission Promoted**) are sent when the next build finishes. This is because we have to wait until the new Incident page is generated and accessible.
+When a new Incident is created or updates, a pending notification item is saved into the `notifications` DB collection with `processed=false` field.
+And finally, as part of the site build process, we processed all pending notifications (`processed=false`), send the emails to all recipients, and update the items with `processed=true` and `sentDate=[now]`.
+
+#### Notifications collection definition
+
+- **Incident Updated**
+    ```
+    {
+        "type": "incident-updated",
+        "incident_id": 374,
+        "processed": false
+    }
+    ```
+- **New Incident Report**
+    ```
+    {
+        "type": "new-report-incident",
+        "incident_id": 374,
+        "report_number": 2172,
+        "processed": false
+    }
+    ```
+- **New Incident**
+    ```
+    {
+        "type": "new-incidents",
+        "incident_id": 374,
+        "processed": false
+    }
+    ```
+- **Entities**
+    ```
+    {
+        "type": "entity",
+        "incident_id": 374,
+        "entity_id": "openai",
+        "isUpdate": true,
+        "processed": false
+    }
+    ```
+- **Submission Promoted**
+    ```
+    {
+        "type": "submission-promoted",
+        "incident_id": 374,
+        "processed": false
+    }
+    ```