Harmonizing default Mixpanel properties across Rails & ingest (SCP-5798)

[bistline]

BACKGROUND && CHANGES

This update aims to harmonize some common Mixpanel event properties across scp-ingest-pipeline & single_cell_portal_core. These include:

• action: Main process being performed by ingest run, like ingest_cluster or differential_expression
• referenceAnnDataFile: T/F indication if this is a "reference" AnnData upload (no data extracted/visualized)
• trigger: this already existed in scp-ingest-pipeline, but the bucket value was never supported, meaning we did not know from ingest if this was an file that used the new "bucket path" feature

This will give us better insight into common Mixpanel events, especially for AnnData files. We can now see the specific action that failed and pair them with the errorTypes array, which contains more detailed error information than a simple exit code.

MANUAL TESTING

1. Initialize your environment and go to the ingest directory
2. Run the sample command to run an AnnData differential expression job:

python ingest_pipeline.py --study-id addedfeed000000000000000 --study-file-id dec0dedfeed1111111111111 differential_expression --annotation-name louvain --annotation-type group --annotation-scope study --matrix-file-path ../tests/data/anndata/trimmed_compliant_pbmc3K.h5ad --matrix-file-type h5ad --annotation-file ../tests/data/anndata/h5ad_frag.metadata.tsv --cluster-file ../tests/data/anndata/h5ad_frag.cluster.X_umap.tsv --cluster-name umap --study-accession SCPdev --differential-expression

3. In the output from the process, you should action listed now in the properties, which should show up as differential_expression:

STATUS after DE [0]
distinct_id: 2f30ec50-a04d-4d43-8fd1-b136a2045079
studyAccession: SCPdev
fileName: dec0dedfeed1111111111111
fileType: input_validation_bypassed
fileSize: 1
trigger: dev-mode
logger: ingest-pipeline
appId: single-cell-portal
action: differential_expression <= new property
file_path ../tests/data/anndata/h5ad_frag.cluster.X_umap.tsv
study_file_id dec0dedfeed1111111111111

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.66%. Comparing base (91f10c7) to head (941b2fe).
Report is 9 commits behind head on development.

Additional details and impacted files

@@               Coverage Diff               @@
##           development     #366      +/-   ##
===============================================
+ Coverage        75.56%   75.66%   +0.09%     
===============================================
  Files               30       30              
  Lines             4383     4392       +9     
===============================================
+ Hits              3312     3323      +11     
+ Misses            1071     1069       -2     

Files with missing lines	Coverage Δ
ingest/config.py	89.71% <100.00%> (+2.46%)	⬆️
ingest/ingest_pipeline.py	56.86% <100.00%> (+0.42%)	⬆️

[eweitz]

Code looks good! Useful observability refinement.

[eweitz]

Some day we'll get optional chaining in Python!

[jlchang]

This will make the Mixpanel data so much more useful! Thank you.