fix: correctly stream process json data normalization to utf8 for bot…

[ddlees]

…h metatag validation and writing to disk

Description

Describe your changes in detail

• correctly stream process json data normalization to utf8
• feed normalized data to metatag validator
• feed normalized data to disk writer

Motivation and Context

This PR addresses: [GitHub issue or Jira ticket number]
BED-4463

Why is this change required? What problem does it solve?

The initial pass at normalizing json data that is prefixed with a byte order mark (BOM) was causing the ValidateMetaTag check to incorrectly fail when preparing to write the data to disk and when it was being read from disk for graph ingest. Additionally, we were reading in the whole payload into an uncompressed []byte which is problematic for very large files. This PR addresses those issues by correcting data normalization via stream processing and feeding the data stream to the validator and disk writer.

How Has This Been Tested?

Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc.

Updated unit tests to reflect necessary interface changes.

Screenshots (optional):

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• I have met the contributing prerequisites
  • Assigned myself to this PR
  • Added the appropriate labels
  • Associated an issue: Issues and Pull Requests README #672
  • Read the Contributing guide: https://github.com/SpecterOps/BloodHound/wiki/Contributing
• I have ensured that related documentation is up-to-date
  • Open API docs
  • Code comments (GoDocs / JSDocs)
• I have followed proper test practices
  • Added/updated tests to cover my changes
  • All new and existing tests passed

[urangel]

This looks to work for uploads of JSON files 👍

It may be out of scope for this work but I believe it is still possible to zip JSON files with BOM's and upload the zip into BH's UI. Since there is no normalization done of the files within the zip, data that is uploaded this way will encounter a similar BOM issue and an error will display that the data is not valid JSON.

[StephenHinck]

This looks to work for uploads of JSON files 👍

It may be out of scope for this work but I believe it is still possible to zip JSON files with BOM's and upload the zip into BH's UI. Since there is no normalization done of the files within the zip, data that is uploaded this way will encounter a similar BOM issue and an error will display that the data is not valid JSON.

This is correct - we should also expect to strip encoding from zipped JSON files.

[ddlees]

This looks to work for uploads of JSON files 👍
It may be out of scope for this work but I believe it is still possible to zip JSON files with BOM's and upload the zip into BH's UI. Since there is no normalization done of the files within the zip, data that is uploaded this way will encounter a similar BOM issue and an error will display that the data is not valid JSON.

This is correct - we should also expect to strip encoding from zipped JSON files.

I looked into what it would take to include normalizing decompressed json files and we would have to do some non-trivial work to accommodate the ReadSeeker interface that we expect when processing the file. SharpHound zips will not include BOM encoded json files but we shouldn't lose sight of this issue because I'm sure someone will bom encode some json files and zip them and try to upload that data to BloodHound at some point. Will attempt to deliver a solution in a separate PR.