RestResource to log results of integrations on parent transaction

[dschibster]

Hi @jongpie,

happy to report that since I presented Nebula Logger at the Salesforce Architects meetup in Düsseldorf, there's hardly a project where I don't see it in use. :)

Right now in one of our projects, we are also using it to log all of our Apex Transactions. However, and this is where things become interesting, we are integrating with MuleSoft. MuleSoft is asynchronously handling the transformation and load into different systems (and Salesforce too). All we get back at the time of sending our payload is a Success.

Of course, we also want to get to know about errors happening in MuleSoft. We were contemplating creating a RestResource for external Systems to give information back to us, for example by creating a new Log (As Parent Log) under the one that triggered the integration itself.

Before we go into further detail creating this ourselves (because architectural guidelines are hestitant about letting us create endpoints), what do you think, would you consider it a worthwile addition to Nebula Logger in general?

BR
Dennis

[jongpie]

Hi @dschibster that's incredible to hear that you're seeing so much usage of Nebula Logger! 🥳

I think the idea of adding a RestResource endpoint is definitely worth considering - I've built & used a few basic prototypes in the past for this, but I never developed the prototypes enough to actually release them here in GitHub. I think that there are 2 valid use cases for having a RestResource class for Nebula Logger:

1. Monitoring of external systems. Your scenario is a perfect example - an external system (Mulesoft, but could be any external system) is doing some off-platform work, but it's relevant to work being done on-platform, so having a way to create & associate logs would provide extra monitoring to Salesforce admins/developers of external systems.
2. Data portability. Right now, Nebula Logger's data is always created - and stays - in a single Salesforce org. But I think there is possible some value in being able to easily move some of Nebula Logger's data to other systems - including from one Salesforce org to another.
   • There have been a few discussions, including this one, about using a standard like OpenTelemetry to send data from a Salesforce org (outbound) to another system, such as an external log analysis tool (like Splunk, DataDog, Beehive, etc.). This wouldn't directly require a RestResource itself, but this is would handle the "outbound" part of improving Nebula Logger's data portability
   • By also adding a RestResource to Nebula Logger (inbound), you could potentially have 1 Salesforce org send its Nebula Logger data to another Salesforce org. This would provide larger/more complex teams with the ability to centralize logging data from multiple Salesforce orgs into a single org (e.g., all logs from a team's scratch orgs and developer sandboxes could have their logs automatically sent to a full-copy sandbox for storage)

I completely agree with being hesitant about introducing new REST endpoints, and I don't think this feature would be needed by everyone using Nebula Logger. But for orgs with a lot of external integrations, multiple environments, and other similar complexities, I think having a RestResource endpoint built into Nebula Logger could be very useful. Depending on how its built, it could be implemented in a way that helps with both monitoring of external systems and data portability.

Let me know what you think!

[jongpie]

@dschibster that sounds incredible, hope you're enjoying your vacation in Japan!

And that sounds good, I'll plan to look into setting up some transformations (using OpenTelemetry's format). I might start sometime in the coming weeks, but I think it'll mostly be exploratory work for now - it might be a few months before I have something fully built out.

[dschibster]

Sounds good! I'll keep you posted. We got an MVP rollout in May so probably thereafter I can shift focus.

[jongpie]

Hi @dschibster - I wanted to check in with you on this feature to see if you've had a chance to make some progress. I've had discussions with a few people over the last few weeks that also need a REST resource for their projects, so I'm considering spending some time building a prototype for this in the coming weeks. Let me know if that works for you!

[dschibster]

Hi @jongpie, So to be fair, the rollout got delayed to july, doing the deed in roughly 1,5 weeks. We have put this topic back in our prioritization.

More interesting to us was listening for platform events in another logging solution, basically summarizing logs the other way round. I do still think that if we have a RestResource we would be able to do something with it, but I'm currently not able to divert time from my own work schedule to work on that. Maybe I can support though, I'd be interested in giving a prototype to our MuleSoft people to have a look how it works out!

[jongpie]

@dschibster no worries & I completely understand if you can't divert any time from your schedule, I'm happy to work on it. I'll let you know once I have a prototype/PR ready, I'd love to get any feedback you & your team want to provide.

And regarding using an external logging solution to store/summarize your data, I've heard of a few other companies/teams using a similar approach, though I haven't really used that setup myself. If you think of any enhancements for Nebula Logger that you think could help with that approach, let me know!

[anengineerdude]

Came across this discussion for a very similar use case. I am implementing several integrations to SF via Mulesoft and heavily debating where Mulesoft logging should lie. While there is still merit in something like OpenTelemetry for mass ingestion of Mule logs, having some consise logging for scheduled integrations and simplistic error handling at a high level that would should up in NebulaLogger would be a great UX for the SF Admins.

A RestResource certainly seems like a great way to doing this. Looking forward to it and happy to test out any ideas. While I am getting good with SF admin and Mule, my package development and Apex skill is still pretty basic, so I am not to much help on the development side.

[dschibster]

Thanks for the feedback! I still have this on my radar, the release timeline that I have to focus on currently though has been a little delayed. When things get softer, I should be able to handle this and get a PR running for @jongpie to review.