Allow to disable workflow deadlock detector

[itayd]

Is your feature request related to a problem? Please describe.

We need to run some function that might block for a while (up to 10s, usually) during a Temporal workflow. That function cannot receive a Temporal workflow context. It also has to run every time the workflow runs, including replays. Further operations during this workflow rely on this function being run first.

Currently when we do this, the deadlock detection mechanism panics.

Describe the solution you'd like

Allow to disable the deadlock detection for a specific scope. Possible just making https://github.com/temporalio/sdk-go/blob/master/internal/workflow_deadlock.go#L51 and its friend public.

Describe alternatives you've considered

We tried running the function in a separate go routine, and while it's running sleeping on Temporal. The problem is that this is non-deterministic, and we don't know how much Sleep we need to get.

Additional context

I initially really really really didn't want to open this issue. I really think that the deadlock detection is a Good Thing. I discussed this with @mfateev and Chad (no idea what's his alias here), and at the end came to a conclusion that there's no avoiding this.

@mfateev suggested (ab)using the DataConverterWithoutDeadlockDetection, but I'd rather do it in a more polite fashion.

[itayd]

Just for fun, this is what I can do to abuse the data convertor:

import (
	"context"

	commonpb "go.temporal.io/api/common/v1"
	"go.temporal.io/sdk/workflow"
)

type deadlockDetectorAbuser func()

func (deadlockDetectorAbuser) ToPayload(interface{}) (*commonpb.Payload, error)      { return nil, nil }
func (deadlockDetectorAbuser) FromPayload(*commonpb.Payload, interface{}) error      { return nil }
func (deadlockDetectorAbuser) ToPayloads(...interface{}) (*commonpb.Payloads, error) { return nil, nil }
func (deadlockDetectorAbuser) FromPayloads(*commonpb.Payloads, ...interface{}) error { return nil }
func (deadlockDetectorAbuser) ToString(*commonpb.Payload) string                     { return "" }

func (dd deadlockDetectorAbuser) ToStrings(*commonpb.Payloads) []string {
	dd()
	return nil
}

func WithoutDeadlockDetection(ctx workflow.Context, f func())  {
	cvt := workflow.DataConverterWithoutDeadlockDetection(deadlockDetectorAbuser(f))
	cvt = cvt.(workflow.ContextAware).WithWorkflowContext(ctx)
	_ = cvt.ToStrings(nil)
}

edit: need to supply workflow context to the converter.

[itayd]

tbh, it's so ugly i kinda like it.

[itayd]

Proposed temporalio/sdk-go#1647.