optimise yarahunter

[ibreakthecloud]

• Using string.Join on million of record is really slow. Using Map here were reads and writes are in o(1)
• using fmt writes the error log to stdout, that then goes to json. which we don't want.

// todo

• size of output(result) json can grow depending on stringsMatch, need fix for that
• size of map will increase the in-mem usage, need to fix this

[ibreakthecloud]

This changes were benchmarked by @ramanan-ravi @ibreakthecloud @gnmahanth @noboruma @akumars1 on different setups

Username	Now	Before
@ramanan-ravi	9m30s	> 30m
@ibreakthecloud	30m	~1 day
@noboruma	< 30m	> 30m
@akumars1	14m	infinite
@gnmahanth	5m	> 30m

cc: @shyam-dev

[shyam-dev]

There is a reason golang calls this library unsafe !! Is there a reason we prefer this over string() or fmt.Sprintf ? @noboruma / @ibreakthecloud ??

[ibreakthecloud]

unsafe.String is faster than using string(), since unsafe doesnot actually copies the data but instead creates a new string header that references the same underlying data as the byte slice. The down-side of this is any changes in byte slice will result in affecting the string, which is shouldn't be any trouble in our case.
cc: @noboruma

also I think we should use -d=checkptr flag while building if we're not already

[ibreakthecloud]

one thing I am not sure if garbage collector will free this prematurely. @noboruma

[noboruma]

There are cases where GC can prematurely release the memory, but not in that scenario because the reference we have created is not unsafe anymore, it's a regular string and this will prevent GC from collecting further