Add rough version of an autodiff refactor #788
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Completely redo the implementation of the Rop/Lop/grad functions and replace them by
pull_back
,push_forward
.grad
can just callpull_back
then. They are currently not really working, but only a sketch of how they could look like.Add new alternative methods
pull_back
andpush_forward
andlinear_transpose
toOp
. In the long-term they should replace thegrad
,Rop
andLop
methods, and give more control and context to the Op when implementing those operations.Invesigate a bit more how we can derive Rop from Lop and the other way round. We can go both ways: Given an implementation of
push_forward
we can use the fact that the mapping between the tangents is linear, and if all involved linear ops implementlinear_transpose
, we can use that to derivepull_back
. Givenpull_back
we can use the fact that pull_back is a push_forward and transpose. And since the push_forward of a linear function is itself, we can apply it twice to effectively remove the transpose. (see pull_back_through_transpose and push_forward_through_pull_back).We can possibly provide hints to the graph optimizer that the values can sometimes be computed in a smarter way if we also want gradients. Ricardo convinced me that we probably don't want to produce new output values in the
push_forward
andpull_back
functions, but maybe if the gradients are computed lazily we could collect those possiblerewrites and apply them during the rewrite that computes the gradients?
Hopefully delete 2000 lines of complicated gradient code once the new stuff is working well.
cc @ricardoV94