Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Normalize loss components #281

Closed
wants to merge 7 commits into from
Closed

Normalize loss components #281

wants to merge 7 commits into from

Conversation

wiederm
Copy link
Member

@wiederm wiederm commented Oct 15, 2024 •
edited
Loading

Pull Request Summary

An easy way to normalize loss components is

$$ L = \sum_i \frac{L_i}{L_i^{\text{detached}}} $$

which uses the componentwise loss and divides it by the detached value, scaling the loss to 1.

# Normalizing losses and combining them
combined_loss = loss_1 / loss_1.detach() + loss_2 / loss_2.detach()

# Backpropagation
combined_loss.backward()

Key changes

Notable points that this PR has either accomplished or will accomplish.

  • add an option to normalize the loss

Associated Issue(s)

Pull Request Checklist

  • Issue(s) raised/addressed and linked
  • Includes appropriate unit test(s)
  • Appropriate docstring(s) added/updated
  • Appropriate .rst doc file(s) added/updated
  • PR is ready for review

@wiederm wiederm self-assigned this Oct 15, 2024
@wiederm wiederm added the enhancement New feature or request label Oct 15, 2024
@wiederm wiederm linked an issue Oct 15, 2024 that may be closed by this pull request
Unsstable multitask learning behavior #271
Open
@codecov-commenter
Copy link

codecov-commenter commented Oct 16, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 79.16667% with 5 lines in your changes missing coverage. Please review.

Project coverage is 85.21%. Comparing base (9b3ed45) to head (7c8fdb0).

Additional details and impacted files

@wiederm
Copy link
Member Author

wiederm commented Oct 17, 2024

@chrisiacovella , even if we don't want to follow up on this kind of normalization, I think it's still helpful to merge this PR. It adds the infrastructure for other loss scaling.

@wiederm wiederm mentioned this pull request Oct 20, 2024
Merged
2 tasks
@wiederm wiederm closed this Oct 21, 2024
auto-merge was automatically disabled October 21, 2024 11:12

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chrisiacovella chrisiacovella Awaiting requested review from chrisiacovella

@MarshallYan MarshallYan Awaiting requested review from MarshallYan

Assignees

@wiederm wiederm

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Unsstable multitask learning behavior
2 participants
@wiederm @codecov-commenter