Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to Calculate Integrated Brier Score (IBS) for a CNN Model with torchsurv? #57

Closed
tadongguk opened this issue Oct 17, 2024 · 2 comments

Comments

@tadongguk
Copy link

I'm currently using the torchsurv library for survival analysis with a CNN model, but the library only supports calculating the C-index and does not provide the Brier score (IBS).

We evaluate the predictive performance of the model using

the [concordance index](https://opensource.nibr.com/torchsurv/_autosummary/torchsurv.metrics.cindex.html) (C-index), which measures the the probability that a model correctly predicts which of two comparable samples will experience an event first based on their estimated risk scores,

the [Area Under the Receiver Operating Characteristic Curve](https://opensource.nibr.com/torchsurv/_autosummary/torchsurv.metrics.auc.html) (AUC), which measures the probability that a model correctly predicts which of two comparable samples will experience an event by time t based on their estimated risk scores.

We cannot use the Brier score because this model is not able to estimate the survival function

What would be the best approach to calculate the Integrated Brier Score (IBS) for a CNN model?

Could I use another library like scikit-survival or lifelines for this, and if so, how could I integrate those with my CNN-based model? Additionally, if using the Weibull model in torchsurv, is it necessary to use the Weibull model to compute IBS, or are there other models I could use?

@melodiemonod
Copy link
Collaborator

Dear @tadongguk,

Thank you for your interest in torchsurv and your question.

1/ To compute the IBS, one needs to be able to estimate the survival function. This is not library-dependent, it is the definition of the metric, see notes here https://opensource.nibr.com/torchsurv/_autosummary/torchsurv.metrics.brier_score.html and relevant reference. The survival function is available for the Weibull Accelerated Time Failure (AFT), but not directly for the Cox Proportional Hazards model.

2/ The NN architecture (in your case a CNN) is defined on the survival model's parameters - the log relative hazard for the cox proportional hazards model, and the Weibull parameters for the Weibull AFT model.
In conclusion, you can use your CNN to estimate the Weibull parameters for the Weibull AFT model, and get an IBS.

@tcoroller
Copy link
Collaborator

Hi @tadongguk!

Thank you for using TorchSurv!

In addition, I would suggest to checkout our tutorial that covers all metrics (including Brier score) and how to use it.

TL;DR:

  1. You need to train a Weibull model, your model will output two (2) values: one for the shape and one for the scale. Your input and architecture in between can be anything, thats the beauty of TorchSurv 🔥
  2. Once trained, compute the survival function for the Weibull log parameters.
  3. Compute the Brier score using the survival function, events and times such as brier_score(surv, event, time)

@tcoroller tcoroller pinned this issue Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants