prediction intervals added #1063
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1063 +/- ##
===========================================
- Coverage 88.22% 77.28% -10.95%
===========================================
Files 130 146 +16
Lines 9311 10100 +789
===========================================
- Hits 8215 7806 -409
- Misses 1096 2294 +1198
|
|
Прежде чем буду содержательно смотреть:
Добавить к одному из имеющихся примеров прогнозирования Fedot-ом интервальную оценку. |
|
fedot/confidence_intervals/tuners.py
Outdated
| class Quantile_Loss_low(QualityMetric): | ||
| default_value = sys.maxsize | ||
| @staticmethod | ||
| def metric(reference: InputData, predicted: OutputData) -> float: | ||
| value = quantile_loss(reference.target, predicted.predict,quantile = low_quantile) | ||
| return value | ||
| class Quantile_Loss_up(QualityMetric): | ||
| default_value = sys.maxsize | ||
| @staticmethod | ||
| def metric(reference: InputData, predicted: OutputData) -> float: | ||
| value = quantile_loss(reference.target, predicted.predict,quantile = up_quantile) | ||
| return value |
There was a problem hiding this comment.
Если есть модуль metrics - то можно их туда и отправить.
fedot/confidence_intervals/main.py
Outdated
| from fedot.confidence_intervals.visualization import plot_confidence_intervals | ||
| from fedot.core.repository.tasks import Task,TaskTypesEnum,TsForecastingParams | ||
|
|
||
| class ConfidenceIntervals: |
There was a problem hiding this comment.
Логика класса в целом понятна, но нужно его доработать, причем больше в сторону упрощения. Сейчас он делает очень много всего за раз.
При этом класс жестко реализует только один способ построение интервалов - хотя явно нужна вариативность, чтобы реализовать и другие методы.
Cтруктурно класс и другие связанные с ним стоит положить в fedot/fedot/core/pipelines/ts/confidence_interval, например - не в корень FEDOT-а.
|
Hello @GrigoriJasnovidov! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
Comment last updated at 2023-08-17 15:28:50 UTC |
GrigoriJasnovidov
changed the title
|
Еще по поводу ноутбуков: я не то, чтобы с интервалами много работал, но то, что я видел, обычно визуализируется заливкой в плотлибе. Делается это не очень просто (когда-то сам делал), но так понятнее, чем смотреть на n графиков, которые сливаются в единое целое. Тот вариант, что сейчас точно нужно оставить, но вариант с заливкой кмк приятнее глазу и не нагружает (пример ниже).
И еще в ноутбуках очень много как будто лишнего вывода (нужно долго скроллить) |
|
Перед вливанием ветки причесанные ноутбуки можно будет перенести сюда как один из примеров: https://github.com/ITMO-NSS-team/fedot-examples |
| from fedot.prediction_intervals.solvers.mutation_of_best_pipeline import solver_mutation_of_best_pipeline | ||
|
|
||
|
|
||
| def solver_prediction_intervals(model, |
There was a problem hiding this comment.
Можно ли это сделать через partial? С первого взгляда выглядит, что да
There was a problem hiding this comment.
добавил примеры
test/unit/pipelines/prediction_intervals/test_solver_mutations.py
Outdated
Show resolved
Hide resolved
fedot/core/pipelines/prediction_intervals/solvers/last_generation_quantile_loss.py
Outdated
Show resolved
Hide resolved
test/unit/pipelines/prediction_intervals/test_prediction_intervals.py
Outdated
Show resolved
Hide resolved
test/unit/pipelines/prediction_intervals/test_solver_mutations.py
Outdated
Show resolved
Hide resolved
|
Замечания позакрывал, осталось синхронизировать с мастером. |
deleted redundant files corrected typos simplifeid code remove redundant correct pep8 issues add example add mutations correct visualization fiting process improve visualization add partial in solver.py add example in .py format examples/confidence_intervals move prediction intervals in core/pipelines delete old examples add unit tests Refactoring of ApiParams and ApiMetrics (aimclub#1041) * WIP refactor ApiParams * Remove explicit ApiParams initialization * Move all params initialization to ApiParams * Minor changes * Remove _divide_params * WIP create ApiParamsBuilder * Add ApiParamsBuilder * Minor * Rename history_folder to history_dir * Remove train_data from ApiParams * Fixes after rebase * Move obtain_metric to ApiMetrics * Fix plot_pareto * Fix Fedot.tune * Refactor Fedot.get_metrics * Fix Fedot.tune * Fix metric names * Fixes after rebase * Remove ApiParamsBuilder * Structure parameters in Fedot docstring * Refactor init_composer_requirements * Refactor init_optimizer_params * Refactor init_optimizer_params * Fix docstrings * Add tests for ApiParamsRepository * Minors * Fix test_api_params * Review fixes * Review fixes * Review fixes Hotfix of pipeline import export example (aimclub#1064) meta rules (aimclub#1057) * initial assumption * final architecture * add meta rules * minor * simplify * add meta rule * fix with cv folds * minors * fix types * minor * golem fixes * add log messages * pep8 * remove log file Fix initial assumptions as list of pipelines (aimclub#1070) `Fedot(..., initial_assumption=...)` is expected to get a sequence of pipelines and pass them as initial graphs to an optimizer via composer. Moreover, FEDOT itself generates more than one initial assumption by default. As the result of this bug, composer passed only one of the initial assumptions to an optimizer. This PR fixes the bug and adds the corresponding test. tests fix (aimclub#1073) * remove some assumptions * update requirements * fix pep8 * update to golem Docs updated, badges added (aimclub#1072) * Docs updated, badges added * Mirror workflow fix Improve API documentation (aimclub#1067) - Moved type hints from method headers to corresponding parameters. - Allowed referencing GOLEM objects in FEDOT documentation. - Replaced all url links to documentation pages with sphinx references - it fixed some broken links. - FEDOT now uses its own directory for cache, instead of using GOLEM folder. - Documented `**composer_tuner_params` of `Fedot` with type hints and default values. All parameters with no usage examples are placed to the separate issue aimclub#1076 minor meta fix (aimclub#1078) * minor fix * indent fix * minor Add catboost to default initial assumptions (aimclub#1081) * add catboost to default initial assumptions for classification * restrict mutating loss function in CatBoost * evaluate f1 as expected in the example * pep8 F1 averaging fix (aimclub#1083) Minor logging fix (aimclub#1082) * fix print instead of logging for memory consumption 1059 timestamp bug (aimclub#1065) 879 FEDOT features (aimclub#1075) describe framework's features add example for surrogate optimizer (aimclub#1085) Example with surrogate optimizer was added. External parameters field was removed from api (now we should use partial) related pull request in GOLEM aimclub/GOLEM#82 has_one_root fix (aimclub#1091) * has_one_root fix * test fix Remove outdated test handled in thegolem (aimclub#1101) 358 Reduce execution time for unit tests (aimclub#1098) Update RTD benchmarks tabular data page (aimclub#1099) * +csv support Golem update requirements (aimclub#1088) * Upd RemoveType in Advisor (golem sync) * Upd requirements.txt for stable GOLEM * Fix few imports Add ts bench (aimclub#1104) Add results from ts benchmark Release 0.7.1 and test workflow updates (aimclub#1105) * Upd release version * Add pre-release tests actions on 'release' branch; Disable auto-publish * Upd GOLEM version * Remove manual-build.yml (dup of integration-build.yml) * Add integration tests badge to README * Revert "Upd GOLEM version" (for PR in master) This reverts commit 257ff16. Hotfix some integration tests for release 071 (aimclub#1107) * Fix integration test of ApiParams * Fix integration test of composition_time * Fix integration test of metocean_forecasting * Fix integration test of nemo_multiple.py * Workaround for sqlite exception raised in tests * pep8 fixes * fix different seed in quality imporvement tests * simplify condition * remove test that barely tested anything * fix condition * remove seed from example * fix typo parallel cache files test fix (aimclub#1109) Add IOptTuner (aimclub#1102) * Refactor search space * Fix ParametersChanges * Redact tuner builder * PEP 8 * Fix examples and correct tuning docs * Add IOpt example to docs * Add simple IOpt example * Fix integration tests (6 steal not work) * Fix integration test test_tuner_builder_with_custom_params (6 steal not work) * Fix pep8 * Add tuners comparison in docs * Update golem version in requirements * Fix warn_requirements * Fix warn_requirements * Fix table in tuning docs * Fix credit_scoring_problem_multiobj.py * Fix extra requirements * Edit docs * Fix table in docs * Change requirements * Test requirements * Set stable branch + h2o benchmark's tabular data values (aimclub#1106) * +h2o vals * change csv to html * specified table uuid * change max values style * add df to html converter --------- Co-authored-by: nicl-nno <[email protected]> Added results for multimodal benchmark into FEDOT.docs (aimclub#1115) * - added results for multimodal benchmark ... add docstrings and type hints add prediction intervals unit test simplified solvers code correct pep issues add class defining PredictionIntervals params fix MutationStrength import issue correct test_data path update imports updated unit tests correct type-hint small corrections take short ts for unit test update pred_int test change ql_models to 'max' in pred_ints test update params for ql method ... update example update tests add base_quantiles visualization fix import get_base_quantiles update examples and deleted ql from tests update pep pep issues pep pep update requirements.txt
GrigoriJasnovidov
force-pushed
the
master
branch
from
ab3f401
to
d821935
Compare
|
Replaced with #1149 |
ConfidenceIntervals class for predicting confidence intervals is created.
The instance of ConfidenceInterval class takes pipelines from the last generation, fit these pipelines in order to minimize quantile_loss metric and creates confidence intervals in this way.
ConfidenceIntervals(model, train_input)
Arguments:
model - a fitted fedot model for ts_forecasting.
train_input – a train input used for building the model.
Methods:
.fit(up_quantile, low_quantile,horizon=None,low_tuner=None,up_tuner=None,number_models=10,pipelines_show =
True)
This method outputs several confidence intervals allowing us to build a final confidence interval using method forecast.
Arguments:
up_quantile/low_quantile – quantiles for upper/low confidence intervals.
horizon – length of time horizon for predicting confidence intervals. If None then it is the same as in
model.task.params; if natural number this number of steps is used for forecasting.
number_models – number models used for forecasting confidence intervals. Can be natural number or ‘max’, in
this case all avaliable models used. If number_models > number of avaliable models then all avaliable
models used.
.forecast(regim = ‘quantile’) forecasts confidence intervals given ConfidenceIntervals fitted. Possible values of regime:
‘quantile’,’median’, etc. In each regime intervals obtained after fitting are used to build confidence intervals
.plot() plot the confidence intervals, forecast and (if given) actual test time series