-
Notifications
You must be signed in to change notification settings - Fork 86
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added results for multimodal benchmark into FEDOT.docs (#1115)
* - added results for multimodal benchmark
- Loading branch information
1 parent
2972367
commit cda892d
Showing
4 changed files
with
350 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,3 +8,4 @@ We make a comparison with existing state-of-the-art AutoML libraries on various | |
|
|
||
| forecasting | ||
| tabular | ||
| multimodal | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| Multimodal data | ||
| --------------- | ||
|
|
||
| Here are overall classification and regression problems results on multimodal datasets across state-of-the-art AutoML frameworks | ||
| using `AutoML Multimodal Benchmark <https://github.com/sxjscience/automl_multimodal_benchmark>`__ test suite: | ||
|
|
||
| .. raw:: html | ||
| :file: multimodal_res.html | ||
|
|
||
|
|
||
| The results are obtained using sever based on Xeon Cascadelake (2900MHz) | ||
| with 12 cores and 24GB memory for experiments with the local infrastructure. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| Dataset name,Metric name,AutoGluon,FEDOT | ||
| prod,accuracy,0.895,0.897 | ||
| salary,accuracy,0.528,0.402 | ||
| airbnb,accuracy,0.466,0.434 | ||
| channel,accuracy,0.550,0.542 | ||
| wine,accuracy,0.842,0.784 | ||
| imdb,auc,0.874,0.872 | ||
| fake,auc,0.968,0.958 | ||
| kick,auc,0.775,0.752 | ||
| jigsaw,auc,0.915,0.998 | ||
| qaa,r2,0.383,0.504 | ||
| qaq,r2,0.426,0.554 | ||
| book,r2,0.6,0.994 | ||
| jc,r2,0.612,0.619 | ||
| cloth,r2,0.654,0.733 | ||
| ae,r2,0.979,0.974 | ||
| pop,r2,0.02,0.02 | ||
| house,r2,0.943,0.928 | ||
| mercari,r2,0.569,0.52 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,318 @@ | ||
| <style type="text/css"> | ||
| #T_benchmark table { | ||
| border: 1px solid #e1e4e5; | ||
| text-align: center; | ||
| font-size: .9rem; | ||
| } | ||
| #T_benchmark th { | ||
| border: 1px solid #e1e4e5; | ||
| text-align: center; | ||
| font-size: .9rem; | ||
| } | ||
| #T_benchmark td { | ||
| border: 1px solid #e1e4e5; | ||
| text-align: center; | ||
| font-size: .9rem; | ||
| } | ||
| #T_benchmark th { | ||
| padding: 8px 16px; | ||
| } | ||
| #T_benchmark td { | ||
| padding: 8px 16px; | ||
| } | ||
| #T_benchmark tr { | ||
| background-color: #fff; | ||
| } | ||
| #T_benchmark tbody tr:nth-child(odd) { | ||
| background-color: #f3f6f6; | ||
| } | ||
| #T_benchmarkrow0_col1, #T_benchmarkrow1_col0, #T_benchmarkrow2_col0, #T_benchmarkrow3_col0, #T_benchmarkrow4_col0, #T_benchmarkrow5_col0, #T_benchmarkrow6_col0, #T_benchmarkrow7_col0, #T_benchmarkrow8_col1, #T_benchmarkrow9_col1, #T_benchmarkrow10_col1, #T_benchmarkrow11_col1, #T_benchmarkrow12_col1, #T_benchmarkrow13_col1, #T_benchmarkrow14_col0, #T_benchmarkrow15_col0, #T_benchmarkrow15_col1, #T_benchmarkrow16_col0, #T_benchmarkrow17_col0 { | ||
| color: blue; | ||
| font-weight: bold; | ||
| } | ||
| </style> | ||
| <div style="overflow: auto;"> | ||
| <table id="T_benchmark" style="width: 100%; border-collapse: collapse; font-family: Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;"> | ||
| <thead> | ||
| <tr> | ||
| <th class="blank"> | ||
| </th> | ||
| <th class="index_name level0"> | ||
| framework | ||
| </th> | ||
| <th class="col_heading level0 col0"> | ||
| AutoGluon | ||
| </th> | ||
| <th class="col_heading level0 col1"> | ||
| FEDOT | ||
| </th> | ||
| </tr> | ||
| <tr> | ||
| <th class="index_name level0"> | ||
| Dataset name | ||
| </th> | ||
| <th class="index_name level1"> | ||
| Metric name | ||
| </th> | ||
| <th class="blank col0"> | ||
| </th> | ||
| <th class="blank col1"> | ||
| </th> | ||
| </tr> | ||
| </thead> | ||
| <tbody> | ||
| <tr> | ||
| <th class="row_heading level0 row0" id="T_benchmarklevel0_row0"> | ||
| prod | ||
| </th> | ||
| <th class="row_heading level1 row0" id="T_benchmarklevel1_row0"> | ||
| accuracy | ||
| </th> | ||
| <td class="data row0 col0" id="T_benchmarkrow0_col0"> | ||
| 0.895 | ||
| </td> | ||
| <td class="data row0 col1" id="T_benchmarkrow0_col1"> | ||
| 0.897 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row1" id="T_benchmarklevel0_row1"> | ||
| salary | ||
| </th> | ||
| <th class="row_heading level1 row1" id="T_benchmarklevel1_row1"> | ||
| accuracy | ||
| </th> | ||
| <td class="data row1 col0" id="T_benchmarkrow1_col0"> | ||
| 0.528 | ||
| </td> | ||
| <td class="data row1 col1" id="T_benchmarkrow1_col1"> | ||
| 0.402 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row2" id="T_benchmarklevel0_row2"> | ||
| airbnb | ||
| </th> | ||
| <th class="row_heading level1 row2" id="T_benchmarklevel1_row2"> | ||
| accuracy | ||
| </th> | ||
| <td class="data row2 col0" id="T_benchmarkrow2_col0"> | ||
| 0.466 | ||
| </td> | ||
| <td class="data row2 col1" id="T_benchmarkrow2_col1"> | ||
| 0.434 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row3" id="T_benchmarklevel0_row3"> | ||
| channel | ||
| </th> | ||
| <th class="row_heading level1 row3" id="T_benchmarklevel1_row3"> | ||
| accuracy | ||
| </th> | ||
| <td class="data row3 col0" id="T_benchmarkrow3_col0"> | ||
| 0.550 | ||
| </td> | ||
| <td class="data row3 col1" id="T_benchmarkrow3_col1"> | ||
| 0.542 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row4" id="T_benchmarklevel0_row4"> | ||
| wine | ||
| </th> | ||
| <th class="row_heading level1 row4" id="T_benchmarklevel1_row4"> | ||
| accuracy | ||
| </th> | ||
| <td class="data row4 col0" id="T_benchmarkrow4_col0"> | ||
| 0.842 | ||
| </td> | ||
| <td class="data row4 col1" id="T_benchmarkrow4_col1"> | ||
| 0.784 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row5" id="T_benchmarklevel0_row5"> | ||
| imdb | ||
| </th> | ||
| <th class="row_heading level1 row5" id="T_benchmarklevel1_row5"> | ||
| auc | ||
| </th> | ||
| <td class="data row5 col0" id="T_benchmarkrow5_col0"> | ||
| 0.874 | ||
| </td> | ||
| <td class="data row5 col1" id="T_benchmarkrow5_col1"> | ||
| 0.872 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row6" id="T_benchmarklevel0_row6"> | ||
| fake | ||
| </th> | ||
| <th class="row_heading level1 row6" id="T_benchmarklevel1_row6"> | ||
| auc | ||
| </th> | ||
| <td class="data row6 col0" id="T_benchmarkrow6_col0"> | ||
| 0.968 | ||
| </td> | ||
| <td class="data row6 col1" id="T_benchmarkrow6_col1"> | ||
| 0.958 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row7" id="T_benchmarklevel0_row7"> | ||
| kick | ||
| </th> | ||
| <th class="row_heading level1 row7" id="T_benchmarklevel1_row7"> | ||
| auc | ||
| </th> | ||
| <td class="data row7 col0" id="T_benchmarkrow7_col0"> | ||
| 0.775 | ||
| </td> | ||
| <td class="data row7 col1" id="T_benchmarkrow7_col1"> | ||
| 0.752 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row8" id="T_benchmarklevel0_row8"> | ||
| jigsaw | ||
| </th> | ||
| <th class="row_heading level1 row8" id="T_benchmarklevel1_row8"> | ||
| auc | ||
| </th> | ||
| <td class="data row8 col0" id="T_benchmarkrow8_col0"> | ||
| 0.915 | ||
| </td> | ||
| <td class="data row8 col1" id="T_benchmarkrow8_col1"> | ||
| 0.998 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row9" id="T_benchmarklevel0_row9"> | ||
| qaa | ||
| </th> | ||
| <th class="row_heading level1 row9" id="T_benchmarklevel1_row9"> | ||
| r2 | ||
| </th> | ||
| <td class="data row9 col0" id="T_benchmarkrow9_col0"> | ||
| 0.383 | ||
| </td> | ||
| <td class="data row9 col1" id="T_benchmarkrow9_col1"> | ||
| 0.504 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row10" id="T_benchmarklevel0_row10"> | ||
| qaq | ||
| </th> | ||
| <th class="row_heading level1 row10" id="T_benchmarklevel1_row10"> | ||
| r2 | ||
| </th> | ||
| <td class="data row10 col0" id="T_benchmarkrow10_col0"> | ||
| 0.426 | ||
| </td> | ||
| <td class="data row10 col1" id="T_benchmarkrow10_col1"> | ||
| 0.554 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row11" id="T_benchmarklevel0_row11"> | ||
| book | ||
| </th> | ||
| <th class="row_heading level1 row11" id="T_benchmarklevel1_row11"> | ||
| r2 | ||
| </th> | ||
| <td class="data row11 col0" id="T_benchmarkrow11_col0"> | ||
| 0.600 | ||
| </td> | ||
| <td class="data row11 col1" id="T_benchmarkrow11_col1"> | ||
| 0.994 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row12" id="T_benchmarklevel0_row12"> | ||
| jc | ||
| </th> | ||
| <th class="row_heading level1 row12" id="T_benchmarklevel1_row12"> | ||
| r2 | ||
| </th> | ||
| <td class="data row12 col0" id="T_benchmarkrow12_col0"> | ||
| 0.612 | ||
| </td> | ||
| <td class="data row12 col1" id="T_benchmarkrow12_col1"> | ||
| 0.619 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row13" id="T_benchmarklevel0_row13"> | ||
| cloth | ||
| </th> | ||
| <th class="row_heading level1 row13" id="T_benchmarklevel1_row13"> | ||
| r2 | ||
| </th> | ||
| <td class="data row13 col0" id="T_benchmarkrow13_col0"> | ||
| 0.654 | ||
| </td> | ||
| <td class="data row13 col1" id="T_benchmarkrow13_col1"> | ||
| 0.733 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row14" id="T_benchmarklevel0_row14"> | ||
| ae | ||
| </th> | ||
| <th class="row_heading level1 row14" id="T_benchmarklevel1_row14"> | ||
| r2 | ||
| </th> | ||
| <td class="data row14 col0" id="T_benchmarkrow14_col0"> | ||
| 0.979 | ||
| </td> | ||
| <td class="data row14 col1" id="T_benchmarkrow14_col1"> | ||
| 0.974 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row15" id="T_benchmarklevel0_row15"> | ||
| pop | ||
| </th> | ||
| <th class="row_heading level1 row15" id="T_benchmarklevel1_row15"> | ||
| r2 | ||
| </th> | ||
| <td class="data row15 col0" id="T_benchmarkrow15_col0"> | ||
| 0.020 | ||
| </td> | ||
| <td class="data row15 col1" id="T_benchmarkrow15_col1"> | ||
| 0.020 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row16" id="T_benchmarklevel0_row16"> | ||
| house | ||
| </th> | ||
| <th class="row_heading level1 row16" id="T_benchmarklevel1_row16"> | ||
| r2 | ||
| </th> | ||
| <td class="data row16 col0" id="T_benchmarkrow16_col0"> | ||
| 0.943 | ||
| </td> | ||
| <td class="data row16 col1" id="T_benchmarkrow16_col1"> | ||
| 0.928 | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th class="row_heading level0 row17" id="T_benchmarklevel0_row17"> | ||
| mercari | ||
| </th> | ||
| <th class="row_heading level1 row17" id="T_benchmarklevel1_row17"> | ||
| r2 | ||
| </th> | ||
| <td class="data row17 col0" id="T_benchmarkrow17_col0"> | ||
| 0.569 | ||
| </td> | ||
| <td class="data row17 col1" id="T_benchmarkrow17_col1"> | ||
| 0.520 | ||
| </td> | ||
| </tr> | ||
| </tbody> | ||
| </table> | ||
| </div> |