Merge pull request #206 from ThibaudReal/feature/extras_requirements

Feature/extras requirements

.github/workflows/main.yml

@@ -8,7 +8,7 @@ jobs:
     strategy:
       max-parallel: 1
       matrix:
-        python-version: [3.6, 3.7, 3.8]
+        python-version: [3.6, 3.7, 3.8, 3.9]
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}

README.md

@@ -30,8 +30,13 @@
   </a>
 </p>
 
+## 🎉 What's new ?
+
+| Version       | New Feature                                                                           | Description                                                                                                                            | Tutorial |
+|:-------------:|:-------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------:|:--------:|
+| 1.4.x         |  Groups of features <br> [demo]()                                                     | You can now regroup features that share common properties together. <br>This option can be useful if your model has a lot of features. |  [<img src="https://raw.githubusercontent.com/MAIF/shapash/master/docs/_static/groups_features.gif" width="120" title="groups-features">](https://github.com/MAIF/shapash/blob/master/tutorial/common/tuto-common01-groups_of_features.ipynb)    | 
+| 1.3.x         |  Shapash Report <br> [demo](https://shapash.readthedocs.io/en/latest/report.html)     | A standalone HTML report that constitutes a basis of an audit document.                                                                |  [<img src="https://raw.githubusercontent.com/MAIF/shapash/master/docs/_static/report-icon.png" width="8%" title="shapash-report">](https://github.com/MAIF/shapash/blob/master/tutorial/report/tuto-shapash-report01.ipynb)    | 
 
-🎉 **We just released Shapash 1.3.2 that includes the generation of a standalone HTML report that constitutes a basis of an audit document. [See an example here](https://shapash.readthedocs.io/en/latest/report.html) that was generated [using this tutorial.](https://github.com/MAIF/shapash/blob/master/tutorial/report/tuto-shapash-report01.ipynb)** 🎉  
 
 ## 🔍 Overview
 
@@ -44,18 +49,22 @@ Shapash also contributes to data science auditing by displaying usefull informat
 
 - Readthedocs: [![documentation badge](https://readthedocs.org/projects/shapash/badge/?version=latest)](https://shapash.readthedocs.io/en/latest/)
 - [Presentation video for french speakers](https://www.youtube.com/watch?v=r1R_A9B9apk)
-- Medium:    
-<a href="https://www.kdnuggets.com/2021/04/shapash-machine-learning-models-understandable.html">
-  <p align="center">
-    <img align="left" src="https://www.kdnuggets.com/images/tkb-2104-g.png?raw=true" width="6%" />
-  </p>
-</a> 
+- Medium:
+  - [Understand your model with Shapash - Towards AI](https://pub.towardsai.net/shapash-making-ml-models-understandable-by-everyone-8f96ad469eb3) 
+  - [Model auditability - Towards DS](https://towardsdatascience.com/shapash-1-3-2-announcing-new-features-for-more-auditable-ai-64a6db71c919)
+
+## 🏆 Awards
+
+<a href="https://raw.githubusercontent.com/MAIF/shapash/master/docs/_static/awards-argus-or.png">
+  <img align="left" src="https://raw.githubusercontent.com/MAIF/shapash/master/docs/_static/awards-argus-or.png" width="130" />
+</a>
 
+<a href="https://www.kdnuggets.com/2021/04/shapash-machine-learning-models-understandable.html">
+  <img src="https://www.kdnuggets.com/images/tkb-2104-g.png?raw=true" width="50" />
+</a>  
 
-- [Understand your model with Shapash - Towards AI](https://pub.towardsai.net/shapash-making-ml-models-understandable-by-everyone-8f96ad469eb3) 
-- [Model auditability - Towards DS](https://towardsdatascience.com/shapash-1-3-2-announcing-new-features-for-more-auditable-ai-64a6db71c919)
 
-## 🔥  Features
+## 🔥 Features
 
 - Display clear and understandable results: plots and outputs use **explicit labels** for each feature and its values
 
@@ -66,9 +75,9 @@ Shapash also contributes to data science auditing by displaying usefull informat
 </p>
 
 <p align="center">
-  <img align="left" src="https://github.com/MAIF/shapash/blob/master/docs/_static/shapash-grid-images-07.png?raw=true" width="28%" />
+  <img align="left" src="https://github.com/MAIF/shapash/blob/master/docs/_static/shapash-grid-images-01.png?raw=true" width="28%" />
   <img src="https://github.com/MAIF/shapash/blob/master/docs/_static/shapash-resize.png?raw=true" width="18%" />
-  <img align="right" src="https://github.com/MAIF/shapash/blob/master/docs/_static/shapash-grid-images-01.png?raw=true" width="28%" /> 
+  <img align="right" src="https://github.com/MAIF/shapash/blob/master/docs/_static/shapash-grid-images-09.png?raw=true" width="28%" /> 
 </p>
 
 <p align="center">
@@ -117,12 +126,20 @@ to compute contributions.
 
 ## 🛠 Installation
 
-Shapash is intended to work with Python versions 3.6 to 3.8. Installation can be done with pip:
+Shapash is intended to work with Python versions 3.6 to 3.9. Installation can be done with pip:
 
 ```
 pip install shapash
 ```
 
+In order to generate the Shapash Report some extra requirements are needed.
+You can install these using the following command :  
+```
+pip install shapash[report]
+```
+
+If you encounter **compatibility issues** you may check the corresponding section in the Shapash documentation [here](https://shapash.readthedocs.io/en/latest/installation-instructions/index.html).
+
 ## 🕐 Quickstart
 
 The 4 steps to display results:
@@ -195,6 +212,7 @@ This github repository offers a lot of tutorials to allow you to start more conc
 - [Launch the webapp with a concrete use case](tutorial/tutorial01-Shapash-Overview-Launch-WebApp.ipynb)
 - [Jupyter Overviews - The main outputs and methods available with the SmartExplainer object](tutorial/tutorial02-Shapash-overview-in-Jupyter.ipynb)
 - [Shapash in production: From model training to deployment (API or Batch Mode)](tutorial/tutorial03-Shapash-overview-model-in-production.ipynb)
+- [Use groups of features](tutorial/common/tuto-common01-groups_of_features.ipynb)
 
 ### More details about charts and plots
 - [**Shapash** Features Importance](tutorial/plot/tuto-plot03-features-importance.ipynb)

docs/installation-instructions/index.rst

@@ -4,12 +4,20 @@ Installation instructions
 Installing
 ----------
 
-Users can install **Shapash** with pip:
+**Shapash** is intended to work with Python versions 3.6 to 3.9. Installation can be done with pip:
 
 .. code:: bash
 
     pip install shapash
 
+
+In order to generate the **Shapash Report** you may need to install specific libraries.
+You can install these using the following command :
+
+.. code:: bash
+
+    pip install shapash[report]
+
 Jupyter
 -------
 
@@ -213,3 +221,24 @@ To test if plotly works:
 
 .. image:: plotly.png
 
+
+Compatibility issues
+--------------------
+
+When using Shapash, you may encounter some compatibility issues related to your environment and the libraries' versions used.
+The extras requirements of Shapash allow you to update your requirements to a compatible version.
+
+For example, if you get an error related to the *xgboost* library, you can use the following command to update it to a working version :
+
+.. code:: bash
+
+    pip install shapash[xgboost]
+
+The full list of extras requirements is listed below (replace xgboost with the corresponding library on the command above) :
+
+* xgboost
+* lightgbm
+* catboost
+* scikit-learn
+* category_encoders
+

requirements.dev.txt

@@ -1,6 +1,6 @@
 pip==20.2.4
 dash==1.17.0
-catboost==0.22
+catboost>=0.22
 category-encoders==2.1.0
 dash-bootstrap-components==0.9.1
 dash-core-components==1.13.0
@@ -23,12 +23,14 @@ sphinxcontrib-qthelp==1.0.3
 sphinxcontrib-serializinghtml==1.1.4
 pytest==5.2.3
 pytest-cov==2.8.1
-scikit-learn==0.23.2
+scikit-learn>=0.23.2
 xgboost==1.0.0
 nbformat>4.2.0
-numba==0.51.2
+numba==0.53.1
 nbconvert==6.0.7
-papermill==2.3.2
-matplotlib==3.3.4
-seaborn==0.11.1
-jupyter==1.0.0
+papermill
+matplotlib
+seaborn
+notebook
+Jinja2
+phik

setup.py

@@ -12,7 +12,7 @@
         'plotly==4.12.0',
         'numpy>1.18.0',
         'pandas>1.0.2',
-        'shap==0.37.0',
+        'shap>=0.36.0',
         'dash==1.17.0',
         'dash-bootstrap-components==0.9.1',
         'dash-core-components==1.13.0',
@@ -21,21 +21,37 @@
         'dash-renderer==1.8.3',
         'dash-table==4.11.0',
         'nbformat>4.2.0',
-        'numba==0.51.2',
-        'nbconvert==6.0.7',
-        'papermill==2.3.2',
-        'matplotlib>=3.0',
-        'seaborn==0.11.1'
+        'numba==0.53.1',
+        'scikit-learn'
     ]
 
+extras = dict()
+
+# This list should be identical to the list in shapash/report/__init__.py
+extras['report'] = [
+    'nbconvert==6.0.7',
+    'papermill',
+    'matplotlib',
+    'seaborn',
+    'notebook',
+    'Jinja2',
+    'phik'
+]
+
+extras['xgboost'] = ['xgboost>=1.0.0']
+extras['lightgbm'] = ['lightgbm>=2.3.0']
+extras['catboost'] = ['catboost>=0.21']
+extras['scikit-learn'] = ['scikit-learn>=0.23.0']
+extras['category_encoders'] = ['category_encoders==2.2.2']
+
 setup_requirements = ['pytest-runner', ]
 
 test_requirements = ['pytest', ]
 
 setup(
-    name="shapash", # Replace with your own username
+    name="shapash",
     version="1.3.2",
-    python_requires='>3.5, < 3.9',
+    python_requires='>3.5, <=3.9',
     url='https://github.com/MAIF/shapash',
     author="Yann Golhen, Sebastien Bidault, Yann Lagre, Maxime Gendre",
     author_email="[email protected]",
@@ -51,6 +67,7 @@
         "Operating System :: OS Independent",
     ],
     install_requires=requirements,
+    extras_require=extras,
     license="Apache Software License 2.0",
     keywords='shapash',
     package_dir={

shapash/explainer/smart_explainer.py

@@ -17,8 +17,8 @@
 from shapash.utils.check import check_model, check_label_dict, check_ypred, check_contribution_object,\
     check_postprocessing, check_features_name
 from shapash.manipulation.select_lines import keep_right_contributions
+from shapash.report import check_report_requirements
 from shapash.manipulation.summarize import create_grouped_features_values
-from shapash.report.generation import execute_report, export_and_save_report
 from .smart_state import SmartState
 from .multi_decorator import MultiDecorator
 from .smart_plotter import SmartPlotter
@@ -1179,6 +1179,10 @@ def generate_report(self,
                 ]
             )
         """
+        check_report_requirements()
+        # Avoid Import Errors with requirements specific to the Shapash Report
+        from shapash.report.generation import execute_report, export_and_save_report
+
         rm_working_dir = False
         if not working_dir:
             working_dir = tempfile.mkdtemp()

shapash/report/__init__.py

@@ -0,0 +1,25 @@
+import importlib
+
+# This list should be identical to the list in setup.py
+report_requirements = [
+    'nbconvert==6.0.7',
+    'papermill',
+    'matplotlib',
+    'seaborn',
+    'notebook',
+    'Jinja2'
+]
+
+
+def check_report_requirements():
+    """
+    Checks that all required packages for the report are installed.
+    This function should be called before executing the report.
+    """
+    for req in report_requirements:
+        pkg = req.split('=')[0]
+        try:
+            importlib.import_module(pkg.lower())
+        except (ModuleNotFoundError, ImportError):
+            raise ModuleNotFoundError(f"The following package is necessary to generate the Shapash Report : {pkg}. "
+                                      f"Try 'pip install shapash[report]' to install all required packages.")

shapash/utils/io.py

@@ -2,7 +2,12 @@
 IO module
 """
 import pickle
-import yaml
+try:
+    import yaml
+    _is_yaml_available = True
+except (ImportError, ModuleNotFoundError):
+    _is_yaml_available = False
+
 
 def save_pickle(obj, path, protocol=pickle.HIGHEST_PROTOCOL):
     """
@@ -72,6 +77,9 @@ def load_yml(path):
     d : dict
         Python dict containing the parsed yml file.
     """
+    if _is_yaml_available is False:
+        raise ModuleNotFoundError('Please install PyYAML using "pip install pyyaml" command.')
+
     if not isinstance(path, str):
         raise ValueError(
             """

tests/unit_tests/explainer/test_smart_explainer.py

@@ -11,6 +11,7 @@
 import numpy as np
 import catboost as cb
 from sklearn.linear_model import LinearRegression
+from sklearn.ensemble import RandomForestClassifier
 from shapash.explainer.smart_explainer import SmartExplainer
 from shapash.explainer.multi_decorator import MultiDecorator
 from shapash.explainer.smart_state import SmartState
@@ -1364,12 +1365,12 @@ def test_to_smartpredictor_1(self):
 
     def test_get_interaction_values_1(self):
         df = pd.DataFrame({
-            "y": np.random.randint(2, size=50),
-            "a": np.random.rand(50),
-            "b": np.random.rand(50),
+            "y": np.random.randint(2, size=10),
+            "a": np.random.rand(10)*10,
+            "b": np.random.rand(10),
         })
 
-        clf = cb.CatBoostClassifier(n_estimators=1).fit(df[['a', 'b']], df['y'])
+        clf = RandomForestClassifier(n_estimators=5).fit(df[['a', 'b']], df['y'])
 
         xpl = SmartExplainer()
         xpl.compile(x=df.drop('y', axis=1), model=clf)
@@ -1400,8 +1401,8 @@ def test_run_app_1(self, mock_get_host_name, mock_custom_thread, mock_smartapp):
 
         assert xpl.y_pred is not None
 
-    @patch('shapash.explainer.smart_explainer.export_and_save_report')
-    @patch('shapash.explainer.smart_explainer.execute_report')
+    @patch('shapash.report.generation.export_and_save_report')
+    @patch('shapash.report.generation.execute_report')
     def test_generate_report(self, mock_execute_report, mock_export_and_save_report):
         """
         Test generate report method

tests/unit_tests/utils/test_load_smartpredictor.py

@@ -30,6 +30,10 @@ def test_load_smartpredictor_1(self):
             pkl_file = path.join(current, 'data/predictor_to_load_36.pkl')
         elif str(sys.version)[0:3] == '3.8':
             pkl_file = path.join(current, 'data/predictor_to_load_38.pkl')
+        elif str(sys.version)[0:3] == '3.9':
+            pkl_file = path.join(current, 'data/predictor_to_load_39.pkl')
+        else:
+            raise NotImplementedError
 
         predictor2 = load_smartpredictor(pkl_file)