Merge pull request #155 from bagustris/master

Add svd DB, roc plot, classification report
felixbur · Aug 29, 2024 · a5f9213 · a5f9213
2 parents f41adf9 + d3c2298
commit a5f9213
Show file tree

Hide file tree

Showing 19 changed files with 2,174 additions and 33 deletions.
diff --git a/.gitignore b/.gitignore
@@ -121,6 +121,7 @@ data/*/**
 # Ignore exp folders
 exp_*/
 results/
+results_*
 *_orig/
 *_test/
 *_tests/
@@ -132,10 +133,11 @@ results/
 ./openXBOW/
 ./pretrained_models/
 
-# ignore specific files
+# ignore specific files/folders
 results_*
 result_*
 test_*
+tests_*
 run_*.sh
 tmp*
 test*

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -475,9 +475,9 @@ Version 0.66.3
 
 Version 0.66.2
 --------------
-* enabled data-pathes with quotes 
+* enabled data-pacthes with quotes 
 * enabled missing category labels
-* used tgdm for progress display
+* used tqdm for progress display
 
 Version 0.66.1
 --------------

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -17,7 +17,7 @@ The preferred way to contribute to nkululeko is to fork the [main repository](ht
 	cd nkululeko
 	```
 
-	-	Using HTTPS&#x3A;
+	-	Using HTTPS:
 
 	```bash
 	git clone https://github.com/YourLogin/nkululeko.git
@@ -64,7 +64,7 @@ The preferred way to contribute to nkululeko is to fork the [main repository](ht
 	git push -u origin my-feature
 	```
 
-9.	Finally, go to the web page of the your nkululeko fork repo, and click 'Pull request' button to send your changes to the maintainers to review.
+9.	Finally, go to the web page of your nkululeko fork repo, and click 'Pull request' button to send your changes to the maintainers to review.
 
 Remarks
 -------
@@ -84,11 +84,11 @@ It is recommended to check that your contribution complies with the following ru
 Filing bugs
 -----------
 
-we use Github issues to track all bugs and feature requests. In the case of coming across a bug, having a question or a feature suggestion etc. please feel free to open an issue. 
+We use Github issues to track all bugs and feature requests. In the case of coming across a bug, having a question or a feature suggestion etc. please feel free to open an issue. 
 
 Please check that your issue complies with the following rules before submitting:
 
--	Verify that your issue is not being currently addressed by other in[issues](https://github.com/felixbur/nkululeko/issues) or [pull requests](https://github.com/felixbur/nkululeko/pulls).
+-	Verify that your issue is not being currently addressed by in other [issues](https://github.com/felixbur/nkululeko/issues) or [pull requests](https://github.com/felixbur/nkululeko/pulls).
 
 -	Please ensure all code snippets and error messages are formatted appropriately. See [Creating and highlighting code blocks](https://help.github.com/articles/creating-and-highlighting-code-blocks).
 

diff --git a/README.md b/README.md
@@ -160,7 +160,7 @@ All of them take *--config <my_config.ini>* as an argument.
   * *configurations*: which experiments to combine
   * *--method* (optional): majority_voting, mean (default), max, sum, uncertainty, uncertainty_weighted, confidence_weighted, performance_weighted  
   * *--threshold*: uncertainty threshold (1.0 means no threshold)
-  * *--weightes*: weights for performance_weighted method (could be from previous UAR, ACC)
+  * *--weights*: weights for performance_weighted method (could be from previous UAR, ACC)
   * *--outfile* (optional): name of CSV file for output (default: ensemble_result.csv)
   * *--no_labels* (optional): indicate that no ground truth is given 
 * **nkululeko.multidb**: do [multiple experiments](http://blog.syntheticspeech.de/2024/01/02/nkululeko-compare-several-databases/), comparing several databases cross and in itself
@@ -176,14 +176,11 @@ All of them take *--config <my_config.ini>* as an argument.
 * **nkululeko.predict**: [predict features](http://blog.syntheticspeech.de/2023/08/16/nkululeko-how-to-predict-labels-for-your-data-from-existing-models-and-check-them/) like SNR, MOS, arousal/valence, age/gender, with DNN models
 * **nkululeko.segment**: [segment a database](http://blog.syntheticspeech.de/2023/07/14/nkululeko-segmenting-a-database/) based on VAD (voice activity detection)
 * **nkululeko.resample**: check on all [sampling rates and change](http://blog.syntheticspeech.de/2023/08/31/how-to-fix-different-sampling-rates-in-a-dataset-with-nkululeko/) to 16kHz 
-* **nkululeko.nkuluflag**: a convenient module to specify configuration parameters on the command-line.
-  * usage: nkuluflag.py [-h] [--config CONFIG] [--data [DATA ...]] [--label [LABEL ...]] [--tuning_params [TUNING_PARAMS ...]] [--layers [LAYERS ...]] [--model MODEL] [--feat FEAT] [--set SET]
-                    [--with_os WITH_OS] [--target TARGET] [--epochs EPOCHS] [--runs RUNS] [--learning_rate LEARNING_RATE] [--drop DROP]
-
-
-
-
+* **nkululeko.nkuluflag**: a convenient module to specify configuration parameters on the command-line. Usage:
 
+  ```bash
+  $ python -m nkululeko.nkuluflag.py [-h] [--config CONFIG] [--data [DATA ...]] [--label [LABEL ...]] [--tuning_params [TUNING_PARAMS ...]] [--layers [LAYERS ...]] [--model MODEL] [--feat FEAT] [--set SET] [--with_os WITH_OS] [--target TARGET] [--epochs EPOCHS] [--runs RUNS] [--learning_rate LEARNING_RATE] [--drop DROP]
+  ```
 
 There's my [blog](http://blog.syntheticspeech.de/?s=nkululeko) with tutorials:
 * [Introduction](http://blog.syntheticspeech.de/2021/08/04/machine-learning-experiment-framework/)

diff --git a/data/README.md b/data/README.md
@@ -11,6 +11,8 @@ The column `access` in the table below indicates the database's accessability. T
 - `restricted`: the database is publicly available on the internet but requires registration or other restrictions to download.
 - `private`: the database is not publicly available on the internet and requires the private information of the owner of the dataset.
 
+To support open science and reproducible research, we only accept PR and recipes for public dataset for now on. 
+
 ## Databases
 
 | #   | name               | target                      | access     | descr.                                  |
@@ -57,7 +59,19 @@ The column `access` in the table below indicates the database's accessability. T
 | 40  | vivae              | emotion                     | public     | English vocal bursts                    |
 | 41  | clac               | healthy speech, age, gender | public     | English                                 |
 | 42  | gerparas           | valence, arousal, dominance | restricted | German                                  |
-
+| 43 | SVD      | pathological speech | public | German |
+| 44 | nEMO | emotion, VAD | public | Polish |
+| 45 | emoUERJ	| emotion |  public | Portuguese |
+| 46 | TTH	| emotion |  public | Vietnamese |
+| 47 | jnv	| emotion |  public | Japanese, non-verbals |
+| 48 | jnvv	| emotion |  public | Japanese, verbal and non-verbal|
+| 49 | kbes	| emotion |  public | Bengali |
+| 50 | eaed	| emotion |  public | Arabic |
+| 51 | banglaser | emotion |  public | Bengali |
+| 52 | kia	| emotion |  public | Korean, wake-up word |
+| 53 | polish | emotion |  public | Polish |
+| 54 | erysac | emotion |  public | Russian, children |
+| 55 | turev-db	| emotion |  public | Turkish |
 # Performance 
 <!-- include performance from images directory -->
 ![Nkululeko performance](../meta/images/nkululeko_ser_20240719.png)
diff --git a/data/svd/README.md b/data/svd/README.md
@@ -0,0 +1,22 @@
+# Nkululeko prerocessing for Saarbruecken Voice Database (SVD) dataset
+
+This is Nkululeko pre-processing recipe for the SVD dataset.
+
+# Filenaming convention (A to H correspond to columns in metadata list files)
+
+A: ID, identification number of recording  
+B: T, type of recording (n:normal, p:pathological)  
+C: D, date of recording   
+D: S, identification number of the speaker  
+E: G, gender of the speaker (w:woman, m:man)  
+F: A, age of the speaker at the time of recording  
+G: Pathologies  
+H: Remark with regard to diagnosis  
+
+```bash
+$ python3 process_database.py
+# Download and extract SVD dataset, see exp.ini for configuration
+$ cd ../..
+$ python3 -m nkululeko.nkululeko --config data/svd/exp.ini
+```
+[1] https://stimmdb.coli.uni-saarland.de/index.php4
diff --git a/data/svd/develop_normal.lst b/data/svd/develop_normal.lst
@@ -0,0 +1,68 @@
+8,n,1997/11/20,10,w,19,,
+21,n,1997/11/21,22,w,20,,
+33,n,1997/12/10,33,w,27,,
+46,n,1997/12/18,46,w,28,,
+56,n,1998/1/9,56,w,22,,
+79,n,1998/1/13,81,w,60,,
+93,n,1998/1/16,109,w,42,,
+115,n,1998/1/23,125,w,22,,
+135,n,1998/1/29,137,w,21,,
+683,n,1998/5/14,611,w,20,,
+697,n,1998/5/15,622,w,20,,
+733,n,1998/5/28,638,w,23,,
+747,n,1998/5/29,651,w,26,,
+840,n,1998/6/25,728,w,17,,
+1006,n,1998/10/29,805,w,32,,
+1094,n,1998/11/26,863,w,19,,
+1102,n,1998/11/26,873,w,18,,
+1124,n,1998/12/3,885,w,24,,
+1137,n,1998/12/4,899,w,25,,
+1169,n,1998/12/10,917,w,18,,
+1177,n,1998/12/10,928,w,27,,
+1209,n,1999/1/14,940,w,26,,
+1356,n,1999/5/20,954,w,20,,
+1361,n,1999/5/21,969,w,20,,
+1372,n,1999/5/21,980,w,21,,
+1504,n,1999/11/25,995,w,20,,
+1530,n,1999/11/26,1011,w,19,,
+1543,n,1999/11/26,1024,w,9,,
+1707,n,2000/5/25,1048,w,23,,
+1732,n,2000/6/8,1063,w,21,,
+1844,n,2000/11/23,1077,w,19,,
+1857,n,2000/11/23,1090,w,19,,
+1879,n,2000/12/7,1104,w,23,,
+1922,n,2001/1/18,1119,w,41,,
+2038,n,2001/5/31,1130,w,24,,
+2050,n,2001/5/31,1142,w,22,,
+2167,n,2001/11/22,1152,w,20,,
+2179,n,2001/11/22,1164,w,19,,
+2207,n,2001/12/13,1179,w,19,,
+2256,n,2002/1/25,1191,w,19,,
+2280,n,2002/2/8,1202,w,22,,
+1529,n,1999/11/26,1912,w,22,,
+40,n,1997/12/11,40,m,20,,
+68,n,1998/1/12,71,m,65,,
+100,n,1998/1/19,117,m,66,,
+698,n,1998/5/15,623,m,20,,
+804,n,1998/6/9,708,m,17,,
+813,n,1998/6/9,718,m,18,,
+941,n,1998/10/26,745,m,50,,
+953,n,1998/10/26,757,m,37,,
+986,n,1998/10/27,769,m,32,,
+975,n,1998/10/27,779,m,35,,
+984,n,1998/10/27,790,m,32,,
+1018,n,1998/10/29,801,m,39,,
+1011,n,1998/10/29,812,m,28,,
+1025,n,1998/11/2,824,m,42,,
+1034,n,1998/11/2,835,m,29,,
+1072,n,1998/11/19,851,m,32,,
+1092,n,1998/11/26,861,m,31,,
+1153,n,1998/12/4,904,m,24,,
+1358,n,1999/5/20,956,m,23,,
+1517,n,1999/11/25,1001,m,23,,
+1586,n,2000/1/13,1032,m,25,,
+1709,n,2000/5/25,1050,m,26,,
+1854,n,2000/11/23,1087,m,20,,
+1923,n,2001/1/18,1120,m,27,,
+2262,n,2002/1/25,1197,m,21,,
+2478,n,2003/2/19,2612,m,55,,