Skip to content

Commit

Permalink
update notebook documentary
Browse files Browse the repository at this point in the history
  • Loading branch information
@poyentung
poyentung committed Sep 21, 2022
1 parent 0674cd4 commit c63c49c
Show file tree
Hide file tree
Showing 2 changed files with 45 additions and 45 deletions.
51 changes: 25 additions & 26 deletions tutorial/tutorial_colab.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,7 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"metadata": {},
"outputs": [],
"source": [
"from google.colab import drive\n",
Expand Down Expand Up @@ -84,7 +82,9 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"metadata": {
"id": "2Wd0omy47rJb"
},
"outputs": [],
"source": [
"!gdown --id '1woNRlyrBbUDIClYp_HNldzA2evdpArsi' -O 'test.bcf'\n",
Expand All @@ -104,9 +104,7 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2Wd0omy47rJb"
},
"metadata": {},
"outputs": [],
"source": [
"from google.colab import files\n",
Expand Down Expand Up @@ -179,11 +177,12 @@
"metadata": {},
"source": [
"### Several (optional) functions to process the dataset:\n",
"1. `sem.rebin_signal(size=(2,2))`: rebin the edx signal with the size of 2x2. After rebinning the dataset, we can access the binned edx or bse data using `sem.edx_bin` or `sem.bse_bin`.\n",
"1. `sem.rebin_signal(size=(2,2))`: rebin the edx signal with the size of 2x2. After rebinning the dataset, we can access the binned edx or bse data using `sem.edx_bin` or `sem.bse_bin`. \n",
"> Note: The size of binning may be changed depending on the number of pixels and signal-to-noise ratio of the EDS spectra. If the input HSI-EDS data contains too many pixels, that may crush the RAM. As for the signal-to-noise, the counts per pixel of the data is better to be higher than 100. In the case of the test dataset, the average counts per pixel is 29.94 before binning and will be 119.76 after binning.\n",
"\n",
"2. `sem.peak_intensity_normalisation()`: normalise the x-ray intensity along energy axis.\n",
"\n",
"3. `sem.remove_fist_peak(end=0.1)`: remove the first x-ray peak (most likely noise) by calling the function with the argument `end`. For example, if one wants to remove the intensity values from 0-0.1 keV, set `end=0.1`.\n",
"3. `sem.remove_fist_peak(end=0.1)`: remove the first x-ray peak (most likely noise) by calling the function with the argument `end`. For example, if one wants to remove the intensity values from 0-0.1 keV, set `end=0.1`. This is an optional step.\n",
"\n",
"4. `visual.plot_intensity_maps`: Plot the elemental intensity maps."
]
Expand Down Expand Up @@ -235,7 +234,11 @@
"source": [
"## Normalisation\n",
"\n",
"Before dimensionality reduction, we normalise the elemental maps use `sem.normalisation()`, where we can pass a list containing (optional) sequential normalisation steps."
"Before dimensionality reduction, we normalise the elemental maps use `sem.normalisation()`, where we can pass a list containing (optional) sequential normalisation steps.\n",
"\n",
"> Note that the pre-processing steps are all optional, but if the user opts for the softmax option, z-score step should be applied beforehand, i.e., the softmax normalization procedure requires input data that is z-score normalized. The purpose of the combinatorial normalization step (z-score + softmax) is to provide the autoencoder with an input that includes global information from all pixels and ‘intentional bias’ within a single pixel. \n",
"\n",
">`neighbour_averaging` is equivilant to apply a 3x3 mean filter to the HSI-EDS data and is an optional step. `zscore` rescales the intensity values within each elemental map such that the mean of all of the values is 0 and the standard deviation is 1 for each map. For example, after z-score normalisation, the Fe Ka map should contain pixels with intensity values that yield `mean=0` and `std=1`. `softmax` is applied within each individual pixel containing z-scores of different elemental peaks. For example, if 5 elemental maps are specified, the 5 corresponding z-scores for each individual pixel will be used to calculate the outputs of softmax."
]
},
{
Expand Down Expand Up @@ -336,13 +339,13 @@
"metadata": {},
"source": [
"**Parameters for `Experiment`**<br>\n",
"> `descriptor`: *str*. The name of the model. It will be used as the model name upon storing the model.<br>\n",
"> `descriptor`: *str*. The name of the model. It will be used as the model name upon saving the model.<br>\n",
"\n",
"> `general_results_dir`: *path*. The folder path to save the model(the model will automatically save in the specified folder).<br>\n",
"\n",
"> `model`: *Autoencoder*. The model to be used for training.<br>\n",
"> `model`: *Autoencoder*. The model to be used for training. At this moment, only the vanilla autoencoder can be used. More models (e.g., variational autoencoder) will be implemented in the future versions.<br>\n",
"\n",
"> `model_args`: *Dict*. Keyword argument for the Autoencoder architecture. The most essential argument is'hidden_layer_sizes' which refers to number of hidden layers and corresponding neurons. For example, if *(512,256,128)*, the encoder will consist of three layers (the first leayer 512 neurons, the second 256 nwurons, and the third 128 neurons). The decoder will also be three layers (the first leayer 128 neurons, the second 256 nwurons, and the third 512 neurons).<br>\n",
"> `model_args`: *Dict*. Keyword argument for the Autoencoder architecture. The most essential argument is'hidden_layer_sizes' which refers to number of hidden layers and corresponding neurons. For example, if *(512,256,128)*, the encoder will consist of three layers (the first leayer 512 neurons, the second 256 nwurons, and the third 128 neurons). The decoder will also be three layers (128 neurons, 256 nwurons, and 512 neurons). The default setting is recommonded in general cases. Increasing the numbers of layers and neurons will increase the complexity of the model, which raise the risk of overfitting.<br>\n",
"\n",
"> `chosen_dataset`: *np.ndarray*. Normalised HSI-EDS data. The size should be (width, height, number of elemental maps).<br>\n",
"\n",
Expand Down Expand Up @@ -391,19 +394,15 @@
"metadata": {},
"source": [
"**Parameters for `ex.run_model`**<br>\n",
"> `num_epochs`: *int*. The number of entire passes through the training dataset.<br>\n",
"\n",
"> `batch_size`: *int*. The number of data points per gradient update.<br>\n",
"\n",
"> `learning_rate`: *float* in a range between 0-1. The learning rate controls how quickly the model is adapted to the problem.<br>\n",
"> `num_epochs`: *int*. The number of entire passes through the training dataset. 50-100 is recommonded for this value. A rule of thumb is that if the loss value stops reducing, that epoch my be a good point to stop. <br>\n",
"\n",
"> `task`: *str*. if 'train_all', all data points will be used for training the autoencoder. If 'train_eval', training will be conducted on the training set (85% dataset) and testing on a testing set (15%) for evaluation.<br>\n",
"> `batch_size`: *int*. The number of data points per gradient update. Values between 32-128 are recommended. smaller batch size means more updates within an epoch, but is more stochastic for the optimisation process.<br>\n",
"\n",
"> `noise_added`: *float* in a range between 0-1. L2 penalty upon optimisation.<br>\n",
"> `learning_rate`: *float* in a range between 0-1. The learning rate controls how quickly the model is adapted to the problem. 1e-4 is recommended. Higher learning rate may yield faster convergence but have a risk to be stuck in an undesirable local minima.<br>\n",
"\n",
"> `criterion`: *str*. IF 'MSE', the criterion is to measure the mean squared error (squared L2 norm) between each element in the input x and target y.<br>\n",
"> `task`: *str*. if 'train_all', all data points will be used for training the autoencoder. If 'train_eval', training will be conducted on the training set (85% dataset) and testing on a testing set (15%) for evaluation. The recommended procedure is to run the 'train_eval' for hyperparameter selection, and 'train_all' for the final analysis.<br>\n",
"\n",
"> `lr_scheduler_args`: *float* in a range between 0-1. L2 penalty upon optimisation.<br>"
"> `criterion`: *str*. If 'MSE', the criterion is to measure the mean squared error (squared L2 norm) between each element in the input x and target y. 'MSE' is the only option. Other criteria will be implemented in the future versions.<br>"
]
},
{
Expand All @@ -416,7 +415,7 @@
"outputs": [],
"source": [
"# Train the model\n",
"ex.run_model(num_epochs=5,\n",
"ex.run_model(num_epochs=50,\n",
" batch_size=64,\n",
" learning_rate=1e-4, \n",
" weight_decay=0.0, \n",
Expand Down Expand Up @@ -474,11 +473,11 @@
"**Parameters for `gui.view_bic`**<br>\n",
"> `latent`: *np.ndarray*. 2D representations learned by the autoencoder. The size of the input array must be (n, 2), where n is number of total data points.<br>\n",
"\n",
"> `model`: *str*. Model for calculation of BIC.<br>\n",
"> `model`: *str*. Model for calculation of BIC. Only 'GaussianMixture' is available for now.<br>\n",
"\n",
"> `n_components`: *int*. If `n_components=20`, it shows the BIC values for GMM using `n_components` from 1 to 20.<br>\n",
"\n",
"> `model_args`: *Dict*. Keyword arguments for the GMM model in sklearn. For example, `random_state` is to specify the random seed for optimisation; `init_params` is to specify the parameter initialisation of the GMM model. See mode detail [here](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html).<br>"
"> `model_args`: *Dict*. Keyword arguments for the GMM model in sklearn. For example, `random_state` is to specify the random seed for optimisation (This can make the results reproduciable); `init_params` is to specify the parameter initialisation of the GMM model ('kmeans' is recommended). See mode detail [here](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html).<br>"
]
},
{
Expand Down Expand Up @@ -513,7 +512,7 @@
"**Parameters for `PixelSegmenter`**<br>\n",
"> `latent`: *np.ndarray*. The size of the input array must be (n, 2), where n is number of total data points.<br>\n",
"\n",
"> `dataset_norm`: *np.ndarray*. Normalised HSI-EDS data. The size should be *(width, height, number of elemental maps)*.<br>\n",
"> `dataset_norm`: *np.ndarray*. Normalised HSI-EDS data. The size should be *(width, height, number of elemental maps)*. <br>\n",
"\n",
"> `sem`: *SEMDataset*. The SEM object created in the beginning.<br>\n",
"\n",
Expand Down
Loading

0 comments on commit c63c49c

Please sign in to comment.