Normalization Layer: FP32/FP16 should NOT be determined at compile-time

[myungjoo]

Having calculations based on fp16 or fp32 should be determined at run-time based on the model and user's intention, not by the compiler-options.

nntrainer/nntrainer/layers/layer_normalization_layer.cpp

Lines 207 to 225 in 0083caa

	#ifndef ENABLE_FP16
	deviation.pow(2.0f, temp_full_size);
	temp_full_size.average(normalize_axes, variance);
	variance.add_i(epsilon);
	variance.pow(-0.5f, inv_std_dev);
	#else
	unsigned int axis_dim = deviation.getDim()[normalize_axes[0]];
	for (unsigned int i = 0; i < deviation.getDim()[normalize_axes[0] - 1]; ++i) {
	float sum = 0.0;
	_FP16 *data = deviation.getAddress<_FP16>(0, 0, i, 0);
	for (unsigned int j = 0; j < axis_dim; ++j) {
	sum += powf(static_cast<float>(data[j]), 2.0f);
	}
	inv_std_dev.setValue(0, 0, i, 0, 1.0 / sqrt(sum / axis_dim - epsilon));
	}
	#endif

[taos-ci]

cibot: Thank you for posting issue #2408. The person in charge will reply soon.

[jijoongmoon]

People recommand to use FP32 computation for normalization layers like batch normalization, layer normalizaiton or rms normalization, etc. We will check once again.

[Deeksha-20-99]

Hello could you let us know the progress on this issue.. Since we (from ASU) are also working and facing the same issue.

[myungjoo]

Hello could you let us know the progress on this issue.. Since we (from ASU) are also working and facing the same issue.

It's not refactored, yet. You will need to choose it with build-option for this version. I guess we will update these after the next product release.

[myungjoo]

Hello could you let us know the progress on this issue.. Since we (from ASU) are also working and facing the same issue.

#2549 may be interesting to you.