[ Tensor ] more types of Tensors are required.

[EunjuYang]

Our awesome TensorV2 has been successfully activated.
As of September 12th, 2024, we support the following types of tensors:

FloatTensor : fp32,
HalfTensor : fp16,
CharTensor : qint8,
ShortTensor : uint16.

For further expansion of our on-device training capabilities, we also need to incorporate tensors using unsigned 32 bit integers (uint32) and unsigned 8 bit integers (uint8).

[taos-ci]

cibot: Thank you for posting issue #2733. The person in charge will reply soon.

[skykongkong8]

Thanks for pointing this issue out!

On top of that, I believe we should discuss format of quantization parameters, along with which quantization method are we going to use.

For example, quantization parameters like zero point and scale factor are NYI as far as I know.
They might be given as a scalar value, or channel-wise vector, or even row / col - wise matrix (in 4-D tensor ).

[EunjuYang]

Great point!
We may consider quantization scheme as well [ref]. This REF considers different quantization parameters according to the quantization scheme; whether simple float / int or lit of float, list of int and axis.

[skykongkong8]

I was planning to bring up this issue after implementing a draft version of qint-using gemm algorithm, but here's a temporal one that I am using :

template <typename T> struct ScalarIOQuantInfo {
...
// input quantization parameters
  T activation_zero_point;
  float activation_scale_factor;

// input information to compute time-of-flight (dynamic) quantization during GEMM
  T activation_min_val;
  T activation_max_val;

// output information to compute static quantization during GEMM
  T output_zero_point;
  float output_scale_factor;

// actual quantization parameter to quantize into u8 GEMM output
  T resultant_zero_point;
  float resultant_scale_factor;
...
};

template <typename T> struct VectorIOQuantInfo {
...
/*
 Information required from the kernel side:
0. list of T or float, obviously
1. axis (per channel?, row?, col?)
2. vector length (maybe using std::vector<T> wil do)
3. quantization algorithm (AFAIK some adds zp before multipluing sf and some do vice versa)
*/
...
};

...

typedef ScalarIOQuantInfo<uint8_t> QInfoU8;
typedef ScalarIOQuantInfo<int8_t> QInfoS8;

Please let me know if there's any suggestions or feedbacks.
And this is just an idea, but many use u8, s8, u16, s16 for unsigned/signed int8/int16 for abbreviation. I think using this terminology is more intuitive because I find the term qint8 quite unclear... does it mean signed int8 ? or signed int8 Tensor with quantization parameter? and if so, does uint16 Tensor have qParam?