cond and uncond quantization issue

[Andy0422]

I just confused that i cannot get the clue about the cond and uncond quantization part you claim in your paper in the code, could you give me a hand?

[A-suozhang]

The classifier-free guidance process consists of two network inference, once with the control signal, and once without, the results are then weighted sum together. These two inference have activation variations, and thus should use different quantization parameters.

Normally, the two inference is conducted in parallel in the batch dimension, and the quantizer automatically calculated the activation scaling factor for each token (of all batches). To calculate the quantization parameter respectively for cond and uncond, we need to conduct two separate inference.

The CFG split code is presented here (apology for it's hidden inside the scheduler), by turning the CFG_SPLIT=True, you could conduct 2 separate inference, and save the quantization parameters for the cond and uncond inference respectively.

https://github.com/A-suozhang/ViDiT-Q/blob/fb2c8486a63aee5071a80d7382802ace0b08aeb7/t2v/opensora/schedulers/iddpm/__init__.py#L140