dense function combines dydense/dyrelu/linearscaling/gating into one function

Add logit gating, dense_layer, stop file, make dyrelu slopes/intercepts trainable

Weight gen (simple_gen) generates attention weights from each square by compressing then doing a batched dense to 64; buckets divides the training data based on material left.

Weight gen, buckets

Dytalking heads at this stage dynamically generates the projection matrices for the attention weights (same for all square pairs). Fixed set_visible_devices error by initializing tensorflow first in TFProcess and making DyDense temperature an instance attribute.

DyDense layers had issue saving sublayers so the design approach of the squeeze-excite layers is used, i.e., sublayers are moved outside into a function.

Fullgen compresses the tokens and combines them to extract global information into attention weights.

Dytalking heads adds residual to matrix specifying linear transformation

Removed old modules which were not useful including yaml references, also removed legacy resnet code. Added arc's encoding with option in yaml and also added example.yaml

update Readme to describe talking heads, fullgen, and dynamic kernel methods. Removed leelalogs and configs, except for example.yaml.

Fixed typo in tfprocess and Readme, made some config stuff oprtional, fixed arc encoding, removed fullgen bias

Added search_loss, which is one over the prediction for the best move, and confident_accuracy, which is the accuracy for positions where there is a clear best move. Removed simple gating. Also updated README to include info on auxiliary losses.

Smolgen is more efficient version of fullgen, also added square relu which adds 0.5% pol acc

self.ffn_gating does not belong

…ro-training into attention-net-body-ergo

Smolgen protobuf updates

Nadam and RMSprop optimizers, ability to choose particular gpu's (not all or one gpu but 0, 1 out of 4 etc, onnx save folder naming changes to fix continuation failure bug)

tfprocess.py typo fix in comment about net to model hack

Nadam, RMSprop, better multigpu, onnx continuation

policy rl is the amount of policy loss that can be reduced (i.e., policy loss minus entropy of target)

FastDepthwise acts as a depthwise convolution but far faster. use_depthwise_process in the yaml adds a dense layer with sqrrelu activation followed by layer norm and FastDepthwise

For use with j33's attention vis

FastLayerNormalization, switch layernorm alphas, mish activation, update depthwise process

Reverted obsolete features and allowed ffn to be specified more cleanly

make_value_buckets did not receive num_buckets

default activation was incorreclty set to None

RMSNorm support, value cat -> value error, more organized outputs (dict rather than tuple)

new stuff: ip_val_err and ip_emb_preproc. In the backend for the engine, RMSNorm and no norm at all should be recognized by the lack of beta/both gamma and beta

This reverts commit 0104192.

Didn't work before with gpus separated by commas

Setting p=x in quantized_conv gives it p bits of precision