Gpu selection

[OfekShochat]

can select gpus for training. syntax is

gpu: 1, 2, ..., n

and

gpu: 1,2,...,n

[OfekShochat]

@Tilps how can I push from gpu-selection branch on the fork to here now? I made the changes

[Tilps]

nothing to do.

[Tilps]

Needs a test and formatting fix and then probably fine to go.

addressed

[dje-dev]

This enhancement will be very useful. However it seems to break backwards compatibility, for example a value of 0 now throws
TypeError: argument of type 'int' is not iterable
instead perhaps the isinstance clause could be added as follows:
elif isinstance(self.cfg['gpu'], str) and "," in self.cfg['gpu']:

[OfekShochat]

yea yea its known. just I didnt know if this is even something to work on. if you are interested sure ill fix it happily!

[OfekShochat]

well now its saying some git shit, but this should work now. can you check @dje-dev? not home...

[dje-dev]

It turns out MirroredStrategy wants a set of string such as ["GPU:0", "GPU:1"].

So I was able to get code as shown below to work.

Note that set_memory_growth is called here, I'm not sure if that is required/desirable.
If it is called, it seems necessary to call it for all GPUs not just the selected ones, otherwise Tensorflow throws an error.

        elif "," in str(self.cfg['gpu']):
            gpus = tf.config.experimental.list_physical_devices('GPU')
            for gpu in gpus:
                tf.config.experimental.set_memory_growth(gpu, True)
            for i in self.cfg['gpu'].split(","):
                active_gpus.append("GPU:" + i)
            self.strategy = tf.distribute.MirroredStrategy(active_gpus)
            tf.distribute.experimental_set_strategy(self.strategy)

[OfekShochat]

Kk
Works now? If it does I'll change my pr