Skip to content
This repository has been archived by the owner on May 3, 2022. It is now read-only.

Latest commit

 

History

History
174 lines (140 loc) · 7.64 KB

ChangeLog.md

File metadata and controls

174 lines (140 loc) · 7.64 KB
Raw

Rinokeras 1.0.0 -- Changelog

Major Changes (Completed)

  • Moved tensorflow 1.x rinokeras to rinokeras.core.v1x
  • Updated folder structure for cleaner API changes
  • Added support for tensorflow 1.13
  • Confirmed support for tensorflow 1.12
  • Confirmed support in Python 3.4,3.5,3.6,3.7
  • Restructure utils folder

Major Changes (In Progress):

  • Added additional testing (regression + unit testing) for 1.x code
  • API changed to conform to Keras standards
  • Added support for tensorflow 2.0

Major Changes (Planned)

  • Changed default to tensorflow 2.0
  • Added TPU support for tensorflow 2.0 libraries
  • Add support for Keras compile/fit

Minor/All Changes

Continuous Integration

  • Added support for code coverage
  • Increased supported testing from Python 3.6 -> Python 3.4, 3.5, 3.6 and 3.7
  • Bumped testing VM to Ubuntu 16.04 (From 14.04)

Library

  • Removed outdated examples
  • Simplified API
    • rinokeras.common.layers moved to rinokeras.layers
    • rinokeras.common.distributions moved to rinokeras.layers.distributions
    • rinokeras.common.gcn moved to rinokeras.layers.gcn
    • rinokeras.common.losses moved to rinokeras.losses
    • rinokeras.common.attention moved to rinokeras.layers.attention
    • rinokeras.common.optimizers moved to rinokeras.train.optimizers
    • rinokeras.common.rnn moved to rinokeras.models.rnn
  • Removed non-functioning code
    • Updated GLOW model to be functional
    • Updated DenseTranspose to be functional
    • Removed Adamax (Re-Added in TF1.13 support)
  • Deprecated EagerGraph v1x API in favor of Rinokeras v2x
  • Moved GroupedConvolution and ResidualBlock from rinokeras.models.resnet to rinokeras.layers.conv and rinokeras.layers.residual respectively

Graphs (1.x)

  • Changed batch_reduce calls to batch_all_reduce calls
  • Changed call_for_each_tower calls to call_for_each_replica calls
  • Added testing for graph generation and execution

Attention Layers (1.x)

  • Luong attention changed so local=True variant takes a tuple of three inputs (target_hidden, source_hidden, position)
  • AttentionQKV renamed to AttentionQKVProjection, altered so that it takes (query, key, value) antecedents instead of (query, memory) antecedents
  • AttentionQKVProjection now takes an optional project_value argument which turns off the value projection
  • Trilinear similarity changed so the pass/return is (context, query) - formerly the tuple passed was (query, context) and returned was a tensor with [bs x context x query]
  • Altered Trilinear similarity to support multiple heads in the case of using it in an attention layer directly (instead of in CQ attention)
  • Changed ScaledDotProductSimilarity to match API of TrilinearSimilarity by taking a tuple (queries, keys) instead of two independent arguments
  • Keyword of ApplyAttentionMask changed from "similarity" to "inputs" to reflect Keras API requirements
  • Fixed bug in ApplyAttentionMask for non-softmax losses
  • AttentionMap changed to comply with Keras API, now takes "inputs" tuple of (queries, keys, values) instead of queries, keys, values as keywords
  • Fixed bug in AttentionMap when using differing numbers of keys and queries
  • Added return_attention_weights argument to AttentionMap layer
  • Added attention_function argument to MultiHeadAttentionMap layer
  • Added trilinear similarity to MultiHeadAttention layer
  • Added attention_function argument to MultiHeadAttention layer
  • Changed MultiHeadAttention layer to take separate query, key and value antecedents
  • Changed MultiHeadAttention layer to take/pass project_value argument to AttentionQKVProjection
  • Changed ContextQueryAttention layer to take tuple of (context, query) as "inputs"
  • Reversed order of "context" and "query" in call to ContextQueryAttention to maintain consistency
  • Changed ContextQueryAttention "attention_method" argument to "similarity_metric" to maintain consistency
  • Added get_config/from_config to SelfAttention, MultiHeadAttention

Activation Layers (1.x)

  • No Changes

Autoregressive Layers (1.x)

  • Fixed breaking bug in CouplingLayer

Conv Layers (1.x)

  • Renamed NormedConv to NormedConvStack to maintain naming consistency
  • Added GroupedConv from rinokeras.models.resnet

Distribution Layers (1.x)

  • Fixed bug in entropy of CategoricalPd

Dropout Layers (1.x)

  • Fixed bug in LayerDropout where layer was never dropped (even during training)

(NEW) Embedding Layers (1.x)

  • Moved VQAImageEmbedding to ImageEmbedding layer

GCN (1.x)

  • Fixed shape bugs in GCN
  • Changed GCN so that inputs are [BS x N x units] instead of [BS x units x N]

Inversion Layers (1.x)

  • Fixed name bug in DenseTranspose

Losses (1.x)

  • No Changes

Masking Layers (1.x)

  • Renamed MaskInput to BERTRandomReplaceMask

Normalization Layers (1.x)

  • No Changes

Position Embedding Layers (1.x)

  • Fixed bug with reproject_embedding in 2D and 3D cases
  • Fixed bug with concatenation in 3D case

Residual Layers (1.x)

  • Changed Highway layer to take a layer as input (expanded functionality)
  • Added ResidualBlock from rinokeras.models.resnet

Stacks (1.x)

  • Added LayerDropoutStack which adds layer-dropout between each layer of the stack

Transformer (1.x)

  • Moved TransformerSelfAttention and TransformerMultiAttention into transformer_attention.py
  • Moved TransformerFeedForward into transformer_ff.py
  • Moved TransformerEncoderBlock and TransformerEncoder into transformer_encoder.py
  • Moved TransformerDecoderBlock and TransformerDecoder into transformer_decoder.py
  • Moved TransformerInputEmbedding into transformer_embedding.py
  • Simplified TransformerSelfAttentionOnlyDecoder (TASODecoder) into transformer_decoder (just pass None for the source)
  • Changed TransformerMultiAttention to conform to keras inputs, now takes input tuple inputs=(source, target)
  • Updated parameter ordering of TransformerFeedForward layer
  • Updated parameter ordering of TransformerEncoderBlock layer
  • Changed TransformerEncoder to take tuple mask=(encoder_mask, conv_mask)
  • Changed TransformerDecoderBlock API to conform to keras inputs, now takes inputs tuple=(encoder_inputs, decoder_inputs) and mask tuple=(self_attention_mask, cross_attention_mask)
  • Updated parameter ordering of TransformerDecoderBlock layer
  • Updated parameter odering of TransformerDeocder layer
  • Changed TransformerDecoder API to conform to keras inputs, now takes inputs tuple=(encoder_inputs, decoder_inputs) and mask tuple=(self_attention_mask, cross_attention_mask)
  • Changed fast_beam_decode to take n_beams instead of beam_size
  • Fixed bugs in fast_decode for non-discrete outputs
  • Added get_config/from_config to TransformerSelfAttention, TransformerMultiAttention, TransformerFeedForward, TransformerEncoderBlock, TransformerEncoder, TransformerDecoderBlock, TransformerDecoder and Transformer
  • Refactored some utilities from transformer_decoder into transformer_utils
  • Updated transformer to conform to Keras API

QANet (1.x)

  • Refactored QANet into multiple files
  • Fixed bug in QANet self attention
  • Reordered arguments in QANet/QANetEncoder
  • Added get_config/from_config to all QANet layers
  • Added support for random (learned) embedding matrices

ResNet (1.x)

  • Moved GroupedConv to Conv layers

GLOW (1.x)

  • Added warnings to GLOW code about functionality

Uitils (1.x)

  • Created unique utilities files for utils:
    • Keras
    • metrics
    • masking
    • optim
    • schedules
    • tensors
  • Added BLEU1, BLEU2, BLEU3, BLEU4 and ROUGE-L metrics as tensorflow ops

Known Errors/Bugs/Questions

Distribution Layers (1.x)

  • Odd shape for entropy of DiagGaussPd () vs (BS,)

Transformer (1)

  • Using a custom output layer may cause problems when reconstructing from config (untested)

QANet (1.x)

  • QANet layers don't handle extra kwargs in get/from config