transformers-keras 0.2.0

Transformer-based models implemented in tensorflow 2.x(Keras)

transformers-keras

Transformer-based models implemented in tensorflow 2.x(Keras).

Installation

pip install -U transformers-keras

Models

• Transformer[DELETED]
  • Attention Is All You Need.
  • Here is a tutorial from tensorflow:Transformer model for language understanding
• BERT
  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• ALBERT
  • ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

BERT

All the bert models pretrained by google-research/bert can be loaded:

from transformers_keras import Bert

model = Bert.from_pretrained('/path/to/pretrained/bert/model')
# then, use this model to fine-tune new model as a keras layer, or to do inference
model(model.dummy_inputs())

ALBERT

All the albert models pretrained by google-research/albert can be loaded:

from transformers_keras import Albert

model = Bert.from_pretrained('/path/to/pretrained/albert/model')
# then, use this model to fine-tune new model as a keras layer, or to do inference
model(model.dummy_inputs())