Transformers

NLP Architect integrated the Transformer models available in pytorch-transformers. Using Transformer models based on a pre-trained models usually done by attaching a classification head on the transformer model and fine-tuning the model (transformer and classifier) on the target (down-stream) task.

Base model

TransformerBase is a base class for handling loading, saving, training and inference of transformer models.

The base model support pytorch-transformers configs, tokenizers and base models as documented in their website (see our base-class for supported models).

In order to use the Transformer models just sub-class the base model and include:

• A classifier (head) for your task.
• sub-method handling of input to tensors used by model.
• any sub-method to evaluate the task, do inference, etc.

Models

Available transformer family models in NLP Architect:

	BERT	Quantized BERT	XLM	XLNet	RoBERTa
Sequence classification	Y	Y	Y	Y	Y
Token classification	Y	Y		Y	Y

Sequence classification

TransformerSequenceClassifier is a transformer model with sentence classification head (the [CLS] token is used as classification label) for sentence classification tasks (classification/regression).

See nlp_architect.procedures.transformers.glue for an example of training sequence classification models on GLUE benchmark tasks.

Training a model on GLUE tasks, using BERT-base uncased base model:

nlp-train transformer_glue \
    --task_name <task name> \
    --model_name_or_path bert-base-uncased \
    --model_type bert \
    --output_dir <output dir> \
    --evaluate_during_training \
    --data_dir </path/to/glue_task> \
    --do_lower_case

Running a model:

nlp-inference run transformer_glue \
    --model_path <path to model> \
    --task_name <task_name> \
    --model_type bert \
    --output_dir <output dir> \
    --data_dir <path to data> \
    --do_lower_case \
    --overwrite_output_dir

To run evaluation on the task’s development set add the flag --evaluate to the command line.

Token classification

TransformerTokenClassifier is a transformer model for token classification for tasks such as NER, POS or chunking.

See example for usage TransformerTokenClassifier NER model description.