ctranslate2 3.5.0

Fast inference engine for Transformer models

CTranslate2

CTranslate2 is a C++ and Python library for efficient inference with Transformer models.

The project implements a custom runtime that applies many performance optimization techniques such as weights quantization, layers fusion, batch reordering, etc., to accelerate and reduce the memory usage of Transformer models on CPU and GPU. The following model types are currently supported:

• Encoder-decoder models: Transformer base/big, M2M-100, NLLB, BART, mBART, Pegasus, T5, Whisper
• Decoder-only models: GPT-2, OPT

Compatible models should be first converted into an optimized model format. The library includes converters for multiple frameworks:

• OpenNMT-py
• OpenNMT-tf
• Fairseq
• Marian
• OPUS-MT
• Transformers

The project is production-oriented and comes with backward compatibility guarantees, but it also includes experimental features related to model compression and inference acceleration.

Key features

• Fast and efficient execution on CPU and GPU
  The execution is significantly faster and requires less resources than general-purpose deep learning frameworks on supported models and tasks thanks to many advanced optimizations: layer fusion, padding removal, batch reordering, in-place operations, caching mechanism, etc.
• Quantization and reduced precision
  The model serialization and computation support weights with reduced precision: 16-bit floating points (FP16), 16-bit integers (INT16), and 8-bit integers (INT8).
• Multiple CPU architectures support
  The project supports x86-64 and AArch64/ARM64 processors and integrates multiple backends that are optimized for these platforms: Intel MKL, oneDNN, OpenBLAS, Ruy, and Apple Accelerate.
• Automatic CPU detection and code dispatch
  One binary can include multiple backends (e.g. Intel MKL and oneDNN) and instruction set architectures (e.g. AVX, AVX2) that are automatically selected at runtime based on the CPU information.
• Parallel and asynchronous execution
  Multiple batches can be processed in parallel and asynchronously using multiple GPUs or CPU cores.
• Dynamic memory usage
  The memory usage changes dynamically depending on the request size while still meeting performance requirements thanks to caching allocators on both CPU and GPU.
• Lightweight on disk
  Quantization can make the models 4 times smaller on disk with minimal accuracy loss.
• Simple integration
  The project has few dependencies and exposes simple APIs in Python and C++ to cover most integration needs.
• Configurable and interactive decoding
  Advanced decoding features allow autocompleting a partial sequence and returning alternatives at a specific location in the sequence.

Some of these features are difficult to achieve with standard deep learning frameworks and are the motivation for this project.

Installation and usage

CTranslate2 can be installed with pip:

pip install ctranslate2

The Python module is used to convert models and can translate or generate text with few lines of code:

translator = ctranslate2.Translator(translation_model_path)
translator.translate_batch(tokens)

generator = ctranslate2.Generator(generation_model_path)
generator.generate_batch(start_tokens)

See the documentation for more information and examples.

Benchmarks

We translate the En->De test set newstest2014 with multiple models:

• OpenNMT-tf WMT14: a base Transformer trained with OpenNMT-tf on the WMT14 dataset (4.5M lines)
• OpenNMT-py WMT14: a base Transformer trained with OpenNMT-py on the WMT14 dataset (4.5M lines)
• OPUS-MT: a base Transformer trained with Marian on all OPUS data available on 2020-02-26 (81.9M lines)

The benchmark reports the number of target tokens generated per second (higher is better). The results are aggregated over multiple runs. See the benchmark scripts for more details and reproduce these numbers.

Please note that the results presented below are only valid for the configuration used during this benchmark: absolute and relative performance may change with different settings.

CPU

	Tokens per second	Max. memory	BLEU
OpenNMT-tf WMT14 model
OpenNMT-tf 2.26.1 (with TensorFlow 2.9.0)	283.0	3475MB	26.93
OpenNMT-py WMT14 model
OpenNMT-py 2.2.0 (with PyTorch 1.11.0)	474.2	1543MB	26.77
- int8	510.6	1455MB	26.72
CTranslate2 2.17.0	1220.2	1072MB	26.77
- int16	1534.8	920MB	26.82
- int8	1737.5	771MB	26.89
- int8 + vmap	2122.4	666MB	26.62
OPUS-MT model
Transformers 4.19.2	230.1	2840MB	27.92
Marian 1.11.0	756.6	13819MB	27.93
- int16	718.4	10395MB	27.65
- int8	853.3	8166MB	27.27
CTranslate2 2.17.0	988.0	995MB	27.92
- int16	1285.7	847MB	27.51
- int8	1469.1	847MB	27.71

Executed with 8 threads on a c5.metal Amazon EC2 instance equipped with an Intel(R) Xeon(R) Platinum 8275CL CPU.

GPU

	Tokens per second	Max. GPU memory	Max. CPU memory	BLEU
OpenNMT-tf WMT14 model
OpenNMT-tf 2.26.1 (with TensorFlow 2.9.0)	1289.3	2667MB	2407MB	26.93
OpenNMT-py WMT14 model
OpenNMT-py 2.2.0 (with PyTorch 1.11.0)	1271.4	2993MB	3553MB	26.77
FasterTransformer 4.0	2941.3	5869MB	2327MB	26.77
- float16	6497.4	3917MB	2325MB	26.83
CTranslate2 2.17.0	3644.1	1231MB	646MB	26.77
- int8	5393.6	975MB	522MB	26.83
- float16	5454.7	815MB	550MB	26.78
- int8 + float16	6158.6	687MB	523MB	26.80
OPUS-MT model
Transformers 4.19.2	811.1	4013MB	3044MB	27.92
Marian 1.11.0	2172.9	3127MB	1869MB	27.92
- float16	2722.0	2985MB	1715MB	27.93
CTranslate2 2.17.0	3042.5	1167MB	486MB	27.92
- int8	4573.1	1007MB	511MB	27.89
- float16	4718.4	783MB	552MB	27.85
- int8 + float16	5300.5	687MB	508MB	27.81

Executed with CUDA 11 on a g4dn.xlarge Amazon EC2 instance equipped with a NVIDIA T4 GPU (driver version: 510.47.03).

Additional resources

• Documentation
• Forum
• Gitter