Skip to main content

A spaCy package for the Rust tokenizations library

Project description

spacy-alignments: Align tokenizations for spaCy + transformers

A spaCy package for Yohei Tamura's Rust tokenizations library with Python bindings.

Installation

pip install -U pip setuptools wheel
pip install spacy-alignments

If no binary wheel is available for your platform, you will need to install Rust in order to build spacy-alignments from source.

spacy-alignments vs. pytokenizations

The spacy_alignments module is a drop-in replacement for tokenizations:

import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
assert a2b == [[0], [0]]
assert b2a == [[0, 1]]

The only difference between this package and the original pytokenizations is that it switches the build system to setuptools-rust to make it easier for us at Explosion to build source and binary packages for a wider range of platforms.

Bug reports and other issues

Please use spaCy's issue tracker to report a bug, or open a new thread on the discussion board for any other issue.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spacy-alignments-0.9.1.tar.gz (7.0 kB view hashes)

Uploaded Source

Built Distributions

spacy_alignments-0.9.1-cp312-cp312-win_amd64.whl (187.7 kB view hashes)

Uploaded CPython 3.12 Windows x86-64

spacy_alignments-0.9.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (314.0 kB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

spacy_alignments-0.9.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (305.8 kB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

spacy_alignments-0.9.1-cp312-cp312-macosx_11_0_arm64.whl (317.3 kB view hashes)

Uploaded CPython 3.12 macOS 11.0+ ARM64

spacy_alignments-0.9.1-cp312-cp312-macosx_10_9_x86_64.whl (312.3 kB view hashes)

Uploaded CPython 3.12 macOS 10.9+ x86-64

spacy_alignments-0.9.1-cp311-cp311-win_amd64.whl (187.7 kB view hashes)

Uploaded CPython 3.11 Windows x86-64

spacy_alignments-0.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (314.0 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

spacy_alignments-0.9.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (305.8 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

spacy_alignments-0.9.1-cp311-cp311-macosx_11_0_arm64.whl (317.3 kB view hashes)

Uploaded CPython 3.11 macOS 11.0+ ARM64

spacy_alignments-0.9.1-cp311-cp311-macosx_10_9_x86_64.whl (312.3 kB view hashes)

Uploaded CPython 3.11 macOS 10.9+ x86-64

spacy_alignments-0.9.1-cp310-cp310-win_amd64.whl (187.7 kB view hashes)

Uploaded CPython 3.10 Windows x86-64

spacy_alignments-0.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (314.0 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

spacy_alignments-0.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (305.8 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

spacy_alignments-0.9.1-cp310-cp310-macosx_11_0_arm64.whl (317.3 kB view hashes)

Uploaded CPython 3.10 macOS 11.0+ ARM64

spacy_alignments-0.9.1-cp310-cp310-macosx_10_9_x86_64.whl (312.3 kB view hashes)

Uploaded CPython 3.10 macOS 10.9+ x86-64

spacy_alignments-0.9.1-cp39-cp39-win_amd64.whl (187.4 kB view hashes)

Uploaded CPython 3.9 Windows x86-64

spacy_alignments-0.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (314.0 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

spacy_alignments-0.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (305.9 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

spacy_alignments-0.9.1-cp39-cp39-macosx_11_0_arm64.whl (317.3 kB view hashes)

Uploaded CPython 3.9 macOS 11.0+ ARM64

spacy_alignments-0.9.1-cp39-cp39-macosx_10_9_x86_64.whl (312.3 kB view hashes)

Uploaded CPython 3.9 macOS 10.9+ x86-64

spacy_alignments-0.9.1-cp38-cp38-win_amd64.whl (187.5 kB view hashes)

Uploaded CPython 3.8 Windows x86-64

spacy_alignments-0.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (314.1 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

spacy_alignments-0.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (305.7 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

spacy_alignments-0.9.1-cp38-cp38-macosx_11_0_arm64.whl (317.4 kB view hashes)

Uploaded CPython 3.8 macOS 11.0+ ARM64

spacy_alignments-0.9.1-cp38-cp38-macosx_10_9_x86_64.whl (312.0 kB view hashes)

Uploaded CPython 3.8 macOS 10.9+ x86-64

spacy_alignments-0.9.1-cp37-cp37m-win_amd64.whl (187.5 kB view hashes)

Uploaded CPython 3.7m Windows x86-64

spacy_alignments-0.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (314.1 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

spacy_alignments-0.9.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (305.7 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

spacy_alignments-0.9.1-cp37-cp37m-macosx_10_9_x86_64.whl (312.0 kB view hashes)

Uploaded CPython 3.7m macOS 10.9+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page