Skip to main content

Reversible String Process Pipeline

Project description

Build Status PyPI version

Reversible string processing pipe. Featuring reproducibility, serializability and performance.

Installation

pip install strpipe

Usage

import strpipe as sp

p = sp.Pipe()
p.add_step_by_op_name('ZhCharTokenizer')
p.add_step_by_op_name('AddSosEos')
p.add_checkpoint()
p.add_step_by_op_name('Pad')
p.add_step_by_op_name('TokenToIndex')

data = [
    '你好啊',
    '早安',
    '你早上好',
]

p.fit(data)
result, tx_info, intermediates = p.transform(data)  # convention: tx => tranform
back_data = p.inverse_transform(result, tx_info)

Serialization

# Save it
p.save_json('/path/of/pipe')

# Load it
p = sp.Pipe.restore_from_json('/path/of/pipe')
result, meta = p.transform(['你好'])

Test

$ make test

Docs

$ make docs

Docs will be built in the `docs/build/html` folder. (Note: this also reinstalls the package because we
need Cython code to be rebuilt.)

Extend Ops

  1. Extend the new ops with BaseOp

  2. Define input_type, output_type

  3. Implement op creation

  4. Implement fit, transform, inverse_transform. If the op is stateless, the fit method should return None.

    Note: It is expected that an ops’s functionality will often be able to be decomposed into several functions. These functions should be written into (or imported from) the toolkit package for easy reuse. Ops in the ops package will, for the most part, be wrappers for functions in toolkit.

  5. Write tests

  6. Register to op_factory

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

strpipe-0.5.0.tar.gz (615.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page