attacut 0.0.4.dev0

Yet Another Tokenizer for Thai

AttaCut

TLDR: 3-Layer dilated CNN on character and syllable features

Installation

# only for beta version
$ pip install attacut

Usage

Command-Line Interface

$ attacut-cli -h
AttaCut: Fast and Reasonably Accurate Tokenizer for Thai

Usage:
  attacut-cli <src> [--dest=<dest>] [--model=<model>]
  attacut-cli (-h | --help)

Options:
  -h --help         Show this screen.
  --model=<model>   Model to be used [default: attacut-sc].
  --dest=<dest>     If not specified, it'll be <src>-tokenized-by-<model>.txt

Higher-Level Inferface

aka. module importing

from attacut import Tokenizer

atta = Tokenizer(model="attacut-sc")
atta.tokenizer(txt)

Development

Please refer to DEVELOPMENT.md

Related Resources

• Tokenization Visualization
• Thai Tokenizer Dockers

Acknowledgements

• This repository was initially done by Pattarawat Chormai, while interning at Dr. Attapol Thamrongrattanarit's NLP Lab, Chulalongkorn University, Bangkok, Thailand.
• Many thanks to my collegeus at Dr. Attapol's lab, PyThaiNLP team, Ekapol Chuangsuwanich , Noom, and Can for comments and feedback.