No project description provided
Project description
hlm-texts
Hlm at your fingertips
This repo currently just has
340-脂砚斋重批红楼梦.txt
: hlm_zhdavid-hawks-the-story-of-the-stone.txt
: hlm_enyang-hlm.txt
: hlm_en1joly-hlm.txt
: hlm_en2
It may expand to other versions. If you wish to have one particular version included, make a pull request
(fork, upload files and click the pull request button) or provide a text file to me.
Special Dependencies
hlm_texts
depends on polyglot that in turn depends on libicu
To install libicu
For Linux/OSX
E.g.
- Ubuntu:
sudo apt install libicu-dev
- Centos:
yum install libicu
- OSX:
brew install icu4c
Then use poetry
or pip
to install PyICU pycld2 Morfessor
, e.g.
pip install PyICU pycld2 Morfessor
or
poetry add PyICU pycld2 Morfessor
For Windows
Download and install the pyicu
and pycld2
(possibly also Morfessor
) whl packages for your OS/Python version from https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyicu and https://www.lfd.uci.edu/~gohlke/pythonlibs/#pycld2 (possibly also Morfessor https://www.lfd.uci.edu/~gohlke/pythonlibs/)
Installation
pip install hlm-texts
or git clone the repo and install from the source
git clone
cd hlm-texts
pip install -r requirements.text
Usage
from hlm_texts import hlm_en, hlm_zh, hlm_en1, hlm_en2
hlm_zh
and hlm_en
are copies of 340-脂砚斋重批红楼梦.txt
and david-hawks-the-story-of-the-stone.txt
, respectively, with blank lines removed and paragraphs retaind.
sent_tokenizer
: to tokenize text or list of texts
from hlm_texts import sent_tokenizer, hlm_en
hlm_en_sents = sent_tokenizer(hlm_en, lang="en")
Tokenizing long English texts for the first time can take a while (3-5 minutes for hlm_en, hlm_en1, hlm_en2). Subsequent operations are, however, instant since sent_tokenizer
is cached in ~/joblib_cache
(\Users\xyz\joblib_cachefor
Windows 10`).
Notes
The repo is for study purpose only. If you believe that your interests have been violated in any way, please let me know. I'll promptly follow it up with appropriate actions.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for hlm_texts-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f7783feb1981fa7c44e9290a289e804cf8f6acadc3ec95287be67e07cb68a2d6 |
|
MD5 | 9a2b66338ef765bff6ca829b4bf7427b |
|
BLAKE2b-256 | 526d37b4232a7227c880ebc7d2f68e04b6771397912239918b15f8328a3e6503 |