A fast and lightweight Python RDF parser which wraps bindings to Rust's Rio using PyO3
Project description
LightRDF
A fast and lightweight Python RDF parser which wraps bindings to Rust's Rio using PyO3.
Features
- Supports N-Triples, Turtle, and RDF/XML
- Handles large-size RDF documents
- Provides HDT-like interfaces
Install
pip install lightrdf
Usage
Iterate over all triples (Parser)
import lightrdf
parser = lightrdf.Parser() # or lightrdf.xml.Parser() for xml
for triple in parser.parse("./go.owl", base_iri=None):
print(triple)
Iterate over all triples (HDT-like)
import lightrdf
doc = lightrdf.RDFDocument("./go.owl")
# ...or lightrdf.RDFDocument("./go.owl", base_iri="", parser=lightrdf.xml.PatternParser) for xml
# `None` matches arbitrary term
for triple in doc.search_triples(None, None, None):
print(triple)
Triple pattern (HDT-like)
import lightrdf
doc = lightrdf.RDFDocument("./go.owl")
for triple in doc.search_triples("http://purl.obolibrary.org/obo/GO_0005840", None, None):
print(triple)
# Output:
# ('http://purl.obolibrary.org/obo/GO_0005840', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type', 'http://www.w3.org/2002/07/owl#Class')
# ('http://purl.obolibrary.org/obo/GO_0005840', 'http://www.w3.org/2000/01/rdf-schema#subClassOf', 'http://purl.obolibrary.org/obo/GO_0043232')
# ...
# ('http://purl.obolibrary.org/obo/GO_0005840', 'http://www.geneontology.org/formats/oboInOwl#inSubset', 'http://purl.obolibrary.org/obo/go#goslim_yeast')
# ('http://purl.obolibrary.org/obo/GO_0005840', 'http://www.w3.org/2000/01/rdf-schema#label', '"ribosome"^^<http://www.w3.org/2001/XMLSchema#string>')
Tip: Open file in Python (Parser)
import lightrdf
parser = lightrdf.Parser()
with open("./go.owl", "rb") as f:
for triple in parser.parse(f, format="owl", base_iri=None):
print(triple)
import lightrdf
parser = lightrdf.xml.Parser()
with open("./go.owl", "rb") as f:
for triple in parser.parse(f, base_iri=None):
print(triple)
Tip: Open file in Python (HDT-like)
import lightrdf
with open("./go.owl", "rb") as f:
doc = lightrdf.RDFDocument(f, parser=lightrdf.xml.PatternParser)
for triple in doc.search_triples("http://purl.obolibrary.org/obo/GO_0005840", None, None):
print(triple)
Tip: Parse from string
import io
import lightrdf
data = """<http://one.example/subject1> <http://one.example/predicate1> <http://one.example/object1> . # comments here
# or on a line by themselves
_:subject1 <http://an.example/predicate1> "object1" .
_:subject2 <http://an.example/predicate2> "object2" ."""
doc = lightrdf.RDFDocument(io.BytesIO(data.encode()), parser=lightrdf.turtle.PatternParser)
for triple in doc.search_triples("http://one.example/subject1", None, None):
print(triple)
Benchmark (WIP)
On MacBook Air (13-inch, 2017), 1.8 GHz Intel Core i5, 8 GB 1600 MHz DDR3
https://gist.github.com/ozekik/b2ae3be0fcaa59670d4dd4759cdffbed
$ wget -q http://purl.obolibrary.org/obo/go.owl
$ gtime python3 count_triples_rdflib_graph.py ./go.owl # RDFLib 4.2.2
1436427
235.29user 2.30system 3:59.56elapsed 99%CPU (0avgtext+0avgdata 1055816maxresident)k
0inputs+0outputs (283major+347896minor)pagefaults 0swaps
$ gtime python3 count_triples_lightrdf_rdfdocument.py ./go.owl # LightRDF 0.1.1
1436427
7.90user 0.22system 0:08.27elapsed 98%CPU (0avgtext+0avgdata 163760maxresident)k
0inputs+0outputs (106major+41389minor)pagefaults 0swaps
$ gtime python3 count_triples_lightrdf_parser.py ./go.owl # LightRDF 0.1.1
1436427
8.00user 0.24system 0:08.47elapsed 97%CPU (0avgtext+0avgdata 163748maxresident)k
0inputs+0outputs (106major+41388minor)pagefaults 0swaps
https://gist.github.com/ozekik/636a8fb521401070e02e010ce591fa92
$ wget -q http://downloads.dbpedia.org/2016-10/dbpedia_2016-10.nt
$ gtime python3 count_triples_rdflib_ntparser.py dbpedia_2016-10.nt # RDFLib 4.2.2
31050
1.63user 0.23system 0:02.47elapsed 75%CPU (0avgtext+0avgdata 26568maxresident)k
0inputs+0outputs (1140major+6118minor)pagefaults 0swaps
$ gtime python3 count_triples_lightrdf_ntparser.py dbpedia_2016-10.nt # LightRDF 0.1.1
31050
0.21user 0.04system 0:00.36elapsed 71%CPU (0avgtext+0avgdata 7628maxresident)k
0inputs+0outputs (534major+1925minor)pagefaults 0swaps
Alternatives
- RDFLib – (Pros) pure-Python, matured, feature-rich / (Cons) takes some time to load triples
- pyHDT – (Pros) extremely fast and efficient / (Cons) requires pre-conversion into HDT
Todo
- Push to PyPI
- Adopt CI
- Handle Base IRI
- Add basic tests
- Support NQuads and TriG
- Add docs
- Add tests for w3c/rdf-tests
- Resume on error
- Allow opening fp
License
Rio and PyO3 are licensed under the Apache-2.0 license.
Copyright 2020 Kentaro Ozeki
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distributions
Close
Hashes for lightrdf-0.2.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3493181dfb0a5602333f74f38e893449298de2e3ddeff911877fe5ffcb2cc8d |
|
MD5 | 836eecb9158cd74bed31e9854e49e877 |
|
BLAKE2b-256 | 394e7985d44b3323131691ecdf52cc15ad6bfc8e84a56125b63a1a51cab31ce8 |
Close
Hashes for lightrdf-0.2.0-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a344a0b8bfcf2b8a0568935fdc13004aada638dca28e9aa87abd25c4d30664c2 |
|
MD5 | be7eb80622e9e5814493557a50688e2c |
|
BLAKE2b-256 | 53eb0f0058dd65f6b598cb0c1584779efda96aa9173b364727e355eb8bbaf48f |
Close
Hashes for lightrdf-0.2.0-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddf10ba9b4681f7b1b5ad821acdd8444a2d50c9d7924cc6155b47cb70e1d8a05 |
|
MD5 | 8247964d8ff8e4a3e0a4b20e0a681d7a |
|
BLAKE2b-256 | 551323eeee502c1917619166df73348f00d28f589f47793848ecf6341fb40c42 |
Close
Hashes for lightrdf-0.2.0-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8879a45393695644b71ae2bffc0ccb89c996a757c9dea113e659a91a55a7b7c2 |
|
MD5 | 453f5e1cd0069594683cfcab18df3bce |
|
BLAKE2b-256 | d79fb64f6725777c1e64440215a667c8e5f696dabce471f5adcfe12085dfc3ec |
Close
Hashes for lightrdf-0.2.0-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a854a0fd60f1ed7e2d7497363711e87663735d1f8c8b3d8363b44bfa32a3ef15 |
|
MD5 | a8e5c5a66756fec296aa61855d899e74 |
|
BLAKE2b-256 | 390431717a7b58974cdeb101de3bb3fa4422fad6dde55368a73edf1cc855b48e |
Close
Hashes for lightrdf-0.2.0-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 095cf5dc385ba20839bebc331dc7d260d5230c0674e29763fe471327bef24caf |
|
MD5 | de9604e1932f8d0ba75ae427a7541b7b |
|
BLAKE2b-256 | d480a5b5b52bf06b57fa84ed8ac998b548136f568109995df0a5760be53f2f01 |
Close
Hashes for lightrdf-0.2.0-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6efc0099f5fcf6a6a5d66ebf96e795167fe4f7630674c110357c2a57532ec9c |
|
MD5 | 452af6d87d22f4fbfaa026cd5a1d47d4 |
|
BLAKE2b-256 | 4544c11a68cc2405fc2a414a4dc6d37ec188fe54c271755c9564738f56e12c81 |
Close
Hashes for lightrdf-0.2.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4589498457a7398386a88e3d7017bb0a800bbac9082b027ed814db295dc48eae |
|
MD5 | 902647afad8f27f9da0e2c1f2d6b5434 |
|
BLAKE2b-256 | f0f9e96d4f66ee350a7f32d7a66d29fdf79398cfc7e84b23b926c2aff71a91c5 |
Close
Hashes for lightrdf-0.2.0-cp36-cp36m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5904f3ee9c0b61d90152cda6914880749c89d8e90a72ddc80c08724c21ce140d |
|
MD5 | b1ff09e311e21f74cb0e69111f0392f6 |
|
BLAKE2b-256 | 6a5c96225c31cd4f25b69080e2e2d9118f47e5abb56ad8bb214375a85c23842d |