Software Heritage Content Indexer
Project description
swh-indexer
Tools to compute multiple indexes on SWH's raw contents:
- content:
- mimetype
- ctags
- language
- fossology-license
- metadata
- revision:
- metadata
An indexer is in charge of:
- looking up objects
- extracting information from those objects
- store those information in the swh-indexer db
There are multiple indexers working on different object types:
- content indexer: works with content sha1 hashes
- revision indexer: works with revision sha1 hashes
- origin indexer: works with origin identifiers
Indexation procedure:
- receive batch of ids
- retrieve the associated data depending on object type
- compute for that object some index
- store the result to swh's storage
Current content indexers:
-
mimetype (queue swh_indexer_content_mimetype): detect the encoding and mimetype
-
language (queue swh_indexer_content_language): detect the programming language
-
ctags (queue swh_indexer_content_ctags): compute tags information
-
fossology-license (queue swh_indexer_fossology_license): compute the license
-
metadata: translate file into translated_metadata dict
Current revision indexers:
- metadata: detects files containing metadata and retrieves translated_metadata in content_metadata table in storage or run content indexer to translate files.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
swh.indexer-0.0.120.tar.gz
(76.0 kB
view hashes)
Built Distribution
swh.indexer-0.0.120-py3-none-any.whl
(109.3 kB
view hashes)
Close
Hashes for swh.indexer-0.0.120-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c235e00f5d6708f3d0b3f84207aad66c03fb337fc6c60aeed7c4a681f08c42a |
|
MD5 | 8cc6f8f511cd1d5ee22ac126e5161f2d |
|
BLAKE2b-256 | 384c65a8c5d8cd164509e2f9850f38aa6b041522124fdc1d038ca36dba9b1903 |