Storing Pandas Data in a MongoDB database
Project description
Antarctic
Project to persist Pandas data structures in a MongoDB database.
Installation
pip install antarctic
Usage
This project (unless the popular arctic project which I admire) is based on top of MongoEngine, see https://pypi.org/project/mongoengine/ MongoEngine is an ORM for MongoDB. MongoDB stores documents. We introduce here two new fields --- one for a Pandas Series and one for a Pandas DataFrame.
from mongoengine import Document, connect
from antarctic.PandasFields import SeriesField, FrameField
# connect with your existing MongoDB (here I am using a popular interface mocking a MongoDB)
client = connect(db="test", host="mongomock://localhost")
# Define the blueprint for a portfolio document
class Portfolio(Document):
nav = SeriesField()
weights = FrameField()
prices = FrameField()
The portfolio objects works exactly the way you think it works
p = Portfolio()
p.nav = pd.Series(...)
p.prices = pd.DataFrame(...)
p.save()
print(p.nav)
print(p.prices)
Behind the scenes we convert the both Series and Frame objects into json documents and store them in a MongoDB database.
We don't apply any clever conversion into compressed bytestreams. Performance is not our main concern here.
Database?
Storing json or bytestream representations of Pandas objects is not exactly a database. Appending is rather expensive as one would have to extract the original Pandas object, append to it and convert the new object back into a json or bytestream representation. Clever sharding can mitigate such effects but at the end of the day you shouldn't update such objects too often. Often practitioners use a small database for recording (e.g. over the last 24h) and update the MongoDB database once a day. It's extremely fast to read the Pandas objects out of such a construction.
Also note that in theory one could try to build this on top of pyarrow and support both R and Python.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for antarctic-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8883ae2f49547ec5323a4e3f6503b396405f248a224c3c00208ff2f9d0eb007d |
|
MD5 | 63fbc3992c30713ddcb33ebdaf801da7 |
|
BLAKE2b-256 | b783104766440f7686d7f2a2a584ec9f8bb61c1f00be154cc5785fedb7c53272 |