Data store

pyABC uses SQL databases for long-term storage of simulation and logging data, thus allowing to analyze and share results later.

Overview

An overview of what can be stored, and how to query the database, is given in the Data store API documentation.

Export the database to other formats

pyABC ships with a small convenience utility that allows to export the SQL databases to other formats. Supported formats are:

• CSV

• feather

• json

• msgpack

• html

• hdf

• stata

An example call might look as follows:

abc-export --db results.db --out exported.feather --format feather

This exports the database “results.db” generated by pyABC to “exported.feather”, which is written in in the feather format as indicated by the option --format feather (the file extension of the exported file is not parsed).

Check abc-export --help for further options to customize the export.

Migration

While we try to avoid it, sometimes changes to pyABC’s storage format are unavoidable. In such cases, we try to provide correct migration of outdated databases to the new format, or in general across versions. If a database is outdated, pyABC will complain when trying to import it via the History class.

We use Alembic to migrate our SQLAlchemy based databases. Install it via:

pip install pyabc[migrate]

To migrate a database, run:

abc-migrate --src=SOURCE.db --dst=DESTINATION.db

with SOURCE.db the name of the current outdated file, and DESTINATION.db the name of the generated migrated file (we recommend using a different file).

See abc-migrate --help for further information.