Welcome to Data Driven Discovery of Models (D3M) metalearing database documentation.

• To insert pipeline runs (and/or other related documents), use HTTP API. (In the future the core package will know how to do this for you.)
• To download documents from the metalearning database, use any ElasticSearch client and point it at https://metalearning.datadrivendiscovery.org/es endpoint. There are also occasional database dumps available. You can get a list of all document collections (ElasticSearch aliases) here.
• To better understand what is stored in the database, see core package documentation and document schemas.
• To coordinate work around metalearning, see the metalearning repository. Code for the HTTP API is available there as well.

As an example of using ElasticSearch client, you can use elasticsearch-dump to obtain all documents from the database:

$ npm install elasticdump
$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipelines --output=pipelines.json --sourceOnly
$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipeline_runs --output=runs.json --sourceOnly

Instead of exporting the entire database always, you can also provide a timestamp and download all documents after that given timestamp for a given index.
The command below will export all data from the pipelines index that were ingested after 2020-01-01T00:00:00Z

$ npm install elasticdump
$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipelines --output=pipelines.json --sourceOnly --searchBody='{ "query": {"range": {"_ingest_timestamp": {"gte": "2020-01-01T00:00:00Z"}}}}'

Pipeline run documents can get very large (due to the predictions stored in them) and exporting them may take a very long time or result in issues. If you are not interested in retrieving the predictions from the pipeline runs you can exclude them from dump.
The command below will export all pipeline runs excluding the predictions field

$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipeline_runs --output=runs.json --sourceOnly --limit=10 --searchBody='{ "query": {"range": {"_ingest_timestamp": {"gte": "2020-03-05T00:00:00Z"}}}, "_source": {"exclude": ["*.predictions"]}}'

The command below will export all pipeline runs excluding the predictions field and the method calls

$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipeline_runs --output=runs.json --sourceOnly --limit=10 --searchBody='{ "query": {"range": {"_ingest_timestamp": {"gte": "2020-03-05T00:00:00Z"}}}, "_source": {"exclude": ["*.predictions", "steps.*.method_calls"]}}'

Alternatively, you can also limit fetching the list of fields by changing

"_source": {"exclude": ["*.predictions"]}}

to

"_source": {"include": ["fields_interested_in"]}}