Welcome to Data Driven Discovery of Models (D3M) metalearing database documentation.

As an example of using ElasticSearch client, you can use elasticsearch-dump to obtain all documents from the database:

$ npm install elasticdump
$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipelines --output=pipelines.json --sourceOnly
$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipeline_runs --output=runs.json --sourceOnly

Instead of exporting the entire database always, you can also provide a timestamp and download all documents after that given timestamp for a given index.
The command below will export all data from the pipelines index that were ingested after 2020-01-01T00:00:00Z

$ npm install elasticdump
$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipelines --output=pipelines.json --sourceOnly --searchBody='{ "query": {"range": {"_ingest_timestamp": {"gte": "2020-01-01T00:00:00Z"}}}}'

Pipeline run documents can get very large (due to the predictions stored in them) and exporting them may take a very long time or result in issues. If you are not interested in retrieving the predictions from the pipeline runs you can exclude them from dump.
The command below will export all pipeline runs excluding the predictions field

$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipeline_runs --output=runs.json --sourceOnly --limit=10 --searchBody='{ "query": {"range": {"_ingest_timestamp": {"gte": "2020-03-05T00:00:00Z"}}}, "_source": {"exclude": ["*.predictions"]}}'

The command below will export all pipeline runs excluding the predictions field and the method calls
$ npx elasticdump --input=https://metalearning.datadrivendiscovery.org/es --input-index=pipeline_runs --output=runs.json --sourceOnly --limit=10 --searchBody='{ "query": {"range": {"_ingest_timestamp": {"gte": "2020-03-05T00:00:00Z"}}}, "_source": {"exclude": ["*.predictions", "steps.*.method_calls"]}}'

Alternatively, you can also limit fetching the list of fields by changing
"_source": {"exclude": ["*.predictions"]}}
to
"_source": {"include": ["fields_interested_in"]}}