Skip to content

Pipeline Development

Pipeline development is one of the main ways CoRE Stack itself is developed.

This section explains how CoRE Stack computes the data it publishes, and how contributors can extend those computations responsibly once the backend is installed.

Read the CoRE Stack Data Structure chapter first if you have not already.

Use it when you want to answer one of these questions:

  • what produced the data I am seeing?
  • how do the pipeline families differ?
  • how do I move from an installed backend to runnable computations?
  • where does the code live?
  • how do I move from a local experiment to an integrated computation?

Follow this order if you want the straightest route through pipeline development:

  1. Read How Our Pipelines Work Algorithmically to understand what each family computes.
  2. Read How They Work Programmatically to understand the code path.
  3. Trace one real request in First Manual API Run.
  4. Start new work from Local Pipeline First.
  5. Add the thin route layer through Integrate With Django Computing API.
  6. Add async execution through Add Celery / Auth / Task Integration only when needed.
  7. Use Computing API Endpoints as the route catalog for the existing compute surface.
  8. Use Build New Pipelines, Pipeline Integrations, and Add and Integrate Public / New Data Resources when extending the stack.

If you are here to understand an existing output rather than build something new, move into Core Workflows or How Current Data Was Computed.

The Common Shape

flowchart LR
    A[Local experiment or API request] --> B[computing/urls.py]
    B --> C[computing/api.py]
    C --> D[Pipeline module]
    D --> E[Earth Engine or local processing]
    E --> F[Dataset metadata]
    E --> G[GeoServer publication]
    E --> H[STAC and public layer metadata]

This shape changes in detail from workflow to workflow, but not in spirit.

If your environment is not ready yet, complete Installer and use Setup Troubleshooting before going deeper into pipeline internals.

Workflow Families

Key Backend Surfaces