Skip to content

Add Celery / Auth / Task Integration

This page comes after:

That order matters.

What This Page Is For

Use it when the computation now needs:

  • async execution
  • retry behavior
  • queue selection
  • auth or access decisions
  • clearer operational boundaries

A Good Task Boundary

Retry-aware task wrapper pattern
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
@app.task(bind=True, max_retries=3, default_retry_delay=60)
def generate_example_task(self, state, district, block, gee_account_id):
    try:
        return generate_example(state, district, block, gee_account_id)  # (1)!
    except Exception as exc:
        logger.error(f"Celery task error: {exc}")                        # (2)!
        try:
            raise self.retry(exc=exc)                                    # (3)!
        except self.MaxRetriesExceededError:
            logger.error(f"Max retries exceeded for {state}/{district}/{block}")
            return False
  1. Keep the analytical function separately callable.
  2. Log at the operational boundary.
  3. Retry where transient infrastructure failures actually matter.

Auth Comes After Meaning

Auth decisions are important, but they are still later than “does the computation actually do the right thing?”

When you add auth or access checks, keep them:

  • explicit
  • documented
  • aligned with the intended audience of the route

For public data, see Public APIs.

For developer-facing computation surfaces, keep documentation consistent with Computing API Endpoints.

When To Stop Adding Layers

Not every computation needs every integration feature immediately.

If the computation is still evolving, it is often enough to stop at:

  • local callable
  • thin Django route
  • basic task execution

and postpone:

  • elaborate auth
  • retries
  • public metadata exposure
  • dataset registration

until the workflow is stable.

When the workflow is stable enough to become a reusable platform surface, continue into Add and Integrate Public / New Data Resources.