orca.celery
Celery application configuration for the ORCA distributed pipeline.
This module initializes the Celery application with broker and backend settings, configures task queues for different workload types (default, cosmology, bandpass, imaging), and sets up task routing to direct specific tasks to appropriate queues.
The Celery app is configured to: - Use JSON serialization for tasks - Limit worker prefetch to 1 task at a time - Restart workers after 20 tasks to prevent memory leaks - Expire results after 2 hours
Queues
- default
General-purpose queue for most tasks.
- cosmology
Queue for cosmology-specific processing tasks.
- bandpass
Queue for bandpass calibration tasks.
- imaging
Queue for imaging pipeline tasks.
- calim00 .. calim10
Per-node queues for NVMe-local subband processing. Each lwacalimNN worker listens on its own
calimNNqueue so that subband data stays on the node’s local NVMe. Tasks are routed dynamically at submit time viatask.apply_async(queue='calim08').
Example
Start a worker for the imaging queue:
celery -A orca.celery worker -Q imaging -c 4
Start a worker on lwacalim08 for subband processing:
celery -A orca.celery worker -Q calim08 --hostname=calim08@lwacalim08 -c 4