orca.autoscaler

Custom Celery autoscaler for the OVRO-LWA Cluster.

Provides a custom autoscaling strategy that considers system load average and available RAM when scaling worker processes.

Classes

CalimScaler

Autoscaler that scales workers based on load and memory constraints.

Classes

CalimScaler

Custom Celery autoscaler for resource-aware worker scaling.

Module Contents

class orca.autoscaler.CalimScaler[source]

Bases: celery.worker.autoscale.Autoscaler

Custom Celery autoscaler for resource-aware worker scaling.

Extends the default Celery autoscaler to consider system load average and available RAM when making scaling decisions. This prevents over-provisioning on shared compute nodes.

Scaling up occurs when:

  • More tasks are queued than processes available

  • System load average is below 20

  • Available RAM exceeds 200 GB

Scaling down occurs when:

  • Fewer tasks than processes, or load is acceptable

Note

Cleanup and retry logic may need further implementation.