Scaling Laravel Horizon Across Multiple ECS Instances: Overcoming Auto-Scaling Pitfalls

Laravel Horizon is a beautiful piece of engineering for managing Redis-backed queues. But when you containerize it and deploy across multiple ECS instances with auto-scaling enabled, the friction becomes immediately apparent. Horizon was designed with single-server deployments in mind, and scaling it horizontally requires deliberate architectural decisions. This guide covers the real-world challenges of Laravel Horizon AWS ECS scaling and the patterns that actually work in production. If your team needs help designing resilient queue infrastructure, consider partnering with a Laravel Agency that specializes in cloud-native Laravel deployments.

The Core Problem: Multiple Instances, Same Jobs

The fundamental issue with scaling Horizon horizontally is that each ECS task spins up its own Horizon process, and each process pulls jobs from the same Redis queue. Without proper coordination, two workers can grab the same job simultaneously—especially when using Redis' blocking pop operations (BRPOP/BLPOP).

Why duplicate processing happens: When an ECS auto-scaling event triggers and a new task spins up, Horizon initializes and begins polling the queue. If a job was already popped but not yet acknowledged by another worker, a brief window exists where both workers claim the same reservation. This is Redis' BRPOP semantics—not a bug, but a feature you need to design around.

APP_NAME Isolation Is Non-Negotiable

Horizon uses the APP_NAME environment variable to namespace its Redis keys, including the processes and supervisors keys that Horizon uses for internal orchestration. If every ECS task shares the same APP_NAME, they'll compete for Horizon's internal locks and tags:

# .env.ecs-task-definition
APP_NAME=laravel-worker-${TASK_ID}
HORIZON_PREFIX=horizon-${TASK_ID}:

Set a unique APP_NAME per ECS task (using an environment variable injected at deploy time, like the ECS task ID or a UUID). This ensures each Horizon instance manages its own supervisor state and doesn't interfere with others.

Split Web, Horizon, and Scheduler into Separate ECS Task Definitions

A common mistake is bundling Horizon with the web server in the same ECS task definition. This means every auto-scaling event for your web tier also spins up a new Horizon worker—even if the queue is empty. The fix is to create three distinct ECS services:

Web service: Runs FPM/Nginx/Ocotane. Scales based on HTTP traffic (CPU/memory/ALB request count).
Horizon service: Runs php artisan horizon. Scales based on queue length (SQS queue depth or Redis list length).
Scheduler service: Runs php artisan schedule:work. Runs as a singleton (desired count = 1) to prevent duplicate scheduled tasks.

Auto-Scaling Horizon Based on Queue Depth

To scale Horizon workers intelligently, you need a custom CloudWatch metric that reflects your actual queue pressure. Here's the approach:

// App\Console\Commands\PublishQueueMetric.php

public function handle()
{
    $queueLength = Redis::connection('horizon')->llen('queues:default:reserved');
    $totalPending = Redis::connection('horizon')->llen('queues:default:pending');

    $client = app('aws.cloudwatch');
    $client->putMetricData([
        'Namespace' => 'Laravel/Horizon',
        'MetricData' => [
            [
                'MetricName' => 'QueueDepth',
                'Value' => $totalPending,
                'Unit' => 'Count',
            ],
        ],
    ]);
}

Then configure an ECS target tracking policy on the Laravel/Horizon QueueDepth metric. This lets you scale Horizon instances up when jobs pile up and scale down to zero when the queue is idle—saving significant costs.

Managing Horizon Tags Across Containers

Horizon's tag system (horizon:tag:{tag}) is stored in Redis and shared across all instances. This is normally fine since tags are read-only metadata. However, if you use Horizon::route() to route jobs based on tags, ensure your routing logic is idempotent—duplicate tag checks across workers are harmless but wasteful.

The safest pattern is to avoid per-instance tag routing and instead use dedicated queues per job type. Route jobs at dispatch time using $job->onQueue('imports'), and configure your Horizon supervisors to watch specific queues exclusively.

Summary

Scaling Laravel Horizon across ECS requires rethinking Horizon's single-server assumptions. Use unique APP_NAME per task, separate your services, scale based on real queue metrics, and design for idempotency. These patterns turn Horizon from a single-server tool into a horizontally-scalable queue worker that handles production traffic gracefully.