Оффлайн
- Регистрация
- 9 Май 2015
- Сообщения
- 1,562
- Баллы
- 155
Ever watched your Laravel app choke during a Black Friday sale or viral marketing campaign? You're not alone. Today, we're diving into the Competing Consumers pattern—a battle-tested approach that'll transform your queue processing from a single-lane road into a superhighway.
The Real-World Pain Point
Picture this: Your SaaS application sends welcome emails, processes payment webhooks, generates PDF reports, and resizes uploaded images. Everything works beautifully... until it doesn't.
Suddenly, 500 users sign up in an hour. Your single queue worker is drowning:
php artisan queue:work
# Processing job 1/847...
# Processing job 2/847...
# Processing job 3/847...
Meanwhile, angry users are tweeting about missing welcome emails, and your Slack is exploding with support tickets.
Enter the Competing Consumers
The Competing Consumers pattern is Laravel's secret weapon for handling unpredictable workloads. Instead of one lonely worker processing jobs sequentially, you deploy multiple worker processes that compete for jobs from the same queue. It's like going from one cashier to ten—all pulling from the same customer line.
Why This Pattern Rocks for Laravel Apps
1. Natural Load Balancing
Laravel's queue system automatically distributes jobs across all available workers. No manual coordination needed—just spin up more workers and watch them compete.
2. Built-in Resilience
If one worker crashes (memory leak, anyone?), the other workers keep chugging along. Your queued jobs don't vanish into the void.
3. Cost-Effective Scaling
Scale up during peak hours, scale down to zero during quiet times. Pay only for what you use.
4. Zero Code Changes
The beauty? Your existing Laravel jobs work without modification. The magic happens at the infrastructure level.
Implementation: From Zero to Hero
Step 1: Set Up Your Queue Driver
First, choose a robust queue driver. While database works for development, production demands something beefier:
# .env
QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
Redis is perfect for competing consumers—it's fast, reliable, and handles concurrent access like a champ.
Step 2: Create a Sample Job
Let's build a realistic example—processing user uploads:
<?php
namespace App\Jobs;
use App\Models\Upload;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Storage;
use Intervention\Image\Facades\Image;
class ProcessImageUpload implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $tries = 3;
public $timeout = 120;
public $backoff = [10, 30, 60];
public function __construct(
public Upload $upload
) {}
public function handle(): void
{
$image = Image::make(Storage::path($this->upload->path));
// Generate thumbnails
$thumbnail = $image->fit(150, 150);
Storage::put(
"thumbnails/{$this->upload->filename}",
$thumbnail->encode()
);
// Optimize original
$optimized = $image->fit(1920, 1080);
Storage::put(
"optimized/{$this->upload->filename}",
$optimized->encode('jpg', 85)
);
$this->upload->update([
'processed' => true,
'thumbnail_path' => "thumbnails/{$this->upload->filename}",
'optimized_path' => "optimized/{$this->upload->filename}",
]);
}
public function failed(\Throwable $exception): void
{
$this->upload->update([
'failed' => true,
'error_message' => $exception->getMessage()
]);
}
}
Step 3: Dispatch Jobs Like a Boss
// In your controller
use App\Jobs\ProcessImageUpload;
public function store(Request $request)
{
$request->validate([
'image' => 'required|image|max:10240'
]);
$path = $request->file('image')->store('uploads');
$upload = Upload::create([
'filename' => $request->file('image')->getClientOriginalName(),
'path' => $path,
'user_id' => auth()->id(),
]);
// Dispatch to queue - let the workers compete!
ProcessImageUpload::dispatch($upload);
return response()->json([
'message' => 'Upload queued for processing',
'upload_id' => $upload->id
]);
}
Step 4: Deploy Your Competing Consumers
Here's where the magic happens. Instead of running one worker, run multiple:
# Terminal 1
php artisan queue:work redis --queue=default --tries=3
# Terminal 2
php artisan queue:work redis --queue=default --tries=3
# Terminal 3
php artisan queue:work redis --queue=default --tries=3
# ... as many as you need!
For production, use Laravel Horizon (Redis) or Supervisor to manage workers:
# /etc/supervisor/conf.d/laravel-worker.conf
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=8
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/worker.log
stopwaitsecs=3600
Notice numprocs=8? That's eight competing workers automatically managed by Supervisor.
Advanced Patterns for Laravel
Horizon: The Premium Experience
Laravel Horizon makes competing consumers beautiful:
composer require laravel/horizon
php artisan horizon:install
Configure workers in config/horizon.php:
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 3,
],
],
],
Horizon automatically scales workers based on queue depth—the perfect competing consumers setup!
Priority Queues
Not all jobs are equal. Process payments before welcome emails:
// High priority
ProcessPayment::dispatch($order)->onQueue('high');
// Normal priority
SendWelcomeEmail::dispatch($user)->onQueue('default');
// Low priority
GenerateMonthlyReport::dispatch()->onQueue('low');
Run workers with priorities:
php artisan queue:work redis --queue=high,default,low
Idempotency: The Secret Sauce
Since multiple workers might grab the same failed job, make your jobs idempotent:
public function handle(): void
{
// Check if already processed
if ($this->upload->processed) {
return;
}
DB::transaction(function () {
// Process image...
// Mark as processed atomically
$this->upload->update(['processed' => true]);
});
}
Dead Letter Queue Handling
Catch poison messages before they tank your workers:
public function retryUntil(): DateTime
{
return now()->addMinutes(30);
}
public function failed(\Throwable $exception): void
{
// Log to monitoring service
report($exception);
// Store in dead letter queue
DeadLetterJob::create([
'job_class' => static::class,
'payload' => serialize($this),
'exception' => $exception->getMessage(),
'failed_at' => now(),
]);
// Notify developers
Notification::send(
User::developers(),
new JobFailedNotification($this, $exception)
);
}
Real-World Scaling Strategy
Here's a battle-tested approach:
Development: 1-2 workers
php artisan queue:work
Staging: 3-5 workers via Supervisor
Production: Horizon with auto-scaling
- Minimum: 2 workers (always available)
- Maximum: 20 workers (during peaks)
- Scale based on queue depth
Black Friday Mode:
- Bump max workers to 50
- Add dedicated high-priority workers
- Monitor with Horizon dashboard
// Check queue health
php artisan queue:monitor redis:default --max=100
// Real-time metrics with Horizon
// Visit /horizon in your browser
// Custom health checks
use Illuminate\Support\Facades\Redis;
$queueSize = Redis::llen('queues:default');
$failedJobs = DB::table('failed_jobs')->count();
if ($queueSize > 1000) {
// Alert: Queue backing up!
}
Common Pitfalls to Avoid
- Memory Leaks: Use --max-time and --max-jobs to restart workers periodically
- Database Connections: Don't store database connections in job properties
- File Storage: Use cloud storage (S3) not local filesystem in multi-server setups
- Timing Issues: Remember jobs run asynchronously—don't expect immediate results
- Jobs must run in strict order (use single worker or serial queues)
- Real-time processing required (use synchronous processing)
- Jobs have complex dependencies on each other (rethink your job design)
The Competing Consumers pattern transforms Laravel's queue system from a potential bottleneck into a scalable, resilient powerhouse. With Redis, Horizon, and proper job design, you can handle traffic spikes that would flatten traditional architectures.
Start small with a few workers, monitor with Horizon, and scale up as needed. Your future self (and your users) will thank you when that next viral moment hits.
Now go forth and queue like a champion!
Pro Tip: Combine this pattern with Laravel's job batching for even more power:
$batch = Bus::batch([
new ProcessImageUpload($upload1),
new ProcessImageUpload($upload2),
new ProcessImageUpload($upload3),
])->then(function (Batch $batch) {
// All jobs completed successfully
})->catch(function (Batch $batch, Throwable $e) {
// First batch job failure
})->finally(function (Batch $batch) {
// Batch has finished
})->dispatch();
Happy queuing!
