Queue Architecture and Configuration

This document covers the queue system architecture, driver configuration, and connection management in Hypervel. It explains how the QueueManager resolves queue drivers, how different driver types (Redis, Database, Sync) are configured, and how queue connections are established.

For information about job dispatching, processing, and lifecycle management, see Job Dispatching and Processing.

Purpose and Scope

The Hypervel queue system provides asynchronous job processing through multiple driver backends. This page documents:

• The QueueManager factory pattern for driver resolution
• Available queue drivers and their characteristics
• Configuration structure for queue connections
• Queue worker process architecture
• Integration with Hyperf's coroutine runtime

Queue Manager Architecture

The queue system follows the Manager pattern (see Manager Pattern for Driver Resolution) to resolve and manage different queue drivers. The QueueManager serves as the central factory for creating queue connection instances based on configuration.

Manager and Driver Resolution

Sources: src/queue/composer.json1-64

The QueueManager is registered as a singleton in the service container and resolves queue connections lazily. Each connection is configured with a driver type and driver-specific options.

Configuration Structure

Queue configuration is typically stored in a config/queue.php file with the following structure:

Configuration Key	Type	Purpose
default	string	Default queue connection name
connections	array	Array of connection configurations
connections.{name}.driver	string	Driver type: redis, database, sync, sqs, beanstalkd
connections.{name}.connection	string	Backend connection name (e.g., Redis connection name)
connections.{name}.queue	string	Default queue name for this connection
connections.{name}.retry_after	int	Job timeout in seconds
connections.{name}.block_for	int	Seconds to block when waiting for jobs (Redis only)
failed.driver	string	Failed job storage driver
failed.database	string	Database connection for failed jobs
failed.table	string	Table name for failed jobs

Sources: src/queue/composer.json22-36

Service Provider Registration

The queue system is registered through the Hypervel\Queue\ConfigProvider class, which integrates with Hyperf's dependency injection system.

Sources: src/queue/composer.json44-46 composer.json261

Queue Drivers

The queue system supports multiple driver implementations, each suited for different use cases and performance requirements.

Redis Driver

The Redis driver stores jobs in Redis lists and is the recommended driver for production deployments with high throughput requirements.

Key Characteristics:

• Blocking Pop: Uses BRPOP for efficient job retrieval with configurable block_for timeout
• Multiple Queues: Supports priority-based queue processing
• Atomic Operations: Leverages Redis atomic operations for job state transitions
• Connection Pooling: Integrates with Hyperf's Redis connection pool for coroutine safety

Configuration Example:

Sources: src/queue/composer.json22-36

Database Driver

The Database driver stores jobs in a database table, making it suitable for applications that need job persistence without additional infrastructure.

Key Characteristics:

• Table Storage: Jobs stored in jobs table with JSON payload
• Transaction Support: Job insertion can be delayed until transaction commits with after_commit
• Simple Setup: Only requires database connection, no additional services
• Limited Throughput: Polling-based retrieval less efficient than Redis

Required Table Structure:

• id (bigint, primary key)
• queue (string, indexed)
• payload (longtext)
• attempts (tinyint)
• reserved_at (int, nullable)
• available_at (int)
• created_at (int)

Configuration Example:

Sources: src/queue/composer.json30 src/queue/composer.json52-54

Sync Driver

The Sync driver executes jobs immediately in the current process without queueing, primarily used for local development and testing.

Key Characteristics:

• Immediate Execution: Jobs run synchronously, blocking until complete
• No Storage: No backend storage required
• Testing Friendly: Simplifies testing by making job execution predictable
• No Worker Required: Does not require separate worker processes

Configuration Example:

Sources: src/queue/composer.json22-36

Optional Drivers

Additional drivers can be enabled by installing their dependencies:

SQS Driver (AWS Simple Queue Service):

• Requires: aws/aws-sdk-php ^3.235.5
• Extension: ext-filter
• Use case: AWS infrastructure, serverless applications
• Configuration: AWS credentials, region, queue URL

Beanstalkd Driver:

• Requires: pda/pheanstalk ^5.0.6|^7.0.0
• Use case: Simple job queueing with priority support
• Configuration: Host, port, queue name

Sources: src/queue/composer.json51-58

Connection Configuration

Multi-Connection Setup

The queue system supports multiple named connections, allowing different queues for different purposes (e.g., high-priority, low-priority, email, notifications).

Sources: src/queue/composer.json1-64

Connection Resolution Flow

Sources: src/queue/composer.json22-36

Queue Worker Configuration

Queue workers are long-running processes that retrieve and process jobs from queue connections. The worker configuration controls process behavior, resource limits, and error handling.

Worker Process Architecture

Sources: src/queue/composer.json22-36 src/queue/composer.json51-58

Worker Configuration Options

Workers are typically started via the queue:work command with various options:

Option	Type	Default	Purpose
--connection	string	default	Queue connection to use
--queue	string	default	Queue name(s) to process (comma-separated for priority)
--daemon	flag	false	Run worker in daemon mode
--once	flag	false	Process one job and exit
--delay	int	0	Seconds to delay failed jobs
--memory	int	128	Memory limit in MB
--timeout	int	60	Job execution timeout in seconds
--sleep	int	3	Seconds to sleep when no jobs available
--tries	int	1	Number of times to attempt a job
--max-jobs	int	0	Process N jobs then exit (0 = infinite)
--max-time	int	0	Run for N seconds then exit (0 = infinite)

Sources: src/queue/composer.json51-58

Process Isolation and Coroutine Safety

The worker leverages Hyperf's coroutine architecture while maintaining process isolation:

Sources: src/queue/composer.json24-25

Failed Job Handling

Failed jobs are stored separately for analysis and manual retry.

Failed Job Storage Configuration

Failed Jobs Table Structure:

• id (bigint, primary key)
• uuid (string, unique)
• connection (text)
• queue (text)
• payload (longtext)
• exception (longtext)
• failed_at (timestamp)

Sources: src/queue/composer.json52-54

Job Serialization and Encryption

The queue system uses Laravel's serializable closures and supports encrypted payloads:

Dependencies:

• laravel/serializable-closure ^1.2.2: Enables closure serialization for queue jobs
• hypervel/encryption ^0.3: Provides optional payload encryption

Use Cases:

• Serializing job closures for inline jobs
• Encrypting sensitive data in job payloads
• Supporting job payloads with complex object graphs

Sources: src/queue/composer.json31 src/queue/composer.json36

Integration with Other Systems

The queue system integrates with several other framework components:

Sources: src/queue/composer.json1-64 src/bus/composer.json54 src/notifications/composer.json40

ConfigProvider Registration

The queue package is registered in Hyperf's configuration system through its ConfigProvider:

Package Registration Flow:

1. composer.json declares Hypervel\Queue\ConfigProvider in extra.hyperf.config
2. Hyperf scans and loads ConfigProvider during bootstrap
3. ConfigProvider registers queue services in the container
4. QueueManager becomes available for dependency injection

Sources: composer.json261 src/queue/composer.json44-46