Database and Beanstalkd Drivers

Purpose and Scope

This document covers the Database and Beanstalkd queue driver implementations. These drivers provide persistent queue storage using relational databases and the Beanstalkd work queue service respectively.

The Database driver stores jobs in a database table, making it suitable for applications that want queue persistence without additional infrastructure dependencies. The Beanstalkd driver integrates with the Beanstalkd work queue service, providing a dedicated queueing system with advanced features like tube management and job prioritization.

For configuration details of other drivers, see SQS Driver and Redis Driver. For general driver configuration patterns, see Driver Configuration.

Sources: publish/queue.php1-144 composer.json1-65

---

Database Driver

Overview

The Database driver (DatabaseQueue) stores queued jobs in a relational database table. This driver is implemented through the DatabaseConnector which instantiates the DatabaseQueue class. Jobs are persisted as serialized payloads in database rows, with metadata for scheduling, retry tracking, and queue assignment.

The Database driver requires the ext-pdo PHP extension and uses Hyperf's database connection infrastructure (hyperf/db-connection).

Sources: publish/queue.php58-65 composer.json30-52

Configuration

The Database driver is configured in the connections.database section of the queue configuration file. The default connection name is set to 'database'.

Configuration Key	Environment Variable	Default	Description
driver	-	'database'	Must be set to 'database'
connection	DB_QUEUE_CONNECTION	null	Database connection name from database config. If null, uses default connection
table	DB_QUEUE_TABLE	'jobs'	Name of the database table storing queue jobs
queue	DB_QUEUE	'default'	Default queue name for this connection
retry_after	DB_QUEUE_RETRY_AFTER	90	Seconds after which reserved jobs are considered abandoned and made available again
after_commit	-	false	Whether to dispatch jobs only after database transaction commits

Configuration Example:

Sources: publish/queue.php58-65

Database Schema

The Database driver requires a table to store job records. While the actual schema is defined in database migrations (not included in the provided files), the configuration indicates the following structure is expected:

Column	Type	Description
id	Primary Key	Auto-incrementing job identifier
queue	String	Queue name (e.g., 'default', 'emails', 'processing')
payload	Text/JSON	Serialized job payload containing class name, data, and metadata
attempts	Integer	Number of times this job has been attempted
reserved_at	Timestamp (nullable)	When the job was reserved by a worker (null if available)
available_at	Timestamp	When the job becomes available for processing
created_at	Timestamp	When the job was initially queued

Jobs transition through states by updating these columns:

• Available: reserved_at is NULL and available_at <= now()
• Reserved: reserved_at is set to current timestamp
• Delayed: available_at is set to future timestamp
• Abandoned: reserved_at is older than retry_after seconds

Sources: publish/queue.php58-65

Use Cases

The Database driver is recommended for:

1. Simple deployments: Applications that want job persistence without deploying additional infrastructure like Redis or Beanstalkd
2. Development environments: Easy setup using SQLite or existing database connections
3. Audit requirements: Jobs stored in database can be easily queried, reported on, and audited using SQL
4. Transaction-aware dispatching: Jobs can be deferred until database transactions commit using after_commit option
5. Small to medium throughput: Suitable for applications processing hundreds to low thousands of jobs per hour

The Database driver may not be optimal for:

• Very high throughput scenarios (thousands of jobs per second)
• Applications requiring advanced queue features like prioritization
• Scenarios where database load is a concern

Sources: publish/queue.php58-65 composer.json52

---

Beanstalkd Driver

Overview

The Beanstalkd driver (BeanstalkdQueue) integrates with the Beanstalkd work queue service, a fast, distributed, in-memory workqueue service. The driver is implemented through the BeanstalkdConnector which instantiates the BeanstalkdQueue class.

Beanstalkd provides advanced features including:

• Tubes: Named queues for job organization
• Job prioritization: Jobs can have priority values
• Buried jobs: Failed jobs can be buried for later inspection
• Time-to-run (TTR): Automatic job timeout handling

The Beanstalkd driver requires the pda/pheanstalk package (version ^5.0.6 or ^7.0.0), a PHP client library for the Beanstalkd protocol.

Sources: publish/queue.php67-80 composer.json58

Configuration

The Beanstalkd driver is configured in the connections.beanstalkd section of the queue configuration file.

Configuration Key	Environment Variable	Default	Description
driver	-	'beanstalkd'	Must be set to 'beanstalkd'
host	BEANSTALKD_QUEUE_HOST	'localhost'	Hostname or IP address of the Beanstalkd server
queue	BEANSTALKD_QUEUE	'default'	Default tube (queue) name
retry_after	BEANSTALKD_QUEUE_RETRY_AFTER	90	Time-to-run (TTR) in seconds for jobs
block_for	-	0	Seconds to block waiting for jobs (0 = non-blocking)
after_commit	-	false	Whether to dispatch jobs only after database transaction commits
pool.min_objects	-	1	Minimum number of pooled connections
pool.max_objects	-	10	Maximum number of pooled connections
pool.wait_timeout	-	3.0	Seconds to wait for available connection
pool.max_lifetime	-	60.0	Maximum lifetime of pooled connection in seconds

Configuration Example:

Sources: publish/queue.php67-80

Connection Pooling

The Beanstalkd driver utilizes connection pooling to efficiently manage TCP connections to the Beanstalkd server. This is implemented through the PoolProxy pattern mentioned in Diagram 2 of the high-level architecture.

Connection pooling provides:

1. Connection Reuse: Existing connections are reused rather than creating new TCP connections for each operation
2. Concurrency Management: Multiple coroutines can share connections from the pool
3. Resource Limits: max_objects prevents excessive connection creation
4. Automatic Cleanup: Connections older than max_lifetime are recycled

The pool configuration ensures that:

• At least min_objects (1) connections are maintained
• No more than max_objects (10) connections are created
• Workers wait up to wait_timeout (3.0s) for an available connection
• Connections are recycled after max_lifetime (60.0s)

Sources: publish/queue.php74-79

Use Cases

The Beanstalkd driver is recommended for:

1. High throughput scenarios: Beanstalkd is designed for fast, in-memory job processing
2. Distributed systems: Multiple workers and applications can connect to a central Beanstalkd server
3. Job prioritization: When jobs need to be processed in priority order
4. Dedicated queue infrastructure: When you want a purpose-built queueing system
5. Simple deployment: Beanstalkd is lightweight and easy to deploy
6. Tube-based organization: When you want to organize jobs into separate tubes (queues)

The Beanstalkd driver may not be optimal for:

• Very small applications where deploying Beanstalkd adds unnecessary complexity
• Scenarios requiring persistence across server restarts (Beanstalkd is in-memory by default)
• Windows environments (Beanstalkd is primarily a Unix-based service)

Sources: publish/queue.php67-80 composer.json58

---

Driver Comparison

The following table compares the key characteristics of the Database and Beanstalkd drivers:

Feature	Database Driver	Beanstalkd Driver
Storage	Relational database (persistent)	In-memory (optional persistence)
Dependencies	ext-pdo, existing database	pda/pheanstalk, Beanstalkd server
Connection Pooling	No	Yes (configured)
Infrastructure	No additional services	Requires Beanstalkd server
Performance	Moderate (depends on DB)	High (in-memory)
Prioritization	No	Yes (native support)
Transaction Awareness	Yes (after_commit)	Yes (after_commit)
Audit Trail	Easy (SQL queries)	Limited (external tools)
Persistence	Always persistent	In-memory (configurable)
Best For	Simple deployments, auditing	High throughput, distributed systems

Sources: publish/queue.php58-80 composer.json52-58

---

Configuration Flow

The following diagram illustrates how both drivers are instantiated through the configuration and connector system:

Sources: publish/queue.php58-80

---

Environment Variable Reference

Both drivers support environment-based configuration for deployment flexibility:

Database Driver Environment Variables

Beanstalkd Driver Environment Variables

Sources: publish/queue.php60-71

---

Integration with Queue System

Both drivers integrate seamlessly with the broader queue system architecture:

1. QueueManager: Both drivers are instantiated through the QueueManager factory using their respective connectors
2. Worker System: Workers can process jobs from either driver using the same Worker class (see Worker Class)
3. Console Commands: All queue commands (queue:work, queue:listen, etc.) support both drivers (see Console Commands)
4. Failed Job Handling: Both drivers integrate with the FailedJobProvider system (see Failed Job Management Commands)
5. Event System: Jobs processed from either driver dispatch the same lifecycle events (see Events and Extension Points)
6. Transaction Awareness: Both support the after_commit configuration for transaction-aware dispatching (see Transaction-Aware Dispatching)

Sources: publish/queue.php58-80