reproq-django

Reproq Django 🚀

License: Apache 2.0 Documentation

Deterministic Background Tasks for Django 6.0+ powered by Go.

Reproq is a production-grade tasks backend that combines the ease of Django with the performance and reliability of the Reproq Worker, a high-performance execution engine written in Go.


🤝 Relationship with Reproq Worker

Reproq is split into two specialized components:

  1. Reproq Django (this repo): Task definition, enqueueing, results, and the Admin dashboard.
  2. Reproq Worker: A Go binary that claims and executes tasks from Postgres.

Key Features

✅ Compatibility

Component Supported
Django Python 3.12–3.14 supports Django 6.0.10 or later in the 6.0 series, or Django 6.1.3 or later in 6.x. Python 3.15 requires Django 6.1.3 or later in 6.x.
Python 3.12–3.15, checked locally with isolated locked environments
Reproq Worker Release tested with this SDK (install/upgrade via uv run python manage.py reproq install / upgrade)

Use uv run python manage.py reproq doctor to verify schema, worker binary, and DSN alignment. The Go process reads its queue connection from REPROQ_DATABASE_URL or explicit configuration/flags. Generic DATABASE_URL remains Django’s business connection; the SDK derives queue connections from the selected Django alias and preserves that business environment for task execution. Queue names must route to the selected alias through REPROQ_QUEUE_DATABASES.

The package requirements exclude the affected Django versions in the November 3 security advisory. The repository lockfile records the dependency versions used by the locked Python checks. Django 6.1.3 added official Python 3.15 support. Package builds require setuptools>=84.1.0 and wheel>=0.46.2.


⚡ Quickstart

These instructions target SDK 0.1.536 and Worker 0.0.148. Source preparation, artifact qualification and publication are separate. Use the PyPI and GitHub installation commands only when the matching artifacts are published; for an unpublished candidate, use the reviewed local SDK and Worker.

SDK 0.1.536 defaults to Reproq Worker v0.0.148 with worker_python_bin:1 and the migration-owned schema-v2/executor-protocol2 contract. Worker v0.0.145 also implements that contract. For upgrades from older queue schemas, follow the historical-schema adoption and draining requirements below.

1. Install

After publication, install SDK 0.1.536 from PyPI:

uv add 'reproq-django==0.1.536'

After publication, alternatively use the version-pinned GitHub release tag:

uv add "reproq-django @ git+https://github.com/adpena/reproq-django.git@v0.1.536"

For reviewed paired development, add the local SDK instead:

uv add --editable /path/to/reviewed/reproq-django

2. Configure Settings

Add reproq_django to your INSTALLED_APPS and configure the TASKS backend:

INSTALLED_APPS = [
    ...,
    "reproq_django",
]

TASKS = {
    "default": {
        "BACKEND": "reproq_django.backend.ReproqBackend",
        "OPTIONS": {
            "DEDUP_ACTIVE": True,
            "TIMEOUT_SECONDS": 900,
            "MAX_ATTEMPTS": 3,
        }
    }
}

3. Define a Task

Use the Django 6.0 Tasks API with a @task decorator:

from django.tasks import task

@task(queue_name="default", priority=0)
def send_welcome_email(user_id: int) -> str:
    # business logic here
    return f"Email sent to {user_id}"

Bootstrap writes a config file, installs the worker binary, and applies committed Django migrations. This package’s migrations own the queue schema.

uv run python manage.py reproq init

For reviewed paired development, select the local v0.0.148 worker and skip the hosted download:

export REPROQ_WORKER_BIN=/absolute/path/to/reviewed/reproq
uv run python manage.py reproq init --skip-install

Use --format toml or --skip-install as needed. --skip-migrate skips the project-wide invocation; --skip-worker-migrate skips the separate ReproQ-only invocation. Both invocations use the same committed Django migrations. Use both flags when migration execution must be deferred.

5. Download the Compatible Worker

SDK 0.1.536 selects the compatible v0.0.148 tag by default:

uv run python manage.py reproq install

The installer detects the OS/architecture and fetches that tag’s pre-built binary from GitHub. Reviewed paired development can use the local selector above.

Downloads must match the publisher’s checksum sidecar or GitHub release-asset SHA-256 digest before the binary is made executable or its version is queried. For reproducible deployment, pin both a release tag and the digest of the binary for the target OS and architecture. For v0.0.148, use the reviewed digest of that release’s binary for the selected platform:

uv run python manage.py reproq install --tag v0.0.148 --sha256 <platform-binary-sha256>

The same options work with init and upgrade. An explicit digest disables download-to-source fallback. Missing or mismatched integrity metadata stops installation and preserves the existing binary; it never triggers a source build. Explicit –build still supports local development and can also verify an expected digest. Successful installation replaces the binary atomically.

Run the complete locked local lint and test suite with make check. The optional hosted CI workflow is manual and calls the same command. Release builds are also manual: they require an existing reviewed tag matching both package version declarations, run the complete locked checks, and create a draft release. Existing releases and their assets are never overwritten by that workflow. Local builds use uv build after make check; publishing and tag retention remain separate reviewed actions.

6. Run Migrations

uv run python manage.py migrate

The historical reproq migrate-worker --database <alias> command now delegates to the same Django migrations for ReproQ. It does not execute another SQL schema or repair tables. The standalone Go worker requires the supported migration-owned schema contract before claiming tasks or scheduling work.

Stop legacy workers and drain all RUNNING tasks before the forward contract migration. Its validated lease constraint makes old claim/completion SQL fail closed. The migration preserves historical task JSON and hashes, adds nullable canonical-spec fields with spec_version=0, and leaves canonical schedule reconciliation to an explicit later operation. Schema rollback requires a verified database backup; this migration deliberately refuses a destructive reverse operation.

Existing databases with legacy Go-created tables or a partially applied Django squash require explicit adoption before ordinary migration. Inspect the selected database first:

uv run python manage.py reproq adopt-schema --database <alias> --inspect

Adoption accepts only checked-in semantic catalog profiles and exact supported historical ledger sets. It checks column types/nullability/defaults, constraints and validation state, indexes, triggers/functions and sequence ownership. Unknown or partial schemas fail without changing migration history. The exact historical Django baseline plus the Go baseline’s auxiliary tables and eight additional indexes is also supported. If the baseline and its replacement records are already coherent, use ordinary forward Django migration without re-adopting history. The physical catalog and forward data checks still apply; a migration record alone never certifies the schema. After reviewing the inspection, supply its exact catalog digest and a new audit file:

uv run python manage.py reproq adopt-schema --database <alias> \
  --expected-catalog-sha256 <inspected-sha256> --audit-output <new-audit-file.json>
uv run python manage.py reproq migrate-worker --database <alias>

Adoption reconciles the replacement markers covered by the published squash; it does not invent code for its never-committed third replacement member. The receipt records before/after history and catalog identity. Adoption requires an outermost autocommit boundary; it refuses nested or manually managed transactions. An interrupted prepared, unconfirmed, or commit_unknown receipt requires independent ledger verification. Losing a COMMIT acknowledgement does not prove rollback. Receipt contents are fsynced and directory entries are flushed where the platform supports it; database and filesystem commits are separate resources. There is no --fake or table-existence shortcut. The known raw Go profile’s unvalidated task-path check requires valid existing rows and is validated by the forward Django migration.

The currently qualified PostgreSQL catalog profiles use the public schema. Other queue namespaces require their own qualified catalog support; the guard fails closed rather than treating a migration record as schema proof. The forward migration also verifies all four core tables and any preexisting auxiliary tables before establishing the new contract. Adoption, migration and runtime checks bind all queue, marker and history relations to ordinary permanent public tables. Temporary objects and shadow schemas cannot substitute for the validated tables, even on a public-first path.

7. Start the Worker

uv run python manage.py reproq worker

8. (Optional) Schedule Periodic Tasks

You have two options (choose one):

Option A: Run Beat (one process per database)

uv run python manage.py reproq beat

Option B: Use Postgres-native scheduling (pg_cron)

uv run python manage.py reproq pg-cron --install

pg_cron requires the Postgres extension to be enabled (see docs/deployment.md).


🧰 Management Commands

Run uv run python manage.py reproq <subcommand> to manage the worker and day-to-day ops. Full reference (examples + exit codes): docs/cli.md.


✅ Why Reproq (vs Celery, RQ, Huey)

Reproq is built for teams who want deterministic background tasks without adding Redis or RabbitMQ.

If you need complex routing, multi-broker support, or huge existing Celery ecosystems, Celery may still fit better. Reproq prioritizes clarity, determinism, and low operational overhead.


📚 API Reference

Reproq fully implements the Django 6.0 Tasks API.

Defining Tasks

Use the standard @task decorator. Reproq respects queue_name and priority.

from django.tasks import task

@task(queue_name="high-priority", priority=100)
def send_welcome_email(user_id):
    # logic here
    return f"Email sent to {user_id}"

You can also annotate a task with a concurrency limit:

from reproq_django.concurrency import limits_concurrency

@limits_concurrency(lambda user_id: f"user:{user_id}", to=1)
@task(queue_name="high-priority", priority=100)
def send_welcome_email(user_id):
    return f"Email sent to {user_id}"

Enqueuing Tasks

Use Django’s standard task API. Every enqueue keyword is a function argument; ReproQ does not reserve names such as priority, run_after or concurrency_key. Delivery options belong on the task, backend configuration or decorators:

from datetime import timedelta
from django.utils import timezone

result = send_welcome_email.enqueue(123)
result = send_welcome_email.using(
    priority=42,
    run_after=timezone.now() + timedelta(minutes=10),
).enqueue(123)

Use @limits_concurrency(lambda user_id: f"user:{user_id}", to=2) for a shared key, as shown above. Named task function attributes can supply custom lock/key options; they never consume function kwargs. Direct, bulk and periodic enqueue use the same typed arguments/spec builder. Strict integer budgets and optional string keys are validated before database writes; bool/float coercion is rejected. Concurrency capacity is the strongest positive candidate/active limit for a shared key. A zero contributes no limit but does not bypass an active key’s cap.

The former delivery-in-function-kwargs extension is retired. Migrate delivery priority/run_after to Task.using and shared concurrency to the decorator; leave same-named function parameters intact.

Async Contexts (ASGI)

If you are in an async view or task producer, use aenqueue() to avoid blocking:

result = await send_welcome_email.aenqueue(123)

In sync/Wsgi code, continue to use enqueue().

Task Context + Metadata

If your task needs context (result id, attempt, metadata), use takes_context=True.

from django.tasks import task

@task(takes_context=True)
def process_upload(context, upload_id):
    context.metadata["stage"] = "starting"
    context.save_metadata()
    # work...
    context.metadata["stage"] = "done"
    context.save_metadata()

Metadata is stored in task_runs.metadata_json and surfaced in task results.

Bulk Enqueuing

Use bulk_enqueue for batches of jobs. It groups database inserts and uses the same task options, argument codec and canonical spec builder as enqueue. Choose a batch size that fits the producer’s memory budget.

from django.tasks import task_backends
from datetime import timedelta
from django.utils import timezone

backend = task_backends["default"]
jobs = []
scheduled_at = timezone.now()

for i in range(1000):
    # (task_func, args, kwargs)
    jobs.append((
        send_welcome_email.using(run_after=scheduled_at + timedelta(seconds=i)),
        (i,),
        {},
    ))

backend.bulk_enqueue(jobs)

⏰ Periodic Tasks

Register source intent through the SDK, then run a scheduler. Migrations do not seed or reset operator schedules. Direct ORM creation leaves a legacy/unassigned row that requires explicit reconciliation; it cannot silently become runnable.

from myapp.tasks import nightly_cleanup
from reproq_django.recurring import register_periodic

register_periodic(
    name="Nightly cleanup",
    schedule="CRON_TZ=America/Chicago 0 2 * * *",
    task=nightly_cleanup,
    queue_name="maintenance",
    scheduler_provider="go",  # or "pg_cron" on a configured database
)

The same canonical SDK serializer handles direct, bulk and periodic task specs, including typed arguments, context, retry/timeout budget, routing and provenance. Registration preserves an existing definition’s enabled, next-run and last-run state. Resetting schedule state or assigning another provider is explicit. The admin permits pausing/resuming definitions; canonical spec fields are read-only.

Each definition belongs to one provider. Long-running Go beat and one-shot Go scheduling use the same implementation:

The read-only stats API and reproq status distinguish disabled definitions, registration problems, past-due Go cursors measured against the database clock, and managed beat being disabled by configuration. Worker records are counted separately as alive or stale against each selected database’s statement clock; their heartbeats do not prove scheduler execution. Admin status uses that same classification. The API’s existing worker_health.dead key counts stale records. Explicit admin and CLI worker pruning also compare against the selected database clock, preserving their existing retention thresholds and selections. Neither status nor doctor runs pruning or changes task history. reproq doctor --strict also fails on these scheduling warnings. A past-due cursor does not identify the cause, and native pg_cron uses its own occurrence clock. Prestart reproq check does not wait for due cursors to advance: beat must first be allowed to start. LOW_MEMORY_MODE=1 does not disable the current supervisor’s explicitly configured beat process.

Set secrets in the environment and reference them by variable name in a custom REPROQ_PRESTART_CMD. The supervisor discards custom prestart stdout and stderr, including Bash syntax diagnostics that can repeat the command. Supervisor status and output from the built-in prestart check remain visible.

uv run python manage.py reproq beat
# Or invoke this from an external timer; do not also run the long-running beat:
uv run python manage.py reproq schedule

Go coalesces an overdue definition to one occurrence at its stored nominal next_run_at, then advances beyond the database clock. It does not invent every missed tick. The occurrence identity (periodic_name, scheduled_for) remains unique after completion or failure, while separate occurrences may overlap subject to the configured concurrency key/limit.

The optional native provider uses the same migration-owned PostgreSQL enqueue primitive and the exact stored SDK bytes. It needs an administrator-installed pg_cron extension and its named-job API; the SDK does not create extensions or functions. Assign definitions to pg_cron, then reconcile only their owned jobs:

uv run python manage.py reproq pg-cron --install --dry-run
uv run python manage.py reproq pg-cron --install
uv run python manage.py reproq pg-cron --remove

--if-supported skips only an unavailable extension/database capability. It does not hide invalid definitions or mismatched zones. The portable grammar uses ASCII numbers/names, lists, ascending ranges and positive wildcard/range steps; Weekdays are0–6 (Sunday0). L/#/wrapping ranges, scalar steps and macros are rejected. Bare cron expressions mean UTC; CRON_TZ/TZ selects a named zone. Native cron.timezone must match that zone (UTC/GMT aliases are accepted). Go supports per-definition zones; pg_cron uses its database server’s configured zone. Native occurrences are identified by the UTC minute when their SQL starts, not an asserted nominal tick or a replay of missed ticks. Native scheduler delays/catch-up follow pg_cron; the SDK does not synthesize a misleading next_run_at for it. Inspect native job run history to verify actual execution. See deployment for prerequisites.

Before upgrading a database with the old native helper, explicitly remove its known owned jobs with the reviewed SDK command while legacy schedulers are stopped. Forward migration0003 owns retiring that exact legacy helper and installing the shared primitive; unknown functions/dependencies/jobs fail closed.

Code-defined registrations can use @recurring with Django’s standard @task:

from django.tasks import task
from reproq_django.recurring import recurring

@recurring(schedule="CRON_TZ=America/Chicago 0 9 * * *", key="daily_report", args=(42,))
@task(queue_name="maintenance")
def send_report(account_id):
    ...

Run uv run python manage.py reproq sync-recurring explicitly after imports/deployment. Named legacy rows can be canonicalized with reconcile-periodic --name NAME (--dry-run previews with transaction rollback). Neither runs on post_migrate. To run a task immediately, call its standard enqueue() method; this does not rewrite its recurring schedule or pretend to be a scheduled occurrence.


⚙️ Configuration

Configure Reproq behavior via the TASKS setting in settings.py.

TASKS = {
    "default": {
        "BACKEND": "reproq_django.backend.ReproqBackend",
        "OPTIONS": {
            # Deduplication (Default: True)
            # If True, enqueuing a task with the exact same arguments as a 
            # READY/RUNNING task will return the existing result_id.
            "DEDUP_ACTIVE": True,

            # Execution Timeout (Default: 900)
            # Max seconds a task can run before being killed by the worker.
            "TIMEOUT_SECONDS": 900,

            # Retry Limit (Default: 3)
            # Max number of attempts for a task.
            "MAX_ATTEMPTS": 3,

            # Expiry (Optional)
            # If set, tasks not picked up by this time will be marked expired.
            "EXPIRES_IN": timedelta(hours=24),
            
            # Provenance (Optional)
            # Metadata stored with the task for auditing.
            "CODE_REF": "git-sha-or-version",
            "PIP_LOCK_HASH": "hash-of-dependencies",
        }
    }
}

Worker config files (reproq.yaml/reproq.toml) are optional. Precedence is: defaults < config file < env vars < CLI flags. Django management commands derive an empty DSN from the selected queue database alias and validate explicit file/environment targets against it. The standalone Go CLI also accepts --dsn.

Executor Python selection

This interpreter binding requires the matching Go/SDK pair and queue migrations described in Quickstart. Its local checks do not establish application workload, native-platform capacity or production deployment qualification.

Run management commands through uv run. The Django worker defaults its executor to that selected sys.executable, passed explicitly through Go’s --python-bin. An explicit worker.python_bin in a config file or --python-bin command option takes precedence. Generated configuration omits this optional value so it remains portable between environments. config --explain, check, and doctor show the effective executable and its source. The worker must advertise worker_python_bin operation version1; the SDK checks this before installation or foreground startup. The standalone Go default remains python3.

Multi-Database Queues

Route specific queues to different Django database aliases.

DATABASES = {
    "default": {...},
    "queues": {...},
}

REPROQ_QUEUE_DATABASES = {
    "high-priority": "queues",
    "bulk-*": "queues",  # glob patterns supported
}

Use REPROQ_DEFAULT_DB_ALIAS to change the fallback alias (defaults to "default"). When multiple queue databases are configured, task result IDs are prefixed with a canonically percent-encoded database alias (queues:123). Multiple databases cannot disable this prefix. Unsupported encoded alias lengths are rejected before producer writes, keeping native result IDs below 64 characters without truncation or guessed routing. To route all Reproq tables to a non-default alias in ORM reads/writes (admin, health checks, etc.), add the router:

DATABASE_ROUTERS = ["reproq_django.db_router.ReproqRouter"]

Run one worker/beat per database:

uv run python manage.py reproq worker --database queues --queues high-priority,bulk-1
uv run python manage.py reproq beat --database queues

Use REPROQ_STATS_DATABASES=["*"] to aggregate stats across all queue databases.


🧪 Development

.python-version selects Python 3.14.10 for local development. The bootstrap installs the locked development dependencies; the compatibility checks still cover Python 3.12–3.15.

bash scripts/dev_bootstrap.sh
uv run pytest

Always run tests with uv run pytest so dependencies and settings stay consistent.


⛓ Workflows (Chains & Groups)

Reproq supports complex task dependencies.

Chains (Sequential)

Execute tasks one after another. If a task fails, the chain stops.

from reproq_django.workflows import chain

# task_a -> task_b -> task_c
c = chain(
    (task_a, (1,), {}),
    task_b, # no args
    (task_c, (), {"param": "val"})
)
results = c.enqueue()
# results[0] is READY, results[1..] are WAITING

Groups (Parallel)

Execute tasks in parallel.

from reproq_django.workflows import group

g = group(
    (resize_image, ("img1.jpg",), {}),
    (resize_image, ("img2.jpg",), {}),
)
results = g.enqueue()

Chords (Group + Callback)

Run a callback once a group finishes.

from reproq_django.workflows import chord

group_results, callback_result = chord(
    (resize_image, ("img1.jpg",), {}),
    (resize_image, ("img2.jpg",), {}),
    callback=notify_done,
).enqueue()

The callback runs only after all group tasks succeed. Failed tasks leave the callback waiting. Failures mark the workflow as failed and the callback will not run.

Automatic retries keep dependents waiting until the task succeeds or exhausts its budget. Terminal workflow members cannot be manually reset in place: that would break dependency order or aggregate counters. Enqueue a fresh workflow to recover the workflow, or explicitly replay one task as a detached new task.


🔁 Retries & Backoff

Optional arrival notifications (Worker 0.0.148 / SDK 0.1.536)

The paired Worker supports --wait-mode notify, REPROQ_WAIT_MODE=notify or worker.wait_mode in its config. The Django foreground command checks exact notification_wait: 1 capability before starting notify mode. Published Worker 0.0.147 does not support it. Polling remains the default.

Set backend OPTIONS["WAKE_NOTIFICATIONS"] = True (a boolean) to send empty PostgreSQL hints after committed enqueue operations. Single/deduplicated enqueue sends one hint; bulk/workflows send one per affected database, with mixed-backend workflows opting in when any node opts in. Rollback discards callbacks, public per-result signals retain their behavior, and a failed hint logs a warning without rejecting already-committed work. No producer thread is created.

Notify mode adds one session connection, requires session-compatible PostgreSQL, and is unsuitable for PgBouncer transaction pooling. SQL claims remain the authority. Recovery intervals remain unchanged for older producers, pg_cron, Go transitions, retries, delayed work and lost hints. Running cancellation stays heartbeat-driven. See the full contract and measurement limits.

Retries are managed by the Go worker. When a task fails and attempts remain, it is re-queued with an exponential backoff:

The worker updates run_after on the failed task, and the backend will only claim it after that timestamp.


🚦 Rate Limiting

Reproq Worker enforces token bucket limits stored in the rate_limits table. You can manage these from Django Admin or via the worker CLI.

Keys:

Example:

reproq limit set --key queue:default --rate 5 --burst 10

Defaults: global rate limiting is disabled until you set a positive rate.


🧰 Worker CLI Ops

The Go worker binary includes operational commands you can run directly:

# Request cancellation of a running task
reproq cancel --dsn "..." --id 12345

# Inspect failed tasks
reproq triage list --dsn "..." --limit 50

🖥 Management Commands

The uv run python manage.py reproq command is your Swiss Army knife.

Subcommand Description
init Bootstraps Reproq in the current project.
worker Starts the Go worker. Flags: --config, --concurrency (default 10), --queues, --allowed-task-modules, --logs-dir, --payload-mode, --python-bin, --metrics-port, --metrics-addr, --metrics-auth-token, --metrics-allow-cidrs, --metrics-tls-cert, --metrics-tls-key, --metrics-tls-client-ca, --database. Auto-configures allow-list when unset (unless config file is used).
beat Starts the scheduler. Flags: --config, --interval (default 30s), --once, --database.
schedule Enqueue due periodic tasks once and exit (alias for beat --once).
pg-cron Syncs Postgres-native schedules. Flags: --install (default), --remove, --prefix, --dry-run, --database.
install Downloads/builds the worker binary.
upgrade Upgrades the worker binary and optionally runs migrate-worker.
migrate-worker Applies committed ReproQ Django migrations on the selected database alias.
adopt-schema Inspects or explicitly adopts a fully verified known legacy PostgreSQL schema.
check Validates binary path, DB connection, and schema health.
doctor Validates DSN, schema, worker binary, and allowlist; --strict fails on warnings.
config Prints effective worker/beat config; use --explain for precedence.
allowlist Prints ALLOWED_TASK_MODULES or writes them to a config file with --write.
sync-recurring Sync code-defined recurring schedules into periodic_tasks.
pause-queue Pause a queue (prevents new claims).
resume-queue Resume a paused queue.
logs Prints logs for a task run using logs_uri (supports aliased result IDs).
cancel Requests cancellation of a task run by result ID (supports aliased result IDs).
reclaim Requeue or fail tasks with expired leases (--database / --all-databases).
prune-workers Delete workers not seen recently (--database / --all-databases).
prune Delete terminal task history by status/age in bounded batches (--database / --all-databases).
stats / status Shows task counts by status and active workers (--database / --all-databases).
systemd Generates systemd service files for production.

📊 Stats API (JSON)

When you include reproq_django.urls in your project, GET /stats/ returns JSON task counts, per-queue task counts, worker records, and periodic task schedules. Access is granted to staff sessions, a signed TUI JWT, or METRICS_AUTH_TOKEN as a bearer token. Additional fields include worker_health, queue_controls, scheduler, and per-database rollups when REPROQ_STATS_DATABASES is set.

🧭 TUI Integration

Set METRICS_AUTH_TOKEN to enable the TUI login flow and sign TUI JWTs. This token is also accepted as a bearer token for /reproq/stats/ and is forwarded to the worker metrics proxy endpoints. If you do not want to expose SSE, set REPROQ_TUI_DISABLE_EVENTS=1 to omit the /reproq/tui/events/ stream from the TUI config payload. If METRICS_AUTH_TOKEN is unset, the TUI auth endpoints return 404.

Set LOW_MEMORY_MODE=1 to disable the worker proxy endpoints (/reproq/tui/metrics/, /reproq/tui/healthz/, /reproq/tui/events/). The endpoints return a 503 with a short hint while low-memory mode is enabled, and events are omitted from the TUI config payload.

Set REPROQ_MEMORY_LOG_INTERVAL=60s to emit periodic memory logs from the Django process (includes RSS on Linux).

Pass the TUI JWT in the Authorization: Bearer ... header when accessing these endpoints programmatically.


🧾 Worker/Beat Config Files

The Go worker/beat support YAML/TOML config files. manage.py reproq worker and manage.py reproq beat will load a config file when --config or REPROQ_CONFIG is set. Otherwise, they look for reproq.yaml, reproq.yml, reproq.toml, .reproq.yaml, .reproq.yml, or .reproq.toml in the current working directory, even when other flags are present. Actual CLI options override environment and config values. Empty queue DSNs inherit from the selected Django queue database; explicit targets must match that alias.

The canonical templates are TOML and YAML. reproq init reads these packaged resources; files in the current directory cannot override them.

Queue selection uses --queues (comma-separated). The worker’s legacy --queue flag has been retired; queue-control commands retain their positional queue name.


⚖️ Scaling: Workers vs Concurrency

Start with one task slot on a small shared web instance. Measure combined web, Go, Python executor and database connection peaks under representative work; reserve headroom before increasing concurrency. Each active Django task has its own Python process, so a small Go footprint does not make a memory-heavy task cheap. Move heavy ETL to a separate process/host when it exceeds the web budget.


🔍 Admin Dashboard

Reproq integrates deeply with the Django Admin.

Results and contexts inherit Django’s frozen, slotted native classes. refresh() and arefresh() update the result and return None. Native attempts and context.attempt derive from the complete worker_ids history, including the current worker as in Django’s built-in backends. The separate budget_attempt extension is ReproQ’s resettable retry-budget counter. Context metadata is an explicit mutable mapping; save it with the fenced save_metadata() or asave_metadata() method rather than replacing the mapping. Native result.task.run_after preserves original enqueue intent. The distinct eligible_after extension reports mutable queue eligibility for periodic slots, retry backoff or replay. Canonical text supplies v1 arguments; a JSONB projection cannot preserve every source detail, including the sign of negative zero.

Task failures retain their qualified Python exception class and bounded stack frames in Django’s native TaskError records. Diagnostics omit source lines, locals and chained details; an explicit marker explains when a message or stack exceeds the budget. Worker or protocol failures without a recoverable Python exception use reproq_django.protocol.WorkerExecutionError. raw_errors retains the original worker records for inspection.

Use the unified reproq check, reproq doctor and reproq worker commands. The standalone reproq_check, reproq_health and reproq_init_executor commands, generated executor stubs and HTTP/CLI stress-test entry points have been retired. The canonical executor module is reproq_django.executor, launched with the configured Python executable.

Timedeltas and saved model references use explicit typed envelopes. Other JSON normalization follows DjangoJSONEncoder: Decimal, UUID and date/time values arrive as strings; tuples arrive as lists. Do not depend on those Python types surviving a queue round trip. Pass explicit task-domain representations when their type or precision is part of the task contract.


🚀 Production Deployment

Generate service files to run the worker and beat processes under systemd 251 or newer.

uv run python manage.py reproq systemd --user myuser --concurrency 20

This generates reproq-worker-<project>.service and reproq-beat-<project>.service. Copy them to /etc/systemd/system/ and enable them. You can pass metrics flags (for example --metrics-addr 127.0.0.1:9090) or use --env-file to load METRICS_AUTH_TOKEN and METRICS_ALLOW_CIDRS from one required file. Relative paths are resolved against the project directory.

To use cron-style scheduling instead of a persistent beat process, generate a timer:

uv run python manage.py reproq systemd --schedule --schedule-on-calendar "*-*-* *:*:00"

Set REPROQ_SCHEDULER_MODE=cron in your EnvironmentFile and enable the timer to ensure only one scheduler runs.

Env Vars

The Go worker relies on standard environment variables:

The SDK derives the queue DSN from the selected Django queue alias; namespaced environment/configuration must match it. Business DATABASE_URL is preserved for task execution.

Worker binary resolution order:

  1. REPROQ_WORKER_BIN (setting or env)
  2. ./.reproq/bin/reproq (installed by reproq install)
  3. reproq_django/bin/reproq (packaged fallback)
  4. PATH

🤝 Contributing

Reproq is split into two repos:

Issues and PRs are welcome in both!

License

Apache License 2.0. See LICENSE.