async-queue-temporal-expert
Unified expert guide for async job queues & durable workflows: BullMQ v5 (Redis queues), Trigger.dev v3 (serverless tasks), Inngest, and Temporal.io (distributed sagas) / Panduan ahli terpadu untuk antrean job asinkron & workflow tahan-gagal: BullMQ v5, Trigger.dev v3, Inngest, dan Temporal.io.
roedyrustam · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add roedyrustam/vibes-plug --skill async-queue-temporal-expert技能说明
name: async-queue-temporal-expert description: "Unified expert guide for async job queues & durable workflows: BullMQ v5 (Redis queues), Trigger.dev v3 (serverless tasks), Inngest, and Temporal.io (distributed sagas) / Panduan ahli terpadu untuk antrean job asinkron & workflow tahan-gagal: BullMQ v5, Trigger.dev v3, Inngest, dan Temporal.io." author: "Roedy Rustam"
version: "3.0.0"
Async Queue & Durable Workflow Expert (2026 Unified Edition)
English
Purpose & Overview
Unified production-grade guide for background execution pipelines, async job queues, and distributed state machines. Covers the full spectrum from simple Redis-backed task queues to complex multi-service sagas with compensating rollbacks.
3-Tier Execution Model
| Tier | Engine | Best For |
|---|---|---|
| Tier 1: Redis Task Queues | BullMQ v5 | High-throughput worker jobs, priority queues, rate limiting, DLQ |
| Tier 2: Serverless Durable Tasks | Trigger.dev v3 / Inngest | Step-checkpointed tasks, automatic resume across crashes, zero infra |
| Tier 3: Distributed Sagas | Temporal.io | Multi-service orchestration, compensating rollbacks, long-running workflows |
Core Capabilities
- Idempotency & Deduplication: Deterministic
jobIdkeys prevent double billing or duplicate emails. - Dead Letter Queues (DLQ): Auto-relocate permanently failing jobs for audit and alerting.
- Exponential Backoff with Jitter: Prevents thundering herds on upstream services.
- Tenant Priority Queues: VIP/enterprise tiers get lower BullMQ priority numbers (higher throughput).
- Durable State Machines: Workflows survive restarts, deployments, and network partitions.
- Saga Compensations: Multi-step transactions with automated reverse-order rollbacks.
Tier 1: BullMQ v5 — Redis Task Queues (TypeScript)
import { Queue, Worker, Job } from 'bullmq';
import Redis from 'ioredis';
const redisConnection = new Redis(process.env.REDIS_URL!, {
maxRetriesPerRequest: null, // Required by BullMQ
});
export interface NotificationPayload {
tenantId: string;
userId: string;
type: 'email' | 'webhook';
payload: Record<string, unknown>;
idempotencyKey: string;
}
// Main Queue
export const notificationQueue = new Queue<NotificationPayload>('notifications', {
connection: redisConnection,
defaultJobOptions: {
attempts: 5,
backoff: { type: 'exponential', delay: 1500 },
removeOnComplete: { age: 86400, count: 5000 },
removeOnFail: false, // Preserved for DLQ audit
},
});
// Dead Letter Queue
export const notificationDLQ = new Queue('notifications-dlq', {
connection: redisConnection,
});
// Enqueue with deduplication & priority
export async function enqueueNotification(data: NotificationPayload, isVip = false) {
return await notificationQueue.add('send_notification', data, {
jobId: `notif_${data.idempotencyKey}`, // Deterministic dedup key
priority: isVip ? 1 : 10,
});
}
// Worker with concurrency & rate limiting
export const notificationWorker = new Worker<NotificationPayload>(
'notifications',
async (job: Job<NotificationPayload>) => {
if (job.data.type === 'email') await deliverEmail(job.data);
},
{
connection: redisConnection,
concurrency: 20,
limiter: { max: 100, duration: 1000 },
}
);
// DLQ forwarding on retry exhaustion
notificationWorker.on('failed', async (job, error) => {
if (job && job.attemptsMade >= (job.opts.attempts || 5)) {
await notificationDLQ.add('failed_notification', {
originalJobId: job.id, failedReason: error.message,
data: job.data, exhaustedAt: new Date().toISOString(),
});
}
});
Tier 2: Trigger.dev v3 — Serverless Durable Tasks
import { task } from '@trigger.dev/sdk/v3';
export const generateReport = task({
id: 'generate-enterprise-report',
retry: { maxAttempts: 4, minTimeoutInMs: 2000, factor: 2, randomize: true },
run: async (payload: { tenantId: string; month: string }, { ctx }) => {
// Each step is a durable checkpoint — survives crashes
const data = await ctx.run('fetch-telemetry', async () => {
return await fetchTelemetryFromWarehouse(payload.tenantId, payload.month);
});
const pdfUrl = await ctx.run('render-pdf', async () => {
return await generateReportPdf(data);
});
await ctx.run('dispatch-webhook', async () => {
return await sendWebhookNotification(payload.tenantId, pdfUrl);
});
return { success: true, pdfUrl };
},
});
Tier 3: Temporal.io — Distributed Saga with Compensations
import { proxyActivities, ApplicationFailure } from '@temporalio/workflow';
import type * as activities from './activities';
const { chargeCustomer, provisionLicense, sendWelcomeEmail, refundCustomer, revokeLicense } =
proxyActivities<typeof activities>({
startToCloseTimeout: '1 minute',
retry: {
initialInterval: '1s', backoffCoefficient: 2, maximumAttempts: 5,
nonRetryableErrorTypes: ['InvalidCardError', 'AccountSuspendedError'],
},
});
export async function subscriptionSagaWorkflow(input: {
customerId: string; planId: string; amountCents: number;
}) {
const compensations: Array<() => Promise<void>> = [];
try {
const charge = await chargeCustomer(input.customerId, input.amountCents);
compensations.unshift(() => refundCustomer(charge.chargeId));
const license = await provisionLicense(input.customerId, input.planId);
compensations.unshift(() => revokeLicense(license.licenseId));
await sendWelcomeEmail(input.customerId, license.licenseKey);
return { status: 'COMPLETED' };
} catch (error) {
for (const compensate of compensations) {
阅读完整来源 ↗