{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Jaival Suthar",
  "title_listed": "Designing a Production-Grade Async Job Queue for Bulk PDF Processing with Redis and BullMQ",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://jaivalsuthar.hashnode.dev/production-grade-async-job-queue-redis-bullmq",
  "publish_date": "2026-03-07",
  "corpus_stage": "",
  "notes": "",
  "id": "d42206d9d0de",
  "collected_at": "2026-10-02T14:12:16",
  "title": "Designing an Async Job Queue with Redis and BullMQ",
  "body_markdown": "Designing a Production-Grade Async Job Queue for Bulk PDF Processing with Redis and BullMQ\nUpdated\nMarch 7, 2026\n•\n23\nmin read\n•\nView as Markdown\nJ\nJaival Suthar\nI write technical articles that analyze real world engineering problems through the lens of production systems.\nMy work focuses on architectural tradeoffs, UI layer system design, performance optimization, and reliability patterns derived from shipping and maintaining scalable applications.\n1. The Real Problem Behind Bulk PDF Processing\nFrom the outside, bulk PDF upload looks simple. User picks a ZIP file, hits upload, system processes it. Done.\nFrom the inside, you're dealing with 50 to 300 PDFs arriving simultaneously, each one needing to be extracted, parsed for text, run through a regex, matched against a database record, conditionally written to object storage, and then have the database updated, all while the user is sitting on the other side expecting the browser not to hang.\nThe synchronous path fails quickly. You start processing PDF number one, and by the time you're done with PDF number three, the HTTP request has already timed out. Even if your reverse proxy gives you 120 seconds, that's not enough for 300 documents. And even if it were, you've now blocked your entire Node.js event loop for the duration. Every other user's request is waiting behind a ZIP file. In effect, a single large upload becomes capable of degrading the responsiveness of the entire application.\nThe real constraint isn't algorithmic. It's environmental. The system operates on a single application server with limited compute and memory resources, connected to a managed database. There’s no distributed processing layer, autoscaling infrastructure, or serverless workers available for offloading heavy tasks. The same server must continue handling normal application traffic while processing large ZIP uploads in the background, so any solution has to work within these constraints without degrading the rest of the system.\nThe deployment window is also real. You don't get to take the system down for an hour to re-architect. You get a narrow window, and the feature has to work when it comes back up.\nThat's the actual problem. Processing PDFs is not the difficult part. That can be solved in an afternoon. The real problem is: how do I process PDFs asynchronously, track progress across an unknown number of files, recover from partial failures, report status back to the client, and do all of this while operating within the existing infrastructure constraints and without introducing additional managed services or increasing operational costs.\n2. System Architecture Overview\nBefore getting into decisions, here's the full async flow as built.\nThe architecture separates request handling from heavy processing as early as possible. The API endpoint accepts the ZIP upload, assigns a batchId, enqueues the job, and immediately returns control to the client. From that point forward, the HTTP request lifecycle is no longer tied to the work being performed.\nThe worker performs extraction and PDF processing independently while the client periodically polls Redis for batch progress. This separation prevents long-running tasks from blocking the application server and keeps the Node.js event loop free from CPU or I/O heavy work triggered by a single request.\nIt also creates a natural progress tracking mechanism where the system can expose partial completion status without maintaining any in-memory session state. The system effectively becomes a set of loosely coupled components: the API accepts work, the queue schedules it, the worker executes it, and Redis tracks the batch progress.\nThe key architectural decision is the decoupling point. The API returns a batchId before any PDF processing begins, allowing the client to poll for progress while the worker processes the batch in the background.\n3. Queue Architecture: BullMQ and Redis vs Database Polling\nThis decision deserves honest treatment because the database-backed queue is a legitimate architecture that I seriously considered and came close to choosing.\nThe database queue pattern\nworks like this: when a job arrives, you insert a row into a\njobs\ntable with status\nPENDING\n. A background process polls that table every few seconds, picks up\nPENDING\nrows, sets them to\nPROCESSING\n, does the work, and marks them\nCOMPLETED\nor\nFAILED\n. You get job persistence for free, you get queryability, and you get zero additional infrastructure.\nThis is genuinely a good pattern. A database polling queue works well when your team is comfortable with SQL and the workload is low-volume and latency-tolerant. If you already operate a database and want to avoid another dependency, it becomes a very reasonable choice. It scales further than most people think, especially with\nSELECT ... FOR UPDATE SKIP LOCKED\nto handle concurrent consumers without collisions.\nWhere it struggles: when you want real-time push semantics, when you want built-in retry with backoff, when your polling interval creates unacceptable latency, or when your database is already under load and you don't want job polling adding to that. Polling a database every 3 seconds is cheap when the table has 50 rows. Less cheap when you're in a high-throughput scenario and your external database instance is already handling application queries.\nWhy BullMQ won here:\nFirst, the cost argument. Redis runs as a Docker Compose service on the same machine. No additional compute cost. The alternative is polling an external managed database. This means every job pickup requires a network round trip to a service that is already serving application queries. Redis on localhost is microseconds. External DB across a network is milliseconds. For a job queue, the difference matters less in throughput than in connection overhead and latency predictability.\nSecond, BullMQ gives you the primitives for free: job state machine, delayed retries with exponential backoff, concurrency control, job events, and stalled job detection. Building equivalent reliability on top of a database polling loop is not hard, but it's code I'd have to write, test, and maintain. BullMQ is that code, already written.\nThird, and this is the honest reason behind the choice, Redis makes batch state tracking much cleaner. I need atomic counter increments across concurrent PDF processing. Redis has\nHINCRBY\n. The database alternative is either a transaction per counter update (expensive) or accepting eventual inconsistency in your progress display (confusing to users). Redis is just the better fit for the state shape I needed. In short, BullMQ reduced the amount of reliability infrastructure I would have had to build myself while Redis provided the atomic primitives needed for progress tracking.\nThe tradeoff I accepted: Redis is now a runtime dependency. If it goes down, the queue stops. The batch state disappears after TTL expiry. These are real risks. But on a single-machine Docker Compose setup where Redis restarts automatically and job data is ephemeral by design, the operational burden is lower than it sounds.\n4. Redis Deployment Strategy\nIn this system Redis serves two roles. It functions as the job queue backend for BullMQ and as the storage layer for batch progress state. Because the application runs on a single server, introducing a separate Redis infrastructure layer would add operational complexity without providing meaningful benefit at the current scale.\nFor this reason Redis runs on the same machine as the application and is managed as a container alongside the Node.js service. This keeps deployment simple while still providing the latency and atomic operations needed for queue management and progress tracking.\nThe Docker Compose configuration for Redis is not complicated, but it contains at least one decision that will cost you hours if you get it wrong.\nservices:\n  app:\n    build: .\n    environment:\n      - REDIS_HOST=redis\n      - REDIS_PORT=6379\n    depends_on:\n      - redis\n    networks:\n      - internal\n\n  redis:\n    image: redis:7-alpine\n    restart: always\n    networks:\n      - internal\n\nnetworks:\n  internal:\n    driver: bridge\nThe service topology:\nThe hostname gotcha.\nInside a Docker Compose network, containers resolve each other by service name.\nredis\nis a valid hostname.\n127.0.0.1\nis not, because inside the app container\nlocalhost\nrefers to the container's own loopback interface, not the Redis container. If you've been testing locally with Redis running on your host machine, your\n.env\nfile probably has\nREDIS_HOST=127.0.0.1\n, and that works fine on your laptop. The moment you build the container, it silently fails. BullMQ will throw connection errors that look like Redis is down, when really it's just a hostname that doesn't resolve to anything useful inside the container's network namespace.\nThe fix is simple:\nREDIS_HOST=redis\nin the container environment,\nREDIS_HOST=127.0.0.1\nin your local\n.env\nwhen running Redis on your host. Keep these in separate env files and don't mix them up.\nLocal development.\nIf you're on WSL2, running Redis directly inside the WSL environment can sometimes produce odd behavior. You may see connection hangs, socket timeouts on the first connection, or issues that disappear on subsequent runs. Docker Desktop running Redis as a local container with the app pointing at\n127.0.0.1:6379\ngives more predictable behavior and closer parity with production.\nrestart: always\n.\nWhen the server reboots, whether for planned maintenance, a kernel update, or another reason, Docker Compose services do not start automatically unless configured. With\nrestart: always\n, Docker will restart the Redis container whenever it stops for any reason, including host reboots, as long as the Docker daemon itself is configured to start on boot. Without this, a reboot means Redis is down, BullMQ can't connect, and your application starts throwing errors. One line in the compose file. Forgetting it is the kind of thing that costs you a Sunday morning.\n5. Batch State Management in Redis\nThe batch state lives entirely in Redis. Not the database. This is a deliberate choice with real tradeoffs.\nWhy not the database?\nThe state changes frequently since it is updated once per processed PDF, and the data is inherently transient. It doesn't need to survive forever; it just needs to survive long enough for the client to see the final result. A 1-hour TTL is sufficient. If this were in the database, you'd be executing UPDATE statements on every PDF completion, under concurrent load, on an external managed instance. That's connection overhead, network latency, and write amplification on infrastructure shared with your application's primary workload. Redis on localhost, updated with atomic increments, is orders of magnitude cheaper for this pattern.\nA Redis hash is used so all batch counters can be stored and updated atomically under a single key.\nThe state schema:\n// Initialized when job is enqueued\nawait redis.hset(`batch:${batchId}`, {\n  status: 'PENDING',\n  totalFiles: 0,         // updated when ZIP is extracted\n  processed: 0,\n  linked: 0,\n  skipped_no_match: 0,\n  skipped_already_exists: 0,\n  failed: 0,\n  createdAt: Date.now()\n});\nawait redis.expire(`batch:${batchId}`, 3600); // 1 hour TTL\nAtomic counter increments with WATCH/MULTI/EXEC:\nThe PDF processing runs with up to 5 concurrent processors via\np-limit\n. All 5 can complete around the same time and try to update the same Redis hash simultaneously. Without coordination, race conditions occur. Two processors might read\nprocessed: 10\nat the same time, both increment it, both write\nprocessed: 11\n, and one update is effectively lost.\nRedis's WATCH/MULTI/EXEC provides optimistic locking:\nasync function incrementBatchCounter(redis, batchId, field) {\n  const key = `batch:${batchId}`;\n  \n  let result = null;\n  while (result === null) {\n    await redis.watch(key);\n    \n    const multi = redis.multi();\n    multi.hincrby(key, field, 1);\n    multi.hincrby(key, 'processed', 1);\n    \n    // Returns null if WATCH key was modified by another client\n    result = await multi.exec();\n  }\n}\nWATCH marks a key for monitoring. If any watched key is modified by another client before EXEC runs, the entire transaction is aborted and EXEC returns null. The while loop retries until the transaction succeeds. Under 5 concurrent processors, contention is low and retries are rare. Under higher concurrency, you'd want Lua scripts for atomicity, which avoid the round-trip overhead of the optimistic retry loop entirely.\nTTL expiry edge cases.\nWhat happens if the job takes longer than an hour and the Redis key expires mid-processing? The worker will start failing on every counter update because the key no longer exists. The client polling for status will get a 404. The job may still complete in BullMQ (the queue is separate from the state key), but the results are lost.\nAt 300 PDFs with 5 concurrent processors, realistic processing time is a few minutes at most. A 1-hour TTL has comfortable headroom. If your file counts grow significantly, extend the TTL. If you're paranoid about it, refresh the TTL on each worker heartbeat. But don't solve a problem that isn't happening yet.\n6. PDF Matching Strategy\nThe matching logic doesn't use the PDF filename at all. This surprises people, but it's the right call.\nFilenames are unreliable. Names like \"John_Resume_Final_v3.pdf\", \"resume.pdf\", or \"cv_2023.pdf\" tell you nothing about whether the person in the PDF exists in your system. The only reliable identifier is an email address embedded in the PDF content.\nExtraction pipeline:\nconst pdfParse = require('pdf-parse');\n\nasync function extractEmailFromPDF(pdfBuffer) {\n  const data = await pdfParse(pdfBuffer);\n  const text = data.text;\n  \n  const emailRegex = /[a-zA-Z0-9._%+\\-]+@[a-zA-Z0-9.\\-]+\\.[a-zA-Z]{2,}/g;\n  const matches = text.match(emailRegex);\n  \n  return matches ? matches[0] : null;\n}\nThe first matched email wins. In practice, documents have one email address, typically near the top. If a document has multiple, the first one encountered is almost always the right one. This is a heuristic, and it works at the scale and document quality we're dealing with.\nThe\npdf-parse\nversion trap.\nThis is genuinely annoying and worth documenting.\npdf-parse\nat recent versions post-1.1.1 changed how the module exports its default. If you\nrequire('pdf-parse')\nand call the result as a function, it works in 1.1.1. On newer versions, the default export is an object and the function lives on one of its properties. Calling it directly results in the runtime error \"pdf-parse is not a function.\"\nPin to\n1.1.1\n:\n{\n  \"dependencies\": {\n    \"pdf-parse\": \"1.1.1\"\n  }\n}\nAlternative libraries like\npdfjs-dist\nand\npdf2json\nexist, but they come with their own quirks. For extracting text from standard text-layer PDFs,\npdf-parse@1.1.1\nworks reliably. Pin the version and move on unless you plan to adapt to the newer export format.\nThe OCR gap.\nScanned PDFs are essentially images rendered as PDF pages, so they have no text layer.\npdf-parse\nwill return empty or near-empty text. The email regex finds nothing. The file is counted as\nskipped_no_match\n. This is the correct behavior given the tooling; the alternative is integrating an OCR engine, which is a different system entirely. Surface the count in your status response so there's visibility into what happened. Don't silently skip and leave the uploader guessing.\nMatching and upload logic:\nasync function processOnePDF(pdfBuffer, batchId, redis, db, s3) {\n  let conn;\n  try {\n    const email = await extractEmailFromPDF(pdfBuffer);\n    if (!email) {\n      await incrementBatchCounter(redis, batchId, 'skipped_no_match');\n      return;\n    }\n\n    conn = await db.getConnection();\n    const [rows] = await conn.query(\n      'SELECT id, pdf_url FROM users WHERE email = ?',\n      [email]\n    );\n\n    if (rows.length === 0) {\n      await incrementBatchCounter(redis, batchId, 'skipped_no_match');\n      return;\n    }\n\n    const user = rows[0];\n    if (user.pdf_url) {\n      await incrementBatchCounter(redis, batchId, 'skipped_already_exists');\n      return;\n    }\n\n    // S3 upload only after confirmed match + no existing PDF\n    const s3Key = `pdfs/\\({user.id}/\\){Date.now()}.pdf`;\n    await s3.putObject({\n      Bucket: process.env.S3_BUCKET,\n      Key: s3Key,\n      Body: pdfBuffer,\n      ContentType: 'application/pdf'\n    }).promise();\n\n    await conn.query(\n      'UPDATE users SET pdf_url = ? WHERE id = ?',\n      [s3Key, user.id]\n    );\n\n    await incrementBatchCounter(redis, batchId, 'linked');\n  } catch (err) {\n    await incrementBatchCounter(redis, batchId, 'failed');\n  } finally {\n    if (conn) conn.release();\n  }\n}\nThe S3 upload placement is deliberate: it happens after you've confirmed the user exists and doesn't have an existing PDF, and before the database update. You don't upload files you don't need. This ordering was chosen after thinking through what a partial failure looks like at each step.\n7. Worker Architecture: Concurrency and Resource Management\nThe worker is responsible for unpacking the ZIP file and processing each PDF while respecting system resource limits. Rather than processing files sequentially or spawning unbounded parallel work, the worker uses controlled concurrency to balance throughput with database load.\nConcurrency structure:\nconst { Worker } = require('bullmq');\nconst pLimit = require('p-limit');\n\nconst worker = new Worker('pdf-bulk-upload', async (job) => {\n  const { batchId, zipBuffer: zipBase64 } = job.data;\n  const zipBuffer = Buffer.from(zipBase64, 'base64');\n  \n  const zip = new AdmZip(zipBuffer);\n  const entries = zip.getEntries().filter(e => \n    e.entryName.toLowerCase().endsWith('.pdf') && !e.isDirectory\n  );\n\n  await redis.hset(`batch:${batchId}`, 'totalFiles', entries.length);\n  await redis.hset(`batch:${batchId}`, 'status', 'PROCESSING');\n\n  const limit = pLimit(5);\n  \n  await Promise.all(\n    entries.map(entry => \n      limit(() => processOnePDF(entry.getData(), batchId, redis, db, s3))\n    )\n  );\n\n  await redis.hset(`batch:${batchId}`, 'status', 'COMPLETED');\n}, {\n  connection: redisConnection,\n  concurrency: 1\n});\nThe per-PDF processing flow:\nWhy\np-limit(5)\nand not a higher number.\nCPU is not the bottleneck in this case.\npdf-parse\nis already fast on text-layer PDFs. The bottleneck is the database. Each PDF needs at least one database query. With 5 concurrent processors, you're holding up to 5 database connections simultaneously. The external managed database has connection limits, and more importantly, you're sharing that instance with your application's other queries. Hammering it with 50 concurrent connections from a bulk upload job while users are actively using the application is a bad trade. Five is conservative enough to be a good neighbor to the rest of the system, while fast enough to process 300 files in a reasonable timeframe.\nOne connection per PDF, released in\nfinally\n.\nEach call acquires a connection from the pool at the start and releases it in\nfinally\n. Non-negotiable. If you forget the\nfinally\nrelease, you leak connections on errors. The pattern is verbose but mechanical. Acquire the connection, use it, and release it every time.\nBullMQ job configuration:\nawait pdfBulkQueue.add('process-batch', {\n  batchId,\n  zipBuffer: zipBuffer.toString('base64')\n}, {\n  attempts: 2,\n  backoff: {\n    type: 'exponential',\n    delay: 5000\n  },\n  removeOnComplete: 100,\n  removeOnFail: 50\n});\nTwo attempts with exponential backoff. If the worker crashes mid-job, BullMQ marks it as stalled after the lock expires and re-enqueues it. The job will restart from the beginning and reprocess all PDFs. This is acceptable given the idempotency logic: if a user already has a PDF set, the duplicate is skipped. The counters will be off (they'll double-count skips), but the actual data integrity is fine.\nWhy\nconcurrency: 1\nat the worker level.\nSingle worker, single job at a time. No race conditions on the BullMQ side. The parallelism lives inside the job via\np-limit\n, not across jobs. This is the right default for a feature that processes large batches. You want predictable resource usage, not two 300-PDF batches competing for database connections at the same time.\nWorker registration:\n// server.js\nrequire('./workers/pdfBulkWorker'); // registers worker on boot\nThe worker runs in the same process as the HTTP server. On a constrained server with a single application container, process isolation buys you nothing and adds deployment complexity. BullMQ handles job persistence. Jobs do not vanish if the process restarts.\n8. Development Issues and Debugging Lessons\nThese bugs all appeared during local development and were resolved before anything reached production. Documented here because they're not obvious, and \"obvious in retrospect\" is a bad standard for learning from mistakes.\nqueue.add is not a function\nThe error message doesn't tell you what's actually wrong. The cause: the queue module was exporting both the queue and the Redis connection object together.\n// pdfBulkQueue.js\nmodule.exports = { pdfBulkQueue, redisConnection };\n\n// In the route file — WRONG\nconst pdfBulkQueue = require('../queues/pdfBulkQueue');\n// pdfBulkQueue is now { pdfBulkQueue, redisConnection }\n// pdfBulkQueue.add is undefined\n\n// CORRECT\nconst { pdfBulkQueue } = require('../queues/pdfBulkQueue');\nJavaScript's module system gives you no type checking on imports. The wrong object gets imported, it quacks enough like a queue (it's an object, it doesn't throw on access), and then you get a runtime error when you actually try to use it. This class of bug doesn't show up in static analysis without TypeScript. Be explicit about destructuring imports and consistent about your module export shapes.\npdf-parse\nis not a function\nSome npm packages changed their module export format between versions. Earlier versions used\nmodule.exports = function(){}\n, while newer versions export an object such as\nmodule.exports = { default: function(){}, ...helpers }\nas part of ESM and CommonJS interoperability changes. When your code calls\npdfParse(buffer)\nand\npdfParse\nis an object instead of a function, JavaScript throws \"pdfParse is not a function\" with no indication that the problem is the import, not the call site.\nVersion pinning to\n1.1.1\nis the fix. The deeper habit: when a function-shaped thing stops being callable, check what you actually imported before assuming the call is wrong.\nDocker networking: why\nlocalhost\nlies inside containers\nInside a Docker container,\nlocalhost\nand\n127.0.0.1\nrefer to the container's own loopback interface. They do not refer to the host machine or any other container. If your Redis container is named\nredis\nin Docker Compose, that's the hostname your app container should use. If you have\nREDIS_HOST=127.0.0.1\nin your environment, left over from local non-container development, the app will attempt to connect to its own loopback, find nothing listening on port 6379, and fail.\nThe error from BullMQ looks like\nECONNREFUSED 127.0.0.1:6379\n. It doesn't say \"wrong hostname.\" If you don't immediately think \"wait, which localhost is this,\" you'll spend time checking whether Redis is running (it is), whether the port is correct (it is). The fix is changing the hostname. The lesson is to understand network namespaces in Docker before you spend an hour debugging a connection string.\n9. Client Progress Tracking: Polling vs WebSockets\nThe client polls for batch status every 3 seconds:\nasync function pollBatchStatus(batchId) {\n  const response = await fetch(`/api/pdf-bulk-upload/${batchId}/status`);\n  const data = await response.json();\n  \n  updateProgressUI(data);\n  \n  if (data.status !== 'COMPLETED' && data.status !== 'FAILED') {\n    setTimeout(() => pollBatchStatus(batchId), 3000);\n  }\n}\nThe server-side status endpoint reads directly from Redis:\napp.get('/pdf-bulk-upload/:batchId/status', async (req, res) => {\n  const { batchId } = req.params;\n  const state = await redis.hgetall(`batch:${batchId}`);\n  \n  if (!state || Object.keys(state).length === 0) {\n    return res.status(404).json({ error: 'Batch not found or expired' });\n  }\n  \n  res.json({\n    status: state.status,\n    totalFiles: parseInt(state.totalFiles) || 0,\n    processed: parseInt(state.processed) || 0,\n    linked: parseInt(state.linked) || 0,\n    skipped_no_match: parseInt(state.skipped_no_match) || 0,\n    skipped_already_exists: parseInt(state.skipped_already_exists) || 0,\n    failed: parseInt(state.failed) || 0\n  });\n});\nWebSockets weren't seriously considered for long. The usual arguments for them are real-time push and the absence of polling overhead. At this scale, those advantages do not hold up. A single user uploading a batch makes one HTTP request every 3 seconds for a few minutes. That's negligible load on any server that can handle a production application at all. WebSockets add connection management complexity, require reconnection handling on the client, and need server-side state to track which socket corresponds to which batch. Polling is stateless, trivially resumable after a page reload, and requires zero additional server infrastructure.\nThe 3-second interval is a judgment call. For a progress bar updating across 300 files processed over a couple of minutes, 3 seconds gives smooth-enough perceived update frequency without hammering the server. If you needed sub-second updates, the calculus changes. You don't, so it doesn't.\nTTL expiry from the client's perspective.\nIf the client polls an hour after the job completed, they'll get a 404. This is correct behavior. The state has expired, so the client should interpret 404 as \"this batch is no longer available\" and stop polling. If your UX needs to persist results longer than an hour, write a summary to the database on job completion and serve it from there. The Redis state is ephemeral progress tracking; persistent results belong in persistent storage.\n10. Production Deployment Checklist\nThese are the things that need to be verified before the feature is live, in the order they'll bite you if you miss them.\nDocker Compose changes:\nservices:\n  redis:\n    image: redis:7-alpine\n    restart: always\n    networks:\n      - internal\n\n  app:\n    environment:\n      - REDIS_HOST=redis\n      - REDIS_PORT=6379\n    depends_on:\n      - redis\n    networks:\n      - internal\nVerify:\ndocker-compose up\nstarts both services, app logs show successful Redis connection,\ndocker-compose ps\nshows both containers running.\nEnvironment variables:\nVariable\nLocal\nContainer\nREDIS_HOST\n127.0.0.1\nredis\nREDIS_PORT\n6379\n6379\nS3_BUCKET\ndev bucket\nprod bucket\nDo not commit the production\n.env\nfile. Commit a\n.env.example\nwith all required variables and no values.\nDocker daemon auto-start on reboot:\nsudo systemctl enable docker\nThis ensures Docker starts on boot. Combined with\nrestart: always\non services, your containers come back up automatically after a reboot. Without it, the Docker daemon doesn't start on reboot, and your containers don't either.\nStuck job recovery.\nBullMQ uses a lock mechanism to detect stalled jobs. If a worker picks up a job and dies mid-processing, BullMQ reclassifies it as stalled after the lock expires and re-enqueues it. For manual recovery, check for jobs in\nACTIVE\nstate older than your expected maximum processing time:\nconst active = await pdfBulkQueue.getActive();\nconst staleThreshold = 10 * 60 * 1000; // 10 minutes\n\nconst stale = active.filter(job => \n  Date.now() - job.timestamp > staleThreshold\n);\n\nfor (const job of stale) {\n  await job.moveToFailed(new Error('Manually failed: stale job'), true);\n}\n11. Scaling Considerations\nAt the current scale, the architecture is appropriately simple. The system supports a small internal team that runs batches a few times a day, typically handling 50 to 300 files at a time. Here is what breaks first as the scale increases.\nThe ZIP in memory.\nThe ZIP buffer is stored in memory during the API request and passed to BullMQ as a base64-encoded string. At 50MB, this is fine. At 200MB, you're putting pressure on a machine that's also running your application and Redis. The fix is to write the ZIP to temporary storage on upload, pass the reference in the job payload, and have the worker retrieve and stream it. This decouples the job payload size from memory consumption.\nSingle Redis on the same machine.\nIf the machine goes down, you lose both the application and the queue. For an internal tool this is probably acceptable. For customer-facing systems with SLA obligations, Redis should run as a managed service with replication, automated failover, and persistence. That way a bad day on the host machine does not take the queue down with it.\nMultiple workers.\nRight now,\nconcurrency: 1\nmeans one batch processes at a time. If two users upload simultaneously, the second batch waits. To support parallel batches, you'd increase\nconcurrency\nor run multiple worker processes. Both approaches require careful database connection pool sizing. Otherwise, 10 workers each holding 5 connections can quickly exhaust your connection limit.\nDistributed locking.\nCurrently there's no lock preventing two workers from processing the same PDF for the same user simultaneously if you run multiple workers. The database UPDATE is effectively the lock, and the existence check happens before the upload. But at higher concurrency with multiple workers, you'd want an explicit distributed lock (Redis SETNX, Redlock) around the per-user PDF assignment to prevent redundant storage uploads from racing updates.\nEmail regex → proper parsing.\nThe email regex works for well-formatted text-layer PDFs. It's fragile in predictable ways: it misses obfuscated emails, it's confused by email addresses appearing elsewhere in the document, and it finds nothing in scanned files. At scale, or if match rates start falling, the right answer is a proper document parsing library or service. The email extraction is a pragmatic shortcut that works at current volume. At 10× volume with more varied document quality, it becomes the primary source of incorrect matches and missed links.\nThe decisions that matter in systems design are rarely the ones with obvious right answers. They are the ones where multiple approaches work, and the correct choice depends on the constraints you are operating under: the infrastructure you have, the team maintaining it, the timeline you are working within, and the failure modes you can tolerate.\nThe architecture described here is simply one set of decisions made under a specific set of constraints and at a particular scale. Change the constraints, and the right decisions will likely change with them.\n#\nsystem-design\n#\nnodejs\n#\nredis\n#\nbackend",
  "metadata": {
    "title": "Designing an Async Job Queue with Redis and BullMQ",
    "description": "How to design a production async job queue for bulk PDF processing using Redis, BullMQ, and Node.js with real-world tradeoffs.",
    "keywords": null,
    "author": null,
    "og:title": "Designing an Async Job Queue with Redis and BullMQ",
    "og:description": "How to design a production async job queue for bulk PDF processing using Redis, BullMQ, and Node.js with real-world tradeoffs.",
    "og:url": "https://jaivalsuthar.hashnode.dev/production-grade-async-job-queue-redis-bullmq",
    "og:site_name": "Jaival Suthar | Engineering Portfolio",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/6ce6cb94-17db-4796-a58c-c69e068abc96.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:creator": "https://x.com/Jaival_Suthar",
    "twitter:title": "Designing an Async Job Queue with Redis and BullMQ",
    "twitter:description": "How to design a production async job queue for bulk PDF processing using Redis, BullMQ, and Node.js with real-world tradeoffs.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/6ce6cb94-17db-4796-a58c-c69e068abc96.png",
    "article:published_time": "2026-03-07T17:10:33.008Z",
    "article:modified_time": "2026-03-07T17:59:17.554Z",
    "article:author": "https://jaivalsuthar.hashnode.dev/@JaivalSuthar"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/d42206d9d0de/code/body-000.txt",
      "chars": 286,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-001.txt",
      "chars": 336,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-002.txt",
      "chars": 396,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-003.txt",
      "chars": 318,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-004.txt",
      "chars": 53,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-005.txt",
      "chars": 1240,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-006.txt",
      "chars": 821,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-007.txt",
      "chars": 222,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-008.txt",
      "chars": 77,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-009.txt",
      "chars": 319,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-010.txt",
      "chars": 311,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-011.txt",
      "chars": 652,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-012.txt",
      "chars": 231,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-013.txt",
      "chars": 29,
      "language": "text"
    },
    {
      "file": "attachments/d42206d9d0de/code/body-014.txt",
      "chars": 289,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/6ce6cb94-17db-4796-a58c-c69e068abc96.png",
      "bytes": 1007710,
      "reused": true,
      "local_path": "attachments/d42206d9d0de/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/6ce6cb94-17db-4796-a58c-c69e068abc96.png",
      "alt": "Designing a Production-Grade Async Job Queue for Bulk PDF Processing with Redis and BullMQ",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/47546953-dab9-4ccd-ba0c-cac18387cfaf.png",
      "bytes": 188430,
      "reused": true,
      "local_path": "attachments/d42206d9d0de/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/47546953-dab9-4ccd-ba0c-cac18387cfaf.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/678b49ab-2394-4ce9-876e-7469b3b43c8e.png",
      "bytes": 28527,
      "reused": true,
      "local_path": "attachments/d42206d9d0de/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/678b49ab-2394-4ce9-876e-7469b3b43c8e.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/f4ecdf76-53bc-45bf-8782-0b368609dcac.png",
      "bytes": 331730,
      "reused": true,
      "local_path": "attachments/d42206d9d0de/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/f4ecdf76-53bc-45bf-8782-0b368609dcac.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://jaivalsuthar.hashnode.dev/production-grade-async-job-queue-redis-bullmq.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@JaivalSuthar",
      "label": "J",
      "type": "a"
    },
    {
      "href": "https://x.com/Jaival_Suthar",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/jaival-suthar-57195b257/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/system-design",
      "label": "# system-design",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/nodejs",
      "label": "# nodejs",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/redis",
      "label": "# redis",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/backend",
      "label": "# backend",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:10:42",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:04Z",
  "source_url": "https://jaivalsuthar.hashnode.dev/production-grade-async-job-queue-redis-bullmq",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 30716,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/d42206d9d0de/pages/source.html",
  "local_html": "attachments/d42206d9d0de/pages/article.html",
  "body_base_path": "attachments/d42206d9d0de/pages/",
  "body_markdown_path": "attachments/d42206d9d0de/pages/article.md"
}