SitePulse

You upload a CSV of up to 200 URLs and get a response within seconds instead of waiting for 200 requests to finish. After that you can watch progress, filter by failure type, retry what failed and export a report.

Once the work sat behind a queue, I had to answer questions one at a time. If you double-click upload, does the job run twice? If a worker dies at URL 87, who finishes the rest? If a message arrives twice, does the progress counter double? Reading the code won't tell you. I had to go looking for where it would break.

• React uploads the CSV. FastAPI validates it, writes one job and up to 200 result rows, puts the job id on Redis and returns 202 right away.
• A Celery worker picks up the message, checks up to 10 URLs at a time and commits each result as soon as it's done. The per-URL commit is what lets you see progress.
• React polls the job once a second until it reaches a terminal state.
• PostgreSQL has two tables, jobs and check_results, joined by a foreign key with cascading deletes.
• Redis does three things: it's the message broker, the counter for per-IP rate limits and the visibility window for unacknowledged messages.
• Logs are one JSON object per line. The request id goes from the HTTP response header into the Celery message, so you can follow one upload through the API, the queue and the worker.

• Idempotent uploads rely on a unique index. An upload can carry an Idempotency-Key, and the file content is hashed with SHA-256 into a fingerprint. The same key with the same bytes returns the original job; the same key with different bytes returns 409. Checking first and writing second leaves a gap when two requests arrive together, so the unique index decides. The request that loses hits the constraint, catches the error and reads the existing job.
• Delivery is at least once, and the worker only processes rows that haven't run. The worker acknowledges a message after it finishes, a lost worker's messages are rejected and redelivered, and unacknowledged messages stay hidden for one hour. To keep two workers off the same job after a redelivery, the worker takes a PostgreSQL session-level advisory lock. The lock belongs to the database connection, so when a worker dies, the connection closes and the lock goes with it; I didn't need any timeout code to clean up stale locks. After a redelivery the worker only processes rows still marked queued, so results already committed aren't counted again.
• Only database errors and OS-level errors are retried, with a backoff of min(60, 5×2ⁿ) seconds. Any other exception marks the job failed and is raised again. I want problems in the environment retried and bugs in the code to show up immediately.
• SSRF checks run again on every hop. A literal IP and every address the hostname resolves to must be public, and the URL is rejected if any resolved address isn't. Redirects aren't followed automatically: each hop is validated and resolved again before the worker follows it. One gap remains, the time between the DNS check and the actual connection. I haven't closed it, and the docs say a production deployment also needs an outbound network policy.
• If Redis fails, the rate limiter lets requests through. The limit is ten job creations per minute per IP, kept in Redis; when Redis raises an error, the limiter logs it and allows the request. Anyone can use the tool, and I didn't want the limiter to take the whole service down with it.

There are 31 backend tests, run against a mocked HTTP transport so they never depend on a live website. They cover CSV parsing and deduplication, the full job lifecycle, idempotent replay and the 409, rate limiting, SSRF validation, size limits and response classification. CI starts real PostgreSQL and Redis containers, runs alembic upgrade head and an alembic check for model drift, then runs the tests, the frontend lint, type check and production build, and validates and builds the Compose stack.

The integration tests use a test client with a mocked transport and stub out the enqueue call, so CI never goes through Redis → Celery → worker. “At-least-once delivery, and a job resumes after a crash” is something I read out of the code; no test has run it. Testing it would mean killing a worker in the middle of a job.

There's also no load test or benchmark (the 150-URL sample file is for manual smoke tests), no metrics or tracing, no coverage gate (the “31 passing tests” figure lives in the README and CI doesn't enforce it), and no frontend or end-to-end tests. The limits are exact: a 1 MB upload, 200 URLs, an 8-second timeout per request, 10 concurrent checks, a 1 MB response body and 5 redirects.

• Once a flow sits behind a queue, any action can happen more than once. Duplicate jobs, doubled counters and two workers on one job were all versions of that.
• A unique index was a better judge of which request wins than application code that checks and then writes. That code always leaves a window under concurrency, and I don't control how wide it is.
• Tying the lock to the database connection saved me from writing timeout and cleanup code. The worker dies, the connection closes, and the lock is released.
• Following one upload through the logs helped me more than adding fields to them. Passing the request id from an HTTP header into the queue message took a few lines, and it's how I find out which of the three services failed.
• Writing the known gaps down, like the DNS-to-connect race and the missing outbound policy, tells whoever works on this next where to start.