What happens between pressing the button and the download
A task, a queue, a worker and a signed URL — and why the work never happens inside the request that asked for it.
4 min read
The web server does not process anything
Resizing a 40-megapixel photograph takes seconds and a great deal of memory. Doing that inside the HTTP request that asked for it means the connection is held open throughout, a browser tab that gives up takes the work with it, and thirty simultaneous uploads can exhaust the machine that is also meant to be serving pages.
So the web server does none of it. It writes down what you want, accepts your files, puts a job on a queue and answers immediately. Separate worker processes take jobs off that queue and do the actual work.
The order of the checks is the design
Before a job is queued, several things are verified: that the task has not already been started, that the file count is within the limit, that the settings are valid for the tool, that the target address is safe when the tool fetches one, and finally that any human check has passed.
Only after all of that is your daily allowance charged. That order is deliberate and it decides who pays for a refused request: a typo in a URL, a batch that is too large or a failed challenge costs you nothing, because the counter is the last thing touched before the work is handed over.
Two queues, because the jobs are not alike
Compressing a JPEG and screenshotting a web page have nothing in common in what they cost. One is a short burst of processor time; the other launches a browser and holds a few hundred megabytes for the duration.
They run on separate queues with separate limits, in separate containers — one built with image tools and no browser, the other with a browser and none of the image tools. A process cannot pick up a job it has no way to run, and a flood of screenshots cannot starve the ordinary tools.
- Per process
- How many jobs one container runs at once.
- Cluster-wide
- A cap held centrally, so adding containers does not multiply the load a queue can place on the machine.
- Per file
- A batch is processed file by file, so one unreadable image fails on its own and the rest still finish.
How the page knows it is done
Progress is streamed to your browser as the worker reports it, rather than the page asking repeatedly. Frames are only sent when something has actually changed, so a task that is waiting its turn costs almost nothing on the wire.
When it finishes, the result is fetched through a signed URL that goes straight to storage rather than back through the application. A single file is a direct download; several arrive as a ZIP that is streamed as it is built, so a large batch never has to be assembled in memory first.
Frequently asked questions
- Why does my task sit at "queued" sometimes?
- Because a worker is busy with someone else. The queues have deliberate limits so that a heavy job cannot take the whole machine, which occasionally means waiting a moment for a slot.
- What happens if one file in my batch is broken?
- That file is marked with a readable reason and the rest are processed normally. One bad image does not cost you the other twenty-nine.
- Can I close the tab while it works?
- The work continues, but the result is only kept for two hours and the link is on the page you closed. Best to wait for it.