Docker Worked Example — The 2 GB Image and the 40-Minute Build
A worked example of fixing a build that had grown intolerable — where the time and the size actually went, the layer ordering that mattered most, and the security findings that came free.
This is an illustrative example. The team, service and figures are invented. The shape — an image and a build time that grew a little at a time until they were the main cost of shipping — is ordinary.
The situation#
A Python service, five years old, actively developed. The image was 2.1 GB and a build took 38 minutes.
The consequences had become the team's working conditions rather than a problem anyone was solving. Pull requests were batched, because nobody would wait 38 minutes twice. Deployments were scheduled rather than routine. Rolling out to a new node took four minutes of image pull before anything started, so scaling up during a traffic spike was useless — capacity arrived after the spike.
Where the 38 minutes went#
Measured, one layer at a time.
| Step | Time | Cached in practice? |
|---|---|---|
| Base image pull | 40 s | Yes |
apt-get update and install build tools | 3 min | No |
pip install — 94 packages | 11 min | No |
| Copy source | 2 s | No |
| Compile one native extension | 6 min | No |
| Run tests inside the build | 14 min | No |
| Package and push | 3 min | No |
Nothing after the base image was cached in practice, and the cause was one line. The Dockerfile copied the whole source tree before installing dependencies. Any code change invalidated that layer, and everything below it rebuilt — including 11 minutes of dependency installation that had not changed in six weeks.
The fixes, in order of what they returned#
Reorder: dependency manifest first, source second. Three lines moved. Dependencies now install in their own layer, invalidated only when the manifest changes. 11 minutes recovered on almost every build, for the smallest change in this story.
Move tests out of the image build. Tests ran inside the Dockerfile, so every build ran the full suite whether or not anything relevant had changed, and a test failure produced no image to inspect. Moved into a pipeline stage against the built image. 14 minutes recovered, and failures became debuggable.
Multi-stage build. The native extension needs a compiler; the running service does not. Compilation moved to a build stage, and only the compiled artefact crosses into the final image. 6 minutes stayed but stopped mattering, because that stage caches independently — and 1.4 GB of toolchain left the image.
Pin the base by digest. Not a speed change. It ended a class of intermittent failure the team had attributed to bad luck: the tag had been re-pointed twice, once introducing a system library version that broke the native build.
What the image contained#
Inspecting the 2.1 GB image, layer by layer, was an afternoon and it was uncomfortable.
| Size | |
|---|---|
| Build toolchain — gcc, headers, make | 780 MB |
| Package manager caches never cleaned | 310 MB |
| Test fixtures, including sample data files | 240 MB |
| Documentation and locale data | 90 MB |
.git directory | 180 MB |
| The application and its runtime dependencies | 210 MB |
Ten per cent of the image was the thing being shipped.
The .git directory was the finding that stopped the review. There was no .dockerignore, so COPY . . had been copying the full history into every image for five years. It contained the complete commit history, and in that history two credentials that had been committed and later removed — which removes them from the working tree and not from the history.
Both were rotated. Neither had been known about.
The result#
| Before | After | |
|---|---|---|
| Image size | 2.1 GB | 185 MB |
| Cold build | 38 min | 9 min |
| Build after a code change | 38 min | 90 s |
| Image pull on a new node | 4 min | 20 s |
| Runs as | root | non-root |
| Root filesystem | writable | read-only |
.git in image | yes | no |
Ninety seconds is the number that changed how the team works. Pull requests stopped being batched. Deploying became routine rather than scheduled. Scaling up during a spike now adds capacity while the spike is still happening.
The security findings that came free#
None of these were the goal, and all of them were found by looking at the image properly.
- Two credentials in the
.githistory, rotated - The service had been running as root for five years, with no reason
- The image shipped a package manager and a shell that nothing used, both removed with the build stage
- A vulnerability scan that had been reporting 140 findings now reports 6, because most had been in the build toolchain rather than in anything that runs
The scanner had been reporting those 140 findings for years. They were dismissed as noise, and they were noise — but only because nobody had asked why a compiler was in a production image.
What was learned#
Layer order is the whole of build performance. One misplaced COPY cost 11 minutes on every build for five years, and moving three lines fixed it.
Tests do not belong in the image build. They were there because it was convenient once, and it cost 14 minutes per build and made every failure undebuggable.
Look inside the image. An afternoon of layer inspection found 1.9 GB of things that should not ship, two leaked credentials and the reason the vulnerability scan was ignored.
.dockerignore is not optional. Without it, COPY . . ships everything — history, fixtures, environment files, anything a developer left in the directory.