What you're seeing is called timer quantization error. One aspect of this error is actually something that's so well understood and accepted that it's even documented in a standard: the POSIX Realtime Extension (1003.1b-1993/1003.1i-1995). This document says that it's all right to delay too much, but it isn't all right to delay too little — the premature firing of a timer is undesirable.
Since the calling of delay() is asynchronous with the running of the clock interrupt, the kernel has to add one clock tick to a relative delay to ensure the correct amount of time (consider what would happen if it didn't, and a one-tick delay was requested just before the clock interrupt went off).
Figure 1. A single 1 ms sleep with error.That normally adds half a millisecond each time, but in the example given, you end up synchronized with the clock interrupt, so the full millisecond gets tacked on each time.
Figure 2. Twelve 1 ms sleeps with each one's error.The small error on each sleep accumulates:
Figure 3. Twelve 1 ms sleeps with the accumulated error.OK, that should make the loop last 2 seconds — where's the extra second coming from?