Even though we pretend that we are calculating ``real'' values, we're making errors (truncation, round-off). Consider the case of the forward-difference formula. In this case, we compute with errors:
and
where e represents round-off error, and h>0.
Now the absolute error E in our derivative calculation will be made up of two parts:
so
where M>0 is a bound on the second derivative on the interval of interest,
and 
is a bound on the size of a truncation or round-off error.
The upshot: we can't just make h as small as we like, and expect approximations to get better and better: round-off error will ultimately kill us. We need to balance the round-off against small h - and we can even guess what value of h is appropriate, given a particular function (and its second derivative), and the size of round-off errors on your particular machine.
Example: #18 Here it is in lisp: