Section 2.5: Analysis of Algorithms

Abstract:

By analysis of algorithms we mean the study of the efficiency of the algorithms. In this section we will measure the efficiency of an algorithm by counting operations (and of course we are shooting for a small number!).

Counting operations directly

In algorithm SequentialSearch (p. 146/148), we search for element x in a list of n items. SequentialSearch is a direct method, by comparison with algorithm BinarySearch (p. 130), which is recursive. Is one algorithm more efficient than the other?

In the SequentialSearch, there are three rather interesting cases:

• we find x on the very first try (total comparisons: 1!). This is called the ``best-case'' scenario.
• we find x on the last try (total comparisons: n). This is the ``worst-case'' scenario.
• On average, we require n/2 comparisons.

We will consider the worst-case scenario as the benchmark.

Counting Using Recurrence Relations

Algorithm BinarySearch is recursive: it calls itself. Starting from a list of length n it makes one comparison and then calls itself with a list of half its initial length. Hence the number of comparisons for the list of length n, C(n), would be

and C(1)=1. Use the ``expand, guess, and verify'' approach: in the worst-case scenario, the algorithm will find the element (or not) on its last check (when it's down to a list of length 1).

Obviously this is only going to work (in the sense that C(n/8), etc., make sense) if n is a power of 2. Assume that , for integer m.

Consider a change of variable: in

we define so that

Note that T(0)=C(1)=1. We can solve easily to get

Hence, . This compares quite favorably with the worst-case estimate from SequentialSearch, which would be n (linear in n).

(For those of you who've forgotten, the function grows much more slowly than a linear function.)

Let's look at the general recurrence relation of the ``divide and conquer'' variety: given

Assume for some integer m. Then

Now we perform the change of variables: let , so that

Using formula (8) of section 2.4, p. 134, we get

Then reindexing, since we start with 0 rather than 1, we get

Finally, substituting back in S and n, we get

Whew!

The BinarySearch algorithm starts with a sorted list, which is not a requirement for the SequentialSearch algorithm; so the comparison isn't really fair. What if we add a sort?

Example: (Exercise 11/13, p. 154/156)

Example: (Exercise 12/14, p. 154/156)

Example: (Exercise 13/15, p. 154/156)

Example: (Exercise 14/16, p. 154/156)

So we can carry out the BinarySearch algorithm following a MergeSort with

or

operations, compared with n operations for SequentialSearch - which wins in this case!

If we had started with a sorted list, however, it would make no sense to use SequentialSearch, since BinarySearch is so much more efficient.

Other criteria

An algorithm should not be analyzed quite so one-dimensionally as we've done here, of course: there may be other issues (such as how easily parallelized an algorithm is, for example) which are more important than simple operation counts.

As demonstrated in the case of the Euclidean Algorithm (or gcd) in this section, we may simply be shooting for an upper bound on the number of operations required (even worse than the worst case scenario!), when actual worst-case numbers are hard to come by.