This page intentionally left blank
Vice President and Editorial Director, ECS EditorinChief Acquisitions Editor Editorial Assistant Vice President, Marketing Marketing Manager Senior Marketing Coordinator Marketing Assistant Vice President, Production Managing Editor Production Project Manager Senior Operations Supervisor Manufacturing Buyer Art Director Text Designer Cover Designer Cover Illustration Media Editor FullService Project Management Composition Printer/Binder Cover Printer Text Font
Marcia Horton Michael Hirsch Matt Goldstein Chelsea Bell Patrice Jones Yezan Alayan Kathryn Ferranti Emma Snider Vince O’Brien Jeff Holcomb Kayla SmithTarbox Alan Fischer Lisa McDowell Anthony Gemmellaro Sandra Rigney Anthony Gemmellaro Jennifer Kohnke Daniel Sandin Windfall Software Windfall Software, using ZzTEX Courier Westford Courier Westford Times Ten
Copyright © 2012, 2007, 2003 Pearson Education, Inc., publishing as AddisonWesley. All rights reserved. Printed in the United States of America. This publication is protected by Copyright, and permission should be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. To obtain permission(s) to use material from this work, please submit a written request to Pearson Education, Inc., Permissions Department, One Lake Street, Upper Saddle River, New Jersey 07458, or you may fax your request to 2012363290. This is the eBook of the printed book and may not include any media, Website access codes or print supplements that may come packaged with the bound book. Many of the designations by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed in initial caps or all caps. Library of Congress CataloginginPublication Data Levitin, Anany. Introduction to the design & analysis of algorithms / Anany Levitin. — 3rd ed. p. cm. Includes bibliographical references and index. ISBN13: 9780132316811 ISBN10: 0132316811 1. Computer algorithms. I. Title. II. Title: Introduction to the design and analysis of algorithms. QA76.9.A43L48 2012 005.1—dc23 2011027089 15 14 13 12 11—CRW—10 9 8 7 6 5 4 3 2 1 ISBN 10: 0132316811 ISBN 13: 9780132316811
Boston Columbus Indianapolis New York San Francisco Upper Saddle River Amsterdam Cape Town Dubai London Madrid Milan Munich Paris Montreal Toronto Delhi Mexico City Sao Paulo Sydney Hong Kong Seoul Singapore Taipei Tokyo
This page intentionally left blank
Brief Contents
New to the Third Edition
xvii
Preface
xix
1 Introduction
1
2 Fundamentals of the Analysis of Algorithm Efﬁciency
41
3 Brute Force and Exhaustive Search
97
4 DecreaseandConquer
131
5 DivideandConquer
169
6 TransformandConquer
201
7 Space and Time TradeOffs
253
8 Dynamic Programming
283
9 Greedy Technique
315
10 Iterative Improvement
345
11 Limitations of Algorithm Power
387
12 Coping with the Limitations of Algorithm Power
423
Epilogue
471
APPENDIX A
Useful Formulas for the Analysis of Algorithms
475
APPENDIX B
Short Tutorial on Recurrence Relations
479
References
493
Hints to Exercises
503
Index
547 v
This page intentionally left blank
Contents
New to the Third Edition
xvii
Preface
xix
1 Introduction
1
1.1 What Is an Algorithm?
3
Exercises 1.1
1.2 Fundamentals of Algorithmic Problem Solving Understanding the Problem Ascertaining the Capabilities of the Computational Device Choosing between Exact and Approximate Problem Solving Algorithm Design Techniques Designing an Algorithm and Data Structures Methods of Specifying an Algorithm Proving an Algorithm’s Correctness Analyzing an Algorithm Coding an Algorithm Exercises 1.2
1.3 Important Problem Types Sorting Searching String Processing Graph Problems Combinatorial Problems Geometric Problems Numerical Problems Exercises 1.3
7 9 9 9 11 11 12 12 13 14 15 17 18 19 20 20 21 21 22 22 23
vii
viii
Contents
1.4 Fundamental Data Structures Linear Data Structures Graphs Trees Sets and Dictionaries Exercises 1.4
Summary
25 25 28 31 35 37 38
2 Fundamentals of the Analysis of Algorithm Efﬁciency
2.1 The Analysis Framework
41
Measuring an Input’s Size Units for Measuring Running Time Orders of Growth WorstCase, BestCase, and AverageCase Efﬁciencies Recapitulation of the Analysis Framework
42 43 44 45 47 50
Exercises 2.1
50
2.2 Asymptotic Notations and Basic Efﬁciency Classes Informal Introduction Onotation notation notation Useful Property Involving the Asymptotic Notations Using Limits for Comparing Orders of Growth Basic Efﬁciency Classes Exercises 2.2
2.3 Mathematical Analysis of Nonrecursive Algorithms Exercises 2.3
2.4 Mathematical Analysis of Recursive Algorithms Exercises 2.4
2.5 Example: Computing the nth Fibonacci Number Exercises 2.5
2.6 Empirical Analysis of Algorithms Exercises 2.6
2.7 Algorithm Visualization Summary
52 52 53 54 55 55 56 58 58 61 67 70 76 80 83 84 89 91 94
Contents
3 Brute Force and Exhaustive Search 3.1 Selection Sort and Bubble Sort Selection Sort Bubble Sort Exercises 3.1
3.2 Sequential Search and BruteForce String Matching Sequential Search BruteForce String Matching Exercises 3.2
3.3 ClosestPair and ConvexHull Problems by Brute Force ClosestPair Problem ConvexHull Problem Exercises 3.3
3.4 Exhaustive Search
ix
97 98 98 100 102 104 104 105 106 108 108 109 113
Traveling Salesman Problem Knapsack Problem Assignment Problem
115 116 116 119
Exercises 3.4
120
3.5 DepthFirst Search and BreadthFirst Search DepthFirst Search BreadthFirst Search Exercises 3.5
Summary
122 122 125 128 130
4 DecreaseandConquer
131
4.1 Insertion Sort
134
Exercises 4.1
4.2 Topological Sorting Exercises 4.2
4.3 Algorithms for Generating Combinatorial Objects Generating Permutations Generating Subsets Exercises 4.3
136 138 142 144 144 146 148
x
Contents
4.4 DecreasebyaConstantFactor Algorithms Binary Search FakeCoin Problem Russian Peasant Multiplication Josephus Problem
150 150 152 153 154
Exercises 4.4
156
4.5 VariableSizeDecrease Algorithms Computing a Median and the Selection Problem Interpolation Search Searching and Insertion in a Binary Search Tree The Game of Nim Exercises 4.5
Summary
157 158 161 163 164 166 167
5 DivideandConquer
169
5.1 Mergesort
172
Exercises 5.1
5.2 Quicksort Exercises 5.2
5.3 Binary Tree Traversals and Related Properties Exercises 5.3
5.4 Multiplication of Large Integers and Strassen’s Matrix Multiplication Multiplication of Large Integers Strassen’s Matrix Multiplication Exercises 5.4
5.5 The ClosestPair and ConvexHull Problems by DivideandConquer The ClosestPair Problem ConvexHull Problem Exercises 5.5
Summary
174 176 181 182 185
186 187 189 191
192 192 195 197 198
Contents
6 TransformandConquer 6.1 Presorting Exercises 6.1
6.2 Gaussian Elimination LU Decomposition Computing a Matrix Inverse Computing a Determinant Exercises 6.2
6.3 Balanced Search Trees
xi
201 202 205 208 212 214 215 216
AVL Trees 23 Trees
218 218 223
Exercises 6.3
225
6.4 Heaps and Heapsort Notion of the Heap Heapsort Exercises 6.4
6.5 Horner’s Rule and Binary Exponentiation Horner’s Rule Binary Exponentiation Exercises 6.5
6.6 Problem Reduction Computing the Least Common Multiple Counting Paths in a Graph Reduction of Optimization Problems Linear Programming Reduction to Graph Problems Exercises 6.6
Summary
226 227 231 233 234 234 236 239 240 241 242 243 244 246 248 250
7 Space and Time TradeOffs
253
7.1 Sorting by Counting
254
Exercises 7.1
7.2 Input Enhancement in String Matching Horspool’s Algorithm
257 258 259
xii
Contents
BoyerMoore Algorithm
263
Exercises 7.2
267
7.3 Hashing Open Hashing (Separate Chaining) Closed Hashing (Open Addressing) Exercises 7.3
7.4 BTrees Exercises 7.4
Summary
269 270 272 274 276 279 280
8 Dynamic Programming
283
8.1 Three Basic Examples
285
Exercises 8.1
8.2 The Knapsack Problem and Memory Functions
290
Memory Functions
292 294
Exercises 8.2
296
8.3 Optimal Binary Search Trees Exercises 8.3
8.4 Warshall’s and Floyd’s Algorithms Warshall’s Algorithm Floyd’s Algorithm for the AllPairs ShortestPaths Problem Exercises 8.4
Summary
297 303 304 304 308 311 312
9 Greedy Technique
315
9.1 Prim’s Algorithm
318
Exercises 9.1
9.2 Kruskal’s Algorithm
322
Disjoint Subsets and UnionFind Algorithms
325 327
Exercises 9.2
331
9.3 Dijkstra’s Algorithm Exercises 9.3
333 337
Contents
9.4 Huffman Trees and Codes Exercises 9.4
Summary
10 Iterative Improvement 10.1 The Simplex Method Geometric Interpretation of Linear Programming An Outline of the Simplex Method Further Notes on the Simplex Method Exercises 10.1
10.2 The MaximumFlow Problem Exercises 10.2
10.3 Maximum Matching in Bipartite Graphs Exercises 10.3
10.4 The Stable Marriage Problem Exercises 10.4
Summary
11 Limitations of Algorithm Power 11.1 LowerBound Arguments Trivial Lower Bounds InformationTheoretic Arguments Adversary Arguments Problem Reduction Exercises 11.1
11.2 Decision Trees Decision Trees for Sorting Decision Trees for Searching a Sorted Array Exercises 11.2
11.3 P , NP , and NPComplete Problems P and NP Problems NP Complete Problems Exercises 11.3
xiii 338 342 344
345 346 347 351 357 359 361 371 372 378 380 383 384
387 388 389 390 390 391 393 394 395 397 399 401 402 406 409
xiv
Contents
11.4 Challenges of Numerical Algorithms Exercises 11.4
Summary
12 Coping with the Limitations of Algorithm Power 12.1 Backtracking
412 419 420
423
nQueens Problem Hamiltonian Circuit Problem SubsetSum Problem General Remarks
424 425 426 427 428
Exercises 12.1
430
12.2 BranchandBound Assignment Problem Knapsack Problem Traveling Salesman Problem
432 433 436 438
Exercises 12.2
440
12.3 Approximation Algorithms for NP Hard Problems Approximation Algorithms for the Traveling Salesman Problem Approximation Algorithms for the Knapsack Problem Exercises 12.3
12.4 Algorithms for Solving Nonlinear Equations
441 443 453 457
Exercises 12.4
459 460 464 464 467
Summary
468
Epilogue
471
Bisection Method Method of False Position Newton’s Method
APPENDIX A
Useful Formulas for the Analysis of Algorithms Properties of Logarithms Combinatorics Important Summation Formulas Sum Manipulation Rules
475 475 475 476 476
Contents
Approximation of a Sum by a Deﬁnite Integral Floor and Ceiling Formulas Miscellaneous
xv 477 477 477
APPENDIX B
Short Tutorial on Recurrence Relations Sequences and Recurrence Relations Methods for Solving Recurrence Relations Common Recurrence Types in Algorithm Analysis
479 479 480 485
References
493
Hints to Exercises
503
Index
547
This page intentionally left blank
New to the Third Edition
Reordering of chapters to introduce decreaseandconquer before divideandconquer Restructuring of chapter 8 on dynamic programming, including all new introductory material and new exercises focusing on wellknown applications More coverage of the applications of the algorithms discussed Reordering of select sections throughout the book to achieve a better alignment of speciﬁc algorithms and general algorithm design techniques Addition of the Lomuto partition and Gray code algorithms Seventy new problems added to the endofchapter exercises, including algorithmic puzzles and questions asked during job interviews
xvii
This page intentionally left blank
Preface
The most valuable acquisitions in a scientiﬁc or technical education are the generalpurpose mental tools which remain serviceable for a lifetime. —George Forsythe, “What to do till the computer scientist comes.” (1968)
A
lgorithms play the central role both in the science and practice of computing. Recognition of this fact has led to the appearance of a considerable number of textbooks on the subject. By and large, they follow one of two alternatives in presenting algorithms. One classiﬁes algorithms according to a problem type. Such a book would have separate chapters on algorithms for sorting, searching, graphs, and so on. The advantage of this approach is that it allows an immediate comparison of, say, the efﬁciency of different algorithms for the same problem. The drawback of this approach is that it emphasizes problem types at the expense of algorithm design techniques. The second alternative organizes the presentation around algorithm design techniques. In this organization, algorithms from different areas of computing are grouped together if they have the same design approach. I share the belief of many (e.g., [BaY95]) that this organization is more appropriate for a basic course on the design and analysis of algorithms. There are three principal reasons for emphasis on algorithm design techniques. First, these techniques provide a student with tools for designing algorithms for new problems. This makes learning algorithm design techniques a very valuable endeavor from a practical standpoint. Second, they seek to classify multitudes of known algorithms according to an underlying design idea. Learning to see such commonality among algorithms from different application areas should be a major goal of computer science education. After all, every science considers classiﬁcation of its principal subject as a major if not the central point of its discipline. Third, in my opinion, algorithm design techniques have utility as general problem solving strategies, applicable to problems beyond computing. xix
xx
Preface
Unfortunately, the traditional classiﬁcation of algorithm design techniques has several serious shortcomings, from both theoretical and educational points of view. The most signiﬁcant of these shortcomings is the failure to classify many important algorithms. This limitation has forced the authors of other textbooks to depart from the design technique organization and to include chapters dealing with speciﬁc problem types. Such a switch leads to a loss of course coherence and almost unavoidably creates a confusion in students’ minds.
New taxonomy of algorithm design techniques My frustration with the shortcomings of the traditional classiﬁcation of algorithm design techniques has motivated me to develop a new taxonomy of them [Lev99], which is the basis of this book. Here are the principal advantages of the new taxonomy: The new taxonomy is more comprehensive than the traditional one. It includes several strategies—bruteforce, decreaseandconquer, transformandconquer, space and time tradeoffs, and iterative improvement—that are rarely if ever recognized as important design paradigms. The new taxonomy covers naturally many classic algorithms (Euclid’s algorithm, heapsort, search trees, hashing, topological sorting, Gaussian elimination, Horner’s rule—to name a few) that the traditional taxonomy cannot classify. As a result, the new taxonomy makes it possible to present the standard body of classic algorithms in a uniﬁed and coherent fashion. It naturally accommodates the existence of important varieties of several design techniques. For example, it recognizes three variations of decreaseandconquer and three variations of transformandconquer. It is better aligned with analytical methods for the efﬁciency analysis (see Appendix B).
Design techniques as general problem solving strategies Most applications of the design techniques in the book are to classic problems of computer science. (The only innovation here is an inclusion of some material on numerical algorithms, which are covered within the same general framework.) But these design techniques can be considered general problem solving tools, whose applications are not limited to traditional computing and mathematical problems. Two factors make this point particularly important. First, more and more computing applications go beyond the traditional domain, and there are reasons to believe that this trend will strengthen in the future. Second, developing students’ problem solving skills has come to be recognized as a major goal of college education. Among all the courses in a computer science curriculum, a course on the design and analysis of algorithms is uniquely suitable for this task because it can offer a student speciﬁc strategies for solving problems. I am not proposing that a course on the design and analysis of algorithms should become a course on general problem solving. But I do believe that the
Preface
xxi
unique opportunity provided by studying the design and analysis of algorithms should not be missed. Toward this goal, the book includes applications to puzzles and puzzlelike games. Although using puzzles in teaching algorithms is certainly not a new idea, the book tries to do this systematically by going well beyond a few standard examples.
Textbook pedagogy My goal was to write a text that would not trivialize the subject but would still be readable by most students on their own. Here are some of the things done toward this objective. Sharing the opinion of George Forsythe expressed in the epigraph, I have sought to stress major ideas underlying the design and analysis of algorithms. In choosing speciﬁc algorithms to illustrate these ideas, I limited the number of covered algorithms to those that demonstrate an underlying design technique or an analysis method most clearly. Fortunately, most classic algorithms satisfy this criterion. In Chapter 2, which is devoted to efﬁciency analysis, the methods used for analyzing nonrecursive algorithms are separated from those typically used for analyzing recursive algorithms. The chapter also includes sections devoted to empirical analysis and algorithm visualization. The narrative is systematically interrupted by questions to the reader. Some of them are asked rhetorically, in anticipation of a concern or doubt, and are answered immediately. The goal of the others is to prevent the reader from drifting through the text without a satisfactory level of comprehension. Each chapter ends with a summary recapping the most important concepts and results discussed in the chapter. The book contains over 600 exercises. Some of them are drills; others make important points about the material covered in the body of the text or introduce algorithms not covered there at all. A few exercises take advantage of Internet resources. More difﬁcult problems—there are not many of them— are marked by special symbols in the Instructor’s Manual. (Because marking problems as difﬁcult may discourage some students from trying to tackle them, problems are not marked in the book itself.) Puzzles, games, and puzzlelike questions are marked in the exercises with a special icon. The book provides hints to all the exercises. Detailed solutions, except for programming projects, are provided in the Instructor’s Manual, available to qualiﬁed adopters through Pearson’s Instructor Resource Center. (Please contact your local Pearson sales representative or go to www.pearsonhighered .com/irc to access this material.) Slides in PowerPoint are available to all readers of this book via anonymous ftp at the CS Support site: http://cssupport .pearsoncmg.com/.
xxii
Preface
Changes for the third edition There are a few changes in the third edition. The most important is the new order of the chapters on decreaseandconquer and divideandconquer. There are several advantages in introducing decreaseandconquer before divideandconquer: Decreaseandconquer is a simpler strategy than divideandconquer. Decreaseandconquer is applicable to more problems than divideandconquer. The new order makes it possible to discuss insertion sort before mergesort and quicksort. The idea of array partitioning is now introduced in conjunction with the selection problem. I took advantage of an opportunity to do this via the onedirectional scan employed by Lomuto’s algorithm, leaving the twodirectional scan used by Hoare’s partitioning to a later discussion in conjunction with quicksort. Binary search is now considered in the section devoted to decreasebyaconstantfactor algorithms, where it belongs. The second important change is restructuring of Chapter 8 on dynamic programming. Speciﬁcally: The introductory section is completely new. It contains three basic examples that provide a much better introduction to this important technique than computing a binomial coefﬁcient, the example used in the ﬁrst two editions. All the exercises for Section 8.1 are new as well; they include wellknown applications not available in the previous editions. I also changed the order of the other sections in this chapter to get a smoother progression from the simpler applications to the more advanced ones. The other changes include the following. More applications of the algorithms discussed are included. The section on the graphtraversal algorithms is moved from the decreaseandconquer chapter to the bruteforce and exhaustivesearch chapter, where it ﬁts better, in my opinion. The Gray code algorithm is added to the section dealing with algorithms for generating combinatorial objects. The divideandconquer algorithm for the closestpair problem is discussed in more detail. Updates include the section on algorithm visualization, approximation algorithms for the traveling salesman problem, and, of course, the bibliography. I also added about 70 new problems to the exercises. Some of them are algorithmic puzzles and questions asked during job interviews.
Prerequisites The book assumes that a reader has gone through an introductory programming course and a standard course on discrete structures. With such a background, he or she should be able to handle the book’s material without undue difﬁculty.
xxiii
Preface
Still, fundamental data structures, necessary summation formulas, and recurrence relations are reviewed in Section 1.4, Appendix A, and Appendix B, respectively. Calculus is used in only three sections (Section 2.2, 11.4, and 12.4), and to a very limited degree; if students lack calculus as an assured part of their background, the relevant portions of these three sections can be omitted without hindering their understanding of the rest of the material.
Use in the curriculum The book can serve as a textbook for a basic course on design and analysis of algorithms organized around algorithm design techniques. It might contain slightly more material than can be covered in a typical onesemester course. By and large, portions of Chapters 3 through 12 can be skipped without the danger of making later parts of the book incomprehensible to the reader. Any portion of the book can be assigned for selfstudy. In particular, Sections 2.6 and 2.7 on empirical analysis and algorithm visualization, respectively, can be assigned in conjunction with projects. Here is a possible plan for a onesemester course; it assumes a 40class meeting format. Lecture
Topic
Sections
1 2, 3 4 5, 6 7 8 9 10, 11 12
Introduction Analysis framework; O, , notations Mathematical analysis of nonrecursive algorithms Mathematical analysis of recursive algorithms Bruteforce algorithms Exhaustive search Depthﬁrst search and breadthﬁrst search Decreasebyone: insertion sort, topological sorting Binary search and other decreasebyaconstantfactor algorithms Variablesizedecrease algorithms Divideandconquer: mergesort, quicksort Other divideandconquer examples Instance simpliﬁcation: presorting, Gaussian elimination, balanced search trees Representation change: heaps and heapsort or Horner’s rule and binary exponentiation Problem reduction Spacetime tradeoffs: string matching, hashing, Btrees Dynamic programming algorithms
1.1–1.3 2.1, 2.2 2.3 2.4, 2.5 (+ App. B) 3.1, 3.2 (+ 3.3) 3.4 3.5 4.1, 4.2 4.4
13 14, 15 16 17–19 20 21 22–24 25–27
4.5 5.1–5.2 5.3 or 5.4 or 5.5 6.1–6.3 6.4 or 6.5 6.6 7.2–7.4 3 from 8.1–8.4
xxiv
Preface 28–30 31–33 34 35 36 37 38 39 40
Greedy algorithms: Prim’s, Kruskal’s, Dijkstra’s, Huffman’s Iterative improvement algorithms Lowerbound arguments Decision trees P, NP, and NPcomplete problems Numerical algorithms Backtracking Branchandbound Approximation algorithms for NPhard problems
9.1–9.4 3 from 10.1–10.4 11.1 11.2 11.3 11.4 (+ 12.4) 12.1 12.2 12.3
Acknowledgments I would like to express my gratitude to the reviewers and many readers who have shared with me their opinions about the ﬁrst two editions of the book and suggested improvements and corrections. The third edition has certainly beneﬁted from the reviews by Andrew Harrington (Loyola University Chicago), David Levine (Saint Bonaventure University), Stefano Lombardi (UC Riverside), Daniel McKee (Mansﬁeld University), Susan Brilliant (Virginia Commonwealth University), David Akers (University of Puget Sound), and two anonymous reviewers. My thanks go to all the people at Pearson and their associates who worked on my book. I am especially grateful to my editor, Matt Goldstein; the editorial assistant, Chelsea Bell; the marketing manager, Yez Alayan; and the production supervisor, Kayla SmithTarbox. I am also grateful to Richard Camp for copyediting the book, Paul Anagnostopoulos of Windfall Software and Jacqui Scarlott for its project management and typesetting, and MaryEllen Oliver for proofreading the book. Finally, I am indebted to two members of my family. Living with a spouse writing a book is probably more trying than doing the actual writing. My wife, Maria, lived through several years of this, helping me any way she could. And help she did: over 400 ﬁgures in the book and the Instructor’s Manual were created by her. My daughter Miriam has been my English prose guru over many years. She read large portions of the book and was instrumental in ﬁnding the chapter epigraphs. Anany Levitin
[email protected] June 2011
This page intentionally left blank
1 Introduction Two ideas lie gleaming on the jeweler’s velvet. The ﬁrst is the calculus, the second, the algorithm. The calculus and the rich body of mathematical analysis to which it gave rise made modern science possible; but it has been the algorithm that has made possible the modern world. —David Berlinski, The Advent of the Algorithm, 2000
W
hy do you need to study algorithms? If you are going to be a computer professional, there are both practical and theoretical reasons to study algorithms. From a practical standpoint, you have to know a standard set of important algorithms from different areas of computing; in addition, you should be able to design new algorithms and analyze their efﬁciency. From the theoretical standpoint, the study of algorithms, sometimes called algorithmics, has come to be recognized as the cornerstone of computer science. David Harel, in his delightful book pointedly titled Algorithmics: the Spirit of Computing, put it as follows: Algorithmics is more than a branch of computer science. It is the core of computer science, and, in all fairness, can be said to be relevant to most of science, business, and technology. [Har92, p. 6] But even if you are not a student in a computingrelated program, there are compelling reasons to study algorithms. To put it bluntly, computer programs would not exist without algorithms. And with computer applications becoming indispensable in almost all aspects of our professional and personal lives, studying algorithms becomes a necessity for more and more people. Another reason for studying algorithms is their usefulness in developing analytical skills. After all, algorithms can be seen as special kinds of solutions to problems—not just answers but precisely deﬁned procedures for getting answers. Consequently, speciﬁc algorithm design techniques can be interpreted as problemsolving strategies that can be useful regardless of whether a computer is involved. Of course, the precision inherently imposed by algorithmic thinking limits the kinds of problems that can be solved with an algorithm. You will not ﬁnd, for example, an algorithm for living a happy life or becoming rich and famous. On 1
2
Introduction
the other hand, this required precision has an important educational advantage. Donald Knuth, one of the most prominent computer scientists in the history of algorithmics, put it as follows: A person welltrained in computer science knows how to deal with algorithms: how to construct them, manipulate them, understand them, analyze them. This knowledge is preparation for much more than writing good computer programs; it is a generalpurpose mental tool that will be a deﬁnite aid to the understanding of other subjects, whether they be chemistry, linguistics, or music, etc. The reason for this may be understood in the following way: It has often been said that a person does not really understand something until after teaching it to someone else. Actually, a person does not really understand something until after teaching it to a computer, i.e., expressing it as an algorithm . . . An attempt to formalize things as algorithms leads to a much deeper understanding than if we simply try to comprehend things in the traditional way. [Knu96, p. 9] We take up the notion of algorithm in Section 1.1. As examples, we use three algorithms for the same problem: computing the greatest common divisor. There are several reasons for this choice. First, it deals with a problem familiar to everybody from their middleschool days. Second, it makes the important point that the same problem can often be solved by several algorithms. Quite typically, these algorithms differ in their idea, level of sophistication, and efﬁciency. Third, one of these algorithms deserves to be introduced ﬁrst, both because of its age—it appeared in Euclid’s famous treatise more than two thousand years ago—and its enduring power and importance. Finally, investigation of these three algorithms leads to some general observations about several important properties of algorithms in general. Section 1.2 deals with algorithmic problem solving. There we discuss several important issues related to the design and analysis of algorithms. The different aspects of algorithmic problem solving range from analysis of the problem and the means of expressing an algorithm to establishing its correctness and analyzing its efﬁciency. The section does not contain a magic recipe for designing an algorithm for an arbitrary problem. It is a wellestablished fact that such a recipe does not exist. Still, the material of Section 1.2 should be useful for organizing your work on designing and analyzing algorithms. Section 1.3 is devoted to a few problem types that have proven to be particularly important to the study of algorithms and their application. In fact, there are textbooks (e.g., [Sed11]) organized around such problem types. I hold the view—shared by many others—that an organization based on algorithm design techniques is superior. In any case, it is very important to be aware of the principal problem types. Not only are they the most commonly encountered problem types in reallife applications, they are used throughout the book to demonstrate particular algorithm design techniques. Section 1.4 contains a review of fundamental data structures. It is meant to serve as a reference rather than a deliberate discussion of this topic. If you need
1.1
What Is an Algorithm?
3
a more detailed exposition, there is a wealth of good books on the subject, most of them tailored to a particular programming language.
1.1 What Is an Algorithm? Although there is no universally agreedon wording to describe this notion, there is general agreement about what the concept means: An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a ﬁnite amount of time. This deﬁnition can be illustrated by a simple diagram (Figure 1.1). The reference to “instructions” in the deﬁnition implies that there is something or someone capable of understanding and following the instructions given. We call this a “computer,” keeping in mind that before the electronic computer was invented, the word “computer” meant a human being involved in performing numeric calculations. Nowadays, of course, “computers” are those ubiquitous electronic devices that have become indispensable in almost everything we do. Note, however, that although the majority of algorithms are indeed intended for eventual computer implementation, the notion of algorithm does not depend on such an assumption. As examples illustrating the notion of the algorithm, we consider in this section three methods for solving the same problem: computing the greatest common divisor of two integers. These examples will help us to illustrate several important points: The nonambiguity requirement for each step of an algorithm cannot be compromised. The range of inputs for which an algorithm works has to be speciﬁed carefully. The same algorithm can be represented in several different ways. There may exist several algorithms for solving the same problem.
problem
algorithm
input
"computer"
FIGURE 1.1 The notion of the algorithm.
output
4
Introduction
Algorithms for the same problem can be based on very different ideas and can solve the problem with dramatically different speeds. Recall that the greatest common divisor of two nonnegative, notbothzero integers m and n, denoted gcd(m, n), is deﬁned as the largest integer that divides both m and n evenly, i.e., with a remainder of zero. Euclid of Alexandria (third century b.c.) outlined an algorithm for solving this problem in one of the volumes of his Elements most famous for its systematic exposition of geometry. In modern terms, Euclid’s algorithm is based on applying repeatedly the equality gcd(m, n) = gcd(n, m mod n), where m mod n is the remainder of the division of m by n, until m mod n is equal to 0. Since gcd(m, 0) = m (why?), the last value of m is also the greatest common divisor of the initial m and n. For example, gcd(60, 24) can be computed as follows: gcd(60, 24) = gcd(24, 12) = gcd(12, 0) = 12. (If you are not impressed by this algorithm, try ﬁnding the greatest common divisor of larger numbers, such as those in Problem 6 in this section’s exercises.) Here is a more structured description of this algorithm: Euclid’s algorithm for computing gcd(m, n) Step 1 If n = 0, return the value of m as the answer and stop; otherwise, proceed to Step 2. Step 2 Divide m by n and assign the value of the remainder to r. Step 3 Assign the value of n to m and the value of r to n. Go to Step 1. Alternatively, we can express the same algorithm in pseudocode: ALGORITHM Euclid(m, n) //Computes gcd(m, n) by Euclid’s algorithm //Input: Two nonnegative, notbothzero integers m and n //Output: Greatest common divisor of m and n while n = 0 do r ← m mod n m←n n←r return m How do we know that Euclid’s algorithm eventually comes to a stop? This follows from the observation that the second integer of the pair gets smaller with each iteration and it cannot become negative. Indeed, the new value of n on the next iteration is m mod n, which is always smaller than n (why?). Hence, the value of the second integer eventually becomes 0, and the algorithm stops.
1.1
What Is an Algorithm?
5
Just as with many other problems, there are several algorithms for computing the greatest common divisor. Let us look at the other two methods for this problem. The ﬁrst is simply based on the deﬁnition of the greatest common divisor of m and n as the largest integer that divides both numbers evenly. Obviously, such a common divisor cannot be greater than the smaller of these numbers, which we will denote by t = min{m, n}. So we can start by checking whether t divides both m and n: if it does, t is the answer; if it does not, we simply decrease t by 1 and try again. (How do we know that the process will eventually stop?) For example, for numbers 60 and 24, the algorithm will try ﬁrst 24, then 23, and so on, until it reaches 12, where it stops. Consecutive integer checking algorithm for computing gcd(m, n) Step 1 Assign the value of min{m, n} to t. Step 2 Divide m by t. If the remainder of this division is 0, go to Step 3; otherwise, go to Step 4. Step 3 Divide n by t. If the remainder of this division is 0, return the value of t as the answer and stop; otherwise, proceed to Step 4. Step 4 Decrease the value of t by 1. Go to Step 2. Note that unlike Euclid’s algorithm, this algorithm, in the form presented, does not work correctly when one of its input numbers is zero. This example illustrates why it is so important to specify the set of an algorithm’s inputs explicitly and carefully. The third procedure for ﬁnding the greatest common divisor should be familiar to you from middle school. Middleschool procedure for computing gcd(m, n) Step 1 Find the prime factors of m. Step 2 Find the prime factors of n. Step 3 Identify all the common factors in the two prime expansions found in Step 1 and Step 2. (If p is a common factor occurring pm and pn times in m and n, respectively, it should be repeated min{pm, pn} times.) Step 4 Compute the product of all the common factors and return it as the greatest common divisor of the numbers given. Thus, for the numbers 60 and 24, we get 60 = 2 . 2 . 3 . 5 24 = 2 . 2 . 2 . 3 gcd(60, 24) = 2 . 2 . 3 = 12. Nostalgia for the days when we learned this method should not prevent us from noting that the last procedure is much more complex and slower than Euclid’s algorithm. (We will discuss methods for ﬁnding and comparing running times of algorithms in the next chapter.) In addition to inferior efﬁciency, the middleschool procedure does not qualify, in the form presented, as a legitimate algorithm. Why? Because the prime factorization steps are not deﬁned unambiguously: they
6
Introduction
require a list of prime numbers, and I strongly suspect that your middleschool math teacher did not explain how to obtain such a list. This is not a matter of unnecessary nitpicking. Unless this issue is resolved, we cannot, say, write a program implementing this procedure. Incidentally, Step 3 is also not deﬁned clearly enough. Its ambiguity is much easier to rectify than that of the factorization steps, however. How would you ﬁnd common elements in two sorted lists? So, let us introduce a simple algorithm for generating consecutive primes not exceeding any given integer n > 1. It was probably invented in ancient Greece and is known as the sieve of Eratosthenes (ca. 200 b.c.). The algorithm starts by initializing a list of prime candidates with consecutive integers from 2 to n. Then, on its ﬁrst iteration, the algorithm eliminates from the list all multiples of 2, i.e., 4, 6, and so on. Then it moves to the next item on the list, which is 3, and eliminates its multiples. (In this straightforward version, there is an overhead because some numbers, such as 6, are eliminated more than once.) No pass for number 4 is needed: since 4 itself and all its multiples are also multiples of 2, they were already eliminated on a previous pass. The next remaining number on the list, which is used on the third pass, is 5. The algorithm continues in this fashion until no more numbers can be eliminated from the list. The remaining integers of the list are the primes needed. As an example, consider the application of the algorithm to ﬁnding the list of primes not exceeding n = 25: 2 2 2 2
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 3 5 7 9 11 13 15 17 19 21 23 25 3 5 7 11 13 17 19 23 25 3 5 7 11 13 17 19 23
For this example, no more passes are needed because they would eliminate numbers already eliminated on previous iterations of the algorithm. The remaining numbers on the list are the consecutive primes less than or equal to 25. What is the largest number p whose multiples can still remain on the list to make further iterations of the algorithm necessary? Before we answer this question, let us ﬁrst note that if p is a number whose multiples are being eliminated on the current pass, then the ﬁrst multiple we should consider is p . p because all its smaller multiples 2p, . . . , (p − 1)p have been eliminated on earlier passes through the list. This observation helps to avoid eliminating the same number more than once. Obviously, p . p shouldnot greater than n, and therefore p cannot exceed √ be √ n rounded down (denoted n using the socalled ﬂoor function). We assume √ n ; in the following pseudocode that there is a function available for computing alternatively, we could check the inequality p . p ≤ n as the loop continuation condition there. ALGORITHM Sieve(n) //Implements the sieve of Eratosthenes //Input: A positive integer n > 1 //Output: Array L of all prime numbers less than or equal to n
1.1
What Is an Algorithm?
7
for p ← 2 to n doA[p] ← p √ for p ← 2 to n do //see note before pseudocode if A[p] = 0 //p hasn’t been eliminated on previous passes j ← p∗p while j ≤ n do A[j ] ← 0 //mark element as eliminated j ←j +p //copy the remaining elements of A to array L of the primes i←0 for p ← 2 to n do if A[p] = 0 L[i] ← A[p] i←i+1 return L So now we can incorporate the sieve of Eratosthenes into the middleschool procedure to get a legitimate algorithm for computing the greatest common divisor of two positive integers. Note that special care needs to be exercised if one or both input numbers are equal to 1: because mathematicians do not consider 1 to be a prime number, strictly speaking, the method does not work for such inputs. Before we leave this section, one more comment is in order. The examples considered in this section notwithstanding, the majority of algorithms in use today—even those that are implemented as computer programs—do not deal with mathematical problems. Look around for algorithms helping us through our daily routines, both professional and personal. May this ubiquity of algorithms in today’s world strengthen your resolve to learn more about these fascinating engines of the information age.
Exercises 1.1 1. Do some research on alKhorezmi (also alKhwarizmi), the man from whose name the word “algorithm” is derived. In particular, you should learn what the origins of the words “algorithm” and “algebra” have in common. 2. Given that the ofﬁcial purpose of the U.S. patent system is the promotion of the “useful arts,” do you think algorithms are patentable in this country? Should they be? 3. a. Write down driving directions for going from your school to your home with the precision required from an algorithm’s description. b. Write down a recipe for cooking your favorite dish with the precision required by an algorithm. √ 4. Design an algorithm for computing n for any positive integer n. Besides assignment and comparison, your algorithm may only use the four basic arithmetical operations.
8
Introduction
5. Design an algorithm to ﬁnd all the common elements in two sorted lists of numbers. For example, for the lists 2, 5, 5, 5 and 2, 2, 3, 5, 5, 7, the output should be 2, 5, 5. What is the maximum number of comparisons your algorithm makes if the lengths of the two given lists are m and n, respectively? 6. a. Find gcd(31415, 14142) by applying Euclid’s algorithm. b. Estimate how many times faster it will be to ﬁnd gcd(31415, 14142) by Euclid’s algorithm compared with the algorithm based on checking consecutive integers from min{m, n} down to gcd(m, n). 7. Prove the equality gcd(m, n) = gcd(n, m mod n) for every pair of positive integers m and n. 8. What does Euclid’s algorithm do for a pair of integers in which the ﬁrst is smaller than the second? What is the maximum number of times this can happen during the algorithm’s execution on such an input? 9. a. What is the minimum number of divisions made by Euclid’s algorithm among all inputs 1 ≤ m, n ≤ 10? b. What is the maximum number of divisions made by Euclid’s algorithm among all inputs 1 ≤ m, n ≤ 10? 10. a. Euclid’s algorithm, as presented in Euclid’s treatise, uses subtractions rather than integer divisions. Write pseudocode for this version of Euclid’s algorithm. b. Euclid’s game (see [Bog]) starts with two unequal positive integers on the board. Two players move in turn. On each move, a player has to write on the board a positive number equal to the difference of two numbers already on the board; this number must be new, i.e., different from all the numbers already on the board. The player who cannot move loses the game. Should you choose to move ﬁrst or second in this game? 11. The extended Euclid’s algorithm determines not only the greatest common divisor d of two positive integers m and n but also integers (not necessarily positive) x and y, such that mx + ny = d. a. Look up a description of the extended Euclid’s algorithm (see, e.g., [KnuI, p. 13]) and implement it in the language of your choice. b. Modify your program to ﬁnd integer solutions to the Diophantine equation ax + by = c with any set of integer coefﬁcients a, b, and c. 12. Locker doors There are n lockers in a hallway, numbered sequentially from 1 to n. Initially, all the locker doors are closed. You make n passes by the lockers, each time starting with locker #1. On the ith pass, i = 1, 2, . . . , n, you toggle the door of every ith locker: if the door is closed, you open it; if it is open, you close it. After the last pass, which locker doors are open and which are closed? How many of them are open?
1.2 Fundamentals of Algorithmic Problem Solving
9
1.2 Fundamentals of Algorithmic Problem Solving Let us start by reiterating an important point made in the introduction to this chapter: We can consider algorithms to be procedural solutions to problems. These solutions are not answers but speciﬁc instructions for getting answers. It is this emphasis on precisely deﬁned constructive procedures that makes computer science distinct from other disciplines. In particular, this distinguishes it from theoretical mathematics, whose practitioners are typically satisﬁed with just proving the existence of a solution to a problem and, possibly, investigating the solution’s properties. We now list and brieﬂy discuss a sequence of steps one typically goes through in designing and analyzing an algorithm (Figure 1.2).
Understanding the Problem From a practical perspective, the ﬁrst thing you need to do before designing an algorithm is to understand completely the problem given. Read the problem’s description carefully and ask questions if you have any doubts about the problem, do a few small examples by hand, think about special cases, and ask questions again if needed. There are a few types of problems that arise in computing applications quite often. We review them in the next section. If the problem in question is one of them, you might be able to use a known algorithm for solving it. Of course, it helps to understand how such an algorithm works and to know its strengths and weaknesses, especially if you have to choose among several available algorithms. But often you will not ﬁnd a readily available algorithm and will have to design your own. The sequence of steps outlined in this section should help you in this exciting but not always easy task. An input to an algorithm speciﬁes an instance of the problem the algorithm solves. It is very important to specify exactly the set of instances the algorithm needs to handle. (As an example, recall the variations in the set of instances for the three greatest common divisor algorithms discussed in the previous section.) If you fail to do this, your algorithm may work correctly for a majority of inputs but crash on some “boundary” value. Remember that a correct algorithm is not one that works most of the time, but one that works correctly for all legitimate inputs. Do not skimp on this ﬁrst step of the algorithmic problemsolving process; otherwise, you will run the risk of unnecessary rework.
Ascertaining the Capabilities of the Computational Device Once you completely understand a problem, you need to ascertain the capabilities of the computational device the algorithm is intended for. The vast majority of
10
Introduction
Understand the problem
Decide on: computational means, exact vs. approximate solving, algorithm design technique
Design an algorithm
Prove correctness
Analyze the algorithm
Code the algorithm
FIGURE 1.2 Algorithm design and analysis process.
algorithms in use today are still destined to be programmed for a computer closely resembling the von Neumann machine—a computer architecture outlined by the prominent HungarianAmerican mathematician John von Neumann (1903– 1957), in collaboration with A. Burks and H. Goldstine, in 1946. The essence of this architecture is captured by the socalled randomaccess machine (RAM). Its central assumption is that instructions are executed one after another, one operation at a time. Accordingly, algorithms designed to be executed on such machines are called sequential algorithms. The central assumption of the RAM model does not hold for some newer computers that can execute operations concurrently, i.e., in parallel. Algorithms that take advantage of this capability are called parallel algorithms. Still, studying the classic techniques for design and analysis of algorithms under the RAM model remains the cornerstone of algorithmics for the foreseeable future.
1.2 Fundamentals of Algorithmic Problem Solving
11
Should you worry about the speed and amount of memory of a computer at your disposal? If you are designing an algorithm as a scientiﬁc exercise, the answer is a qualiﬁed no. As you will see in Section 2.1, most computer scientists prefer to study algorithms in terms independent of speciﬁcation parameters for a particular computer. If you are designing an algorithm as a practical tool, the answer may depend on a problem you need to solve. Even the “slow” computers of today are almost unimaginably fast. Consequently, in many situations you need not worry about a computer being too slow for the task. There are important problems, however, that are very complex by their nature, or have to process huge volumes of data, or deal with applications where the time is critical. In such situations, it is imperative to be aware of the speed and memory available on a particular computer system.
Choosing between Exact and Approximate Problem Solving The next principal decision is to choose between solving the problem exactly or solving it approximately. In the former case, an algorithm is called an exact algorithm; in the latter case, an algorithm is called an approximation algorithm. Why would one opt for an approximation algorithm? First, there are important problems that simply cannot be solved exactly for most of their instances; examples include extracting square roots, solving nonlinear equations, and evaluating definite integrals. Second, available algorithms for solving a problem exactly can be unacceptably slow because of the problem’s intrinsic complexity. This happens, in particular, for many problems involving a very large number of choices; you will see examples of such difﬁcult problems in Chapters 3, 11, and 12. Third, an approximation algorithm can be a part of a more sophisticated algorithm that solves a problem exactly.
Algorithm Design Techniques Now, with all the components of the algorithmic problem solving in place, how do you design an algorithm to solve a given problem? This is the main question this book seeks to answer by teaching you several general design techniques. What is an algorithm design technique? An algorithm design technique (or “strategy” or “paradigm”) is a general approach to solving problems algorithmically that is applicable to a variety of problems from different areas of computing. Check this book’s table of contents and you will see that a majority of its chapters are devoted to individual design techniques. They distill a few key ideas that have proven to be useful in designing algorithms. Learning these techniques is of utmost importance for the following reasons. First, they provide guidance for designing algorithms for new problems, i.e., problems for which there is no known satisfactory algorithm. Therefore—to use the language of a famous proverb—learning such techniques is akin to learning
12
Introduction
to ﬁsh as opposed to being given a ﬁsh caught by somebody else. It is not true, of course, that each of these general techniques will be necessarily applicable to every problem you may encounter. But taken together, they do constitute a powerful collection of tools that you will ﬁnd quite handy in your studies and work. Second, algorithms are the cornerstone of computer science. Every science is interested in classifying its principal subject, and computer science is no exception. Algorithm design techniques make it possible to classify algorithms according to an underlying design idea; therefore, they can serve as a natural way to both categorize and study algorithms.
Designing an Algorithm and Data Structures While the algorithm design techniques do provide a powerful set of general approaches to algorithmic problem solving, designing an algorithm for a particular problem may still be a challenging task. Some design techniques can be simply inapplicable to the problem in question. Sometimes, several techniques need to be combined, and there are algorithms that are hard to pinpoint as applications of the known design techniques. Even when a particular design technique is applicable, getting an algorithm often requires a nontrivial ingenuity on the part of the algorithm designer. With practice, both tasks—choosing among the general techniques and applying them—get easier, but they are rarely easy. Of course, one should pay close attention to choosing data structures appropriate for the operations performed by the algorithm. For example, the sieve of Eratosthenes introduced in Section 1.1 would run longer if we used a linked list instead of an array in its implementation (why?). Also note that some of the algorithm design techniques discussed in Chapters 6 and 7 depend intimately on structuring or restructuring data specifying a problem’s instance. Many years ago, an inﬂuential textbook proclaimed the fundamental importance of both algorithms and data structures for computer programming by its very title: Algorithms + Data Structures = Programs [Wir76]. In the new world of objectoriented programming, data structures remain crucially important for both design and analysis of algorithms. We review basic data structures in Section 1.4.
Methods of Specifying an Algorithm Once you have designed an algorithm, you need to specify it in some fashion. In Section 1.1, to give you an example, Euclid’s algorithm is described in words (in a free and also a stepbystep form) and in pseudocode. These are the two options that are most widely used nowadays for specifying algorithms. Using a natural language has an obvious appeal; however, the inherent ambiguity of any natural language makes a succinct and clear description of algorithms surprisingly difﬁcult. Nevertheless, being able to do this is an important skill that you should strive to develop in the process of learning algorithms. Pseudocode is a mixture of a natural language and programming languagelike constructs. Pseudocode is usually more precise than natural language, and its
1.2 Fundamentals of Algorithmic Problem Solving
13
usage often yields more succinct algorithm descriptions. Surprisingly, computer scientists have never agreed on a single form of pseudocode, leaving textbook authors with a need to design their own “dialects.” Fortunately, these dialects are so close to each other that anyone familiar with a modern programming language should be able to understand them all. This book’s dialect was selected to cause minimal difﬁculty for a reader. For the sake of simplicity, we omit declarations of variables and use indentation to show the scope of such statements as for, if, and while. As you saw in the previous section, we use an arrow “←” for the assignment operation and two slashes “//” for comments. In the earlier days of computing, the dominant vehicle for specifying algorithms was a ﬂowchart, a method of expressing an algorithm by a collection of connected geometric shapes containing descriptions of the algorithm’s steps. This representation technique has proved to be inconvenient for all but very simple algorithms; nowadays, it can be found only in old algorithm books. The state of the art of computing has not yet reached a point where an algorithm’s description—be it in a natural language or pseudocode—can be fed into an electronic computer directly. Instead, it needs to be converted into a computer program written in a particular computer language. We can look at such a program as yet another way of specifying the algorithm, although it is preferable to consider it as the algorithm’s implementation.
Proving an Algorithm’s Correctness Once an algorithm has been speciﬁed, you have to prove its correctness. That is, you have to prove that the algorithm yields a required result for every legitimate input in a ﬁnite amount of time. For example, the correctness of Euclid’s algorithm for computing the greatest common divisor stems from the correctness of the equality gcd(m, n) = gcd(n, m mod n) (which, in turn, needs a proof; see Problem 7 in Exercises 1.1), the simple observation that the second integer gets smaller on every iteration of the algorithm, and the fact that the algorithm stops when the second integer becomes 0. For some algorithms, a proof of correctness is quite easy; for others, it can be quite complex. A common technique for proving correctness is to use mathematical induction because an algorithm’s iterations provide a natural sequence of steps needed for such proofs. It might be worth mentioning that although tracing the algorithm’s performance for a few speciﬁc inputs can be a very worthwhile activity, it cannot prove the algorithm’s correctness conclusively. But in order to show that an algorithm is incorrect, you need just one instance of its input for which the algorithm fails. The notion of correctness for approximation algorithms is less straightforward than it is for exact algorithms. For an approximation algorithm, we usually would like to be able to show that the error produced by the algorithm does not exceed a predeﬁned limit. You can ﬁnd examples of such investigations in Chapter 12.
14
Introduction
Analyzing an Algorithm We usually want our algorithms to possess several qualities. After correctness, by far the most important is efﬁciency. In fact, there are two kinds of algorithm efﬁciency: time efﬁciency, indicating how fast the algorithm runs, and space efﬁciency, indicating how much extra memory it uses. A general framework and speciﬁc techniques for analyzing an algorithm’s efﬁciency appear in Chapter 2. Another desirable characteristic of an algorithm is simplicity. Unlike efﬁciency, which can be precisely deﬁned and investigated with mathematical rigor, simplicity, like beauty, is to a considerable degree in the eye of the beholder. For example, most people would agree that Euclid’s algorithm is simpler than the middleschool procedure for computing gcd(m, n), but it is not clear whether Euclid’s algorithm is simpler than the consecutive integer checking algorithm. Still, simplicity is an important algorithm characteristic to strive for. Why? Because simpler algorithms are easier to understand and easier to program; consequently, the resulting programs usually contain fewer bugs. There is also the undeniable aesthetic appeal of simplicity. Sometimes simpler algorithms are also more efﬁcient than more complicated alternatives. Unfortunately, it is not always true, in which case a judicious compromise needs to be made. Yet another desirable characteristic of an algorithm is generality. There are, in fact, two issues here: generality of the problem the algorithm solves and the set of inputs it accepts. On the ﬁrst issue, note that it is sometimes easier to design an algorithm for a problem posed in more general terms. Consider, for example, the problem of determining whether two integers are relatively prime, i.e., whether their only common divisor is equal to 1. It is easier to design an algorithm for a more general problem of computing the greatest common divisor of two integers and, to solve the former problem, check whether the gcd is 1 or not. There are situations, however, where designing a more general algorithm is unnecessary or difﬁcult or even impossible. For example, it is unnecessary to sort a list of n numbers to ﬁnd its median, which is its n/2th smallest element. To give another example, the standard formula for roots of a quadratic equation cannot be generalized to handle polynomials of arbitrary degrees. As to the set of inputs, your main concern should be designing an algorithm that can handle a set of inputs that is natural for the problem at hand. For example, excluding integers equal to 1 as possible inputs for a greatest common divisor algorithm would be quite unnatural. On the other hand, although the standard formula for the roots of a quadratic equation holds for complex coefﬁcients, we would normally not implement it on this level of generality unless this capability is explicitly required. If you are not satisﬁed with the algorithm’s efﬁciency, simplicity, or generality, you must return to the drawing board and redesign the algorithm. In fact, even if your evaluation is positive, it is still worth searching for other algorithmic solutions. Recall the three different algorithms in the previous section for computing the greatest common divisor: generally, you should not expect to get the best algorithm on the ﬁrst try. At the very least, you should try to ﬁnetune the algorithm you
1.2 Fundamentals of Algorithmic Problem Solving
15
already have. For example, we made several improvements in our implementation of the sieve of Eratosthenes compared with its initial outline in Section 1.1. (Can you identify them?) You will do well if you keep in mind the following observation ´ the French writer, pilot, and aircraft designer: “A of Antoine de SaintExupery, designer knows he has arrived at perfection not when there is no longer anything to add, but when there is no longer anything to take away.”1
Coding an Algorithm Most algorithms are destined to be ultimately implemented as computer programs. Programming an algorithm presents both a peril and an opportunity. The peril lies in the possibility of making the transition from an algorithm to a program either incorrectly or very inefﬁciently. Some inﬂuential computer scientists strongly believe that unless the correctness of a computer program is proven with full mathematical rigor, the program cannot be considered correct. They have developed special techniques for doing such proofs (see [Gri81]), but the power of these techniques of formal veriﬁcation is limited so far to very small programs. As a practical matter, the validity of programs is still established by testing. Testing of computer programs is an art rather than a science, but that does not mean that there is nothing in it to learn. Look up books devoted to testing and debugging; even more important, test and debug your program thoroughly whenever you implement an algorithm. Also note that throughout the book, we assume that inputs to algorithms belong to the speciﬁed sets and hence require no veriﬁcation. When implementing algorithms as programs to be used in actual applications, you should provide such veriﬁcations. Of course, implementing an algorithm correctly is necessary but not sufﬁcient: you would not like to diminish your algorithm’s power by an inefﬁcient implementation. Modern compilers do provide a certain safety net in this regard, especially when they are used in their code optimization mode. Still, you need to be aware of such standard tricks as computing a loop’s invariant (an expression that does not change its value) outside the loop, collecting common subexpressions, replacing expensive operations by cheap ones, and so on. (See [Ker99] and [Ben00] for a good discussion of code tuning and other issues related to algorithm programming.) Typically, such improvements can speed up a program only by a constant factor, whereas a better algorithm can make a difference in running time by orders of magnitude. But once an algorithm is selected, a 10–50% speedup may be worth an effort.
1.
I found this call for design simplicity in an essay collection by Jon Bentley [Ben00]; the essays deal with a variety of issues in algorithm design and implementation and are justiﬁably titled Programming ´ Pearls. I wholeheartedly recommend the writings of both Jon Bentley and Antoine de SaintExupery.
16
Introduction
A working program provides an additional opportunity in allowing an empirical analysis of the underlying algorithm. Such an analysis is based on timing the program on several inputs and then analyzing the results obtained. We discuss the advantages and disadvantages of this approach to analyzing algorithms in Section 2.6. In conclusion, let us emphasize again the main lesson of the process depicted in Figure 1.2: As a rule, a good algorithm is a result of repeated effort and rework. Even if you have been fortunate enough to get an algorithmic idea that seems perfect, you should still try to see whether it can be improved. Actually, this is good news since it makes the ultimate result so much more enjoyable. (Yes, I did think of naming this book The Joy of Algorithms.) On the other hand, how does one know when to stop? In the real world, more often than not a project’s schedule or the impatience of your boss will stop you. And so it should be: perfection is expensive and in fact not always called for. Designing an algorithm is an engineeringlike activity that calls for compromises among competing goals under the constraints of available resources, with the designer’s time being one of the resources. In the academic world, the question leads to an interesting but usually difﬁcult investigation of an algorithm’s optimality. Actually, this question is not about the efﬁciency of an algorithm but about the complexity of the problem it solves: What is the minimum amount of effort any algorithm will need to exert to solve the problem? For some problems, the answer to this question is known. For example, any algorithm that sorts an array by comparing values of its elements needs about n log2 n comparisons for some arrays of size n (see Section 11.2). But for many seemingly easy problems such as integer multiplication, computer scientists do not yet have a ﬁnal answer. Another important issue of algorithmic problem solving is the question of whether or not every problem can be solved by an algorithm. We are not talking here about problems that do not have a solution, such as ﬁnding real roots of a quadratic equation with a negative discriminant. For such cases, an output indicating that the problem does not have a solution is all we can and should expect from an algorithm. Nor are we talking about ambiguously stated problems. Even some unambiguous problems that must have a simple yes or no answer are “undecidable,” i.e., unsolvable by any algorithm. An important example of such a problem appears in Section 11.3. Fortunately, a vast majority of problems in practical computing can be solved by an algorithm. Before leaving this section, let us be sure that you do not have the misconception—possibly caused by the somewhat mechanical nature of the diagram of Figure 1.2—that designing an algorithm is a dull activity. There is nothing further from the truth: inventing (or discovering?) algorithms is a very creative and rewarding process. This book is designed to convince you that this is the case.
1.2 Fundamentals of Algorithmic Problem Solving
17
Exercises 1.2 1. Old World puzzle A peasant ﬁnds himself on a riverbank with a wolf, a goat, and a head of cabbage. He needs to transport all three to the other side of the river in his boat. However, the boat has room for only the peasant himself and one other item (either the wolf, the goat, or the cabbage). In his absence, the wolf would eat the goat, and the goat would eat the cabbage. Solve this problem for the peasant or prove it has no solution. (Note: The peasant is a vegetarian but does not like cabbage and hence can eat neither the goat nor the cabbage to help him solve the problem. And it goes without saying that the wolf is a protected species.) 2. New World puzzle There are four people who want to cross a rickety bridge; they all begin on the same side. You have 17 minutes to get them all across to the other side. It is night, and they have one ﬂashlight. A maximum of two people can cross the bridge at one time. Any party that crosses, either one or two people, must have the ﬂashlight with them. The ﬂashlight must be walked back and forth; it cannot be thrown, for example. Person 1 takes 1 minute to cross the bridge, person 2 takes 2 minutes, person 3 takes 5 minutes, and person 4 takes 10 minutes. A pair must walk together at the rate of the slower person’s pace. (Note: According to a rumor on the Internet, interviewers at a wellknown software company located near Seattle have given this problem to interviewees.) 3. Which of the following formulas can be considered an algorithm for computing the area of a triangle whose side lengths are given positive numbers a, b, and c? a. S = p(p − a)(p − b)(p − c), where p = (a + b + c)/2 b. S = 21 bc sin A, where A is the angle between sides b and c c. S = 21 aha , where ha is the height to base a 4. Write pseudocode for an algorithm for ﬁnding real roots of equation ax 2 + bx + c = 0 for arbitrary real coefﬁcients a, b, and c. (You may assume the availability of the square root function sqrt (x).) 5. Describe the standard algorithm for ﬁnding the binary representation of a positive decimal integer a. in English. b. in pseudocode. 6. Describe the algorithm used by your favorite ATM machine in dispensing cash. (You may give your description in either English or pseudocode, whichever you ﬁnd more convenient.) 7. a. Can the problem of computing the number π be solved exactly? b. How many instances does this problem have? c. Look up an algorithm for this problem on the Internet.
18
Introduction
8. Give an example of a problem other than computing the greatest common divisor for which you know more than one algorithm. Which of them is simpler? Which is more efﬁcient? 9. Consider the following algorithm for ﬁnding the distance between the two closest elements in an array of numbers. ALGORITHM MinDistance(A[0..n − 1]) //Input: Array A[0..n − 1] of numbers //Output: Minimum distance between two of its elements dmin ← ∞ for i ← 0 to n − 1 do for j ← 0 to n − 1 do if i = j and A[i] − A[j ] < dmin dmin ← A[i] − A[j ] return dmin Make as many improvements as you can in this algorithmic solution to the problem. If you need to, you may change the algorithm altogether; if not, improve the implementation given. 10. One of the most inﬂuential books on problem solving, titled How To Solve It [Pol57], was written by the HungarianAmerican mathematician George ´ ´ Polya (1887–1985). Polya summarized his ideas in a fourpoint summary. Find this summary on the Internet or, better yet, in his book, and compare it with the plan outlined in Section 1.2. What do they have in common? How are they different?
1.3 Important Problem Types In the limitless sea of problems one encounters in computing, there are a few areas that have attracted particular attention from researchers. By and large, their interest has been driven either by the problem’s practical importance or by some speciﬁc characteristics making the problem an interesting research subject; fortunately, these two motivating forces reinforce each other in most cases. In this section, we are going to introduce the most important problem types: Sorting Searching String processing Graph problems Combinatorial problems Geometric problems Numerical problems
1.3
Important Problem Types
19
These problems are used in subsequent chapters of the book to illustrate different algorithm design techniques and methods of algorithm analysis.
Sorting The sorting problem is to rearrange the items of a given list in nondecreasing order. Of course, for this problem to be meaningful, the nature of the list items must allow such an ordering. (Mathematicians would say that there must exist a relation of total ordering.) As a practical matter, we usually need to sort lists of numbers, characters from an alphabet, character strings, and, most important, records similar to those maintained by schools about their students, libraries about their holdings, and companies about their employees. In the case of records, we need to choose a piece of information to guide sorting. For example, we can choose to sort student records in alphabetical order of names or by student number or by student gradepoint average. Such a specially chosen piece of information is called a key. Computer scientists often talk about sorting a list of keys even when the list’s items are not records but, say, just integers. Why would we want a sorted list? To begin with, a sorted list can be a required output of a task such as ranking Internet search results or ranking students by their GPA scores. Further, sorting makes many questions about the list easier to answer. The most important of them is searching: it is why dictionaries, telephone books, class lists, and so on are sorted. You will see other examples of the usefulness of list presorting in Section 6.1. In a similar vein, sorting is used as an auxiliary step in several important algorithms in other areas, e.g., geometric algorithms and data compression. The greedy approach—an important algorithm design technique discussed later in the book—requires a sorted input. By now, computer scientists have discovered dozens of different sorting algorithms. In fact, inventing a new sorting algorithm has been likened to designing the proverbial mousetrap. And I am happy to report that the hunt for a better sorting mousetrap continues. This perseverance is admirable in view of the following facts. On the one hand, there are a few good sorting algorithms that sort an arbitrary array of size n using about n log2 n comparisons. On the other hand, no algorithm that sorts by key comparisons (as opposed to, say, comparing small pieces of keys) can do substantially better than that. There is a reason for this embarrassment of algorithmic riches in the land of sorting. Although some algorithms are indeed better than others, there is no algorithm that would be the best solution in all situations. Some of the algorithms are simple but relatively slow, while others are faster but more complex; some work better on randomly ordered inputs, while others do better on almostsorted lists; some are suitable only for lists residing in the fast memory, while others can be adapted for sorting large ﬁles stored on a disk; and so on. Two properties of sorting algorithms deserve special mention. A sorting algorithm is called stable if it preserves the relative order of any two equal elements in its input. In other words, if an input list contains two equal elements in positions i and j where i < j, then in the sorted list they have to be in positions i and j ,
20
Introduction
respectively, such that i < j . This property can be desirable if, for example, we have a list of students sorted alphabetically and we want to sort it according to student GPA: a stable algorithm will yield a list in which students with the same GPA will still be sorted alphabetically. Generally speaking, algorithms that can exchange keys located far apart are not stable, but they usually work faster; you will see how this general comment applies to important sorting algorithms later in the book. The second notable feature of a sorting algorithm is the amount of extra memory the algorithm requires. An algorithm is said to be inplace if it does not require extra memory, except, possibly, for a few memory units. There are important sorting algorithms that are inplace and those that are not.
Searching The searching problem deals with ﬁnding a given value, called a search key, in a given set (or a multiset, which permits several elements to have the same value). There are plenty of searching algorithms to choose from. They range from the straightforward sequential search to a spectacularly efﬁcient but limited binary search and algorithms based on representing the underlying set in a different form more conducive to searching. The latter algorithms are of particular importance for realworld applications because they are indispensable for storing and retrieving information from large databases. For searching, too, there is no single algorithm that ﬁts all situations best. Some algorithms work faster than others but require more memory; some are very fast but applicable only to sorted arrays; and so on. Unlike with sorting algorithms, there is no stability problem, but different issues arise. Speciﬁcally, in applications where the underlying data may change frequently relative to the number of searches, searching has to be considered in conjunction with two other operations: an addition to and deletion from the data set of an item. In such situations, data structures and algorithms should be chosen to strike a balance among the requirements of each operation. Also, organizing very large data sets for efﬁcient searching poses special challenges with important implications for realworld applications.
String Processing In recent decades, the rapid proliferation of applications dealing with nonnumerical data has intensiﬁed the interest of researchers and computing practitioners in stringhandling algorithms. A string is a sequence of characters from an alphabet. Strings of particular interest are text strings, which comprise letters, numbers, and special characters; bit strings, which comprise zeros and ones; and gene sequences, which can be modeled by strings of characters from the fourcharacter alphabet {A, C, G, T}. It should be pointed out, however, that stringprocessing algorithms have been important for computer science for a long time in conjunction with computer languages and compiling issues.
1.3
Important Problem Types
21
One particular problem—that of searching for a given word in a text—has attracted special attention from researchers. They call it string matching. Several algorithms that exploit the special nature of this type of searching have been invented. We introduce one very simple algorithm in Chapter 3 and discuss two algorithms based on a remarkable idea by R. Boyer and J. Moore in Chapter 7.
Graph Problems One of the oldest and most interesting areas in algorithmics is graph algorithms. Informally, a graph can be thought of as a collection of points called vertices, some of which are connected by line segments called edges. (A more formal deﬁnition is given in the next section.) Graphs are an interesting subject to study, for both theoretical and practical reasons. Graphs can be used for modeling a wide variety of applications, including transportation, communication, social and economic networks, project scheduling, and games. Studying different technical and social aspects of the Internet in particular is one of the active areas of current research involving computer scientists, economists, and social scientists (see, e.g., [Eas10]). Basic graph algorithms include graphtraversal algorithms (how can one reach all the points in a network?), shortestpath algorithms (what is the best route between two cities?), and topological sorting for graphs with directed edges (is a set of courses with their prerequisites consistent or selfcontradictory?). Fortunately, these algorithms can be considered illustrations of general design techniques; accordingly, you will ﬁnd them in corresponding chapters of the book. Some graph problems are computationally very hard; the most wellknown examples are the traveling salesman problem and the graphcoloring problem. The traveling salesman problem (TSP) is the problem of ﬁnding the shortest tour through n cities that visits every city exactly once. In addition to obvious applications involving route planning, it arises in such modern applications as circuit board and VLSI chip fabrication, Xray crystallography, and genetic engineering. The graphcoloring problem seeks to assign the smallest number of colors to the vertices of a graph so that no two adjacent vertices are the same color. This problem arises in several applications, such as event scheduling: if the events are represented by vertices that are connected by an edge if and only if the corresponding events cannot be scheduled at the same time, a solution to the graphcoloring problem yields an optimal schedule.
Combinatorial Problems From a more abstract perspective, the traveling salesman problem and the graphcoloring problem are examples of combinatorial problems. These are problems that ask, explicitly or implicitly, to ﬁnd a combinatorial object—such as a permutation, a combination, or a subset—that satisﬁes certain constraints. A desired combinatorial object may also be required to have some additional property such as a maximum value or a minimum cost.
22
Introduction
Generally speaking, combinatorial problems are the most difﬁcult problems in computing, from both a theoretical and practical standpoint. Their difﬁculty stems from the following facts. First, the number of combinatorial objects typically grows extremely fast with a problem’s size, reaching unimaginable magnitudes even for moderatesized instances. Second, there are no known algorithms for solving most such problems exactly in an acceptable amount of time. Moreover, most computer scientists believe that such algorithms do not exist. This conjecture has been neither proved nor disproved, and it remains the most important unresolved issue in theoretical computer science. We discuss this topic in more detail in Section 11.3. Some combinatorial problems can be solved by efﬁcient algorithms, but they should be considered fortunate exceptions to the rule. The shortestpath problem mentioned earlier is among such exceptions.
Geometric Problems Geometric algorithms deal with geometric objects such as points, lines, and polygons. The ancient Greeks were very much interested in developing procedures (they did not call them algorithms, of course) for solving a variety of geometric problems, including problems of constructing simple geometric shapes—triangles, circles, and so on—with an unmarked ruler and a compass. Then, for about 2000 years, intense interest in geometric algorithms disappeared, to be resurrected in the age of computers—no more rulers and compasses, just bits, bytes, and good old human ingenuity. Of course, today people are interested in geometric algorithms with quite different applications in mind, such as computer graphics, robotics, and tomography. We will discuss algorithms for only two classic problems of computational geometry: the closestpair problem and the convexhull problem. The closestpair problem is selfexplanatory: given n points in the plane, ﬁnd the closest pair among them. The convexhull problem asks to ﬁnd the smallest convex polygon that would include all the points of a given set. If you are interested in other geometric algorithms, you will ﬁnd a wealth of material in such specialized monographs as [deB10], [ORo98], and [Pre85].
Numerical Problems Numerical problems, another large special area of applications, are problems that involve mathematical objects of continuous nature: solving equations and systems of equations, computing deﬁnite integrals, evaluating functions, and so on. The majority of such mathematical problems can be solved only approximately. Another principal difﬁculty stems from the fact that such problems typically require manipulating real numbers, which can be represented in a computer only approximately. Moreover, a large number of arithmetic operations performed on approximately represented numbers can lead to an accumulation of the roundoff
1.3
Important Problem Types
23
error to a point where it can drastically distort an output produced by a seemingly sound algorithm. Many sophisticated algorithms have been developed over the years in this area, and they continue to play a critical role in many scientiﬁc and engineering applications. But in the last 30 years or so, the computing industry has shifted its focus to business applications. These new applications require primarily algorithms for information storage, retrieval, transportation through networks, and presentation to users. As a result of this revolutionary change, numerical analysis has lost its formerly dominating position in both industry and computer science programs. Still, it is important for any computerliterate person to have at least a rudimentary idea about numerical algorithms. We discuss several classical numerical algorithms in Sections 6.2, 11.4, and 12.4.
Exercises 1.3 1. Consider the algorithm for the sorting problem that sorts an array by counting, for each of its elements, the number of smaller elements and then uses this information to put the element in its appropriate position in the sorted array: ALGORITHM ComparisonCountingSort(A[0..n − 1]) //Sorts an array by comparison counting //Input: Array A[0..n − 1] of orderable values //Output: Array S[0..n − 1] of A’s elements sorted // in nondecreasing order for i ← 0 to n − 1 do Count[i] ← 0 for i ← 0 to n − 2 do for j ← i + 1 to n − 1 do if A[i] < A[j ] Count[j ] ← Count[j ] + 1 else Count[i] ← Count[i] + 1 for i ← 0 to n − 1 do S[Count[i]] ← A[i] return S a. Apply this algorithm to sorting the list 60, 35, 81, 98, 14, 47. b. Is this algorithm stable? c. Is it inplace? 2. Name the algorithms for the searching problem that you already know. Give a good succinct description of each algorithm in English. If you know no such algorithms, use this opportunity to design one. 3. Design a simple algorithm for the stringmatching problem.
24
Introduction
¨ 4. Konigsberg ¨ bridges The Konigsberg bridge puzzle is universally accepted as the problem that gave birth to graph theory. It was solved by the great Swissborn mathematician Leonhard Euler (1707–1783). The problem asked whether one could, in a single stroll, cross all seven bridges of the city of ¨ Konigsberg exactly once and return to a starting point. Following is a sketch of the river with its two islands and seven bridges:
a. State the problem as a graph problem. b. Does this problem have a solution? If you believe it does, draw such a stroll; if you believe it does not, explain why and indicate the smallest number of new bridges that would be required to make such a stroll possible. 5. Icosian Game A century after Euler’s discovery (see Problem 4), another famous puzzle—this one invented by the renowned Irish mathematician Sir William Hamilton (1805–1865)—was presented to the world under the name of the Icosian Game. The game’s board was a circular wooden board on which the following graph was carved:
Find a Hamiltonian circuit—a path that visits all the graph’s vertices exactly once before returning to the starting vertex—for this graph. 6. Consider the following problem: Design an algorithm to determine the best route for a subway passenger to take from one designated station to another in a welldeveloped subway system similar to those in such cities as Washington, D.C., and London, UK.
1.4 Fundamental Data Structures
25
a. The problem’s statement is somewhat vague, which is typical of reallife problems. In particular, what reasonable criterion can be used for deﬁning the “best” route? b. How would you model this problem by a graph? 7. a. Rephrase the travelingsalesman problem in combinatorial object terms. b. Rephrase the graphcoloring problem in combinatorial object terms. 8. Consider the following map: b a d c e f
a. Explain how we can use the graphcoloring problem to color the map so that no two neighboring regions are colored the same. b. Use your answer to part (a) to color the map with the smallest number of colors. 9. Design an algorithm for the following problem: Given a set of n points in the Cartesian plane, determine whether all of them lie on the same circumference. 10. Write a program that reads as its inputs the (x, y) coordinates of the endpoints of two line segments P1Q1 and P2 Q2 and determines whether the segments have a common point.
1.4 Fundamental Data Structures Since the vast majority of algorithms of interest operate on data, particular ways of organizing data play a critical role in the design and analysis of algorithms. A data structure can be deﬁned as a particular scheme of organizing related data items. The nature of the data items is dictated by the problem at hand; they can range from elementary data types (e.g., integers or characters) to data structures (e.g., a onedimensional array of onedimensional arrays is often used for implementing matrices). There are a few data structures that have proved to be particularly important for computer algorithms. Since you are undoubtedly familiar with most if not all of them, just a quick review is provided here.
Linear Data Structures The two most important elementary data structures are the array and the linked list. A (onedimensional) array is a sequence of n items of the same data type that
26
Introduction
are stored contiguously in computer memory and made accessible by specifying a value of the array’s index (Figure 1.3). In the majority of cases, the index is an integer either between 0 and n − 1 (as shown in Figure 1.3) or between 1 and n. Some computer languages allow an array index to range between any two integer bounds low and high, and some even permit nonnumerical indices to specify, for example, data items corresponding to the 12 months of the year by the month names. Each and every element of an array can be accessed in the same constant amount of time regardless of where in the array the element in question is located. This feature positively distinguishes arrays from linked lists, discussed below. Arrays are used for implementing a variety of other data structures. Prominent among them is the string, a sequence of characters from an alphabet terminated by a special character indicating the string’s end. Strings composed of zeros and ones are called binary strings or bit strings. Strings are indispensable for processing textual data, deﬁning computer languages and compiling programs written in them, and studying abstract computational models. Operations we usually perform on strings differ from those we typically perform on other arrays (say, arrays of numbers). They include computing the string length, comparing two strings to determine which one precedes the other in lexicographic (i.e., alphabetical) order, and concatenating two strings (forming one string from two given strings by appending the second to the end of the ﬁrst). A linked list is a sequence of zero or more elements called nodes, each containing two kinds of information: some data and one or more links called pointers to other nodes of the linked list. (A special pointer called “null” is used to indicate the absence of a node’s successor.) In a singly linked list, each node except the last one contains a single pointer to the next element (Figure 1.4). To access a particular node of a linked list, one starts with the list’s ﬁrst node and traverses the pointer chain until the particular node is reached. Thus, the time needed to access an element of a singly linked list, unlike that of an array, depends on where in the list the element is located. On the positive side, linked lists do
Item [0]
Item [1]
Item [n –1]
FIGURE 1.3 Array of n elements.
Item 0
Item 1
FIGURE 1.4 Singly linked list of n elements.
Item n –1 null
27
1.4 Fundamental Data Structures
null Item 0
Item 1
Item n –1 null
FIGURE 1.5 Doubly linked list of n elements.
not require any preliminary reservation of the computer memory, and insertions and deletions can be made quite efﬁciently in a linked list by reconnecting a few appropriate pointers. We can exploit ﬂexibility of the linked list structure in a variety of ways. For example, it is often convenient to start a linked list with a special node called the header. This node may contain information about the linked list itself, such as its current length; it may also contain, in addition to a pointer to the ﬁrst element, a pointer to the linked list’s last element. Another extension is the structure called the doubly linked list, in which every node, except the ﬁrst and the last, contains pointers to both its successor and its predecessor (Figure 1.5). The array and linked list are two principal choices in representing a more abstract data structure called a linear list or simply a list. A list is a ﬁnite sequence of data items, i.e., a collection of data items arranged in a certain linear order. The basic operations performed on this data structure are searching for, inserting, and deleting an element. Two special types of lists, stacks and queues, are particularly important. A stack is a list in which insertions and deletions can be done only at the end. This end is called the top because a stack is usually visualized not horizontally but vertically—akin to a stack of plates whose “operations” it mimics very closely. As a result, when elements are added to (pushed onto) a stack and deleted from (popped off) it, the structure operates in a “lastin–ﬁrstout” (LIFO) fashion— exactly like a stack of plates if we can add or remove a plate only from the top. Stacks have a multitude of applications; in particular, they are indispensable for implementing recursive algorithms. A queue, on the other hand, is a list from which elements are deleted from one end of the structure, called the front (this operation is called dequeue), and new elements are added to the other end, called the rear (this operation is called enqueue). Consequently, a queue operates in a “ﬁrstin–ﬁrstout” (FIFO) fashion—akin to a queue of customers served by a single teller in a bank. Queues also have many important applications, including several algorithms for graph problems. Many important applications require selection of an item of the highest priority among a dynamically changing set of candidates. A data structure that seeks to satisfy the needs of such applications is called a priority queue. A priority queue is a collection of data items from a totally ordered universe (most often,
28
Introduction
integer or real numbers). The principal operations on a priority queue are ﬁnding its largest element, deleting its largest element, and adding a new element. Of course, a priority queue must be implemented so that the last two operations yield another priority queue. Straightforward implementations of this data structure can be based on either an array or a sorted array, but neither of these options yields the most efﬁcient solution possible. A better implementation of a priority queue is based on an ingenious data structure called the heap. We discuss heaps and an important sorting algorithm based on them in Section 6.4.
Graphs As we mentioned in the previous section, a graph is informally thought of as a collection of points in the plane called “vertices” or “nodes,” some of them connected by line segments called “edges” or “arcs.” Formally, a graph G = V , E is deﬁned by a pair of two sets: a ﬁnite nonempty set V of items called vertices and a set E of pairs of these items called edges. If these pairs of vertices are unordered, i.e., a pair of vertices (u, v) is the same as the pair (v, u), we say that the vertices u and v are adjacent to each other and that they are connected by the undirected edge (u, v). We call the vertices u and v endpoints of the edge (u, v) and say that u and v are incident to this edge; we also say that the edge (u, v) is incident to its endpoints u and v. A graph G is called undirected if every edge in it is undirected. If a pair of vertices (u, v) is not the same as the pair (v, u), we say that the edge (u, v) is directed from the vertex u, called the edge’s tail, to the vertex v, called the edge’s head. We also say that the edge (u, v) leaves u and enters v. A graph whose every edge is directed is called directed. Directed graphs are also called digraphs. It is normally convenient to label vertices of a graph or a digraph with letters, integer numbers, or, if an application calls for it, character strings (Figure 1.6). The graph depicted in Figure 1.6a has six vertices and seven undirected edges: V = {a, b, c, d, e, f }, E = {(a, c), (a, d), (b, c), (b, f ), (c, e), (d, e), (e, f )}. The digraph depicted in Figure 1.6b has six vertices and eight directed edges: V = {a, b, c, d, e, f }, E = {(a, c), (b, c), (b, f ), (c, e), (d, a), (d, e), (e, c), (e, f )}.
a
c
b
a
c
b
d
e
f
d
e
f
(a) FIGURE 1.6 (a) Undirected graph. (b) Digraph.
(b)
29
1.4 Fundamental Data Structures
Our deﬁnition of a graph does not forbid loops, or edges connecting vertices to themselves. Unless explicitly stated otherwise, we will consider graphs without loops. Since our deﬁnition disallows multiple edges between the same vertices of an undirected graph, we have the following inequality for the number of edges E possible in an undirected graph with V  vertices and no loops: 0 ≤ E ≤ V (V  − 1)/2. (We get the largest number of edges in a graph if there is an edge connecting each of its V  vertices with all V  − 1 other vertices. We have to divide product V (V  − 1) by 2, however, because it includes every edge twice.) A graph with every pair of its vertices connected by an edge is called complete. A standard notation for the complete graph with V  vertices is KV . A graph with relatively few possible edges missing is called dense; a graph with few edges relative to the number of its vertices is called sparse. Whether we are dealing with a dense or sparse graph may inﬂuence how we choose to represent the graph and, consequently, the running time of an algorithm being designed or used.
Graph Representations Graphs for computer algorithms are usually represented in one of two ways: the adjacency matrix and adjacency lists. The adjacency matrix of a graph with n vertices is an n × n boolean matrix with one row and one column for each of the graph’s vertices, in which the element in the ith row and the j th column is equal to 1 if there is an edge from the ith vertex to the j th vertex, and equal to 0 if there is no such edge. For example, the adjacency matrix for the graph of Figure 1.6a is given in Figure 1.7a. Note that the adjacency matrix of an undirected graph is always symmetric, i.e., A[i, j ] = A[j, i] for every 0 ≤ i, j ≤ n − 1 (why?). The adjacency lists of a graph or a digraph is a collection of linked lists, one for each vertex, that contain all the vertices adjacent to the list’s vertex (i.e., all the vertices connected to it by an edge). Usually, such lists start with a header identifying a vertex for which the list is compiled. For example, Figure 1.7b represents the graph in Figure 1.6a via its adjacency lists. To put it another way,
a b c d e f
a 0 0 1 1 0 0
b 0 0 1 0 0 1
c d 1 1 1 0 0 0 0 0 1 1 0 0
(a)
e 0 0 1 1 0 1
f 0 1 0 0 1 0
a b c d e f
→ → → → → →
c c a a c b
→ → → → → →
d f b e d e
→
e
→
f
(b)
FIGURE 1.7 (a) Adjacency matrix and (b) adjacency lists of the graph in Figure 1.6a.
30
Introduction
adjacency lists indicate columns of the adjacency matrix that, for a given vertex, contain 1’s. If a graph is sparse, the adjacency list representation may use less space than the corresponding adjacency matrix despite the extra storage consumed by pointers of the linked lists; the situation is exactly opposite for dense graphs. In general, which of the two representations is more convenient depends on the nature of the problem, on the algorithm used for solving it, and, possibly, on the type of input graph (sparse or dense).
Weighted Graphs A weighted graph (or weighted digraph) is a graph (or digraph) with numbers assigned to its edges. These numbers are called weights or costs. An interest in such graphs is motivated by numerous realworld applications, such as ﬁnding the shortest path between two points in a transportation or communication network or the traveling salesman problem mentioned earlier. Both principal representations of a graph can be easily adopted to accommodate weighted graphs. If a weighted graph is represented by its adjacency matrix, then its element A[i, j ] will simply contain the weight of the edge from the ith to the j th vertex if there is such an edge and a special symbol, e.g., ∞, if there is no such edge. Such a matrix is called the weight matrix or cost matrix. This approach is illustrated in Figure 1.8b for the weighted graph in Figure 1.8a. (For some applications, it is more convenient to put 0’s on the main diagonal of the adjacency matrix.) Adjacency lists for a weighted graph have to include in their nodes not only the name of an adjacent vertex but also the weight of the corresponding edge (Figure 1.8c). Paths and Cycles Among the many properties of graphs, two are important for a great number of applications: connectivity and acyclicity. Both are based on the notion of a path. A path from vertex u to vertex v of a graph G can be deﬁned as a sequence of adjacent (connected by an edge) vertices that starts with u and ends with v. If all vertices of a path are distinct, the path is said to be simple. The length of a path is the total number of vertices in the vertex sequence deﬁning the path minus 1, which is the same as the number of edges in the path. For example, a, c, b, f is a simple path of length 3 from a to f in the graph in Figure 1.6a, whereas a, c, e, c, b, f is a path (not simple) of length 5 from a to f.
a
5
b
7
1
c
2
(a)
4
d
a b c d
a
b 5
5 1
∞
∞ ∞
7 4
c 1 7
∞
2
∞
∞
(b)
d
4 2
a b c d
→ → → →
b, 5 a, 5 a, 1 b, 4
→ → → →
c, 1 c, 7 → d, 4 b, 7 → d, 2 c, 2
(c)
FIGURE 1.8 (a) Weighted graph. (b) Its weight matrix. (c) Its adjacency lists.
1.4 Fundamental Data Structures
a
b
c
31
f
e
d
g
h
i
FIGURE 1.9 Graph that is not connected.
In the case of a directed graph, we are usually interested in directed paths. A directed path is a sequence of vertices in which every consecutive pair of the vertices is connected by an edge directed from the vertex listed ﬁrst to the vertex listed next. For example, a, c, e, f is a directed path from a to f in the graph in Figure 1.6b. A graph is said to be connected if for every pair of its vertices u and v there is a path from u to v. If we make a model of a connected graph by connecting some balls representing the graph’s vertices with strings representing the edges, it will be a single piece. If a graph is not connected, such a model will consist of several connected pieces that are called connected components of the graph. Formally, a connected component is a maximal (not expandable by including another vertex and an edge) connected subgraph2 of a given graph. For example, the graphs in Figures 1.6a and 1.8a are connected, whereas the graph in Figure 1.9 is not, because there is no path, for example, from a to f. The graph in Figure 1.9 has two connected components with vertices {a, b, c, d, e} and {f, g, h, i}, respectively. Graphs with several connected components do happen in realworld applications. A graph representing the Interstate highway system of the United States would be an example (why?). It is important to know for many applications whether or not a graph under consideration has cycles. A cycle is a path of a positive length that starts and ends at the same vertex and does not traverse the same edge more than once. For example, f , h, i, g, f is a cycle in the graph in Figure 1.9. A graph with no cycles is said to be acyclic. We discuss acyclic graphs in the next subsection.
Trees A tree (more accurately, a free tree) is a connected acyclic graph (Figure 1.10a). A graph that has no cycles but is not necessarily connected is called a forest: each of its connected components is a tree (Figure 1.10b). 2.
A subgraph of a given graph G = V , E is a graph G = V , E such that V ⊆ V and E ⊆ E.
32
Introduction a
b
a
b
c
d
c
d
f
g
f
g
(a)
h
i
e
j
(b)
FIGURE 1.10 (a) Tree. (b) Forest.
i
d a
c
a
b
b
e c
h
g
f
(a)
h
d g
e f
i
(b)
FIGURE 1.11 (a) Free tree. (b) Its transformation into a rooted tree.
Trees have several important properties other graphs do not have. In particular, the number of edges in a tree is always one less than the number of its vertices: E = V  − 1. As the graph in Figure 1.9 demonstrates, this property is necessary but not sufﬁcient for a graph to be a tree. However, for connected graphs it is sufﬁcient and hence provides a convenient way of checking whether a connected graph has a cycle.
Rooted Trees Another very important property of trees is the fact that for every two vertices in a tree, there always exists exactly one simple path from one of these vertices to the other. This property makes it possible to select an arbitrary vertex in a free tree and consider it as the root of the socalled rooted tree. A rooted tree is usually depicted by placing its root on the top (level 0 of the tree), the vertices adjacent to the root below it (level 1), the vertices two edges apart from the root still below (level 2), and so on. Figure 1.11 presents such a transformation from a free tree to a rooted tree.
1.4 Fundamental Data Structures
33
Rooted trees play a very important role in computer science, a much more important one than free trees do; in fact, for the sake of brevity, they are often referred to as simply “trees.” An obvious application of trees is for describing hierarchies, from ﬁle directories to organizational charts of enterprises. There are many less obvious applications, such as implementing dictionaries (see below), efﬁcient access to very large data sets (Section 7.4), and data encoding (Section 9.4). As we discuss in Chapter 2, trees also are helpful in analysis of recursive algorithms. To ﬁnish this farfromcomplete list of tree applications, we should mention the socalled statespace trees that underline two important algorithm design techniques: backtracking and branchandbound (Sections 12.1 and 12.2). For any vertex v in a tree T , all the vertices on the simple path from the root to that vertex are called ancestors of v. The vertex itself is usually considered its own ancestor; the set of ancestors that excludes the vertex itself is referred to as the set of proper ancestors. If (u, v) is the last edge of the simple path from the root to vertex v (and u = v), u is said to be the parent of v and v is called a child of u; vertices that have the same parent are said to be siblings. A vertex with no children is called a leaf ; a vertex with at least one child is called parental. All the vertices for which a vertex v is an ancestor are said to be descendants of v; the proper descendants exclude the vertex v itself. All the descendants of a vertex v with all the edges connecting them form the subtree of T rooted at that vertex. Thus, for the tree in Figure 1.11b, the root of the tree is a; vertices d, g, f, h, and i are leaves, and vertices a, b, e, and c are parental; the parent of b is a; the children of b are c and g; the siblings of b are d and e; and the vertices of the subtree rooted at b are {b, c, g, h, i}. The depth of a vertex v is the length of the simple path from the root to v. The height of a tree is the length of the longest simple path from the root to a leaf. For example, the depth of vertex c of the tree in Figure 1.11b is 2, and the height of the tree is 3. Thus, if we count tree levels top down starting with 0 for the root’s level, the depth of a vertex is simply its level in the tree, and the tree’s height is the maximum level of its vertices. (You should be alert to the fact that some authors deﬁne the height of a tree as the number of levels in it; this makes the height of a tree larger by 1 than the height deﬁned as the length of the longest simple path from the root to a leaf.)
Ordered Trees An ordered tree is a rooted tree in which all the children of each vertex are ordered. It is convenient to assume that in a tree’s diagram, all the children are ordered left to right. A binary tree can be deﬁned as an ordered tree in which every vertex has no more than two children and each child is designated as either a left child or a right child of its parent; a binary tree may also be empty. An example of a binary tree is given in Figure 1.12a. The binary tree with its root at the left (right) child of a vertex in a binary tree is called the left (right) subtree of that vertex. Since left and right subtrees are binary trees as well, a binary tree can also be deﬁned recursively. This makes it possible to solve many problems involving binary trees by recursive algorithms.
34
Introduction
9 5 1
12 7
10
4
(a)
(b)
FIGURE 1.12 (a) Binary tree. (b) Binary search tree.
9
5
null
12
1
null
null
4
7
null
null
10
null
null
null
FIGURE 1.13 Standard implementation of the binary search tree in Figure 1.12b.
In Figure 1.12b, some numbers are assigned to vertices of the binary tree in Figure 1.12a. Note that a number assigned to each parental vertex is larger than all the numbers in its left subtree and smaller than all the numbers in its right subtree. Such trees are called binary search trees. Binary trees and binary search trees have a wide variety of applications in computer science; you will encounter some of them throughout the book. In particular, binary search trees can be generalized to more general types of search trees called multiway search trees, which are indispensable for efﬁcient access to very large data sets. As you will see later in the book, the efﬁciency of most important algorithms for binary search trees and their extensions depends on the tree’s height. Therefore, the following inequalities for the height h of a binary tree with n nodes are especially important for analysis of such algorithms: log2 n ≤ h ≤ n − 1.
35
1.4 Fundamental Data Structures
A binary tree is usually implemented for computing purposes by a collection of nodes corresponding to vertices of the tree. Each node contains some information associated with the vertex (its name or some value assigned to it) and two pointers to the nodes representing the left child and right child of the vertex, respectively. Figure 1.13 illustrates such an implementation for the binary search tree in Figure 1.12b. A computer representation of an arbitrary ordered tree can be done by simply providing a parental vertex with the number of pointers equal to the number of its children. This representation may prove to be inconvenient if the number of children varies widely among the nodes. We can avoid this inconvenience by using nodes with just two pointers, as we did for binary trees. Here, however, the left pointer will point to the ﬁrst child of the vertex, and the right pointer will point to its next sibling. Accordingly, this representation is called the ﬁrst child–next sibling representation. Thus, all the siblings of a vertex are linked via the nodes’ right pointers in a singly linked list, with the ﬁrst element of the list pointed to by the left pointer of their parent. Figure 1.14a illustrates this representation for the tree in Figure 1.11b. It is not difﬁcult to see that this representation effectively transforms an ordered tree into a binary tree said to be associated with the ordered tree. We get this representation by “rotating” the pointers about 45 degrees clockwise (see Figure 1.14b).
Sets and Dictionaries The notion of a set plays a central role in mathematics. A set can be described as an unordered collection (possibly empty) of distinct items called elements of the
a
null
a
null
b
null
d
c
null
g
null
h
null
i
null
null
e
null
f
null
b
d
c
g
h
i
(a)
e
f
(b)
FIGURE 1.14 (a) First child–next sibling representation of the tree in Figure 1.11b. (b) Its binary tree representation.
36
Introduction
set. A speciﬁc set is deﬁned either by an explicit listing of its elements (e.g., S = {2, 3, 5, 7}) or by specifying a property that all the set’s elements and only they must satisfy (e.g., S = {n: n is a prime number smaller than 10}). The most important set operations are: checking membership of a given item in a given set; ﬁnding the union of two sets, which comprises all the elements in either or both of them; and ﬁnding the intersection of two sets, which comprises all the common elements in the sets. Sets can be implemented in computer applications in two ways. The ﬁrst considers only sets that are subsets of some large set U, called the universal set. If set U has n elements, then any subset S of U can be represented by a bit string of size n, called a bit vector, in which the ith element is 1 if and only if the ith element of U is included in set S. Thus, to continue with our example, if U = {1, 2, 3, 4, 5, 6, 7, 8, 9}, then S = {2, 3, 5, 7} is represented by the bit string 011010100. This way of representing sets makes it possible to implement the standard set operations very fast, but at the expense of potentially using a large amount of storage. The second and more common way to represent a set for computing purposes is to use the list structure to indicate the set’s elements. Of course, this option, too, is feasible only for ﬁnite sets; fortunately, unlike mathematics, this is the kind of sets most computer applications need. Note, however, the two principal points of distinction between sets and lists. First, a set cannot contain identical elements; a list can. This requirement for uniqueness is sometimes circumvented by the introduction of a multiset, or bag, an unordered collection of items that are not necessarily distinct. Second, a set is an unordered collection of items; therefore, changing the order of its elements does not change the set. A list, deﬁned as an ordered collection of items, is exactly the opposite. This is an important theoretical distinction, but fortunately it is not important for many applications. It is also worth mentioning that if a set is represented by a list, depending on the application at hand, it might be worth maintaining the list in a sorted order. In computing, the operations we need to perform for a set or a multiset most often are searching for a given item, adding a new item, and deleting an item from the collection. A data structure that implements these three operations is called the dictionary. Note the relationship between this data structure and the problem of searching mentioned in Section 1.3; obviously, we are dealing here with searching in a dynamic context. Consequently, an efﬁcient implementation of a dictionary has to strike a compromise between the efﬁciency of searching and the efﬁciencies of the other two operations. There are quite a few ways a dictionary can be implemented. They range from an unsophisticated use of arrays (sorted or not) to much more sophisticated techniques such as hashing and balanced search trees, which we discuss later in the book. A number of applications in computing require a dynamic partition of some nelement set into a collection of disjoint subsets. After being initialized as a collection of n oneelement subsets, the collection is subjected to a sequence of intermixed union and search operations. This problem is called the set union problem. We discuss efﬁcient algorithmic solutions to this problem in Section 9.2, in conjunction with one of its important applications.
1.4 Fundamental Data Structures
37
You may have noticed that in our review of basic data structures we almost always mentioned speciﬁc operations that are typically performed for the structure in question. This intimate relationship between the data and operations has been recognized by computer scientists for a long time. It has led them in particular to the idea of an abstract data type (ADT): a set of abstract objects representing data items with a collection of operations that can be performed on them. As illustrations of this notion, reread, say, our deﬁnitions of the priority queue and dictionary. Although abstract data types could be implemented in older procedural languages such as Pascal (see, e.g., [Aho83]), it is much more convenient to do this in objectoriented languages such as C++ and Java, which support abstract data types by means of classes.
Exercises 1.4 1. Describe how one can implement each of the following operations on an array so that the time it takes does not depend on the array’s size n. a. Delete the ith element of an array (1 ≤ i ≤ n). b. Delete the ith element of a sorted array (the remaining array has to stay sorted, of course). 2. If you have to solve the searching problem for a list of n numbers, how can you take advantage of the fact that the list is known to be sorted? Give separate answers for a. lists represented as arrays. b. lists represented as linked lists. 3. a. Show the stack after each operation of the following sequence that starts with the empty stack: push(a), push(b), pop, push(c), push(d), pop b. Show the queue after each operation of the following sequence that starts with the empty queue: enqueue(a), enqueue(b), dequeue, enqueue(c), enqueue(d), dequeue 4. a. Let A be the adjacency matrix of an undirected graph. Explain what property of the matrix indicates that i. the graph is complete. ii. the graph has a loop, i.e., an edge connecting a vertex to itself. iii. the graph has an isolated vertex, i.e., a vertex with no edges incident to it. b. Answer the same questions for the adjacency list representation. 5. Give a detailed description of an algorithm for transforming a free tree into a tree rooted at a given vertex of the free tree.
38
Introduction
6. Prove the inequalities that bracket the height of a binary tree with n vertices: log2 n ≤ h ≤ n − 1. 7. Indicate how the ADT priority queue can be implemented as a. an (unsorted) array. b. a sorted array. c. a binary search tree. 8. How would you implement a dictionary of a reasonably small size n if you knew that all its elements are distinct (e.g., names of the 50 states of the United States)? Specify an implementation of each dictionary operation. 9. For each of the following applications, indicate the most appropriate data structure: a. answering telephone calls in the order of their known priorities b. sending backlog orders to customers in the order they have been received c. implementing a calculator for computing simple arithmetical expressions 10. Anagram checking Design an algorithm for checking whether two given words are anagrams, i.e., whether one word can be obtained by permuting the letters of the other. For example, the words tea and eat are anagrams.
SUMMARY An algorithm is a sequence of nonambiguous instructions for solving a problem in a ﬁnite amount of time. An input to an algorithm speciﬁes an instance of the problem the algorithm solves. Algorithms can be speciﬁed in a natural language or pseudocode; they can also be implemented as computer programs. Among several ways to classify algorithms, the two principal alternatives are: . to group algorithms according to types of problems they solve . to group algorithms according to underlying design techniques they are based upon The important problem types are sorting, searching, string processing, graph problems, combinatorial problems, geometric problems, and numerical problems. Algorithm design techniques (or “strategies” or “paradigms”) are general approaches to solving problems algorithmically, applicable to a variety of problems from different areas of computing.
Summary
39
Although designing an algorithm is undoubtedly a creative activity, one can identify a sequence of interrelated actions involved in such a process. They are summarized in Figure 1.2. A good algorithm is usually the result of repeated efforts and rework. The same problem can often be solved by several algorithms. For example, three algorithms were given for computing the greatest common divisor of two integers: Euclid’s algorithm, the consecutive integer checking algorithm, and the middleschool method enhanced by the sieve of Eratosthenes for generating a list of primes. Algorithms operate on data. This makes the issue of data structuring critical for efﬁcient algorithmic problem solving. The most important elementary data structures are the array and the linked list. They are used for representing more abstract data structures such as the list, the stack, the queue, the graph (via its adjacency matrix or adjacency lists), the binary tree, and the set. An abstract collection of objects with several operations that can be performed on them is called an abstract data type (ADT). The list, the stack, the queue, the priority queue, and the dictionary are important examples of abstract data types. Modern objectoriented languages support implementation of ADTs by means of classes.
This page intentionally left blank
2 Fundamentals of the Analysis of Algorithm Efficiency I often say that when you can measure what you are speaking about and express it in numbers you know something about it; but when you cannot express it in numbers your knowledge is a meagre and unsatisfactory kind: it may be the beginning of knowledge but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be. —Lord Kelvin (1824–1907)
Not everything that can be counted counts, and not everything that counts can be counted. —Albert Einstein (1879–1955)
T
his chapter is devoted to analysis of algorithms. The American Heritage Dictionary deﬁnes “analysis” as “the separation of an intellectual or substantial whole into its constituent parts for individual study.” Accordingly, each of the principal dimensions of an algorithm pointed out in Section 1.2 is both a legitimate and desirable subject of study. But the term “analysis of algorithms” is usually used in a narrower, technical sense to mean an investigation of an algorithm’s efﬁciency with respect to two resources: running time and memory space. This emphasis on efﬁciency is easy to explain. First, unlike such dimensions as simplicity and generality, efﬁciency can be studied in precise quantitative terms. Second, one can argue—although this is hardly always the case, given the speed and memory of today’s computers—that the efﬁciency considerations are of primary importance from a practical point of view. In this chapter, we too will limit the discussion to an algorithm’s efﬁciency.
41
42
Fundamentals of the Analysis of Algorithm Efﬁciency
We start with a general framework for analyzing algorithm efﬁciency in Section 2.1. This section is arguably the most important in the chapter; the fundamental nature of the topic makes it also one of the most important sections in the entire book. In Section 2.2, we introduce three notations: O (“big oh”), (“big omega”), and (“big theta”). Borrowed from mathematics, these notations have become the language for discussing the efﬁciency of algorithms. In Section 2.3, we show how the general framework outlined in Section 2.1 can be systematically applied to analyzing the efﬁciency of nonrecursive algorithms. The main tool of such an analysis is setting up a sum representing the algorithm’s running time and then simplifying the sum by using standard sum manipulation techniques. In Section 2.4, we show how the general framework outlined in Section 2.1 can be systematically applied to analyzing the efﬁciency of recursive algorithms. Here, the main tool is not a summation but a special kind of equation called a recurrence relation. We explain how such recurrence relations can be set up and then introduce a method for solving them. Although we illustrate the analysis framework and the methods of its applications by a variety of examples in the ﬁrst four sections of this chapter, Section 2.5 is devoted to yet another example—that of the Fibonacci numbers. Discovered 800 years ago, this remarkable sequence appears in a variety of applications both within and outside computer science. A discussion of the Fibonacci sequence serves as a natural vehicle for introducing an important class of recurrence relations not solvable by the method of Section 2.4. We also discuss several algorithms for computing the Fibonacci numbers, mostly for the sake of a few general observations about the efﬁciency of algorithms and methods of analyzing them. The methods of Sections 2.3 and 2.4 provide a powerful technique for analyzing the efﬁciency of many algorithms with mathematical clarity and precision, but these methods are far from being foolproof. The last two sections of the chapter deal with two approaches—empirical analysis and algorithm visualization—that complement the pure mathematical techniques of Sections 2.3 and 2.4. Much newer and, hence, less developed than their mathematical counterparts, these approaches promise to play an important role among the tools available for analysis of algorithm efﬁciency.
2.1 The Analysis Framework In this section, we outline a general framework for analyzing the efﬁciency of algorithms. We already mentioned in Section 1.2 that there are two kinds of efﬁciency: time efﬁciency and space efﬁciency. Time efﬁciency, also called time complexity, indicates how fast an algorithm in question runs. Space efﬁciency, also called space complexity, refers to the amount of memory units required by the algorithm in addition to the space needed for its input and output. In the early days of electronic computing, both resources—time and space—were at a premium. Half a century
2.1 The Analysis Framework
43
of relentless technological innovations have improved the computer’s speed and memory size by many orders of magnitude. Now the amount of extra space required by an algorithm is typically not of as much concern, with the caveat that there is still, of course, a difference between the fast main memory, the slower secondary memory, and the cache. The time issue has not diminished quite to the same extent, however. In addition, the research experience has shown that for most problems, we can achieve much more spectacular progress in speed than in space. Therefore, following a wellestablished tradition of algorithm textbooks, we primarily concentrate on time efﬁciency, but the analytical framework introduced here is applicable to analyzing space efﬁciency as well.
Measuring an Input’s Size Let’s start with the obvious observation that almost all algorithms run longer on larger inputs. For example, it takes longer to sort larger arrays, multiply larger matrices, and so on. Therefore, it is logical to investigate an algorithm’s efﬁciency as a function of some parameter n indicating the algorithm’s input size.1 In most cases, selecting such a parameter is quite straightforward. For example, it will be the size of the list for problems of sorting, searching, ﬁnding the list’s smallest element, and most other problems dealing with lists. For the problem of evaluating a polynomial p(x) = anx n + . . . + a0 of degree n, it will be the polynomial’s degree or the number of its coefﬁcients, which is larger by 1 than its degree. You’ll see from the discussion that such a minor difference is inconsequential for the efﬁciency analysis. There are situations, of course, where the choice of a parameter indicating an input size does matter. One such example is computing the product of two n × n matrices. There are two natural measures of size for this problem. The ﬁrst and more frequently used is the matrix order n. But the other natural contender is the total number of elements N in the matrices being multiplied. (The latter is also more general since it is applicable to matrices that are not necessarily square.) Since there is a simple formula relating these two measures, we can easily switch from one to the other, but the answer about an algorithm’s efﬁciency will be qualitatively different depending on which of these two measures we use (see Problem 2 in this section’s exercises). The choice of an appropriate size metric can be inﬂuenced by operations of the algorithm in question. For example, how should we measure an input’s size for a spellchecking algorithm? If the algorithm examines individual characters of its input, we should measure the size by the number of characters; if it works by processing words, we should count their number in the input. We should make a special note about measuring input size for algorithms solving problems such as checking primality of a positive integer n. Here, the input is just one number, and it is this number’s magnitude that determines the input 1.
Some algorithms require more than one parameter to indicate the size of their inputs (e.g., the number of vertices and the number of edges for algorithms on graphs represented by their adjacency lists).
44
Fundamentals of the Analysis of Algorithm Efﬁciency
size. In such situations, it is preferable to measure size by the number b of bits in the n’s binary representation: b = log2 n + 1.
(2.1)
This metric usually gives a better idea about the efﬁciency of algorithms in question.
Units for Measuring Running Time The next issue concerns units for measuring an algorithm’s running time. Of course, we can simply use some standard unit of time measurement—a second, or millisecond, and so on—to measure the running time of a program implementing the algorithm. There are obvious drawbacks to such an approach, however: dependence on the speed of a particular computer, dependence on the quality of a program implementing the algorithm and of the compiler used in generating the machine code, and the difﬁculty of clocking the actual running time of the program. Since we are after a measure of an algorithm’s efﬁciency, we would like to have a metric that does not depend on these extraneous factors. One possible approach is to count the number of times each of the algorithm’s operations is executed. This approach is both excessively difﬁcult and, as we shall see, usually unnecessary. The thing to do is to identify the most important operation of the algorithm, called the basic operation, the operation contributing the most to the total running time, and compute the number of times the basic operation is executed. As a rule, it is not difﬁcult to identify the basic operation of an algorithm: it is usually the most timeconsuming operation in the algorithm’s innermost loop. For example, most sorting algorithms work by comparing elements (keys) of a list being sorted with each other; for such algorithms, the basic operation is a key comparison. As another example, algorithms for mathematical problems typically involve some or all of the four arithmetical operations: addition, subtraction, multiplication, and division. Of the four, the most timeconsuming operation is division, followed by multiplication and then addition and subtraction, with the last two usually considered together.2 Thus, the established framework for the analysis of an algorithm’s time efﬁciency suggests measuring it by counting the number of times the algorithm’s basic operation is executed on inputs of size n. We will ﬁnd out how to compute such a count for nonrecursive and recursive algorithms in Sections 2.3 and 2.4, respectively. Here is an important application. Let cop be the execution time of an algorithm’s basic operation on a particular computer, and let C(n) be the number of times this operation needs to be executed for this algorithm. Then we can estimate 2.
On some computers, multiplication does not take longer than addition/subtraction (see, for example, the timing data provided by Kernighan and Pike in [Ker99, pp. 185–186]).
2.1 The Analysis Framework
45
the running time T (n) of a program implementing this algorithm on that computer by the formula T (n) ≈ cop C(n). Of course, this formula should be used with caution. The count C(n) does not contain any information about operations that are not basic, and, in fact, the count itself is often computed only approximately. Further, the constant cop is also an approximation whose reliability is not always easy to assess. Still, unless n is extremely large or very small, the formula can give a reasonable estimate of the algorithm’s running time. It also makes it possible to answer such questions as “How much faster would this algorithm run on a machine that is 10 times faster than the one we have?” The answer is, obviously, 10 times. Or, assuming that C(n) = 21 n(n − 1), how much longer will the algorithm run if we double its input size? The answer is about four times longer. Indeed, for all but very small values of n, 1 1 1 1 C(n) = n(n − 1) = n2 − n ≈ n2 2 2 2 2 and therefore 1 (2n)2 T (2n) cop C(2n) ≈ ≈ 21 = 4. 2 T (n) cop C(n) 2n
Note that we were able to answer the last question without actually knowing the value of cop : it was neatly cancelled out in the ratio. Also note that 21 , the multiplicative constant in the formula for the count C(n), was also cancelled out. It is for these reasons that the efﬁciency analysis framework ignores multiplicative constants and concentrates on the count’s order of growth to within a constant multiple for largesize inputs.
Orders of Growth Why this emphasis on the count’s order of growth for large input sizes? A difference in running times on small inputs is not what really distinguishes efﬁcient algorithms from inefﬁcient ones. When we have to compute, for example, the greatest common divisor of two small numbers, it is not immediately clear how much more efﬁcient Euclid’s algorithm is compared to the other two algorithms discussed in Section 1.1 or even why we should care which of them is faster and by how much. It is only when we have to ﬁnd the greatest common divisor of two large numbers that the difference in algorithm efﬁciencies becomes both clear and important. For large values of n, it is the function’s order of growth that counts: just look at Table 2.1, which contains values of a few functions particularly important for analysis of algorithms. The magnitude of the numbers in Table 2.1 has a profound signiﬁcance for the analysis of algorithms. The function growing the slowest among these is the logarithmic function. It grows so slowly, in fact, that we should expect a program
46
Fundamentals of the Analysis of Algorithm Efﬁciency TABLE 2.1 Values (some approximate) of several functions important for analysis of algorithms
n
log2 n
n
10 102 103 104 105 106
3.3 6.6 10 13 17 20
101 102 103 104 105 106
n log2 n 3.3.101 6.6.102 1.0.104 1.3.105 1.7.106 2.0.107
n2
n3
2n
n!
102 104 106 108 1010 1012
103 106 109 1012 1015 1018
103 1.3.1030
3.6.106 9.3.10157
implementing an algorithm with a logarithmic basicoperation count to run practically instantaneously on inputs of all realistic sizes. Also note that although speciﬁc values of such a count depend, of course, on the logarithm’s base, the formula loga n = loga b logb n makes it possible to switch from one base to another, leaving the count logarithmic but with a new multiplicative constant. This is why we omit a logarithm’s base and write simply log n in situations where we are interested just in a function’s order of growth to within a multiplicative constant. On the other end of the spectrum are the exponential function 2n and the factorial function n! Both these functions grow so fast that their values become astronomically large even for rather small values of n. (This is the reason why we did not include their values for n > 102 in Table 2.1.) For example, it would take about 4 . 1010 years for a computer making a trillion (1012 ) operations per second to execute 2100 operations. Though this is incomparably faster than it would have taken to execute 100! operations, it is still longer than 4.5 billion (4.5 . 109) years— the estimated age of the planet Earth. There is a tremendous difference between the orders of growth of the functions 2n and n!, yet both are often referred to as “exponentialgrowth functions” (or simply “exponential”) despite the fact that, strictly speaking, only the former should be referred to as such. The bottom line, which is important to remember, is this: Algorithms that require an exponential number of operations are practical for solving only problems of very small sizes. Another way to appreciate the qualitative difference among the orders of growth of the functions in Table 2.1 is to consider how they react to, say, a twofold increase in the value of their argument n. The function log2 n increases in value by just 1 (because log2 2n = log2 2 + log2 n = 1 + log2 n); the linear function increases twofold, the linearithmic function n log2 n increases slightly more than twofold; the quadratic function n2 and cubic function n3 increase fourfold and
2.1 The Analysis Framework
47
eightfold, respectively (because (2n)2 = 4n2 and (2n)3 = 8n3); the value of 2n gets squared (because 22n = (2n)2 ); and n! increases much more than that (yes, even mathematics refuses to cooperate to give a neat answer for n!).
WorstCase, BestCase, and AverageCase Efﬁciencies In the beginning of this section, we established that it is reasonable to measure an algorithm’s efﬁciency as a function of a parameter indicating the size of the algorithm’s input. But there are many algorithms for which running time depends not only on an input size but also on the speciﬁcs of a particular input. Consider, as an example, sequential search. This is a straightforward algorithm that searches for a given item (some search key K) in a list of n elements by checking successive elements of the list until either a match with the search key is found or the list is exhausted. Here is the algorithm’s pseudocode, in which, for simplicity, a list is implemented as an array. It also assumes that the second condition A[i] = K will not be checked if the ﬁrst one, which checks that the array’s index does not exceed its upper bound, fails. ALGORITHM SequentialSearch(A[0..n − 1], K) //Searches for a given value in a given array by sequential search //Input: An array A[0..n − 1] and a search key K //Output: The index of the ﬁrst element in A that matches K // or −1 if there are no matching elements i←0 while i < n and A[i] = K do i ←i+1 if i < n return i else return −1 Clearly, the running time of this algorithm can be quite different for the same list size n. In the worst case, when there are no matching elements or the ﬁrst matching element happens to be the last one on the list, the algorithm makes the largest number of key comparisons among all possible inputs of size n: Cworst (n) = n. The worstcase efﬁciency of an algorithm is its efﬁciency for the worstcase input of size n, which is an input (or inputs) of size n for which the algorithm runs the longest among all possible inputs of that size. The way to determine the worstcase efﬁciency of an algorithm is, in principle, quite straightforward: analyze the algorithm to see what kind of inputs yield the largest value of the basic operation’s count C(n) among all possible inputs of size n and then compute this worstcase value Cworst (n). (For sequential search, the answer was obvious. The methods for handling less trivial situations are explained in subsequent sections of this chapter.) Clearly, the worstcase analysis provides very important information about an algorithm’s efﬁciency by bounding its running time from above. In other
48
Fundamentals of the Analysis of Algorithm Efﬁciency
words, it guarantees that for any instance of size n, the running time will not exceed Cworst (n), its running time on the worstcase inputs. The bestcase efﬁciency of an algorithm is its efﬁciency for the bestcase input of size n, which is an input (or inputs) of size n for which the algorithm runs the fastest among all possible inputs of that size. Accordingly, we can analyze the bestcase efﬁciency as follows. First, we determine the kind of inputs for which the count C(n) will be the smallest among all possible inputs of size n. (Note that the best case does not mean the smallest input; it means the input of size n for which the algorithm runs the fastest.) Then we ascertain the value of C(n) on these most convenient inputs. For example, the bestcase inputs for sequential search are lists of size n with their ﬁrst element equal to a search key; accordingly, Cbest (n) = 1 for this algorithm. The analysis of the bestcase efﬁciency is not nearly as important as that of the worstcase efﬁciency. But it is not completely useless, either. Though we should not expect to get bestcase inputs, we might be able to take advantage of the fact that for some algorithms a good bestcase performance extends to some useful types of inputs close to being the bestcase ones. For example, there is a sorting algorithm (insertion sort) for which the bestcase inputs are already sorted arrays on which the algorithm works very fast. Moreover, the bestcase efﬁciency deteriorates only slightly for almostsorted arrays. Therefore, such an algorithm might well be the method of choice for applications dealing with almostsorted arrays. And, of course, if the bestcase efﬁciency of an algorithm is unsatisfactory, we can immediately discard it without further analysis. It should be clear from our discussion, however, that neither the worstcase analysis nor its bestcase counterpart yields the necessary information about an algorithm’s behavior on a “typical” or “random” input. This is the information that the averagecase efﬁciency seeks to provide. To analyze the algorithm’s averagecase efﬁciency, we must make some assumptions about possible inputs of size n. Let’s consider again sequential search. The standard assumptions are that (a) the probability of a successful search is equal to p (0 ≤ p ≤ 1) and (b) the probability of the ﬁrst match occurring in the ith position of the list is the same for every i. Under these assumptions—the validity of which is usually difﬁcult to verify, their reasonableness notwithstanding—we can ﬁnd the average number of key comparisons Cavg (n) as follows. In the case of a successful search, the probability of the ﬁrst match occurring in the ith position of the list is p/n for every i, and the number of comparisons made by the algorithm in such a situation is obviously i. In the case of an unsuccessful search, the number of comparisons will be n with the probability of such a search being (1 − p). Therefore, Cavg (n) = [1 .
p p p p + 2 . + . . . + i . + . . . + n . ] + n . (1 − p) n n n n
p [1 + 2 + . . . + i + . . . + n] + n(1 − p) n p(n + 1) p n(n + 1) + n(1 − p) = + n(1 − p). = n 2 2 =
2.1 The Analysis Framework
49
This general formula yields some quite reasonable answers. For example, if p = 1 (the search must be successful), the average number of key comparisons made by sequential search is (n + 1)/2; that is, the algorithm will inspect, on average, about half of the list’s elements. If p = 0 (the search must be unsuccessful), the average number of key comparisons will be n because the algorithm will inspect all n elements on all such inputs. As you can see from this very elementary example, investigation of the averagecase efﬁciency is considerably more difﬁcult than investigation of the worstcase and bestcase efﬁciencies. The direct approach for doing this involves dividing all instances of size n into several classes so that for each instance of the class the number of times the algorithm’s basic operation is executed is the same. (What were these classes for sequential search?) Then a probability distribution of inputs is obtained or assumed so that the expected value of the basic operation’s count can be found. The technical implementation of this plan is rarely easy, however, and probabilistic assumptions underlying it in each particular case are usually difﬁcult to verify. Given our quest for simplicity, we will mostly quote known results about the averagecase efﬁciency of algorithms under discussion. If you are interested in derivations of these results, consult such books as [Baa00], [Sed96], [KnuI], [KnuII], and [KnuIII]. It should be clear from the preceding discussion that the averagecase efﬁciency cannot be obtained by taking the average of the worstcase and the bestcase efﬁciencies. Even though this average does occasionally coincide with the averagecase cost, it is not a legitimate way of performing the averagecase analysis. Does one really need the averagecase efﬁciency information? The answer is unequivocally yes: there are many important algorithms for which the averagecase efﬁciency is much better than the overly pessimistic worstcase efﬁciency would lead us to believe. So, without the averagecase analysis, computer scientists could have missed many important algorithms. Yet another type of efﬁciency is called amortized efﬁciency. It applies not to a single run of an algorithm but rather to a sequence of operations performed on the same data structure. It turns out that in some situations a single operation can be expensive, but the total time for an entire sequence of n such operations is always signiﬁcantly better than the worstcase efﬁciency of that single operation multiplied by n. So we can “amortize” the high cost of such a worstcase occurrence over the entire sequence in a manner similar to the way a business would amortize the cost of an expensive item over the years of the item’s productive life. This sophisticated approach was discovered by the American computer scientist Robert Tarjan, who used it, among other applications, in developing an interesting variation of the classic binary search tree (see [Tar87] for a quite readable nontechnical discussion and [Tar85] for a technical account). We will see an example of the usefulness of amortized efﬁciency in Section 9.2, when we consider algorithms for ﬁnding unions of disjoint sets.
50
Fundamentals of the Analysis of Algorithm Efﬁciency
Recapitulation of the Analysis Framework Before we leave this section, let us summarize the main points of the framework outlined above. Both time and space efﬁciencies are measured as functions of the algorithm’s input size. Time efﬁciency is measured by counting the number of times the algorithm’s basic operation is executed. Space efﬁciency is measured by counting the number of extra memory units consumed by the algorithm. The efﬁciencies of some algorithms may differ signiﬁcantly for inputs of the same size. For such algorithms, we need to distinguish between the worstcase, averagecase, and bestcase efﬁciencies. The framework’s primary interest lies in the order of growth of the algorithm’s running time (extra memory units consumed) as its input size goes to inﬁnity. In the next section, we look at formal means to investigate orders of growth. In Sections 2.3 and 2.4, we discuss particular methods for investigating nonrecursive and recursive algorithms, respectively. It is there that you will see how the analysis framework outlined here can be applied to investigating the efﬁciency of speciﬁc algorithms. You will encounter many more examples throughout the rest of the book.
Exercises 2.1 1. For each of the following algorithms, indicate (i) a natural size metric for its inputs, (ii) its basic operation, and (iii) whether the basic operation count can be different for inputs of the same size: a. computing the sum of n numbers b. computing n! c. d. e. f.
ﬁnding the largest element in a list of n numbers Euclid’s algorithm sieve of Eratosthenes penandpencil algorithm for multiplying two ndigit decimal integers
2. a. Consider the deﬁnitionbased algorithm for adding two n × n matrices. What is its basic operation? How many times is it performed as a function of the matrix order n? As a function of the total number of elements in the input matrices? b. Answer the same questions for the deﬁnitionbased algorithm for matrix multiplication.
2.1 The Analysis Framework
51
3. Consider a variation of sequential search that scans a list to return the number of occurrences of a given search key in the list. Does its efﬁciency differ from the efﬁciency of classic sequential search? 4. a. Glove selection There are 22 gloves in a drawer: 5 pairs of red gloves, 4 pairs of yellow, and 2 pairs of green. You select the gloves in the dark and can check them only after a selection has been made. What is the smallest number of gloves you need to select to have at least one matching pair in the best case? In the worst case? b. Missing socks Imagine that after washing 5 distinct pairs of socks, you discover that two socks are missing. Of course, you would like to have the largest number of complete pairs remaining. Thus, you are left with 4 complete pairs in the bestcase scenario and with 3 complete pairs in the worst case. Assuming that the probability of disappearance for each of the 10 socks is the same, ﬁnd the probability of the bestcase scenario; the probability of the worstcase scenario; the number of pairs you should expect in the average case. 5. a. Prove formula (2.1) for the number of bits in the binary representation of a positive decimal integer. b. Prove the alternative formula for the number of bits in the binary representation of a positive integer n: b = log2 (n + 1). c. What would be the analogous formulas for the number of decimal digits? d. Explain why, within the accepted analysis framework, it does not matter whether we use binary or decimal digits in measuring n’s size. 6. Suggest how any sorting algorithm can be augmented in a way to make the bestcase count of its key comparisons equal to just n − 1 (n is a list’s size, of course). Do you think it would be a worthwhile addition to any sorting algorithm? 7. Gaussian elimination, the classic algorithm for solving systems of n linear equations in n unknowns, requires about 31 n3 multiplications, which is the algorithm’s basic operation. a. How much longer should you expect Gaussian elimination to work on a system of 1000 equations versus a system of 500 equations? b. You are considering buying a computer that is 1000 times faster than the one you currently have. By what factor will the faster computer increase the sizes of systems solvable in the same amount of time as on the old computer? 8. For each of the following functions, indicate how much the function’s value will change if its argument is increased fourfold. √ b. n c. n d. n2 e. n3 f. 2n a. log2 n
52
Fundamentals of the Analysis of Algorithm Efﬁciency
9. For each of the following pairs of functions, indicate whether the ﬁrst function of each of the following pairs has a lower, same, or higher order of growth (to within a constant multiple) than the second function. a. n(n + 1) and 2000n2 b. 100n2 and 0.01n3 c. log2 n and ln n
d. log22 n and log2 n2
e. 2n−1 and 2n
f. (n − 1)! and n!
10. Invention of chess a. According to a wellknown legend, the game of chess was invented many centuries ago in northwestern India by a certain sage. When he took his invention to his king, the king liked the game so much that he offered the inventor any reward he wanted. The inventor asked for some grain to be obtained as follows: just a single grain of wheat was to be placed on the ﬁrst square of the chessboard, two on the second, four on the third, eight on the fourth, and so on, until all 64 squares had been ﬁlled. If it took just 1 second to count each grain, how long would it take to count all the grain due to him? b. How long would it take if instead of doubling the number of grains for each square of the chessboard, the inventor asked for adding two grains?
2.2 Asymptotic Notations and Basic Efﬁciency Classes As pointed out in the previous section, the efﬁciency analysis framework concentrates on the order of growth of an algorithm’s basic operation count as the principal indicator of the algorithm’s efﬁciency. To compare and rank such orders of growth, computer scientists use three notations: O (big oh), (big omega), and (big theta). First, we introduce these notations informally, and then, after several examples, formal deﬁnitions are given. In the following discussion, t (n) and g(n) can be any nonnegative functions deﬁned on the set of natural numbers. In the context we are interested in, t (n) will be an algorithm’s running time (usually indicated by its basic operation count C(n)), and g(n) will be some simple function to compare the count with.
Informal Introduction Informally, O(g(n)) is the set of all functions with a lower or same order of growth as g(n) (to within a constant multiple, as n goes to inﬁnity). Thus, to give a few examples, the following assertions are all true: n ∈ O(n2 ),
100n + 5 ∈ O(n2 ),
1 n(n − 1) ∈ O(n2 ). 2
2.2
Asymptotic Notations and Basic Efﬁciency Classes
53
Indeed, the ﬁrst two functions are linear and hence have a lower order of growth than g(n) = n2 , while the last one is quadratic and hence has the same order of growth as n2 . On the other hand, n3 ∈ O(n2 ),
0.00001n3 ∈ O(n2 ),
n4 + n + 1 ∈ O(n2 ).
Indeed, the functions n3 and 0.00001n3 are both cubic and hence have a higher order of growth than n2 , and so has the fourthdegree polynomial n4 + n + 1. The second notation, (g(n)), stands for the set of all functions with a higher or same order of growth as g(n) (to within a constant multiple, as n goes to inﬁnity). For example, n3 ∈ (n2 ),
1 n(n − 1) ∈ (n2 ), 2
but 100n + 5 ∈ (n2 ).
Finally, (g(n)) is the set of all functions that have the same order of growth as g(n) (to within a constant multiple, as n goes to inﬁnity). Thus, every quadratic function an2 + bn + c with a > 0 is in (n2 ), but so are, among inﬁnitely many others, n2 + sin n and n2 + log n. (Can you explain why?) Hopefully, this informal introduction has made you comfortable with the idea behind the three asymptotic notations. So now come the formal deﬁnitions.
Onotation DEFINITION A function t (n) is said to be in O(g(n)), denoted t (n) ∈ O(g(n)), if t (n) is bounded above by some constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n0 such that t (n) ≤ cg(n)
for all n ≥ n0.
The deﬁnition is illustrated in Figure 2.1 where, for the sake of visual clarity, n is extended to be a real number. As an example, let us formally prove one of the assertions made in the introduction: 100n + 5 ∈ O(n2 ). Indeed, 100n + 5 ≤ 100n + n (for all n ≥ 5) = 101n ≤ 101n2 . Thus, as values of the constants c and n0 required by the deﬁnition, we can take 101 and 5, respectively. Note that the deﬁnition gives us a lot of freedom in choosing speciﬁc values for constants c and n0. For example, we could also reason that 100n + 5 ≤ 100n + 5n (for all n ≥ 1) = 105n to complete the proof with c = 105 and n0 = 1.
54
Fundamentals of the Analysis of Algorithm Efﬁciency
cg (n ) t (n )
doesn't matter
n
n0
FIGURE 2.1 Bigoh notation: t (n) ∈ O(g(n)).
t (n ) cg (n )
doesn't matter
n
n0
FIGURE 2.2 Bigomega notation: t (n) ∈ (g(n)).
notation DEFINITION A function t (n) is said to be in (g(n)), denoted t (n) ∈ (g(n)), if t (n) is bounded below by some positive constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n0 such that t (n) ≥ cg(n)
for all n ≥ n0.
The deﬁnition is illustrated in Figure 2.2. Here is an example of the formal proof that n3 ∈ (n2 ): n3 ≥ n2 i.e., we can select c = 1 and n0 = 0.
for all n ≥ 0,
2.2
Asymptotic Notations and Basic Efﬁciency Classes
55
c1g (n ) t (n ) c2g (n )
doesn't matter
n
n0
FIGURE 2.3 Bigtheta notation: t (n) ∈ (g(n)).
notation DEFINITION A function t (n) is said to be in (g(n)), denoted t (n) ∈ (g(n)), if t (n) is bounded both above and below by some positive constant multiples of g(n) for all large n, i.e., if there exist some positive constants c1 and c2 and some nonnegative integer n0 such that c2 g(n) ≤ t (n) ≤ c1g(n)
for all n ≥ n0.
The deﬁnition is illustrated in Figure 2.3. For example, let us prove that 21 n(n − 1) ∈ (n2 ). First, we prove the right inequality (the upper bound): 1 1 1 1 n(n − 1) = n2 − n ≤ n2 2 2 2 2
for all n ≥ 0.
Second, we prove the left inequality (the lower bound): 1 1 1 1 1 1 1 n(n − 1) = n2 − n ≥ n2 − n n (for all n ≥ 2) = n2 . 2 2 2 2 2 2 4 Hence, we can select c2 = 41 , c1 = 21 , and n0 = 2.
Useful Property Involving the Asymptotic Notations Using the formal deﬁnitions of the asymptotic notations, we can prove their general properties (see Problem 7 in this section’s exercises for a few simple examples). The following property, in particular, is useful in analyzing algorithms that comprise two consecutively executed parts.
56
Fundamentals of the Analysis of Algorithm Efﬁciency
THEOREM
If t1(n) ∈ O(g1(n)) and t2 (n) ∈ O(g2 (n)), then t1(n) + t2 (n) ∈ O(max{g1(n), g2 (n)}).
(The analogous assertions are true for the and notations as well.) PROOF The proof extends to orders of growth the following simple fact about four arbitrary real numbers a1, b1, a2 , b2 : if a1 ≤ b1 and a2 ≤ b2 , then a1 + a2 ≤ 2 max{b1, b2 }. Since t1(n) ∈ O(g1(n)), there exist some positive constant c1 and some nonnegative integer n1 such that t1(n) ≤ c1g1(n)
for all n ≥ n1.
Similarly, since t2 (n) ∈ O(g2 (n)), t2 (n) ≤ c2 g2 (n)
for all n ≥ n2 .
Let us denote c3 = max{c1, c2 } and consider n ≥ max{n1, n2 } so that we can use both inequalities. Adding them yields the following: t1(n) + t2 (n) ≤ c1g1(n) + c2 g2 (n) ≤ c3g1(n) + c3g2 (n) = c3[g1(n) + g2 (n)] ≤ c32 max{g1(n), g2 (n)}. Hence, t1(n) + t2 (n) ∈ O(max{g1(n), g2 (n)}), with the constants c and n0 required by the O deﬁnition being 2c3 = 2 max{c1, c2 } and max{n1, n2 }, respectively. So what does this property imply for an algorithm that comprises two consecutively executed parts? It implies that the algorithm’s overall efﬁciency is determined by the part with a higher order of growth, i.e., its least efﬁcient part: t1(n) ∈ O(g1(n)) t1(n) + t2 (n) ∈ O(max{g1(n), g2 (n)}). t2 (n) ∈ O(g2 (n)) For example, we can check whether an array has equal elements by the following twopart algorithm: ﬁrst, sort the array by applying some known sorting algorithm; second, scan the sorted array to check its consecutive elements for equality. If, for example, a sorting algorithm used in the ﬁrst part makes no more than 21 n(n − 1) comparisons (and hence is in O(n2 )) while the second part makes no more than n − 1 comparisons (and hence is in O(n)), the efﬁciency of the entire algorithm will be in O(max{n2 , n}) = O(n2 ).
Using Limits for Comparing Orders of Growth Though the formal deﬁnitions of O, , and are indispensable for proving their abstract properties, they are rarely used for comparing the orders of growth of two speciﬁc functions. A much more convenient method for doing so is based on
2.2
Asymptotic Notations and Basic Efﬁciency Classes
57
computing the limit of the ratio of two functions in question. Three principal cases may arise: ⎧ 0 implies that t (n) has a smaller order of growth than g(n), t (n) ⎨ implies that t (n) has the same order of growth as g(n), lim = c n→∞ g(n) ⎩ ∞ implies that t (n) has a larger order of growth than g(n).3 Note that the ﬁrst two cases mean that t (n) ∈ O(g(n)), the last two mean that t (n) ∈ (g(n)), and the second case means that t (n) ∈ (g(n)). The limitbased approach is often more convenient than the one based on the deﬁnitions because it can take advantage of the powerful calculus techniques ˆ developed for computing limits, such as L’Hopital’s rule lim
n→∞
t (n) t (n) = lim
g(n) n→∞ g (n)
and Stirling’s formula n √ n for large values of n. n! ≈ 2π n e Here are three examples of using the limitbased approach to comparing orders of growth of two functions. EXAMPLE 1 Compare the orders of growth of 21 n(n − 1) and n2 . (This is one of the examples we used at the beginning of this section to illustrate the deﬁnitions.) 1 2 n(n − 1) n→∞ n2
lim
=
1 n2 − n 1 1 1 lim = lim (1 − ) = . 2 n→∞ n→∞ 2 n 2 n 2
Since the limit is equal to a positive constant, the functions have the same order of growth or, symbolically, 21 n(n − 1) ∈ (n2 ). √ EXAMPLE 2 Compare the orders of growth of log2 n and n. (Unlike Example 1, the answer here is not immediately obvious.)
log2 e n1 log2 n log2 n 1 = 2 log2 e lim √ = 0. lim √ = lim √ = lim 1 n→∞ n→∞ n→∞ n→∞ √ n n n 2 n √ Since the limit is equal to zero, log2 n has a smaller order of growth than n. (Since √ log n limn→∞ √2n = 0, we can use the socalled littleoh notation: log2 n ∈ o( n). Unlike the bigOh, the littleoh notation is rarely used in analysis of algorithms.)
3.
The fourth case, in which such a limit does not exist, rarely happens in the actual practice of analyzing algorithms. Still, this possibility makes the limitbased approach to comparing orders of growth less general than the one based on the deﬁnitions of O, , and .
58
Fundamentals of the Analysis of Algorithm Efﬁciency
EXAMPLE 3 Compare the orders of growth of n! and 2n. (We discussed this informally in Section 2.1.) Taking advantage of Stirling’s formula, we get √ n n √ √ 2π n ne n n! nn = lim = lim 2πn n n = lim 2π n = ∞. lim n n→∞ 2n n→∞ n→∞ n→∞ 2 2 e 2e Thus, though 2n grows very fast, n!grows still faster. We can write symbolically that n! ∈ (2n); note, however, that while the bigOmega notation does not preclude the possibility that n! and 2n have the same order of growth, the limit computed here certainly does.
Basic Efﬁciency Classes Even though the efﬁciency analysis framework puts together all the functions whose orders of growth differ by a constant multiple, there are still inﬁnitely many such classes. (For example, the exponential functions a n have different orders of growth for different values of base a.) Therefore, it may come as a surprise that the time efﬁciencies of a large number of algorithms fall into only a few classes. These classes are listed in Table 2.2 in increasing order of their orders of growth, along with their names and a few comments. You could raise a concern that classifying algorithms by their asymptotic efﬁciency would be of little practical use since the values of multiplicative constants are usually left unspeciﬁed. This leaves open the possibility of an algorithm in a worse efﬁciency class running faster than an algorithm in a better efﬁciency class for inputs of realistic sizes. For example, if the running time of one algorithm is n3 while the running time of the other is 106n2 , the cubic algorithm will outperform the quadratic algorithm unless n exceeds 106. A few such anomalies are indeed known. Fortunately, multiplicative constants usually do not differ that drastically. As a rule, you should expect an algorithm from a better asymptotic efﬁciency class to outperform an algorithm from a worse class even for moderately sized inputs. This observation is especially true for an algorithm with a better than exponential running time versus an exponential (or worse) algorithm.
Exercises 2.2 1. Use the most appropriate notation among O, , and to indicate the time efﬁciency class of sequential search (see Section 2.1) a. in the worst case. b. in the best case. c. in the average case. 2. Use the informal deﬁnitions of O, , and to determine whether the following assertions are true or false.
2.2
Asymptotic Notations and Basic Efﬁciency Classes
59
TABLE 2.2 Basic asymptotic efﬁciency classes
Class
Name
Comments
1
constant
log n
logarithmic
n
linear
n log n
linearithmic
n2
quadratic
n3
cubic
2n
exponential
n!
factorial
Short of bestcase efﬁciencies, very few reasonable examples can be given since an algorithm’s running time typically goes to inﬁnity when its input size grows inﬁnitely large. Typically, a result of cutting a problem’s size by a constant factor on each iteration of the algorithm (see Section 4.4). Note that a logarithmic algorithm cannot take into account all its input or even a ﬁxed fraction of it: any algorithm that does so will have at least linear running time. Algorithms that scan a list of size n (e.g., sequential search) belong to this class. Many divideandconquer algorithms (see Chapter 5), including mergesort and quicksort in the average case, fall into this category. Typically, characterizes efﬁciency of algorithms with two embedded loops (see the next section). Elementary sorting algorithms and certain operations on n × n matrices are standard examples. Typically, characterizes efﬁciency of algorithms with three embedded loops (see the next section). Several nontrivial algorithms from linear algebra fall into this class. Typical for algorithms that generate all subsets of an nelement set. Often, the term “exponential” is used in a broader sense to include this and larger orders of growth as well. Typical for algorithms that generate all permutations of an nelement set.
a. n(n + 1)/2 ∈ O(n3) c. n(n + 1)/2 ∈ (n3)
b. n(n + 1)/2 ∈ O(n2 ) d. n(n + 1)/2 ∈ (n)
3. For each of the following functions, indicate the class (g(n)) the function belongs to. (Use the simplest g(n) possible in your answers.) Prove your assertions. a. (n2 + 1)10 b. 10n2 + 7n + 3 c. 2n lg(n + 2)2 + (n + 2)2 lg n2 d. 2n+1 + 3n−1 e. log2 n
60
Fundamentals of the Analysis of Algorithm Efﬁciency
4. a. Table 2.1 contains values of several functions that often arise in the analysis of algorithms. These values certainly suggest that the functions log n,
n,
n log2 n,
n2 ,
n3 ,
2n ,
n!
are listed in increasing order of their order of growth. Do these values prove this fact with mathematical certainty? b. Prove that the functions are indeed listed in increasing order of their order of growth. 5. List the following functions according to their order of growth from the lowest to the highest: √ (n − 2)!, 5 lg(n + 100)10, 22n, 0.001n4 + 3n3 + 1, ln2 n, 3 n, 3n. 6. a. Prove that every polynomial of degree k, p(n) = ak nk + ak−1nk−1 + . . . + a0 with ak > 0, belongs to (nk ). b. Prove that exponential functions a n have different orders of growth for different values of base a > 0. 7. Prove the following assertions by using the deﬁnitions of the notations involved, or disprove them by giving a speciﬁc counterexample. a. If t (n) ∈ O(g(n)), then g(n) ∈ (t (n)). b. (αg(n)) = (g(n)), where α > 0. c. (g(n)) = O(g(n)) ∩ (g(n)). d. For any two nonnegative functions t (n) and g(n) deﬁned on the set of nonnegative integers, either t (n) ∈ O(g(n)), or t (n) ∈ (g(n)), or both. 8. Prove the section’s theorem for a. notation. b. notation. 9. We mentioned in this section that one can check whether all elements of an array are distinct by a twopart algorithm based on the array’s presorting. a. If the presorting is done by an algorithm with a time efﬁciency in (n log n), what will be a timeefﬁciency class of the entire algorithm? b. If the sorting algorithm used for presorting needs an extra array of size n, what will be the spaceefﬁciency class of the entire algorithm? 10. The range of a ﬁnite nonempty set of n real numbers S is deﬁned as the difference between the largest and smallest elements of S. For each representation of S given below, describe in English an algorithm to compute the range. Indicate the time efﬁciency classes of these algorithms using the most appropriate notation (O, , or ). a. An unsorted array b. A sorted array c. A sorted singly linked list d. A binary search tree
2.3
Mathematical Analysis of Nonrecursive Algorithms
61
11. Lighter or heavier? You have n > 2 identicallooking coins and a twopan balance scale with no weights. One of the coins is a fake, but you do not know whether it is lighter or heavier than the genuine coins, which all weigh the same. Design a (1) algorithm to determine whether the fake coin is lighter or heavier than the others. 12. Door in a wall You are facing a wall that stretches inﬁnitely in both directions. There is a door in the wall, but you know neither how far away nor in which direction. You can see the door only when you are right next to it. Design an algorithm that enables you to reach the door by walking at most O(n) steps where n is the (unknown to you) number of steps between your initial position and the door. [Par95]
2.3 Mathematical Analysis of Nonrecursive Algorithms In this section, we systematically apply the general framework outlined in Section 2.1 to analyzing the time efﬁciency of nonrecursive algorithms. Let us start with a very simple example that demonstrates all the principal steps typically taken in analyzing such algorithms. EXAMPLE 1 Consider the problem of ﬁnding the value of the largest element in a list of n numbers. For simplicity, we assume that the list is implemented as an array. The following is pseudocode of a standard algorithm for solving the problem. ALGORITHM MaxElement(A[0..n − 1]) //Determines the value of the largest element in a given array //Input: An array A[0..n − 1] of real numbers //Output: The value of the largest element in A maxval ← A[0] for i ← 1 to n − 1 do if A[i] > maxval maxval ← A[i] return maxval The obvious measure of an input’s size here is the number of elements in the array, i.e., n. The operations that are going to be executed most often are in the algorithm’s for loop. There are two operations in the loop’s body: the comparison A[i] > maxval and the assignment maxval ← A[i]. Which of these two operations should we consider basic? Since the comparison is executed on each repetition of the loop and the assignment is not, we should consider the comparison to be the algorithm’s basic operation. Note that the number of comparisons will be the same for all arrays of size n; therefore, in terms of this metric, there is no need to distinguish among the worst, average, and best cases here.
62
Fundamentals of the Analysis of Algorithm Efﬁciency
Let us denote C(n) the number of times this comparison is executed and try to ﬁnd a formula expressing it as a function of size n. The algorithm makes one comparison on each execution of the loop, which is repeated for each value of the loop’s variable i within the bounds 1 and n − 1, inclusive. Therefore, we get the following sum for C(n): C(n) =
n−1
1.
i=1
This is an easy sum to compute because it is nothing other than 1 repeated n − 1 times. Thus, C(n) =
n−1
1 = n − 1 ∈ (n).
i=1
Here is a general plan to follow in analyzing nonrecursive algorithms. General Plan for Analyzing the Time Efﬁciency of Nonrecursive Algorithms 1. Decide on a parameter (or parameters) indicating an input’s size. 2. Identify the algorithm’s basic operation. (As a rule, it is located in the innermost loop.) 3. Check whether the number of times the basic operation is executed depends only on the size of an input. If it also depends on some additional property, the worstcase, averagecase, and, if necessary, bestcase efﬁciencies have to be investigated separately. 4. Set up a sum expressing the number of times the algorithm’s basic operation is executed.4 5. Using standard formulas and rules of sum manipulation, either ﬁnd a closedform formula for the count or, at the very least, establish its order of growth. Before proceeding with further examples, you may want to review Appendix A, which contains a list of summation formulas and rules that are often useful in analysis of algorithms. In particular, we use especially frequently two basic rules of sum manipulation u
cai = c
i=l u
i=l
4.
(ai ± bi ) =
u
(R1)
ai ,
i=l u
u
i=l
i=l
ai ±
bi ,
(R2)
Sometimes, an analysis of a nonrecursive algorithm requires setting up not a sum but a recurrence relation for the number of times its basic operation is executed. Using recurrence relations is much more typical for analyzing recursive algorithms (see Section 2.4).
2.3
Mathematical Analysis of Nonrecursive Algorithms
63
and two summation formulas u
i=l n
i=0
1 = u − l + 1 where l ≤ u are some lower and upper integer limits, (S1) i=
n
i=1
i =1+2 +...+n=
n(n + 1) 1 2 ≈ n ∈ (n2 ). 2 2
(S2)
Note that the formula n−1 i=1 1 = n − 1, which we used in Example 1, is a special case of formula (S1) for l = 1 and u = n − 1. EXAMPLE 2 Consider the element uniqueness problem: check whether all the elements in a given array of n elements are distinct. This problem can be solved by the following straightforward algorithm. ALGORITHM UniqueElements(A[0..n − 1]) //Determines whether all the elements in a given array are distinct //Input: An array A[0..n − 1] //Output: Returns “true” if all the elements in A are distinct // and “false” otherwise for i ← 0 to n − 2 do for j ← i + 1 to n − 1 do if A[i] = A[j ] return false return true
The natural measure of the input’s size here is again n, the number of elements in the array. Since the innermost loop contains a single operation (the comparison of two elements), we should consider it as the algorithm’s basic operation. Note, however, that the number of element comparisons depends not only on n but also on whether there are equal elements in the array and, if there are, which array positions they occupy. We will limit our investigation to the worst case only. By deﬁnition, the worst case input is an array for which the number of element comparisons Cworst (n) is the largest among all arrays of size n. An inspection of the innermost loop reveals that there are two kinds of worstcase inputs—inputs for which the algorithm does not exit the loop prematurely: arrays with no equal elements and arrays in which the last two elements are the only pair of equal elements. For such inputs, one comparison is made for each repetition of the innermost loop, i.e., for each value of the loop variable j between its limits i + 1 and n − 1; this is repeated for each value of the outer loop, i.e., for each value of the loop variable i between its limits 0 and n − 2. Accordingly, we get
64
Fundamentals of the Analysis of Algorithm Efﬁciency
Cworst (n) =
n−1 n−2
1=
i=0 j =i+1
n−2 n−2
[(n − 1) − (i + 1) + 1] = (n − 1 − i) i=0
i=0
n−2 n−2 n−2
(n − 2)(n − 1) = (n − 1) − i = (n − 1) 1− 2 i=0 i=0 i=0
(n − 2)(n − 1) (n − 1)n 1 2 = ≈ n ∈ (n2 ). 2 2 2 n−2 We also could have computed the sum i=0 (n − 1 − i) faster as follows: = (n − 1)2 −
n−2
(n − 1)n , (n − 1 − i) = (n − 1) + (n − 2) + . . . + 1 = 2 i=0
where the last equality is obtained by applying summation formula (S2). Note that this result was perfectly predictable: in the worst case, the algorithm needs to compare all n(n − 1)/2 distinct pairs of its n elements. EXAMPLE 3 Given two n × n matrices A and B, ﬁnd the time efﬁciency of the deﬁnitionbased algorithm for computing their product C = AB. By deﬁnition, C is an n × n matrix whose elements are computed as the scalar (dot) products of the rows of matrix A and the columns of matrix B:
A
C
B *
= C [i, j]
row i col. j
where C[i, j ] = A[i, 0]B[0, j ] + . . . + A[i, k]B[k, j ] + . . . + A[i, n − 1]B[n − 1, j ] for every pair of indices 0 ≤ i, j ≤ n − 1. ALGORITHM MatrixMultiplication(A[0..n − 1, 0..n − 1], B[0..n − 1, 0..n − 1]) //Multiplies two square matrices of order n by the deﬁnitionbased algorithm //Input: Two n × n matrices A and B //Output: Matrix C = AB for i ← 0 to n − 1 do for j ← 0 to n − 1 do C[i, j ] ← 0.0 for k ← 0 to n − 1 do C[i, j ] ← C[i, j ] + A[i, k] ∗ B[k, j ] return C
2.3
Mathematical Analysis of Nonrecursive Algorithms
65
We measure an input’s size by matrix order n. There are two arithmetical operations in the innermost loop here—multiplication and addition—that, in principle, can compete for designation as the algorithm’s basic operation. Actually, we do not have to choose between them, because on each repetition of the innermost loop each of the two is executed exactly once. So by counting one we automatically count the other. Still, following a wellestablished tradition, we consider multiplication as the basic operation (see Section 2.1). Let us set up a sum for the total number of multiplications M(n) executed by the algorithm. (Since this count depends only on the size of the input matrices, we do not have to investigate the worstcase, averagecase, and bestcase efﬁciencies separately.) Obviously, there is just one multiplication executed on each repetition of the algorithm’s innermost loop, which is governed by the variable k ranging from the lower bound 0 to the upper bound n − 1. Therefore, the number of multiplications made for every pair of speciﬁc values of variables i and j is n−1
1,
k=0
and the total number of multiplications M(n) is expressed by the following triple sum: M(n) =
n−1 n−1 n−1
1.
i=0 j =0 k=0
Now, we can compute this sum by using formula (S1) and rule (R1) given above. Starting with the innermost sum n−1 k=0 1, which is equal to n (why?), we get M(n) =
n−1 n−1 n−1
i=0 j =0 k=0
1=
n−1 n−1
n=
i=0 j =0
n−1
n2 = n3.
i=0
This example is simple enough so that we could get this result without all the summation machinations. How? The algorithm computes n2 elements of the product matrix. Each of the product’s elements is computed as the scalar (dot) product of an nelement row of the ﬁrst matrix and an nelement column of the second matrix, which takes n multiplications. So the total number of multiplications is n . n2 = n3. (It is this kind of reasoning that we expected you to employ when answering this question in Problem 2 of Exercises 2.1.) If we now want to estimate the running time of the algorithm on a particular machine, we can do it by the product T (n) ≈ cmM(n) = cmn3, where cm is the time of one multiplication on the machine in question. We would get a more accurate estimate if we took into account the time spent on the additions, too: T (n) ≈ cmM(n) + ca A(n) = cmn3 + ca n3 = (cm + ca )n3,
66
Fundamentals of the Analysis of Algorithm Efﬁciency
where ca is the time of one addition. Note that the estimates differ only by their multiplicative constants and not by their order of growth. You should not have the erroneous impression that the plan outlined above always succeeds in analyzing a nonrecursive algorithm. An irregular change in a loop variable, a sum too complicated to analyze, and the difﬁculties intrinsic to the average case analysis are just some of the obstacles that can prove to be insurmountable. These caveats notwithstanding, the plan does work for many simple nonrecursive algorithms, as you will see throughout the subsequent chapters of the book. As a last example, let us consider an algorithm in which the loop’s variable changes in a different manner from that of the previous examples. EXAMPLE 4 The following algorithm ﬁnds the number of binary digits in the binary representation of a positive decimal integer. ALGORITHM Binary(n) //Input: A positive decimal integer n //Output: The number of binary digits in n’s binary representation count ← 1 while n > 1 do count ← count + 1 n ← n/2 return count First, notice that the most frequently executed operation here is not inside the while loop but rather the comparison n > 1 that determines whether the loop’s body will be executed. Since the number of times the comparison will be executed is larger than the number of repetitions of the loop’s body by exactly 1, the choice is not that important. A more signiﬁcant feature of this example is the fact that the loop variable takes on only a few values between its lower and upper limits; therefore, we have to use an alternative way of computing the number of times the loop is executed. Since the value of n is about halved on each repetition of the loop, the answer should be about log2 n. The exact formula for the number of times the comparison n > 1 will be executed is actually log2 n + 1—the number of bits in the binary representation of n according to formula (2.1). We could also get this answer by applying the analysis technique based on recurrence relations; we discuss this technique in the next section because it is more pertinent to the analysis of recursive algorithms.
2.3
Mathematical Analysis of Nonrecursive Algorithms
67
Exercises 2.3 1. Compute the following sums. a. 1 + 3 + 5 + 7 + . . . + 999 b. 2 + 4 + 8 + 16 + . . . + 1024 n+1 n+1 c. d. i i=3 1 n ni=3 n j +1 f. g. j =1 3 i=1 j =1 ij
n−1 i(i + 1) i=0 n h. i=1 1/ i(i + 1) e.
2. Find the order of growth of the following sums. Use the (g(n)) notation with the simplest function g(n) possible. n−1 2 2 n−1 2 a. b. i=0 (i +1) i=2 lg i n n−1 i−1 i−1 c. d. i=1(i + 1)2 i=0 j =0 (i + j ) 3. The sample variance of n measurements x1, . . . , xn can be computed as either n n xi ¯ 2 i=1(xi − x) where x¯ = i=1 n−1 n or n
2 i=1 xi
−(
n
2 i=1 xi ) /n
n−1
.
Find and compare the number of divisions, multiplications, and additions/ subtractions (additions and subtractions are usually bunched together) that are required for computing the variance according to each of these formulas. 4. Consider the following algorithm. ALGORITHM Mystery(n) //Input: A nonnegative integer n S←0 for i ← 1 to n do S←S+i∗i return S a. b. c. d. e.
What does this algorithm compute? What is its basic operation? How many times is the basic operation executed? What is the efﬁciency class of this algorithm? Suggest an improvement, or a better algorithm altogether, and indicate its efﬁciency class. If you cannot do it, try to prove that, in fact, it cannot be done.
68
Fundamentals of the Analysis of Algorithm Efﬁciency
5. Consider the following algorithm. ALGORITHM Secret(A[0..n − 1]) //Input: An array A[0..n − 1] of n real numbers minval ← A[0]; maxval ← A[0] for i ← 1 to n − 1 do if A[i] < minval minval ← A[i] if A[i] > maxval maxval ← A[i] return maxval − minval Answer questions (a)–(e) of Problem 4 about this algorithm. 6. Consider the following algorithm. ALGORITHM
Enigma(A[0..n − 1, 0..n − 1])
//Input: A matrix A[0..n − 1, 0..n − 1] of real numbers for i ← 0 to n − 2 do for j ← i + 1 to n − 1 do if A[i, j ] = A[j, i] return false return true Answer questions (a)–(e) of Problem 4 about this algorithm. 7. Improve the implementation of the matrix multiplication algorithm (see Example 3) by reducing the number of additions made by the algorithm. What effect will this change have on the algorithm’s efﬁciency? 8. Determine the asymptotic order of growth for the total number of times all the doors are toggled in the locker doors puzzle (Problem 12 in Exercises 1.1). 9. Prove the formula n
i=1
i =1+2 +...+n=
n(n + 1) 2
either by mathematical induction or by following the insight of a 10yearold school boy named Carl Friedrich Gauss (1777–1855) who grew up to become one of the greatest mathematicians of all times.
2.3
Mathematical Analysis of Nonrecursive Algorithms
69
10. Mental arithmetic A 10×10 table is ﬁlled with repeating numbers on its diagonals as shown below. Calculate the total sum of the table’s numbers in your head (after [Cra07, Question 1.33]).
2
2
3
…
3
9
10
9
10
11
9
10
11
9
10
11
9
10
11
9
10
11
9
10
11
9
10
11
9
10
11
10
11
…
3
…
1
17
…
17
17
18
18
19
11. Consider the following version of an important algorithm that we will study later in the book. ALGORITHM GE(A[0..n − 1, 0..n]) //Input: An n × (n + 1) matrix A[0..n − 1, 0..n] of real numbers for i ← 0 to n − 2 do for j ← i + 1 to n − 1 do for k ← i to n do A[j, k] ← A[j, k] − A[i, k] ∗ A[j, i] / A[i, i] a. Find the time efﬁciency class of this algorithm. b. What glaring inefﬁciency does this pseudocode contain and how can it be eliminated to speed the algorithm up? 12. von Neumann’s neighborhood Consider the algorithm that starts with a single square and on each of its n iterations adds new squares all around the outside. How many onebyone squares are there after n iterations? [Gar99] (In the parlance of cellular automata theory, the answer is the number of cells in the von Neumann neighborhood of range n.) The results for n = 0, 1, and 2 are illustrated below.
70
Fundamentals of the Analysis of Algorithm Efﬁciency
n=0
n=1
n=2
13. Page numbering Find the total number of decimal digits needed for numbering pages in a book of 1000 pages. Assume that the pages are numbered consecutively starting with 1.
2.4 Mathematical Analysis of Recursive Algorithms In this section, we will see how to apply the general framework for analysis of algorithms to recursive algorithms. We start with an example often used to introduce novices to the idea of a recursive algorithm. EXAMPLE 1 Compute the factorial function F (n) = n! for an arbitrary nonnegative integer n. Since n! = 1 . . . . . (n − 1) . n = (n − 1)! . n
for n ≥ 1
and 0! = 1 by deﬁnition, we can compute F (n) = F (n − 1) . n with the following recursive algorithm. ALGORITHM F(n) //Computes n! recursively //Input: A nonnegative integer n //Output: The value of n! if n = 0 return 1 else return F (n − 1) ∗ n For simplicity, we consider n itself as an indicator of this algorithm’s input size (rather than the number of bits in its binary expansion). The basic operation of the algorithm is multiplication,5 whose number of executions we denote M(n). Since the function F (n) is computed according to the formula F (n) = F (n − 1) . n 5.
for n > 0,
Alternatively, we could count the number of times the comparison n = 0 is executed, which is the same as counting the total number of calls made by the algorithm (see Problem 2 in this section’s exercises).
2.4
71
Mathematical Analysis of Recursive Algorithms
the number of multiplications M(n) needed to compute it must satisfy the equality M(n) = M(n − 1) + to compute F (n−1)
1
for n > 0.
to multiply F (n−1) by n
Indeed, M(n − 1) multiplications are spent to compute F (n − 1), and one more multiplication is needed to multiply the result by n. The last equation deﬁnes the sequence M(n) that we need to ﬁnd. This equation deﬁnes M(n) not explicitly, i.e., as a function of n, but implicitly as a function of its value at another point, namely n − 1. Such equations are called recurrence relations or, for brevity, recurrences. Recurrence relations play an important role not only in analysis of algorithms but also in some areas of applied mathematics. They are usually studied in detail in courses on discrete mathematics or discrete structures; a very brief tutorial on them is provided in Appendix B. Our goal now is to solve the recurrence relation M(n) = M(n − 1) + 1, i.e., to ﬁnd an explicit formula for M(n) in terms of n only. Note, however, that there is not one but inﬁnitely many sequences that satisfy this recurrence. (Can you give examples of, say, two of them?) To determine a solution uniquely, we need an initial condition that tells us the value with which the sequence starts. We can obtain this value by inspecting the condition that makes the algorithm stop its recursive calls: if n = 0 return 1. This tells us two things. First, since the calls stop when n = 0, the smallest value of n for which this algorithm is executed and hence M(n) deﬁned is 0. Second, by inspecting the pseudocode’s exiting line, we can see that when n = 0, the algorithm performs no multiplications. Therefore, the initial condition we are after is M(0) = 0. the calls stop when n = 0
no multiplications when n = 0
Thus, we succeeded in setting up the recurrence relation and initial condition for the algorithm’s number of multiplications M(n): M(n) = M(n − 1) + 1 for n > 0, M(0) = 0.
(2.2)
Before we embark on a discussion of how to solve this recurrence, let us pause to reiterate an important point. We are dealing here with two recursively deﬁned functions. The ﬁrst is the factorial function F (n) itself; it is deﬁned by the recurrence F (n) = F (n − 1) . n for every n > 0, F (0) = 1. The second is the number of multiplications M(n) needed to compute F (n) by the recursive algorithm whose pseudocode was given at the beginning of the section.
72
Fundamentals of the Analysis of Algorithm Efﬁciency
As we just showed, M(n) is deﬁned by recurrence (2.2). And it is recurrence (2.2) that we need to solve now. Though it is not difﬁcult to “guess” the solution here (what sequence starts with 0 when n = 0 and increases by 1 on each step?), it will be more useful to arrive at it in a systematic fashion. From the several techniques available for solving recurrence relations, we use what can be called the method of backward substitutions. The method’s idea (and the reason for the name) is immediately clear from the way it applies to solving our particular recurrence: M(n) = M(n − 1) + 1
substitute M(n − 1) = M(n − 2) + 1
= [M(n − 2) + 1] + 1 = M(n − 2) + 2
substitute M(n − 2) = M(n − 3) + 1
= [M(n − 3) + 1] + 2 = M(n − 3) + 3. After inspecting the ﬁrst three lines, we see an emerging pattern, which makes it possible to predict not only the next line (what would it be?) but also a general formula for the pattern: M(n) = M(n − i) + i. Strictly speaking, the correctness of this formula should be proved by mathematical induction, but it is easier to get to the solution as follows and then verify its correctness. What remains to be done is to take advantage of the initial condition given. Since it is speciﬁed for n = 0, we have to substitute i = n in the pattern’s formula to get the ultimate result of our backward substitutions: M(n) = M(n − 1) + 1 = . . . = M(n − i) + i = . . . = M(n − n) + n = n. You should not be disappointed after exerting so much effort to get this “obvious” answer. The beneﬁts of the method illustrated in this simple example will become clear very soon, when we have to solve more difﬁcult recurrences. Also, note that the simple iterative algorithm that accumulates the product of n consecutive integers requires the same number of multiplications, and it does so without the overhead of time and space used for maintaining the recursion’s stack. The issue of time efﬁciency is actually not that important for the problem of computing n!, however. As we saw in Section 2.1, the function’s values get so large so fast that we can realistically compute exact values of n! only for very small n’s. Again, we use this example just as a simple and convenient vehicle to introduce the standard approach to analyzing recursive algorithms. Generalizing our experience with investigating the recursive algorithm for computing n!, we can now outline a general plan for investigating recursive algorithms. General Plan for Analyzing the Time Efﬁciency of Recursive Algorithms 1. 2.
Decide on a parameter (or parameters) indicating an input’s size. Identify the algorithm’s basic operation.
2.4
3.
4. 5.
Mathematical Analysis of Recursive Algorithms
73
Check whether the number of times the basic operation is executed can vary on different inputs of the same size; if it can, the worstcase, averagecase, and bestcase efﬁciencies must be investigated separately. Set up a recurrence relation, with an appropriate initial condition, for the number of times the basic operation is executed. Solve the recurrence or, at least, ascertain the order of growth of its solution.
EXAMPLE 2 As our next example, we consider another educational workhorse of recursive algorithms: the Tower of Hanoi puzzle. In this puzzle, we (or mythical monks, if you do not like to move disks) have n disks of different sizes that can slide onto any of three pegs. Initially, all the disks are on the ﬁrst peg in order of size, the largest on the bottom and the smallest on top. The goal is to move all the disks to the third peg, using the second one as an auxiliary, if necessary. We can move only one disk at a time, and it is forbidden to place a larger disk on top of a smaller one. The problem has an elegant recursive solution, which is illustrated in Figure 2.4. To move n > 1 disks from peg 1 to peg 3 (with peg 2 as auxiliary), we ﬁrst move recursively n − 1 disks from peg 1 to peg 2 (with peg 3 as auxiliary), then move the largest disk directly from peg 1 to peg 3, and, ﬁnally, move recursively n − 1 disks from peg 2 to peg 3 (using peg 1 as auxiliary). Of course, if n = 1, we simply move the single disk directly from the source peg to the destination peg.
1
3
2
FIGURE 2.4 Recursive solution to the Tower of Hanoi puzzle.
74
Fundamentals of the Analysis of Algorithm Efﬁciency
Let us apply the general plan outlined above to the Tower of Hanoi problem. The number of disks n is the obvious choice for the input’s size indicator, and so is moving one disk as the algorithm’s basic operation. Clearly, the number of moves M(n) depends on n only, and we get the following recurrence equation for it: M(n) = M(n − 1) + 1 + M(n − 1)
for n > 1.
With the obvious initial condition M(1) = 1, we have the following recurrence relation for the number of moves M(n): M(n) = 2M(n − 1) + 1 for n > 1, M(1) = 1.
(2.3)
We solve this recurrence by the same method of backward substitutions: M(n) = 2M(n − 1) + 1
sub. M(n − 1) = 2M(n − 2) + 1
= 2[2M(n − 2) + 1] + 1 = 2 M(n − 2) + 2 + 1 sub. M(n − 2) = 2M(n − 3) + 1 2
= 22 [2M(n − 3) + 1] + 2 + 1 = 23M(n − 3) + 22 + 2 + 1. The pattern of the ﬁrst three sums on the left suggests that the next one will be 24M(n − 4) + 23 + 22 + 2 + 1, and generally, after i substitutions, we get M(n) = 2i M(n − i) + 2i−1 + 2i−2 + . . . + 2 + 1 = 2i M(n − i) + 2i − 1. Since the initial condition is speciﬁed for n = 1, which is achieved for i = n − 1, we get the following formula for the solution to recurrence (2.3): M(n) = 2n−1M(n − (n − 1)) + 2n−1 − 1 = 2n−1M(1) + 2n−1 − 1 = 2n−1 + 2n−1 − 1 = 2n − 1. Thus, we have an exponential algorithm, which will run for an unimaginably long time even for moderate values of n (see Problem 5 in this section’s exercises). This is not due to the fact that this particular algorithm is poor; in fact, it is not difﬁcult to prove that this is the most efﬁcient algorithm possible for this problem. It is the problem’s intrinsic difﬁculty that makes it so computationally hard. Still, this example makes an important general point: One should be careful with recursive algorithms because their succinctness may mask their inefﬁciency. When a recursive algorithm makes more than a single call to itself, it can be useful for analysis purposes to construct a tree of its recursive calls. In this tree, nodes correspond to recursive calls, and we can label them with the value of the parameter (or, more generally, parameters) of the calls. For the Tower of Hanoi example, the tree is given in Figure 2.5. By counting the number of nodes in the tree, we can get the total number of calls made by the Tower of Hanoi algorithm: C(n) =
n−1
l=0
2l (where l is the level in the tree in Figure 2.5) = 2n − 1.
2.4
75
Mathematical Analysis of Recursive Algorithms n n–1
n–1 n–2 2 1
n–2
n–2
2 1
1
n–2 2
1
1
2 1
1
1
FIGURE 2.5 Tree of recursive calls made by the recursive algorithm for the Tower of Hanoi puzzle.
The number agrees, as it should, with the move count obtained earlier. EXAMPLE 3 As our next example, we investigate a recursive version of the algorithm discussed at the end of Section 2.3. ALGORITHM BinRec(n) //Input: A positive decimal integer n //Output: The number of binary digits in n’s binary representation if n = 1 return 1 else return BinRec(n/2) + 1 Let us set up a recurrence and an initial condition for the number of additions A(n) made by the algorithm. The number of additions made in computing BinRec(n/2) is A(n/2), plus one more addition is made by the algorithm to increase the returned value by 1. This leads to the recurrence A(n) = A(n/2) + 1 for n > 1.
(2.4)
Since the recursive calls end when n is equal to 1 and there are no additions made then, the initial condition is A(1) = 0. The presence of n/2 in the function’s argument makes the method of backward substitutions stumble on values of n that are not powers of 2. Therefore, the standard approach to solving such a recurrence is to solve it only for n = 2k and then take advantage of the theorem called the smoothness rule (see Appendix B), which claims that under very broad assumptions the order of growth observed for n = 2k gives a correct answer about the order of growth for all values of n. (Alternatively, after getting a solution for powers of 2, we can sometimes ﬁnetune this solution to get a formula valid for an arbitrary n.) So let us apply this recipe to our recurrence, which for n = 2k takes the form
76
Fundamentals of the Analysis of Algorithm Efﬁciency
A(2k ) = A(2k−1) + 1 for k > 0, A(20) = 0. Now backward substitutions encounter no problems: A(2k ) = A(2k−1) + 1
substitute A(2k−1) = A(2k−2 ) + 1
= [A(2k−2 ) + 1] + 1 = A(2k−2 ) + 2
substitute A(2k−2 ) = A(2k−3) + 1
= [A(2k−3) + 1] + 2 = A(2k−3) + 3 ... ...
...
= A(2k−i ) + i = A(2k−k ) + k.
Thus, we end up with A(2k ) = A(1) + k = k, or, after returning to the original variable n = 2k and hence k = log2 n, A(n) = log2 n ∈ (log n). In fact, one can prove (Problem 7 in this section’s exercises) that the exact solution for an arbitrary value of n is given by just a slightly more reﬁned formula A(n) = log2 n. This section provides an introduction to the analysis of recursive algorithms. These techniques will be used throughout the book and expanded further as necessary. In the next section, we discuss the Fibonacci numbers; their analysis involves more difﬁcult recurrence relations to be solved by a method different from backward substitutions.
Exercises 2.4 1. Solve the following recurrence relations. a. x(n) = x(n − 1) + 5 for n > 1, x(1) = 0 b. x(n) = 3x(n − 1) for n > 1, x(1) = 4 c. x(n) = x(n − 1) + n for n > 0, x(0) = 0 d. x(n) = x(n/2) + n for n > 1, x(1) = 1 (solve for n = 2k ) e. x(n) = x(n/3) + 1 for n > 1,
x(1) = 1 (solve for n = 3k )
2. Set up and solve a recurrence relation for the number of calls made by F (n), the recursive algorithm for computing n!. 3. Consider the following recursive algorithm for computing the sum of the ﬁrst n cubes: S(n) = 13 + 23 + . . . + n3.
2.4
Mathematical Analysis of Recursive Algorithms
ALGORITHM
77
S(n)
//Input: A positive integer n //Output: The sum of the ﬁrst n cubes if n = 1 return 1 else return S(n − 1) + n ∗ n ∗ n a. Set up and solve a recurrence relation for the number of times the algorithm’s basic operation is executed. b. How does this algorithm compare with the straightforward nonrecursive algorithm for computing this sum? 4. Consider the following recursive algorithm. ALGORITHM Q(n) //Input: A positive integer n if n = 1 return 1 else return Q(n − 1) + 2 ∗ n − 1 a. Set up a recurrence relation for this function’s values and solve it to determine what this algorithm computes. b. Set up a recurrence relation for the number of multiplications made by this algorithm and solve it. c. Set up a recurrence relation for the number of additions/subtractions made by this algorithm and solve it. 5. Tower of Hanoi a. In the original version of the Tower of Hanoi puzzle, as it was published in ´ the 1890s by Edouard Lucas, a French mathematician, the world will end after 64 disks have been moved from a mystical Tower of Brahma. Estimate the number of years it will take if monks could move one disk per minute. (Assume that monks do not eat, sleep, or die.) b. How many moves are made by the ith largest disk (1 ≤ i ≤ n) in this algorithm? c. Find a nonrecursive algorithm for the Tower of Hanoi puzzle and implement it in the language of your choice. 6. Restricted Tower of Hanoi Consider the version of the Tower of Hanoi puzzle in which n disks have to be moved from peg A to peg C using peg B so that any move should either place a disk on peg B or move a disk from that peg. (Of course, the prohibition of placing a larger disk on top of a smaller one remains in place, too.) Design a recursive algorithm for this problem and ﬁnd the number of moves made by it.
78
Fundamentals of the Analysis of Algorithm Efﬁciency
7. a. Prove that the exact number of additions made by the recursive algorithm BinRec(n) for an arbitrary positive decimal integer n is log2 n. b. Set up a recurrence relation for the number of additions made by the nonrecursive version of this algorithm (see Section 2.3, Example 4) and solve it. 8. a. Design a recursive algorithm for computing 2n for any nonnegative integer n that is based on the formula 2n = 2n−1 + 2n−1. b. Set up a recurrence relation for the number of additions made by the algorithm and solve it. c. Draw a tree of recursive calls for this algorithm and count the number of calls made by the algorithm. d. Is it a good algorithm for solving this problem? 9. Consider the following recursive algorithm. ALGORITHM Riddle(A[0..n − 1]) //Input: An array A[0..n − 1] of real numbers if n = 1 return A[0] else temp ← Riddle(A[0..n − 2]) if temp ≤ A[n − 1] return temp else return A[n − 1] a. What does this algorithm compute? b. Set up a recurrence relation for the algorithm’s basic operation count and solve it. 10. Consider the following algorithm to check whether a graph deﬁned by its adjacency matrix is complete. ALGORITHM GraphComplete(A[0..n − 1, 0..n − 1]) //Input: Adjacency matrix A[0..n − 1, 0..n − 1]) of an undirected graph G //Output: 1 (true) if G is complete and 0 (false) otherwise if n = 1 return 1 //onevertex graph is complete by deﬁnition else if not GraphComplete(A[0..n − 2, 0..n − 2]) return 0 else for j ← 0 to n − 2 do if A[n − 1, j ] = 0 return 0 return 1 What is the algorithm’s efﬁciency class in the worst case? 11. The determinant of an n × n matrix
2.4
Mathematical Analysis of Recursive Algorithms
⎡
a0 0 ⎢ a1 0 A=⎢ ⎣ ... an−1 0
... ... ...
79
⎤ a0 n−1 a1 n−1 ⎥ ⎥, .. ⎦ . an−1 n−1
denoted det A, can be deﬁned as a00 for n = 1 and, for n > 1, by the recursive formula det A =
n−1
sj a0 j det Aj ,
j =0
where sj is +1 if j is even and −1 if j is odd, a0 j is the element in row 0 and column j , and Aj is the (n − 1) × (n − 1) matrix obtained from matrix A by deleting its row 0 and column j . a. Set up a recurrence relation for the number of multiplications made by the algorithm implementing this recursive deﬁnition. b. Without solving the recurrence, what can you say about the solution’s order of growth as compared to n!? 12. von Neumann’s neighborhood revisited Find the number of cells in the von Neumann neighborhood of range n (Problem 12 in Exercises 2.3) by setting up and solving a recurrence relation. 13. Frying hamburgers There are n hamburgers to be fried on a small grill that can hold only two hamburgers at a time. Each hamburger has to be fried on both sides; frying one side of a hamburger takes 1 minute, regardless of whether one or two hamburgers are fried at the same time. Consider the following recursive algorithm for executing this task in the minimum amount of time. If n ≤ 2, fry the hamburger or the two hamburgers together on each side. If n > 2, fry any two hamburgers together on each side and then apply the same procedure recursively to the remaining n − 2 hamburgers. a. Set up and solve the recurrence for the amount of time this algorithm needs to fry n hamburgers. b. Explain why this algorithm does not fry the hamburgers in the minimum amount of time for all n > 0. c. Give a correct recursive algorithm that executes the task in the minimum amount of time. 14. Celebrity problem A celebrity among a group of n people is a person who knows nobody but is known by everybody else. The task is to identify a celebrity by only asking questions to people of the form “Do you know him/her?” Design an efﬁcient algorithm to identify a celebrity or determine that the group has no such person. How many questions does your algorithm need in the worst case?
80
Fundamentals of the Analysis of Algorithm Efﬁciency
2.5 Example: Computing the nth Fibonacci Number In this section, we consider the Fibonacci numbers, a famous sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .
(2.5)
that can be deﬁned by the simple recurrence F (n) = F (n − 1) + F (n − 2)
for n > 1
(2.6)
and two initial conditions F (0) = 0,
F (1) = 1.
(2.7)
The Fibonacci numbers were introduced by Leonardo Fibonacci in 1202 as a solution to a problem about the size of a rabbit population (Problem 2 in this section’s exercises). Many more examples of Fibonaccilike numbers have since been discovered in the natural world, and they have even been used in predicting the prices of stocks and commodities. There are some interesting applications of the Fibonacci numbers in computer science as well. For example, worstcase inputs for Euclid’s algorithm discussed in Section 1.1 happen to be consecutive elements of the Fibonacci sequence. In this section, we brieﬂy consider algorithms for computing the nth element of this sequence. Among other beneﬁts, the discussion will provide us with an opportunity to introduce another method for solving recurrence relations useful for analysis of recursive algorithms. To start, let us get an explicit formula for F (n). If we try to apply the method of backward substitutions to solve recurrence (2.6), we will fail to get an easily discernible pattern. Instead, we can take advantage of a theorem that describes solutions to a homogeneous secondorder linear recurrence with constant coefﬁcients ax(n) + bx(n − 1) + cx(n − 2) = 0,
(2.8)
where a, b, and c are some ﬁxed real numbers (a = 0) called the coefﬁcients of the recurrence and x(n) is the generic term of an unknown sequence to be found. Applying this theorem to our recurrence with the initial conditions given—see Appendix B—we obtain the formula 1 F (n) = √ (φ n − φˆ n), 5
(2.9)
√ where φ = (1 + 5)/2 ≈ 1.61803 and φˆ = −1/φ ≈ −0.61803.6 It is hard to believe that formula (2.9), which includes arbitrary integer powers of irrational numbers, yields nothing else but all the elements of Fibonacci sequence (2.5), but it does! One of the beneﬁts of formula (2.9) is that it immediately implies that F (n) grows exponentially (remember Fibonacci’s rabbits?), i.e., F (n) ∈ (φ n). This 6.
Constant φ is known as the golden ratio. Since antiquity, it has been considered the most pleasing ratio of a rectangle’s two sides to the human eye and might have been consciously used by ancient architects and sculptors.
81
2.5 Example: Computing the nth Fibonacci Number
follows from the observation that φˆ is a fraction between −1 and 0, and hence φˆ n gets inﬁnitely small as n goes to inﬁnity. In fact, one can prove that the impact of the second term √1 φˆ n on the value of F (n) can be obtained by rounding off the 5 value of the ﬁrst term to the nearest integer. In other words, for every nonnegative integer n, 1 F (n) = √ φ n 5
rounded to the nearest integer.
(2.10)
In the algorithms that follow, we consider, for the sake of simplicity, such operations as additions and multiplications at unit cost. Since the Fibonacci numbers grow inﬁnitely large (and grow very rapidly), a more detailed analysis than the one offered here is warranted. In fact, it is the size of the numbers rather than a timeefﬁcient method for computing them that should be of primary concern here. Still, these caveats notwithstanding, the algorithms we outline and their analysis provide useful examples for a student of the design and analysis of algorithms. To begin with, we can use recurrence (2.6) and initial conditions (2.7) for the obvious recursive algorithm for computing F (n). ALGORITHM F (n) //Computes the nth Fibonacci number recursively by using its deﬁnition //Input: A nonnegative integer n //Output: The nth Fibonacci number if n ≤ 1 return n else return F (n − 1) + F (n − 2) Before embarking on its formal analysis, can you tell whether this is an efﬁcient algorithm? Well, we need to do a formal analysis anyway. The algorithm’s basic operation is clearly addition, so let A(n) be the number of additions performed by the algorithm in computing F (n). Then the numbers of additions needed for computing F (n − 1) and F (n − 2) are A(n − 1) and A(n − 2), respectively, and the algorithm needs one more addition to compute their sum. Thus, we get the following recurrence for A(n): A(n) = A(n − 1) + A(n − 2) + 1 for n > 1, A(0) = 0, A(1) = 0.
(2.11)
The recurrence A(n) − A(n − 1) − A(n − 2) = 1 is quite similar to recurrence F (n) − F (n − 1) − F (n − 2) = 0, but its righthand side is not equal to zero. Such recurrences are called inhomogeneous. There are general techniques for solving inhomogeneous recurrences (see Appendix B or any textbook on discrete mathematics), but for this particular recurrence, a special trick leads to a faster solution. We can reduce our inhomogeneous recurrence to a homogeneous one by rewriting it as [A(n) + 1] − [A(n − 1) + 1] − [A(n − 2) + 1] = 0 and substituting B(n) = A(n) + 1:
82
Fundamentals of the Analysis of Algorithm Efﬁciency
B(n) − B(n − 1) − B(n − 2) = 0, B(0) = 1, B(1) = 1. This homogeneous recurrence can be solved exactly in the same manner as recurrence (2.6) was solved to ﬁnd an explicit formula for F (n). But it can actually be avoided by noting that B(n) is, in fact, the same recurrence as F (n) except that it starts with two 1’s and thus runs one step ahead of F (n). So B(n) = F (n + 1), and 1 A(n) = B(n) − 1 = F (n + 1) − 1 = √ (φ n+1 − φˆ n+1) − 1. 5 Hence, A(n) ∈ (φ n), and if we measure the size of n by the number of bits b = log2 n + 1 in its binary representation, the efﬁciency class will be even worse, b namely, doubly exponential: A(b) ∈ (φ 2 ). The poor efﬁciency class of the algorithm could be anticipated by the nature of recurrence (2.11). Indeed, it contains two recursive calls with the sizes of smaller instances only slightly smaller than size n. (Have you encountered such a situation before?) We can also see the reason behind the algorithm’s inefﬁciency by looking at a recursive tree of calls tracing the algorithm’s execution. An example of such a tree for n = 5 is given in Figure 2.6. Note that the same values of the function are being evaluated here again and again, which is clearly extremely inefﬁcient. We can obtain a much faster algorithm by simply computing the successive elements of the Fibonacci sequence iteratively, as is done in the following algorithm. ALGORITHM Fib(n) //Computes the nth Fibonacci number iteratively by using its deﬁnition //Input: A nonnegative integer n //Output: The nth Fibonacci number F [0] ← 0; F [1] ← 1 for i ← 2 to n do F [i] ← F [i − 1] + F [i − 2] return F [n]
F (5) F (3)
F (4) F (3) F (2) F (1)
F (2) F (1)
F (1)
F (2) F (0)
F (1)
F (1) F (0)
F (0)
FIGURE 2.6 Tree of recursive calls for computing the 5th Fibonacci number by the deﬁnitionbased algorithm.
2.5 Example: Computing the nth Fibonacci Number
83
This algorithm clearly makes n − 1 additions. Hence, it is linear as a function of n and “only” exponential as a function of the number of bits b in n’s binary representation. Note that using an extra array for storing all the preceding elements of the Fibonacci sequence can be avoided: storing just two values is necessary to accomplish the task (see Problem 8 in this section’s exercises). The third alternative for computing the nth Fibonacci number lies in using formula (2.10). The efﬁciency of the algorithm will obviously be determined by the efﬁciency of an exponentiation algorithm used for computing φ n. If it is done by simply multiplying φ by itself n − 1 times, the algorithm will be in (n) = (2b ). There are faster algorithms for the exponentiation problem. For example, we will discuss (log n) = (b) algorithms for this problem in Chapters 4 and 6. Note also that special care should be exercised in implementing this approach to computing the nth Fibonacci number. Since all its intermediate results are irrational numbers, we would have to make sure that their approximations in the computer are accurate enough so that the ﬁnal roundoff yields a correct result. Finally, there exists a (log n) algorithm for computing the nth Fibonacci number that manipulates only integers. It is based on the equality n F (n − 1) F (n) 0 1 = for n ≥ 1 1 1 F (n) F (n + 1) and an efﬁcient way of computing matrix powers.
Exercises 2.5 1. Find a Web site dedicated to applications of the Fibonacci numbers and study it. 2. Fibonacci’s rabbits problem A man put a pair of rabbits in a place surrounded by a wall. How many pairs of rabbits will be there in a year if the initial pair of rabbits (male and female) are newborn and all rabbit pairs are not fertile during their ﬁrst month of life but thereafter give birth to one new male/female pair at the end of every month? 3. Climbing stairs Find the number of different ways to climb an nstair staircase if each step is either one or two stairs. For example, a 3stair staircase can be climbed three ways: 111, 12, and 21. 4. How many even numbers are there among the ﬁrst n Fibonacci numbers, i.e., among the numbers F (0), F (1), . . . , F (n − 1)? Give a closedform formula valid for every n > 0. 5. Check by direct substitutions that the function recurrence (2.6) and initial conditions (2.7).
√1 (φ n 5
− φˆ n) indeed satisﬁes
6. The maximum values of the Java primitive types int and long are 231 − 1 and 263 − 1, respectively. Find the smallest n for which the nth Fibonacci number is not going to ﬁt in a memory allocated for
84
Fundamentals of the Analysis of Algorithm Efﬁciency
a. the type int.
b. the type long.
7. Consider the recursive deﬁnitionbased algorithm for computing the nth Fibonacci number F (n). Let C(n) and Z(n) be the number of times F (1) and F (0) are computed, respectively. Prove that a. C(n) = F (n). b. Z(n) = F (n − 1). 8. Improve algorithm F ib of the text so that it requires only (1) space. 9. Prove the equality n F (n − 1) F (n) 0 1 = F (n) F (n + 1) 1 1
for n ≥ 1.
10. How many modulo divisions are made by Euclid’s algorithm on two consecutive Fibonacci numbers F (n) and F (n − 1) as the algorithm’s input? 11. Dissecting a Fibonacci rectangle Given a rectangle whose sides are two consecutive Fibonacci numbers, design an algorithm to dissect it into squares with no more than two squares being the same size. What is the time efﬁciency class of your algorithm? 12. In the language of your choice, implement two algorithms for computing the last ﬁve digits of the nth Fibonacci number that are based on (a) the recursive deﬁnitionbased algorithm F(n); (b) the iterative deﬁnitionbased algorithm Fib(n). Perform an experiment to ﬁnd the largest value of n for which your programs run under 1 minute on your computer.
2.6 Empirical Analysis of Algorithms In Sections 2.3 and 2.4, we saw how algorithms, both nonrecursive and recursive, can be analyzed mathematically. Though these techniques can be applied successfully to many simple algorithms, the power of mathematics, even when enhanced with more advanced techniques (see [Sed96], [Pur04], [Gra94], and [Gre07]), is far from limitless. In fact, even some seemingly simple algorithms have proved to be very difﬁcult to analyze with mathematical precision and certainty. As we pointed out in Section 2.1, this is especially true for the averagecase analysis. The principal alternative to the mathematical analysis of an algorithm’s efﬁciency is its empirical analysis. This approach implies steps spelled out in the following plan. General Plan for the Empirical Analysis of Algorithm Time Efﬁciency 1. Understand the experiment’s purpose. 2. Decide on the efﬁciency metric M to be measured and the measurement unit (an operation count vs. a time unit). 3. Decide on characteristics of the input sample (its range, size, and so on). 4. Prepare a program implementing the algorithm (or algorithms) for the experimentation.
2.6 Empirical Analysis of Algorithms
5. 6. 7.
85
Generate a sample of inputs. Run the algorithm (or algorithms) on the sample’s inputs and record the data observed. Analyze the data obtained.
Let us discuss these steps one at a time. There are several different goals one can pursue in analyzing algorithms empirically. They include checking the accuracy of a theoretical assertion about the algorithm’s efﬁciency, comparing the efﬁciency of several algorithms for solving the same problem or different implementations of the same algorithm, developing a hypothesis about the algorithm’s efﬁciency class, and ascertaining the efﬁciency of the program implementing the algorithm on a particular machine. Obviously, an experiment’s design should depend on the question the experimenter seeks to answer. In particular, the goal of the experiment should inﬂuence, if not dictate, how the algorithm’s efﬁciency is to be measured. The ﬁrst alternative is to insert a counter (or counters) into a program implementing the algorithm to count the number of times the algorithm’s basic operation is executed. This is usually a straightforward operation; you should only be mindful of the possibility that the basic operation is executed in several places in the program and that all its executions need to be accounted for. As straightforward as this task usually is, you should always test the modiﬁed program to ensure that it works correctly, in terms of both the problem it solves and the counts it yields. The second alternative is to time the program implementing the algorithm in question. The easiest way to do this is to use a system’s command, such as the time command in UNIX. Alternatively, one can measure the running time of a code fragment by asking for the system time right before the fragment’s start (tstart ) and just after its completion (tﬁnish), and then computing the difference between the two (tﬁnish− tstart ).7 In C and C++, you can use the function clock for this purpose; in Java, the method currentTimeMillis() in the System class is available. It is important to keep several facts in mind, however. First, a system’s time is typically not very accurate, and you might get somewhat different results on repeated runs of the same program on the same inputs. An obvious remedy is to make several such measurements and then take their average (or the median) as the sample’s observation point. Second, given the high speed of modern computers, the running time may fail to register at all and be reported as zero. The standard trick to overcome this obstacle is to run the program in an extra loop many times, measure the total running time, and then divide it by the number of the loop’s repetitions. Third, on a computer running under a timesharing system such as UNIX, the reported time may include the time spent by the CPU on other programs, which obviously defeats the purpose of the experiment. Therefore, you should take care to ask the system for the time devoted speciﬁcally to execution of
7.
If the system time is given in units called “ticks,” the difference should be divided by a constant indicating the number of ticks per time unit.
86
Fundamentals of the Analysis of Algorithm Efﬁciency
your program. (In UNIX, this time is called the “user time,” and it is automatically provided by the time command.) Thus, measuring the physical running time has several disadvantages, both principal (dependence on a particular machine being the most important of them) and technical, not shared by counting the executions of a basic operation. On the other hand, the physical running time provides very speciﬁc information about an algorithm’s performance in a particular computing environment, which can be of more importance to the experimenter than, say, the algorithm’s asymptotic efﬁciency class. In addition, measuring time spent on different segments of a program can pinpoint a bottleneck in the program’s performance that can be missed by an abstract deliberation about the algorithm’s basic operation. Getting such data—called proﬁling—is an important resource in the empirical analysis of an algorithm’s running time; the data in question can usually be obtained from the system tools available in most computing environments. Whether you decide to measure the efﬁciency by basic operation counting or by time clocking, you will need to decide on a sample of inputs for the experiment. Often, the goal is to use a sample representing a “typical” input; so the challenge is to understand what a “typical” input is. For some classes of algorithms—e.g., for algorithms for the traveling salesman problem that we are going to discuss later in the book—researchers have developed a set of instances they use for benchmarking. But much more often than not, an input sample has to be developed by the experimenter. Typically, you will have to make decisions about the sample size (it is sensible to start with a relatively small sample and increase it later if necessary), the range of instance sizes (typically neither trivially small nor excessively large), and a procedure for generating instances in the range chosen. The instance sizes can either adhere to some pattern (e.g., 1000, 2000, 3000, . . . , 10,000 or 500, 1000, 2000, 4000, . . . , 128,000) or be generated randomly within the range chosen. The principal advantage of size changing according to a pattern is that its impact is easier to analyze. For example, if a sample’s sizes are generated by doubling, you can compute the ratios M(2n)/M(n) of the observed metric M (the count or the time) to see whether the ratios exhibit a behavior typical of algorithms in one of the basic efﬁciency classes discussed in Section 2.2. The major disadvantage of nonrandom sizes is the possibility that the algorithm under investigation exhibits atypical behavior on the sample chosen. For example, if all the sizes in a sample are even and your algorithm runs much more slowly on oddsize inputs, the empirical results will be quite misleading. Another important issue concerning sizes in an experiment’s sample is whether several instances of the same size should be included. If you expect the observed metric to vary considerably on instances of the same size, it would be probably wise to include several instances for every size in the sample. (There are welldeveloped methods in statistics to help the experimenter make such decisions; you will ﬁnd no shortage of books on this subject.) Of course, if several instances of the same size are included in the sample, the averages or medians of the observed values for each size should be computed and investigated instead of or in addition to individual sample points.
2.6 Empirical Analysis of Algorithms
87
Much more often than not, an empirical analysis requires generating random numbers. Even if you decide to use a pattern for input sizes, you will typically want instances themselves generated randomly. Generating random numbers on a digital computer is known to present a difﬁcult problem because, in principle, the problem can be solved only approximately. This is the reason computer scientists prefer to call such numbers pseudorandom. As a practical matter, the easiest and most natural way of getting such numbers is to take advantage of a random number generator available in computer language libraries. Typically, its output will be a value of a (pseudo)random variable uniformly distributed in the interval between 0 and 1. If a different (pseudo)random variable is desired, an appropriate transformation needs to be made. For example, if x is a continuous random variable uniformly distributed on the interval 0 ≤ x < 1, the variable y = l+ x(r − l) will be uniformly distributed among the integer values between integers l and r − 1 (l < r). Alternatively, you can implement one of several known algorithms for generating (pseudo)random numbers. The most widely used and thoroughly studied of such algorithms is the linear congruential method. ALGORITHM Random(n, m, seed, a, b) //Generates a sequence of n pseudorandom numbers according to the linear // congruential method //Input: A positive integer n and positive integer parameters m, seed, a, b //Output: A sequence r1, . . . , rn of n pseudorandom integers uniformly // distributed among integer values between 0 and m − 1 //Note: Pseudorandom numbers between 0 and 1 can be obtained // by treating the integers generated as digits after the decimal point r0 ← seed for i ← 1 to n do ri ← (a ∗ ri−1 + b) mod m The simplicity of this pseudocode is misleading because the devil lies in the details of choosing the algorithm’s parameters. Here is a partial list of recommendations based on the results of a sophisticated mathematical analysis (see [KnuII, pp. 184–185] for details): seed may be chosen arbitrarily and is often set to the current date and time; m should be large and may be conveniently taken as 2w , where w is the computer’s word size; a should be selected as an integer between 0.01m and 0.99m with no particular pattern in its digits but such that a mod 8 = 5; and the value of b can be chosen as 1. The empirical data obtained as the result of an experiment need to be recorded and then presented for an analysis. Data can be presented numerically in a table or graphically in a scatterplot, i.e., by points in a Cartesian coordinate system. It is a good idea to use both these options whenever it is feasible because both methods have their unique strengths and weaknesses.
88
Fundamentals of the Analysis of Algorithm Efﬁciency
The principal advantage of tabulated data lies in the opportunity to manipulate it easily. For example, one can compute the ratios M(n)/g(n) where g(n) is a candidate to represent the efﬁciency class of the algorithm in question. If the algorithm is indeed in (g(n)), most likely these ratios will converge to some positive constant as n gets large. (Note that careless novices sometimes assume that this constant must be 1, which is, of course, incorrect according to the deﬁnition of (g(n)).) Or one can compute the ratios M(2n)/M(n) and see how the running time reacts to doubling of its input size. As we discussed in Section 2.2, such ratios should change only slightly for logarithmic algorithms and most likely converge to 2, 4, and 8 for linear, quadratic, and cubic algorithms, respectively—to name the most obvious and convenient cases. On the other hand, the form of a scatterplot may also help in ascertaining the algorithm’s probable efﬁciency class. For a logarithmic algorithm, the scatterplot will have a concave shape (Figure 2.7a); this fact distinguishes it from all the other basic efﬁciency classes. For a linear algorithm, the points will tend to aggregate around a straight line or, more generally, to be contained between two straight lines (Figure 2.7b). Scatterplots of functions in (n lg n) and (n2 ) will have a convex shape (Figure 2.7c), making them difﬁcult to differentiate. A scatterplot of a cubic algorithm will also have a convex shape, but it will show a much more rapid increase in the metric’s values. An exponential algorithm will most probably require a logarithmic scale for the vertical axis, in which the values of loga M(n) rather than those of M(n) are plotted. (The commonly used logarithm base is 2 or 10.) In such a coordinate system, a scatterplot of a truly exponential algorithm should resemble a linear function because M(n) ≈ ca n implies logb M(n) ≈ logb c + n logb a, and vice versa. One of the possible applications of the empirical analysis is to predict the algorithm’s performance on an instance not included in the experiment sample. For example, if you observe that the ratios M(n)/g(n) are close to some constant c for the sample instances, it could be sensible to approximate M(n) by the product cg(n) for other instances, too. This approach should be used with caution, especially for values of n outside the sample range. (Mathematicians call such predictions extrapolation, as opposed to interpolation, which deals with values within the sample range.) Of course, you can try unleashing the standard techniques of statistical data analysis and prediction. Note, however, that the majority of such techniques are based on speciﬁc probabilistic assumptions that may or may not be valid for the experimental data in question. It seems appropriate to end this section by pointing out the basic differences between mathematical and empirical analyses of algorithms. The principal strength of the mathematical analysis is its independence of speciﬁc inputs; its principal weakness is its limited applicability, especially for investigating the averagecase efﬁciency. The principal strength of the empirical analysis lies in its applicability to any algorithm, but its results can depend on the particular sample of instances and the computer used in the experiment.
89
2.6 Empirical Analysis of Algorithms count or time
count or time
n
n
(a)
(b)
count or time
n
(c) FIGURE 2.7 Typical scatter plots. (a) Logarithmic. (b) Linear. (c) One of the convex functions.
Exercises 2.6 1. Consider the following wellknown sorting algorithm, which is studied later in the book, with a counter inserted to count the number of key comparisons. ALGORITHM SortAnalysis(A[0..n − 1]) //Input: An array A[0..n − 1] of n orderable elements //Output: The total number of key comparisons made count ← 0 for i ← 1 to n − 1 do
90
Fundamentals of the Analysis of Algorithm Efﬁciency
v ← A[i] j ←i−1 while j ≥ 0 and A[j ] > v do count ← count + 1 A[j + 1] ← A[j ] j ←j −1 A[j + 1] ← v return count Is the comparison counter inserted in the right place? If you believe it is, prove it; if you believe it is not, make an appropriate correction. 2. a. Run the program of Problem 1, with a properly inserted counter (or counters) for the number of key comparisons, on 20 random arrays of sizes 1000, 2000, 3000, . . . , 20,000. b. Analyze the data obtained to form a hypothesis about the algorithm’s averagecase efﬁciency. c. Estimate the number of key comparisons we should expect for a randomly generated array of size 25,000 sorted by the same algorithm. 3. Repeat Problem 2 by measuring the program’s running time in milliseconds. 4. Hypothesize a likely efﬁciency class of an algorithm based on the following empirical observations of its basic operation’s count: size
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
count 11,966 24,303 39,992 53,010 67,272 78,692 91,274 113,063 129,799 140,538 5. What scale transformation will make a logarithmic scatterplot look like a linear one? 6. How can one distinguish a scatterplot for an algorithm in (lg lg n) from a scatterplot for an algorithm in (lg n)? 7. a. Find empirically the largest number of divisions made by Euclid’s algorithm for computing gcd(m, n) for 1≤ n ≤ m ≤ 100. b. For each positive integer k, ﬁnd empirically the smallest pair of integers 1≤ n ≤ m ≤ 100 for which Euclid’s algorithm needs to make k divisions in order to ﬁnd gcd(m, n). 8. The averagecase efﬁciency of Euclid’s algorithm on inputs of size n can be measured by the average number of divisions Davg (n) made by the algorithm in computing gcd(n, 1), gcd(n, 2), . . . , gcd(n, n). For example, 1 Davg (5) = (1 + 2 + 3 + 2 + 1) = 1.8. 5
2.7
Algorithm Visualization
91
Produce a scatterplot of Davg (n) and indicate the algorithm’s likely averagecase efﬁciency class. 9. Run an experiment to ascertain the efﬁciency class of the sieve of Eratosthenes (see Section 1.1). 10. Run a timing experiment for the three algorithms for computing gcd(m, n) presented in Section 1.1.
2.7 Algorithm Visualization In addition to the mathematical and empirical analyses of algorithms, there is yet a third way to study algorithms. It is called algorithm visualization and can be deﬁned as the use of images to convey some useful information about algorithms. That information can be a visual illustration of an algorithm’s operation, of its performance on different kinds of inputs, or of its execution speed versus that of other algorithms for the same problem. To accomplish this goal, an algorithm visualization uses graphic elements—points, line segments, two or threedimensional bars, and so on—to represent some “interesting events” in the algorithm’s operation. There are two principal variations of algorithm visualization: Static algorithm visualization Dynamic algorithm visualization, also called algorithm animation Static algorithm visualization shows an algorithm’s progress through a series of still images. Algorithm animation, on the other hand, shows a continuous, movielike presentation of an algorithm’s operations. Animation is an arguably more sophisticated option, which, of course, is much more difﬁcult to implement. Early efforts in the area of algorithm visualization go back to the 1970s. The watershed event happened in 1981 with the appearance of a 30minute color sound ﬁlm titled Sorting Out Sorting. This algorithm visualization classic was produced at the University of Toronto by Ronald Baecker with the assistance of D. Sherman [Bae81, Bae98]. It contained visualizations of nine wellknown sorting algorithms (more than half of them are discussed later in the book) and provided quite a convincing demonstration of their relative speeds. The success of Sorting Out Sorting made sorting algorithms a perennial favorite for algorithm animation. Indeed, the sorting problem lends itself quite naturally to visual presentation via vertical or horizontal bars or sticks of different heights or lengths, which need to be rearranged according to their sizes (Figure 2.8). This presentation is convenient, however, only for illustrating actions of a typical sorting algorithm on small inputs. For larger ﬁles, Sorting Out Sorting used the ingenious idea of presenting data by a scatterplot of points on a coordinate plane, with the ﬁrst coordinate representing an item’s position in the ﬁle and the second one representing the item’s value; with such a representation, the process of sorting looks like a transformation of a “random” scatterplot of points into the points along a frame’s diagonal (Figure 2.9). In addition, most sorting algorithms
92
Fundamentals of the Analysis of Algorithm Efﬁciency
FIGURE 2.8 Initial and ﬁnal screens of a typical visualization of a sorting algorithm using the bar representation.
work by comparing and exchanging two given items at a time—an event that can be animated relatively easily. Since the appearance of Sorting Out Sorting, a great number of algorithm animations have been created, especially after the appearance of Java and the
2.7
Algorithm Visualization
93
FIGURE 2.9 Initial and ﬁnal screens of a typical visualization of a sorting algorithm using the scatterplot representation.
World Wide Web in the 1990s. They range in scope from one particular algorithm to a group of algorithms for the same problem (e.g., sorting) or the same application area (e.g., geometric algorithms) to generalpurpose animation systems. At the end of 2010, a catalog of links to existing visualizations, maintained under the
94
Fundamentals of the Analysis of Algorithm Efﬁciency
NSFsupported AlgoVizProject, contained over 500 links. Unfortunately, a survey of existing visualizations found most of them to be of low quality, with the content heavily skewed toward easier topics such as sorting [Sha07]. There are two principal applications of algorithm visualization: research and education. Potential beneﬁts for researchers are based on expectations that algorithm visualization may help uncover some unknown features of algorithms. For example, one researcher used a visualization of the recursive Tower of Hanoi algorithm in which odd and evennumbered disks were colored in two different colors. He noticed that two disks of the same color never came in direct contact during the algorithm’s execution. This observation helped him in developing a better nonrecursive version of the classic algorithm. To give another example, Bentley and McIlroy [Ben93] mentioned using an algorithm animation system in their work on improving a library implementation of a leading sorting algorithm. The application of algorithm visualization to education seeks to help students learning algorithms. The available evidence of its effectiveness is decisively mixed. Although some experiments did register positive learning outcomes, others failed to do so. The increasing body of evidence indicates that creating sophisticated software systems is not going to be enough. In fact, it appears that the level of student involvement with visualization might be more important than speciﬁc features of visualization software. In some experiments, lowtech visualizations prepared by students were more effective than passive exposure to sophisticated software systems. To summarize, although some successes in both research and education have been reported in the literature, they are not as impressive as one might expect. A deeper understanding of human perception of images will be required before the true potential of algorithm visualization is fulﬁlled.
SUMMARY There are two kinds of algorithm efﬁciency: time efﬁciency and space efﬁciency. Time efﬁciency indicates how fast the algorithm runs; space efﬁciency deals with the extra space it requires. An algorithm’s time efﬁciency is principally measured as a function of its input size by counting the number of times its basic operation is executed. A basic operation is the operation that contributes the most to running time. Typically, it is the most timeconsuming operation in the algorithm’s innermost loop. For some algorithms, the running time can differ considerably for inputs of the same size, leading to worstcase efﬁciency, averagecase efﬁciency, and bestcase efﬁciency. The established framework for analyzing time efﬁciency is primarily grounded in the order of growth of the algorithm’s running time as its input size goes to inﬁnity.
Summary
95
The notations O, , and are used to indicate and compare the asymptotic orders of growth of functions expressing algorithm efﬁciencies. The efﬁciencies of a large number of algorithms fall into the following few classes: constant, logarithmic, linear, linearithmic, quadratic, cubic, and exponential. The main tool for analyzing the time efﬁciency of a nonrecursive algorithm is to set up a sum expressing the number of executions of its basic operation and ascertain the sum’s order of growth. The main tool for analyzing the time efﬁciency of a recursive algorithm is to set up a recurrence relation expressing the number of executions of its basic operation and ascertain the solution’s order of growth. Succinctness of a recursive algorithm may mask its inefﬁciency. The Fibonacci numbers are an important sequence of integers in which every element is equal to the sum of its two immediate predecessors. There are several algorithms for computing the Fibonacci numbers, with drastically different efﬁciencies. Empirical analysis of an algorithm is performed by running a program implementing the algorithm on a sample of inputs and analyzing the data observed (the basic operation’s count or physical running time). This often involves generating pseudorandom numbers. The applicability to any algorithm is the principal strength of this approach; the dependence of results on the particular computer and instance sample is its main weakness. Algorithm visualization is the use of images to convey useful information about algorithms. The two principal variations of algorithm visualization are static algorithm visualization and dynamic algorithm visualization (also called algorithm animation).
This page intentionally left blank
3 Brute Force and Exhaustive Search Science is as far removed from brute force as this sword from a crowbar. —Edward Lytton (1803–1873), Leila, Book II, Chapter I
Doing a thing well is often a waste of time. —Robert Byrne, a master pool and billiards player and a writer
A
fter introducing the framework and methods for algorithm analysis in the preceding chapter, we are ready to embark on a discussion of algorithm design strategies. Each of the next eight chapters is devoted to a particular design strategy. The subject of this chapter is brute force and its important special case, exhaustive search. Brute force can be described as follows: Brute force is a straightforward approach to solving a problem, usually directly based on the problem statement and deﬁnitions of the concepts involved. The “force” implied by the strategy’s deﬁnition is that of a computer and not that of one’s intellect. “Just do it!” would be another way to describe the prescription of the bruteforce approach. And often, the bruteforce strategy is indeed the one that is easiest to apply. As an example, consider the exponentiation problem: compute a n for a nonzero number a and a nonnegative integer n. Although this problem might seem trivial, it provides a useful vehicle for illustrating several algorithm design strategies, including the brute force. (Also note that computing a n mod m for some large integers is a principal component of a leading encryption algorithm.) By the deﬁnition of exponentiation, ... an = a ∗ ∗ a . n times
97
98
Brute Force and Exhaustive Search
This suggests simply computing a n by multiplying 1 by a n times. We have already encountered at least two bruteforce algorithms in the book: the consecutive integer checking algorithm for computing gcd(m, n) in Section 1.1 and the deﬁnitionbased algorithm for matrix multiplication in Section 2.3. Many other examples are given later in this chapter. (Can you identify a few algorithms you already know as being based on the bruteforce approach?) Though rarely a source of clever or efﬁcient algorithms, the bruteforce approach should not be overlooked as an important algorithm design strategy. First, unlike some of the other strategies, brute force is applicable to a very wide variety of problems. In fact, it seems to be the only general approach for which it is more difﬁcult to point out problems it cannot tackle. Second, for some important problems—e.g., sorting, searching, matrix multiplication, string matching— the bruteforce approach yields reasonable algorithms of at least some practical value with no limitation on instance size. Third, the expense of designing a more efﬁcient algorithm may be unjustiﬁable if only a few instances of a problem need to be solved and a bruteforce algorithm can solve those instances with acceptable speed. Fourth, even if too inefﬁcient in general, a bruteforce algorithm can still be useful for solving smallsize instances of a problem. Finally, a bruteforce algorithm can serve an important theoretical or educational purpose as a yardstick with which to judge more efﬁcient alternatives for solving a problem.
3.1 Selection Sort and Bubble Sort In this section, we consider the application of the bruteforce approach to the problem of sorting: given a list of n orderable items (e.g., numbers, characters from some alphabet, character strings), rearrange them in nondecreasing order. As we mentioned in Section 1.3, dozens of algorithms have been developed for solving this very important problem. You might have learned several of them in the past. If you have, try to forget them for the time being and look at the problem afresh. Now, after your mind is unburdened of previous knowledge of sorting algorithms, ask yourself a question: “What would be the most straightforward method for solving the sorting problem?” Reasonable people may disagree on the answer to this question. The two algorithms discussed here—selection sort and bubble sort—seem to be the two prime candidates.
Selection Sort We start selection sort by scanning the entire given list to ﬁnd its smallest element and exchange it with the ﬁrst element, putting the smallest element in its ﬁnal position in the sorted list. Then we scan the list, starting with the second element, to ﬁnd the smallest among the last n − 1 elements and exchange it with the second element, putting the second smallest element in its ﬁnal position. Generally, on the
3.1 Selection Sort and Bubble Sort
99
ith pass through the list, which we number from 0 to n − 2, the algorithm searches for the smallest item among the last n − i elements and swaps it with Ai :
A0 ≤ A1 ≤ . . . ≤ Ai–1 ⏐ Ai, . . . , Amin, . . . , An–1 in their final positions
the last n – i elements
After n − 1 passes, the list is sorted. Here is pseudocode of this algorithm, which, for simplicity, assumes that the list is implemented as an array: ALGORITHM SelectionSort(A[0..n − 1]) //Sorts a given array by selection sort //Input: An array A[0..n − 1] of orderable elements //Output: Array A[0..n − 1] sorted in nondecreasing order for i ← 0 to n − 2 do min ← i for j ← i + 1 to n − 1 do if A[j ] < A[min] min ← j swap A[i] and A[min] As an example, the action of the algorithm on the list 89, 45, 68, 90, 29, 34, 17 is illustrated in Figure 3.1. The analysis of selection sort is straightforward. The input size is given by the number of elements n; the basic operation is the key comparison A[j ] < A[min]. The number of times it is executed depends only on the array size and is given by the following sum: C(n) =
n−2 n−1
i=0 j =i+1
1=
n−2 n−2
[(n − 1) − (i + 1) + 1] = (n − 1 − i). i=0
 89 45 68 90 29 34 17  45 68 90 29 34 17 29  68 90 45 34 17 29 34  90 45 68 17 29 34 45  90 68 17 29 34 45 68  90 17 29 34 45 68 89 
i=0
17 89 89 89 89 89 90
FIGURE 3.1 Example of sorting with selection sort. Each line corresponds to one iteration of the algorithm, i.e., a pass through the list’s tail to the right of the vertical bar; an element in bold indicates the smallest element found. Elements to the left of the vertical bar are in their ﬁnal positions and are not considered in this and subsequent iterations.
100
Brute Force and Exhaustive Search
Since we have already encountered the last sum in analyzing the algorithm of Example 2 in Section 2.3, you should be able to compute it now on your own. Whether you compute this sum by distributing the summation symbol or by immediately getting the sum of decreasing integers, the answer, of course, must be the same: C(n) =
n−2 n−1
i=0 j =i+1
1=
n−2
(n − 1)n . (n − 1 − i) = 2 i=0
Thus, selection sort is a (n2 ) algorithm on all inputs. Note, however, that the number of key swaps is only (n), or, more precisely, n − 1 (one for each repetition of the i loop). This property distinguishes selection sort positively from many other sorting algorithms.
Bubble Sort Another bruteforce application to the sorting problem is to compare adjacent elements of the list and exchange them if they are out of order. By doing it repeatedly, we end up “bubbling up” the largest element to the last position on the list. The next pass bubbles up the second largest element, and so on, until after n − 1 passes the list is sorted. Pass i (0 ≤ i ≤ n − 2) of bubble sort can be represented by the following diagram: ?
A0, . . . , Aj ↔ Aj +1, . . . , An−i−1  An−i ≤ . . . ≤ An−1 in their ﬁnal positions
Here is pseudocode of this algorithm. ALGORITHM BubbleSort(A[0..n − 1]) //Sorts a given array by bubble sort //Input: An array A[0..n − 1] of orderable elements //Output: Array A[0..n − 1] sorted in nondecreasing order for i ← 0 to n − 2 do for j ← 0 to n − 2 − i do if A[j + 1] < A[j ] swap A[j ] and A[j + 1] The action of the algorithm on the list 89, 45, 68, 90, 29, 34, 17 is illustrated as an example in Figure 3.2. The number of key comparisons for the bubblesort version given above is the same for all arrays of size n; it is obtained by a sum that is almost identical to the sum for selection sort:
101
3.1 Selection Sort and Bubble Sort
89 45
? ↔ 45
68 ? 89 ↔ 68
90
29
34
17
90
29
34
17
34 ? ↔ 34
17
45
68
? 89 ↔
45
68
89
29
90
45
68
89
29
34
45
68
89
29
34
17 ? 90 ↔ 17 17  90
29
34
17

90
34
17 ? ↔ 17

90

90
45
? ? ? ↔ 68 ↔ 89 ↔
? 90 ↔ 29
45
68
29
? 89 ↔
45
68
29
34
89
45
68
29
34
17

89
90
etc.
FIGURE 3.2 First two passes of bubble sort on the list 89, 45, 68, 90, 29, 34, 17. A new line is shown after a swap of two elements is done. The elements to the right of the vertical bar are in their ﬁnal positions and are not considered in subsequent iterations of the algorithm.
C(n) =
n−2 n−2−i
i=0 j =0
=
1=
n−2
[(n − 2 − i) − 0 + 1] i=0
n−2
(n − 1)n ∈ (n2 ). (n − 1 − i) = 2 i=0
The number of key swaps, however, depends on the input. In the worst case of decreasing arrays, it is the same as the number of key comparisons: Sworst (n) = C(n) =
(n − 1)n ∈ (n2 ). 2
As is often the case with an application of the bruteforce strategy, the ﬁrst version of an algorithm obtained can often be improved upon with a modest amount of effort. Speciﬁcally, we can improve the crude version of bubble sort given above by exploiting the following observation: if a pass through the list makes no exchanges, the list has been sorted and we can stop the algorithm (Problem 12a in this section’s exercises). Though the new version runs faster on some inputs, it is still in (n2 ) in the worst and average cases. In fact, even among elementary sorting methods, bubble sort is an inferior choice, and if it were not for its catchy name, you would probably have never heard of it. However, the general lesson you just learned is important and worth repeating: A ﬁrst application of the bruteforce approach often results in an algorithm that can be improved with a modest amount of effort.
102
Brute Force and Exhaustive Search
Exercises 3.1 1. a. Give an example of an algorithm that should not be considered an application of the bruteforce approach. b. Give an example of a problem that cannot be solved by a bruteforce algorithm. 2. a. What is the time efﬁciency of the bruteforce algorithm for computing a n as a function of n? As a function of the number of bits in the binary representation of n? b. If you are to compute a n mod m where a > 1 and n is a large positive integer, how would you circumvent the problem of a very large magnitude of a n? 3. For each of the algorithms in Problems 4, 5, and 6 of Exercises 2.3, tell whether or not the algorithm is based on the bruteforce approach. 4. a. Design a bruteforce algorithm for computing the value of a polynomial p(x) = anx n + an−1x n−1 + . . . + a1x + a0 at a given point x0 and determine its worstcase efﬁciency class. b. If the algorithm you designed is in (n2 ), design a linear algorithm for this problem. c. Is it possible to design an algorithm with a betterthanlinear efﬁciency for this problem? 5. A network topology speciﬁes how computers, printers, and other devices are connected over a network. The ﬁgure below illustrates three common topologies of networks: the ring, the star, and the fully connected mesh.
ring
star
fully connected mesh
You are given a boolean matrix A[0..n − 1, 0..n − 1], where n > 3, which is supposed to be the adjacency matrix of a graph modeling a network with one of these topologies. Your task is to determine which of these three topologies, if any, the matrix represents. Design a bruteforce algorithm for this task and indicate its time efﬁciency class. 6. Tetromino tilings Tetrominoes are tiles made of four 1 × 1 squares. There are ﬁve types of tetrominoes shown below:
103
3.1 Selection Sort and Bubble Sort
straight tetromino
square tetromino
Ltetromino
Ttetromino
Ztetromino
Is it possible to tile—i.e., cover exactly without overlaps—an 8 × 8 chessboard with a. straight tetrominoes? b. square tetrominoes? c. Ltetrominoes? d. Ttetrominoes? e. Ztetrominoes? 7. A stack of fake coins There are n stacks of n identicallooking coins. All of the coins in one of these stacks are counterfeit, while all the coins in the other stacks are genuine. Every genuine coin weighs 10 grams; every fake weighs 11 grams. You have an analytical scale that can determine the exact weight of any number of coins. a. Devise a bruteforce algorithm to identify the stack with the fake coins and determine its worstcase efﬁciency class. b. What is the minimum number of weighings needed to identify the stack with the fake coins? 8. Sort the list E, X, A, M, P , L, E in alphabetical order by selection sort. 9. Is selection sort stable? (The deﬁnition of a stable sorting algorithm was given in Section 1.3.) 10. Is it possible to implement selection sort for linked lists with the same (n2 ) efﬁciency as the array version? 11. Sort the list E, X, A, M, P , L, E in alphabetical order by bubble sort. 12. a. Prove that if bubble sort makes no exchanges on its pass through a list, the list is sorted and the algorithm can be stopped. b. Write pseudocode of the method that incorporates this improvement. c. Prove that the worstcase efﬁciency of the improved version is quadratic. 13. Is bubble sort stable? 14. Alternating disks You have a row of 2n disks of two colors, n dark and n light. They alternate: dark, light, dark, light, and so on. You want to get all the dark disks to the righthand end, and all the light disks to the lefthand end. The only moves you are allowed to make are those that interchange the positions of two neighboring disks.
Design an algorithm for solving this puzzle and determine the number of moves it takes. [Gar99]
104
Brute Force and Exhaustive Search
3.2 Sequential Search and BruteForce String Matching We saw in the previous section two applications of the bruteforce approach to the sorting porblem. Here we discuss two applications of this strategy to the problem of searching. The ﬁrst deals with the canonical problem of searching for an item of a given value in a given list. The second is different in that it deals with the stringmatching problem.
Sequential Search We have already encountered a bruteforce algorithm for the general searching problem: it is called sequential search (see Section 2.1). To repeat, the algorithm simply compares successive elements of a given list with a given search key until either a match is encountered (successful search) or the list is exhausted without ﬁnding a match (unsuccessful search). A simple extra trick is often employed in implementing sequential search: if we append the search key to the end of the list, the search for the key will have to be successful, and therefore we can eliminate the end of list check altogether. Here is pseudocode of this enhanced version. ALGORITHM SequentialSearch2(A[0..n], K) //Implements sequential search with a search key as a sentinel //Input: An array A of n elements and a search key K //Output: The index of the ﬁrst element in A[0..n − 1] whose value is // equal to K or −1 if no such element is found A[n] ← K i←0 while A[i] = K do i←i+1 if i < n return i else return −1 Another straightforward improvement can be incorporated in sequential search if a given list is known to be sorted: searching in such a list can be stopped as soon as an element greater than or equal to the search key is encountered. Sequential search provides an excellent illustration of the bruteforce approach, with its characteristic strength (simplicity) and weakness (inferior efﬁciency). The efﬁciency results obtained in Section 2.1 for the standard version of sequential search change for the enhanced version only very slightly, so that the algorithm remains linear in both the worst and average cases. We discuss later in the book several searching algorithms with a better time efﬁciency.
3.2
105
Sequential Search and BruteForce String Matching
BruteForce String Matching Recall the stringmatching problem introduced in Section 1.3: given a string of n characters called the text and a string of m characters (m ≤ n) called the pattern, ﬁnd a substring of the text that matches the pattern. To put it more precisely, we want to ﬁnd i—the index of the leftmost character of the ﬁrst matching substring in the text—such that ti = p0, . . . , ti+j = pj , . . . , ti+m−1 = pm−1: t0
...
ti p0
... ...
ti+j pj
... ...
ti+m−1 pm−1
...
tn−1
text T
pattern P
If matches other than the ﬁrst one need to be found, a stringmatching algorithm can simply continue working until the entire text is exhausted. A bruteforce algorithm for the stringmatching problem is quite obvious: align the pattern against the ﬁrst m characters of the text and start matching the corresponding pairs of characters from left to right until either all the m pairs of the characters match (then the algorithm can stop) or a mismatching pair is encountered. In the latter case, shift the pattern one position to the right and resume the character comparisons, starting again with the ﬁrst character of the pattern and its counterpart in the text. Note that the last position in the text that can still be a beginning of a matching substring is n − m (provided the text positions are indexed from 0 to n − 1). Beyond that position, there are not enough characters to match the entire pattern; hence, the algorithm need not make any comparisons there. ALGORITHM BruteForceStringMatch(T [0..n − 1], P [0..m − 1]) //Implements bruteforce string matching //Input: An array T [0..n − 1] of n characters representing a text and // an array P [0..m − 1] of m characters representing a pattern //Output: The index of the ﬁrst character in the text that starts a // matching substring or −1 if the search is unsuccessful for i ← 0 to n − m do j ←0 while j < m and P [j ] = T [i + j ] do j ←j +1 if j = m return i return −1 An operation of the algorithm is illustrated in Figure 3.3. Note that for this example, the algorithm shifts the pattern almost always after a single character comparison. The worst case is much worse: the algorithm may have to make all m comparisons before shifting the pattern, and this can happen for each of the n − m + 1 tries. (Problem 6 in this section’s exercises asks you to give a speciﬁc example of such a situation.) Thus, in the worst case, the algorithm makes
106
Brute Force and Exhaustive Search
N N
O O N
B T O N
O T O N
D
T O N
Y
T O N
_
T O N
N
O
T
T O N
T O
T
I
C
E
D _ H
I
M
FIGURE 3.3 Example of bruteforce string matching. The pattern’s characters that are compared with their text counterparts are in bold type.
m(n − m + 1) character comparisons, which puts it in the O(nm) class. For a typical word search in a natural language text, however, we should expect that most shifts would happen after very few comparisons (check the example again). Therefore, the averagecase efﬁciency should be considerably better than the worstcase efﬁciency. Indeed it is: for searching in random texts, it has been shown to be linear, i.e., (n). There are several more sophisticated and more efﬁcient algorithms for string searching. The most widely known of them—by R. Boyer and J. Moore—is outlined in Section 7.2 along with its simpliﬁcation suggested by R. Horspool.
Exercises 3.2 1. Find the number of comparisons made by the sentinel version of sequential search a. in the worst case. b. in the average case if the probability of a successful search is p (0 ≤ p ≤ 1). 2. As shown in Section 2.1, the average number of key comparisons made by sequential search (without a sentinel, under standard assumptions about its inputs) is given by the formula Cavg (n) =
p(n + 1) + n(1 − p), 2
where p is the probability of a successful search. Determine, for a ﬁxed n, the values of p (0 ≤ p ≤ 1) for which this formula yields the maximum value of Cavg (n) and the minimum value of Cavg (n). 3. Gadget testing A ﬁrm wants to determine the highest ﬂoor of its nstory headquarters from which a gadget can fall without breaking. The ﬁrm has two identical gadgets to experiment with. If one of them gets broken, it cannot be repaired, and the experiment will have to be completed with the remaining gadget. Design an algorithm in the best efﬁciency class you can to solve this problem.
3.2
Sequential Search and BruteForce String Matching
107
4. Determine the number of character comparisons made by the bruteforce algorithm in searching for the pattern GANDHI in the text THERE_IS_MORE_TO_LIFE_THAN_INCREASING_ITS_SPEED
Assume that the length of the text—it is 47 characters long—is known before the search starts. 5. How many comparisons (both successful and unsuccessful) will be made by the bruteforce algorithm in searching for each of the following patterns in the binary text of one thousand zeros? a. 00001 b. 10000 c. 01010 6. Give an example of a text of length n and a pattern of length m that constitutes a worstcase input for the bruteforce stringmatching algorithm. Exactly how many character comparisons will be made for such input? 7. In solving the stringmatching problem, would there be any advantage in comparing pattern and text characters righttoleft instead of lefttoright? 8. Consider the problem of counting, in a given text, the number of substrings that start with an A and end with a B. For example, there are four such substrings in CABAAXBYA. a. Design a bruteforce algorithm for this problem and determine its efﬁciency class. b. Design a more efﬁcient algorithm for this problem. [Gin04] 9. Write a visualization program for the bruteforce stringmatching algorithm. 10. Word Find A popular diversion in the United States, “word ﬁnd” (or “word search”) puzzles ask the player to ﬁnd each of a given set of words in a square table ﬁlled with single letters. A word can read horizontally (left or right), vertically (up or down), or along a 45 degree diagonal (in any of the four directions) formed by consecutively adjacent cells of the table; it may wrap around the table’s boundaries, but it must read in the same direction with no zigzagging. The same cell of the table may be used in different words, but, in a given word, the same cell may be used no more than once. Write a computer program for solving this puzzle. 11. Battleship game Write a program based on a version of bruteforce pattern matching for playing the game Battleship on the computer. The rules of the game are as follows. There are two opponents in the game (in this case, a human player and the computer). The game is played on two identical boards (10 × 10 tables of squares) on which each opponent places his or her ships, not seen by the opponent. Each player has ﬁve ships, each of which occupies a certain number of squares on the board: a destroyer (two squares), a submarine (three squares), a cruiser (three squares), a battleship (four squares), and an aircraft carrier (ﬁve squares). Each ship is placed either horizontally or vertically, with no two ships touching each other. The game is played by the opponents taking turns “shooting” at each other’s ships. The
108
Brute Force and Exhaustive Search
result of every shot is displayed as either a hit or a miss. In case of a hit, the player gets to go again and keeps playing until missing. The goal is to sink all the opponent’s ships before the opponent succeeds in doing it ﬁrst. To sink a ship, all squares occupied by the ship must be hit.
3.3 ClosestPair and ConvexHull Problems by Brute Force In this section, we consider a straightforward approach to two wellknown problems dealing with a ﬁnite set of points in the plane. These problems, aside from their theoretical interest, arise in two important applied areas: computational geometry and operations research.
ClosestPair Problem The closestpair problem calls for ﬁnding the two closest points in a set of n points. It is the simplest of a variety of problems in computational geometry that deals with proximity of points in the plane or higherdimensional spaces. Points in question can represent such physical objects as airplanes or post ofﬁces as well as database records, statistical samples, DNA sequences, and so on. An airtrafﬁc controller might be interested in two closest planes as the most probable collision candidates. A regional postal service manager might need a solution to the closestpair problem to ﬁnd candidate postofﬁce locations to be closed. One of the important applications of the closestpair problem is cluster analysis in statistics. Based on n data points, hierarchical cluster analysis seeks to organize them in a hierarchy of clusters based on some similarity metric. For numerical data, this metric is usually the Euclidean distance; for text and other nonnumerical data, metrics such as the Hamming distance (see Problem 5 in this section’s exercises) are used. A bottomup algorithm begins with each element as a separate cluster and merges them into successively larger clusters by combining the closest pair of clusters. For simplicity, we consider the twodimensional case of the closestpair problem. We assume that the points in question are speciﬁed in a standard fashion by their (x, y) Cartesian coordinates and that the distance between two points pi (xi , yi ) and pj (xj , yj ) is the standard Euclidean distance d(pi , pj ) = (xi − xj )2 + (yi − yj )2 . The bruteforce approach to solving this problem leads to the following obvious algorithm: compute the distance between each pair of distinct points and ﬁnd a pair with the smallest distance. Of course, we do not want to compute the distance between the same pair of points twice. To avoid doing so, we consider only the pairs of points (pi , pj ) for which i < j .
3.3 ClosestPair and ConvexHull Problems by Brute Force
109
Pseudocode below computes the distance between the two closest points; getting the closest points themselves requires just a trivial modiﬁcation. ALGORITHM
BruteForceClosestPair(P )
//Finds distance between two closest points in the plane by brute force //Input: A list P of n (n ≥ 2) points p1(x1, y1), . . . , pn(xn, yn) //Output: The distance between the closest pair of points d←∞ for i ← 1 to n − 1 do for j ← i + 1 to n do d ← min(d, sqrt((xi − xj )2 + (yi − yj )2 )) //sqrt is square root return d The basic operation of the algorithm is computing the square root. In the age of electronic calculators with a squareroot button, one might be led to believe that computing the square root is as simple an operation as, say, addition or multiplication. Of course, it is not. For starters, even for most integers, square roots are irrational numbers that therefore can be found only approximately. Moreover, computing such approximations is not a trivial matter. But, in fact, computing square roots in the loop can be avoided! (Can you think how?) The trick is to realize that we can simply ignore the squareroot function and compare the values (xi − xj )2 + (yi − yj )2 themselves. We can do this because the smaller a number of which we take the square root, the smaller its square root, or, as mathematicians say, the squareroot function is strictly increasing. Then the basic operation of the algorithm will be squaring a number. The number of times it will be executed can be computed as follows: C(n) =
n−1 n
i=1 j =i+1
2=2
n−1
(n − i) i=1
= 2[(n − 1) + (n − 2) + . . . + 1] = (n − 1)n ∈ (n2 ). Of course, speeding up the innermost loop of the algorithm could only decrease the algorithm’s running time by a constant factor (see Problem 1 in this section’s exercises), but it cannot improve its asymptotic efﬁciency class. In Chapter 5, we discuss a linearithmic algorithm for this problem, which is based on a more sophisticated design technique.
ConvexHull Problem On to the other problem—that of computing the convex hull. Finding the convex hull for a given set of points in the plane or a higher dimensional space is one of the most important—some people believe the most important—problems in computational geometry. This prominence is due to a variety of applications in which
110
Brute Force and Exhaustive Search
this problem needs to be solved, either by itself or as a part of a larger task. Several such applications are based on the fact that convex hulls provide convenient approximations of object shapes and data sets given. For example, in computer animation, replacing objects by their convex hulls speeds up collision detection; the same idea is used in path planning for Mars mission rovers. Convex hulls are used in computing accessibility maps produced from satellite images by Geographic Information Systems. They are also used for detecting outliers by some statistical techniques. An efﬁcient algorithm for computing a diameter of a set of points, which is the largest distance between two of the points, needs the set’s convex hull to ﬁnd the largest distance between two of its extreme points (see below). Finally, convex hulls are important for solving many optimization problems, because their extreme points provide a limited set of solution candidates. We start with a deﬁnition of a convex set. DEFINITION A set of points (ﬁnite or inﬁnite) in the plane is called convex if for any two points p and q in the set, the entire line segment with the endpoints at p and q belongs to the set. All the sets depicted in Figure 3.4a are convex, and so are a straight line, a triangle, a rectangle, and, more generally, any convex polygon,1 a circle, and the entire plane. On the other hand, the sets depicted in Figure 3.4b, any ﬁnite set of two or more distinct points, the boundary of any convex polygon, and a circumference are examples of sets that are not convex. Now we are ready for the notion of the convex hull. Intuitively, the convex hull of a set of n points in the plane is the smallest convex polygon that contains all of them either inside or on its boundary. If this formulation does not ﬁre up your enthusiasm, consider the problem as one of barricading n sleeping tigers by a fence of the shortest length. This interpretation is due to D. Harel [Har92]; it is somewhat lively, however, because the fenceposts have to be erected right at the spots where some of the tigers sleep! There is another, much tamer interpretation of this notion. Imagine that the points in question are represented by nails driven into a large sheet of plywood representing the plane. Take a rubber band and stretch it to include all the nails, then let it snap into place. The convex hull is the area bounded by the snapped rubber band (Figure 3.5). A formal deﬁnition of the convex hull that is applicable to arbitrary sets, including sets of points that happen to lie on the same line, follows. DEFINITION The convex hull of a set S of points is the smallest convex set containing S. (The “smallest” requirement means that the convex hull of S must be a subset of any convex set containing S.) If S is convex, its convex hull is obviously S itself. If S is a set of two points, its convex hull is the line segment connecting these points. If S is a set of three 1.
By “a triangle, a rectangle, and, more generally, any convex polygon,” we mean here a region, i.e., the set of points both inside and on the boundary of the shape in question.
3.3 ClosestPair and ConvexHull Problems by Brute Force
(a)
111
(b)
FIGURE 3.4 (a) Convex sets. (b) Sets that are not convex.
FIGURE 3.5 Rubberband interpretation of the convex hull.
points not on the same line, its convex hull is the triangle with the vertices at the three points given; if the three points do lie on the same line, the convex hull is the line segment with its endpoints at the two points that are farthest apart. For an example of the convex hull for a larger set, see Figure 3.6. A study of the examples makes the following theorem an expected result. THEOREM The convex hull of any set S of n > 2 points not all on the same line is a convex polygon with the vertices at some of the points of S. (If all the points do lie on the same line, the polygon degenerates to a line segment but still with the endpoints at two points of S.)
112
Brute Force and Exhaustive Search p6 p7 p2
p8
p3
p4
p5
p1
FIGURE 3.6 The convex hull for this set of eight points is the convex polygon with vertices at p1, p5, p6, p7, and p3.
The convexhull problem is the problem of constructing the convex hull for a given set S of n points. To solve it, we need to ﬁnd the points that will serve as the vertices of the polygon in question. Mathematicians call the vertices of such a polygon “extreme points.” By deﬁnition, an extreme point of a convex set is a point of this set that is not a middle point of any line segment with endpoints in the set. For example, the extreme points of a triangle are its three vertices, the extreme points of a circle are all the points of its circumference, and the extreme points of the convex hull of the set of eight points in Figure 3.6 are p1, p5, p6, p7, and p3. Extreme points have several special properties other points of a convex set do not have. One of them is exploited by the simplex method, a very important algorithm discussed in Section 10.1. This algorithm solves linear programming problems, which are problems of ﬁnding a minimum or a maximum of a linear function of n variables subject to linear constraints (see Problem 12 in this section’s exercises for an example and Sections 6.6 and 10.1 for a general discussion). Here, however, we are interested in extreme points because their identiﬁcation solves the convexhull problem. Actually, to solve this problem completely, we need to know a bit more than just which of n points of a given set are extreme points of the set’s convex hull: we need to know which pairs of points need to be connected to form the boundary of the convex hull. Note that this issue can also be addressed by listing the extreme points in a clockwise or a counterclockwise order. So how can we solve the convexhull problem in a bruteforce manner? If you do not see an immediate plan for a frontal attack, do not be dismayed: the convexhull problem is one with no obvious algorithmic solution. Nevertheless, there is a simple but inefﬁcient algorithm that is based on the following observation about line segments making up the boundary of a convex hull: a line segment connecting two points pi and pj of a set of n points is a part of the convex hull’s boundary if and
3.3 ClosestPair and ConvexHull Problems by Brute Force
113
only if all the other points of the set lie on the same side of the straight line through these two points.2 (Verify this property for the set in Figure 3.6.) Repeating this test for every pair of points yields a list of line segments that make up the convex hull’s boundary. A few elementary facts from analytical geometry are needed to implement this algorithm. First, the straight line through two points (x1, y1), (x2 , y2 ) in the coordinate plane can be deﬁned by the equation ax + by = c, where a = y2 − y1, b = x1 − x2 , c = x1y2 − y1x2 . Second, such a line divides the plane into two halfplanes: for all the points in one of them, ax + by > c, while for all the points in the other, ax + by < c. (For the points on the line itself, of course, ax + by = c.) Thus, to check whether certain points lie on the same side of the line, we can simply check whether the expression ax + by − c has the same sign for each of these points. We leave the implementation details as an exercise. What is the time efﬁciency of this algorithm? It is in O(n3): for each of n(n − 1)/2 pairs of distinct points, we may need to ﬁnd the sign of ax + by − c for each of the other n − 2 points. There are much more efﬁcient algorithms for this important problem, and we discuss one of them later in the book.
Exercises 3.3 1. Assuming that sqrt takes about 10 times longer than each of the other operations in the innermost loop of BruteForceClosestPoints, which are assumed to take the same amount of time, estimate how much faster the algorithm will run after the improvement discussed in Section 3.3. 2. Can you design a more efﬁcient algorithm than the one based on the bruteforce strategy to solve the closestpair problem for n points x1, x2 , . . . , xn on the real line? 3. Let x1 < x2 < . . . < xn be real numbers representing coordinates of n villages located along a straight road. A post ofﬁce needs to be built in one of these villages. a. Design an efﬁcient algorithm to ﬁnd the postofﬁce location minimizing the average distance between the villages and the post ofﬁce. b. Design an efﬁcient algorithm to ﬁnd the postofﬁce location minimizing the maximum distance from a village to the post ofﬁce.
2.
For the sake of simplicity, we assume here that no three points of a given set lie on the same line. A modiﬁcation needed for the general case is left for the exercises.
114
Brute Force and Exhaustive Search
4. a. There are several alternative ways to deﬁne a distance between two points p1(x1, y1) and p2 (x2 , y2 ) in the Cartesian plane. In particular, the Manhattan distance is deﬁned as dM (p1, p2 ) = x1 − x2  + y1 − y2 . Prove that dM satisﬁes the following axioms, which every distance function must satisfy: i. dM (p1, p2 ) ≥ 0 for any two points p1 and p2 , and dM (p1, p2 ) = 0 if and only if p1 = p2 ii. dM (p1, p2 ) = dM (p2 , p1) iii. dM (p1, p2 ) ≤ dM (p1, p3) + dM (p3, p2 ) for any p1, p2 , and p3 b. Sketch all the points in the Cartesian plane whose Manhattan distance to the origin (0, 0) is equal to 1. Do the same for the Euclidean distance. c. True or false: A solution to the closestpair problem does not depend on which of the two metrics—dE (Euclidean) or dM (Manhattan)—is used? 5. The Hamming distance between two strings of equal length is deﬁned as the number of positions at which the corresponding symbols are different. It is named after Richard Hamming (1915–1998), a prominent American scientist and engineer, who introduced it in his seminal paper on errordetecting and errorcorrecting codes. a. Does the Hamming distance satisfy the three axioms of a distance metric listed in Problem 4? b. What is the time efﬁciency class of the bruteforce algorithm for the closestpair problem if the points in question are strings of m symbols long and the distance between two of them is measured by the Hamming distance? 6. Odd pie ﬁght There are n ≥ 3 people positioned on a ﬁeld (Euclidean plane) so that each has a unique nearest neighbor. Each person has a cream pie. At a signal, everybody hurls his or her pie at the nearest neighbor. Assuming that n is odd and that nobody can miss his or her target, true or false: There always remains at least one person not hit by a pie. [Car79] 7. The closestpair problem can be posed in the kdimensional space, in which the Euclidean distance between two points p (x1 , . . . , xk ) and p
(x1
, . . . , xk
) is deﬁned as
k
d(p , p ) = (xs − xs
)2 . s=1
What is the timeefﬁciency class of the bruteforce algorithm for the kdimensional closestpair problem? 8. Find the convex hulls of the following sets and identify their extreme points (if they have any): a. a line segment
3.4
Exhaustive Search
115
b. a square c. the boundary of a square d. a straight line 9. Design a lineartime algorithm to determine two extreme points of the convex hull of a given set of n > 1 points in the plane. 10. What modiﬁcation needs to be made in the bruteforce algorithm for the convexhull problem to handle more than two points on the same straight line? 11. Write a program implementing the bruteforce algorithm for the convexhull problem. 12. Consider the following small instance of the linear programming problem: maximize subject to
3x + 5y x+ y≤4 x + 3y ≤ 6 x ≥ 0, y ≥ 0.
a. Sketch, in the Cartesian plane, the problem’s feasible region, deﬁned as the set of points satisfying all the problem’s constraints. b. Identify the region’s extreme points. c. Solve this optimization problem by using the following theorem: A linear programming problem with a nonempty bounded feasible region always has a solution, which can be found at one of the extreme points of its feasible region.
3.4 Exhaustive Search Many important problems require ﬁnding an element with a special property in a domain that grows exponentially (or faster) with an instance size. Typically, such problems arise in situations that involve—explicitly or implicitly—combinatorial objects such as permutations, combinations, and subsets of a given set. Many such problems are optimization problems: they ask to ﬁnd an element that maximizes or minimizes some desired characteristic such as a path length or an assignment cost. Exhaustive search is simply a bruteforce approach to combinatorial problems. It suggests generating each and every element of the problem domain, selecting those of them that satisfy all the constraints, and then ﬁnding a desired element (e.g., the one that optimizes some objective function). Note that although the idea of exhaustive search is quite straightforward, its implementation typically requires an algorithm for generating certain combinatorial objects. We delay a discussion of such algorithms until the next chapter and assume here that they exist.
116
Brute Force and Exhaustive Search
We illustrate exhaustive search by applying it to three important problems: the traveling salesman problem, the knapsack problem, and the assignment problem.
Traveling Salesman Problem The traveling salesman problem (TSP) has been intriguing researchers for the last 150 years by its seemingly simple formulation, important applications, and interesting connections to other combinatorial problems. In layman’s terms, the problem asks to ﬁnd the shortest tour through a given set of n cities that visits each city exactly once before returning to the city where it started. The problem can be conveniently modeled by a weighted graph, with the graph’s vertices representing the cities and the edge weights specifying the distances. Then the problem can be stated as the problem of ﬁnding the shortest Hamiltonian circuit of the graph. (A Hamiltonian circuit is deﬁned as a cycle that passes through all the vertices of the graph exactly once. It is named after the Irish mathematician Sir William Rowan Hamilton (1805–1865), who became interested in such cycles as an application of his algebraic discoveries.) It is easy to see that a Hamiltonian circuit can also be deﬁned as a sequence of n + 1 adjacent vertices vi0 , vi1, . . . , vin−1, vi0 , where the ﬁrst vertex of the sequence is the same as the last one and all the other n − 1 vertices are distinct. Further, we can assume, with no loss of generality, that all circuits start and end at one particular vertex (they are cycles after all, are they not?). Thus, we can get all the tours by generating all the permutations of n − 1 intermediate cities, compute the tour lengths, and ﬁnd the shortest among them. Figure 3.7 presents a small instance of the problem and its solution by this method. An inspection of Figure 3.7 reveals three pairs of tours that differ only by their direction. Hence, we could cut the number of vertex permutations by half. We could, for example, choose any two intermediate vertices, say, b and c, and then consider only permutations in which b precedes c. (This trick implicitly deﬁnes a tour’s direction.) This improvement cannot brighten the efﬁciency picture much, however. The total number of permutations needed is still 21 (n − 1)!, which makes the exhaustivesearch approach impractical for all but very small values of n. On the other hand, if you always see your glass as halffull, you can claim that cutting the work by half is nothing to sneeze at, even if you solve a small instance of the problem, especially by hand. Also note that had we not limited our investigation to the circuits starting at the same vertex, the number of permutations would have been even larger, by a factor of n.
Knapsack Problem Here is another wellknown problem in algorithmics. Given n items of known weights w1, w2 , . . . , wn and values v1, v2 , . . . , vn and a knapsack of capacity W , ﬁnd the most valuable subset of the items that ﬁt into the knapsack. If you do not like the idea of putting yourself in the shoes of a thief who wants to steal the most
3.4 2
a
5
8
c Tour ——
b
3
7
1
117
Exhaustive Search
d Length ———
a > b > c > d > a
I = 2 + 8 + 1 + 7 = 18
a > b > d > c > a
I = 2 + 3 + 1 + 5 = 11
a > c > b > d > a
I = 5 + 8 + 3 + 7 = 23
a > c > d > b > a
I = 5 + 1 + 3 + 2 = 11
a > d > b > c > a
I = 7 + 3 + 8 + 5 = 23
a > d > c > b > a
I = 7 + 1 + 8 + 2 = 18
optimal
optimal
FIGURE 3.7 Solution to a small instance of the traveling salesman problem by exhaustive search.
valuable loot that ﬁts into his knapsack, think about a transport plane that has to deliver the most valuable set of items to a remote location without exceeding the plane’s capacity. Figure 3.8a presents a small instance of the knapsack problem. The exhaustivesearch approach to this problem leads to generating all the subsets of the set of n items given, computing the total weight of each subset in order to identify feasible subsets (i.e., the ones with the total weight not exceeding the knapsack capacity), and ﬁnding a subset of the largest value among them. As an example, the solution to the instance of Figure 3.8a is given in Figure 3.8b. Since the number of subsets of an nelement set is 2n, the exhaustive search leads to a (2n) algorithm, no matter how efﬁciently individual subsets are generated. Thus, for both the traveling salesman and knapsack problems considered above, exhaustive search leads to algorithms that are extremely inefﬁcient on every input. In fact, these two problems are the bestknown examples of socalled NPhard problems. No polynomialtime algorithm is known for any NPhard problem. Moreover, most computer scientists believe that such algorithms do not exist, although this very important conjecture has never been proven. Moresophisticated approaches—backtracking and branchandbound (see Sections 12.1 and 12.2)—enable us to solve some but not all instances of these and
118
Brute Force and Exhaustive Search
10
knapsack
w1 = 7 v1 = $42
w2 = 3 v2 = $12
w3 = 4 v3 = $40
w4 = 5 v4 = $25
item 1
item 2
item 3
item 4
(a)
Subset
Total weight
Total value
∅ {1} {2} {3} {4} {1, 2} {1, 3} {1, 4} {2, 3} {2, 4} 3, 4 {1, 2, 3} {1, 2, 4} {1, 3, 4} {2, 3, 4} {1, 2, 3, 4}
0 7 3 4 5 10 11 12 7 8 9 14 15 16 12 19
$0 $42 $12 $40 $25 $54 not feasible not feasible $52 $37 $65 not feasible not feasible not feasible not feasible not feasible
(b) FIGURE 3.8 (a) Instance of the knapsack problem. (b) Its solution by exhaustive search. The information about the optimal selection is in bold.
3.4
119
Exhaustive Search
similar problems in less than exponential time. Alternatively, we can use one of many approximation algorithms, such as those described in Section 12.3.
Assignment Problem In our third example of a problem that can be solved by exhaustive search, there are n people who need to be assigned to execute n jobs, one person per job. (That is, each person is assigned to exactly one job and each job is assigned to exactly one person.) The cost that would accrue if the ith person is assigned to the j th job is a known quantity C[i, j ] for each pair i, j = 1, 2, . . . , n. The problem is to ﬁnd an assignment with the minimum total cost. A small instance of this problem follows, with the table entries representing the assignment costs C[i, j ]:
Person 1 Person 2 Person 3 Person 4
Job 1
Job 2
Job 3
Job 4
9 6 5 7
2 4 8 6
7 3 1 9
8 7 8 4
It is easy to see that an instance of the assignment problem is completely speciﬁed by its cost matrix C. In terms of this matrix, the problem is to select one element in each row of the matrix so that all selected elements are in different columns and the total sum of the selected elements is the smallest possible. Note that no obvious strategy for ﬁnding a solution works here. For example, we cannot select the smallest element in each row, because the smallest elements may happen to be in the same column. In fact, the smallest element in the entire matrix need not be a component of an optimal solution. Thus, opting for the exhaustive search may appear as an unavoidable evil. We can describe feasible solutions to the assignment problem as ntuples j1, . . . , jn in which the ith component, i = 1, . . . , n, indicates the column of the element selected in the ith row (i.e., the job number assigned to the ith person). For example, for the cost matrix above, 2, 3, 4, 1 indicates the assignment of Person 1 to Job 2, Person 2 to Job 3, Person 3 to Job 4, and Person 4 to Job 1. The requirements of the assignment problem imply that there is a onetoone correspondence between feasible assignments and permutations of the ﬁrst n integers. Therefore, the exhaustivesearch approach to the assignment problem would require generating all the permutations of integers 1, 2, . . . , n, computing the total cost of each assignment by summing up the corresponding elements of the cost matrix, and ﬁnally selecting the one with the smallest sum. A few ﬁrst iterations of applying this algorithm to the instance given above are shown in Figure 3.9; you are asked to complete it in the exercises.
120
Brute Force and Exhaustive Search
C=
9 6 5 7
2 4 8 6
7 3 1 9
8 7 8 4
cost = 9 + 4 + 1 + 4 = 18 cost = 9 + 4 + 8 + 9 = 30 cost = 9 + 3 + 8 + 4 = 24 cost = 9 + 3 + 8 + 6 = 26 cost = 9 + 7 + 8 + 9 = 33 cost = 9 + 7 + 1 + 6 = 23
etc.
FIGURE 3.9 First few iterations of solving a small instance of the assignment problem by exhaustive search.
Since the number of permutations to be considered for the general case of the assignment problem is n!, exhaustive search is impractical for all but very small instances of the problem. Fortunately, there is a much more efﬁcient algorithm for this problem called the Hungarian method after the Hungarian mathematicians ¨ ´ whose work underlies the method (see, e.g., [Kol95]). Konig and Egervary, This is good news: the fact that a problem domain grows exponentially or faster does not necessarily imply that there can be no efﬁcient algorithm for solving it. In fact, we present several other examples of such problems later in the book. However, such examples are more of an exception to the rule. More often than not, there are no known polynomialtime algorithms for problems whose domain grows exponentially with instance size, provided we want to solve them exactly. And, as we mentioned above, such algorithms quite possibly do not exist.
Exercises 3.4 1. a. Assuming that each tour can be generated in constant time, what will be the efﬁciency class of the exhaustivesearch algorithm outlined in the text for the traveling salesman problem? b. If this algorithm is programmed on a computer that makes ten billion additions per second, estimate the maximum number of cities for which the problem can be solved in i. 1 hour. ii. 24 hours. iii. 1 year. iv. 1 century. 2. Outline an exhaustivesearch algorithm for the Hamiltonian circuit problem. 3. Outline an algorithm to determine whether a connected graph represented by its adjacency matrix has an Eulerian circuit. What is the efﬁciency class of your algorithm? 4. Complete the application of exhaustive search to the instance of the assignment problem started in the text. 5. Give an example of the assignment problem whose optimal solution does not include the smallest element of its cost matrix.
3.4
Exhaustive Search
121
6. Consider the partition problem: given n positive integers, partition them into two disjoint subsets with the same sum of their elements. (Of course, the problem does not always have a solution.) Design an exhaustivesearch algorithm for this problem. Try to minimize the number of subsets the algorithm needs to generate. 7. Consider the clique problem: given a graph G and a positive integer k, determine whether the graph contains a clique of size k, i.e., a complete subgraph of k vertices. Design an exhaustivesearch algorithm for this problem. 8. Explain how exhaustive search can be applied to the sorting problem and determine the efﬁciency class of such an algorithm. 9. Eightqueens problem Consider the classic puzzle of placing eight queens on an 8 × 8 chessboard so that no two queens are in the same row or in the same column or on the same diagonal. How many different positions are there so that a. no two queens are on the same square? b. no two queens are in the same row? c. no two queens are in the same row or in the same column? Also estimate how long it would take to ﬁnd all the solutions to the problem by exhaustive search based on each of these approaches on a computer capable of checking 10 billion positions per second. 10. Magic squares A magic square of order n is an arrangement of the integers from 1 to n2 in an n × n matrix, with each number occurring exactly once, so that each row, each column, and each main diagonal has the same sum. a. Prove that if a magic square of order n exists, the sum in question must be equal to n(n2 + 1)/2. b. Design an exhaustivesearch algorithm for generating all magic squares of order n. c. Go to the Internet or your library and ﬁnd a better algorithm for generating magic squares. d. Implement the two algorithms—the exhaustive search and the one you have found—and run an experiment to determine the largest value of n for which each of the algorithms is able to ﬁnd a magic square of order n in less than 1 minute on your computer. 11. Famous alphametic A puzzle in which the digits in a correct mathematical expression, such as a sum, are replaced by letters is called cryptarithm; if, in addition, the puzzle’s words make sense, it is said to be an alphametic. The most wellknown alphametic was published by the renowned British puzzlist Henry E. Dudeney (1857–1930):
122
Brute Force and Exhaustive Search
S END + MORE MO NEY
Two conditions are assumed: ﬁrst, the correspondence between letters and decimal digits is onetoone, i.e., each letter represents one digit only and different letters represent different digits. Second, the digit zero does not appear as the leftmost digit in any of the numbers. To solve an alphametic means to ﬁnd which digit each letter represents. Note that a solution’s uniqueness cannot be assumed and has to be veriﬁed by the solver. a. Write a program for solving cryptarithms by exhaustive search. Assume that a given cryptarithm is a sum of two words. b. Solve Dudeney’s puzzle the way it was expected to be solved when it was ﬁrst published in 1924.
3.5 DepthFirst Search and BreadthFirst Search The term “exhaustive search” can also be applied to two very important algorithms that systematically process all vertices and edges of a graph. These two traversal algorithms are depthﬁrst search (DFS) and breadthﬁrst search (BFS). These algorithms have proved to be very useful for many applications involving graphs in artiﬁcial intelligence and operations research. In addition, they are indispensable for efﬁcient investigation of fundamental properties of graphs such as connectivity and cycle presence.
DepthFirst Search Depthﬁrst search starts a graph’s traversal at an arbitrary vertex by marking it as visited. On each iteration, the algorithm proceeds to an unvisited vertex that is adjacent to the one it is currently in. (If there are several such vertices, a tie can be resolved arbitrarily. As a practical matter, which of the adjacent unvisited candidates is chosen is dictated by the data structure representing the graph. In our examples, we always break ties by the alphabetical order of the vertices.) This process continues until a dead end—a vertex with no adjacent unvisited vertices— is encountered. At a dead end, the algorithm backs up one edge to the vertex it came from and tries to continue visiting unvisited vertices from there. The algorithm eventually halts after backing up to the starting vertex, with the latter being a dead end. By then, all the vertices in the same connected component as the starting vertex have been visited. If unvisited vertices still remain, the depthﬁrst search must be restarted at any one of them. It is convenient to use a stack to trace the operation of depthﬁrst search. We push a vertex onto the stack when the vertex is reached for the ﬁrst time (i.e., the
3.5
123
DepthFirst Search and BreadthFirst Search
g
a
g
c
h
h a
e c
d3, 1 c2, 5 a1, 6
f
d
b
j
e 6, 2 b 5, 3 f4, 4
j10,7 i 9, 8 h8, 9 g7,10
d
f
i
b
j
i e
(a)
(b)
(c)
FIGURE 3.10 Example of a DFS traversal. (a) Graph. (b) Traversal’s stack (the ﬁrst subscript number indicates the order in which a vertex is visited, i.e., pushed onto the stack; the second one indicates the order in which it becomes a deadend, i.e., popped off the stack). (c) DFS forest with the tree and back edges shown with solid and dashed lines, respectively.
visit of the vertex starts), and we pop a vertex off the stack when it becomes a dead end (i.e., the visit of the vertex ends). It is also very useful to accompany a depthﬁrst search traversal by constructing the socalled depthﬁrst search forest. The starting vertex of the traversal serves as the root of the ﬁrst tree in such a forest. Whenever a new unvisited vertex is reached for the ﬁrst time, it is attached as a child to the vertex from which it is being reached. Such an edge is called a tree edge because the set of all such edges forms a forest. The algorithm may also encounter an edge leading to a previously visited vertex other than its immediate predecessor (i.e., its parent in the tree). Such an edge is called a back edge because it connects a vertex to its ancestor, other than the parent, in the depthﬁrst search forest. Figure 3.10 provides an example of a depthﬁrst search traversal, with the traversal stack and corresponding depthﬁrst search forest shown as well. Here is pseudocode of the depthﬁrst search. ALGORITHM
DFS(G)
//Implements a depthﬁrst search traversal of a given graph //Input: Graph G = V , E //Output: Graph G with its vertices marked with consecutive integers // in the order they are ﬁrst encountered by the DFS traversal mark each vertex in V with 0 as a mark of being “unvisited” count ← 0 for each vertex v in V do if v is marked with 0 dfs(v)
124
Brute Force and Exhaustive Search
dfs(v) //visits recursively all the unvisited vertices connected to vertex v //by a path and numbers them in the order they are encountered //via global variable count count ← count + 1; mark v with count for each vertex w in V adjacent to v do if w is marked with 0 dfs(w) The brevity of the DFS pseudocode and the ease with which it can be performed by hand may create a wrong impression about the level of sophistication of this algorithm. To appreciate its true power and depth, you should trace the algorithm’s action by looking not at a graph’s diagram but at its adjacency matrix or adjacency lists. (Try it for the graph in Figure 3.10 or a smaller example.) How efﬁcient is depthﬁrst search? It is not difﬁcult to see that this algorithm is, in fact, quite efﬁcient since it takes just the time proportional to the size of the data structure used for representing the graph in question. Thus, for the adjacency matrix representation, the traversal time is in (V 2 ), and for the adjacency list representation, it is in (V  + E) where V  and E are the number of the graph’s vertices and edges, respectively. A DFS forest, which is obtained as a byproduct of a DFS traversal, deserves a few comments, too. To begin with, it is not actually a forest. Rather, we can look at it as the given graph with its edges classiﬁed by the DFS traversal into two disjoint classes: tree edges and back edges. (No other types are possible for a DFS forest of an undirected graph.) Again, tree edges are edges used by the DFS traversal to reach previously unvisited vertices. If we consider only the edges in this class, we will indeed get a forest. Back edges connect vertices to previously visited vertices other than their immediate predecessors in the traversal. They connect vertices to their ancestors in the forest other than their parents. A DFS traversal itself and the forestlike representation of the graph it provides have proved to be extremely helpful for the development of efﬁcient algorithms for checking many important properties of graphs.3 Note that the DFS yields two orderings of vertices: the order in which the vertices are reached for the ﬁrst time (pushed onto the stack) and the order in which the vertices become dead ends (popped off the stack). These orders are qualitatively different, and various applications can take advantage of either of them. Important elementary applications of DFS include checking connectivity and checking acyclicity of a graph. Since dfs halts after visiting all the vertices con3.
The discovery of several such applications was an important breakthrough achieved by the two American computer scientists John Hopcroft and Robert Tarjan in the 1970s. For this and other contributions, they were given the Turing Award—the most prestigious prize in the computing ﬁeld [Hop87, Tar87].
3.5
DepthFirst Search and BreadthFirst Search
125
nected by a path to the starting vertex, checking a graph’s connectivity can be done as follows. Start a DFS traversal at an arbitrary vertex and check, after the algorithm halts, whether all the vertices of the graph will have been visited. If they have, the graph is connected; otherwise, it is not connected. More generally, we can use DFS for identifying connected components of a graph (how?). As for checking for a cycle presence in a graph, we can take advantage of the graph’s representation in the form of a DFS forest. If the latter does not have back edges, the graph is clearly acyclic. If there is a back edge from some vertex u to its ancestor v (e.g., the back edge from d to a in Figure 3.10c), the graph has a cycle that comprises the path from v to u via a sequence of tree edges in the DFS forest followed by the back edge from u to v. You will ﬁnd a few other applications of DFS later in the book, although more sophisticated applications, such as ﬁnding articulation points of a graph, are not included. (A vertex of a connected graph is said to be its articulation point if its removal with all edges incident to it breaks the graph into disjoint pieces.)
BreadthFirst Search If depthﬁrst search is a traversal for the brave (the algorithm goes as far from “home” as it can), breadthﬁrst search is a traversal for the cautious. It proceeds in a concentric manner by visiting ﬁrst all the vertices that are adjacent to a starting vertex, then all unvisited vertices two edges apart from it, and so on, until all the vertices in the same connected component as the starting vertex are visited. If there still remain unvisited vertices, the algorithm has to be restarted at an arbitrary vertex of another connected component of the graph. It is convenient to use a queue (note the difference from depthﬁrst search!) to trace the operation of breadthﬁrst search. The queue is initialized with the traversal’s starting vertex, which is marked as visited. On each iteration, the algorithm identiﬁes all unvisited vertices that are adjacent to the front vertex, marks them as visited, and adds them to the queue; after that, the front vertex is removed from the queue. Similarly to a DFS traversal, it is useful to accompany a BFS traversal by constructing the socalled breadthﬁrst search forest. The traversal’s starting vertex serves as the root of the ﬁrst tree in such a forest. Whenever a new unvisited vertex is reached for the ﬁrst time, the vertex is attached as a child to the vertex it is being reached from with an edge called a tree edge. If an edge leading to a previously visited vertex other than its immediate predecessor (i.e., its parent in the tree) is encountered, the edge is noted as a cross edge. Figure 3.11 provides an example of a breadthﬁrst search traversal, with the traversal queue and corresponding breadthﬁrst search forest shown.
126
Brute Force and Exhaustive Search
g
h a a
g
e c
a1 c2 d3 e4 f5 b6 g7 h8 j9 i10
f b
d j
c f
d
e
h
b
i
j
i
(a)
(b)
(c)
FIGURE 3.11 Example of a BFS traversal. (a) Graph. (b) Traversal queue, with the numbers indicating the order in which the vertices are visited, i.e., added to (and removed from) the queue. (c) BFS forest with the tree and cross edges shown with solid and dotted lines, respectively.
Here is pseudocode of the breadthﬁrst search. ALGORITHM
BFS(G)
//Implements a breadthﬁrst search traversal of a given graph //Input: Graph G = V , E //Output: Graph G with its vertices marked with consecutive integers // in the order they are visited by the BFS traversal mark each vertex in V with 0 as a mark of being “unvisited” count ← 0 for each vertex v in V do if v is marked with 0 bfs(v) bfs(v) //visits all the unvisited vertices connected to vertex v //by a path and numbers them in the order they are visited //via global variable count count ← count + 1; mark v with count and initialize a queue with v while the queue is not empty do for each vertex w in V adjacent to the front vertex do if w is marked with 0 count ← count + 1; mark w with count add w to the queue remove the front vertex from the queue
3.5
127
DepthFirst Search and BreadthFirst Search
a a
e
b
c
f
g
d
c
h d
(a)
e
b f g
(b)
FIGURE 3.12 Illustration of the BFSbased algorithm for ﬁnding a minimumedge path. (a) Graph. (b) Part of its BFS tree that identiﬁes the minimumedge path from a to g.
Breadthﬁrst search has the same efﬁciency as depthﬁrst search: it is in (V 2 ) for the adjacency matrix representation and in (V  + E) for the adjacency list representation. Unlike depthﬁrst search, it yields a single ordering of vertices because the queue is a FIFO (ﬁrstin ﬁrstout) structure and hence the order in which vertices are added to the queue is the same order in which they are removed from it. As to the structure of a BFS forest of an undirected graph, it can also have two kinds of edges: tree edges and cross edges. Tree edges are the ones used to reach previously unvisited vertices. Cross edges connect vertices to those visited before, but, unlike back edges in a DFS tree, they connect vertices either on the same or adjacent levels of a BFS tree. BFS can be used to check connectivity and acyclicity of a graph, essentially in the same manner as DFS can. It is not applicable, however, for several less straightforward applications such as ﬁnding articulation points. On the other hand, it can be helpful in some situations where DFS cannot. For example, BFS can be used for ﬁnding a path with the fewest number of edges between two given vertices. To do this, we start a BFS traversal at one of the two vertices and stop it as soon as the other vertex is reached. The simple path from the root of the BFS tree to the second vertex is the path sought. For example, path a − b − c − g in the graph in Figure 3.12 has the fewest number of edges among all the paths between vertices a and g. Although the correctness of this application appears to stem immediately from the way BFS operates, a mathematical proof of its validity is not quite elementary (see, e.g., [Cor09, Section 22.2]). Table 3.1 summarizes the main facts about depthﬁrst search and breadthﬁrst search.
128
Brute Force and Exhaustive Search TABLE 3.1 Main facts about depthﬁrst search (DFS) and breadthﬁrst search (BFS)
Data structure Number of vertex orderings Edge types (undirected graphs) Applications
Efﬁciency for adjacency matrix Efﬁciency for adjacency lists
DFS
BFS
a stack two orderings tree and back edges connectivity, acyclicity, articulation points (V 2 ) (V  + E)
a queue one ordering tree and cross edges connectivity, acyclicity, minimumedge paths (V 2 ) (V  + E)
Exercises 3.5 1. Consider the following graph. c
b
f d
a
g e
a. Write down the adjacency matrix and adjacency lists specifying this graph. (Assume that the matrix rows and columns and vertices in the adjacency lists follow in the alphabetical order of the vertex labels.) b. Starting at vertex a and resolving ties by the vertex alphabetical order, traverse the graph by depthﬁrst search and construct the corresponding depthﬁrst search tree. Give the order in which the vertices were reached for the ﬁrst time (pushed onto the traversal stack) and the order in which the vertices became dead ends (popped off the stack). 2. If we deﬁne sparse graphs as graphs for which E ∈ O(V ), which implementation of DFS will have a better time efﬁciency for such graphs, the one that uses the adjacency matrix or the one that uses the adjacency lists? 3. Let G be a graph with n vertices and m edges. a. True or false: All its DFS forests (for traversals starting at different vertices) will have the same number of trees? b. True or false: All its DFS forests will have the same number of tree edges and the same number of back edges? 4. Traverse the graph of Problem 1 by breadthﬁrst search and construct the corresponding breadthﬁrst search tree. Start the traversal at vertex a and resolve ties by the vertex alphabetical order.
3.5
DepthFirst Search and BreadthFirst Search
129
5. Prove that a cross edge in a BFS tree of an undirected graph can connect vertices only on either the same level or on two adjacent levels of a BFS tree. 6. a. Explain how one can check a graph’s acyclicity by using breadthﬁrst search. b. Does either of the two traversals—DFS or BFS—always ﬁnd a cycle faster than the other? If you answer yes, indicate which of them is better and explain why it is the case; if you answer no, give two examples supporting your answer. 7. Explain how one can identify connected components of a graph by using a. a depthﬁrst search. b. a breadthﬁrst search. 8. A graph is said to be bipartite if all its vertices can be partitioned into two disjoint subsets X and Y so that every edge connects a vertex in X with a vertex in Y . (One can also say that a graph is bipartite if its vertices can be colored in two colors so that every edge has its vertices colored in different colors; such graphs are also called 2colorable.) For example, graph (i) is bipartite while graph (ii) is not. x1
y1
x3
a
b
y2
x2
y3
c
d
(i)
(ii)
a. Design a DFSbased algorithm for checking whether a graph is bipartite. b. Design a BFSbased algorithm for checking whether a graph is bipartite. 9. Write a program that, for a given graph, outputs: a. vertices of each connected component b. its cycle or a message that the graph is acyclic 10. One can model a maze by having a vertex for a starting point, a ﬁnishing point, dead ends, and all the points in the maze where more than one path can be taken, and then connecting the vertices according to the paths in the maze. a. Construct such a graph for the following maze.
130
Brute Force and Exhaustive Search
b. Which traversal—DFS or BFS—would you use if you found yourself in a maze and why? ´ Denis Poisson (1781–1840), a famous French mathemati11. Three Jugs Simeon cian and physicist, is said to have become interested in mathematics after encountering some version of the following old puzzle. Given an 8pint jug full of water and two empty jugs of 5 and 3pint capacity, get exactly 4 pints of water in one of the jugs by completely ﬁlling up and/or emptying jugs into others. Solve this puzzle by using breadthﬁrst search.
SUMMARY Brute force is a straightforward approach to solving a problem, usually directly based on the problem statement and deﬁnitions of the concepts involved. The principal strengths of the bruteforce approach are wide applicability and simplicity; its principal weakness is the subpar efﬁciency of most bruteforce algorithms. A ﬁrst application of the bruteforce approach often results in an algorithm that can be improved with a modest amount of effort. The following noted algorithms can be considered as examples of the bruteforce approach: . deﬁnitionbased algorithm for matrix multiplication . selection sort . sequential search . straightforward stringmatching algorithm Exhaustive search is a bruteforce approach to combinatorial problems. It suggests generating each and every combinatorial object of the problem, selecting those of them that satisfy all the constraints, and then ﬁnding a desired object. The traveling salesman problem, the knapsack problem, and the assignment problem are typical examples of problems that can be solved, at least theoretically, by exhaustivesearch algorithms. Exhaustive search is impractical for all but very small instances of problems it can be applied to. Depthﬁrst search (DFS) and breadthﬁrst search (BFS) are two principal graphtraversal algorithms. By representing a graph in a form of a depthﬁrst or breadthﬁrst search forest, they help in the investigation of many important properties of the graph. Both algorithms have the same time efﬁciency: (V 2 ) for the adjacency matrix representation and (V  + E) for the adjacency list representation.
4 DecreaseandConquer Plutarch says that Sertorius, in order to teach his soldiers that perseverance and wit are better than brute force, had two horses brought before them, and set two men to pull out their tails. One of the men was a burly Hercules, who tugged and tugged, but all to no purpose; the other was a sharp, weaselfaced tailor, who plucked one hair at a time, amidst roars of laughter, and soon left the tail quite bare. —E. Cobham Brewer, Dictionary of Phrase and Fable, 1898
T
he decreaseandconquer technique is based on exploiting the relationship between a solution to a given instance of a problem and a solution to its smaller instance. Once such a relationship is established, it can be exploited either top down or bottom up. The former leads naturally to a recursive implementation, although, as one can see from several examples in this chapter, an ultimate implementation may well be nonrecursive. The bottomup variation is usually implemented iteratively, starting with a solution to the smallest instance of the problem; it is called sometimes the incremental approach. There are three major variations of decreaseandconquer: decrease by a constant decrease by a constant factor variable size decrease In the decreasebyaconstant variation, the size of an instance is reduced by the same constant on each iteration of the algorithm. Typically, this constant is equal to one (Figure 4.1), although other constant size reductions do happen occasionally. Consider, as an example, the exponentiation problem of computing a n where a = 0 and n is a nonnegative integer. The relationship between a solution to an instance of size n and an instance of size n − 1 is obtained by the obvious formula a n = a n−1 . a. So the function f (n) = a n can be computed either “top down” by using its recursive deﬁnition 131
132
DecreaseandConquer
problem of size n
subproblem of size n –1
solution to the subproblem
solution to the original problem
FIGURE 4.1 Decrease(by one)andconquer technique.
f (n) =
f (n − 1) . a 1
if n > 0, if n = 0,
(4.1)
or “bottom up” by multiplying 1 by a n times. (Yes, it is the same as the bruteforce algorithm, but we have come to it by a different thought process.) More interesting examples of decreasebyone algorithms appear in Sections 4.1–4.3. The decreasebyaconstantfactor technique suggests reducing a problem instance by the same constant factor on each iteration of the algorithm. In most applications, this constant factor is equal to two. (Can you give an example of such an algorithm?) The decreasebyhalf idea is illustrated in Figure 4.2. For an example, let us revisit the exponentiation problem. If the instance of size n is to compute a n, the instance of half its size is to compute a n/2 , with the obvious relationship between the two: a n = (a n/2 )2 . But since we consider here instances with integer exponents only, the former does not work for odd n. If n is odd, we have to compute a n−1 by using the rule for evenvalued exponents and then multiply the result by a. To summarize, we have the following formula:
DecreaseandConquer
problem
133
of size n
subproblem of size n/2
solution to the subproblem
solution to the original problem
FIGURE 4.2 Decrease(by half)andconquer technique.
⎧ ⎨ (a n/2 )2 n a = (a (n−1)/2 )2 . a ⎩ 1
if n is even and positive, if n is odd, if n = 0.
(4.2)
If we compute a n recursively according to formula (4.2) and measure the algorithm’s efﬁciency by the number of multiplications, we should expect the algorithm to be in (log n) because, on each iteration, the size is reduced by about a half at the expense of one or two multiplications. A few other examples of decreasebyaconstantfactor algorithms are given in Section 4.4 and its exercises. Such algorithms are so efﬁcient, however, that there are few examples of this kind. Finally, in the variablesizedecrease variety of decreaseandconquer, the sizereduction pattern varies from one iteration of an algorithm to another. Euclid’s algorithm for computing the greatest common divisor provides a good example of such a situation. Recall that this algorithm is based on the formula gcd(m, n) = gcd(n, m mod n).
134
DecreaseandConquer
Though the value of the second argument is always smaller on the righthand side than on the lefthand side, it decreases neither by a constant nor by a constant factor. A few other examples of such algorithms appear in Section 4.5.
4.1 Insertion Sort In this section, we consider an application of the decreasebyone technique to sorting an array A[0..n − 1]. Following the technique’s idea, we assume that the smaller problem of sorting the array A[0..n − 2] has already been solved to give us a sorted array of size n − 1: A[0] ≤ . . . ≤ A[n − 2]. How can we take advantage of this solution to the smaller problem to get a solution to the original problem by taking into account the element A[n − 1]? Obviously, all we need is to ﬁnd an appropriate position for A[n − 1] among the sorted elements and insert it there. This is usually done by scanning the sorted subarray from right to left until the ﬁrst element smaller than or equal to A[n − 1] is encountered to insert A[n − 1] right after that element. The resulting algorithm is called straight insertion sort or simply insertion sort. Though insertion sort is clearly based on a recursive idea, it is more efﬁcient to implement this algorithm bottom up, i.e., iteratively. As shown in Figure 4.3, starting with A[1] and ending with A[n − 1], A[i] is inserted in its appropriate place among the ﬁrst i elements of the array that have been already sorted (but, unlike selection sort, are generally not in their ﬁnal positions). Here is pseudocode of this algorithm. ALGORITHM
InsertionSort(A[0..n − 1])
//Sorts a given array by insertion sort //Input: An array A[0..n − 1] of n orderable elements //Output: Array A[0..n − 1] sorted in nondecreasing order for i ← 1 to n − 1 do v ← A[i] j ←i−1 while j ≥ 0 and A[j ] > v do A[j + 1] ← A[j ] j ←j −1 A[j + 1] ← v
A[0] ≤ . . . ≤ A[ j] < A[ j + 1] ≤ . . . ≤ A[i – 1] ⏐ A[i] . . . A[n – 1] smaller than or equal to A[i]
greater than A[i]
FIGURE 4.3 Iteration of insertion sort: A[i] is inserted in its proper position among the preceding elements previously sorted.
4.1 Insertion Sort
135
89  45 68 90 29 34 17 45 89  68 90 29 34 17 45 68 89  90 29 34 17 45 68 89 90  29 34 17 29 45 68 89 90  34 17 29 34 45 68 89 90  17 17 29 34 45 68 89 90
FIGURE 4.4 Example of sorting with insertion sort. A vertical bar separates the sorted part of the array from the remaining elements; the element being inserted is in bold.
The operation of the algorithm is illustrated in Figure 4.4. The basic operation of the algorithm is the key comparison A[j ] > v. (Why not j ≥ 0? Because it is almost certainly faster than the former in an actual computer implementation. Moreover, it is not germane to the algorithm: a better implementation with a sentinel—see Problem 8 in this section’s exercises—eliminates it altogether.) The number of key comparisons in this algorithm obviously depends on the nature of the input. In the worst case, A[j ] > v is executed the largest number of times, i.e., for every j = i − 1, . . . , 0. Since v = A[i], it happens if and only if A[j ] > A[i] for j = i − 1, . . . , 0. (Note that we are using the fact that on the ith iteration of insertion sort all the elements preceding A[i] are the ﬁrst i elements in the input, albeit in the sorted order.) Thus, for the worstcase input, we get A[0] > A[1] (for i = 1), A[1] > A[2] (for i = 2), . . . , A[n − 2] > A[n − 1] (for i = n − 1). In other words, the worstcase input is an array of strictly decreasing values. The number of key comparisons for such an input is Cworst (n) =
i−1 n−1
1=
i=1 j =0
n−1
i=1
i=
(n − 1)n ∈ (n2 ). 2
Thus, in the worst case, insertion sort makes exactly the same number of comparisons as selection sort (see Section 3.1). In the best case, the comparison A[j ] > v is executed only once on every iteration of the outer loop. It happens if and only if A[i − 1] ≤ A[i] for every i = 1, . . . , n − 1, i.e., if the input array is already sorted in nondecreasing order. (Though it “makes sense” that the best case of an algorithm happens when the problem is already solved, it is not always the case, as you are going to see in our discussion of quicksort in Chapter 5.) Thus, for sorted arrays, the number of key comparisons is Cbest (n) =
n−1
i=1
1 = n − 1 ∈ (n).
136
DecreaseandConquer
This very good performance in the best case of sorted arrays is not very useful by itself, because we cannot expect such convenient inputs. However, almostsorted ﬁles do arise in a variety of applications, and insertion sort preserves its excellent performance on such inputs. A rigorous analysis of the algorithm’s averagecase efﬁciency is based on investigating the number of element pairs that are out of order (see Problem 11 in this section’s exercises). It shows that on randomly ordered arrays, insertion sort makes on average half as many comparisons as on decreasing arrays, i.e., Cavg (n) ≈
n2 ∈ (n2 ). 4
This twiceasfast averagecase performance coupled with an excellent efﬁciency on almostsorted arrays makes insertion sort stand out among its principal competitors among elementary sorting algorithms, selection sort and bubble sort. In addition, its extension named shellsort, after its inventor D. L. Shell [She59], gives us an even better algorithm for sorting moderately large ﬁles (see Problem 12 in this section’s exercises).
Exercises 4.1 1. Ferrying soldiers A detachment of n soldiers must cross a wide and deep river with no bridge in sight. They notice two 12yearold boys playing in a rowboat by the shore. The boat is so tiny, however, that it can only hold two boys or one soldier. How can the soldiers get across the river and leave the boys in joint possession of the boat? How many times need the boat pass from shore to shore? 2. Alternating glasses a. There are 2n glasses standing next to each other in a row, the ﬁrst n of them ﬁlled with a soda drink and the remaining n glasses empty. Make the glasses alternate in a ﬁlledemptyﬁlledempty pattern in the minimum number of glass moves. [Gar78]
b. Solve the same problem if 2n glasses—n with a drink and n empty—are initially in a random order. 3. Marking cells Design an algorithm for the following task. For any even n, mark n cells on an inﬁnite sheet of graph paper so that each marked cell has an odd number of marked neighbors. Two cells are considered neighbors if they are next to each other either horizontally or vertically but not diagonally. The marked cells must form a contiguous region, i.e., a region in which there is a path between any pair of marked cells that goes through a sequence of marked neighbors. [Kor05]
4.1 Insertion Sort
137
4. Design a decreasebyone algorithm for generating the power set of a set of n elements. (The power set of a set S is the set of all the subsets of S, including the empty set and S itself.) 5. Consider the following algorithm to check connectivity of a graph deﬁned by its adjacency matrix. ALGORITHM Connected(A[0..n − 1, 0..n − 1]) //Input: Adjacency matrix A[0..n − 1, 0..n − 1]) of an undirected graph G //Output: 1 (true) if G is connected and 0 (false) if it is not if n = 1 return 1 //onevertex graph is connected by deﬁnition else if not Connected(A[0..n − 2, 0..n − 2]) return 0 else for j ← 0 to n − 2 do if A[n − 1, j ] return 1 return 0 Does this algorithm work correctly for every undirected graph with n > 0 vertices? If you answer yes, indicate the algorithm’s efﬁciency class in the worst case; if you answer no, explain why. 6. Team ordering You have the results of a completed roundrobin tournament in which n teams played each other once. Each game ended either with a victory for one of the teams or with a tie. Design an algorithm that lists the teams in a sequence so that every team did not lose the game with the team listed immediately after it. What is the time efﬁciency class of your algorithm? 7. Apply insertion sort to sort the list E, X, A, M, P , L, E in alphabetical order. 8. a. What sentinel should be put before the ﬁrst element of an array being sorted in order to avoid checking the inbound condition j ≥ 0 on each iteration of the inner loop of insertion sort? b. Is the sentinel version in the same efﬁciency class as the original version? 9. Is it possible to implement insertion sort for sorting linked lists? Will it have the same O(n2 ) time efﬁciency as the array version? 10. Compare the text’s implementation of insertion sort with the following version. ALGORITHM InsertSort2(A[0..n − 1]) for i ← 1 to n − 1 do j ←i−1 while j ≥ 0 and A[j ] > A[j + 1] do swap(A[j ], A[j + 1]) j ←j −1
138
DecreaseandConquer
What is the time efﬁciency of this algorithm? How is it compared to that of the version given in Section 4.1? 11. Let A[0..n − 1] be an array of n sortable elements. (For simplicity, you may assume that all the elements are distinct.) A pair (A[i], A[j ]) is called an inversion if i < j and A[i] > A[j ]. a. What arrays of size n have the largest number of inversions and what is this number? Answer the same questions for the smallest number of inversions. b. Show that the averagecase number of key comparisons in insertion sort is given by the formula Cavg (n) ≈
n2 . 4
12. Shellsort (more accurately Shell’s sort) is an important sorting algorithm that works by applying insertion sort to each of several interleaving sublists of a given list. On each pass through the list, the sublists in question are formed by stepping through the list with an increment hi taken from some predeﬁned decreasing sequence of step sizes, h1 > . . . > hi > . . . > 1, which must end with 1. (The algorithm works for any such sequence, though some sequences are known to yield a better efﬁciency than others. For example, the sequence 1, 4, 13, 40, 121, . . . , used, of course, in reverse, is known to be among the best for this purpose.) a. Apply shellsort to the list S, H, E, L, L, S, O, R, T , I, S, U, S, E, F, U, L b. Is shellsort a stable sorting algorithm? c. Implement shellsort, straight insertion sort, selection sort, and bubble sort in the language of your choice and compare their performance on random arrays of sizes 10n for n = 2, 3, 4, 5, and 6 as well as on increasing and decreasing arrays of these sizes.
4.2 Topological Sorting In this section, we discuss an important problem for directed graphs, with a variety of applications involving prerequisiterestricted tasks. Before we pose this problem, though, let us review a few basic facts about directed graphs themselves. A directed graph, or digraph for short, is a graph with directions speciﬁed for all its edges (Figure 4.5a is an example). The adjacency matrix and adjacency lists are still two principal means of representing a digraph. There are only two notable differences between undirected and directed graphs in representing them: (1) the adjacency matrix of a directed graph does not have to be symmetric; (2) an edge in a directed graph has just one (not two) corresponding nodes in the digraph’s adjacency lists.
4.2 Topological Sorting
a
a
d
c
b
e
d
c
b
139
e
(a)
(b)
FIGURE 4.5 (a) Digraph. (b) DFS forest of the digraph for the DFS traversal started at a.
Depthﬁrst search and breadthﬁrst search are principal traversal algorithms for traversing digraphs as well, but the structure of corresponding forests can be more complex than for undirected graphs. Thus, even for the simple example of Figure 4.5a, the depthﬁrst search forest (Figure 4.5b) exhibits all four types of edges possible in a DFS forest of a directed graph: tree edges (ab, bc, de), back edges (ba) from vertices to their ancestors, forward edges (ac) from vertices to their descendants in the tree other than their children, and cross edges (dc), which are none of the aforementioned types. Note that a back edge in a DFS forest of a directed graph can connect a vertex to its parent. Whether or not it is the case, the presence of a back edge indicates that the digraph has a directed cycle. A directed cycle in a digraph is a sequence of three or more of its vertices that starts and ends with the same vertex and in which every vertex is connected to its immediate predecessor by an edge directed from the predecessor to the successor. For example, a, b, a is a directed cycle in the digraph in Figure 4.5a. Conversely, if a DFS forest of a digraph has no back edges, the digraph is a dag, an acronym for directed acyclic graph. Edge directions lead to new questions about digraphs that are either meaningless or trivial for undirected graphs. In this section, we discuss one such question. As a motivating example, consider a set of ﬁve required courses {C1, C2, C3, C4, C5} a parttime student has to take in some degree program. The courses can be taken in any order as long as the following course prerequisites are met: C1 and C2 have no prerequisites, C3 requires C1 and C2, C4 requires C3, and C5 requires C3 and C4. The student can take only one course per term. In which order should the student take the courses? The situation can be modeled by a digraph in which vertices represent courses and directed edges indicate prerequisite requirements (Figure 4.6). In terms of this digraph, the question is whether we can list its vertices in such an order that for every edge in the graph, the vertex where the edge starts is listed before the vertex where the edge ends. (Can you ﬁnd such an ordering of this digraph’s vertices?) This problem is called topological sorting. It can be posed for an
140
DecreaseandConquer
C1
C4 C3
C2
C5
FIGURE 4.6 Digraph representing the prerequisite structure of ﬁve courses.
C1
C4
C 51 C 42 C 33
C5
C 14 C 25
C3 C2
(a)
(b)
The poppingoff order: C 5, C4, C 3, C 1, C 2 The topologically sorted list: C2 C1 C3 C4 C5
(c)
FIGURE 4.7 (a) Digraph for which the topological sorting problem needs to be solved. (b) DFS traversal stack with the subscript numbers indicating the poppingoff order. (c) Solution to the problem.
arbitrary digraph, but it is easy to see that the problem cannot have a solution if a digraph has a directed cycle. Thus, for topological sorting to be possible, a digraph in question must be a dag. It turns out that being a dag is not only necessary but also sufﬁcient for topological sorting to be possible; i.e., if a digraph has no directed cycles, the topological sorting problem for it has a solution. Moreover, there are two efﬁcient algorithms that both verify whether a digraph is a dag and, if it is, produce an ordering of vertices that solves the topological sorting problem. The ﬁrst algorithm is a simple application of depthﬁrst search: perform a DFS traversal and note the order in which vertices become deadends (i.e., popped off the traversal stack). Reversing this order yields a solution to the topological sorting problem, provided, of course, no back edge has been encountered during the traversal. If a back edge has been encountered, the digraph is not a dag, and topological sorting of its vertices is impossible. Why does the algorithm work? When a vertex v is popped off a DFS stack, no vertex u with an edge from u to v can be among the vertices popped off before v. (Otherwise, (u, v) would have been a back edge.) Hence, any such vertex u will be listed after v in the poppedoff order list, and before v in the reversed list. Figure 4.7 illustrates an application of this algorithm to the digraph in Figure 4.6. Note that in Figure 4.7c, we have drawn the edges of the digraph, and they all point from left to right as the problem’s statement requires. It is a convenient way to check visually the correctness of a solution to an instance of the topological sorting problem.
141
4.2 Topological Sorting
C1
C4 delete C 1
C3 C2
C4
C2
delete C 3
delete C 2
C3
C5 C4
C5
C4
C5
C3 C5
delete C 4
delete C 5
C5
The solution obtained is C 1, C 2, C 3, C 4, C 5
FIGURE 4.8 Illustration of the sourceremoval algorithm for the topological sorting problem. On each iteration, a vertex with no incoming edges is deleted from the digraph.
The second algorithm is based on a direct implementation of the decrease(by one)andconquer technique: repeatedly, identify in a remaining digraph a source, which is a vertex with no incoming edges, and delete it along with all the edges outgoing from it. (If there are several sources, break the tie arbitrarily. If there are none, stop because the problem cannot be solved—see Problem 6a in this section’s exercises.) The order in which the vertices are deleted yields a solution to the topological sorting problem. The application of this algorithm to the same digraph representing the ﬁve courses is given in Figure 4.8. Note that the solution obtained by the sourceremoval algorithm is different from the one obtained by the DFSbased algorithm. Both of them are correct, of course; the topological sorting problem may have several alternative solutions. The tiny size of the example we used might create a wrong impression about the topological sorting problem. But imagine a large project—e.g., in construction, research, or software development—that involves a multitude of interrelated tasks with known prerequisites. The ﬁrst thing to do in such a situation is to make sure that the set of given prerequisites is not contradictory. The convenient way of doing this is to solve the topological sorting problem for the project’s digraph. Only then can one start thinking about scheduling tasks to, say, minimize the total completion time of the project. This would require, of course, other algorithms that you can ﬁnd in general books on operations research or in special ones on CPM (Critical Path Method) and PERT (Program Evaluation and Review Technique) methodologies. As to applications of topological sorting in computer science, they include instruction scheduling in program compilation, cell evaluation ordering in spreadsheet formulas, and resolving symbol dependencies in linkers.
142
DecreaseandConquer
Exercises 4.2 1. Apply the DFSbased algorithm to solve the topological sorting problem for the following digraphs:
a c
a
b d
(a)
c
d
e g
f
b
e
f
g
(b)
2. a. Prove that the topological sorting problem has a solution if and only if it is a dag. b. For a digraph with n vertices, what is the largest number of distinct solutions the topological sorting problem can have? 3. a. What is the time efﬁciency of the DFSbased algorithm for topological sorting? b. How can one modify the DFSbased algorithm to avoid reversing the vertex ordering generated by DFS? 4. Can one use the order in which vertices are pushed onto the DFS stack (instead of the order they are popped off it) to solve the topological sorting problem? 5. Apply the sourceremoval algorithm to the digraphs of Problem 1 above. 6. a. Prove that a nonempty dag must have at least one source. b. How would you ﬁnd a source (or determine that such a vertex does not exist) in a digraph represented by its adjacency matrix? What is the time efﬁciency of this operation? c. How would you ﬁnd a source (or determine that such a vertex does not exist) in a digraph represented by its adjacency lists? What is the time efﬁciency of this operation? 7. Can you implement the sourceremoval algorithm for a digraph represented by its adjacency lists so that its running time is in O(V  + E)? 8. Implement the two topological sorting algorithms in the language of your choice. Run an experiment to compare their running times. 9. A digraph is called strongly connected if for any pair of two distinct vertices u and v there exists a directed path from u to v and a directed path from v to u. In general, a digraph’s vertices can be partitioned into disjoint maximal subsets of vertices that are mutually accessible via directed paths; these subsets are called strongly connected components of the digraph. There are two DFS
4.2 Topological Sorting
143
based algorithms for identifying strongly connected components. Here is the simpler (but somewhat less efﬁcient) one of the two: Step 1 Perform a DFS traversal of the digraph given and number its vertices in the order they become dead ends. Step 2 Reverse the directions of all the edges of the digraph. Step 3 Perform a DFS traversal of the new digraph by starting (and, if necessary, restarting) the traversal at the highest numbered vertex among still unvisited vertices. The strongly connected components are exactly the vertices of the DFS trees obtained during the last traversal. a. Apply this algorithm to the following digraph to determine its strongly connected components: a
c
b
e
d f
g
h
b. What is the time efﬁciency class of this algorithm? Give separate answers for the adjacency matrix representation and adjacency list representation of an input digraph. c. How many strongly connected components does a dag have? 10. Spider’s web A spider sits at the bottom (point S) of its web, and a ﬂy sits at the top (F). How many different ways can the spider reach the ﬂy by moving along the web’s lines in the directions indicated by the arrows? [Kor05] F
S
144
DecreaseandConquer
4.3 Algorithms for Generating Combinatorial Objects In this section, we keep our promise to discuss algorithms for generating combinatorial objects. The most important types of combinatorial objects are permutations, combinations, and subsets of a given set. They typically arise in problems that require a consideration of different choices. We already encountered them in Chapter 3 when we discussed exhaustive search. Combinatorial objects are studied in a branch of discrete mathematics called combinatorics. Mathematicians, of course, are primarily interested in different counting formulas; we should be grateful for such formulas because they tell us how many items need to be generated. In particular, they warn us that the number of combinatorial objects typically grows exponentially or even faster as a function of the problem size. But our primary interest here lies in algorithms for generating combinatorial objects, not just in counting them.
Generating Permutations We start with permutations. For simplicity, we assume that the underlying set whose elements need to be permuted is simply the set of integers from 1 to n; more generally, they can be interpreted as indices of elements in an nelement set {a1, . . . , an}. What would the decreasebyone technique suggest for the problem of generating all n! permutations of {1, . . . , n}? The smallerbyone problem is to generate all (n − 1)! permutations. Assuming that the smaller problem is solved, we can get a solution to the larger one by inserting n in each of the n possible positions among elements of every permutation of n − 1 elements. All the permutations obtained in this fashion will be distinct (why?), and their total number will be n(n − 1)! = n!. Hence, we will obtain all the permutations of {1, . . . , n}. We can insert n in the previously generated permutations either left to right or right to left. It turns out that it is beneﬁcial to start with inserting n into 12 . . . (n − 1) by moving right to left and then switch direction every time a new permutation of {1, . . . , n − 1} needs to be processed. An example of applying this approach bottom up for n = 3 is given in Figure 4.9. The advantage of this order of generating permutations stems from the fact that it satisﬁes the minimalchange requirement: each permutation can be obtained from its immediate predecessor by exchanging just two elements in it. (For the method being discussed, these two elements are always adjacent to each other.
start insert 2 into 1 right to left insert 3 into 12 right to left insert 3 into 21 left to right
1 12 123 321
FIGURE 4.9 Generating permutations bottom up.
21 132 231
312 213
4.3
Algorithms for Generating Combinatorial Objects
145
Check this for the permutations generated in Figure 4.9.) The minimalchange requirement is beneﬁcial both for the algorithm’s speed and for applications using the permutations. For example, in Section 3.4, we needed permutations of cities to solve the traveling salesman problem by exhaustive search. If such permutations are generated by a minimalchange algorithm, we can compute the length of a new tour from the length of its predecessor in constant rather than linear time (how?). It is possible to get the same ordering of permutations of n elements without explicitly generating permutations for smaller values of n. It can be done by associating a direction with each element k in a permutation. We indicate such a direction by a small arrow written above the element in question, e.g., →←→←
3 2 4 1.
The element k is said to be mobile in such an arrowmarked permutation if its arrow points to a smaller number adjacent to it. For example, for the permutation →←→← 3 2 4 1, 3 and 4 are mobile while 2 and 1 are not. Using the notion of a mobile element, we can give the following description of the JohnsonTrotter algorithm for generating permutations. ALGORITHM
JohnsonTrotter(n)
//Implements JohnsonTrotter algorithm for generating permutations //Input: A positive integer n //Output: A list of all permutations of {1, . . . , n} ←← ← initialize the ﬁrst permutation with 1 2 . . . n while the last permutation has a mobile element do ﬁnd its largest mobile element k swap k with the adjacent element k’s arrow points to reverse the direction of all the elements that are larger than k add the new permutation to the list Here is an application of this algorithm for n = 3, with the largest mobile element shown in bold: ←←←
123
←←←
132
←←←
312
→←←
32 1
←→←
23 1
←←→
2 1 3.
This algorithm is one of the most efﬁcient for generating permutations; it can be implemented to run in time proportional to the number of permutations, i.e., in (n!). Of course, it is horribly slow for all but very small values of n; however, this is not the algorithm’s “fault” but rather the fault of the problem: it simply asks to generate too many items. One can argue that the permutation ordering generated by the JohnsonTrotter algorithm is not quite natural; for example, the natural place for permutation n(n − 1) . . . 1 seems to be the last one on the list. This would be the case if permutations were listed in increasing order—also called the lexicographic or
146
DecreaseandConquer
der—which is the order in which they would be listed in a dictionary if the numbers were interpreted as letters of an alphabet. For example, for n = 3, 123 132
213 231 312
321.
So how can we generate the permutation following a1a2 . . . an−1an in lexicographic order? If an−1 < an, which is the case for exactly one half of all the permutations, we can simply transpose these last two elements. For example, 123 is followed by 132. If an−1 > an, we ﬁnd the permutation’s longest decreasing sufﬁx ai+1 > ai+2 > . . . > an (but ai < ai+1); increase ai by exchanging it with the smallest element of the sufﬁx that is greater than ai ; and reverse the new sufﬁx to put it in increasing order. For example, 362541 is followed by 364125. Here is pseudocode of this simple algorithm whose origins go as far back as 14thcentury India. ALGORITHM LexicographicPermute(n) //Generates permutations in lexicographic order //Input: A positive integer n //Output: A list of all permutations of {1, . . . , n} in lexicographic order initialize the ﬁrst permutation with 12 . . . n while last permutation has two consecutive elements in increasing order do let i be its largest index such that ai < ai+1 //ai+1 > ai+2 > . . . > an ﬁnd the largest index j such that ai < aj //j ≥ i + 1 since ai < ai+1 swap ai with aj //ai+1ai+2 . . . an will remain in decreasing order reverse the order of the elements from ai+1 to an inclusive add the new permutation to the list
Generating Subsets Recall that in Section 3.4 we examined the knapsack problem, which asks to ﬁnd the most valuable subset of items that ﬁts a knapsack of a given capacity. The exhaustivesearch approach to solving this problem discussed there was based on generating all subsets of a given set of items. In this section, we discuss algorithms for generating all 2n subsets of an abstract set A = {a1, . . . , an}. (Mathematicians call the set of all subsets of a set its power set.) The decreasebyone idea is immediately applicable to this problem, too. All subsets of A = {a1, . . . , an} can be divided into two groups: those that do not contain an and those that do. The former group is nothing but all the subsets of {a1, . . . , an−1}, while each and every element of the latter can be obtained by adding an to a subset of {a1, . . . , an−1}. Thus, once we have a list of all subsets of {a1, . . . , an−1}, we can get all the subsets of {a1, . . . , an} by adding to the list all its elements with an put into each of them. An application of this algorithm to generate all subsets of {a1, a2 , a3} is illustrated in Figure 4.10. Similarly to generating permutations, we do not have to generate power sets of smaller sets. A convenient way of solving the problem directly is based on a onetoone correspondence between all 2n subsets of an n element set A = {a1, . . . , an}
4.3
subsets
n 0 1 2 3
147
Algorithms for Generating Combinatorial Objects
∅ ∅ ∅ ∅
{a1} {a1} {a1}
{a2 } {a2 }
{a1, a2 } {a1, a2 }
{a3}
{a1, a3}
{a2 , a3}
{a1, a2 , a3}
FIGURE 4.10 Generating subsets bottom up.
and all 2n bit strings b1, . . . , bn of length n. The easiest way to establish such a correspondence is to assign to a subset the bit string in which bi = 1 if ai belongs to the subset and bi = 0 if ai does not belong to it. (We mentioned this idea of bit vectors in Section 1.4.) For example, the bit string 000 will correspond to the empty subset of a threeelement set, 111 will correspond to the set itself, i.e., {a1, a2 , a3}, and 110 will represent {a1, a2 }. With this correspondence in place, we can generate all the bit strings of length n by generating successive binary numbers from 0 to 2n − 1, padded, when necessary, with an appropriate number of leading 0’s. For example, for the case of n = 3, we obtain bit strings subsets
000 ∅
001 {a3}
010 {a2 }
011 {a2 , a3}
100 {a1}
101 {a1, a3}
110 {a1, a2 }
111 {a1, a2 , a3}
Note that although the bit strings are generated by this algorithm in lexicographic order (in the twosymbol alphabet of 0 and 1), the order of the subsets looks anything but natural. For example, we might want to have the socalled squashed order, in which any subset involving aj can be listed only after all the subsets involving a1, . . . , aj −1, as was the case for the list of the threeelement set in Figure 4.10. It is easy to adjust the bit string–based algorithm above to yield a squashed ordering of the subsets involved (see Problem 6 in this section’s exercises). A more challenging question is whether there exists a minimalchange algorithm for generating bit strings so that every one of them differs from its immediate predecessor by only a single bit. (In the language of subsets, we want every subset to differ from its immediate predecessor by either an addition or a deletion, but not both, of a single element.) The answer to this question is yes. For example, for n = 3, we can get 000
001 011 010
110
111 101 100.
Such a sequence of bit strings is called the binary reﬂected Gray code. Frank Gray, a researcher at AT&T Bell Laboratories, reinvented it in the 1940s to minimize the effect of errors in transmitting digital signals (see, e.g., [Ros07], pp. 642– ´ 643). Seventy years earlier, the French engineer Emile Baudot used such codes
148
DecreaseandConquer
in telegraphy. Here is pseudocode that generates the binary reﬂected Gray code recursively. ALGORITHM BRGC(n) //Generates recursively the binary reﬂected Gray code of order n //Input: A positive integer n //Output: A list of all bit strings of length n composing the Gray code if n = 1 make list L containing bit strings 0 and 1 in this order else generate list L1 of bit strings of size n − 1 by calling BRGC(n − 1) copy list L1 to list L2 in reversed order add 0 in front of each bit string in list L1 add 1 in front of each bit string in list L2 append L2 to L1 to get list L return L The correctness of the algorithm stems from the fact that it generates 2n bit strings and all of them are distinct. Both these assertions are easy to check by mathematical induction. Note that the binary reﬂected Gray code is cyclic: its last bit string differs from the ﬁrst one by a single bit. For a nonrecursive algorithm for generating the binary reﬂected Gray code see Problem 9 in this section’s exercises.
Exercises 4.3 1. Is it realistic to implement an algorithm that requires generating all permutations of a 25element set on your computer? What about all the subsets of such a set? 2. Generate all permutations of {1, 2, 3, 4} by a. the bottomup minimalchange algorithm. b. the JohnsonTrotter algorithm. c. the lexicographicorder algorithm. 3. Apply LexicographicPermute to multiset {1, 2, 2, 3}. Does it generate correctly all the permutations in lexicographic order? 4. Consider the following implementation of the algorithm for generating permutations discovered by B. Heap [Hea63]. ALGORITHM HeapPermute(n) //Implements Heap’s algorithm for generating permutations //Input: A positive integer n and a global array A[1..n] //Output: All permutations of elements of A if n = 1 write A
4.3
Algorithms for Generating Combinatorial Objects
149
else for i ← 1 to n do HeapPermute(n − 1) if n is odd swap A[1] and A[n] else swap A[i] and A[n] a. Trace the algorithm by hand for n = 2, 3, and 4. b. Prove the correctness of Heap’s algorithm. c. What is the time efﬁciency of HeapPermute? 5. Generate all the subsets of a fourelement set A = {a1, a2 , a3, a4} by each of the two algorithms outlined in this section. 6. What simple trick would make the bit string–based algorithm generate subsets in squashed order? 7. Write pseudocode for a recursive algorithm for generating all 2n bit strings of length n. 8. Write a nonrecursive algorithm for generating 2n bit strings of length n that implements bit strings as arrays and does not use binary additions. 9. a. Generate the binary reﬂexive Gray code of order 4. b. Trace the following nonrecursive algorithm to generate the binary reﬂexive Gray code of order 4. Start with the nbit string of all 0’s. For i = 1, 2, . . . , 2n−1, generate the ith bit string by ﬂipping bit b in the previous bit string, where b is the position of the least signiﬁcant 1 in the binary representation of i. 10. Design a decreaseandconquer algorithm for generating all combinations of k items chosen from n, i.e., all kelement subsets of a given nelement set. Is your algorithm a minimalchange algorithm? 11. Gray code and the Tower of Hanoi a. Show that the disk moves made in the classic recursive algorithm for the Tower of Hanoi puzzle can be used for generating the binary reﬂected Gray code. b. Show how the binary reﬂected Gray code can be used for solving the Tower of Hanoi puzzle. 12. Fair attraction In olden days, one could encounter the following attraction at a fair. A light bulb was connected to several switches in such a way that it lighted up only when all the switches were closed. Each switch was controlled by a push button; pressing the button toggled the switch, but there was no way to know the state of the switch. The object was to turn the light bulb on. Design an algorithm to turn on the light bulb with the minimum number of button pushes needed in the worst case for n switches.
150
DecreaseandConquer
4.4 DecreasebyaConstantFactor Algorithms You may recall from the introduction to this chapter that decreasebyaconstantfactor is the second major variety of decreaseandconquer. As an example of an algorithm based on this technique, we mentioned there exponentiation by squaring deﬁned by formula (4.2). In this section, you will ﬁnd a few other examples of such algorithms.. The most important and wellknown of them is binary search. Decreasebyaconstantfactor algorithms usually run in logarithmic time, and, being very efﬁcient, do not happen often; a reduction by a factor other than two is especially rare.
Binary Search Binary search is a remarkably efﬁcient algorithm for searching in a sorted array. It works by comparing a search key K with the array’s middle element A[m]. If they match, the algorithm stops; otherwise, the same operation is repeated recursively for the ﬁrst half of the array if K < A[m], and for the second half if K > A[m]: K A[0] . . . A[m − 1] A[m] A[m + 1] . . . A[n − 1] . search here if KA[m]
As an example, let us apply binary search to searching for K = 70 in the array
3
14 27 31 39 42 55 70 74 81 85 93 98
The iterations of the algorithm are given in the following table:
index value
0 3
iteration 1
l
1 2 3 4 5 6 7 8 9 10 11 12 14 27 31 39 42 55 70 74 81 85 93 98 m
r
iteration 2
l
iteration 3
l,m
m
r
r
Though binary search is clearly based on a recursive idea, it can be easily implemented as a nonrecursive algorithm, too. Here is pseudocode of this nonrecursive version.
4.4 DecreasebyaConstantFactor Algorithms
ALGORITHM
151
BinarySearch(A[0..n − 1], K)
//Implements nonrecursive binary search //Input: An array A[0..n − 1] sorted in ascending order and // a search key K //Output: An index of the array’s element that is equal to K // or −1 if there is no such element l ← 0; r ← n − 1 while l ≤ r do m ← (l + r)/2 if K = A[m] return m else if K < A[m] r ← m − 1 else l ← m + 1 return −1 The standard way to analyze the efﬁciency of binary search is to count the number of times the search key is compared with an element of the array. Moreover, for the sake of simplicity, we will count the socalled threeway comparisons. This assumes that after one comparison of K with A[m], the algorithm can determine whether K is smaller, equal to, or larger than A[m]. How many such comparisons does the algorithm make on an array of n elements? The answer obviously depends not only on n but also on the speciﬁcs of a particular instance of the problem. Let us ﬁnd the number of key comparisons in the worst case Cworst (n). The worstcase inputs include all arrays that do not contain a given search key, as well as some successful searches. Since after one comparison the algorithm faces the same situation but for an array half the size, we get the following recurrence relation for Cworst (n): Cworst (n) = Cworst (n/2) + 1 for n > 1, Cworst (1) = 1.
(4.3)
(Stop and convince yourself that n/2 must be, indeed, rounded down and that the initial condition must be written as speciﬁed.) We already encountered recurrence (4.3), with a different initial condition, in Section 2.4 (see recurrence (2.4) and its solution there for n = 2k ). For the initial condition Cworst (1) = 1, we obtain Cworst (2k ) = k + 1 = log2 n + 1.
(4.4)
Further, similarly to the case of recurrence (2.4) (Problem 7 in Exercises 2.4), the solution given by formula (4.4) for n = 2k can be tweaked to get a solution valid for an arbitrary positive integer n: Cworst (n) = log2 n + 1 = log2 (n + 1).
(4.5)
Formula (4.5) deserves attention. First, it implies that the worstcase time efﬁciency of binary search is in (log n). Second, it is the answer we should have
152
DecreaseandConquer
fully expected: since the algorithm simply reduces the size of the remaining array by about half on each iteration, the number of such iterations needed to reduce the initial size n to the ﬁnal size 1 has to be about log2 n. Third, to reiterate the point made in Section 2.1, the logarithmic function grows so slowly that its values remain small even for very large values of n. In particular, according to formula (4.5), it will take no more than log2 (103 + 1) = 10 threeway comparisons to ﬁnd an element of a given value (or establish that there is no such element) in any sorted array of one thousand elements, and it will take no more than log2 (106 + 1) = 20 comparisons to do it for any sorted array of size one million! What can we say about the averagecase efﬁciency of binary search? A sophisticated analysis shows that the average number of key comparisons made by binary search is only slightly smaller than that in the worst case: Cavg (n) ≈ log2 n. (More accurate formulas for the average number of comparisons in a successful yes (n) ≈ log n − 1 and C no (n) ≈ log (n + 1), and an unsuccessful search are Cavg 2 2 avg respectively.) Though binary search is an optimal searching algorithm if we restrict our operations only to comparisons between keys (see Section 11.2), there are searching algorithms (see interpolation search in Section 4.5 and hashing in Section 7.3) with a better averagecase time efﬁciency, and one of them (hashing) does not even require the array to be sorted! These algorithms do require some special calculations in addition to key comparisons, however. Finally, the idea behind binary search has several applications beyond searching (see, e.g., [Ben00]). In addition, it can be applied to solving nonlinear equations in one unknown; we discuss this continuous analogue of binary search, called the method of bisection, in Section 12.4.
FakeCoin Problem Of several versions of the fakecoin identiﬁcation problem, we consider here the one that best illustrates the decreasebyaconstantfactor strategy. Among n identicallooking coins, one is fake. With a balance scale, we can compare any two sets of coins. That is, by tipping to the left, to the right, or staying even, the balance scale will tell whether the sets weigh the same or which of the sets is heavier than the other but not by how much. The problem is to design an efﬁcient algorithm for detecting the fake coin. An easier version of the problem—the one we discuss here—assumes that the fake coin is known to be, say, lighter than the genuine one.1 The most natural idea for solving this problem is to divide n coins into two piles of n/2 coins each, leaving one extra coin aside if n is odd, and put the two 1.
A much more challenging version assumes no additional information about the relative weights of the fake and genuine coins or even the presence of the fake coin among n given coins. We pursue this more difﬁcult version in the exercises for Section 11.2.
4.4 DecreasebyaConstantFactor Algorithms
153
piles on the scale. If the piles weigh the same, the coin put aside must be fake; otherwise, we can proceed in the same manner with the lighter pile, which must be the one with the fake coin. We can easily set up a recurrence relation for the number of weighings W (n) needed by this algorithm in the worst case: W (n) = W (n/2) + 1 for n > 1, W (1) = 0. This recurrence should look familiar to you. Indeed, it is almost identical to the one for the worstcase number of comparisons in binary search. (The difference is in the initial condition.) This similarity is not really surprising, since both algorithms are based on the same technique of halving an instance size. The solution to the recurrence for the number of weighings is also very similar to the one we had for binary search: W (n) = log2 n. This stuff should look elementary by now, if not outright boring. But wait: the interesting point here is the fact that the above algorithm is not the most efﬁcient solution. It would be more efﬁcient to divide the coins not into two but into three piles of about n/3 coins each. (Details of a precise formulation are developed in this section’s exercises. Do not miss it! If your instructor forgets, demand the instructor to assign Problem 10.) After weighing two of the piles, we can reduce the instance size by a factor of three. Accordingly, we should expect the number of weighings to be about log3 n, which is smaller than log2 n.
Russian Peasant Multiplication Now we consider a nonorthodox algorithm for multiplying two positive integers called multiplication a` la russe or the Russian peasant method. Let n and m be positive integers whose product we want to compute, and let us measure the instance size by the value of n. Now, if n is even, an instance of half the size has to deal with n/2, and we have an obvious formula relating the solution to the problem’s larger instance to the solution to the smaller one: n.m=
n. 2m. 2
If n is odd, we need only a slight adjustment of this formula: n−1 . 2m + m. 2 Using these formulas and the trivial case of 1 . m = m to stop, we can compute product n . m either recursively or iteratively. An example of computing 50 . 65 with this algorithm is given in Figure 4.11. Note that all the extra addends shown in parentheses in Figure 4.11a are in the rows that have odd values in the ﬁrst column. Therefore, we can ﬁnd the product by simply adding all the elements in the m column that have an odd number in the n column (Figure 4.11b). Also note that the algorithm involves just the simple operations of halving, doubling, and adding—a feature that might be attractive, for example, to those n.m=
154
DecreaseandConquer
n
m
n
m
50 25 12 6 3 1
65 130 260 520 1040 2080
50 25 12 6 3 1
65 130 260 520 1040 2080
2080
(+130)
(+1040) +(130 + 1040) = 3250 (a)
130
1040 2080 3250
(b)
FIGURE 4.11 Computing 50 . 65 by the Russian peasant method.
who do not want to memorize the table of multiplications. It is this feature of the algorithm that most probably made it attractive to Russian peasants who, according to Western visitors, used it widely in the nineteenth century and for whom the method is named. (In fact, the method was known to Egyptian mathematicians as early as 1650 b.c. [Cha98, p. 16].) It also leads to very fast hardware implementation since doubling and halving of binary numbers can be performed using shifts, which are among the most basic operations at the machine level.
Josephus Problem Our last example is the Josephus problem, named for Flavius Josephus, a famous Jewish historian who participated in and chronicled the Jewish revolt of 66–70 c.e. against the Romans. Josephus, as a general, managed to hold the fortress of Jotapata for 47 days, but after the fall of the city he took refuge with 40 diehards in a nearby cave. There, the rebels voted to perish rather than surrender. Josephus proposed that each man in turn should dispatch his neighbor, the order to be determined by casting lots. Josephus contrived to draw the last lot, and, as one of the two surviving men in the cave, he prevailed upon his intended victim to surrender to the Romans. So let n people numbered 1 to n stand in a circle. Starting the grim count with person number 1, we eliminate every second person until only one survivor is left. The problem is to determine the survivor’s number J (n). For example (Figure 4.12), if n is 6, people in positions 2, 4, and 6 will be eliminated on the ﬁrst pass through the circle, and people in initial positions 3 and 1 will be eliminated on the second pass, leaving a sole survivor in initial position 5—thus, J (6) = 5. To give another example, if n is 7, people in positions 2, 4, 6, and 1 will be eliminated on the ﬁrst pass (it is more convenient to include 1 in the ﬁrst pass) and people in positions 5 and, for convenience, 3 on the second—thus, J (7) = 7.
155
4.4 DecreasebyaConstantFactor Algorithms 12
11
61
21
5
32
7
21
61
41
32
41
52
(a)
(b)
FIGURE 4.12 Instances of the Josephus problem for (a) n = 6 and (b) n = 7. Subscript numbers indicate the pass on which the person in that position is eliminated. The solutions are J (6) = 5 and J (7) = 7, respectively.
It is convenient to consider the cases of even and odd n’s separately. If n is even, i.e., n = 2k, the ﬁrst pass through the circle yields an instance of exactly the same problem but half its initial size. The only difference is in position numbering; for example, a person in initial position 3 will be in position 2 for the second pass, a person in initial position 5 will be in position 3, and so on (check Figure 4.12a). It is easy to see that to get the initial position of a person, we simply need to multiply his new position by 2 and subtract 1. This relationship will hold, in particular, for the survivor, i.e., J (2k) = 2J (k) − 1. Let us now consider the case of an odd n (n > 1), i.e., n = 2k + 1. The ﬁrst pass eliminates people in all even positions. If we add to this the elimination of the person in position 1 right after that, we are left with an instance of size k. Here, to get the initial position that corresponds to the new position numbering, we have to multiply the new position number by 2 and add 1 (check Figure 4.12b). Thus, for odd values of n, we get J (2k + 1) = 2J (k) + 1. Can we get a closedform solution to the twocase recurrence subject to the initial condition J (1) = 1? The answer is yes, though getting it requires more ingenuity than just applying backward substitutions. In fact, one way to ﬁnd a solution is to apply forward substitutions to get, say, the ﬁrst 15 values of J (n), discern a pattern, and then prove its general validity by mathematical induction. We leave the execution of this plan to the exercises; alternatively, you can look it up in [GKP94], whose exposition of the Josephus problem we have been following. Interestingly, the most elegant form of the closedform answer involves the binary representation of size n: J (n) can be obtained by a 1bit cyclic shift left of n itself! For example, J (6) = J (1102 ) = 1012 = 5 and J (7) = J (1112 ) = 1112 = 7.
156
DecreaseandConquer
Exercises 4.4 1. Cutting a stick A stick n inches long needs to be cut into n 1inch pieces. Outline an algorithm that performs this task with the minimum number of cuts if several pieces of the stick can be cut at the same time. Also give a formula for the minimum number of cuts. 2. Design a decreasebyhalf algorithm for computing log2 n and determine its time efﬁciency. 3. a. What is the largest number of key comparisons made by binary search in searching for a key in the following array?
3
14
27 31 39
42 55
70 74 81 85 93
98
b. List all the keys of this array that will require the largest number of key comparisons when searched for by binary search. c. Find the average number of key comparisons made by binary search in a successful search in this array. Assume that each key is searched for with the same probability. d. Find the average number of key comparisons made by binary search in an unsuccessful search in this array. Assume that searches for keys in each of the 14 intervals formed by the array’s elements are equally likely. 4. Estimate how many times faster an average successful search will be in a sorted array of one million elements if it is done by binary search versus sequential search. 5. The time efﬁciency of sequential search does not depend on whether a list is implemented as an array or as a linked list. Is it also true for searching a sorted list by binary search? 6. a. Design a version of binary search that uses only twoway comparisons such as ≤ and =. Implement your algorithm in the language of your choice and carefully debug it: such programs are notorious for being prone to bugs. b. Analyze the time efﬁciency of the twoway comparison version designed in part a. 7. Picture guessing A version of the popular problemsolving task involves presenting people with an array of 42 pictures—seven rows of six pictures each— and asking them to identify the target picture by asking questions that can be answered yes or no. Further, people are then required to identify the picture with as few questions as possible. Suggest the most efﬁcient algorithm for this problem and indicate the largest number of questions that may be necessary. 8. Consider ternary search—the following algorithm for searching in a sorted array A[0..n − 1]. If n = 1, simply compare the search key K with the single
4.5 VariableSizeDecrease Algorithms
157
element of the array; otherwise, search recursively by comparing K with A[n/3], and if K is larger, compare it with A[2n/3] to determine in which third of the array to continue the search. a. What design technique is this algorithm based on? b. Set up a recurrence for the number of key comparisons in the worst case. You may assume that n = 3k . c. Solve the recurrence for n = 3k . d. Compare this algorithm’s efﬁciency with that of binary search. 9. An array A[0..n − 2] contains n − 1 integers from 1 to n in increasing order. (Thus one integer in this range is missing.) Design the most efﬁcient algorithm you can to ﬁnd the missing integer and indicate its time efﬁciency. 10. a. Write pseudocode for the divideintothree algorithm for the fakecoin problem. Make sure that your algorithm handles properly all values of n, not only those that are multiples of 3. b. Set up a recurrence relation for the number of weighings in the divideintothree algorithm for the fakecoin problem and solve it for n = 3k . c. For large values of n, about how many times faster is this algorithm than the one based on dividing coins into two piles? Your answer should not depend on n. 11. a. Apply the Russian peasant algorithm to compute 26 . 47. b. From the standpoint of time efﬁciency, does it matter whether we multiply n by m or m by n by the Russian peasant algorithm? 12. a. Write pseudocode for the Russian peasant multiplication algorithm. b. What is the time efﬁciency class of Russian peasant multiplication? 13. Find J (40)—the solution to the Josephus problem for n = 40. 14. Prove that the solution to the Josephus problem is 1 for every n that is a power of 2. 15. For the Josephus problem, a. compute J (n) for n = 1, 2, . . . , 15. b. discern a pattern in the solutions for the ﬁrst ﬁfteen values of n and prove its general validity. c. prove the validity of getting J (n) by a 1bit cyclic shift left of the binary representation of n.
4.5 VariableSizeDecrease Algorithms In the third principal variety of decreaseandconquer, the size reduction pattern varies from one iteration of the algorithm to another. Euclid’s algorithm for computing the greatest common divisor (Section 1.1) provides a good example
158
DecreaseandConquer
of this kind of algorithm. In this section, we encounter a few more examples of this variety.
Computing a Median and the Selection Problem The selection problem is the problem of ﬁnding the kth smallest element in a list of n numbers. This number is called the kth order statistic. Of course, for k = 1 or k = n, we can simply scan the list in question to ﬁnd the smallest or largest element, respectively. A more interesting case of this problem is for k = n/2, which asks to ﬁnd an element that is not larger than one half of the list’s elements and not smaller than the other half. This middle value is called the median, and it is one of the most important notions in mathematical statistics. Obviously, we can ﬁnd the kth smallest element in a list by sorting the list ﬁrst and then selecting the kth element in the output of a sorting algorithm. The time of such an algorithm is determined by the efﬁciency of the sorting algorithm used. Thus, with a fast sorting algorithm such as mergesort (discussed in the next chapter), the algorithm’s efﬁciency is in O(n log n). You should immediately suspect, however, that sorting the entire list is most likely overkill since the problem asks not to order the entire list but just to ﬁnd its kth smallest element. Indeed, we can take advantage of the idea of partitioning a given list around some value p of, say, its ﬁrst element. In general, this is a rearrangement of the list’s elements so that the left part contains all the elements smaller than or equal to p, followed by the pivot p itself, followed by all the elements greater than or equal to p. p
all are ≤ p
p
all are ≥ p
Of the two principal algorithmic alternatives to partition an array, here we discuss the Lomuto partitioning [Ben00, p. 117]; we introduce the better known Hoare’s algorithm in the next chapter. To get the idea behind the Lomuto partitioning, it is helpful to think of an array—or, more generally, a subarray A[l..r] (0 ≤ l ≤ r ≤ n − 1)—under consideration as composed of three contiguous segments. Listed in the order they follow pivot p, they are as follows: a segment with elements known to be smaller than p, the segment of elements known to be greater than or equal to p, and the segment of elements yet to be compared to p (see Figure 4.13a). Note that the segments can be empty; for example, it is always the case for the ﬁrst two segments before the algorithm starts. Starting with i = l + 1, the algorithm scans the subarray A[l..r] left to right, maintaining this structure until a partition is achieved. On each iteration, it compares the ﬁrst element in the unknown segment (pointed to by the scanning index i in Figure 4.13a) with the pivot p. If A[i] ≥ p, i is simply incremented to expand the segment of the elements greater than or equal to p while shrinking the unprocessed segment. If A[i] < p, it is the segment of the elements smaller than p that needs to be expanded. This is done by incrementing s, the index of the last
159
4.5 VariableSizeDecrease Algorithms l p
s
i
r
≥p
K(r). Thus, on each iteration of the algorithm, the problem of searching in a binary search tree is reduced to searching in a smaller binary search tree. The most sensible measure of the size of a search tree is its height; obviously, the decrease in a tree’s height normally changes from one iteration to another of the binary tree search—thus giving us an excellent example of a variablesizedecrease algorithm. In the worst case of the binary tree search, the tree is severely skewed. This happens, in particular, if a tree is constructed by successive insertions of an increasing or decreasing sequence of keys (Figure 4.15). Obviously, the search for an−1 in such a tree requires n comparisons, making the worstcase efﬁciency of the search operation fall into (n). Fortunately, the averagecase efﬁciency turns out to be in (log n). More precisely, the number of key comparisons needed for a search in a binary search tree built from n random keys is about 2ln n ≈ 1.39 log2 n. Since insertion of a new key into a binary search tree is almost identical to that of searching there, it also exempliﬁes the variablesizedecrease technique and has the same efﬁciency characteristics as the search operation.
164
DecreaseandConquer
a0
a0 a1
..
. an–2
an–2 an–1
(a)
. ..
a1
an–1
(b)
FIGURE 4.15 Binary search trees for (a) an increasing sequence of keys and (b) a decreasing sequence of keys.
The Game of Nim There are several wellknown games that share the following features. There are two players, who move in turn. No randomness or hidden information is permitted: all players know all information about gameplay. A game is impartial: each player has the same moves available from the same game position. Each of a ﬁnite number of available moves leads to a smaller instance of the same game. The game ends with a win by one of the players (there are no ties). The winner is the last player who is able to move. A prototypical example of such games is Nim. Generally, the game is played with several piles of chips, but we consider the onepile version ﬁrst. Thus, there is a single pile of n chips. Two players take turns by removing from the pile at least one and at most m chips; the number of chips taken may vary from one move to another, but both the lower and upper limits stay the same. Who wins the game by taking the last chip, the player moving ﬁrst or second, if both players make the best moves possible? Let us call an instance of the game a winning position for the player to move next if that player has a winning strategy, i.e., a sequence of moves that results in a victory no matter what moves the opponent makes. Let us call an instance of the game a losing position for the player to move next if every move available for that player leads to a winning position for the opponent. The standard approach to determining which positions are winning and which are losing is to investigate small values of n ﬁrst. It is logical to consider the instance of n = 0 as a losing one for the player to move next because this player is the ﬁrst one who cannot make a move. Any instance with 1 ≤ n ≤ m chips is obviously a winning position for the player to move next (why?). The instance with n = m + 1 chips is a losing one because taking any allowed number of chips puts the opponent in a winning position. (See an illustration for m = 4 in Figure 4.16.) Any instance with m + 2 ≤ n ≤ 2m + 1 chips is a winning position for the player to move next because there is a move that leaves the opponent with m + 1 chips, which is a losing
165
4.5 VariableSizeDecrease Algorithms 1
6
2 0
7 5
10
3
8
4
9
FIGURE 4.16 Illustration of onepile Nim with the maximum number of chips that may be taken on each move m = 4. The numbers indicate n, the number of chips in the pile. The losing positions for the player to move are circled. Only winning moves from the winning positions are shown (in bold).
position. 2m + 2 = 2(m + 1) chips is the next losing position, and so on. It is not difﬁcult to see the pattern that can be formally proved by mathematical induction: an instance with n chips is a winning position for the player to move next if and only if n is not a multiple of m + 1. The winning strategy is to take n mod(m + 1) chips on every move; any deviation from this strategy puts the opponent in a winning position. Onepile Nim has been known for a very long time. It appeared, in particular, as the summation game in the ﬁrst published book on recreational mathematics, authored by ClaudeGaspar Bachet, a French aristocrat and mathematician, in 1612: a player picks a positive integer less than, say, 10, and then his opponent and he take turns adding any integer less than 10; the ﬁrst player to reach 100 exactly is the winner [Dud70]. In general, Nim is played with I > 1 piles of chips of sizes n1, n2 , . . . , nI . On each move, a player can take any available number of chips, including all of them, from any single pile. The goal is the same—to be the last player able to make a move. Note that for I = 2, it is easy to ﬁgure out who wins this game and how. Here is a hint: the answer for the game’s instances with n1 = n2 differs from the answer for those with n1 = n2 . A solution to the general case of Nim is quite unexpected because it is based on the binary representation of the pile sizes. Let b1, b2 , . . . , bI be the pile sizes in binary. Compute their binary digital sum, also known as the nim sum, deﬁned as the sum of binary digits discarding any carry. (In other words, a binary digit si in the sum is 0 if the number of 1’s in the ith position in the addends is even, and it is 1 if the number of 1’s is odd.) It turns out that an instance of Nim is a winning one for the player to move next if and only if its nim sum contains at least one 1; consequently, Nim’s instance is a losing instance if and only if its nim sum contains only zeros. For example, for the commonly played instance with n1 = 3, n2 = 4, n3 = 5, the nim sum is
166
DecreaseandConquer
011 100 101 010
Since this sum contains a 1, the instance is a winning one for the player moving ﬁrst. To ﬁnd a winning move from this position, the player needs to change one of the three bit strings so that the new nim sum contains only 0’s. It is not difﬁcult to see that the only way to accomplish this is to remove two chips from the ﬁrst pile. This ingenious solution to the game of Nim was discovered by Harvard mathematics professor C. L. Bouton more than 100 years ago. Since then, mathematicians have developed a much more general theory of such games. An excellent account of this theory, with applications to many speciﬁc games, is given in the monograph by E. R. Berlekamp, J. H. Conway, and R. K. Guy [Ber03].
Exercises 4.5 1. a. If we measure an instance size of computing the greatest common divisor of m and n by the size of the second number n, by how much can the size decrease after one iteration of Euclid’s algorithm? b. Prove that an instance size will always decrease at least by a factor of two after two successive iterations of Euclid’s algorithm. 2. Apply quickselect to ﬁnd the median of the list of numbers 9, 12, 5, 17, 20, 30, 8. 3. Write pseudocode for a nonrecursive implementation of quickselect. 4. Derive the formula underlying interpolation search. 5. Give an example of the worstcase input for interpolation search and show that the algorithm is linear in the worst case. 6. a. Find the smallest value of n for which log2 log2 n + 1 is greater than 6. b. Determine which, if any, of the following assertions are true: i. log log n ∈ o(log n) ii. log log n ∈ (log n) iii. log log n ∈ (log n) 7. a. Outline an algorithm for ﬁnding the largest key in a binary search tree. Would you classify your algorithm as a variablesizedecrease algorithm? b. What is the time efﬁciency class of your algorithm in the worst case? 8. a. Outline an algorithm for deleting a key from a binary search tree. Would you classify this algorithm as a variablesizedecrease algorithm? b. What is the time efﬁciency class of your algorithm in the worst case? 9. Outline a variablesizedecrease algorithm for constructing an Eulerian circuit in a connected graph with all vertices of even degrees.
Summary
167
10. Mis`ere onepile Nim Consider the socalled mis`ere version of the onepile Nim, in which the player taking the last chip loses the game. All the other conditions of the game remain the same, i.e., the pile contains n chips and on each move a player takes at least one but no more than m chips. Identify the winning and losing positions (for the player to move next) in this game. 11. a. Moldy chocolate Two players take turns by breaking an m × n chocolate bar, which has one spoiled 1 × 1 square. Each break must be a single straight line cutting all the way across the bar along the boundaries between the squares. After each break, the player who broke the bar last eats the piece that does not contain the spoiled square. The player left with the spoiled square loses the game. Is it better to go ﬁrst or second in this game? b. Write an interactive program to play this game with the computer. Your program should make a winning move in a winning position and a random legitimate move in a losing position. 12. Flipping pancakes There are n pancakes all of different sizes that are stacked on top of each other. You are allowed to slip a ﬂipper under one of the pancakes and ﬂip over the whole stack above the ﬂipper. The purpose is to arrange pancakes according to their size with the biggest at the bottom. (You can see a visualization of this puzzle on the Interactive Mathematics Miscellany and Puzzles site [Bog].) Design an algorithm for solving this puzzle. 13. You need to search for a given number in an n × n matrix in which every row and every column is sorted in increasing order. Can you design a O(n) algorithm for this problem? [Laa10]
SUMMARY Decreaseandconquer is a general algorithm design technique, based on exploiting a relationship between a solution to a given instance of a problem and a solution to a smaller instance of the same problem. Once such a relationship is established, it can be exploited either top down (usually recursively) or bottom up. There are three major variations of decreaseandconquer: . decreasebyaconstant, most often by one (e.g., insertion sort) . decreasebyaconstantfactor, most often by the factor of two (e.g., binary search) . variablesizedecrease (e.g., Euclid’s algorithm) Insertion sort is a direct application of the decrease(by one)andconquer technique to the sorting problem. It is a (n2 ) algorithm both in the worst and average cases, but it is about twice as fast on average than in the worst case. The algorithm’s notable advantage is a good performance on almostsorted arrays.
168
DecreaseandConquer
A digraph is a graph with directions on its edges. The topological sorting problem asks to list vertices of a digraph in an order such that for every edge of the digraph, the vertex it starts at is listed before the vertex it points to. This problem has a solution if and only if a digraph is a dag (directed acyclic graph), i.e., it has no directed cycles. There are two algorithms for solving the topological sorting problem. The ﬁrst one is based on depthﬁrst search; the second is based on a direct application of the decreasebyone technique. The decreasebyone technique is a natural approach to developing algorithms for generating elementary combinatorial objects. The most efﬁcient class of such algorithms are minimalchange algorithms. However, the number of combinatorial objects grows so fast that even the best algorithms are of practical interest only for very small instances of such problems. Binary search is a very efﬁcient algorithm for searching in a sorted array. It is a principal example of a decreasebyaconstantfactor algorithm. Other examples include exponentiation by squaring, identifying a fake coin with a balance scale, Russian peasant multiplication, and the Josephus problem. For some decreaseandconquer algorithms, the size reduction varies from one iteration of the algorithm to another. Examples of such variablesizedecrease algorithms include Euclid’s algorithm, the partitionbased algorithm for the selection problem, interpolation search, and searching and insertion in a binary search tree. Nim exempliﬁes games that proceed through a series of diminishing instances of the same game.
5 DivideandConquer Whatever man prays for, he prays for a miracle. Every prayer reduces itself to this—Great God, grant that twice two be not four. —Ivan Turgenev (1818–1883), Russian novelist and shortstory writer
D
ivideandconquer is probably the bestknown general algorithm design technique. Though its fame may have something to do with its catchy name, it is well deserved: quite a few very efﬁcient algorithms are speciﬁc implementations of this general strategy. Divideandconquer algorithms work according to the following general plan: 1. 2.
3.
A problem is divided into several subproblems of the same type, ideally of about equal size. The subproblems are solved (typically recursively, though sometimes a different algorithm is employed, especially when subproblems become small enough). If necessary, the solutions to the subproblems are combined to get a solution to the original problem.
The divideandconquer technique is diagrammed in Figure 5.1, which depicts the case of dividing a problem into two smaller subproblems, by far the most widely occurring case (at least for divideandconquer algorithms designed to be executed on a singleprocessor computer). As an example, let us consider the problem of computing the sum of n numbers a0, . . . , an−1. If n > 1, we can divide the problem into two instances of the same problem: to compute the sum of the ﬁrst n/2 numbers and to compute the sum of the remaining n/2 numbers. (Of course, if n = 1, we simply return a0 as the answer.) Once each of these two sums is computed by applying the same method recursively, we can add their values to get the sum in question: a0 + . . . + an−1 = (a0 + . . . + an/2−1) + (an/2 + . . . + an−1). Is this an efﬁcient way to compute the sum of n numbers? A moment of reﬂection (why could it be more efﬁcient than the bruteforce summation?), a 169
170
DivideandConquer
problem
of size n
subproblem 1 of size n /2
subproblem 2 of size n /2
solution to subproblem 1
solution to subproblem 2
solution to the original problem
FIGURE 5.1 Divideandconquer technique (typical case).
small example of summing, say, four numbers by this algorithm, a formal analysis (which follows), and common sense (we do not normally compute sums this way, do we?) all lead to a negative answer to this question.1 Thus, not every divideandconquer algorithm is necessarily more efﬁcient than even a bruteforce solution. But often our prayers to the Goddess of Algorithmics—see the chapter’s epigraph—are answered, and the time spent on executing the divideandconquer plan turns out to be signiﬁcantly smaller than solving a problem by a different method. In fact, the divideandconquer approach yields some of the most important and efﬁcient algorithms in computer science. We discuss a few classic examples of such algorithms in this chapter. Though we consider only sequential algorithms here, it is worth keeping in mind that the divideandconquer technique is ideally suited for parallel computations, in which each subproblem can be solved simultaneously by its own processor.
1.
Actually, the divideandconquer algorithm, called the pairwise summation, may substantially reduce the accumulated roundoff error of the sum of numbers that can be represented only approximately in a digital computer [Hig93].
DivideandConquer
171
As mentioned above, in the most typical case of divideandconquer a problem’s instance of size n is divided into two instances of size n/2. More generally, an instance of size n can be divided into b instances of size n/b, with a of them needing to be solved. (Here, a and b are constants; a ≥ 1 and b > 1.) Assuming that size n is a power of b to simplify our analysis, we get the following recurrence for the running time T (n): T (n) = aT (n/b) + f (n),
(5.1)
where f (n) is a function that accounts for the time spent on dividing an instance of size n into instances of size n/b and combining their solutions. (For the sum example above, a = b = 2 and f (n) = 1.) Recurrence (5.1) is called the general divideandconquer recurrence. Obviously, the order of growth of its solution T (n) depends on the values of the constants a and b and the order of growth of the function f (n). The efﬁciency analysis of many divideandconquer algorithms is greatly simpliﬁed by the following theorem (see Appendix B). Master Theorem
If f (n) ∈ (nd ) where d ≥ 0 in recurrence (5.1), then ⎧ ⎨ (nd ) if a < bd , d T (n) ∈ (n log n) if a = bd , ⎩ if a > bd . (nlogb a )
Analogous results hold for the O and notations, too. For example, the recurrence for the number of additions A(n) made by the divideandconquer sumcomputation algorithm (see above) on inputs of size n = 2k is A(n) = 2A(n/2) + 1. Thus, for this example, a = 2, b = 2, and d = 0; hence, since a > bd , A(n) ∈ (nlogb a ) = (nlog2 2 ) = (n). Note that we were able to ﬁnd the solution’s efﬁciency class without going through the drudgery of solving the recurrence. But, of course, this approach can only establish a solution’s order of growth to within an unknown multiplicative constant, whereas solving a recurrence equation with a speciﬁc initial condition yields an exact answer (at least for n’s that are powers of b). It is also worth pointing out that if a = 1, recurrence (5.1) covers decreasebyaconstantfactor algorithms discussed in the previous chapter. In fact, some people consider such algorithms as binary search degenerate cases of divideandconquer, where just one of two subproblems of half the size needs to be solved. It is better not to do this and consider decreasebyaconstantfactor and divideandconquer as different design paradigms.
172
DivideandConquer
5.1 Mergesort Mergesort is a perfect example of a successful application of the divideandconquer technique. It sorts a given array A[0..n − 1] by dividing it into two halves A[0..n/2 − 1] and A[n/2..n − 1], sorting each of them recursively, and then merging the two smaller sorted arrays into a single sorted one. ALGORITHM Mergesort(A[0..n − 1]) //Sorts array A[0..n − 1] by recursive mergesort //Input: An array A[0..n − 1] of orderable elements //Output: Array A[0..n − 1] sorted in nondecreasing order if n > 1 copy A[0..n/2 − 1] to B[0..n/2 − 1] copy A[n/2..n − 1] to C[0..n/2 − 1] Mergesort(B[0..n/2 − 1]) Mergesort(C[0..n/2 − 1]) Merge(B, C, A) //see below The merging of two sorted arrays can be done as follows. Two pointers (array indices) are initialized to point to the ﬁrst elements of the arrays being merged. The elements pointed to are compared, and the smaller of them is added to a new array being constructed; after that, the index of the smaller element is incremented to point to its immediate successor in the array it was copied from. This operation is repeated until one of the two given arrays is exhausted, and then the remaining elements of the other array are copied to the end of the new array. ALGORITHM Merge(B[0..p − 1], C[0..q − 1], A[0..p + q − 1]) //Merges two sorted arrays into one sorted array //Input: Arrays B[0..p − 1] and C[0..q − 1] both sorted //Output: Sorted array A[0..p + q − 1] of the elements of B and C i ← 0; j ← 0; k ← 0 while i < p and j < q do if B[i] ≤ C[j ] A[k] ← B[i]; i ← i + 1 else A[k] ← C[j ]; j ← j + 1 k←k+1 if i = p copy C[j..q − 1] to A[k..p + q − 1] else copy B[i..p − 1] to A[k..p + q − 1] The operation of the algorithm on the list 8, 3, 2, 9, 7, 1, 5, 4 is illustrated in Figure 5.2.
173
5.1 Mergesort
8 3 2 9 7 1 5 4
8 3 2 9
8 3
8
7 1 5 4
2 9
3
2
3 8
7 1
9
2 9
7
5 4
1
5
1 7
2 3 8 9
4
4 5
1 4 5 7
1 2 3 4 5 7 8 9
FIGURE 5.2 Example of mergesort operation.
How efﬁcient is mergesort? Assuming for simplicity that n is a power of 2, the recurrence relation for the number of key comparisons C(n) is C(n) = 2C(n/2) + Cmerge (n)
for n > 1, C(1) = 0.
Let us analyze Cmerge (n), the number of key comparisons performed during the merging stage. At each step, exactly one comparison is made, after which the total number of elements in the two arrays still needing to be processed is reduced by 1. In the worst case, neither of the two arrays becomes empty before the other one contains just one element (e.g., smaller elements may come from the alternating arrays). Therefore, for the worst case, Cmerge (n) = n − 1, and we have the recurrence Cworst (n) = 2Cworst (n/2) + n − 1 for n > 1, Cworst (1) = 0. Hence, according to the Master Theorem, Cworst (n) ∈ (n log n) (why?). In fact, it is easy to ﬁnd the exact solution to the worstcase recurrence for n = 2k : Cworst (n) = n log2 n − n + 1.
174
DivideandConquer
The number of key comparisons made by mergesort in the worst case comes very close to the theoretical minimum2 that any general comparisonbased sorting algorithm can have. For large n, the number of comparisons made by this algorithm in the average case turns out to be about 0.25n less (see [Gon91, p. 173]) and hence is also in (n log n). A noteworthy advantage of mergesort over quicksort and heapsort—the two important advanced sorting algorithms to be discussed later—is its stability (see Problem 7 in this section’s exercises). The principal shortcoming of mergesort is the linear amount of extra storage the algorithm requires. Though merging can be done inplace, the resulting algorithm is quite complicated and of theoretical interest only. There are two main ideas leading to several variations of mergesort. First, the algorithm can be implemented bottom up by merging pairs of the array’s elements, then merging the sorted pairs, and so on. (If n is not a power of 2, only slight bookkeeping complications arise.) This avoids the time and space overhead of using a stack to handle recursive calls. Second, we can divide a list to be sorted in more than two parts, sort each recursively, and then merge them together. This scheme, which is particularly useful for sorting ﬁles residing on secondary memory devices, is called multiway mergesort.
Exercises 5.1 1. a. Write pseudocode for a divideandconquer algorithm for ﬁnding the position of the largest element in an array of n numbers. b. What will be your algorithm’s output for arrays with several elements of the largest value? c. Set up and solve a recurrence relation for the number of key comparisons made by your algorithm. d. How does this algorithm compare with the bruteforce algorithm for this problem? 2. a. Write pseudocode for a divideandconquer algorithm for ﬁnding values of both the largest and smallest elements in an array of n numbers. b. Set up and solve (for n = 2k ) a recurrence relation for the number of key comparisons made by your algorithm. c. How does this algorithm compare with the bruteforce algorithm for this problem? 3. a. Write pseudocode for a divideandconquer algorithm for the exponentiation problem of computing a n where n is a positive integer. b. Set up and solve a recurrence relation for the number of multiplications made by this algorithm. 2.
As we shall see in Section 11.2, this theoretical minimum is log2 n! ≈ n log2 n − 1.44n.
5.1 Mergesort
175
c. How does this algorithm compare with the bruteforce algorithm for this problem? 4. As mentioned in Chapter 2, logarithm bases are irrelevant in most contexts arising in analyzing an algorithm’s efﬁciency class. Is this true for both assertions of the Master Theorem that include logarithms? 5. Find the order of growth for solutions of the following recurrences. a. T (n) = 4T (n/2) + n, T (1) = 1 b. T (n) = 4T (n/2) + n2 , T (1) = 1 c. T (n) = 4T (n/2) + n3, T (1) = 1 6. Apply mergesort to sort the list E, X, A, M, P, L, E in alphabetical order. 7. Is mergesort a stable sorting algorithm? 8. a. Solve the recurrence relation for the number of key comparisons made by mergesort in the worst case. You may assume that n = 2k . b. Set up a recurrence relation for the number of key comparisons made by mergesort on bestcase inputs and solve it for n = 2k . c. Set up a recurrence relation for the number of key moves made by the version of mergesort given in Section 5.1. Does taking the number of key moves into account change the algorithm’s efﬁciency class? 9. Let A[0..n − 1] be an array of n real numbers. A pair (A[i], A[j ]) is said to be an inversion if these numbers are out of order, i.e., i < j but A[i] > A[j ]. Design an O(n log n) algorithm for counting the number of inversions. 10. Implement the bottomup version of mergesort in the language of your choice. 11. Tromino puzzle A tromino (more accurately, a right tromino) is an Lshaped tile formed by three 1 × 1 squares. The problem is to cover any 2n × 2n chessboard with a missing square with trominoes. Trominoes can be oriented in an arbitrary way, but they should cover all the squares of the board except the missing one exactly and with no overlaps. [Gol94]
Design a divideandconquer algorithm for this problem.
176
DivideandConquer
5.2 Quicksort Quicksort is the other important sorting algorithm that is based on the divideandconquer approach. Unlike mergesort, which divides its input elements according to their position in the array, quicksort divides them according to their value. We already encountered this idea of an array partition in Section 4.5, where we discussed the selection problem. A partition is an arrangement of the array’s elements so that all the elements to the left of some element A[s] are less than or equal to A[s], and all the elements to the right of A[s] are greater than or equal to it: A[0] . . . A[s − 1] A[s] A[s + 1] . . . A[n − 1] all are ≤A[s]
all are ≥A[s]
Obviously, after a partition is achieved, A[s] will be in its ﬁnal position in the sorted array, and we can continue sorting the two subarrays to the left and to the right of A[s] independently (e.g., by the same method). Note the difference with mergesort: there, the division of the problem into two subproblems is immediate and the entire work happens in combining their solutions; here, the entire work happens in the division stage, with no work required to combine the solutions to the subproblems. Here is pseudocode of quicksort: call Quicksort(A[0..n − 1]) where ALGORITHM Quicksort(A[l..r]) //Sorts a subarray by quicksort //Input: Subarray of array A[0..n − 1], deﬁned by its left and right // indices l and r //Output: Subarray A[l..r] sorted in nondecreasing order if l < r s ←Partition(A[l..r]) //s is a split position Quicksort(A[l..s − 1]) Quicksort(A[s + 1..r]) As a partition algorithm, we can certainly use the Lomuto partition discussed in Section 4.5. Alternatively, we can partition A[0..n − 1] and, more generally, its subarray A[l..r] (0 ≤ l < r ≤ n − 1) by the more sophisticated method suggested by C.A.R. Hoare, the prominent British computer scientist who invented quicksort.3 3.
C.A.R. Hoare, at age 26, invented his algorithm in 1960 while trying to sort words for a machine translation project from Russian to English. Says Hoare, “My ﬁrst thought on how to do this was bubblesort and, by an amazing stroke of luck, my second thought was Quicksort.” It is hard to disagree with his overall assessment: “I have been very lucky. What a wonderful way to start a career in Computing, by discovering a new sorting algorithm!” [Hoa96]. Twenty years later, he received the Turing Award for “fundamental contributions to the deﬁnition and design of programming languages”; in 1980, he was also knighted for services to education and computer science.
5.2
Quicksort
177
As before, we start by selecting a pivot—an element with respect to whose value we are going to divide the subarray. There are several different strategies for selecting a pivot; we will return to this issue when we analyze the algorithm’s efﬁciency. For now, we use the simplest strategy of selecting the subarray’s ﬁrst element: p = A[l]. Unlike the Lomuto algorithm, we will now scan the subarray from both ends, comparing the subarray’s elements to the pivot. The lefttoright scan, denoted below by index pointer i, starts with the second element. Since we want elements smaller than the pivot to be in the left part of the subarray, this scan skips over elements that are smaller than the pivot and stops upon encountering the ﬁrst element greater than or equal to the pivot. The righttoleft scan, denoted below by index pointer j, starts with the last element of the subarray. Since we want elements larger than the pivot to be in the right part of the subarray, this scan skips over elements that are larger than the pivot and stops on encountering the ﬁrst element smaller than or equal to the pivot. (Why is it worth stopping the scans after encountering an element equal to the pivot? Because doing this tends to yield more even splits for arrays with a lot of duplicates, which makes the algorithm run faster. For example, if we did otherwise for an array of n equal elements, we would have gotten a split into subarrays of sizes n − 1 and 0, reducing the problem size just by 1 after scanning the entire array.) After both scans stop, three situations may arise, depending on whether or not the scanning indices have crossed. If scanning indices i and j have not crossed, i.e., i < j, we simply exchange A[i] and A[j ] and resume the scans by incrementing i and decrementing j, respectively: i→ p
all are ≤ p
←j ...
≥p
≤p
all are ≥ p
If the scanning indices have crossed over, i.e., i > j, we will have partitioned the subarray after exchanging the pivot with A[j ]: ←j p
all are ≤ p
i→
≤p
≥p
all are ≥ p
Finally, if the scanning indices stop while pointing to the same element, i.e., i = j, the value they are pointing to must be equal to p (why?). Thus, we have the subarray partitioned, with the split position s = i = j : ←j=i→ p
all are ≤ p
=p
all are ≥ p
We can combine the last case with the case of crossedover indices (i > j ) by exchanging the pivot with A[j ] whenever i ≥ j . Here is pseudocode implementing this partitioning procedure.
178
DivideandConquer
ALGORITHM HoarePartition(A[l..r]) //Partitions a subarray by Hoare’s algorithm, using the ﬁrst element // as a pivot //Input: Subarray of array A[0..n − 1], deﬁned by its left and right // indices l and r (l < r) //Output: Partition of A[l..r], with the split position returned as // this function’s value p ← A[l] i ← l; j ← r + 1 repeat repeat i ← i + 1 until A[i] ≥ p repeat j ← j − 1 until A[j ] ≤ p swap(A[i], A[j ]) until i ≥ j swap(A[i], A[j ]) //undo last swap when i ≥ j swap(A[l], A[j ]) return j
Note that index i can go out of the subarray’s bounds in this pseudocode. Rather than checking for this possibility every time index i is incremented, we can append to array A[0..n − 1] a “sentinel” that would prevent index i from advancing beyond position n. Note that the more sophisticated method of pivot selection mentioned at the end of the section makes such a sentinel unnecessary. An example of sorting an array by quicksort is given in Figure 5.3. We start our discussion of quicksort’s efﬁciency by noting that the number of key comparisons made before a partition is achieved is n + 1 if the scanning indices cross over and n if they coincide (why?). If all the splits happen in the middle of corresponding subarrays, we will have the best case. The number of key comparisons in the best case satisﬁes the recurrence Cbest (n) = 2Cbest (n/2) + n
for n > 1, Cbest (1) = 0.
According to the Master Theorem, Cbest (n) ∈ (n log2 n); solving it exactly for n = 2k yields Cbest (n) = n log2 n. In the worst case, all the splits will be skewed to the extreme: one of the two subarrays will be empty, and the size of the other will be just 1 less than the size of the subarray being partitioned. This unfortunate situation will happen, in particular, for increasing arrays, i.e., for inputs for which the problem is already solved! Indeed, if A[0..n − 1] is a strictly increasing array and we use A[0] as the pivot, the lefttoright scan will stop on A[1] while the righttoleft scan will go all the way to reach A[0], indicating the split at position 0:
5.2 0
1 i
2
3
4
5
6
7
8
2
4
7
8
2
4
j
5
3
1
9
5
3
1
9
j
i
5 5
3 3 3
1 1 1
4
1
4
2
3 i
j
2
3
1
i
j
1
3
j
i
1 2
3 3
8
2
i
j
8
2
i
j
2
8
j
i
2 5
8 8
9
7
9
7
9
7
9 9
7 7
I=0, r =7 s =4
j
i
2 1 1
4
4 4
3 3
2
4
1 1
5 2
7
j
i
5
179
Quicksort
4
I=0, r =3 s =1
4 4 4
I=0, r =0
I=2, r =3 s =2
ij
3
I=5, r =7 s =6
I=5, r =5
I=7, r =7
4
j
i
3
4 4
I=2, r =1 8 8 8 7 7
i
j
9
7
i
j
7
9
j
i
7 8
9 9
I=3, r =3
(b)
9
(a) FIGURE 5.3 Example of quicksort operation. (a) Array’s transformations with pivots shown in bold. (b) Tree of recursive calls to Quicksort with input values l and r of subarray bounds and split position s of a partition obtained.
←j A[0]
i→ A[1]
. . .
A[n–1]
So, after making n + 1 comparisons to get to this partition and exchanging the pivot A[0] with itself, the algorithm will be left with the strictly increasing array A[1..n − 1] to sort. This sorting of strictly increasing arrays of diminishing sizes will
180
DivideandConquer
continue until the last one A[n − 2..n − 1] has been processed. The total number of key comparisons made will be equal to Cworst (n) = (n + 1) + n + . . . + 3 =
(n + 1)(n + 2) − 3 ∈ (n2 ). 2
Thus, the question about the utility of quicksort comes down to its averagecase behavior. Let Cavg (n) be the average number of key comparisons made by quicksort on a randomly ordered array of size n. A partition can happen in any position s (0 ≤ s ≤ n − 1) after n + 1 comparisons are made to achieve the partition. After the partition, the left and right subarrays will have s and n − 1 − s elements, respectively. Assuming that the partition split can happen in each position s with the same probability 1/n, we get the following recurrence relation: Cavg (n) =
n−1 1 [(n + 1) + Cavg (s) + Cavg (n − 1 − s)] for n > 1, n s=0
Cavg (0) = 0,
Cavg (1) = 0.
Its solution, which is much trickier than the worst and bestcase analyses, turns out to be Cavg (n) ≈ 2n ln n ≈ 1.39n log2 n. Thus, on the average, quicksort makes only 39% more comparisons than in the best case. Moreover, its innermost loop is so efﬁcient that it usually runs faster than mergesort (and heapsort, another n log n algorithm that we discuss in Chapter 6) on randomly ordered arrays of nontrivial sizes. This certainly justiﬁes the name given to the algorithm by its inventor. Because of quicksort’s importance, there have been persistent efforts over the years to reﬁne the basic algorithm. Among several improvements discovered by researchers are: better pivot selection methods such as randomized quicksort that uses a random element or the medianofthree method that uses the median of the leftmost, rightmost, and the middle element of the array switching to insertion sort on very small subarrays (between 5 and 15 elements for most computer systems) or not sorting small subarrays at all and ﬁnishing the algorithm with insertion sort applied to the entire nearly sorted array modiﬁcations of the partitioning algorithm such as the threeway partition into segments smaller than, equal to, and larger than the pivot (see Problem 9 in this section’s exercises) According to Robert Sedgewick [Sed11, p. 296], the world’s leading expert on quicksort, such improvements in combination can cut the running time of the algorithm by 20%–30%. Like any sorting algorithm, quicksort has weaknesses. It is not stable. It requires a stack to store parameters of subarrays that are yet to be sorted. While
5.2
Quicksort
181
the size of this stack can be made to be in O(log n) by always sorting ﬁrst the smaller of two subarrays obtained by partitioning, it is worse than the O(1) space efﬁciency of heapsort. Although more sophisticated ways of choosing a pivot make the quadratic running time of the worst case very unlikely, they do not eliminate it completely. And even the performance on randomly ordered arrays is known to be sensitive not only to implementation details of the algorithm but also to both computer architecture and data type. Still, the January/February 2000 issue of Computing in Science & Engineering, a joint publication of the American Institute of Physics and the IEEE Computer Society, selected quicksort as one of the 10 algorithms “with the greatest inﬂuence on the development and practice of science and engineering in the 20th century.”
Exercises 5.2 1. Apply quicksort to sort the list E, X, A, M, P , L, E in alphabetical order. Draw the tree of the recursive calls made. 2. For the partitioning procedure outlined in this section: a. Prove that if the scanning indices stop while pointing to the same element, i.e., i = j, the value they are pointing to must be equal to p. b. Prove that when the scanning indices stop, j cannot point to an element more than one position to the left of the one pointed to by i. 3. Give an example showing that quicksort is not a stable sorting algorithm. 4. Give an example of an array of n elements for which the sentinel mentioned in the text is actually needed. What should be its value? Also explain why a single sentinel sufﬁces for any input. 5. For the version of quicksort given in this section: a. Are arrays made up of all equal elements the worstcase input, the bestcase input, or neither? b. Are strictly decreasing arrays the worstcase input, the bestcase input, or neither? 6. a. For quicksort with the medianofthree pivot selection, are strictly increasing arrays the worstcase input, the bestcase input, or neither? b. Answer the same question for strictly decreasing arrays. 7. a. Estimate how many times faster quicksort will sort an array of one million random numbers than insertion sort. b. True or false: For every n > 1, there are nelement arrays that are sorted faster by insertion sort than by quicksort? 8. Design an algorithm to rearrange elements of a given array of n real numbers so that all its negative elements precede all its positive elements. Your algorithm should be both time efﬁcient and space efﬁcient.
182
DivideandConquer
9. a. The Dutch national ﬂag problem is to rearrange an array of characters R, W, and B (red, white, and blue are the colors of the Dutch national ﬂag) so that all the R’s come ﬁrst, the W’s come next, and the B’s come last. [Dij76] Design a linear inplace algorithm for this problem. b. Explain how a solution to the Dutch national ﬂag problem can be used in quicksort. 10. Implement quicksort in the language of your choice. Run your program on a sample of inputs to verify the theoretical assertions about the algorithm’s efﬁciency. 11. Nuts and bolts You are given a collection of n bolts of different widths and n corresponding nuts. You are allowed to try a nut and bolt together, from which you can determine whether the nut is larger than the bolt, smaller than the bolt, or matches the bolt exactly. However, there is no way to compare two nuts together or two bolts together. The problem is to match each bolt to its nut. Design an algorithm for this problem with averagecase efﬁciency in (n log n). [Raw91]
5.3 Binary Tree Traversals and Related Properties In this section, we see how the divideandconquer technique can be applied to binary trees. A binary tree T is deﬁned as a ﬁnite set of nodes that is either empty or consists of a root and two disjoint binary trees TL and TR called, respectively, the left and right subtree of the root. We usually think of a binary tree as a special case of an ordered tree (Figure 5.4). (This standard interpretation was an alternative deﬁnition of a binary tree in Section 1.4.) Since the deﬁnition itself divides a binary tree into two smaller structures of the same type, the left subtree and the right subtree, many problems about binary trees can be solved by applying the divideandconquer technique. As an example, let us consider a recursive algorithm for computing the height of a binary tree. Recall that the height is deﬁned as the length of the longest path from the root to a leaf. Hence, it can be computed as the maximum of the heights of the root’s left
Tleft
Tright
FIGURE 5.4 Standard representation of a binary tree.
5.3
Binary Tree Traversals and Related Properties
183
and right subtrees plus 1. (We have to add 1 to account for the extra level of the root.) Also note that it is convenient to deﬁne the height of the empty tree as −1. Thus, we have the following recursive algorithm. ALGORITHM Height(T ) //Computes recursively the height of a binary tree //Input: A binary tree T //Output: The height of T if T = ∅ return −1 else return max{Height(Tlef t ), Height(Tright )} + 1 We measure the problem’s instance size by the number of nodes n(T ) in a given binary tree T . Obviously, the number of comparisons made to compute the maximum of two numbers and the number of additions A(n(T )) made by the algorithm are the same. We have the following recurrence relation for A(n(T )): A(n(T )) = A(n(Tlef t )) + A(n(Tright )) + 1 for n(T ) > 0, A(0) = 0. Before we solve this recurrence (can you tell what its solution is?), let us note that addition is not the most frequently executed operation of this algorithm. What is? Checking—and this is very typical for binary tree algorithms—that the tree is not empty. For example, for the empty tree, the comparison T = ∅ is executed once but there are no additions, and for a singlenode tree, the comparison and addition numbers are 3 and 1, respectively. It helps in the analysis of tree algorithms to draw the tree’s extension by replacing the empty subtrees by special nodes. The extra nodes (shown by little squares in Figure 5.5) are called external; the original nodes (shown by little circles) are called internal. By deﬁnition, the extension of the empty binary tree is a single external node. It is easy to see that the Height algorithm makes exactly one addition for every internal node of the extended tree, and it makes one comparison to check whether
(a)
(b)
FIGURE 5.5 Binary tree (on the left) and its extension (on the right). Internal nodes are shown as circles; external nodes are shown as squares.
184
DivideandConquer
the tree is empty for every internal and external node. Therefore, to ascertain the algorithm’s efﬁciency, we need to know how many external nodes an extended binary tree with n internal nodes can have. After checking Figure 5.5 and a few similar examples, it is easy to hypothesize that the number of external nodes x is always 1 more than the number of internal nodes n: x = n + 1.
(5.2)
To prove this equality, consider the total number of nodes, both internal and external. Since every node, except the root, is one of the two children of an internal node, we have the equation 2n + 1 = x + n, which immediately implies equality (5.2). Note that equality (5.2) also applies to any nonempty full binary tree, in which, by deﬁnition, every node has either zero or two children: for a full binary tree, n and x denote the numbers of parental nodes and leaves, respectively. Returning to algorithm Height, the number of comparisons to check whether the tree is empty is C(n) = n + x = 2n + 1, and the number of additions is A(n) = n. The most important divideandconquer algorithms for binary trees are the three classic traversals: preorder, inorder, and postorder. All three traversals visit nodes of a binary tree recursively, i.e., by visiting the tree’s root and its left and right subtrees. They differ only by the timing of the root’s visit: In the preorder traversal, the root is visited before the left and right subtrees are visited (in that order). In the inorder traversal, the root is visited after visiting its left subtree but before visiting the right subtree. In the postorder traversal, the root is visited after visiting the left and right subtrees (in that order). These traversals are illustrated in Figure 5.6. Their pseudocodes are quite straightforward, repeating the descriptions given above. (These traversals are also a standard feature of data structures textbooks.) As to their efﬁciency analysis, it is identical to the above analysis of the Height algorithm because a recursive call is made for each node of an extended binary tree. Finally, we should note that, obviously, not all questions about binary trees require traversals of both left and right subtrees. For example, the search and insert operations for a binary search tree require processing only one of the two subtrees. Accordingly, we considered them in Section 4.5 not as applications of divideandconquer but rather as examples of the variablesizedecrease technique.
5.3
Binary Tree Traversals and Related Properties
185
a b
d
c
e
f
preorder: a, b, d, g, e, c, f inorder: d, g, b, e, a, f, c postorder: g, d, e, b, f, c, a
g
FIGURE 5.6 Binary tree and its traversals.
Exercises 5.3 1. Design a divideandconquer algorithm for computing the number of levels in a binary tree. (In particular, the algorithm must return 0 and 1 for the empty and singlenode trees, respectively.) What is the time efﬁciency class of your algorithm? 2. The following algorithm seeks to compute the number of leaves in a binary tree. ALGORITHM LeafCounter(T ) //Computes recursively the number of leaves in a binary tree //Input: A binary tree T //Output: The number of leaves in T if T = ∅ return 0 else return LeafCounter(Tlef t )+ LeafCounter(Tright ) Is this algorithm correct? If it is, prove it; if it is not, make an appropriate correction. 3. Can you compute the height of a binary tree with the same asymptotic efﬁciency as the section’s divideandconquer algorithm but without using a stack explicitly or implicitly? Of course, you may use a different algorithm altogether. 4. Prove equality (5.2) by mathematical induction. 5. Traverse the following binary tree a. in preorder. b. in inorder. c. in postorder.
186
DivideandConquer
a c
b
d
e
f
6. Write pseudocode for one of the classic traversal algorithms (preorder, inorder, and postorder) for binary trees. Assuming that your algorithm is recursive, ﬁnd the number of recursive calls made. 7. Which of the three classic traversal algorithms yields a sorted list if applied to a binary search tree? Prove this property. 8. a. Draw a binary tree with 10 nodes labeled 0, 1, . . . , 9 in such a way that the inorder and postorder traversals of the tree yield the following lists: 9, 3, 1, 0, 4, 2, 7, 6, 8, 5 (inorder) and 9, 1, 4, 0, 3, 6, 7, 5, 8, 2 (postorder). b. Give an example of two permutations of the same n labels 0, 1, . . . , n − 1 that cannot be inorder and postorder traversal lists of the same binary tree. c. Design an algorithm that constructs a binary tree for which two given lists of n labels 0, 1, . . . , n − 1 are generated by the inorder and postorder traversals of the tree. Your algorithm should also identify inputs for which the problem has no solution. 9. The internal path length I of an extended binary tree is deﬁned as the sum of the lengths of the paths—taken over all internal nodes—from the root to each internal node. Similarly, the external path length E of an extended binary tree is deﬁned as the sum of the lengths of the paths—taken over all external nodes—from the root to each external node. Prove that E = I + 2n where n is the number of internal nodes in the tree. 10. Write a program for computing the internal path length of an extended binary tree. Use it to investigate empirically the average number of key comparisons for searching in a randomly generated binary search tree. 11. Chocolate bar puzzle Given an n × m chocolate bar, you need to break it into nm 1 × 1 pieces. You can break a bar only in a straight line, and only one bar can be broken at a time. Design an algorithm that solves the problem with the minimum number of bar breaks. What is this minimum number? Justify your answer by using properties of a binary tree.
5.4 Multiplication of Large Integers and Strassen’s Matrix Multiplication In this section, we examine two surprising algorithms for seemingly straightforward tasks: multiplying two integers and multiplying two square matrices. Both
5.4
Multiplication of Large Integers and Strassen’s Matrix Multiplication
187
achieve a better asymptotic efﬁciency by ingenious application of the divideandconquer technique.
Multiplication of Large Integers Some applications, notably modern cryptography, require manipulation of integers that are over 100 decimal digits long. Since such integers are too long to ﬁt in a single word of a modern computer, they require special treatment. This practical need supports investigations of algorithms for efﬁcient manipulation of large integers. In this section, we outline an interesting algorithm for multiplying such numbers. Obviously, if we use the conventional penandpencil algorithm for multiplying two ndigit integers, each of the n digits of the ﬁrst number is multiplied by each of the n digits of the second number for the total of n2 digit multiplications. (If one of the numbers has fewer digits than the other, we can pad the shorter number with leading zeros to equalize their lengths.) Though it might appear that it would be impossible to design an algorithm with fewer than n2 digit multiplications, this turns out not to be the case. The miracle of divideandconquer comes to the rescue to accomplish this feat. To demonstrate the basic idea of the algorithm, let us start with a case of twodigit integers, say, 23 and 14. These numbers can be represented as follows: 23 = 2 . 101 + 3 . 100
and
14 = 1 . 101 + 4 . 100.
Now let us multiply them: 23 ∗ 14 = (2 . 101 + 3 . 100) ∗ (1 . 101 + 4 . 100) = (2 ∗ 1)102 + (2 ∗ 4 + 3 ∗ 1)101 + (3 ∗ 4)100. The last formula yields the correct answer of 322, of course, but it uses the same four digit multiplications as the penandpencil algorithm. Fortunately, we can compute the middle term with just one digit multiplication by taking advantage of the products 2 ∗ 1 and 3 ∗ 4 that need to be computed anyway: 2 ∗ 4 + 3 ∗ 1 = (2 + 3) ∗ (1 + 4) − 2 ∗ 1 − 3 ∗ 4. Of course, there is nothing special about the numbers we just multiplied. For any pair of twodigit numbers a = a1a0 and b = b1b0, their product c can be computed by the formula c = a ∗ b = c2 102 + c1101 + c0, where c2 = a1 ∗ b1 is the product of their ﬁrst digits, c0 = a0 ∗ b0 is the product of their second digits, c1 = (a1 + a0) ∗ (b1 + b0) − (c2 + c0) is the product of the sum of the a’s digits and the sum of the b’s digits minus the sum of c2 and c0.
188
DivideandConquer
Now we apply this trick to multiplying two ndigit integers a and b where n is a positive even number. Let us divide both numbers in the middle—after all, we promised to take advantage of the divideandconquer technique. We denote the ﬁrst half of the a’s digits by a1 and the second half by a0; for b, the notations are b1 and b0, respectively. In these notations, a = a1a0 implies that a = a110n/2 + a0 and b = b1b0 implies that b = b110n/2 + b0. Therefore, taking advantage of the same trick we used for twodigit numbers, we get c = a ∗ b = (a110n/2 + a0) ∗ (b110n/2 + b0) = (a1 ∗ b1)10n + (a1 ∗ b0 + a0 ∗ b1)10n/2 + (a0 ∗ b0) = c2 10n + c110n/2 + c0, where c2 = a1 ∗ b1 is the product of their ﬁrst halves, c0 = a0 ∗ b0 is the product of their second halves, c1 = (a1 + a0) ∗ (b1 + b0) − (c2 + c0) is the product of the sum of the a’s halves and the sum of the b’s halves minus the sum of c2 and c0. If n/2 is even, we can apply the same method for computing the products c2 , c0, and c1. Thus, if n is a power of 2, we have a recursive algorithm for computing the product of two ndigit integers. In its pure form, the recursion is stopped when n becomes 1. It can also be stopped when we deem n small enough to multiply the numbers of that size directly. How many digit multiplications does this algorithm make? Since multiplication of ndigit numbers requires three multiplications of n/2digit numbers, the recurrence for the number of multiplications M(n) is M(n) = 3M(n/2) for n > 1, M(1) = 1. Solving it by backward substitutions for n = 2k yields M(2k ) = 3M(2k−1) = 3[3M(2k−2 )] = 32 M(2k−2 ) = . . . = 3i M(2k−i ) = . . . = 3k M(2k−k ) = 3k . Since k = log2 n, M(n) = 3log2 n = nlog2 3 ≈ n1.585. (On the last step, we took advantage of the following property of logarithms: a logb c = clogb a .) But what about additions and subtractions? Have we not decreased the number of multiplications by requiring more of those operations? Let A(n) be the number of digit additions and subtractions executed by the above algorithm in multiplying two ndigit decimal integers. Besides 3A(n/2) of these operations needed to compute the three products of n/2digit numbers, the above formulas
5.4
Multiplication of Large Integers and Strassen’s Matrix Multiplication
189
require ﬁve additions and one subtraction. Hence, we have the recurrence A(n) = 3A(n/2) + cn
for n > 1, A(1) = 1.
Applying the Master Theorem, which was stated in the beginning of the chapter, we obtain A(n) ∈ (nlog2 3), which means that the total number of additions and subtractions have the same asymptotic order of growth as the number of multiplications. The asymptotic advantage of this algorithm notwithstanding, how practical is it? The answer depends, of course, on the computer system and program quality implementing the algorithm, which might explain the rather wide disparity of reported results. On some machines, the divideandconquer algorithm has been reported to outperform the conventional method on numbers only 8 decimal digits long and to run more than twice faster with numbers over 300 decimal digits long—the area of particular importance for modern cryptography. Whatever this outperformance “crossover point” happens to be on a particular machine, it is worth switching to the conventional algorithm after the multiplicands become smaller than the crossover point. Finally, if you program in an objectoriented language such as Java, C++, or Smalltalk, you should also be aware that these languages have special classes for dealing with large integers. Discovered by 23yearold Russian mathematician Anatoly Karatsuba in 1960, the divideandconquer algorithm proved wrong the thenprevailing opinion that the time efﬁciency of any integer multiplication algorithm must be in (n2 ). The discovery encouraged researchers to look for even (asymptotically) faster algorithms for this and other algebraic problems. We will see such an algorithm in the next section.
Strassen’s Matrix Multiplication Now that we have seen that the divideandconquer approach can reduce the number of onedigit multiplications in multiplying two integers, we should not be surprised that a similar feat can be accomplished for multiplying matrices. Such an algorithm was published by V. Strassen in 1969 [Str69]. The principal insight of the algorithm lies in the discovery that we can ﬁnd the product C of two 2 × 2 matrices A and B with just seven multiplications as opposed to the eight required by the bruteforce algorithm (see Example 3 in Section 2.3). This is accomplished by using the following formulas:
where
c00 c10
c01 c11
a00 a01 b00 b01 = ∗ a10 a11 b10 b11 m1 + m4 − m5 + m7 m3 + m5 , = m2 + m 4 m1 + m 3 − m 2 + m 6
190
DivideandConquer
m1 = (a00 + a11) ∗ (b00 + b11), m2 = (a10 + a11) ∗ b00, m3 = a00 ∗ (b01 − b11), m4 = a11 ∗ (b10 − b00), m5 = (a00 + a01) ∗ b11, m6 = (a10 − a00) ∗ (b00 + b01), m7 = (a01 − a11) ∗ (b10 + b11). Thus, to multiply two 2 × 2 matrices, Strassen’s algorithm makes seven multiplications and 18 additions/subtractions, whereas the bruteforce algorithm requires eight multiplications and four additions. These numbers should not lead us to multiplying 2 × 2 matrices by Strassen’s algorithm. Its importance stems from its asymptotic superiority as matrix order n goes to inﬁnity. Let A and B be two n × n matrices where n is a power of 2. (If n is not a power of 2, matrices can be padded with rows and columns of zeros.) We can divide A, B, and their product C into four n/2 × n/2 submatrices each as follows: # " # " # " A00 A01 C00 C01 B00 B01 = ∗ . C10 C11 A10 A11 B10 B11 It is not difﬁcult to verify that one can treat these submatrices as numbers to get the correct product. For example, C00 can be computed either as A00 ∗ B00 + A01 ∗ B10 or as M1 + M4 − M5 + M7 where M1, M4, M5, and M7 are found by Strassen’s formulas, with the numbers replaced by the corresponding submatrices. If the seven products of n/2 × n/2 matrices are computed recursively by the same method, we have Strassen’s algorithm for matrix multiplication. Let us evaluate the asymptotic efﬁciency of this algorithm. If M(n) is the number of multiplications made by Strassen’s algorithm in multiplying two n × n matrices (where n is a power of 2), we get the following recurrence relation for it: M(n) = 7M(n/2) for n > 1, M(1) = 1. Since
n = 2k , M(2k ) = 7M(2k−1) = 7[7M(2k−2 )] = 72 M(2k−2 ) = . . . = 7i M(2k−i ) . . . = 7k M(2k−k ) = 7k .
Since k = log2 n, M(n) = 7log2 n = nlog2 7 ≈ n2.807, which is smaller than n3 required by the bruteforce algorithm. Since this savings in the number of multiplications was achieved at the expense of making extra additions, we must check the number of additions A(n) made by Strassen’s algorithm. To multiply two matrices of order n > 1, the algorithm needs to multiply seven matrices of order n/2 and make 18 additions/subtractions of matrices of size n/2; when n = 1, no additions are made since two numbers are
5.4
Multiplication of Large Integers and Strassen’s Matrix Multiplication
191
simply multiplied. These observations yield the following recurrence relation: A(n) = 7A(n/2) + 18(n/2)2
for n > 1, A(1) = 0.
Though one can obtain a closedform solution to this recurrence (see Problem 8 in this section’s exercises), here we simply establish the solution’s order of growth. According to the Master Theorem, A(n) ∈ (nlog2 7). In other words, the number of additions has the same order of growth as the number of multiplications. This puts Strassen’s algorithm in (nlog2 7), which is a better efﬁciency class than (n3) of the bruteforce method. Since the time of Strassen’s discovery, several other algorithms for multiplying two n × n matrices of real numbers in O(nα ) time with progressively smaller constants α have been invented. The fastest algorithm so far is that of Coopersmith and Winograd [Coo87] with its efﬁciency in O(n2.376). The decreasing values of the exponents have been obtained at the expense of the increasing complexity of these algorithms. Because of large multiplicative constants, none of them is of practical value. However, they are interesting from a theoretical point of view. On one hand, they get closer and closer to the best theoretical lower bound known for matrix multiplication, which is n2 multiplications, though the gap between this bound and the best available algorithm remains unresolved. On the other hand, matrix multiplication is known to be computationally equivalent to some other important problems, such as solving systems of linear equations (discussed in the next chapter).
Exercises 5.4 1. What are the smallest and largest numbers of digits the product of two decimal ndigit integers can have? 2. Compute 2101 ∗ 1130 by applying the divideandconquer algorithm outlined in the text. 3. a. Prove the equality a logb c = clogb a , which was used in Section 5.4. b. Why is nlog2 3 better than 3log2 n as a closedform formula for M(n)? 4. a. Why did we not include multiplications by 10n in the multiplication count M(n) of the largeinteger multiplication algorithm? b. In addition to assuming that n is a power of 2, we made, for the sake of simplicity, another, more subtle, assumption in setting up the recurrences for M(n) and A(n), which is not always true (it does not change the ﬁnal answers, however). What is this assumption? 5. How many onedigit additions are made by the penandpencil algorithm in multiplying two ndigit integers? You may disregard potential carries. 6. Verify the formulas underlying Strassen’s algorithm for multiplying 2 × 2 matrices.
192
DivideandConquer
7. Apply Strassen’s algorithm to compute ⎡ ⎤ ⎡ ⎤ 1 0 2 1 0 1 0 1 ⎢4 1 1 0⎥ ⎢2 1 0 4⎥ ⎢ ⎥ ⎢ ⎥ ⎣0 1 3 0⎦∗⎣2 0 1 1⎦ 5 0 2 1 1 3 5 0 exiting the recursion when n = 2, i.e., computing the products of 2 × 2 matrices by the bruteforce algorithm. 8. Solve the recurrence for the number of additions required by Strassen’s algorithm. Assume that n is a power of 2. 9. V. Pan [Pan78] has discovered a divideandconquer matrix multiplication algorithm that is based on multiplying two 70 × 70 matrices using 143,640 multiplications. Find the asymptotic efﬁciency of Pan’s algorithm (you may ignore additions) and compare it with that of Strassen’s algorithm. 10. Practical implementations of Strassen’s algorithm usually switch to the bruteforce method after matrix sizes become smaller than some crossover point. Run an experiment to determine such a crossover point on your computer system.
5.5 The ClosestPair and ConvexHull Problems by DivideandConquer In Section 3.3, we discussed the bruteforce approach to solving two classic problems of computational geometry: the closestpair problem and the convexhull problem. We saw that the twodimensional versions of these problems can be solved by bruteforce algorithms in (n2 ) and O(n3) time, respectively. In this section, we discuss more sophisticated and asymptotically more efﬁcient algorithms for these problems, which are based on the divideandconquer technique.
The ClosestPair Problem Let P be a set of n > 1 points in the Cartesian plane. For the sake of simplicity, we assume that the points are distinct. We can also assume that the points are ordered in nondecreasing order of their x coordinate. (If they were not, we could sort them ﬁrst by an efﬁceint sorting algorithm such as mergesort.) It will also be convenient to have the points sorted in a separate list in nondecreasing order of the y coordinate; we will denote such a list Q. If 2 ≤ n ≤ 3, the problem can be solved by the obvious bruteforce algorithm. If n > 3, we can divide the points into two subsets Pl and Pr of n/2 and n/2 points, respectively, by drawing a vertical line through the median m of their x coordinates so that n/2 points lie to the left of or on the line itself, and n/2 points lie to the right of or on the line. Then we can solve the closestpair problem
193
5.5 The ClosestPair and ConvexHull Problems by DivideandConquer x=m
dl
dr x=m
d
d
d min p
d
d
(a)
(b)
FIGURE 5.7 (a) Idea of the divideandconquer algorithm for the closestpair problem. (b) Rectangle that may contain points closer than dmin to point p.
recursively for subsets Pl and Pr . Let dl and dr be the smallest distances between pairs of points in Pl and Pr , respectively, and let d = min{dl , dr }. Note that d is not necessarily the smallest distance between all the point pairs because points of a closer pair can lie on the opposite sides of the separating line. Therefore, as a step combining the solutions to the smaller subproblems, we need to examine such points. Obviously, we can limit our attention to the points inside the symmetric vertical strip of width 2d around the separating line, since the distance between any other pair of points is at least d (Figure 5.7a).
194
DivideandConquer
Let S be the list of points inside the strip of width 2d around the separating line, obtained from Q and hence ordered in nondecreasing order of their y coordinate. We will scan this list, updating the information about dmin , the minimum distance seen so far, if we encounter a closer pair of points. Initially, dmin = d, and subsequently dmin ≤ d. Let p(x, y) be a point on this list. For a point p (x , y ) to have a chance to be closer to p than dmin , the point must follow p on list S and the difference between their y coordinates must be less than dmin (why?). Geometrically, this means that p must belong to the rectangle shown in Figure 5.7b. The principal insight exploited by the algorithm is the observation that the rectangle can contain just a few such points, because the points in each half (left and right) of the rectangle must be at least distance d apart. It is easy to prove that the total number of such points in the rectangle, including p, does not exceed eight (Problem 2 in this section’s exercises); a more careful analysis reduces this number to six (see [Joh04, p. 695]). Thus, the algorithm can consider no more than ﬁve next points following p on the list S, before moving up to the next point. Here is pseudocode of the algorithm. We follow the advice given in Section 3.3 to avoid computing square roots inside the innermost loop of the algorithm. ALGORITHM EfﬁcientClosestPair(P , Q) //Solves the closestpair problem by divideandconquer //Input: An array P of n ≥ 2 points in the Cartesian plane sorted in // nondecreasing order of their x coordinates and an array Q of the // same points sorted in nondecreasing order of the y coordinates //Output: Euclidean distance between the closest pair of points if n ≤ 3 return the minimal distance found by the bruteforce algorithm else copy the ﬁrst n/2 points of P to array Pl copy the same n/2 points from Q to array Ql copy the remaining n/2 points of P to array Pr copy the same n/2 points from Q to array Qr dl ← EfﬁcientClosestPair(Pl , Ql ) dr ← EfﬁcientClosestPair(Pr , Qr ) d ←min{dl , dr } m ← P [n/2 − 1].x copy all the points of Q for which x − m < d into array S[0..num − 1] dminsq ← d 2 for i ← 0 to num − 2 do k←i+1 while k ≤ num − 1 and (S[k].y − S[i].y)2 < dminsq dminsq ← min((S[k].x − S[i].x)2 + (S[k].y − S[i].y)2 , dminsq) k←k+1 return sqrt(dminsq)
5.5 The ClosestPair and ConvexHull Problems by DivideandConquer
195
The algorithm spends linear time both for dividing the problem into two problems half the size and combining the obtained solutions. Therefore, assuming as usual that n is a power of 2, we have the following recurrence for the running time of the algorithm: T (n) = 2T (n/2) + f (n), where f (n) ∈ (n). Applying the Master Theorem (with a = 2, b = 2, and d = 1), we get T (n) ∈ (n log n). The necessity to presort input points does not change the overall efﬁciency class if sorting is done by a O(n log n) algorithm such as mergesort. In fact, this is the best efﬁciency class one can achieve, because it has been proved that any algorithm for this problem must be in (n log n) under some natural assumptions about operations an algorithm can perform (see [Pre85, p. 188]).
ConvexHull Problem Let us revisit the convexhull problem, introduced in Section 3.3: ﬁnd the smallest convex polygon that contains n given points in the plane. We consider here a divideandconquer algorithm called quickhull because of its resemblance to quicksort. Let S be a set of n > 1 points p1(x1, y1), . . . , pn(xn, yn) in the Cartesian plane. We assume that the points are sorted in nondecreasing order of their x coordinates, with ties resolved by increasing order of the y coordinates of the points involved. It is not difﬁcult to prove the geometrically obvious fact that the leftmost point p1 and the rightmost point pn are two distinct extreme points of the set’s convex −−−→ hull (Figure 5.8). Let p1p n be the straight line through points p1 and pn directed from p1 to pn. This line separates the points of S into two sets: S1 is the set of points to the left of this line, and S2 is the set of points to the right of this line. −−−−→ (We say that point q3 is to the left of the line q1q2 directed from point q1 to point q2 if q1q2 q3 forms a counterclockwise cycle. Later, we cite an analytical way to check this condition, based on checking the sign of a determinant formed by the −−−→ coordinates of the three points.) The points of S on the line p1p n, other than p1 and pn, cannot be extreme points of the convex hull and hence are excluded from further consideration. The boundary of the convex hull of S is made up of two polygonal chains: an “upper” boundary and a “lower” boundary. The “upper” boundary, called the upper hull, is a sequence of line segments with vertices at p1, some of the points in S1 (if S1 is not empty) and pn. The “lower” boundary, called the lower hull, is a sequence of line segments with vertices at p1, some of the points in S2 (if S2 is not empty) and pn. The fact that the convex hull of the entire set S is composed of the upper and lower hulls, which can be constructed independently and in a similar fashion, is a very useful observation exploited by several algorithms for this problem. For concreteness, let us discuss how quickhull proceeds to construct the upper hull; the lower hull can be constructed in the same manner. If S1 is empty, the
196
DivideandConquer
pn p1
FIGURE 5.8 Upper and lower hulls of a set of points.
pmax
pn p1
FIGURE 5.9 The idea of quickhull.
upper hull is simply the line segment with the endpoints at p1 and pn. If S1 is not empty, the algorithm identiﬁes point pmax in S1, which is the farthest from the line −−−→ p1p n (Figure 5.9). If there is a tie, the point that maximizes the angle pmax ppn can be selected. (Note that point pmax maximizes the area of the triangle with two vertices at p1 and pn and the third one at some other point of S1.) Then the −−−→ algorithm identiﬁes all the points of set S1 that are to the left of the line p1p max ; these are the points that will make up the set S1,1. The points of S1 to the left of −−−−−−→ the line pmax p n will make up the set S1,2 . It is not difﬁcult to prove the following: pmax is a vertex of the upper hull. The points inside p1pmax pn cannot be vertices of the upper hull (and hence can be eliminated from further consideration). −−−→ −−−−−−→ There are no points to the left of both lines p1p max and pmax p n. Therefore, the algorithm can continue constructing the upper hulls of p1 ∪ S1,1 ∪ pmax and pmax ∪ S1,2 ∪ pn recursively and then simply concatenate them to get the upper hull of the entire set p1 ∪ S1 ∪ pn.
5.5 The ClosestPair and ConvexHull Problems by DivideandConquer
197
Now we have to ﬁgure out how the algorithm’s geometric operations can be actually implemented. Fortunately, we can take advantage of the following very useful fact from analytical geometry: if q1(x1, y1), q2 (x2 , y2 ), and q3(x3, y3) are three arbitrary points in the Cartesian plane, then the area of the triangle q1q2 q3 is equal to onehalf of the magnitude of the determinant $ $ $ x 1 y1 1 $ $ $ $ x2 y2 1 $ = x1y2 + x3y1 + x2 y3 − x3y2 − x2 y1 − x1y3, $ $ $ x3 y3 1 $ while the sign of this expression is positive if and only if the point q3 = (x3, y3) is to −−−→ the left of the line q1q2 . Using this formula, we can check in constant time whether a point lies to the left of the line determined by two other points as well as ﬁnd the distance from the point to the line. Quickhull has the same (n2 ) worstcase efﬁciency as quicksort (Problem 9 in this section’s exercises). In the average case, however, we should expect a much better performance. First, the algorithm should beneﬁt from the quicksortlike savings from the onaverage balanced split of the problem into two smaller subproblems. Second, a signiﬁcant fraction of the points—namely, those inside p1pmax pn (see Figure 5.9)—are eliminated from further processing. Under a natural assumption that points given are chosen randomly from a uniform distribution over some convex region (e.g., a circle or a rectangle), the averagecase efﬁciency of quickhull turns out to be linear [Ove80].
Exercises 5.5 1. a. For the onedimensional version of the closestpair problem, i.e., for the problem of ﬁnding two closest numbers among a given set of n real numbers, design an algorithm that is directly based on the divideandconquer technique and determine its efﬁciency class. b. Is it a good algorithm for this problem? 2. Prove that the divideandconquer algorithm for the closestpair problem examines, for every point p in the vertical strip (see Figures 5.7a and 5.7b), no more than seven other points that can be closer to p than dmin , the minimum distance between two points encountered by the algorithm up to that point. 3. Consider the version of the divideandconquer twodimensional closestpair algorithm in which, instead of presorting input set P , we simply sort each of the two sets Pl and Pr in nondecreasing order of their y coordinates on each recursive call. Assuming that sorting is done by mergesort, set up a recurrence relation for the running time in the worst case and solve it for n = 2k . 4. Implement the divideandconquer closestpair algorithm, outlined in this section, in the language of your choice.
198
DivideandConquer
5. Find on the Web a visualization of an algorithm for the closestpair problem. What algorithm does this visualization represent? 6. The Voronoi polygon for a point p of a set S of points in the plane is deﬁned to be the perimeter of the set of all points in the plane closer to p than to any other point in S. The union of all the Voronoi polygons of the points in S is called the Voronoi diagram of S. a. What is the Voronoi diagram for a set of three points? b. Find a visualization of an algorithm for generating the Voronoi diagram on the Web and study a few examples of such diagrams. Based on your observations, can you tell how the solution to the previous question is generalized to the general case? 7. Explain how one can ﬁnd point pmax in the quickhull algorithm analytically. 8. What is the bestcase efﬁciency of quickhull? 9. Give a speciﬁc example of inputs that make quickhull run in quadratic time. 10. Implement quickhull in the language of your choice. 11. Creating decagons There are 1000 points in the plane, no three of them on the same line. Devise an algorithm to construct 100 decagons with their vertices at these points. The decagons need not be convex, but each of them has to be simple, i.e., its boundary should not cross itself, and no two decagons may have a common point. 12. Shortest path around There is a fenced area in the twodimensional Euclidean plane in the shape of a convex polygon with vertices at points p1(x1, y1), p2 (x2 , y2 ), . . . , pn(xn, yn) (not necessarily in this order). There are two more points, a(xa , ya ) and b(xb , yb ) such that xa < min{x1, x2 , . . . , xn} and xb > max{x1, x2 , . . . , xn}. Design a reasonably efﬁcient algorithm for computing the length of the shortest path between a and b. [ORo98]
SUMMARY Divideandconquer is a general algorithm design technique that solves a problem by dividing it into several smaller subproblems of the same type (ideally, of about equal size), solving each of them recursively, and then combining their solutions to get a solution to the original problem. Many efﬁcient algorithms are based on this technique, although it can be both inapplicable and inferior to simpler algorithmic solutions. Running time T (n) of many divideandconquer algorithms satisﬁes the recurrence T (n) = aT (n/b) + f (n). The Master Theorem establishes the order of growth of its solutions. Mergesort is a divideandconquer sorting algorithm. It works by dividing an input array into two halves, sorting them recursively, and then merging the two
Summary
199
sorted halves to get the original array sorted. The algorithm’s time efﬁciency is in (n log n) in all cases, with the number of key comparisons being very close to the theoretical minimum. Its principal drawback is a signiﬁcant extra storage requirement. Quicksort is a divideandconquer sorting algorithm that works by partitioning its input elements according to their value relative to some preselected element. Quicksort is noted for its superior efﬁciency among n log n algorithms for sorting randomly ordered arrays but also for the quadratic worstcase efﬁciency. The classic traversals of a binary tree—preorder, inorder, and postorder— and similar algorithms that require recursive processing of both left and right subtrees can be considered examples of the divideandconquer technique. Their analysis is helped by replacing all the empty subtrees of a given tree by special external nodes. There is a divideandconquer algorithm for multiplying two ndigit integers that requires about n1.585 onedigit multiplications. Strassen’s algorithm needs only seven multiplications to multiply two 2 × 2 matrices. By exploiting the divideandconquer technique, this algorithm can multiply two n × n matrices with about n2.807 multiplications. The divideandconquer technique can be successfully applied to two important problems of computational geometry: the closestpair problem and the convexhull problem.
This page intentionally left blank
6 TransformandConquer That’s the secret to life . . . replace one worry with another. —Charles M. Schulz (1922–2000), American cartoonist, the creator of Peanuts
T
his chapter deals with a group of design methods that are based on the idea of transformation. We call this general technique transformandconquer because these methods work as twostage procedures. First, in the transformation stage, the problem’s instance is modiﬁed to be, for one reason or another, more amenable to solution. Then, in the second or conquering stage, it is solved. There are three major variations of this idea that differ by what we transform a given instance to (Figure 6.1): Transformation to a simpler or more convenient instance of the same problem—we call it instance simpliﬁcation. Transformation to a different representation of the same instance—we call it representation change. Transformation to an instance of a different problem for which an algorithm is already available—we call it problem reduction. In the ﬁrst three sections of this chapter, we encounter examples of the instancesimpliﬁcation variety. Section 6.1 deals with the simple but fruitful idea of presorting. Many algorithmic problems are easier to solve if their input is sorted. Of course, the beneﬁts of sorting should more than compensate for the
problem's instance
simpler instance or another representation or another problem's instance
solution
FIGURE 6.1 Transformandconquer strategy.
201
202
TransformandConquer
time spent on it; otherwise, we would be better off dealing with an unsorted input directly. Section 6.2 introduces one of the most important algorithms in applied mathematics: Gaussian elimination. This algorithm solves a system of linear equations by ﬁrst transforming it to another system with a special property that makes ﬁnding a solution quite easy. In Section 6.3, the ideas of instance simpliﬁcation and representation change are applied to search trees. The results are AVL trees and multiway balanced search trees; of the latter we consider the simplest case, 23 trees. Section 6.4 presents heaps and heapsort. Even if you are already familiar with this important data structure and its application to sorting, you can still beneﬁt from looking at them in this new light of transformandconquer design. In Section 6.5, we discuss Horner’s rule, a remarkable algorithm for evaluating polynomials. If there were an Algorithm Hall of Fame, Horner’s rule would be a serious candidate for induction based on the algorithm’s elegance and efﬁciency. We also consider there two interesting algorithms for the exponentiation problem, both based on the representationchange idea. The chapter concludes with a review of several applications of the third variety of transformandconquer: problem reduction. This variety should be considered the most radical of the three: one problem is reduced to another, i.e., transformed into an entirely different problem. This is a very powerful idea, and it is extensively used in the complexity theory (Chapter 11). Its application to designing practical algorithms is not trivial, however. First, we need to identify a new problem into which the given problem should be transformed. Then we must make sure that the transformation algorithm followed by the algorithm for solving the new problem is time efﬁcient compared to other algorithmic alternatives. Among several examples, we discuss an important special case of mathematical modeling, or expressing a problem in terms of purely mathematical objects such as variables, functions, and equations.
6.1 Presorting Presorting is an old idea in computer science. In fact, interest in sorting algorithms is due, to a signiﬁcant degree, to the fact that many questions about a list are easier to answer if the list is sorted. Obviously, the time efﬁciency of algorithms that involve sorting may depend on the efﬁciency of the sorting algorithm being used. For the sake of simplicity, we assume throughout this section that lists are implemented as arrays, because some sorting algorithms are easier to implement for the array representation. So far, we have discussed three elementary sorting algorithms—selection sort, bubble sort, and insertion sort—that are quadratic in the worst and average cases, and two advanced algorithms—mergesort, which is always in (n log n), and quicksort, whose efﬁciency is also (n log n) in the average case but is quadratic in the worst case. Are there faster sorting algorithms? As we have already stated in Section 1.3 (see also Section 11.2), no general comparisonbased sorting algorithm
6.1
Presorting
203
can have a better efﬁciency than n log n in the worst case, and the same result holds for the averagecase efﬁciency.1 Following are three examples that illustrate the idea of presorting. More examples can be found in this section’s exercises. EXAMPLE 1 Checking element uniqueness in an array If this element uniqueness problem looks familiar to you, it should; we considered a bruteforce algorithm for the problem in Section 2.3 (see Example 2). The bruteforce algorithm compared pairs of the array’s elements until either two equal elements were found or no more pairs were left. Its worstcase efﬁciency was in (n2 ). Alternatively, we can sort the array ﬁrst and then check only its consecutive elements: if the array has equal elements, a pair of them must be next to each other, and vice versa. ALGORITHM PresortElementUniqueness(A[0..n − 1]) //Solves the element uniqueness problem by sorting the array ﬁrst //Input: An array A[0..n − 1] of orderable elements //Output: Returns “true” if A has no equal elements, “false” otherwise sort the array A for i ← 0 to n − 2 do if A[i] = A[i + 1] return false return true The running time of this algorithm is the sum of the time spent on sorting and the time spent on checking consecutive elements. Since the former requires at least n log n comparisons and the latter needs no more than n − 1 comparisons, it is the sorting part that will determine the overall efﬁciency of the algorithm. So, if we use a quadratic sorting algorithm here, the entire algorithm will not be more efﬁcient than the bruteforce one. But if we use a good sorting algorithm, such as mergesort, with worstcase efﬁciency in (n log n), the worstcase efﬁciency of the entire presortingbased algorithm will be also in (n log n): T (n) = Tsort (n) + Tscan(n) ∈ (n log n) + (n) = (n log n). EXAMPLE 2 Computing a mode A mode is a value that occurs most often in a given list of numbers. For example, for 5, 1, 5, 7, 6, 5, 7, the mode is 5. (If several different values occur most often, any of them can be considered a mode.) The bruteforce approach to computing a mode would scan the list and compute the frequencies of all its distinct values, then ﬁnd the value with the largest frequency. 1.
Sorting algorithms called radix sorts are linear but in terms of the total number of input bits. These algorithms work by comparing individual bits or pieces of keys rather than keys in their entirety. Although the running time of these algorithms is proportional to the number of input bits, they are still essentially n log n algorithms because the number of bits per key must be at least log2 n in order to accommodate n distinct keys of input.
204
TransformandConquer
In order to implement this idea, we can store the values already encountered, along with their frequencies, in a separate list. On each iteration, the ith element of the original list is compared with the values already encountered by traversing this auxiliary list. If a matching value is found, its frequency is incremented; otherwise, the current element is added to the list of distinct values seen so far with a frequency of 1. It is not difﬁcult to see that the worstcase input for this algorithm is a list with no equal elements. For such a list, its ith element is compared with i − 1 elements of the auxiliary list of distinct values seen so far before being added to the list with a frequency of 1. As a result, the worstcase number of comparisons made by this algorithm in creating the frequency list is C(n) =
n
(n − 1)n ∈ (n2 ). (i − 1) = 0 + 1 + . . . + (n − 1) = 2 i=1
The additional n − 1 comparisons needed to ﬁnd the largest frequency in the auxiliary list do not change the quadratic worstcase efﬁciency class of the algorithm. As an alternative, let us ﬁrst sort the input. Then all equal values will be adjacent to each other. To compute the mode, all we need to do is to ﬁnd the longest run of adjacent equal values in the sorted array. ALGORITHM
PresortMode(A[0..n − 1])
//Computes the mode of an array by sorting it ﬁrst //Input: An array A[0..n − 1] of orderable elements //Output: The array’s mode sort the array A i←0 //current run begins at position i modef requency ← 0 //highest frequency seen so far while i ≤ n − 1 do runlength ← 1; runvalue ← A[i] while i + runlength ≤ n − 1 and A[i + runlength] = runvalue runlength ← runlength + 1 if runlength > modef requency modef requency ← runlength; modevalue ← runvalue i ← i + runlength return modevalue The analysis here is similar to the analysis of Example 1: the running time of the algorithm will be dominated by the time spent on sorting since the remainder of the algorithm takes linear time (why?). Consequently, with an n log n sort, this method’s worstcase efﬁciency will be in a better asymptotic class than the worstcase efﬁciency of the bruteforce algorithm.
6.1
Presorting
205
EXAMPLE 3 Searching problem Consider the problem of searching for a given value v in a given array of n sortable items. The bruteforce solution here is sequential search (Section 3.1), which needs n comparisons in the worst case. If the array is sorted ﬁrst, we can then apply binary search, which requires only log2 n + 1 comparisons in the worst case. Assuming the most efﬁcient n log n sort, the total running time of such a searching algorithm in the worst case will be T (n) = Tsort (n) + Tsearch(n) = (n log n) + (log n) = (n log n), which is inferior to sequential search. The same will also be true for the averagecase efﬁciency. Of course, if we are to search in the same list more than once, the time spent on sorting might well be justiﬁed. (Problem 4 in this section’s exercises asks to estimate the minimum number of searches needed to justify presorting.)
Before we ﬁnish our discussion of presorting, we should mention that many, if not most, geometric algorithms dealing with sets of points use presorting in one way or another. Points can be sorted by one of their coordinates, or by their distance from a particular line, or by some angle, and so on. For example, presorting was used in the divideandconquer algorithms for the closestpair problem and for the convexhull problem, which were discussed in Section 5.5. Further, some problems for directed acyclic graphs can be solved more easily after topologically sorting the digraph in question. The problems of ﬁnding the longest and shortest paths in such digraphs (see the exercises for Sections 8.1 and 9.3) illustrate this point. Finally, most algorithms based on the greedy technique, which is the subject of Chapter 9, require presorting of their inputs as an intrinsic part of their operations.
Exercises 6.1 1. Consider the problem of ﬁnding the distance between the two closest numbers in an array of n numbers. (The distance between two numbers x and y is computed as x − y.) a. Design a presortingbased algorithm for solving this problem and determine its efﬁciency class. b. Compare the efﬁciency of this algorithm with that of the bruteforce algorithm (see Problem 9 in Exercises 1.2). 2. Let A = {a1, . . . , an} and B = {b1, . . . , bm} be two sets of numbers. Consider the problem of ﬁnding their intersection, i.e., the set C of all the numbers that are in both A and B. a. Design a bruteforce algorithm for solving this problem and determine its efﬁciency class. b. Design a presortingbased algorithm for solving this problem and determine its efﬁciency class.
206
TransformandConquer
3. Consider the problem of ﬁnding the smallest and largest elements in an array of n numbers. a. Design a presortingbased algorithm for solving this problem and determine its efﬁciency class. b. Compare the efﬁciency of the three algorithms: (i) the bruteforce algorithm, (ii) this presortingbased algorithm, and (iii) the divideandconquer algorithm (see Problem 2 in Exercises 5.1). 4. Estimate how many searches will be needed to justify time spent on presorting an array of 103 elements if sorting is done by mergesort and searching is done by binary search. (You may assume that all searches are for elements known to be in the array.) What about an array of 106 elements? 5. To sort or not to sort? Design a reasonably efﬁcient algorithm for solving each of the following problems and determine its efﬁciency class. a. You are given n telephone bills and m checks sent to pay the bills (n ≥ m). Assuming that telephone numbers are written on the checks, ﬁnd out who failed to pay. (For simplicity, you may also assume that only one check is written for a particular bill and that it covers the bill in full.) b. You have a ﬁle of n student records indicating each student’s number, name, home address, and date of birth. Find out the number of students from each of the 50 U.S. states. 6. Given a set of n ≥ 3 points in the Cartesian plane, connect them in a simple polygon, i.e., a closed path through all the points so that its line segments (the polygon’s edges) do not intersect (except for neighboring edges at their common vertex). For example, P3
P3 P1
P6 P2
P1
P6 P2
P4
P4 P5
P5
a. Does the problem always have a solution? Does it always have a unique solution? b. Design a reasonably efﬁcient algorithm for solving this problem and indicate its efﬁciency class. 7. You have an array of n real numbers and another integer s. Find out whether the array contains two elements whose sum is s. (For example, for the array 5, 9, 1, 3 and s = 6, the answer is yes, but for the same array and s = 7, the answer
6.1
Presorting
207
is no.) Design an algorithm for this problem with a better than quadratic time efﬁciency. 8. You have a list of n open intervals (a1, b1), (a2 , b2 ), . . . , (an, bn) on the real line. (An open interval (a, b) comprises all the points strictly between its endpoints a and b, i.e., (a, b) = {x a < x < b}.) Find the maximum number of these intervals that have a common point. For example, for the intervals (1, 4), (0, 3), (−1.5, 2), (3.6, 5), this maximum number is 3. Design an algorithm for this problem with a better than quadratic time efﬁciency. 9. Number placement Given a list of n distinct integers and a sequence of n boxes with preset inequality signs inserted between them, design an algorithm that places the numbers into the boxes to satisfy those inequalities. For example, the numbers 4, 6, 3, 1, 8 can be placed in the ﬁve boxes as shown below: 1 < 8 > 3 < 4 < 6 10. Maxima search a. A point (xi , yi ) in the Cartesian plane is said to be dominated by point (xj , yj ) if xi ≤ xj and yi ≤ yj with at least one of the two inequalities being strict. Given a set of n points, one of them is said to be a maximum of the set if it is not dominated by any other point in the set. For example, in the ﬁgure below, all the maximum points of the set of 10 points are circled.
y
x
Design an efﬁcient algorithm for ﬁnding all the maximum points of a given set of n points in the Cartesian plane. What is the time efﬁciency class of your algorithm? b. Give a few realworld applications of this algorithm.
208
TransformandConquer
11. Anagram detection a. Design an efﬁcient algorithm for ﬁnding all sets of anagrams in a large ﬁle such as a dictionary of English words [Ben00]. For example, eat, ate, and tea belong to one such set. b. Write a program implementing the algorithm.
6.2 Gaussian Elimination You are certainly familiar with systems of two linear equations in two unknowns: a11x + a12 y = b1 a21x + a22 y = b2 . Recall that unless the coefﬁcients of one equation are proportional to the coefﬁcients of the other, the system has a unique solution. The standard method for ﬁnding this solution is to use either equation to express one of the variables as a function of the other and then substitute the result into the other equation, yielding a linear equation whose solution is then used to ﬁnd the value of the second variable. In many applications, we need to solve a system of n equations in n unknowns: a11x1 + a12 x2 + . . . + a1nxn = b1 a21x1 + a22 x2 + . . . + a2nxn = b2 .. . an1x1 + an2 x2 + . . . + annxn = bn where n is a large number. Theoretically, we can solve such a system by generalizing the substitution method for solving systems of two linear equations (what general design technique would such a method be based upon?); however, the resulting algorithm would be extremely cumbersome. Fortunately, there is a much more elegant algorithm for solving systems of linear equations called Gaussian elimination.2 The idea of Gaussian elimination is to transform a system of n linear equations in n unknowns to an equivalent system (i.e., a system with the same solution as the original one) with an uppertriangular coefﬁcient matrix, a matrix with all zeros below its main diagonal:
2.
The method is named after Carl Friedrich Gauss (1777–1855), who—like other giants in the history of mathematics such as Isaac Newton and Leonhard Euler—made numerous fundamental contributions to both theoretical and computational mathematics. The method was known to the Chinese 1800 years before the Europeans rediscovered it.
6.2
a11 x1 + a12 x2 + . . . + a1n xn = b1
a11x1 + a12 x2 + . . . + a1nxn = b1 a21x1 + a22 x2 + . . . + a2nxn = b2 .. . an1x1 + an2 x2 + . . . + annxn = bn
209
Gaussian Elimination
⇒
a22 x2 + . . . + a2n xn = b2
.. .
ann xn = bn .
In matrix notations, we can write this as Ax = b where ⎡
a11 a12 ⎢ a21 a22 A=⎢ ⎣ ... an1 an2
... ... ...
⇒
A x = b ,
⎤ ⎡ ⎤ ⎡
a11 a12 b1 a1n
⎢ ⎥ ⎢ 0 a22 a2n ⎥ ⎥, b = ⎢ b.2 ⎥, A = ⎢ . ⎦ ⎣ .. ⎦ ⎣ . . ann bn 0 0
... ... ...
⎡ ⎤
⎤ a1n b1
⎢ b2 ⎥ a2n ⎥ ⎥, b = ⎢ . ⎥ . ⎦ ⎣.⎦ .
bn
ann
(We added primes to the matrix elements and righthand sides of the new system to stress the point that their values differ from their counterparts in the original system.) Why is the system with the uppertriangular coefﬁcient matrix better than a system with an arbitrary coefﬁcient matrix? Because we can easily solve the system with an uppertriangular coefﬁcient matrix by back substitutions as follows. First, we can immediately ﬁnd the value of xn from the last equation; then we can substitute this value into the next to last equation to get xn−1, and so on, until we substitute the known values of the last n − 1 variables into the ﬁrst equation, from which we ﬁnd the value of x1. So how can we get from a system with an arbitrary coefﬁcient matrix A to an equivalent system with an uppertriangular coefﬁcient matrix A ? We can do that through a series of the socalled elementary operations: exchanging two equations of the system replacing an equation with its nonzero multiple replacing an equation with a sum or difference of this equation and some multiple of another equation Since no elementary operation can change a solution to a system, any system that is obtained through a series of such operations will have the same solution as the original one. Let us see how we can get to a system with an uppertriangular matrix. First, we use a11 as a pivot to make all x1 coefﬁcients zeros in the equations below the ﬁrst one. Speciﬁcally, we replace the second equation with the difference between it and the ﬁrst equation multiplied by a21/a11 to get an equation with a zero coefﬁcient for x1. Doing the same for the third, fourth, and ﬁnally nth equation—with the multiples a31/a11, a41/a11, . . . , an1/a11 of the ﬁrst equation, respectively—makes all the coefﬁcients of x1 below the ﬁrst equation zero. Then we get rid of all the coefﬁcients of x2 by subtracting an appropriate multiple of the second equation from each of the equations below the second one. Repeating this
210
TransformandConquer
elimination for each of the ﬁrst n − 1 variables ultimately yields a system with an uppertriangular coefﬁcient matrix. Before we look at an example of Gaussian elimination, let us note that we can operate with just a system’s coefﬁcient matrix augmented, as its (n + 1)st column, with the equations’ righthand side values. In other words, we need to write explicitly neither the variable names nor the plus and equality signs. EXAMPLE 1
Solve the system by Gaussian elimination. 2x1 − x2 + x3 = 1 4x1 + x2 − x3 = 5 x1 + x2 + x3 = 0. ⎤ 2 −1 1 1 ⎣4 1 −1 5 ⎦ row 2 − 24 row 1 1 1 1 0 row 3 − 21 row 1 ⎤ ⎡ 2 −1 1 1 ⎣0 3 −3 3⎦ 3 1 1 row 3 − 21 row 2 − 0 2 2 2 ⎡
⎡
2 ⎣0 0
⎤ −1 1 1 3 −3 3⎦ 0 2 −2
Now we can obtain the solution by back substitutions: x3 = (−2)/2 = −1, x2 = (3 − (−3)x3)/3 = 0, and x1 = (1 − x3 − (−1)x2 )/2 = 1.
Here is pseudocode of the ﬁrst stage, called forward elimination, of the algorithm. ALGORITHM ForwardElimination(A[1..n, 1..n], b[1..n]) //Applies Gaussian elimination to matrix A of a system’s coefﬁcients, //augmented with vector b of the system’s righthand side values //Input: Matrix A[1..n, 1..n] and columnvector b[1..n] //Output: An equivalent uppertriangular matrix in place of A with the //corresponding righthand side values in the (n + 1)st column for i ← 1 to n do A[i, n + 1] ← b[i] //augments the matrix for i ← 1 to n − 1 do for j ← i + 1 to n do for k ← i to n + 1 do A[j, k] ← A[j, k] − A[i, k] ∗ A[j, i] / A[i, i]
6.2
Gaussian Elimination
211
There are two important observations to make about this pseudocode. First, it is not always correct: if A[i, i] = 0, we cannot divide by it and hence cannot use the ith row as a pivot for the ith iteration of the algorithm. In such a case, we should take advantage of the ﬁrst elementary operation and exchange the ith row with some row below it that has a nonzero coefﬁcient in the ith column. (If the system has a unique solution, which is the normal case for systems under consideration, such a row must exist.) Since we have to be prepared for the possibility of row exchanges anyway, we can take care of another potential difﬁculty: the possibility that A[i, i] is so small and consequently the scaling factor A[j, i]/A[i, i] so large that the new value of A[j, k] might become distorted by a roundoff error caused by a subtraction of two numbers of greatly different magnitudes.3 To avoid this problem, we can always look for a row with the largest absolute value of the coefﬁcient in the ith column, exchange it with the ith row, and then use the new A[i, i] as the ith iteration’s pivot. This modiﬁcation, called partial pivoting, guarantees that the magnitude of the scaling factor will never exceed 1. The second observation is the fact that the innermost loop is written with a glaring inefﬁciency. Can you ﬁnd it before checking the following pseudocode, which both incorporates partial pivoting and eliminates this inefﬁciency? ALGORITHM BetterForwardElimination(A[1..n, 1..n], b[1..n]) //Implements Gaussian elimination with partial pivoting //Input: Matrix A[1..n, 1..n] and columnvector b[1..n] //Output: An equivalent uppertriangular matrix in place of A and the //corresponding righthand side values in place of the (n + 1)st column for i ← 1 to n do A[i, n + 1] ← b[i] //appends b to A as the last column for i ← 1 to n − 1 do pivotrow ← i for j ← i + 1 to n do if A[j, i] > A[pivotrow, i] pivotrow ← j for k ← i to n + 1 do swap(A[i, k], A[pivotrow, k]) for j ← i + 1 to n do temp ← A[j, i] / A[i, i] for k ← i to n + 1 do A[j, k] ← A[j, k] − A[i, k] ∗ temp Let us ﬁnd the time efﬁciency of this algorithm. Its innermost loop consists of a single line, A[j, k] ← A[j, k] − A[i, k] ∗ temp, 3.
We discuss roundoff errors in more detail in Section 11.4.
212
TransformandConquer
which contains one multiplication and one subtraction. On most computers, multiplication is unquestionably more expensive than addition/subtraction, and hence it is multiplication that is usually quoted as the algorithm’s basic operation.4 The standard summation formulas and rules reviewed in Section 2.3 (see also Appendix A) are very helpful in the following derivation: C(n) =
n n+1 n−1
i=1 j =i+1 k=i
=
1=
n n−1
(n + 1 − i + 1) =
i=1 j =i+1
(n + 2 − i)
i=1 j =i+1
n−1
n−1
i=1
i=1
(n + 2 − i)(n − (i + 1) + 1) =
n−1 n
(n + 2 − i)(n − i)
= (n + 1)(n − 1) + n(n − 2) + . . . + 3 . 1 =
n−1 n−1 n−1
(n − 1)n (n − 1)n(2n − 1) +2 (j + 2)j = j2 + 2j = 6 2 j =1 j =1 j =1
=
n(n − 1)(2n + 5) 1 3 ≈ n ∈ (n3). 6 3
Since the second (back substitution) stage of Gaussian elimination is in (n2 ), as you are asked to show in the exercises, the running time is dominated by the cubic elimination stage, making the entire algorithm cubic as well. Theoretically, Gaussian elimination always either yields an exact solution to a system of linear equations when the system has a unique solution or discovers that no such solution exists. In the latter case, the system will have either no solutions or inﬁnitely many of them. In practice, solving systems of signiﬁcant size on a computer by this method is not nearly so straightforward as the method would lead us to believe. The principal difﬁculty lies in preventing an accumulation of roundoff errors (see Section 11.4). Consult textbooks on numerical analysis that analyze this and other implementation issues in great detail.
LU Decomposition Gaussian elimination has an interesting and very useful byproduct called LU decomposition of the coefﬁcient matrix. In fact, modern commercial implementations of Gaussian elimination are based on such a decomposition rather than on the basic algorithm outlined above. EXAMPLE Let us return to the example in the beginning of this section, where we applied Gaussian elimination to the matrix 4.
As we mentioned in Section 2.1, on some computers multiplication is not necessarily more expensive than addition/subtraction. For this algorithm, this point is moot since we can simply count the number of times the innermost loop is executed, which is, of course, exactly the same number as the number of multiplications and the number of subtractions there.
6.2
Gaussian Elimination
213
⎡
⎤ 2 −1 1 A=⎣4 1 −1 ⎦ . 1 1 1 Consider the lowertriangular matrix L made up of 1’s on its main diagonal and the row multiples used in the forward elimination process ⎡ ⎤ 1 0 0 L=⎣2 1 0⎦ 1 1 1 2 2 and the uppertriangular matrix U that was the result of this elimination ⎡ ⎤ 2 −1 1 U =⎣0 3 −3 ⎦ . 0 0 2 It turns out that the product LU of these matrices is equal to matrix A. (For this particular pair of L and U , you can verify this fact by direct multiplication, but as a general proposition, it needs, of course, a proof, which we omit here.) Therefore, solving the system Ax = b is equivalent to solving the system LU x = b. The latter system can be solved as follows. Denote y = U x, then Ly = b. Solve the system Ly = b ﬁrst, which is easy to do because L is a lowertriangular matrix; then solve the system U x = y, with the uppertriangular matrix U , to ﬁnd x. Thus, for the system at the beginning of this section, we ﬁrst solve Ly = b: ⎡ ⎤⎡ ⎤ ⎡ ⎤ 1 0 0 1 y1 ⎣ 2 1 0 ⎦ ⎣ y2 ⎦ = ⎣ 5 ⎦ . 1 1 0 1 y3 2 2 Its solution is y1 = 1,
y2 = 5 − 2y1 = 3,
1 1 y3 = 0 − y1 − y2 = −2. 2 2
Solving U x = y means solving ⎤ ⎡ ⎤⎡ ⎤ ⎡ 1 2 −1 1 x1 ⎣0 3 −3 ⎦ ⎣ x2 ⎦ = ⎣ 3 ⎦ , −2 x3 0 0 2 and the solution is x3 = (−2)/2 = −1, x2 = (3 − (−3)x3)/3 = 0, x1 = (1 − x3 − (−1)x2 )/2 = 1. Note that once we have the LU decomposition of matrix A, we can solve systems Ax = b with as many righthand side vectors b as we want to, one at a time. This is a distinct advantage over the classic Gaussian elimination discussed earlier. Also note that the LU decomposition does not actually require extra memory, because we can store the nonzero part of U in the uppertriangular part of A
214
TransformandConquer
(including the main diagonal) and store the nontrivial part of L below the main diagonal of A.
Computing a Matrix Inverse Gaussian elimination is a very useful algorithm that tackles one of the most important problems of applied mathematics: solving systems of linear equations. In fact, Gaussian elimination can also be applied to several other problems of linear algebra, such as computing a matrix inverse. The inverse of an n × n matrix A is an n × n matrix, denoted A−1, such that AA−1 = I, where I is the n × n identity matrix (the matrix with all zero elements except the main diagonal elements, which are all ones). Not every square matrix has an inverse, but when it exists, the inverse is unique. If a matrix A does not have an inverse, it is called singular. One can prove that a matrix is singular if and only if one of its rows is a linear combination (a sum of some multiples) of the other rows. A convenient way to check whether a matrix is nonsingular is to apply Gaussian elimination: if it yields an uppertriangular matrix with no zeros on the main diagonal, the matrix is nonsingular; otherwise, it is singular. So being singular is a very special situation, and most square matrices do have their inverses. Theoretically, inverse matrices are very important because they play the role of reciprocals in matrix algebra, overcoming the absence of the explicit division operation for matrices. For example, in a complete analogy with a linear equation in one unknown ax = b whose solution can be written as x = a −1b (if a is not zero), we can express a solution to a system of n equations in n unknowns Ax = b as x = A−1b (if A is nonsingular) where b is, of course, a vector, not a number. According to the deﬁnition of the inverse matrix for a nonsingular n × n matrix A, to compute it we need to ﬁnd n2 numbers xij , 1 ≤ i, j ≤ n, such that ⎡ ⎤ ⎤⎡ ⎤ ⎡ a11 a12 . . . a1n x11 x12 . . . x1n 1 0 ... 0 ⎢ a21 a22 . . . a2n ⎥ ⎢ x21 x22 . . . x2n ⎥ ⎢ 0 1 . . . 0 ⎥ ⎢ . ⎥. ⎥⎢ . ⎥=⎢ . ⎣ .. ⎦ ⎦ ⎣ .. ⎦ ⎣ .. an1 an2
...
ann
xn1 xn2
...
xnn
0
0
...
1
We can ﬁnd the unknowns by solving n systems of linear equations that have the same coefﬁcient matrix A, the vector of unknowns x j is the j th column of the inverse, and the righthand side vector ej is the j th column of the identity matrix (1 ≤ j ≤ n): Ax j = ej . We can solve these systems by applying Gaussian elimination to matrix A augmented by the n × n identity matrix. Better yet, we can use forward elimination to ﬁnd the LU decomposition of A and then solve the systems LU x j = ej , j = 1, . . . , n, as explained earlier.
6.2
Gaussian Elimination
215
Computing a Determinant Another problem that can be solved by Gaussian elimination is computing a determinant. The determinant of an n × n matrix A, denoted det A or A, is a number whose value can be deﬁned recursively as follows. If n = 1, i.e., if A consists of a single element a11, det A is equal to a11; for n > 1, det A is computed by the recursive formula det A =
n
sj a1j det Aj ,
j =1
where sj is +1 if j is odd and −1 if j is even, a1j is the element in row 1 and column j , and Aj is the (n − 1) × (n − 1) matrix obtained from matrix A by deleting its row 1 and column j . In particular, for a 2 × 2 matrix, the deﬁnition implies a formula that is easy to remember: a11 a12 = a11 det [a22 ] − a12 det [a21] = a11a22 − a12 a21. det a21 a22 In other words, the determinant of a 2 × 2 matrix is simply equal to the difference between the products of its diagonal elements. For a 3 × 3 matrix, we get ⎤ ⎡ a11 a12 a13 det ⎣ a21 a22 a23 ⎦ a31 a32 a33 a21 a23 a21 a22 a22 a23 = a11 det − a12 det + a13 det a32 a33 a31 a33 a31 a32 = a11a22 a33 + a12 a23a31 + a13a21a32 − a11a23a32 − a12 a21a33 − a13a22 a31. Incidentally, this formula is very handy in a variety of applications. In particular, we used it twice already in Section 5.5 as a part of the quickhull algorithm. But what if we need to compute a determinant of a large matrix? Although this is a task that is rarely needed in practice, it is worth discussing nevertheless. Using the recursive deﬁnition can be of little help because it implies computing the sum of n!terms. Here, Gaussian elimination comes to the rescue again. The central point is the fact that the determinant of an uppertriangular matrix is equal to the product of elements on its main diagonal, and it is easy to see how elementary operations employed by the elimination algorithm inﬂuence the determinant’s value. (Basically, it either remains unchanged or changes a sign or is multiplied by the constant used by the elimination algorithm.) As a result, we can compute the determinant of an n × n matrix in cubic time. Determinants play an important role in the theory of systems of linear equations. Speciﬁcally, a system of n linear equations in n unknowns Ax = b has a unique solution if and only if the determinant of its coefﬁcient matrix det A is
216
TransformandConquer
not equal to zero. Moreover, this solution can be found by the formulas called Cramer’s rule, det Aj det An det A1 , . . . , xj = , . . . , xn = , det A det A det A where det Aj is the determinant of the matrix obtained by replacing the j th column of A by the column b. You are asked to investigate in the exercises whether using Cramer’s rule is a good algorithm for solving systems of linear equations. x1 =
Exercises 6.2 1. Solve the following system by Gaussian elimination: x1 + x 2 + x 3 = 2 2x1 + x2 + x3 = 3 x1 − x2 + 3x3 = 8. 2. a. Solve the system of the previous question by the LU decomposition method. b. From the standpoint of general algorithm design techniques, how would you classify the LU decomposition method? 3. Solve the system of Problem 1 by computing the inverse of its coefﬁcient matrix and then multiplying it by the vector on the righthand side. 4. Would it be correct to get the efﬁciency class of the forward elimination stage of Gaussian elimination as follows? C(n) =
n n+1 n−1
i=1 j =i+1 k=i
=
1=
n−1
(n + 2 − i)(n − i) i=1
n−1
[(n + 2)n − i(2n + 2) + i 2 ]
i=1 n−1 n−1 n−1
(n + 2)n − (2n + 2)i + i 2. = i=1
i=1
i=1
3 (n + 2)n ∈ (n3), s2 (n) = n−1 Since s1(n) = n−1 i=1 i=1 (2n + 2)i ∈ (n ), and n−1 2 s3(n) = i=1 i ∈ (n3), s1(n) − s2 (n) + s3(n) ∈ (n3). 5. Write pseudocode for the backsubstitution stage of Gaussian elimination and show that its running time is in (n2 ). 6. Assuming that division of two numbers takes three times longer than their multiplication, estimate how much faster BetterForwardElimination is than ForwardElimination. (Of course, you should also assume that a compiler is not going to eliminate the inefﬁciency in ForwardElimination.)
6.2
Gaussian Elimination
217
7. a. Give an example of a system of two linear equations in two unknowns that has a unique solution and solve it by Gaussian elimination. b. Give an example of a system of two linear equations in two unknowns that has no solution and apply Gaussian elimination to it. c. Give an example of a system of two linear equations in two unknowns that has inﬁnitely many solutions and apply Gaussian elimination to it. 8. The GaussJordan elimination method differs from Gaussian elimination in that the elements above the main diagonal of the coefﬁcient matrix are made zero at the same time and by the same use of a pivot row as the elements below the main diagonal. a. Apply the GaussJordan method to the system of Problem 1 of these exercises. b. What general design strategy is this algorithm based on? c. In general, how many multiplications are made by this method in solving a system of n equations in n unknowns? How does this compare with the number of multiplications made by the Gaussian elimination method in both its elimination and backsubstitution stages? 9. A system Ax = b of n linear equations in n unknowns has a unique solution if and only if det A = 0. Is it a good idea to check this condition before applying Gaussian elimination to the system? 10. a. Apply Cramer’s rule to solve the system of Problem 1 of these exercises. b. Estimate how many times longer it will take to solve a system of n linear equations in n unknowns by Cramer’s rule than by Gaussian elimination. Assume that all the determinants in Cramer’s rule formulas are computed independently by Gaussian elimination. 11. Lights out This oneperson game is played on an n × n board composed of 1 × 1 light panels. Each panel has a switch that can be turned on and off, thereby toggling the on/off state of this and four vertically and horizontally adjacent panels. (Of course, toggling a corner square affects a total of three panels, and toggling a noncorner panel on the board’s border affects a total of four squares.) Given an initial subset of lighted squares, the goal is to turn all the lights off. a. Show that an answer can be found by solving a system of linear equations with 0/1 coefﬁcients and righthand sides using the modulo 2 arithmetic. b. Use Gaussian elimination to solve the 2 × 2 “allones” instance of this problem, where all the panels of the 2 × 2 board are initially lit. c. Use Gaussian elimination to solve the 3 × 3 “allones” instance of this problem, where all the panels of the 3 × 3 board are initially lit.
218
TransformandConquer
6.3 Balanced Search Trees In Sections 1.4, 4.5, and 5.3, we discussed the binary search tree—one of the principal data structures for implementing dictionaries. It is a binary tree whose nodes contain elements of a set of orderable items, one element per node, so that all elements in the left subtree are smaller than the element in the subtree’s root, and all the elements in the right subtree are greater than it. Note that this transformation from a set to a binary search tree is an example of the representationchange technique. What do we gain by such transformation compared to the straightforward implementation of a dictionary by, say, an array? We gain in the time efﬁciency of searching, insertion, and deletion, which are all in (log n), but only in the average case. In the worst case, these operations are in (n) because the tree can degenerate into a severely unbalanced one with its height equal to n − 1. Computer scientists have expended a lot of effort in trying to ﬁnd a structure that preserves the good properties of the classical binary search tree—principally, the logarithmic efﬁciency of the dictionary operations and having the set’s elements sorted—but avoids its worstcase degeneracy. They have come up with two approaches. The ﬁrst approach is of the instancesimpliﬁcation variety: an unbalanced binary search tree is transformed into a balanced one. Because of this, such trees are called selfbalancing. Speciﬁc implementations of this idea differ by their deﬁnition of balance. An AVL tree requires the difference between the heights of the left and right subtrees of every node never exceed 1. A redblack tree tolerates the height of one subtree being twice as large as the other subtree of the same node. If an insertion or deletion of a new node creates a tree with a violated balance requirement, the tree is restructured by one of a family of special transformations called rotations that restore the balance required. In this section, we will discuss only AVL trees. Information about other types of binary search trees that utilize the idea of rebalancing via rotations, including redblack trees and splay trees, can be found in the references [Cor09], [Sed02], and [Tar83]. The second approach is of the representationchange variety: allow more than one element in a node of a search tree. Speciﬁc cases of such trees are 23 trees, 234 trees, and more general and important Btrees. They differ in the number of elements admissible in a single node of a search tree, but all are perfectly balanced. We discuss the simplest case of such trees, the 23 tree, in this section, leaving the discussion of Btrees for Chapter 7.
AVL Trees AVL trees were invented in 1962 by two Russian scientists, G. M. AdelsonVelsky and E. M. Landis [Ade62], after whom this data structure is named.
6.3 1
2
10
10 1
0 5 1 4 0 2
219
Balanced Search Trees
0 5
20 –1 7
0 12 0 8
(a)
0 2
0 20
1
–1
4
7 0 8
(b)
FIGURE 6.2 (a) AVL tree. (b) Binary search tree that is not an AVL tree. The numbers above the nodes indicate the nodes’ balance factors.
DEFINITION An AVL tree is a binary search tree in which the balance factor of every node, which is deﬁned as the difference between the heights of the node’s left and right subtrees, is either 0 or +1 or −1. (The height of the empty tree is deﬁned as −1. Of course, the balance factor can also be computed as the difference between the numbers of levels rather than the height difference of the node’s left and right subtrees.) For example, the binary search tree in Figure 6.2a is an AVL tree but the one in Figure 6.2b is not. If an insertion of a new node makes an AVL tree unbalanced, we transform the tree by a rotation. A rotation in an AVL tree is a local transformation of its subtree rooted at a node whose balance has become either +2 or −2. If there are several such nodes, we rotate the tree rooted at the unbalanced node that is the closest to the newly inserted leaf. There are only four types of rotations; in fact, two of them are mirror images of the other two. In their simplest form, the four rotations are shown in Figure 6.3. The ﬁrst rotation type is called the single right rotation, or Rrotation. (Imagine rotating the edge connecting the root and its left child in the binary tree in Figure 6.3a to the right.) Figure 6.4 presents the single Rrotation in its most general form. Note that this rotation is performed after a new key is inserted into the left subtree of the left child of a tree whose root had the balance of +1 before the insertion. The symmetric single left rotation, or Lrotation, is the mirror image of the single Rrotation. It is performed after a new key is inserted into the right subtree of the right child of a tree whose root had the balance of −1 before the insertion. (You are asked to draw a diagram of the general case of the single Lrotation in the exercises.)
220
TransformandConquer 0 2
2 3 1 2
R
0 1
0 3
0 1
(a) –2
0 2
1 –1 2
L
0 3
0 1
0 3
(b) 0 2
2 3 –1 1
LR
0 1
0 3
0 2
(c) 0 2
–2 1 1 3
RL
0 1
0 3
0 2
(d)
FIGURE 6.3 Four rotation types for AVL trees with three nodes. (a) Single Rrotation. (b) Single Lrotation. (c) Double LRrotation. (d) Double RLrotation.
The second rotation type is called the double leftright rotation (LRrotation). It is, in fact, a combination of two rotations: we perform the Lrotation of the left subtree of root r followed by the Rrotation of the new tree rooted at r (Figure 6.5). It is performed after a new key is inserted into the right subtree of the left child of a tree whose root had the balance of +1 before the insertion.
6.3
221
Balanced Search Trees
single Rrotation r
c r
c
T1
T3 T1
T2
T2
T3
FIGURE 6.4 General form of the Rrotation in the AVL tree. A shaded node is the last one inserted.
double LRrotation g
r
r
c
c g T4 T1
T2
T3
T2
T3
T1
T4 or
or
FIGURE 6.5 General form of the double LRrotation in the AVL tree. A shaded node is the last one inserted. It can be either in the left subtree or in the right subtree of the root’s grandchild.
The double rightleft rotation (RLrotation) is the mirror image of the double LRrotation and is left for the exercises. Note that the rotations are not trivial transformations, though fortunately they can be done in constant time. Not only should they guarantee that a resulting tree is balanced, but they should also preserve the basic requirements of a binary search tree. For example, in the initial tree of Figure 6.4, all the keys of subtree T1 are smaller than c, which is smaller than all the keys of subtree T2 , which are smaller than r, which is smaller than all the keys of subtree T3. And the same relationships among the key values hold, as they must, for the balanced tree after the rotation.
222
TransformandConquer –1 5
0 5
–2 5
0 6
L(5) 0 5
–1 6
0 6
0 8
0 8 1 6 1 5
1 6
2 6 2 5
0 8
0 8
0 3
R (5) 0 2
1 3
0 3
0 8 0 5
0 2
0 5
2 6 0 8
–1 3
0 3
LR (6) 0 2
1 5
0 2
–1 6 0 4
0 8
0 4
0 3 0 2
–1 5
–2 6 1 8
0 4
RL(6)
0 5
0 3 0 2
0 7 0 4
0 6
0 8
0 7
FIGURE 6.6 Construction of an AVL tree for the list 5, 6, 8, 3, 2, 4, 7 by successive insertions. The parenthesized number of a rotation’s abbreviation indicates the root of the tree being reorganized.
An example of constructing an AVL tree for a given list of numbers is shown in Figure 6.6. As you trace the algorithm’s operations, keep in mind that if there are several nodes with the ±2 balance, the rotation is done for the tree rooted at the unbalanced node that is the closest to the newly inserted leaf. How efﬁcient are AVL trees? As with any search tree, the critical characteristic is the tree’s height. It turns out that it is bounded both above and below
6.3
Balanced Search Trees
223
by logarithmic functions. Speciﬁcally, the height h of any AVL tree with n nodes satisﬁes the inequalities log2 n ≤ h < 1.4405 log2 (n + 2) − 1.3277. (These weirdlooking constants are roundoffs of some irrational numbers related to Fibonacci numbers and the golden ratio—see Section 2.5.) The inequalities immediately imply that the operations of searching and insertion are (log n) in the worst case. Getting an exact formula for the average height of an AVL tree constructed for random lists of keys has proved to be difﬁcult, but it is known from extensive experiments that it is about 1.01log2 n + 0.1 except when n is small [KnuIII, p. 468]. Thus, searching in an AVL tree requires, on average, almost the same number of comparisons as searching in a sorted array by binary search. The operation of key deletion in an AVL tree is considerably more difﬁcult than insertion, but fortunately it turns out to be in the same efﬁciency class as insertion, i.e., logarithmic. These impressive efﬁciency characteristics come at a price, however. The drawbacks of AVL trees are frequent rotations and the need to maintain balances for its nodes. These drawbacks have prevented AVL trees from becoming the standard structure for implementing dictionaries. At the same time, their underlying idea—that of rebalancing a binary search tree via rotations—has proved to be very fruitful and has led to discoveries of other interesting variations of the classical binary search tree.
23 Trees As mentioned at the beginning of this section, the second idea of balancing a search tree is to allow more than one key in the same node of such a tree. The simplest implementation of this idea is 23 trees, introduced by the U.S. computer scientist John Hopcroft in 1970. A 23 tree is a tree that can have nodes of two kinds: 2nodes and 3nodes. A 2node contains a single key K and has two children: the left child serves as the root of a subtree whose keys are less than K, and the right child serves as the root of a subtree whose keys are greater than K. (In other words, a 2node is the same kind of node we have in the classical binary search tree.) A 3node contains two ordered keys K1 and K2 (K1 < K2 ) and has three children. The leftmost child serves as the root of a subtree with keys less than K1, the middle child serves as the root of a subtree with keys between K1 and K2 , and the rightmost child serves as the root of a subtree with keys greater than K2 (Figure 6.7). The last requirement of the 23 tree is that all its leaves must be on the same level. In other words, a 23 tree is always perfectly heightbalanced: the length of a path from the root to a leaf is the same for every leaf. It is this property that we “buy” by allowing more than one key in the same node of a search tree. Searching for a given key K in a 23 tree is quite straightforward. We start at the root. If the root is a 2node, we act as if it were a binary search tree: we either stop if K is equal to the root’s key or continue the search in the left or right
224
TransformandConquer
K
< K1
(K1, K2)
> K2
FIGURE 6.7 Two kinds of nodes of a 23 tree.
subtree if K is, respectively, smaller or larger than the root’s key. If the root is a 3node, we know after no more than two key comparisons whether the search can be stopped (if K is equal to one of the root’s keys) or in which of the root’s three subtrees it needs to be continued. Inserting a new key in a 23 tree is done as follows. First of all, we always insert a new key K in a leaf, except for the empty tree. The appropriate leaf is found by performing a search for K. If the leaf in question is a 2node, we insert K there as either the ﬁrst or the second key, depending on whether K is smaller or larger than the node’s old key. If the leaf is a 3node, we split the leaf in two: the smallest of the three keys (two old ones and the new key) is put in the ﬁrst leaf, the largest key is put in the second leaf, and the middle key is promoted to the old leaf’s parent. (If the leaf happens to be the tree’s root, a new root is created to accept the middle key.) Note that promotion of a middle key to its parent can cause the parent’s overﬂow (if it was a 3node) and hence can lead to several node splits along the chain of the leaf’s ancestors. An example of a 23 tree construction is given in Figure 6.8. As for any search tree, the efﬁciency of the dictionary operations depends on the tree’s height. So let us ﬁrst ﬁnd an upper bound for it. A 23 tree of height h with the smallest number of keys is a full tree of 2nodes (such as the ﬁnal tree in Figure 6.8 for h = 2). Therefore, for any 23 tree of height h with n nodes, we get the inequality n ≥ 1 + 2 + . . . + 2h = 2h+1 − 1, and hence h ≤ log2 (n + 1) − 1. On the other hand, a 23 tree of height h with the largest number of keys is a full tree of 3nodes, each with two keys and three children. Therefore, for any 23 tree with n nodes, n ≤ 2 . 1 + 2 . 3 + . . . + 2 . 3h = 2(1 + 3 + . . . + 3h) = 3h+1 − 1
6.3
225
Balanced Search Trees
8 9
5, 8, 9
5, 9
8
5
9
3, 5
3, 8
8 9
2, 3, 5
2
5
9
3, 8 9
2
9
4, 5
5 3, 5, 8
3, 8 2
4, 5, 7
9
2
4
3
7
9
8
2
4
7
9
FIGURE 6.8 Construction of a 23 tree for the list 9, 5, 8, 3, 2, 4, 7.
and hence h ≥ log3(n + 1) − 1. These lower and upper bounds on height h, log3(n + 1) − 1 ≤ h ≤ log2 (n + 1) − 1, imply that the time efﬁciencies of searching, insertion, and deletion are all in (log n) in both the worst and average case. We consider a very important generalization of 23 trees, called Btrees, in Section 7.4.
Exercises 6.3 1. Which of the following binary trees are AVL trees? 5 3
5 6
2
4 8
6
2 1
(a)
5 3 8
3
7
(b)
1
6 2
7
9
9
(c)
2. a. For n = 1, 2, 3, 4, and 5, draw all the binary trees with n nodes that satisfy the balance requirement of AVL trees.
226
TransformandConquer
b. Draw a binary tree of height 4 that can be an AVL tree and has the smallest number of nodes among all such trees. 3. Draw diagrams of the single Lrotation and of the double RLrotation in their general form. 4. For each of the following lists, construct an AVL tree by inserting their elements successively, starting with the empty tree. a. 1, 2, 3, 4, 5, 6 b. 6, 5, 4, 3, 2, 1 c. 3, 6, 5, 1, 2, 4 5. a. For an AVL tree containing real numbers, design an algorithm for computing the range (i.e., the difference between the largest and smallest numbers in the tree) and determine its worstcase efﬁciency. b. True or false: The smallest and the largest keys in an AVL tree can always be found on either the last level or the nexttolast level? 6. Write a program for constructing an AVL tree for a given list of n distinct integers. 7. a. Construct a 23 tree for the list C, O, M, P, U, T, I, N, G. Use the alphabetical order of the letters and insert them successively starting with the empty tree. b. Assuming that the probabilities of searching for each of the keys (i.e., the letters) are the same, ﬁnd the largest number and the average number of key comparisons for successful searches in this tree. 8. Let TB and T23 be, respectively, a classical binary search tree and a 23 tree constructed for the same list of keys inserted in the corresponding trees in the same order. True or false: Searching for the same key in T23 always takes fewer or the same number of key comparisons as searching in TB ? 9. For a 23 tree containing real numbers, design an algorithm for computing the range (i.e., the difference between the largest and smallest numbers in the tree) and determine its worstcase efﬁciency. 10. Write a program for constructing a 23 tree for a given list of n integers.
6.4 Heaps and Heapsort The data structure called the “heap” is deﬁnitely not a disordered pile of items as the word’s deﬁnition in a standard dictionary might suggest. Rather, it is a clever, partially ordered data structure that is especially suitable for implementing priority queues. Recall that a priority queue is a multiset of items with an orderable characteristic called an item’s priority, with the following operations:
6.4
10
10
5 4
7 2
227
Heaps and Heapsort
1
10
5
7 2
1
5 6
7 2
1
FIGURE 6.9 Illustration of the deﬁnition of heap: only the leftmost tree is a heap.
ﬁnding an item with the highest (i.e., largest) priority deleting an item with the highest priority adding a new item to the multiset It is primarily an efﬁcient implementation of these operations that makes the heap both interesting and useful. Priority queues arise naturally in such applications as scheduling job executions by computer operating systems and trafﬁc management by communication networks. They also arise in several important algorithms, e.g., Prim’s algorithm (Section 9.1), Dijkstra’s algorithm (Section 9.3), Huffman encoding (Section 9.4), and branchandbound applications (Section 12.2). The heap is also the data structure that serves as a cornerstone of a theoretically important sorting algorithm called heapsort. We discuss this algorithm after we deﬁne the heap and investigate its basic properties.
Notion of the Heap DEFINITION A heap can be deﬁned as a binary tree with keys assigned to its nodes, one key per node, provided the following two conditions are met: 1.
2.
The shape property—the binary tree is essentially complete (or simply complete), i.e., all its levels are full except possibly the last level, where only some rightmost leaves may be missing. The parental dominance or heap property—the key in each node is greater than or equal to the keys in its children. (This condition is considered automatically satisﬁed for all leaves.)5
For example, consider the trees of Figure 6.9. The ﬁrst tree is a heap. The second one is not a heap, because the tree’s shape property is violated. And the third one is not a heap, because the parental dominance fails for the node with key 5. Note that key values in a heap are ordered top down; i.e., a sequence of values on any path from the root to a leaf is decreasing (nonincreasing, if equal keys are allowed). However, there is no lefttoright order in key values; i.e., there is no 5.
Some authors require the key at each node to be less than or equal to the keys at its children. We call this variation a minheap.
228
TransformandConquer
10 the array representation 8
7
5
2
1
6
index value
0
2
3
4
5
6
7
8
9
10
10 8
1
7
5
2
1
6
3
5
1
parents 3
5
leaves
1
FIGURE 6.10 Heap and its array representation.
relationship among key values for nodes either on the same level of the tree or, more generally, in the left and right subtrees of the same node. Here is a list of important properties of heaps, which are not difﬁcult to prove (check these properties for the heap of Figure 6.10, as an example). 1. 2. 3. 4.
There exists exactly one essentially complete binary tree with n nodes. Its height is equal to log2 n. The root of a heap always contains its largest element. A node of a heap considered with all its descendants is also a heap. A heap can be implemented as an array by recording its elements in the topdown, lefttoright fashion. It is convenient to store the heap’s elements in positions 1 through n of such an array, leaving H [0] either unused or putting there a sentinel whose value is greater than every element in the heap. In such a representation, a. the parental node keys will be in the ﬁrst n/2 positions of the array, while the leaf keys will occupy the last n/2 positions; b. the children of a key in the array’s parental position i (1 ≤ i ≤ n/2) will be in positions 2i and 2i + 1, and, correspondingly, the parent of a key in position i (2 ≤ i ≤ n) will be in position i/2.
Thus, we could also deﬁne a heap as an array H [1..n] in which every element in position i in the ﬁrst half of the array is greater than or equal to the elements in positions 2i and 2i + 1, i.e., H [i] ≥ max{H [2i], H [2i + 1]}
for i = 1, . . . , n/2.
(Of course, if 2i + 1 > n, just H [i] ≥ H [2i] needs to be satisﬁed.) While the ideas behind the majority of algorithms dealing with heaps are easier to understand if we think of heaps as binary trees, their actual implementations are usually much simpler and more efﬁcient with arrays. How can we construct a heap for a given list of keys? There are two principal alternatives for doing this. The ﬁrst is the bottomup heap construction algorithm illustrated in Figure 6.11. It initializes the essentially complete binary tree with n nodes by placing keys in the order given and then “heapiﬁes” the tree as follows. Starting with the last parental node, the algorithm checks whether the parental
6.4
2
2
9 6
7 5
8
6
8 5
7
9
8
6
5
7
9
9
8 5
2
9
2
6
229
Heaps and Heapsort
7
9
2 6
8 5
7
6 2
8 5
7
FIGURE 6.11 Bottomup construction of a heap for the list 2, 9, 7, 6, 5, 8. The doubleheaded arrows show key comparisons verifying the parental dominance.
dominance holds for the key in this node. If it does not, the algorithm exchanges the node’s key K with the larger key of its children and checks whether the parental dominance holds for K in its new position. This process continues until the parental dominance for K is satisﬁed. (Eventually, it has to because it holds automatically for any key in a leaf.) After completing the “heapiﬁcation” of the subtree rooted at the current parental node, the algorithm proceeds to do the same for the node’s immediate predecessor. The algorithm stops after this is done for the root of the tree. ALGORITHM HeapBottomUp(H [1..n]) //Constructs a heap from elements of a given array // by the bottomup algorithm //Input: An array H [1..n] of orderable items //Output: A heap H [1..n] for i ← n/2 downto 1 do k ← i; v ← H [k] heap ← false while not heap and 2 ∗ k ≤ n do j ←2∗k if j < n //there are two children if H [j ] < H [j + 1] j ← j + 1 if v ≥ H [j ] heap ← true else H [k] ← H [j ]; k ← j H [k] ← v How efﬁcient is this algorithm in the worst case? Assume, for simplicity, that n = 2k − 1 so that a heap’s tree is full, i.e., the largest possible number of
230
TransformandConquer
nodes occurs on each level. Let h be the height of the tree. According to the ﬁrst property of heaps in the list at the beginning of the section, h = log2 n or just log2 (n + 1) − 1 = k − 1 for the speciﬁc values of n we are considering. Each key on level i of the tree will travel to the leaf level h in the worst case of the heap construction algorithm. Since moving to the next level down requires two comparisons—one to ﬁnd the larger child and the other to determine whether the exchange is required—the total number of key comparisons involving a key on level i will be 2(h − i). Therefore, the total number of key comparisons in the worst case will be Cworst (n) =
h−1
2(h − i) =
i=0 level i keys
h−1
2(h − i)2i = 2(n − log2 (n + 1)),
i=0
where the validity of the last equality can be proved either by using the closedform formula for the sum hi=1 i2i (see Appendix A) or by mathematical induction on h. Thus, with this bottomup algorithm, a heap of size n can be constructed with fewer than 2n comparisons. The alternative (and less efﬁcient) algorithm constructs a heap by successive insertions of a new key into a previously constructed heap; some people call it the topdown heap construction algorithm. So how can we insert a new key K into a heap? First, attach a new node with key K in it after the last leaf of the existing heap. Then sift K up to its appropriate place in the new heap as follows. Compare K with its parent’s key: if the latter is greater than or equal to K, stop (the structure is a heap); otherwise, swap these two keys and compare K with its new parent. This swapping continues until K is not greater than its last parent or it reaches the root (illustrated in Figure 6.12). Obviously, this insertion operation cannot require more key comparisons than the heap’s height. Since the height of a heap with n nodes is about log2 n, the time efﬁciency of insertion is in O(log n). How can we delete an item from a heap? We consider here only the most important case of deleting the root’s key, leaving the question about deleting an arbitrary key in a heap for the exercises. (Authors of textbooks like to do such things to their readers, do they not?) Deleting the root’s key from a heap can be done with the following algorithm, illustrated in Figure 6.13.
9
9
6 2
8 5
7
10
6 10
2
10 5
7
6 8
2
9 5
7
8
FIGURE 6.12 Inserting a key (10) into the heap constructed in Figure 6.11. The new key is sifted up via a swap with its parent until it is not larger than its parent (or is in the root).
6.4
231
Heaps and Heapsort
9 8 2
6 5
1
1
1
Step 1
8
Step 2 8 2
Step 3
6 5
9
8 2
6 5
5 2
6 1
FIGURE 6.13 Deleting the root’s key from a heap. The key to be deleted is swapped with the last key after which the smaller tree is “heapiﬁed” by exchanging the new key in its root with the larger key in its children until the parental dominance requirement is satisﬁed.
Maximum Key Deletion from a heap Step 1 Exchange the root’s key with the last key K of the heap. Step 2 Decrease the heap’s size by 1. Step 3 “Heapify” the smaller tree by sifting K down the tree exactly in the same way we did it in the bottomup heap construction algorithm. That is, verify the parental dominance for K: if it holds, we are done; if not, swap K with the larger of its children and repeat this operation until the parental dominance condition holds for K in its new position. The efﬁciency of deletion is determined by the number of key comparisons needed to “heapify” the tree after the swap has been made and the size of the tree is decreased by 1. Since this cannot require more key comparisons than twice the heap’s height, the time efﬁciency of deletion is in O(log n) as well.
Heapsort Now we can describe heapsort—an interesting sorting algorithm discovered by J. W. J. Williams [Wil64]. This is a twostage algorithm that works as follows. Stage 1 (heap construction): Construct a heap for a given array. Stage 2 (maximum deletions): Apply the rootdeletion operation n − 1 times to the remaining heap. As a result, the array elements are eliminated in decreasing order. But since under the array implementation of heaps an element being deleted is placed last, the resulting array will be exactly the original array sorted in increasing order. Heapsort is traced on a speciﬁc input in Figure 6.14. (The same input as the one
232
TransformandConquer Stage 1 (heap construction)
Stage 2 (maximum deletions)
2
9
7
6
5
8
9
6
8
2
5
2
9
8
6
5
7
7
6
8
2
5  9
2
9
8
6
5
7
8
6
7
2
5
9
2
8
6
5
7
5
6
7
2  8
9
6
8
2
5
7
7
6
5
2
2
6
5  7
6
2
5
5
2  6
5
2
7
2  5 2
FIGURE 6.14 Sorting the array 2, 9, 7, 6, 5, 8 by heapsort.
in Figure 6.11 is intentionally used so that you can compare the tree and array implementations of the bottomup heap construction algorithm.) Since we already know that the heap construction stage of the algorithm is in O(n), we have to investigate just the time efﬁciency of the second stage. For the number of key comparisons, C(n), needed for eliminating the root keys from the heaps of diminishing sizes from n to 2, we get the following inequality: C(n) ≤ 2log2 (n − 1) + 2log2 (n − 2) + . . . + 2log2 1 ≤ 2
n−1
log2 i
i=1
≤2
n−1
log2 (n − 1) = 2(n − 1) log2 (n − 1) ≤ 2n log2 n.
i=1
This means that C(n) ∈ O(n log n) for the second stage of heapsort. For both stages, we get O(n) + O(n log n) = O(n log n). A more detailed analysis shows that the time efﬁciency of heapsort is, in fact, in (n log n) in both the worst and average cases. Thus, heapsort’s time efﬁciency falls in the same class as that of mergesort. Unlike the latter, heapsort is inplace, i.e., it does not require any extra storage. Timing experiments on random ﬁles show that heapsort runs more slowly than quicksort but can be competitive with mergesort.
6.4
Heaps and Heapsort
233
Exercises 6.4 1. a. Construct a heap for the list 1, 8, 6, 5, 3, 7, 4 by the bottomup algorithm. b. Construct a heap for the list 1, 8, 6, 5, 3, 7, 4 by successive key insertions (topdown algorithm). c. Is it always true that the bottomup and topdown algorithms yield the same heap for the same input? 2. Outline an algorithm for checking whether an array H [1..n] is a heap and determine its time efﬁciency. 3. a. Find the smallest and the largest number of keys that a heap of height h can contain. b. Prove that the height of a heap with n nodes is equal to log2 n. 4. Prove the following equality used in Section 6.4: h−1
2(h − i)2i = 2(n − log2 (n + 1)),
where n = 2h+1 − 1.
i=0
5. a. Design an efﬁcient algorithm for ﬁnding and deleting an element of the smallest value in a heap and determine its time efﬁciency. b. Design an efﬁcient algorithm for ﬁnding and deleting an element of a given value v in a heap H and determine its time efﬁciency. 6. Indicate the time efﬁciency classes of the three main operations of the priority queue implemented as a. an unsorted array. b. a sorted array. c. a binary search tree. d. an AVL tree. e. a heap. 7. Sort the following lists by heapsort by using the array representation of heaps. a. 1, 2, 3, 4, 5 (in increasing order) b. 5, 4, 3, 2, 1 (in increasing order) c. S, O, R, T, I, N, G (in alphabetical order) 8. Is heapsort a stable sorting algorithm? 9. What variety of the transformandconquer technique does heapsort represent? 10. Which sorting algorithm other than heapsort uses a priority queue? 11. Implement three advanced sorting algorithms—mergesort, quicksort, and heapsort—in the language of your choice and investigate their performance on arrays of sizes n = 103, 104, 105, and 106. For each of these sizes consider
234
TransformandConquer
a. randomly generated ﬁles of integers in the range [1..n]. b. increasing ﬁles of integers 1, 2, . . . , n. c. decreasing ﬁles of integers n, n − 1, . . . , 1. 12. Spaghetti sort Imagine a handful of uncooked spaghetti, individual rods whose lengths represent numbers that need to be sorted. a. Outline a “spaghetti sort”—a sorting algorithm that takes advantage of this unorthodox representation. b. What does this example of computer science folklore (see [Dew93]) have to do with the topic of this chapter in general and heapsort in particular?
6.5 Horner’s Rule and Binary Exponentiation In this section, we discuss the problem of computing the value of a polynomial p(x) = anx n + an−1x n−1 + . . . + a1x + a0
(6.1) x n.
Polynomials at a given point x and its important special case of computing constitute the most important class of functions because they possess a wealth of good properties on the one hand and can be used for approximating other types of functions on the other. The problem of manipulating polynomials efﬁciently has been important for several centuries; new discoveries were still being made the last 50 years. By far the most important of them was the fast Fourier transform (FFT). The practical importance of this remarkable algorithm, which is based on representing a polynomial by its values at specially chosen points, was such that some people consider it one of the most important algorithmic discoveries of all time. Because of its relative complexity, we do not discuss the FFT algorithm in this book. An interested reader will ﬁnd a wealth of literature on the subject, including reasonably accessible treatments in such textbooks as [Kle06] and [Cor09].
Horner’s Rule Horner’s rule is an old but very elegant and efﬁcient algorithm for evaluating a polynomial. It is named after the British mathematician W. G. Horner, who published it in the early 19th century. But according to Knuth [KnuII, p. 486], the method was used by Isaac Newton 150 years before Horner. You will appreciate this method much more if you ﬁrst design an algorithm for the polynomial evaluation problem by yourself and investigate its efﬁciency (see Problems 1 and 2 in this section’s exercises). Horner’s rule is a good example of the representationchange technique since it is based on representing p(x) by a formula different from (6.1). This new formula is obtained from (6.1) by successively taking x as a common factor in the remaining polynomials of diminishing degrees: p(x) = (. . . (anx + an−1)x + . . .)x + a0.
(6.2)
6.5
Horner’s Rule and Binary Exponentiation
235
For example, for the polynomial p(x) = 2x 4 − x 3 + 3x 2 + x − 5, we get p(x) = 2x 4 − x 3 + 3x 2 + x − 5 = x(2x 3 − x 2 + 3x + 1) − 5 = x(x(2x 2 − x + 3) + 1) − 5 = x(x(x(2x − 1) + 3) + 1) − 5.
(6.3)
It is in formula (6.2) that we will substitute a value of x at which the polynomial needs to be evaluated. It is hard to believe that this is a way to an efﬁcient algorithm, but the unpleasant appearance of formula (6.2) is just that, an appearance. As we shall see, there is no need to go explicitly through the transformation leading to it: all we need is an original list of the polynomial’s coefﬁcients. The penandpencil calculation can be conveniently organized with a tworow table. The ﬁrst row contains the polynomial’s coefﬁcients (including all the coefﬁcients equal to zero, if any) listed from the highest an to the lowest a0. Except for its ﬁrst entry, which is an, the second row is ﬁlled left to right as follows: the next entry is computed as the x’s value times the last entry in the second row plus the next coefﬁcient from the ﬁrst row. The ﬁnal entry computed in this fashion is the value being sought. EXAMPLE 1 coefﬁcients 2 x=3 2
Evaluate p(x) = 2x 4 − x 3 + 3x 2 + x − 5 at x = 3. −1 3 1 −5 3 . 2 + (−1) = 5 3 . 5 + 3 = 18 3 . 18 + 1 = 55 3 . 55 + (−5) = 160
Thus, p(3) = 160. (On comparing the table’s entries with formula (6.3), you will see that 3 . 2 + (−1) = 5 is the value of 2x − 1 at x = 3, 3 . 5 + 3 = 18 is the value of x(2x − 1) + 3 at x = 3, 3 . 18 + 1 = 55 is the value of x(x(2x − 1) + 3) + 1 at x = 3, and, ﬁnally, 3 . 55 + (−5) = 160 is the value of x(x(x(2x − 1) + 3) + 1) − 5 = p(x) at x = 3.) Pseudocode of this algorithm is the shortest one imaginable for a nontrivial algorithm: ALGORITHM Horner(P [0..n], x) //Evaluates a polynomial at a given point by Horner’s rule //Input: An array P [0..n] of coefﬁcients of a polynomial of degree n, // stored from the lowest to the highest and a number x //Output: The value of the polynomial at x p ← P [n] for i ← n − 1 downto 0 do p ← x ∗ p + P [i] return p
236
TransformandConquer
The number of multiplications and the number of additions are given by the same sum: n−1
1 = n. M(n) = A(n) = i=0
To appreciate how efﬁcient Horner’s rule is, consider only the ﬁrst term of a polynomial of degree n: anx n. Just computing this single term by the bruteforce algorithm would require n multiplications, whereas Horner’s rule computes, in addition to this term, n − 1 other terms, and it still uses the same number of multiplications! It is not surprising that Horner’s rule is an optimal algorithm for polynomial evaluation without preprocessing the polynomial’s coefﬁcients. But it took scientists 150 years after Horner’s publication to come to the realization that such a question was worth investigating. Horner’s rule also has some useful byproducts. The intermediate numbers generated by the algorithm in the process of evaluating p(x) at some point x0 turn out to be the coefﬁcients of the quotient of the division of p(x) by x − x0, and the ﬁnal result, in addition to being p(x0), is equal to the remainder of this division. Thus, according to Example 1, the quotient and the remainder of the division of 2x 4 − x 3 + 3x 2 + x − 5 by x − 3 are 2x 3 + 5x 2 + 18x + 55 and 160, respectively. This division algorithm, known as synthetic division, is more convenient than socalled long division.
Binary Exponentiation The amazing efﬁciency of Horner’s rule fades if the method is applied to computing a n, which is the value of x n at x = a. In fact, it degenerates to the bruteforce multiplication of a by itself, with wasteful additions of zeros in between. Since computing a n (actually, a n mod m) is an essential operation in several important primalitytesting and encryption methods, we consider now two algorithms for computing a n that are based on the representationchange idea. They both exploit the binary representation of exponent n, but one of them processes this binary string left to right, whereas the second does it right to left. Let n = b I . . . bi . . . b0 be the bit string representing a positive integer n in the binary number system. This means that the value of n can be computed as the value of the polynomial p(x) = bI x I + . . . + bi x i + . . . + b0
(6.4)
at x = 2. For example, if n = 13, its binary representation is 1101 and 13 = 1 . 23 + 1 . 22 + 0 . 21 + 1 . 20. Let us now compute the value of this polynomial by applying Horner’s rule and see what the method’s operations imply for computing the power a n = a p(2) = a bI 2
...+bi 2i +...+b0
I+
.
6.5
237
Horner’s Rule and Binary Exponentiation
Horner’s rule for the binary polynomial p(2)
Implications for a n = a p(2)
p ← 1 //the leading digit is always 1 for n ≥ 1 for i ← I − 1 downto 0 do p ← 2p + bi
ap ← a1 for i ← I − 1 downto 0 do a p ← a 2p+bi
But
a 2p+bi = a 2p . a bi = (a p )2 . a bi =
(a p )2 (a p )2 . a
if bi = 0, if bi = 1.
Thus, after initializing the accumulator’s value to a, we can scan the bit string representing the exponent n to always square the last value of the accumulator and, if the current binary digit is 1, also to multiply it by a. These observations lead to the following lefttoright binary exponentiation method of computing a n. ALGORITHM
LeftRightBinaryExponentiation(a, b(n))
//Computes a n by the lefttoright binary exponentiation algorithm //Input: A number a and a list b(n) of binary digits bI , . . . , b0 // in the binary expansion of a positive integer n //Output: The value of a n product ← a for i ← I − 1 downto 0 do product ← product ∗ product if bi = 1 product ← product ∗ a return product EXAMPLE 2 Compute a 13 by the lefttoright binary exponentiation algorithm. Here, n = 13 = 11012 . So we have binary digits of n product accumulator
1 a
1 a2 . a = a3
0 (a 3)2 = a 6
1 (a 6)2 . a = a 13
Since the algorithm makes one or two multiplications on each repetition of its only loop, the total number of multiplications M(n) made by it in computing a n is (b − 1) ≤ M(n) ≤ 2(b − 1), where b is the length of the bit string representing the exponent n. Taking into account that b − 1 = log2 n, we can conclude that the efﬁciency of the lefttoright binary exponentiation is logarithmic. Thus, this algorithm is in a better efﬁciency class than the bruteforce exponentiation, which always requires n − 1 multiplications.
238
TransformandConquer
The righttoleft binary exponentiation uses the same binary polynomial p(2) (see (6.4)) yielding the value of n. But rather than applying Horner’s rule to it as the previous method did, this one exploits it differently: a n = a bI 2
I+
...+bi 2i +...+b0
I i = a bI 2 . . . . . a bi 2 . . . . . a b0 .
Thus, a n can be computed as the product of the terms i 2 i if bi = 1, a bi 2 = a 1 if bi = 0, i
i.e., the product of consecutive terms a 2 , skipping those for which the binary digit i bi is zero. In addition, we can compute a 2 by simply squaring the same term we i i−1 computed for the previous value of i since a 2 = (a 2 )2 . So we compute all such powers of a from the smallest to the largest (from right to left), but we include in the product accumulator only those whose corresponding binary digit is 1. Here is pseudocode of this algorithm. ALGORITHM RightLeftBinaryExponentiation(a, b(n)) //Computes a n by the righttoleft binary exponentiation algorithm //Input: A number a and a list b(n) of binary digits bI , . . . , b0 // in the binary expansion of a nonnegative integer n //Output: The value of a n i term ← a //initializes a 2 if b0 = 1 product ← a else product ← 1 for i ← 1 to I do term ← term ∗ term if bi = 1 product ← product ∗ term return product EXAMPLE 3 Compute a 13 by the righttoleft binary exponentiation method. Here, n = 13 = 11012 . So we have the following table ﬁlled in from right to left:
1
1
0
1
binary digits of n
a8 5 a . a 8 = a 13
a4 a . a4 = a5
a2
a a
terms a 2 product accumulator
i
Obviously, the algorithm’s efﬁciency is also logarithmic for the same reason the lefttoright binary multiplication is. The usefulness of both binary exponentiation algorithms is reduced somewhat by their reliance on availability of the explicit
6.5
Horner’s Rule and Binary Exponentiation
239
binary expansion of exponent n. Problem 9 in this section’s exercises asks you to design an algorithm that does not have this shortcoming.
Exercises 6.5 1. Consider the following bruteforce algorithm for evaluating a polynomial. ALGORITHM BruteForcePolynomialEvaluation(P [0..n], x) //Computes the value of polynomial P at a given point x //by the “highest to lowest term” bruteforce algorithm //Input: An array P [0..n] of the coefﬁcients of a polynomial of degree n, // stored from the lowest to the highest and a number x //Output: The value of the polynomial at the point x p ← 0.0 for i ← n downto 0 do power ← 1 for j ← 1 to i do power ← power ∗ x p ← p + P [i] ∗ power return p Find the total number of multiplications and the total number of additions made by this algorithm. 2. Write pseudocode for the bruteforce polynomial evaluation that stems from substituting a given value of the variable into the polynomial’s formula and evaluating it from the lowest term to the highest one. Determine the number of multiplications and the number of additions made by this algorithm. 3. a. Estimate how much faster Horner’s rule is compared to the “lowesttohighest term” bruteforce algorithm of Problem 2 if (i) the time of one multiplication is signiﬁcantly larger than the time of one addition; (ii) the time of one multiplication is about the same as the time of one addition. b. Is Horner’s rule more time efﬁcient at the expense of being less space efﬁcient than the bruteforce algorithm? 4. a. Apply Horner’s rule to evaluate the polynomial p(x) = 3x 4 − x 3 + 2x + 5 at x = −2. b. Use the results of the above application of Horner’s rule to ﬁnd the quotient and remainder of the division of p(x) by x + 2. 5. Apply Horner’s rule to convert 110100101 from binary to decimal. 6. Compare the number of multiplications and additions/subtractions needed by the “long division” of a polynomial p(x) = anx n + an−1x n−1 + . . . + a0 by
240
TransformandConquer
x − c, where c is some constant, with the number of these operations in the “synthetic division.” 7. a. Apply the lefttoright binary exponentiation algorithm to compute a 17. b. Is it possible to extend the lefttoright binary exponentiation algorithm to work for every nonnegative integer exponent? 8. Apply the righttoleft binary exponentiation algorithm to compute a 17. 9. Design a nonrecursive algorithm for computing a n that mimics the righttoleft binary exponentiation but does not explicitly use the binary representation of n. 10. Is it a good idea to use a generalpurpose polynomialevaluation algorithm such as Horner’s rule to evaluate the polynomial p(x) = x n + x n−1 + . . . + x + 1? 11. According to the corollary of the Fundamental Theorem of Algebra, every polynomial p(x) = anx n + an−1x n−1 + . . . + a0 can be represented in the form p(x) = an(x − x1)(x − x2 ) . . . (x − xn) where x1, x2 , . . . , xn are the roots of the polynomial (generally, complex and not necessarily distinct). Discuss which of the two representations is more convenient for each of the following operations: a. polynomial evaluation at a given point b. addition of two polynomials c. multiplication of two polynomials 12. Polynomial interpolation Given a set of n data points (xi , yi ) where no two xi are the same, ﬁnd a polynomial p(x) of degree at most n − 1 such that p(xi ) = yi for every i = 1, 2, . . . , n.
6.6 Problem Reduction Here is my version of a wellknown joke about mathematicians. Professor X, a noted mathematician, noticed that when his wife wanted to boil water for their tea, she took their kettle from their cupboard, ﬁlled it with water, and put it on the stove. Once, when his wife was away (if you have to know, she was signing her bestseller in a local bookstore), the professor had to boil water by himself. He saw that the kettle was sitting on the kitchen counter. What did Professor X do? He put the kettle in the cupboard ﬁrst and then proceeded to follow his wife’s routine.
241
6.6 Problem Reduction reduction Problem 1 (to be solved)
alg. A Problem 2 (solvable by alg. A )
solution to Problem 2
FIGURE 6.15 Problem reduction strategy.
The way Professor X approached his task is an example of an important problemsolving strategy called problem reduction. If you need to solve a problem, reduce it to another problem that you know how to solve (Figure 6.15). The joke about the professor notwithstanding, the idea of problem reduction plays a central role in theoretical computer science, where it is used to classify problems according to their complexity. We will touch on this classiﬁcation in Chapter 11. But the strategy can be used for actual problem solving, too. The practical difﬁculty in applying it lies, of course, in ﬁnding a problem to which the problem at hand should be reduced. Moreover, if we want our efforts to be of practical value, we need our reductionbased algorithm to be more efﬁcient than solving the original problem directly. Note that we have already encountered this technique earlier in the book. In Section 6.5, for example, we mentioned the socalled synthetic division done by applying Horner’s rule for polynomial evaluation. In Section 5.5, we used the following fact from analytical geometry: if p1(x1, y1), p2 (x2 , y2 ), and p3(x3, y3) are three arbitrary points in the plane, then the determinant $ $ $ x1 y 1 1 $ $ $ $ x2 y2 1 $ = x1y2 + x3y1 + x2 y3 − x3y2 − x1y3 − x2 y1 $ $ $ x3 y3 1 $ −−−−→
is positive if and only if the point p3 is to the left of the directed line p1p2 through points p1 and p2 . In other words, we reduced a geometric question about the relative locations of three points to a question about the sign of a determinant. In fact, the entire idea of analytical geometry is based on reducing geometric problems to algebraic ones. And the vast majority of geometric algorithms take advantage of this historic insight by Rene´ Descartes (1596–1650). In this section, we give a few more examples of algorithms based on the strategy of problem reduction.
Computing the Least Common Multiple Recall that the least common multiple of two positive integers m and n, denoted lcm(m, n), is deﬁned as the smallest integer that is divisible by both m and n. For example, lcm(24, 60) = 120, and lcm(11, 5) = 55. The least common multiple is one of the most important notions in elementary arithmetic and algebra. Perhaps you remember the following middleschool method for computing it: Given the prime factorizations of m and n, compute the product of all the common prime
242
TransformandConquer
factors of m and n, all the prime factors of m that are not in n, and all the prime factors of n that are not in m. For example, 24 = 2 . 2 . 2 . 3, 60 = 2 . 2 . 3 . 5, lcm(24, 60) = (2 . 2 . 3) . 2 . 5 = 120. As a computational procedure, this algorithm has the same drawbacks as the middleschool algorithm for computing the greatest common divisor discussed in Section 1.1: it is inefﬁcient and requires a list of consecutive primes. A much more efﬁcient algorithm for computing the least common multiple can be devised by using problem reduction. After all, there is a very efﬁcient algorithm (Euclid’s algorithm) for ﬁnding the greatest common divisor, which is a product of all the common prime factors of m and n. Can we ﬁnd a formula relating lcm(m, n) and gcd(m, n)? It is not difﬁcult to see that the product of lcm(m, n) and gcd(m, n) includes every factor of m and n exactly once and hence is simply equal to the product of m and n. This observation leads to the formula m.n , lcm(m, n) = gcd(m, n) where gcd(m, n) can be computed very efﬁciently by Euclid’s algorithm.
Counting Paths in a Graph As our next example, we consider the problem of counting paths between two vertices in a graph. It is not difﬁcult to prove by mathematical induction that the number of different paths of length k > 0 from the ith vertex to the j th vertex of a graph (undirected or directed) equals the (i, j )th element of Ak where A is the adjacency matrix of the graph. Therefore, the problem of counting a graph’s paths can be solved with an algorithm for computing an appropriate power of its adjacency matrix. Note that the exponentiation algorithms we discussed before for computing powers of numbers are applicable to matrices as well. As a speciﬁc example, consider the graph of Figure 6.16. Its adjacency matrix A and its square A2 indicate the numbers of paths of length 1 and 2, respectively, between the corresponding vertices of the graph. In particular, there are three
a
b
c
d
a b A= c d
a 0 1 1 1
b 1 0 0 0
c 1 0 0 1
d 1 0 1 0
a b A2 = c d
a 3 0 1 1
b 0 1 1 1
c 1 1 2 1
d 1 1 1 2
FIGURE 6.16 A graph, its adjacency matrix A, and its square A2 . The elements of A and A2 indicate the numbers of paths of lengths 1 and 2, respectively.
243
6.6 Problem Reduction
paths of length 2 that start and end at vertex a (a − b − a, a − c − a, and a − d − a); but there is only one path of length 2 from a to c (a − d − c).
Reduction of Optimization Problems Our next example deals with solving optimization problems. If a problem asks to ﬁnd a maximum of some function, it is said to be a maximization problem; if it asks to ﬁnd a function’s minimum, it is called a minimization problem. Suppose now that you need to ﬁnd a minimum of some function f (x) and you have an algorithm for function maximization. How can you take advantage of the latter? The answer lies in the simple formula min f (x) = − max[−f (x)]. In other words, to minimize a function, we can maximize its negative instead and, to get a correct minimal value of the function itself, change the sign of the answer. This property is illustrated for a function of one real variable in Figure 6.17. Of course, the formula max f (x) = − min[−f (x)] is valid as well; it shows how a maximization problem can be reduced to an equivalent minimization problem. This relationship between minimization and maximization problems is very general: it holds for functions deﬁned on any domain D. In particular, we can
y
f (x )
f (x*) x*
x
–f (x*)
–f (x )
FIGURE 6.17 Relationship between minimization and maximization problems: min f (x) = − max[−f (x)].
244
TransformandConquer
apply it to functions of several variables subject to additional constraints. A very important class of such problems is introduced below in this section. Now that we are on the topic of function optimization, it is worth pointing out that the standard calculus procedure for ﬁnding extremum points of a function is, in fact, also based on problem reduction. Indeed, it suggests ﬁnding the function’s derivative f (x) and then solving the equation f (x) = 0 to ﬁnd the function’s critical points. In other words, the optimization problem is reduced to the problem of solving an equation as the principal part of ﬁnding extremum points. Note that we are not calling the calculus procedure an algorithm, since it is not clearly deﬁned. In fact, there is no general method for solving equations. A little secret of calculus textbooks is that problems are carefully selected so that critical points can always be found without difﬁculty. This makes the lives of both students and instructors easier but, in the process, may unintentionally create a wrong impression in students’ minds.
Linear Programming Many problems of optimal decision making can be reduced to an instance of the linear programming problem—a problem of optimizing a linear function of several variables subject to constraints in the form of linear equations and linear inequalities. EXAMPLE 1 Consider a university endowment that needs to invest $100 million. This sum has to be split between three types of investments: stocks, bonds, and cash. The endowment managers expect an annual return of 10%, 7%, and 3% for their stock, bond, and cash investments, respectively. Since stocks are more risky than bonds, the endowment rules require the amount invested in stocks to be no more than onethird of the moneys invested in bonds. In addition, at least 25% of the total amount invested in stocks and bonds must be invested in cash. How should the managers invest the money to maximize the return? Let us create a mathematical model of this problem. Let x, y, and z be the amounts (in millions of dollars) invested in stocks, bonds, and cash, respectively. By using these variables, we can pose the following optimization problem: maximize
0.10x + 0.07y + 0.03z
subject to
x + y + z = 100 x ≤ 31 y z ≥ 0.25(x + y) x ≥ 0,
y ≥ 0,
z ≥ 0.
Although this example is both small and simple, it does show how a problem of optimal decision making can be reduced to an instance of the general linear programming problem
6.6 Problem Reduction
c1x1 + . . . + cnxn subject to ai1x1 + . . . + ainxn ≤ (or ≥ or =) bi
245
maximize (or minimize)
for i = 1, . . . , m
x1 ≥ 0, . . . , xn ≥ 0. (The last group of constraints—called the nonnegativity constraints—are, strictly speaking, unnecessary because they are special cases of more general constraints ai1x1 + . . . + ainxn ≥ bi , but it is convenient to treat them separately.) Linear programming has proved to be ﬂexible enough to model a wide variety of important applications, such as airline crew scheduling, transportation and communication network planning, oil exploration and reﬁning, and industrial production optimization. In fact, linear programming is considered by many as one of the most important achievements in the history of applied mathematics. The classic algorithm for this problem is called the simplex method (Section 10.1). It was discovered by the U.S. mathematician George Dantzig in the 1940s [Dan63]. Although the worstcase efﬁciency of this algorithm is known to be exponential, it performs very well on typical inputs. Moreover, a more recent algorithm by Narendra Karmarkar [Kar84] not only has a proven polynomial worstcase efﬁciency but has also performed competitively with the simplex method in empirical tests. It is important to stress, however, that the simplex method and Karmarkar’s algorithm can successfully handle only linear programming problems that do not limit its variables to integer values. When variables of a linear programming problem are required to be integers, the linear programming problem is said to be an integer linear programming problem. Except for some special cases (e.g., the assignment problem and the problems discussed in Sections 10.2–10.4), integer linear programming problems are much more difﬁcult. There is no known polynomialtime algorithm for solving an arbitrary instance of the general integer linear programming problem and, as we see in Chapter 11, such an algorithm quite possibly does not exist. Other approaches such as the branchandbound technique discussed in Section 12.2 are typically used for solving integer linear programming problems. EXAMPLE 2 Let us see how the knapsack problem can be reduced to a linear programming problem. Recall from Section 3.4 that the knapsack problem can be posed as follows. Given a knapsack of capacity W and n items of weights w1, . . . , wn and values v1, . . . , vn, ﬁnd the most valuable subset of the items that ﬁts into the knapsack. We consider ﬁrst the continuous (or fractional) version of the problem, in which any fraction of any item given can be taken into the knapsack. Let xj , j = 1, . . . , n, be a variable representing a fraction of item j taken into the knapsack. Obviously, xj must satisfy the inequality 0 ≤ xj ≤ 1. Then the total weight of the selected items can be expressed by the sum nj=1 wj xj , and their total value by the sum nj=1 vj xj . Thus, the continuous version of the knapsack problem can be posed as the following linear programming problem:
246
TransformandConquer
maximize
subject to
n
j =1 n
vj x j wj xj ≤ W
j =1
0 ≤ xj ≤ 1 for j = 1, . . . , n. There is no need to apply a general method for solving linear programming problems here: this particular problem can be solved by a simple special algorithm that is introduced in Section 12.3. (But why wait? Try to discover it on your own now.) This reduction of the knapsack problem to an instance of the linear programming problem is still useful, though, to prove the correctness of the algorithm in question. In the discrete (or 01) version of the knapsack problem, we are only allowed either to take a whole item or not to take it at all. Hence, we have the following integer linear programming problem for this version: maximize
n
vj xj
j =1
subject to
n
wj xj ≤ W
j =1
xj ∈ {0, 1}
for j = 1, . . . , n.
This seemingly minor modiﬁcation makes a drastic difference for the complexity of this and similar problems constrained to take only discrete values in their potential ranges. Despite the fact that the 01 version might seem to be easier because it can ignore any subset of the continuous version that has a fractional value of an item, the 01 version is, in fact, much more complicated than its continuous counterpart. The reader interested in speciﬁc algorithms for solving this problem will ﬁnd a wealth of literature on the subject, including the monographs [Mar90] and [Kel04].
Reduction to Graph Problems As we pointed out in Section 1.3, many problems can be solved by a reduction to one of the standard graph problems. This is true, in particular, for a variety of puzzles and games. In these applications, vertices of a graph typically represent possible states of the problem in question, and edges indicate permitted transitions among such states. One of the graph’s vertices represents an initial state and another represents a goal state of the problem. (There might be several vertices of the latter kind.) Such a graph is called a statespace graph. Thus, the transformation just described reduces the problem to the question about a path from the initialstate vertex to a goalstate vertex.
6.6 Problem Reduction
Pwgc  
Pg
247
wc   Pg
P Pwc   g
Pgc   w
Pwg   c
Pw Pc
Pg
Pc
Pg
c   Pwg
w   Pgc
Pw
g   Pwc
P Pg   wc
Pg
  Pwgc
FIGURE 6.18 Statespace graph for the peasant, wolf, goat, and cabbage puzzle.
EXAMPLE Let us revisit the classic rivercrossing puzzle that was included in the exercises for Section 1.2. A peasant ﬁnds himself on a river bank with a wolf, a goat, and a head of cabbage. He needs to transport all three to the other side of the river in his boat. However, the boat has room only for the peasant himself and one other item (either the wolf, the goat, or the cabbage). In his absence, the wolf would eat the goat, and the goat would eat the cabbage. Find a way for the peasant to solve his problem or prove that it has no solution. The statespace graph for this problem is given in Figure 6.18. Its vertices are labeled to indicate the states they represent: P, w, g, c stand for the peasant, the wolf, the goat, and the cabbage, respectively; the two bars   denote the river; for convenience, we also label the edges by indicating the boat’s occupants for each crossing. In terms of this graph, we are interested in ﬁnding a path from the initialstate vertex labeled Pwgc  to the ﬁnalstate vertex labeled  Pwgc. It is easy to see that there exist two distinct simple paths from the initialstate vertex to the ﬁnal state vertex (what are they?). If we ﬁnd them by applying breadthﬁrst search, we get a formal proof that these paths have the smallest number of edges possible. Hence, this puzzle has two solutions requiring seven river crossings, which is the minimum number of crossings needed. Our success in solving this simple puzzle should not lead you to believe that generating and investigating statespace graphs is always a straightforward task. To get a better appreciation of them, consult books on artiﬁcial intelligence (AI), the branch of computer science in which statespace graphs are a principal subject.
248
TransformandConquer
In this book, we deal with an important special case of statespace graphs in Sections 12.1 and 12.2.
Exercises 6.6 1. a. Prove the equality lcm(m, n) =
m.n gcd(m, n)
that underlies the algorithm for computing lcm(m, n). b. Euclid’s algorithm is known to be in O(log n). If it is the algorithm that is used for computing gcd(m, n), what is the efﬁciency of the algorithm for computing lcm(m, n)? 2. You are given a list of numbers for which you need to construct a minheap. (A minheap is a complete binary tree in which every key is less than or equal to the keys in its children.) How would you use an algorithm for constructing a maxheap (a heap as deﬁned in Section 6.4) to construct a minheap? 3. Prove that the number of different paths of length k > 0 from the ith vertex to the j th vertex in a graph (undirected or directed) equals the (i, j )th element of Ak where A is the adjacency matrix of the graph. 4. a. Design an algorithm with a time efﬁciency better than cubic for checking whether a graph with n vertices contains a cycle of length 3 [Man89]. b. Consider the following algorithm for the same problem. Starting at an arbitrary vertex, traverse the graph by depthﬁrst search and check whether its depthﬁrst search forest has a vertex with a back edge leading to its grandparent. If it does, the graph contains a triangle; if it does not, the graph does not contain a triangle as its subgraph. Is this algorithm correct? 5. Given n > 3 points P1 = (x1, y1), . . . , Pn = (xn, yn) in the coordinate plane, design an algorithm to check whether all the points lie within a triangle with its vertices at three of the points given. (You can either design an algorithm from scratch or reduce the problem to another one with a known algorithm.) 6. Consider the problem of ﬁnding, for a given positive integer n, the pair of integers whose sum is n and whose product is as large as possible. Design an efﬁcient algorithm for this problem and indicate its efﬁciency class. 7. The assignment problem introduced in Section 3.4 can be stated as follows: There are n people who need to be assigned to execute n jobs, one person per job. (That is, each person is assigned to exactly one job and each job is assigned to exactly one person.) The cost that would accrue if the ith person is assigned to the j th job is a known quantity C[i, j ] for each pair i, j = 1, . . . , n. The problem is to assign the people to the jobs to minimize the total cost of
6.6 Problem Reduction
249
the assignment. Express the assignment problem as a 01 linear programming problem. 8. Solve the instance of the linear programming problem given in Section 6.6: maximize
0.10x + 0.07y + 0.03z
subject to
x + y + z = 100 x ≤ 31 y z ≥ 0.25(x + y) x ≥ 0,
y ≥ 0,
z ≥ 0.
9. The graphcoloring problem is usually stated as the vertexcoloring problem: Assign the smallest number of colors to vertices of a given graph so that no two adjacent vertices are the same color. Consider the edgecoloring problem: Assign the smallest number of colors possible to edges of a given graph so that no two edges with the same endpoint are the same color. Explain how the edgecoloring problem can be reduced to a vertexcoloring problem. 10. Consider the twodimensional post ofﬁce location problem: given n points (x1, y1), . . . , (xn, yn) in the Cartesian plane, ﬁnd a location (x, y) for a post ofﬁce that minimizes n1 ni=1(xi − x + yi − y), the average Manhattan distance from the post ofﬁce to these points. Explain how this problem can be efﬁciently solved by the problem reduction technique, provided the post ofﬁce does not have to be located at one of the input points. 11. Jealous husbands There are n ≥ 2 married couples who need to cross a river. They have a boat that can hold no more than two people at a time. To complicate matters, all the husbands are jealous and will not agree on any crossing procedure that would put a wife on the same bank of the river with another woman’s husband without the wife’s husband being there too, even if there are other people on the same bank. Can they cross the river under such constraints? a. Solve the problem for n = 2. b. Solve the problem for n = 3, which is the classical version of this problem. c. Does the problem have a solution for n ≥ 4? If it does, indicate how many river crossings it will take; if it does not, explain why. 12. Doublen dominoes Dominoes are small rectangular tiles with dots called spots or pips embossed at both halves of the tiles. A standard “doublesix” domino set has 28 tiles: one for each unordered pair of integers from (0, 0) to (6, 6). In general, a “doublen” domino set would consist of domino tiles for each unordered pair of integers from (0, 0) to (n, n). Determine all values of n for which one constructs a ring made up of all the tiles in a doublen domino set.
250
TransformandConquer
SUMMARY Transformandconquer is the fourth general algorithm design (and problemsolving) strategy discussed in the book. It is, in fact, a group of techniques based on the idea of transformation to a problem that is easier to solve. There are three principal varieties of the transformandconquer strategy: instance simpliﬁcation, representation change, and problem reduction. Instance simpliﬁcation is transforming an instance of a problem to an instance of the same problem with some special property that makes the problem easier to solve. List presorting, Gaussian elimination, and rotations in AVL trees are good examples of this strategy. Representation change implies changing one representation of a problem’s instance to another representation of the same instance. Examples discussed in this chapter include representation of a set by a 23 tree, heaps and heapsort, Horner’s rule for polynomial evaluation, and two binary exponentiation algorithms. Problem reduction calls for transforming a given problem to another problem that can be solved by a known algorithm. Among examples of applying this idea to algorithmic problem solving (see Section 6.6), reductions to linear programming and reductions to graph problems are especially important. Some examples used to illustrate transformandconquer happen to be very important data structures and algorithms. They are: heaps and heapsort, AVL and 23 trees, Gaussian elimination, and Horner’s rule. A heap is an essentially complete binary tree with keys (one per node) satisfying the parental dominance requirement. Though deﬁned as binary trees, heaps are normally implemented as arrays. Heaps are most important for the efﬁcient implementation of priority queues; they also underlie heapsort. Heapsort is a theoretically important sorting algorithm based on arranging elements of an array in a heap and then successively removing the largest element from a remaining heap. The algorithm’s running time is in (n log n) both in the worst case and in the average case; in addition, it is inplace. AVL trees are binary search trees that are always balanced to the extent possible for a binary tree. The balance is maintained by transformations of four types called rotations. All basic operations on AVL trees are in O(log n); it eliminates the bad worstcase efﬁciency of classic binary search trees. 23 trees achieve a perfect balance in a search tree by allowing a node to contain up to two ordered keys and have up to three children. This idea can be generalized to yield very important Btrees, discussed later in the book.
Summary
251
Gaussian elimination—an algorithm for solving systems of linear equations— is a principal algorithm in linear algebra. It solves a system by transforming it to an equivalent system with an uppertriangular coefﬁcient matrix, which is easy to solve by back substitutions. Gaussian elimination requires about 31 n3 multiplications. Horner’s rule is an optimal algorithm for polynomial evaluation without coefﬁcient preprocessing. It requires only n multiplications and n additions to evaluate an ndegree polynomial at a given point. Horner’s rule also has a few useful byproducts, such as the synthetic division algorithm. Two binary exponentiation algorithms for computing a n are introduced in Section 6.5. Both of them exploit the binary representation of the exponent n, but they process it in the opposite directions: left to right and right to left. Linear programming concerns optimizing a linear function of several variables subject to constraints in the form of linear equations and linear inequalities. There are efﬁcient algorithms capable of solving very large instances of this problem with many thousands of variables and constraints, provided the variables are not required to be integers. The latter, called integer linear programming, constitute a much more difﬁcult class of problems.
This page intentionally left blank
7 Space and Time TradeOffs Things which matter most must never be at the mercy of things which matter less. ¨ —Johann Wolfgang von Goethe (1749–1832)
S
pace and time tradeoffs in algorithm design are a wellknown issue for both theoreticians and practitioners of computing. Consider, as an example, the problem of computing values of a function at many points in its domain. If it is time that is at a premium, we can precompute the function’s values and store them in a table. This is exactly what human computers had to do before the advent of electronic computers, in the process burdening libraries with thick volumes of mathematical tables. Though such tables have lost much of their appeal with the widespread use of electronic computers, the underlying idea has proven to be quite useful in the development of several important algorithms for other problems. In somewhat more general terms, the idea is to preprocess the problem’s input, in whole or in part, and store the additional information obtained to accelerate solving the problem afterward. We call this approach input enhancement 1 and discuss the following algorithms based on it: counting methods for sorting (Section 7.1) BoyerMoore algorithm for string matching and its simpliﬁed version suggested by Horspool (Section 7.2) The other type of technique that exploits spacefortime tradeoffs simply uses extra space to facilitate faster and/or more ﬂexible access to the data. We call this approach prestructuring. This name highlights two facets of this variation of the spacefortime tradeoff: some processing is done before a problem in question
1.
The standard terms used synonymously for this technique are preprocessing and preconditioning. Confusingly, these terms can also be applied to methods that use the idea of preprocessing but do not use extra space (see Chapter 6). Thus, in order to avoid confusion, we use “input enhancement” as a special name for the spacefortime tradeoff technique being discussed here.
253
254
Space and Time TradeOffs
is actually solved but, unlike the inputenhancement variety, it deals with access structuring. We illustrate this approach by: hashing (Section 7.3) indexing with Btrees (Section 7.4) There is one more algorithm design technique related to the spacefortime tradeoff idea: dynamic programming. This strategy is based on recording solutions to overlapping subproblems of a given problem in a table from which a solution to the problem in question is then obtained. We discuss this welldeveloped technique separately, in the next chapter of the book. Two ﬁnal comments about the interplay between time and space in algorithm design need to be made. First, the two resources—time and space—do not have to compete with each other in all design situations. In fact, they can align to bring an algorithmic solution that minimizes both the running time and the space consumed. Such a situation arises, in particular, when an algorithm uses a spaceefﬁcient data structure to represent a problem’s input, which leads, in turn, to a faster algorithm. Consider, as an example, the problem of traversing graphs. Recall that the time efﬁciency of the two principal traversal algorithms—depthﬁrst search and breadthﬁrst search—depends on the data structure used for representing graphs: it is (n2 ) for the adjacency matrix representation and (n + m) for the adjacency list representation, where n and m are the numbers of vertices and edges, respectively. If input graphs are sparse, i.e., have few edges relative to the number of vertices (say, m ∈ O(n)), the adjacency list representation may well be more efﬁcient from both the space and the runningtime points of view. The same situation arises in the manipulation of sparse matrices and sparse polynomials: if the percentage of zeros in such objects is sufﬁciently high, we can save both space and time by ignoring zeros in the objects’ representation and processing. Second, one cannot discuss spacetime tradeoffs without mentioning the hugely important area of data compression. Note, however, that in data compression, size reduction is the goal rather than a technique for solving another problem. We discuss just one data compression algorithm, in the next chapter. The reader interested in this topic will ﬁnd a wealth of algorithms in such books as [Say05].
7.1 Sorting by Counting As a ﬁrst example of applying the inputenhancement technique, we discuss its application to the sorting problem. One rather obvious idea is to count, for each element of a list to be sorted, the total number of elements smaller than this element and record the results in a table. These numbers will indicate the positions of the elements in the sorted list: e.g., if the count is 10 for some element, it should be in the 11th position (with index 10, if we start counting with 0) in the sorted array. Thus, we will be able to sort the list by simply copying its elements to their appropriate positions in a new, sorted list. This algorithm is called comparisoncounting sort (Figure 7.1).
255
7.1 Sorting by Counting Array A [0..5] Initially After pass i After pass i After pass i After pass i After pass i Final state
=0 =1 =2 =3 =4
Array S [0..5]
Count [] Count [] Count [] Count [] Count [] Count [] Count []
62
31
84
96
19
47
0 3
0 0 1
0 1 2 4
0 1 2 3 5
3
1
4
5
0 0 0 0 0 0 0
0 0 1 1 1 2 2
19
31
47
62
84
96
FIGURE 7.1 Example of sorting by comparison counting.
ALGORITHM ComparisonCountingSort(A[0..n − 1]) //Sorts an array by comparison counting //Input: An array A[0..n − 1] of orderable elements //Output: Array S[0..n − 1] of A’s elements sorted in nondecreasing order for i ← 0 to n − 1 do Count[i] ← 0 for i ← 0 to n − 2 do for j ← i + 1 to n − 1 do if A[i] < A[j ] Count[j ] ← Count[j ] + 1 else Count[i] ← Count[i] + 1 for i ← 0 to n − 1 do S[Count[i]] ← A[i] return S What is the time efﬁciency of this algorithm? It should be quadratic because the algorithm considers all the different pairs of an nelement array. More formally, the number of times its basic operation, the comparison A[i] < A[j ], is executed is equal to the sum we have encountered several times already: C(n) =
n−2 n−1
i=0 j =i+1
1=
n−2 n−2
n(n − 1) . [(n − 1) − (i + 1) + 1] = (n − 1 − i) = 2 i=0 i=0
Thus, the algorithm makes the same number of key comparisons as selection sort and in addition uses a linear amount of extra space. On the positive side, the algorithm makes the minimum number of key moves possible, placing each of them directly in their ﬁnal position in a sorted array. The counting idea does work productively in a situation in which elements to be sorted belong to a known small set of values. Assume, for example, that we have to sort a list whose values can be either 1 or 2. Rather than applying a general sorting algorithm, we should be able to take advantage of this additional
256
Space and Time TradeOffs
information about values to be sorted. Indeed, we can scan the list to compute the number of 1’s and the number of 2’s in it and then, on the second pass, simply make the appropriate number of the ﬁrst elements equal to 1 and the remaining elements equal to 2. More generally, if element values are integers between some lower bound l and upper bound u, we can compute the frequency of each of those values and store them in array F [0..u − l]. Then the ﬁrst F [0] positions in the sorted list must be ﬁlled with l, the next F [1] positions with l + 1, and so on. All this can be done, of course, only if we can overwrite the given elements. Let us consider a more realistic situation of sorting a list of items with some other information associated with their keys so that we cannot overwrite the list’s elements. Then we can copy elements into a new array S[0..n − 1]to hold the sorted list as follows. The elements of A whose values are equal to the lowest possible value l are copied into the ﬁrst F [0] elements of S, i.e., positions 0 through F [0] − 1; the elements of value l + 1 are copied to positions from F [0] to (F [0] + F [1]) − 1; and so on. Since such accumulated sums of frequencies are called a distribution in statistics, the method itself is known as distribution counting. EXAMPLE
Consider sorting the array 13
11
12
13
12
12
whose values are known to come from the set {11, 12, 13} and should not be overwritten in the process of sorting. The frequency and distribution arrays are as follows:
Array values Frequencies Distribution values
11
12
13
1 1
3 4
2 6
Note that the distribution values indicate the proper positions for the last occurrences of their elements in the ﬁnal sorted array. If we index array positions from 0 to n − 1, the distribution values must be reduced by 1 to get corresponding element positions. It is more convenient to process the input array right to left. For the example, the last element is 12, and, since its distribution value is 4, we place this 12 in position 4 − 1 = 3 of the array S that will hold the sorted list. Then we decrease the 12’s distribution value by 1 and proceed to the next (from the right) element in the given array. The entire processing of this example is depicted in Figure 7.2.
257
7.1 Sorting by Counting
A [5] = 12 A [4] = 12 A [3] = 13 A [2] = 12 A [1] = 11 A [0] = 13
1 1 1 1 1 0
D [0..2] 4 3 2 2 1 1
6 6 6 5 5 5
S [0..5] 12 12 13 12 11 13
FIGURE 7.2 Example of sorting by distribution counting. The distribution values being decremented are shown in bold.
Here is pseudocode of this algorithm. ALGORITHM DistributionCountingSort(A[0..n − 1], l, u) //Sorts an array of integers from a limited range by distribution counting //Input: An array A[0..n − 1] of integers between l and u (l ≤ u) //Output: Array S[0..n − 1] of A’s elements sorted in nondecreasing order for j ← 0 to u − l do D[j ] ← 0 //initialize frequencies for i ← 0 to n − 1 do D[A[i] − l] ← D[A[i] − l] + 1 //compute frequencies for j ← 1 to u − l do D[j ] ← D[j − 1] + D[j ] //reuse for distribution for i ← n − 1 downto 0 do j ← A[i] − l S[D[j ] − 1] ← A[i] D[j ] ← D[j ] − 1 return S Assuming that the range of array values is ﬁxed, this is obviously a linear algorithm because it makes just two consecutive passes through its input array A. This is a better timeefﬁciency class than that of the most efﬁcient sorting algorithms—mergesort, quicksort, and heapsort—we have encountered. It is important to remember, however, that this efﬁciency is obtained by exploiting the speciﬁc nature of inputs for which sorting by distribution counting works, in addition to trading space for time.
Exercises 7.1 1. Is it possible to exchange numeric values of two variables, say, u and v, without using any extra storage? 2. Will the comparisoncounting algorithm work correctly for arrays with equal values? 3. Assuming that the set of possible list values is {a, b, c, d}, sort the following list in alphabetical order by the distributioncounting algorithm: b, c, d, c, b, a, a, b.
258
Space and Time TradeOffs
4. Is the distributioncounting algorithm stable? 5. Design a oneline algorithm for sorting any array of size n whose values are n distinct integers from 1 to n. 6. The ancestry problem asks to determine whether a vertex u is an ancestor of vertex v in a given binary (or, more generally, rooted ordered) tree of n vertices. Design a O(n) inputenhancement algorithm that provides sufﬁcient information to solve this problem for any pair of the tree’s vertices in constant time. 7. The following technique, known as virtual initialization, provides a timeefﬁcient way to initialize just some elements of a given array A[0..n − 1] so that for each of its elements, we can say in constant time whether it has been initialized and, if it has been, with which value. This is done by utilizing a variable counter for the number of initialized elements in A and two auxiliary arrays of the same size, say B[0..n − 1] and C[0..n − 1], deﬁned as follows. B[0], . . . , B[counter − 1] contain the indices of the elements of A that were initialized: B[0] contains the index of the element initialized ﬁrst, B[1] contains the index of the element initialized second, etc. Furthermore, if A[i] was the kth element (0 ≤ k ≤ counter − 1) to be initialized, C[i] contains k. a. Sketch the state of arrays A[0..7], B[0..7], and C[0..7] after the three assignments A[3] ← x;
A[7] ← z;
A[1] ← y.
b. In general, how can we check with this scheme whether A[i] has been initialized and, if it has been, with which value? 8. Least distance sorting There are 10 Egyptian stone statues standing in a row in an art gallery hall. A new curator wants to move them so that the statues are ordered by their height. How should this be done to minimize the total distance that the statues are moved? You may assume for simplicity that all the statues have different heights. [Azi10] 9. a. Write a program for multiplying two sparse matrices, a p × q matrix A and a q × r matrix B. b. Write a program for multiplying two sparse polynomials p(x) and q(x) of degrees m and n, respectively. 10. Is it a good idea to write a program that plays the classic game of tictactoe with the human user by storing all possible positions on the game’s 3 × 3 board along with the best move for each of them?
7.2 Input Enhancement in String Matching In this section, we see how the technique of input enhancement can be applied to the problem of string matching. Recall that the problem of string matching
7.2
Input Enhancement in String Matching
259
requires ﬁnding an occurrence of a given string of m characters called the pattern in a longer string of n characters called the text. We discussed the bruteforce algorithm for this problem in Section 3.2: it simply matches corresponding pairs of characters in the pattern and the text left to right and, if a mismatch occurs, shifts the pattern one position to the right for the next trial. Since the maximum number of such trials is n − m + 1 and, in the worst case, m comparisons need to be made on each of them, the worstcase efﬁciency of the bruteforce algorithm is in the O(nm) class. On average, however, we should expect just a few comparisons before a pattern’s shift, and for random naturallanguage texts, the averagecase efﬁciency indeed turns out to be in O(n + m). Several faster algorithms have been discovered. Most of them exploit the inputenhancement idea: preprocess the pattern to get some information about it, store this information in a table, and then use this information during an actual search for the pattern in a given text. This is exactly the idea behind the two bestknown algorithms of this type: the KnuthMorrisPratt algorithm [Knu77] and the BoyerMoore algorithm [Boy77]. The principal difference between these two algorithms lies in the way they compare characters of a pattern with their counterparts in a text: the KnuthMorrisPratt algorithm does it left to right, whereas the BoyerMoore algorithm does it right to left. Since the latter idea leads to simpler algorithms, it is the only one that we will pursue here. (Note that the BoyerMoore algorithm starts by aligning the pattern against the beginning characters of the text; if the ﬁrst trial fails, it shifts the pattern to the right. It is comparisons within a trial that the algorithm does right to left, starting with the last character in the pattern.) Although the underlying idea of the BoyerMoore algorithm is simple, its actual implementation in a working method is less so. Therefore, we start our discussion with a simpliﬁed version of the BoyerMoore algorithm suggested by R. Horspool [Hor80]. In addition to being simpler, Horspool’s algorithm is not necessarily less efﬁcient than the BoyerMoore algorithm on random strings.
Horspool’s Algorithm Consider, as an example, searching for the pattern BARBER in some text: s0
...
c
...
sn−1
B A R B E R
Starting with the last R of the pattern and moving right to left, we compare the corresponding pairs of characters in the pattern and the text. If all the pattern’s characters match successfully, a matching substring is found. Then the search can be either stopped altogether or continued if another occurrence of the same pattern is desired. If a mismatch occurs, we need to shift the pattern to the right. Clearly, we would like to make as large a shift as possible without risking the possibility of missing a matching substring in the text. Horspool’s algorithm determines the size
260
Space and Time TradeOffs
of such a shift by looking at the character c of the text that is aligned against the last character of the pattern. This is the case even if character c itself matches its counterpart in the pattern. In general, the following four possibilities can occur. Case 1 If there are no c’s in the pattern—e.g., c is letter S in our example— we can safely shift the pattern by its entire length (if we shift less, some character of the pattern would be aligned against the text’s character c that is known not to be in the pattern): ...
s0
S
...
sn−1
B A R B E R B A R B E R
Case 2 If there are occurrences of character c in the pattern but it is not the last one there—e.g., c is letter B in our example—the shift should align the rightmost occurrence of c in the pattern with the c in the text: s0
B
...
...
sn−1
B A R B E R B A R B E R
Case 3 If c happens to be the last character in the pattern but there are no c’s among its other m − 1 characters—e.g., c is letter R in our example—the situation is similar to that of Case 1 and the pattern should be shifted by the entire pattern’s length m: s0
...
M E R
...
sn−1
L E A D E R L E A D E R
Case 4 Finally, if c happens to be the last character in the pattern and there are other c’s among its ﬁrst m − 1 characters—e.g., c is letter R in our example— the situation is similar to that of Case 2 and the rightmost occurrence of c among the ﬁrst m − 1 characters in the pattern should be aligned with the text’s c: s0
...
A R
...
sn−1
R E O R D E R R E O R D E R
These examples clearly demonstrate that righttoleft character comparisons can lead to farther shifts of the pattern than the shifts by only one position
7.2
Input Enhancement in String Matching
261
always made by the bruteforce algorithm. However, if such an algorithm had to check all the characters of the pattern on every trial, it would lose much of this superiority. Fortunately, the idea of input enhancement makes repetitive comparisons unnecessary. We can precompute shift sizes and store them in a table. The table will be indexed by all possible characters that can be encountered in a text, including, for natural language texts, the space, punctuation symbols, and other special characters. (Note that no other information about the text in which eventual searching will be done is required.) The table’s entries will indicate the shift sizes computed by the formula ⎧ the pattern’s length m, ⎪ ⎪ ⎪ ⎪ ⎨ if c is not among the ﬁrst m − 1 characters of the pattern; (7.1) t (c) = ⎪ ⎪ the distance from the rightmost c among the ﬁrst m − 1 characters ⎪ ⎪ ⎩ of the pattern to its last character, otherwise. For example, for the pattern BARBER, all the table’s entries will be equal to 6, except for the entries for E, B, R, and A, which will be 1, 2, 3, and 4, respectively. Here is a simple algorithm for computing the shift table entries. Initialize all the entries to the pattern’s length m and scan the pattern left to right repeating the following step m − 1 times: for the j th character of the pattern (0 ≤ j ≤ m − 2), overwrite its entry in the table with m − 1 − j , which is the character’s distance to the last character of the pattern. Note that since the algorithm scans the pattern from left to right, the last overwrite will happen for the character’s rightmost occurrence—exactly as we would like it to be. ALGORITHM ShiftTable(P [0..m − 1]) //Fills the shift table used by Horspool’s and BoyerMoore algorithms //Input: Pattern P [0..m − 1] and an alphabet of possible characters //Output: Table[0..size − 1] indexed by the alphabet’s characters and // ﬁlled with shift sizes computed by formula (7.1) for i ← 0 to size − 1 do Table[i] ← m for j ← 0 to m − 2 do Table[P [j ]] ← m − 1 − j return Table Now, we can summarize the algorithm as follows: Horspool’s algorithm Step 1 For a given pattern of length m and the alphabet used in both the pattern and text, construct the shift table as described above. Step 2 Align the pattern against the beginning of the text. Step 3 Repeat the following until either a matching substring is found or the pattern reaches beyond the last character of the text. Starting with the last character in the pattern, compare the corresponding characters in the pattern and text until either all m characters are matched (then
262
Space and Time TradeOffs
stop) or a mismatching pair is encountered. In the latter case, retrieve the entry t (c) from the c’s column of the shift table where c is the text’s character currently aligned against the last character of the pattern, and shift the pattern by t (c) characters to the right along the text. Here is pseudocode of Horspool’s algorithm. ALGORITHM HorspoolMatching(P [0..m − 1], T [0..n − 1]) //Implements Horspool’s algorithm for string matching //Input: Pattern P [0..m − 1] and text T [0..n − 1] //Output: The index of the left end of the ﬁrst matching substring // or −1 if there are no matches ShiftTable(P [0..m − 1]) //generate Table of shifts i ←m−1 //position of the pattern’s right end while i ≤ n − 1 do k←0 //number of matched characters while k ≤ m − 1 and P [m − 1 − k] = T [i − k] do k←k+1 if k = m return i − m + 1 else i ← i + Table[T [i]] return −1 EXAMPLE As an example of a complete application of Horspool’s algorithm, consider searching for the pattern BARBER in a text that comprises English letters and spaces (denoted by underscores). The shift table, as we mentioned, is ﬁlled as follows: character c
A
B
C
D
E
F
...
R
...
Z
_
shift t (c)
4
2
6
6
1
6
6
3
6
6
6
The actual search in a particular text proceeds as follows: J I M _ S A W _ M E _ I N _ A _ B A R B E R S H O P B A R B E R B A R B E R B A R B E R B A R B E R B A R B E R B A R B E R
A simple example can demonstrate that the worstcase efﬁciency of Horspool’s algorithm is in O(nm) (Problem 4 in this section’s exercises). But for random texts, it is in (n), and, although in the same efﬁciency class, Horspool’s algorithm is obviously faster on average than the bruteforce algorithm. In fact, as mentioned, it is often at least as efﬁcient as its more sophisticated predecessor discovered by R. Boyer and J. Moore.
7.2
263
Input Enhancement in String Matching
BoyerMoore Algorithm Now we outline the BoyerMoore algorithm itself. If the ﬁrst comparison of the rightmost character in the pattern with the corresponding character c in the text fails, the algorithm does exactly the same thing as Horspool’s algorithm. Namely, it shifts the pattern to the right by the number of characters retrieved from the table precomputed as explained earlier. The two algorithms act differently, however, after some positive number k (0 < k < m) of the pattern’s characters are matched successfully before a mismatch is encountered: s0
c
...
p0
pm−k−1
...
si−k+1 pm−k
si
... ...
...
sn−1
pm−1
text pattern
In this situation, the BoyerMoore algorithm determines the shift size by considering two quantities. The ﬁrst one is guided by the text’s character c that caused a mismatch with its counterpart in the pattern. Accordingly, it is called the badsymbol shift. The reasoning behind this shift is the reasoning we used in Horspool’s algorithm. If c is not in the pattern, we shift the pattern to just pass this c in the text. Conveniently, the size of this shift can be computed by the formula t1(c) − k where t1(c) is the entry in the precomputed table used by Horspool’s algorithm (see above) and k is the number of matched characters: s0
c
...
p0
...
pm−k−1
si−k+1 pm−k p0
... ... ...
si
...
sn−1
pm−1
text pattern
pm−1
For example, if we search for the pattern BARBER in some text and match the last two characters before failing on letter S in the text, we can shift the pattern by t1(S) − 2 = 6 − 2 = 4 positions: s0
...
S E R
...
sn−1
B A R B E R B A R B E R
The same formula can also be used when the mismatching character c of the text occurs in the pattern, provided t1(c) − k > 0. For example, if we search for the pattern BARBER in some text and match the last two characters before failing on letter A, we can shift the pattern by t1(A) − 2 = 4 − 2 = 2 positions: s0
...
A E R
B A R B E R B A R B E R
...
sn−1
264
Space and Time TradeOffs
If t1(c) − k ≤ 0, we obviously do not want to shift the pattern by 0 or a negative number of positions. Rather, we can fall back on the bruteforce thinking and simply shift the pattern by one position to the right. To summarize, the badsymbol shift d1 is computed by the BoyerMoore algorithm either as t1(c) − k if this quantity is positive and as 1 if it is negative or zero. This can be expressed by the following compact formula: d1 = max{t1(c) − k, 1}.
(7.2)
The second type of shift is guided by a successful match of the last k > 0 characters of the pattern. We refer to the ending portion of the pattern as its sufﬁx of size k and denote it suff (k). Accordingly, we call this type of shift the goodsufﬁx shift. We now apply the reasoning that guided us in ﬁlling the badsymbol shift table, which was based on a single alphabet character c, to the pattern’s sufﬁxes of sizes 1, . . . , m − 1 to ﬁll in the goodsufﬁx shift table. Let us ﬁrst consider the case when there is another occurrence of suff (k) in the pattern or, to be more accurate, there is another occurrence of suff (k) not preceded by the same character as in its rightmost occurrence. (It would be useless to shift the pattern to match another occurrence of suff (k) preceded by the same character because this would simply repeat a failed trial.) In this case, we can shift the pattern by the distance d2 between such a second rightmost occurrence (not preceded by the same character as in the rightmost occurrence) of suff (k) and its rightmost occurrence. For example, for the pattern ABCBAB, these distances for k = 1 and 2 will be 2 and 4, respectively: k
pattern
d2
1 2
ABCBAB ABCBAB
2 4
What is to be done if there is no other occurrence of suff (k) not preceded by the same character as in its rightmost occurrence? In most cases, we can shift the pattern by its entire length m. For example, for the pattern DBCBAB and k = 3, we can shift the pattern by its entire length of 6 characters: s0
...
c B A B
...
sn−1
D B C B A B D B C B A B
Unfortunately, shifting the pattern by its entire length when there is no other occurrence of suff (k) not preceded by the same character as in its rightmost occurrence is not always correct. For example, for the pattern ABCBAB and k = 3, shifting by 6 could miss a matching substring that starts with the text’s AB aligned with the last two characters of the pattern:
7.2
s0
...
265
Input Enhancement in String Matching
c B A B C B A B
...
sn−1
A B C B A B A B C B A B
Note that the shift by 6 is correct for the pattern DBCBAB but not for ABCBAB, because the latter pattern has the same substring AB as its preﬁx (beginning part of the pattern) and as its sufﬁx (ending part of the pattern). To avoid such an erroneous shift based on a sufﬁx of size k, for which there is no other occurrence in the pattern not preceded by the same character as in its rightmost occurrence, we need to ﬁnd the longest preﬁx of size l < k that matches the sufﬁx of the same size l. If such a preﬁx exists, the shift size d2 is computed as the distance between this preﬁx and the corresponding sufﬁx; otherwise, d2 is set to the pattern’s length m. As an example, here is the complete list of the d2 values—the goodsufﬁx table of the BoyerMoore algorithm—for the pattern ABCBAB:
k
pattern
d2
1 2 3 4 5
ABCBAB ABCBAB ABCBAB ABCBAB ABCBAB
2 4 4 4 4
Now we are prepared to summarize the BoyerMoore algorithm in its entirety. The BoyerMoore algorithm Step 1 For a given pattern and the alphabet used in both the pattern and the text, construct the badsymbol shift table as described earlier. Step 2 Using the pattern, construct the goodsufﬁx shift table as described earlier. Step 3 Align the pattern against the beginning of the text. Step 4 Repeat the following step until either a matching substring is found or the pattern reaches beyond the last character of the text. Starting with the last character in the pattern, compare the corresponding characters in the pattern and the text until either all m character pairs are matched (then stop) or a mismatching pair is encountered after k ≥ 0 character pairs are matched successfully. In the latter case, retrieve the entry t1(c) from the c’s column of the badsymbol table where c is the text’s mismatched character. If k > 0, also retrieve the corresponding d2 entry from the goodsufﬁx table. Shift the pattern to the right by the
266
Space and Time TradeOffs
number of positions computed by the formula if k = 0, d1 d= max{d1, d2 } if k > 0,
(7.3)
where d1 = max{t1(c) − k, 1}. Shifting by the maximum of the two available shifts when k > 0 is quite logical. The two shifts are based on the observations—the ﬁrst one about a text’s mismatched character, and the second one about a matched group of the pattern’s rightmost characters—that imply that shifting by less than d1 and d2 characters, respectively, cannot lead to aligning the pattern with a matching substring in the text. Since we are interested in shifting the pattern as far as possible without missing a possible matching substring, we take the maximum of these two numbers. EXAMPLE
As a complete example, let us consider searching for the pattern
BAOBAB in a text made of English letters and spaces. The badsymbol table looks
as follows: c
A
B
C
D
...
O
...
Z
_
t1(c)
1
2
6
6
6
3
6
6
6
The goodsufﬁx table is ﬁlled as follows: k
pattern
d2
1 2 3 4 5
BAOBAB BAOBAB BAOBAB BAOBAB BAOBAB
2 5 5 5 5
The actual search for this pattern in the text given in Figure 7.3 proceeds as follows. After the last B of the pattern fails to match its counterpart K in the text, the algorithm retrieves t1(K) = 6 from the badsymbol table and shifts the pattern by d1 = max{t1(K) − 0, 1} = 6 positions to the right. The new try successfully matches two pairs of characters. After the failure of the third comparison on the space character in the text, the algorithm retrieves t1( ) = 6 from the badsymbol table and d2 = 5 from the goodsufﬁx table to shift the pattern by max{d1, d2 } = max{6 − 2, 5} = 5. Note that on this iteration it is the goodsufﬁx rule that leads to a farther shift of the pattern. The next try successfully matches just one pair of B’s. After the failure of the next comparison on the space character in the text, the algorithm retrieves t1( ) = 6 from the badsymbol table and d2 = 2 from the goodsufﬁx table to shift
7.2
B
E
S
S
_
K
B
A
O
B
A
B
d1 = t1(K) − 0 = 6
267
Input Enhancement in String Matching
N
E
W
_
A
B
O
U
T
_
B
B
A
O
B
A
B
d1 = t1( ) − 2 = 4
B
A
O
B
A
B
d2 = 5
d1 = t1( ) − 1 = 5
d = max{4, 5} = 5
d2 = 2
A
O
B
A
B
A
O
B
A
B
S
d = max{5, 2} = 5 B
FIGURE 7.3 Example of string matching with the BoyerMoore algorithm.
the pattern by max{d1,d2 } = max{6 − 1, 2} = 5. Note that on this iteration it is the badsymbol rule that leads to a farther shift of the pattern. The next try ﬁnds a matching substring in the text after successfully matching all six characters of the pattern with their counterparts in the text. When searching for the ﬁrst occurrence of the pattern, the worstcase efﬁciency of the BoyerMoore algorithm is known to be linear. Though this algorithm runs very fast, especially on large alphabets (relative to the length of the pattern), many people prefer its simpliﬁed versions, such as Horspool’s algorithm, when dealing with naturallanguage–like strings.
Exercises 7.2 1. Apply Horspool’s algorithm to search for the pattern BAOBAB in the text BESS KNEW ABOUT BAOBABS
2. Consider the problem of searching for genes in DNA sequences using Horspool’s algorithm. A DNA sequence is represented by a text on the alphabet {A, C, G, T}, and the gene or gene segment is the pattern. a. Construct the shift table for the following gene segment of your chromosome 10: TCCTATTCTT
b. Apply Horspool’s algorithm to locate the above pattern in the following DNA sequence: TTATAGATCTCGTATTCTTTTATAGATCTCCTATTCTT
268
Space and Time TradeOffs
3. How many character comparisons will be made by Horspool’s algorithm in searching for each of the following patterns in the binary text of 1000 zeros? a. 00001 b. 10000 c. 01010 4. For searching in a text of length n for a pattern of length m (n ≥ m) with Horspool’s algorithm, give an example of a. worstcase input. b. bestcase input. 5. Is it possible for Horspool’s algorithm to make more character comparisons than the bruteforce algorithm would make in searching for the same pattern in the same text? 6. If Horspool’s algorithm discovers a matching substring, how large a shift should it make to search for a next possible match? 7. How many character comparisons will the BoyerMoore algorithm make in searching for each of the following patterns in the binary text of 1000 zeros? a. 00001 b. 10000 c. 01010 8. a. Would the BoyerMoore algorithm work correctly with just the badsymbol table to guide pattern shifts? b. Would the BoyerMoore algorithm work correctly with just the goodsufﬁx table to guide pattern shifts? 9. a. If the last characters of a pattern and its counterpart in the text do match, does Horspool’s algorithm have to check other characters right to left, or can it check them left to right too? b. Answer the same question for the BoyerMoore algorithm. 10. Implement Horspool’s algorithm, the BoyerMoore algorithm, and the bruteforce algorithm of Section 3.2 in the language of your choice and run an experiment to compare their efﬁciencies for matching a. random binary patterns in random binary texts. b. random naturallanguage patterns in naturallanguage texts. 11. You are given two strings S and T , each n characters long. You have to establish whether one of them is a right cyclic shift of the other. For example, PLEA is a right cyclic shift of LEAP, and vice versa. (Formally, T is a right cyclic shift of S if T can be obtained by concatenating the (n − i)character sufﬁx of S and the icharacter preﬁx of S for some 1 ≤ i ≤ n.) a. Design a spaceefﬁcient algorithm for the task. Indicate the space and time efﬁciencies of your algorithm. b. Design a timeefﬁcient algorithm for the task. Indicate the time and space efﬁciencies of your algorithm.
7.3 Hashing
269
7.3 Hashing In this section, we consider a very efﬁcient way to implement dictionaries. Recall that a dictionary is an abstract data type, namely, a set with the operations of searching (lookup), insertion, and deletion deﬁned on its elements. The elements of this set can be of an arbitrary nature: numbers, characters of some alphabet, character strings, and so on. In practice, the most important case is that of records (student records in a school, citizen records in a governmental ofﬁce, book records in a library). Typically, records comprise several ﬁelds, each responsible for keeping a particular type of information about an entity the record represents. For example, a student record may contain ﬁelds for the student’s ID, name, date of birth, sex, home address, major, and so on. Among record ﬁelds there is usually at least one called a key that is used for identifying entities represented by the records (e.g., the student’s ID). In the discussion below, we assume that we have to implement a dictionary of n records with keys K1, K2 , . . . , Kn. Hashing is based on the idea of distributing keys among a onedimensional array H [0..m − 1] called a hash table. The distribution is done by computing, for each of the keys, the value of some predeﬁned function h called the hash function. This function assigns an integer between 0 and m − 1, called the hash address, to a key. For example, if keys are nonnegative integers, a hash function can be of the form h(K) = K mod m; obviously, the remainder of division by m is always between 0 and m − 1. If keys are letters of some alphabet, we can ﬁrst assign a letter its position in the alphabet, denoted here ord(K), and then apply the same kind of a function used for integers. Finally, if K is a character string c0c1 . . . cs−1, we can use, as a very unsophisticated option, ( s−1 i=0 ord(ci )) mod m. A better option is to compute h(K) as follows:2 h ← 0;
for i ← 0 to s − 1 do h ← (h ∗ C + ord(ci )) mod m,
where C is a constant larger than every ord(ci ). In general, a hash function needs to satisfy somewhat conﬂicting requirements: A hash table’s size should not be excessively large compared to the number of keys, but it should be sufﬁcient to not jeopardize the implementation’s time efﬁciency (see below). A hash function needs to distribute keys among the cells of the hash table as evenly as possible. (This requirement makes it desirable, for most applications, to have a hash function dependent on all bits of a key, not just some of them.) A hash function has to be easy to compute. 2.
This can be obtained by treating ord(ci ) as digits of a number in the Cbased system, computing its decimal value by Horner’s rule, and ﬁnding the remainder of the number after dividing it by m.
270
Space and Time TradeOffs Ki
. 0
.
Kj
.
.
.
.
b
m –1
FIGURE 7.4 Collision of two keys in hashing: h(Ki ) = h(Kj ).
Obviously, if we choose a hash table’s size m to be smaller than the number of keys n, we will get collisions—a phenomenon of two (or more) keys being hashed into the same cell of the hash table (Figure 7.4). But collisions should be expected even if m is considerably larger than n (see Problem 5 in this section’s exercises). In fact, in the worst case, all the keys could be hashed to the same cell of the hash table. Fortunately, with an appropriately chosen hash table size and a good hash function, this situation happens very rarely. Still, every hashing scheme must have a collision resolution mechanism. This mechanism is different in the two principal versions of hashing: open hashing (also called separate chaining) and closed hashing (also called open addressing).
Open Hashing (Separate Chaining) In open hashing, keys are stored in linked lists attached to cells of a hash table. Each list contains all the keys hashed to its cell. Consider, as an example, the following list of words: A, FOOL, AND, HIS, MONEY, ARE, SOON, PARTED.
As a hash function, we will use the simple function for strings mentioned above, i.e., we will add the positions of a word’s letters in the alphabet and compute the sum’s remainder after division by 13. We start with the empty table. The ﬁrst key is the word A; its hash value is h(A) = 1 mod 13 = 1. The second key—the word FOOL—is installed in the ninth cell since (6 + 15 + 15 + 12) mod 13 = 9, and so on. The ﬁnal result of this process is given in Figure 7.5; note a collision of the keys ARE and SOON because h(ARE) = (1 + 18 + 5) mod 13 = 11 and h(SOON) = (19 + 15 + 15 + 14) mod 13 = 11. How do we search in a dictionary implemented as such a table of linked lists? We do this by simply applying to a search key the same procedure that was used for creating the table. To illustrate, if we want to search for the key KID in the hash table of Figure 7.5, we ﬁrst compute the value of the same hash function for the key: h(KID) = 11. Since the list attached to cell 11 is not empty, its linked list may contain the search key. But because of possible collisions, we cannot tell whether this is the case until we traverse this linked list. After comparing the string KID ﬁrst with the string ARE and then with the string SOON, we end up with an unsuccessful search. In general, the efﬁciency of searching depends on the lengths of the linked lists, which, in turn, depend on the dictionary and table sizes, as well as the quality
271
7.3 Hashing
keys
A
FOOL
AND
HIS
MONEY
ARE
SOON
PARTED
hash addresses
1
9
6
10
7
11
11
12
0
1
↓ A
2
3
4
5
6
↓
AND
7
8
↓
9
↓
MONEY
FOOL
10
↓
HIS
11
↓
ARE
12
↓
PARTED
↓
SOON
FIGURE 7.5 Example of a hash table construction with separate chaining.
of the hash function. If the hash function distributes n keys among m cells of the hash table about evenly, each list will be about n/m keys long. The ratio α = n/m, called the load factor of the hash table, plays a crucial role in the efﬁciency of hashing. In particular, the average number of pointers (chain links) inspected in successful searches, S, and unsuccessful searches, U, turns out to be S ≈1+
α 2
and
U = α,
(7.4)
respectively, under the standard assumptions of searching for a randomly selected element and a hash function distributing keys uniformly among the table’s cells. These results are quite natural. Indeed, they are almost identical to searching sequentially in a linked list; what we have gained by hashing is a reduction in average list size by a factor of m, the size of the hash table. Normally, we want the load factor to be not far from 1. Having it too small would imply a lot of empty lists and hence inefﬁcient use of space; having it too large would mean longer linked lists and hence longer search times. But if we do have the load factor around 1, we have an amazingly efﬁcient scheme that makes it possible to search for a given key for, on average, the price of one or two comparisons! True, in addition to comparisons, we need to spend time on computing the value of the hash function for a search key, but it is a constanttime operation, independent from n and m. Note that we are getting this remarkable efﬁciency not only as a result of the method’s ingenuity but also at the expense of extra space. The two other dictionary operations—insertion and deletion—are almost identical to searching. Insertions are normally done at the end of a list (but see Problem 6 in this section’s exercises for a possible modiﬁcation of this rule). Deletion is performed by searching for a key to be deleted and then removing it from its list. Hence, the efﬁciency of these operations is identical to that of searching, and they are all (1) in the average case if the number of keys n is about equal to the hash table’s size m.
272
Space and Time TradeOffs
Closed Hashing (Open Addressing) In closed hashing, all keys are stored in the hash table itself without the use of linked lists. (Of course, this implies that the table size m must be at least as large as the number of keys n.) Different strategies can be employed for collision resolution. The simplest one—called linear probing—checks the cell following the one where the collision occurs. If that cell is empty, the new key is installed there; if the next cell is already occupied, the availability of that cell’s immediate successor is checked, and so on. Note that if the end of the hash table is reached, the search is wrapped to the beginning of the table; i.e., it is treated as a circular array. This method is illustrated in Figure 7.6 with the same word list and hash function used above to illustrate separate chaining. To search for a given key K, we start by computing h(K) where h is the hash function used in the table construction. If the cell h(K) is empty, the search is unsuccessful. If the cell is not empty, we must compare K with the cell’s occupant: if they are equal, we have found a matching key; if they are not, we compare K with a key in the next cell and continue in this manner until we encounter either a matching key (a successful search) or an empty cell (unsuccessful search). For example, if we search for the word LIT in the table of Figure 7.6, we will get h(LIT) = (12 + 9 + 20) mod 13 = 2 and, since cell 2 is empty, we can stop immediately. However, if we search for KID with h(KID) = (11 + 9 + 4) mod 13 = 11, we will have to compare KID with ARE, SOON, PARTED, and A before we can declare the search unsuccessful. Although the search and insertion operations are straightforward for this version of hashing, deletion is not. For example, if we simply delete the key ARE from the last state of the hash table in Figure 7.6, we will be unable to ﬁnd the key SOON afterward. Indeed, after computing h(SOON) = 11, the algorithm would ﬁnd this location empty and report the unsuccessful search result. A simple solution
keys
A
FOOL
AND
HIS
MONEY
ARE
SOON
PARTED
hash addresses
1
9
6
10
7
11
11
12
0
1
2
3
4
5
6
7
8
9
10
11
12
A FOOL
A
PARTED
A
AND
A
AND
FOOL
HIS
A
AND
MONEY
FOOL
HIS
A
AND
MONEY
FOOL
HIS
ARE
A
AND
MONEY
FOOL
HIS
ARE
SOON
A
AND
MONEY
FOOL
HIS
ARE
SOON
FOOL
FIGURE 7.6 Example of a hash table construction with linear probing.
273
7.3 Hashing
is to use “lazy deletion,” i.e., to mark previously occupied locations by a special symbol to distinguish them from locations that have not been occupied. The mathematical analysis of linear probing is a much more difﬁcult problem than that of separate chaining.3 The simpliﬁed versions of these results state that the average number of times the algorithm must access the hash table with the load factor α in successful and unsuccessful searches is, respectively, 1 1 ) S ≈ (1 + 2 1−α
and
1 1 U ≈ (1 + ) 2 (1 − α)2
(7.5)
(and the accuracy of these approximations increases with larger sizes of the hash table). These numbers are surprisingly small even for densely populated tables, i.e., for large percentage values of α: α
1 1 2 (1 + 1−α )
1 1 2 (1 + (1−α)2 )
1.5 2.5 5.5
2.5 8.5 50.5
50% 75% 90%
Still, as the hash table gets closer to being full, the performance of linear probing deteriorates because of a phenomenon called clustering. A cluster in linear probing is a sequence of contiguously occupied cells (with a possible wrapping). For example, the ﬁnal state of the hash table of Figure 7.6 has two clusters. Clusters are bad news in hashing because they make the dictionary operations less efﬁcient. As clusters become larger, the probability that a new element will be attached to a cluster increases; in addition, large clusters increase the probability that two clusters will coalesce after a new key’s insertion, causing even more clustering. Several other collision resolution strategies have been suggested to alleviate this problem. One of the most important is double hashing. Under this scheme, we use another hash function, s(K), to determine a ﬁxed increment for the probing sequence to be used after a collision at location l = h(K): (l + s(K)) mod m,
(l + 2s(K)) mod m,
... .
(7.6)
To guarantee that every location in the table is probed by sequence (7.6), the increment s(k) and the table size m must be relatively prime, i.e., their only common divisor must be 1. (This condition is satisﬁed automatically if m itself is prime.) Some functions recommended in the literature are s(k) = m − 2 − k mod (m − 2) and s(k) = 8 − (k mod 8) for small tables and s(k) = k mod 97 + 1 for larger ones. 3.
This problem was solved in 1962 by a young graduate student in mathematics named Donald E. Knuth. Knuth went on to become one of the most important computer scientists of our time. His multivolume treatise The Art of Computer Programming [KnuI, KnuII, KnuIII, KnuIV] remains the most comprehensive and inﬂuential book on algorithmics ever published.
274
Space and Time TradeOffs
Mathematical analysis of double hashing has proved to be quite difﬁcult. Some partial results and considerable practical experience with the method suggest that with good hashing functions—both primary and secondary—double hashing is superior to linear probing. But its performance also deteriorates when the table gets close to being full. A natural solution in such a situation is rehashing: the current table is scanned, and all its keys are relocated into a larger table. It is worthwhile to compare the main properties of hashing with balanced search trees—its principal competitor for implementing dictionaries. Asymptotic time efﬁciency With hashing, searching, insertion, and deletion can be implemented to take (1) time on the average but (n) time in the very unlikely worst case. For balanced search trees, the average time efﬁciencies are (log n) for both the average and worst cases. Ordering preservation Unlike balanced search trees, hashing does not assume existence of key ordering and usually does not preserve it. This makes hashing less suitable for applications that need to iterate over the keys in order or require range queries such as counting the number of keys between some lower and upper bounds. Since its discovery in the 1950s by IBM researchers, hashing has found many important applications. In particular, it has become a standard technique for storing a symbol table—a table of a computer program’s symbols generated during compilation. Hashing is quite handy for such AI applications as checking whether positions generated by a chessplaying computer program have already been considered. With some modiﬁcations, it has also proved to be useful for storing very large dictionaries on disks; this variation of hashing is called extendible hashing. Since disk access is expensive compared with probes performed in the main memory, it is preferable to make many more probes than disk accesses. Accordingly, a location computed by a hash function in extendible hashing indicates a disk address of a bucket that can hold up to b keys. When a key’s bucket is identiﬁed, all its keys are read into main memory and then searched for the key in question. In the next section, we discuss Btrees, a principal alternative for storing large dictionaries.
Exercises 7.3 1. For the input 30, 20, 56, 75, 31, 19 and hash function h(K) = K mod 11 a. construct the open hash table. b. ﬁnd the largest number of key comparisons in a successful search in this table. c. ﬁnd the average number of key comparisons in a successful search in this table. 2. For the input 30, 20, 56, 75, 31, 19 and hash function h(K) = K mod 11 a. construct the closed hash table.
275
7.3 Hashing
b. ﬁnd the largest number of key comparisons in a successful search in this table. c. ﬁnd the average number of key comparisons in a successful search in this table. 3. Why is it not a good idea for a hash function to depend on just one letter (say, the ﬁrst one) of a naturallanguage word? 4. Find the probability of all n keys being hashed to the same cell of a hash table of size m if the hash function distributes keys evenly among all the cells of the table. 5. Birthday paradox The birthday paradox asks how many people should be in a room so that the chances are better than even that two of them will have the same birthday (month and day). Find the quite unexpected answer to this problem. What implication for hashing does this result have? 6. Answer the following questions for the separatechaining version of hashing. a. Where would you insert keys if you knew that all the keys in the dictionary are distinct? Which dictionary operations, if any, would beneﬁt from this modiﬁcation? b. We could keep keys of the same linked list sorted. Which of the dictionary operations would beneﬁt from this modiﬁcation? How could we take advantage of this if all the keys stored in the entire table need to be sorted? 7. Explain how to use hashing to check whether all elements of a list are distinct. What is the time efﬁciency of this application? Compare its efﬁciency with that of the bruteforce algorithm (Section 2.3) and of the presortingbased algorithm (Section 6.1). 8. Fill in the following table with the averagecase (as the ﬁrst entry) and worstcase (as the second entry) efﬁciency classes for the ﬁve implementations of the ADT dictionary: unordered array
ordered array
binary search tree
balanced search tree
hashing
search insertion deletion
9. We have discussed hashing in the context of techniques based on space–time tradeoffs. But it also takes advantage of another general strategy. Which one? 10. Write a computer program that uses hashing for the following problem. Given a naturallanguage text, generate a list of distinct words with the number of occurrences of each word in the text. Insert appropriate counters in the program to compare the empirical efﬁciency of hashing with the corresponding theoretical results.
276
Space and Time TradeOffs
p0
T0
K1
p1
T1
. . .
pi –1
Ti –1
Ki
. . .
pi
Ti
pn –2 Kn –1 pn –1
Tn –2
Tn –1
FIGURE 7.7 Parental node of a Btree.
7.4 BTrees The idea of using extra space to facilitate faster access to a given data set is particularly important if the data set in question contains a very large number of records that need to be stored on a disk. A principal device in organizing such data sets is an index, which provides some information about the location of records with indicated key values. For data sets of structured records (as opposed to “unstructured” data such as text, images, sound, and video), the most important index organization is the Btree, introduced by R. Bayer and E. McGreight [Bay72]. It extends the idea of the 23 tree (see Section 6.3) by permitting more than a single key in the same node of a search tree. In the Btree version we consider here, all data records (or record keys) are stored at the leaves, in increasing order of the keys. The parental nodes are used for indexing. Speciﬁcally, each parental node contains n − 1 ordered keys K1 < . . . < Kn−1 assumed, for the sake of simplicity, to be distinct. The keys are interposed with n pointers to the node’s children so that all the keys in subtree T0 are smaller than K1, all the keys in subtree T1 are greater than or equal to K1 and smaller than K2 with K1 being equal to the smallest key in T1, and so on, through the last subtree Tn−1 whose keys are greater than or equal to Kn−1 with Kn−1 being equal to the smallest key in Tn−1 (see Figure 7.7).4 In addition, a Btree of order m ≥ 2 must satisfy the following structural properties: The root is either a leaf or has between 2 and m children. Each node, except for the root and the leaves, has between m/2 and m children (and hence between m/2 − 1 and m − 1 keys). The tree is (perfectly) balanced, i.e., all its leaves are at the same level.
4.
The node depicted in Figure 7.7 is called the nnode. Thus, all the nodes in a classic binary search tree are 2nodes; a 23 tree introduced in Section 6.3 comprises 2nodes and 3nodes.
7.4
11
4, 7, 10
15
11, 14
15, 16, 19
20, 24
20
51
25
34
25, 28
277
BTrees
40
34, 38
60
40, 43, 46
51, 55
60, 68, 80
FIGURE 7.8 Example of a Btree of order 4.
An example of a Btree of order 4 is given in Figure 7.8. Searching in a Btree is very similar to searching in the binary search tree, and even more so in the 23 tree. Starting with the root, we follow a chain of pointers to the leaf that may contain the search key. Then we search for the search key among the keys of that leaf. Note that since keys are stored in sorted order, at both parental nodes and leaves, we can use binary search if the number of keys at a node is large enough to make it worthwhile. It is not the number of key comparisons, however, that we should be concerned about in a typical application of this data structure. When used for storing a large data ﬁle on a disk, the nodes of a Btree normally correspond to the disk pages. Since the time needed to access a disk page is typically several orders of magnitude larger than the time needed to compare keys in the fast computer memory, it is the number of disk accesses that becomes the principal indicator of the efﬁciency of this and similar data structures. How many nodes of a Btree do we need to access during a search for a record with a given key value? This number is, obviously, equal to the height of the tree plus 1. To estimate the height, let us ﬁnd the smallest number of keys a Btree of order m and positive height h can have. The root of the tree will contain at least one key. Level 1 will have at least two nodes with at least m/2 − 1 keys in each of them, for the total minimum number of keys 2(m/2 − 1). Level 2 will have at least 2m/2 nodes (the children of the nodes on level 1) with at least m/2 − 1 in each of them, for the total minimum number of keys 2m/2(m/2 − 1). In general, the nodes of level i, 1 ≤ i ≤ h − 1, will contain at least 2m/2i−1(m/2 − 1) keys. Finally, level h, the leaf level, will have at least 2m/2h−1 nodes with at least one key in each. Thus, for any Btree of order m with n nodes and height h > 0, we have the following inequality: n≥1+
h−1
2m/2i−1(m/2 − 1) + 2m/2h−1.
i=1
After a series of standard simpliﬁcations (see Problem 2 in this section’s exercises), this inequality reduces to
278
Space and Time TradeOffs
n ≥ 4m/2h−1 − 1, which, in turn, yields the following upper bound on the height h of the Btree of order m with n nodes: h ≤ logm/2
n+1 + 1. 4
(7.7)
Inequality (7.7) immediately implies that searching in a Btree is a O(log n) operation. But it is important to ascertain here not just the efﬁciency class but the actual number of disk accesses implied by this formula. The following table contains the values of the righthandside estimates for a ﬁle of 100 million records and a few typical values of the tree’s order m: order m h’s upper bound
50 6
100 5
250 4
Keep in mind that the table’s entries are upper estimates for the number of disk accesses. In actual applications, this number rarely exceeds 3, with the Btree’s root and sometimes ﬁrstlevel nodes stored in the fast memory to minimize the number of disk accesses. The operations of insertion and deletion are less straightforward than searching, but both can also be done in O(log n) time. Here we outline an insertion algorithm only; a deletion algorithm can be found in the references (e.g., [Aho83], [Cor09]). The most straightforward algorithm for inserting a new record into a Btree is quite similar to the algorithm for insertion into a 23 tree outlined in Section 6.3. First, we apply the search procedure to the new record’s key K to ﬁnd the appropriate leaf for the new record. If there is room for the record in that leaf, we place it there (in an appropriate position so that the keys remain sorted) and we are done. If there is no room for the record, the leaf is split in half by sending the second half of the records to a new node. After that, the smallest key K in the new node and the pointer to it are inserted into the old leaf’s parent (immediately after the key and pointer to the old leaf). This recursive procedure may percolate up to the tree’s root. If the root is already full too, a new root is created with the two halves of the old root’s keys split between two children of the new root. As an example, Figure 7.9 shows the result of inserting 65 into the Btree in Figure 7.8 under the restriction that the leaves cannot contain more than three items. You should be aware that there are other algorithms for implementing insertions into a Btree. For example, to avoid the possibility of recursive node splits, we can split full nodes encountered in searching for an appropriate leaf for the new record. Another possibility is to avoid some node splits by moving a key to the node’s sibling. For example, inserting 65 into the Btree in Figure 7.8 can be done by moving 60, the smallest key of the full leaf, to its sibling with keys 51 and 55, and replacing the key value of their parent by 65, the new smallest value in
7.4
11
4, 7, 10
15
11, 14
15, 16, 19
20, 24
20
51
25
34
25, 28
60
40
34, 38
279
BTrees
40, 43, 46
51, 55
68
60, 65
68, 80
FIGURE 7.9 Btree obtained after inserting 65 into the Btree in Figure 7.8.
the second child. This modiﬁcation tends to save some space at the expense of a slightly more complicated algorithm. A Btree does not have to be always associated with the indexing of a large ﬁle, and it can be considered as one of several search tree varieties. As with other types of search trees—such as binary search trees, AVL trees, and 23 trees—a Btree can be constructed by successive insertions of data records into the initially empty tree. (The empty tree is considered to be a Btree, too.) When all keys reside in the leaves and the upper levels are organized as a Btree comprising an index, the entire structure is usually called, in fact, a B+ tree.
Exercises 7.4 1. Give examples of using an index in reallife applications that do not involve computers. 2. a. Prove the equality 1+
h−1
2m/2i−1(m/2 − 1) + 2m/2h−1 = 4m/2h−1 − 1,
i=1
which was used in the derivation of upper bound (7.7) for the height of a Btree. b. Complete the derivation of inequality (7.7). 3. Find the minimum order of the Btree that guarantees that the number of disk accesses in searching in a ﬁle of 100 million records does not exceed 3. Assume that the root’s page is stored in main memory. 4. Draw the Btree obtained after inserting 30 and then 31 in the Btree in Figure 7.8. Assume that a leaf cannot contain more than three items. 5. Outline an algorithm for ﬁnding the largest key in a Btree. 6. a. A topdown 234 tree is a Btree of order 4 with the following modiﬁcation of the insert operation: Whenever a search for a leaf for a new key
280
Space and Time TradeOffs
encounters a full node (i.e., a node with three keys), the node is split into two nodes by sending its middle key to the node’s parent, or, if the full node happens to be the root, the new root for the middle key is created. Construct a topdown 234 tree by inserting the following list of keys in the initially empty tree: 10, 6, 15, 31, 20, 27, 50, 44, 18. b. What is the principal advantage of this insertion procedure compared with the one used for 23 trees in Section 6.3? What is its disadvantage? 7. a. Write a program implementing a key insertion algorithm in a Btree. b. Write a program for visualization of a key insertion algorithm in a Btree.
SUMMARY Space and time tradeoffs in algorithm design are a wellknown issue for both theoreticians and practitioners of computing. As an algorithm design technique, trading space for time is much more prevalent than trading time for space. Input enhancement is one of the two principal varieties of trading space for time in algorithm design. Its idea is to preprocess the problem’s input, in whole or in part, and store the additional information obtained in order to accelerate solving the problem afterward. Sorting by distribution counting and several important algorithms for string matching are examples of algorithms based on this technique. Distribution counting is a special method for sorting lists of elements from a small set of possible values. Horspool’s algorithm for string matching can be considered a simpliﬁed version of the BoyerMoore algorithm.Both algorithms are based on the ideas of input enhancement and righttoleft comparisons of a pattern’s characters. Both algorithms use the same badsymbol shift table; the BoyerMoore also uses a second table, called the goodsufﬁx shift table. Prestructuring—the second type of technique that exploits spacefortime tradeoffs—uses extra space to facilitate a faster and/or more ﬂexible access to the data. Hashing and B+trees are important examples of prestructuring. Hashing is a very efﬁcient approach to implementing dictionaries. It is based on the idea of mapping keys into a onedimensional table. The size limitations of such a table make it necessary to employ a collision resolution mechanism. The two principal varieties of hashing are open hashing or separate chaining (with keys stored in linked lists outside of the hash table) and closed hashing
Summary
281
or open addressing (with keys stored inside the table). Both enable searching, insertion, and deletion in (1) time, on average. The Btree is a balanced search tree that generalizes the idea of the 23 tree by allowing multiple keys at the same node. Its principal application, called the B+tree, is for keeping indexlike information about data stored on a disk. By choosing the order of the tree appropriately, one can implement the operations of searching, insertion, and deletion with just a few disk accesses even for extremely large ﬁles.
This page intentionally left blank
8 Dynamic Programming An idea, like a ghost . . . must be spoken to a little before it will explain itself. —Charles Dickens (1812–1870)
D
ynamic programming is an algorithm design technique with a rather interesting history. It was invented by a prominent U.S. mathematician, Richard Bellman, in the 1950s as a general method for optimizing multistage decision processes. Thus, the word “programming” in the name of this technique stands for “planning” and does not refer to computer programming. After proving its worth as an important tool of applied mathematics, dynamic programming has eventually come to be considered, at least in computer science circles, as a general algorithm design technique that does not have to be limited to special types of optimization problems. It is from this point of view that we will consider this technique here. Dynamic programming is a technique for solving problems with overlapping subproblems. Typically, these subproblems arise from a recurrence relating a given problem’s solution to solutions of its smaller subproblems. Rather than solving overlapping subproblems again and again, dynamic programming suggests solving each of the smaller subproblems only once and recording the results in a table from which a solution to the original problem can then be obtained. This technique can be illustrated by revisiting the Fibonacci numbers discussed in Section 2.5. (If you have not read that section, you will be able to follow the discussion anyway. But it is a beautiful topic, so if you feel a temptation to read it, do succumb to it.) The Fibonacci numbers are the elements of the sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . . ,
which can be deﬁned by the simple recurrence F (n) = F (n − 1) + F (n − 2)
for n > 1
(8.1) 283
284
Dynamic Programming
and two initial conditions F (0) = 0,
F (1) = 1.
(8.2)
If we try to use recurrence (8.1) directly to compute the nth Fibonacci number F (n), we would have to recompute the same values of this function many times (see Figure 2.6 for an example). Note that the problem of computing F (n) is expressed in terms of its smaller and overlapping subproblems of computing F (n − 1) and F (n − 2). So we can simply ﬁll elements of a onedimensional array with the n + 1 consecutive values of F (n) by starting, in view of initial conditions (8.2), with 0 and 1 and using equation (8.1) as the rule for producing all the other elements. Obviously, the last element of this array will contain F (n). Singleloop pseudocode of this very simple algorithm can be found in Section 2.5. Note that we can, in fact, avoid using an extra array to accomplish this task by recording the values of just the last two elements of the Fibonacci sequence (Problem 8 in Exercises 2.5). This phenomenon is not unusual, and we shall encounter it in a few more examples in this chapter. Thus, although a straightforward application of dynamic programming can be interpreted as a special variety of spacefortime tradeoff, a dynamic programming algorithm can sometimes be reﬁned to avoid using extra space. Certain algorithms compute the nth Fibonacci number without computing all the preceding elements of this sequence (see Section 2.5). It is typical of an algorithm based on the classic bottomup dynamic programming approach, however, to solve all smaller subproblems of a given problem. One variation of the dynamic programming approach seeks to avoid solving unnecessary subproblems. This technique, illustrated in Section 8.2, exploits socalled memory functions and can be considered a topdown variation of dynamic programming. Whether one uses the classical bottomup version of dynamic programming or its topdown variation, the crucial step in designing such an algorithm remains the same: deriving a recurrence relating a solution to the problem to solutions to its smaller subproblems. The immediate availability of equation (8.1) for computing the nth Fibonacci number is one of the few exceptions to this rule. Since a majority of dynamic programming applications deal with optimization problems, we also need to mention a general principle that underlines such applications. Richard Bellman called it the principle of optimality. In terms somewhat different from its original formulation, it says that an optimal solution to any instance of an optimization problem is composed of optimal solutions to its subinstances. The principle of optimality holds much more often than not. (To give a rather rare example, it fails for ﬁnding the longest simple path in a graph.) Although its applicability to a particular problem needs to be checked, of course, such a check is usually not a principal difﬁculty in developing a dynamic programming algorithm. In the sections and exercises of this chapter are a few standard examples of dynamic programming algorithms. (The algorithms in Section 8.4 were, in fact,
8.1 Three Basic Examples
285
invented independently of the discovery of dynamic programming and only later came to be viewed as examples of this technique’s applications.) Numerous other applications range from the optimal way of breaking text into lines (e.g., [Baa00]) to image resizing [Avi07] to a variety of applications to sophisticated engineering problems (e.g., [Ber01]).
8.1 Three Basic Examples The goal of this section is to introduce dynamic programming via three typical examples. EXAMPLE 1 Coinrow problem There is a row of n coins whose values are some positive integers c1, c2 , . . . , cn, not necessarily distinct. The goal is to pick up the maximum amount of money subject to the constraint that no two coins adjacent in the initial row can be picked up. Let F (n) be the maximum amount that can be picked up from the row of n coins. To derive a recurrence for F (n), we partition all the allowed coin selections into two groups: those that include the last coin and those without it. The largest amount we can get from the ﬁrst group is equal to cn + F (n − 2)—the value of the nth coin plus the maximum amount we can pick up from the ﬁrst n − 2 coins. The maximum amount we can get from the second group is equal to F (n − 1) by the deﬁnition of F (n). Thus, we have the following recurrence subject to the obvious initial conditions: F (n) = max{cn + F (n − 2), F (n − 1)} F (0) = 0,
F (1) = c1.
for n > 1,
(8.3)
We can compute F (n) by ﬁlling the onerow table left to right in the manner similar to the way it was done for the nth Fibonacci number by Algorithm Fib(n) in Section 2.5. ALGORITHM CoinRow(C[1..n]) //Applies formula (8.3) bottom up to ﬁnd the maximum amount of money //that can be picked up from a coin row without picking two adjacent coins //Input: Array C[1..n] of positive integers indicating the coin values //Output: The maximum amount of money that can be picked up F [0] ← 0; F [1] ← C[1] for i ← 2 to n do F [i] ← max(C[i] + F [i − 2], F [i − 1]) return F [n] The application of the algorithm to the coin row of denominations 5, 1, 2, 10, 6, 2 is shown in Figure 8.1. It yields the maximum amount of 17. It is worth pointing
286
Dynamic Programming
F [0] = 0, F [1] = c1 = 5
index 0 1 2 3 4 5 6 5 1 2 10 6 2 C F 0 5
F [2] = max{1 + 0, 5} = 5
index 0 1 2 3 4 5 6 5 1 2 10 6 2 C F 0 5 5
F [3] = max{2 + 5, 5} = 7
index 0 1 2 3 4 5 6 5 1 2 10 6 2 C F 0 5 5 7
F [4] = max{10 + 5, 7} = 15
index 0 1 2 3 4 5 6 5 1 2 10 6 2 C F 0 5 5 7 15
F [5] = max{6 + 7, 15} = 15
index 0 1 2 3 4 5 6 5 1 2 10 6 2 C F 0 5 5 7 15 15
F [6] = max{2 + 15, 15} = 17
index 0 1 2 3 4 5 6 5 1 2 10 6 2 C F 0 5 5 7 15 15 17
FIGURE 8.1 Solving the coinrow problem by dynamic programming for the coin row 5, 1, 2, 10, 6, 2.
out that, in fact, we also solved the problem for the ﬁrst i coins in the row given for every 1 ≤ i ≤ 6. For example, for i = 3, the maximum amount is F (3) = 7. To ﬁnd the coins with the maximum total value found, we need to backtrace the computations to see which of the two possibilities—cn + F (n − 2) or F (n − 1)—produced the maxima in formula (8.3). In the last application of the formula, it was the sum c6 + F (4), which means that the coin c6 = 2 is a part of an optimal solution. Moving to computing F (4), the maximum was produced by the sum c4 + F (2), which means that the coin c4 = 10 is a part of an optimal solution as well. Finally, the maximum in computing F (2) was produced by F (1), implying that the coin c2 is not the part of an optimal solution and the coin c1 = 5 is. Thus, the optimal solution is {c1, c4, c6}. To avoid repeating the same computations during the backtracing, the information about which of the two terms in (8.3) was larger can be recorded in an extra array when the values of F are computed. Using the CoinRow to ﬁnd F (n), the largest amount of money that can be picked up, as well as the coins composing an optimal set, clearly takes (n) time and (n) space. This is by far superior to the alternatives: the straightforward top
8.1 Three Basic Examples
287
down application of recurrence (8.3) and solving the problem by exhaustive search (Problem 3 in this section’s exercises). EXAMPLE 2 Changemaking problem Consider the general instance of the following wellknown problem. Give change for amount n using the minimum number of coins of denominations d1 < d2 < . . . < dm. For the coin denominations used in the United States, as for those used in most if not all other countries, there is a very simple and efﬁcient algorithm discussed in the next chapter. Here, we consider a dynamic programming algorithm for the general case, assuming availability of unlimited quantities of coins for each of the m denominations d1 < d2 < . . . < dm where d1 = 1. Let F (n) be the minimum number of coins whose values add up to n; it is convenient to deﬁne F (0) = 0. The amount n can only be obtained by adding one coin of denomination dj to the amount n − dj for j = 1, 2, . . . , m such that n ≥ dj . Therefore, we can consider all such denominations and select the one minimizing F (n − dj ) + 1. Since 1 is a constant, we can, of course, ﬁnd the smallest F (n − dj ) ﬁrst and then add 1 to it. Hence, we have the following recurrence for F (n): F (n) = min {F (n − dj )} + 1 for n > 0, j : n≥dj
F (0) = 0.
(8.4)
We can compute F (n) by ﬁlling a onerow table left to right in the manner similar to the way it was done above for the coinrow problem, but computing a table entry here requires ﬁnding the minimum of up to m numbers. ALGORITHM ChangeMaking(D[1..m], n) //Applies dynamic programming to ﬁnd the minimum number of coins //of denominations d1 < d2 < . . . < dm where d1 = 1 that add up to a //given amount n //Input: Positive integer n and array D[1..m] of increasing positive // integers indicating the coin denominations where D[1] = 1 //Output: The minimum number of coins that add up to n F [0] ← 0 for i ← 1 to n do temp ← ∞; j ← 1 while j ≤ m and i ≥ D[j ] do temp ← min(F [i − D[j ]], temp) j ←j +1 F [i] ← temp + 1 return F [n] The application of the algorithm to amount n = 6 and denominations 1, 3, 4 is shown in Figure 8.2. The answer it yields is two coins. The time and space efﬁciencies of the algorithm are obviously O(nm) and (n), respectively.
288
Dynamic Programming
n 0 F 0
1
2
3
4
5
6
F [0] = 0
n 0 1 F 0 1
2
3
4
5
6
F [1] = min{F [1 – 1]} + 1 = 1
n 0 1 2 F 0 1 2
3
4
5
6
F [2] = min{F [2 – 1]} + 1 = 2
n 0 1 2 3 F 0 1 2 1
4
5
6
F [3] = min{F [3 – 1], F [3 – 3]} + 1 = 1
n 0 1 2 3 4 F 0 1 2 1 1
5
6
F [4] = min{F [4 – 1], F [4 – 3], F [4 – 4]} + 1 = 1
n 0 1 2 3 4 5 F 0 1 2 1 1 2
6
F [5] = min{F[5 – 1], F [5 – 3], F [5 – 4]} + 1 = 2
F [6] = min{F[6 – 1], F [6 – 3], F [6 – 4]} + 1 = 2
n 0 1 2 3 4 5 6 F 0 1 2 1 1 2 2
FIGURE 8.2 Application of Algorithm MinCoinChange to amount n = 6 and coin denominations 1, 3, and 4.
To ﬁnd the coins of an optimal solution, we need to backtrace the computations to see which of the denominations produced the minima in formula (8.4). For the instance considered, the last application of the formula (for n = 6), the minimum was produced by d2 = 3. The second minimum (for n = 6 − 3) was also produced for a coin of that denomination. Thus, the minimumcoin set for n = 6 is two 3’s. EXAMPLE 3 Coincollecting problem Several coins are placed in cells of an n × m board, no more than one coin per cell. A robot, located in the upper left cell of the board, needs to collect as many of the coins as possible and bring them to the bottom right cell. On each step, the robot can move either one cell to the right or one cell down from its current location. When the robot visits a cell with a coin, it always picks up that coin. Design an algorithm to ﬁnd the maximum number of coins the robot can collect and a path it needs to follow to do this. Let F (i, j ) be the largest number of coins the robot can collect and bring to the cell (i, j ) in the ith row and j th column of the board. It can reach this cell either from the adjacent cell (i − 1, j ) above it or from the adjacent cell (i, j − 1) to the left of it. The largest numbers of coins that can be brought to these cells are F (i − 1, j ) and F (i, j − 1), respectively. Of course, there are no adjacent cells
8.1 Three Basic Examples
289
above the cells in the ﬁrst row, and there are no adjacent cells to the left of the cells in the ﬁrst column. For those cells, we assume that F (i − 1, j ) and F (i, j − 1) are equal to 0 for their nonexistent neighbors. Therefore, the largest number of coins the robot can bring to cell (i, j ) is the maximum of these two numbers plus one possible coin at cell (i, j ) itself. In other words, we have the following formula for F (i, j ): F (i, j ) = max{F (i − 1, j ), F (i, j − 1)} + cij F (0, j ) = 0 for 1 ≤ j ≤ m
and
for 1 ≤ i ≤ n, 1 ≤ j ≤ m
F (i, 0) = 0 for 1 ≤ i ≤ n,
(8.5)
where cij = 1 if there is a coin in cell (i, j ), and cij = 0 otherwise. Using these formulas, we can ﬁll in the n × m table of F (i, j ) values either row by row or column by column, as is typical for dynamic programming algorithms involving twodimensional tables.
ALGORITHM RobotCoinCollection(C[1..n, 1..m]) //Applies dynamic programming to compute the largest number of //coins a robot can collect on an n × m board by starting at (1, 1) //and moving right and down from upper left to down right corner //Input: Matrix C[1..n, 1..m] whose elements are equal to 1 and 0 //for cells with and without a coin, respectively //Output: Largest number of coins the robot can bring to cell (n, m) F [1, 1] ← C[1, 1]; for j ← 2 to m do F [1, j ] ← F [1, j − 1] + C[1, j ] for i ← 2 to n do F [i, 1] ← F [i − 1, 1] + C[i, 1] for j ← 2 to m do F [i, j ] ← max(F [i − 1, j ], F [i, j − 1]) + C[i, j ] return F [n, m]
The algorithm is illustrated in Figure 8.3b for the coin setup in Figure 8.3a. Since computing the value of F (i, j ) by formula (8.5) for each cell of the table takes constant time, the time efﬁciency of the algorithm is (nm). Its space efﬁciency is, obviously, also (nm). Tracing the computations backward makes it possible to get an optimal path: if F (i − 1, j ) > F (i, j − 1), an optimal path to cell (i, j ) must come down from the adjacent cell above it; if F (i − 1, j ) < F (i, j − 1), an optimal path to cell (i, j ) must come from the adjacent cell on the left; and if F (i − 1, j ) = F (i, j − 1), it can reach cell (i, j ) from either direction. This yields two optimal paths for the instance in Figure 8.3a, which are shown in Figure 8.3c. If ties are ignored, one optimal path can be obtained in (n + m) time.
290
Dynamic Programming 1
2
3
4
5
6
1
2
3
4
5
6
1
1
0
0
0
0
1
1
2
2
0
1
1
2
2
2
3
3
0
1
1
3
3
4
4
4
0
1
2
3
3
5
5
5
1
1
2
3
4
5
(b)
(a) 1
2
3
4
5
6
1 2 3 4 5 (c)
FIGURE 8.3 (a) Coins to collect. (b) Dynamic programming algorithm results. (c) Two paths to collect 5 coins, the maximum number of coins possible.
Exercises 8.1 1. What does dynamic programming have in common with divideandconquer? What is a principal difference between them? 2. Solve the instance 5, 1, 2, 10, 6 of the coinrow problem. 3. a. Show that the time efﬁciency of solving the coinrow problem by straightforward application of recurrence (8.3) is exponential. b. Show that the time efﬁciency of solving the coinrow problem by exhaustive search is at least exponential. 4. Apply the dynamic programming algorithm to ﬁnd all the solutions to the changemaking problem for the denominations 1, 3, 5 and the amount n = 9.
8.1 Three Basic Examples
291
5. How would you modify the dynamic programming algorithm for the coincollecting problem if some cells on the board are inaccessible for the robot? Apply your algorithm to the board below, where the inaccessible cells are shown by X’s. How many optimal paths are there for this board? 1
2
3
4
5
6
1 2 3 4 5
6. Rodcutting problem Design a dynamic programming algorithm for the following problem. Find the maximum total sale price that can be obtained by cutting a rod of n units long into integerlength pieces if the sale price of a piece i units long is pi for i = 1, 2, . . . , n. What are the time and space efﬁciencies of your algorithm? 7. Shortestpath counting A chess rook can move horizontally or vertically to any square in the same row or in the same column of a chessboard. Find the number of shortest paths by which a rook can move from one corner of a chessboard to the diagonally opposite corner. The length of a path is measured by the number of squares it passes through, including the ﬁrst and the last squares. Solve the problem a. by a dynamic programming algorithm. b. by using elementary combinatorics. 8. Minimumsum descent Some positive integers are arranged in an equilateral triangle with n numbers in its base like the one shown in the ﬁgure below for n = 4. The problem is to ﬁnd the smallest sum in a descent from the triangle apex to its base through a sequence of adjacent numbers (shown in the ﬁgure by the circles). Design a dynamic programming algorithm for this problem and indicate its time efﬁciency. 2 5
1 8
4
4 6
7 9
6
292
Dynamic Programming
9. Binomial coefﬁcient Design an efﬁcient algorithm for computing the binomial coefﬁcient C(n, k) that uses no multiplications. What are the time and space efﬁciencies of your algorithm? 10. Longest path in a dag a. Design an efﬁcient algorithm for ﬁnding the length of the longest path in a dag. (This problem is important both as a prototype of many other dynamic programming applications and in its own right because it determines the minimal time needed for completing a project comprising precedenceconstrained tasks.) b. Show how to reduce the coinrow problem discussed in this section to the problem of ﬁnding a longest path in a dag. 11. Maximum square submatrix Given an m × n boolean matrix B, ﬁnd its largest square submatrix whose elements are all zeros. Design a dynamic programming algorithm and indicate its time efﬁciency. (The algorithm may be useful for, say, ﬁnding the largest free square area on a computer screen or for selecting a construction site.) 12. World Series odds Consider two teams, A and B, playing a series of games until one of the teams wins n games. Assume that the probability of A winning a game is the same for each game and equal to p, and the probability of A losing a game is q = 1 − p. (Hence, there are no ties.) Let P (i, j ) be the probability of A winning the series if A needs i more games to win the series and B needs j more games to win the series. a. Set up a recurrence relation for P (i, j ) that can be used by a dynamic programming algorithm. b. Find the probability of team A winning a sevengame series if the probability of it winning a game is 0.4. c. Write pseudocode of the dynamic programming algorithm for solving this problem and determine its time and space efﬁciencies.
8.2 The Knapsack Problem and Memory Functions We start this section with designing a dynamic programming algorithm for the knapsack problem: given n items of known weights w1, . . . , wn and values v1, . . . , vn and a knapsack of capacity W , ﬁnd the most valuable subset of the items that ﬁt into the knapsack. (This problem was introduced in Section 3.4, where we discussed solving it by exhaustive search.) We assume here that all the weights and the knapsack capacity are positive integers; the item values do not have to be integers. To design a dynamic programming algorithm, we need to derive a recurrence relation that expresses a solution to an instance of the knapsack problem in terms
8.2
293
The Knapsack Problem and Memory Functions
of solutions to its smaller subinstances. Let us consider an instance deﬁned by the ﬁrst i items, 1 ≤ i ≤ n, with weights w1, . . . , wi , values v1, . . . , vi , and knapsack capacity j, 1 ≤ j ≤ W. Let F (i, j ) be the value of an optimal solution to this instance, i.e., the value of the most valuable subset of the ﬁrst i items that ﬁt into the knapsack of capacity j. We can divide all the subsets of the ﬁrst i items that ﬁt the knapsack of capacity j into two categories: those that do not include the ith item and those that do. Note the following: 1. 2.
Among the subsets that do not include the ith item, the value of an optimal subset is, by deﬁnition, F (i − 1, j ). Among the subsets that do include the ith item (hence, j − wi ≥ 0), an optimal subset is made up of this item and an optimal subset of the ﬁrst i − 1 items that ﬁts into the knapsack of capacity j − wi . The value of such an optimal subset is vi + F (i − 1, j − wi ).
Thus, the value of an optimal solution among all feasible subsets of the ﬁrst i items is the maximum of these two values. Of course, if the ith item does not ﬁt into the knapsack, the value of an optimal subset selected from the ﬁrst i items is the same as the value of an optimal subset selected from the ﬁrst i − 1 items. These observations lead to the following recurrence: max{F (i − 1, j ), vi + F (i − 1, j − wi )} if j − wi ≥ 0, (8.6) F (i, j ) = F (i − 1, j ) if j − wi < 0. It is convenient to deﬁne the initial conditions as follows: F (0, j ) = 0 for j ≥ 0
and
F (i, 0) = 0 for i ≥ 0.
(8.7)
Our goal is to ﬁnd F (n, W ), the maximal value of a subset of the n given items that ﬁt into the knapsack of capacity W, and an optimal subset itself. Figure 8.4 illustrates the values involved in equations (8.6) and (8.7). For i, j > 0, to compute the entry in the ith row and the j th column, F (i, j ), we compute the maximum of the entry in the previous row and the same column and the sum of vi and the entry in the previous row and wi columns to the left. The table can be ﬁlled either row by row or column by column.
wi, vi
0
j –wi
j
W
0
0
0
0
0
i –1 i
0 0
F (i –1, j – wi )
F (i –1, j ) F (i, j )
n
0
goal
FIGURE 8.4 Table for solving the knapsack problem by dynamic programming.
294
Dynamic Programming
capacity j i 0 1 2 3 4
w1 = 2, v1 = 12 w2 = 1, v2 = 10 w3 = 3, v3 = 20 w4 = 2, v4 = 15
0 0 0 0 0 0
1 0 0 10 10 10
2 0 12 12 12 15
3 0 12 22 22 25
4 0 12 22 30 30
5 0 12 22 32 37
FIGURE 8.5 Example of solving an instance of the knapsack problem by the dynamic programming algorithm.
EXAMPLE 1
Let us consider the instance given by the following data: item
weight
1 2 3 4
2 1 3 2
value $12 $10 $20 $15
capacity W = 5.
The dynamic programming table, ﬁlled by applying formulas (8.6) and (8.7), is shown in Figure 8.5. Thus, the maximal value is F (4, 5) = $37. We can ﬁnd the composition of an optimal subset by backtracing the computations of this entry in the table. Since F (4, 5) > F (3, 5), item 4 has to be included in an optimal solution along with an optimal subset for ﬁlling 5 − 2 = 3 remaining units of the knapsack capacity. The value of the latter is F (3, 3). Since F (3, 3) = F (2, 3), item 3 need not be in an optimal subset. Since F (2, 3) > F (1, 3), item 2 is a part of an optimal selection, which leaves element F (1, 3 − 1) to specify its remaining composition. Similarly, since F (1, 2) > F (0, 2), item 1 is the ﬁnal part of the optimal solution {item 1, item 2, item 4}. The time efﬁciency and space efﬁciency of this algorithm are both in (nW ). The time needed to ﬁnd the composition of an optimal solution is in O(n). You are asked to prove these assertions in the exercises.
Memory Functions As we discussed at the beginning of this chapter and illustrated in subsequent sections, dynamic programming deals with problems whose solutions satisfy a recurrence relation with overlapping subproblems. The direct topdown approach to ﬁnding a solution to such a recurrence leads to an algorithm that solves common subproblems more than once and hence is very inefﬁcient (typically, exponential
8.2
The Knapsack Problem and Memory Functions
295
or worse). The classic dynamic programming approach, on the other hand, works bottom up: it ﬁlls a table with solutions to all smaller subproblems, but each of them is solved only once. An unsatisfying aspect of this approach is that solutions to some of these smaller subproblems are often not necessary for getting a solution to the problem given. Since this drawback is not present in the topdown approach, it is natural to try to combine the strengths of the topdown and bottomup approaches. The goal is to get a method that solves only subproblems that are necessary and does so only once. Such a method exists; it is based on using memory functions. This method solves a given problem in the topdown manner but, in addition, maintains a table of the kind that would have been used by a bottomup dynamic programming algorithm. Initially, all the table’s entries are initialized with a special “null” symbol to indicate that they have not yet been calculated. Thereafter, whenever a new value needs to be calculated, the method checks the corresponding entry in the table ﬁrst: if this entry is not “null,” it is simply retrieved from the table; otherwise, it is computed by the recursive call whose result is then recorded in the table. The following algorithm implements this idea for the knapsack problem. After initializing the table, the recursive function needs to be called with i = n (the number of items) and j = W (the knapsack capacity). ALGORITHM MFKnapsack(i, j ) //Implements the memory function method for the knapsack problem //Input: A nonnegative integer i indicating the number of the ﬁrst // items being considered and a nonnegative integer j indicating // the knapsack capacity //Output: The value of an optimal feasible subset of the ﬁrst i items //Note: Uses as global variables input arrays W eights[1..n], V alues[1..n], //and table F [0..n, 0..W ] whose entries are initialized with −1’s except for //row 0 and column 0 initialized with 0’s if F [i, j ] < 0 if j < Weights[i] value ← MFKnapsack(i − 1, j ) else value ← max(MFKnapsack(i − 1, j ), Values[i] + MFKnapsack(i − 1, j − Weights[i])) F [i, j ] ← value return F [i, j ] EXAMPLE 2 Let us apply the memory function method to the instance considered in Example 1. The table in Figure 8.6 gives the results. Only 11 out of 20 nontrivial values (i.e., not those in row 0 or in column 0) have been computed.
296
Dynamic Programming
capacity j i 0 1 2 3 4
w1 = 2, v1 = 12 w2 = 1, v2 = 10 w3 = 3, v3 = 20 w4 = 2, v4 = 15
0 0 0 0 0 0
1 0 0 — — —
2 0 12 12 — —
3 0 12 22 22 —
4 0 12 — — —
5 0 12 22 32 37
FIGURE 8.6 Example of solving an instance of the knapsack problem by the memory function algorithm.
Just one nontrivial entry, V (1, 2), is retrieved rather than being recomputed. For larger instances, the proportion of such entries can be signiﬁcantly larger. In general, we cannot expect more than a constantfactor gain in using the memory function method for the knapsack problem, because its time efﬁciency class is the same as that of the bottomup algorithm (why?). A more signiﬁcant improvement can be expected for dynamic programming algorithms in which a computation of one value takes more than constant time. You should also keep in mind that a memory function algorithm may be less spaceefﬁcient than a spaceefﬁcient version of a bottomup algorithm.
Exercises 8.2 1. a. Apply the bottomup dynamic programming algorithm to the following instance of the knapsack problem: item
weight
1 2 3 4 5
3 2 1 4 5
value $25 $20 $15 $40 $50
capacity W = 6.
b. How many different optimal subsets does the instance of part (a) have? c. In general, how can we use the table generated by the dynamic programming algorithm to tell whether there is more than one optimal subset for the knapsack problem’s instance?
8.3 Optimal Binary Search Trees
297
2. a. Write pseudocode of the bottomup dynamic programming algorithm for the knapsack problem. b. Write pseudocode of the algorithm that ﬁnds the composition of an optimal subset from the table generated by the bottomup dynamic programming algorithm for the knapsack problem. 3. For the bottomup dynamic programming algorithm for the knapsack problem, prove that a. its time efﬁciency is (nW ). b. its space efﬁciency is (nW ). c. the time needed to ﬁnd the composition of an optimal subset from a ﬁlled dynamic programming table is O(n). 4. a. True or false: A sequence of values in a row of the dynamic programming table for the knapsack problem is always nondecreasing? b. True or false: A sequence of values in a column of the dynamic programming table for the knapsack problem is always nondecreasing? 5. Design a dynamic programming algorithm for the version of the knapsack problem in which there are unlimited quantities of copies for each of the n item kinds given. Indicate the time efﬁciency of the algorithm. 6. Apply the memory function method to the instance of the knapsack problem given in Problem 1. Indicate the entries of the dynamic programming table that are (i) never computed by the memory function method, (ii) retrieved without a recomputation. 7. Prove that the efﬁciency class of the memory function algorithm for the knapsack problem is the same as that of the bottomup algorithm (see Problem 3). 8. Explain why the memory function approach is unattractive for the problem of computing a binomial coefﬁcient by the formula C(n, k) = C(n − 1, k − 1) + C(n − 1, k). 9. Write a research report on one of the following wellknown applications of dynamic programming: a. ﬁnding the longest common subsequence in two sequences b. optimal string editing c. minimal triangulation of a polygon
8.3 Optimal Binary Search Trees A binary search tree is one of the most important data structures in computer science. One of its principal applications is to implement a dictionary, a set of elements with the operations of searching, insertion, and deletion. If probabilities
298
Dynamic Programming
B
A A
B C
C D
D
FIGURE 8.7 Two out of 14 possible binary search trees with keys A, B, C, and D.
of searching for elements of a set are known—e.g., from accumulated data about past searches—it is natural to pose a question about an optimal binary search tree for which the average number of comparisons in a search is the smallest possible. For simplicity, we limit our discussion to minimizing the average number of comparisons in a successful search. The method can be extended to include unsuccessful searches as well. As an example, consider four keys A, B, C, and D to be searched for with probabilities 0.1, 0.2, 0.4, and 0.3, respectively. Figure 8.7 depicts two out of 14 possible binary search trees containing these keys. The average number of comparisons in a successful search in the ﬁrst of these trees is 0.1 . 1 + 0.2 . 2 + 0.4 . 3 + 0.3 . 4 = 2.9, and for the second one it is 0.1 . 2 + 0.2 . 1 + 0.4 . 2 + 0.3 . 3 = 2.1. Neither of these two trees is, in fact, optimal. (Can you tell which binary tree is optimal?) For our tiny example, we could ﬁnd the optimal tree by generating all 14 binary search trees with these keys. As a general algorithm, this exhaustivesearch approach is unrealistic: the total number of binary search trees with n keys is equal to the nth Catalan number,
2n 1 for n > 0, c(0) = 1, c(n) = n+1 n which grows to inﬁnity as fast as 4n/n1.5 (see Problem 7 in this section’s exercises). So let a1, . . . , an be distinct keys ordered from the smallest to the largest and let p1, . . . , pn be the probabilities of searching for them. Let C(i, j ) be the smallest average number of comparisons made in a successful search in a binary search tree j Ti made up of keys ai , . . . , aj , where i, j are some integer indices, 1 ≤ i ≤ j ≤ n. Following the classic dynamic programming approach, we will ﬁnd values of C(i, j ) for all smaller instances of the problem, although we are interested just in C(1, n). To derive a recurrence underlying a dynamic programming algorithm, we will consider all possible ways to choose a root ak among the keys ai , . . . , aj . For such a binary search tree (Figure 8.8), the root contains key ak , the left subtree j Tik−1 contains keys ai , . . . , ak−1 optimally arranged, and the right subtree Tk+1
299
8.3 Optimal Binary Search Trees
ak
Optimal BST for ai , . . . ,ak–1
Optimal BST for ak +1 , . . . ,aj
FIGURE 8.8 Binary search tree (BST) with root ak and two optimal binary search subtrees j Tik−1 and Tk+1.
contains keys ak+1, . . . , aj also optimally arranged. (Note how we are taking advantage of the principle of optimality here.) If we count tree levels starting with 1 to make the comparison numbers equal the keys’ levels, the following recurrence relation is obtained: C(i, j ) = min {pk . 1 + i≤k≤j
+
k−1
ps . (level of as in Tik−1 + 1)
s=i j
j
ps . (level of as in Tk+1 + 1)}
s=k+1 k−1
= min { i≤k≤j
ps . level of as in Tik−1 +
s=i
j
j
ps . level of as in Tk+1 +
i≤k≤j
j
ps }
s=i
s=k+1
= min {C(i, k − 1) + C(k + 1, j )} +
j
ps .
s=i
Thus, we have the recurrence C(i, j ) = min {C(i, k − 1) + C(k + 1, j )} + i≤k≤j
j
ps
for 1 ≤ i ≤ j ≤ n.
(8.8)
s=i
We assume in formula (8.8) that C(i, i − 1) = 0 for 1 ≤ i ≤ n + 1, which can be interpreted as the number of comparisons in the empty tree. Note that this formula implies that C(i, i) = pi
for 1 ≤ i ≤ n,
as it should be for a onenode binary search tree containing ai .
300
Dynamic Programming
1
0
1
0
p1 0
i
j
n goal
p2 C [i,j ]
pn n +1
0
FIGURE 8.9 Table of the dynamic programming algorithm for constructing an optimal binary search tree.
The twodimensional table in Figure 8.9 shows the values needed for computing C(i, j ) by formula (8.8): they are in row i and the columns to the left of column j and in column j and the rows below row i. The arrows point to the pairs of entries whose sums are computed in order to ﬁnd the smallest one to be recorded as the value of C(i, j ). This suggests ﬁlling the table along its diagonals, starting with all zeros on the main diagonal and given probabilities pi , 1 ≤ i ≤ n, right above it and moving toward the upper right corner. The algorithm we just sketched computes C(1, n)—the average number of comparisons for successful searches in the optimal binary tree. If we also want to get the optimal tree itself, we need to maintain another twodimensional table to record the value of k for which the minimum in (8.8) is achieved. The table has the same shape as the table in Figure 8.9 and is ﬁlled in the same manner, starting with entries R(i, i) = i for 1 ≤ i ≤ n. When the table is ﬁlled, its entries indicate indices of the roots of the optimal subtrees, which makes it possible to reconstruct an optimal tree for the entire set given. EXAMPLE Let us illustrate the algorithm by applying it to the fourkey set we used at the beginning of this section:
301
8.3 Optimal Binary Search Trees
key probability
A 0.1
B 0.2
C 0.4
D 0.3
The initial tables look like this:
1 2 3 4 5
0 0
1 0.1 0
main table 2 3 0.2 0
0.4 0
0
4
0.3 0
1 2 3 4 5
root table 1 2 3 1 2 3
4
4
Let us compute C(1, 2): & ' k = 1: C(1, 0) + C(2, 2) + 2s=1 ps = 0 + 0.2 + 0.3 = 0.5 C(1, 2) = min k = 2: C(1, 1) + C(3, 2) + 2s=1 ps = 0.1 + 0 + 0.3 = 0.4 = 0.4. Thus, out of two possible binary trees containing the ﬁrst two keys, A and B, the root of the optimal tree has index 2 (i.e., it contains B), and the average number of comparisons in a successful search in this tree is 0.4. We will ask you to ﬁnish the computations in the exercises. You should arrive at the following ﬁnal tables:
1 2 3 4 5
0 0
1 0.1 0
main table 2 3 0.4 1.1 0.2 0.8 0 0.4 0
4 1.7 1.4 1.0 0.3 0
0 1 2 3 4 5
root table 1 2 3 1 2 3 2 3 3
4 3 3 3 4
Thus, the average number of key comparisons in the optimal tree is equal to 1.7. Since R(1, 4) = 3, the root of the optimal tree contains the third key, i.e., C. Its left subtree is made up of keys A and B, and its right subtree contains just key D (why?). To ﬁnd the speciﬁc structure of these subtrees, we ﬁnd ﬁrst their roots by consulting the root table again as follows. Since R(1, 2) = 2, the root of the optimal tree containing A and B is B, with A being its left child (and the root of the onenode tree: R(1, 1) = 1). Since R(4, 4) = 4, the root of this onenode optimal tree is its only key D. Figure 8.10 presents the optimal tree in its entirety.
302
Dynamic Programming
C B
D
A
FIGURE 8.10 Optimal binary search tree for the example.
Here is pseudocode of the dynamic programming algorithm. ALGORITHM OptimalBST(P [1..n]) //Finds an optimal binary search tree by dynamic programming //Input: An array P [1..n] of search probabilities for a sorted list of n keys //Output: Average number of comparisons in successful searches in the // optimal BST and table R of subtrees’ roots in the optimal BST for i ← 1 to n do C[i, i − 1] ← 0 C[i, i] ← P [i] R[i, i] ← i C[n + 1, n] ← 0 for d ← 1 to n − 1 do //diagonal count for i ← 1 to n − d do j ←i+d minval ← ∞ for k ← i to j do if C[i, k − 1] + C[k + 1, j ] < minval minval ← C[i, k − 1] + C[k + 1, j ]; kmin ← k R[i, j ] ← kmin sum ← P [i]; for s ← i + 1 to j do sum ← sum + P [s] C[i, j ] ← minval + sum return C[1, n], R The algorithm’s space efﬁciency is clearly quadratic; the time efﬁciency of this version of the algorithm is cubic (why?). A more careful analysis shows that entries in the root table are always nondecreasing along each row and column. This limits values for R(i, j ) to the range R(i, j − 1), . . . , R(i + 1, j ) and makes it possible to reduce the running time of the algorithm to (n2 ).
8.3 Optimal Binary Search Trees
303
Exercises 8.3 1. Finish the computations started in the section’s example of constructing an optimal binary search tree. 2. a. Why is the time efﬁciency of algorithm OptimalBST cubic? b. Why is the space efﬁciency of algorithm OptimalBST quadratic? 3. Write pseudocode for a lineartime algorithm that generates the optimal binary search tree from the root table. j 4. Devise a way to compute the sums s=i ps , which are used in the dynamic programming algorithm for constructing an optimal binary search tree, in constant time (per sum). 5. True or false: The root of an optimal binary search tree always contains the key with the highest search probability? 6. How would you construct an optimal binary search tree for a set of n keys if all the keys are equally likely to be searched for? What will be the average number of comparisons in a successful search in such a tree if n = 2k ? 7. a. Show that the number of distinct binary search trees b(n) that can be constructed for a set of n orderable keys satisﬁes the recurrence relation b(n) =
n−1
b(k)b(n − 1 − k) for n > 0, b(0) = 1.
k=0
b. It is known that the solution to this recurrence is given by the Catalan numbers. Verify this assertion for n = 1, 2, . . . , 5. c. Find the order of growth of b(n). What implication does the answer to this question have for the exhaustivesearch algorithm for constructing an optimal binary search tree? 8. Design a (n2 ) algorithm for ﬁnding an optimal binary search tree. 9. Generalize the optimal binary search algorithm by taking into account unsuccessful searches. 10. Write pseudocode of a memory function for the optimal binary search tree problem. You may limit your function to ﬁnding the smallest number of key comparisons in a successful search. 11. Matrix chain multiplication Consider the problem of minimizing the total number of multiplications made in computing the product of n matrices A1 . A2 . . . . . An whose dimensions are d0 × d1, d1 × d2 , . . . , dn−1 × dn, respectively. Assume that all intermediate products of two matrices are computed by the bruteforce (deﬁnitionbased) algorithm.
304
Dynamic Programming
a. Give an example of three matrices for which the number of multiplications in (A1 . A2 ) . A3 and A1 . (A2 . A3) differ at least by a factor of 1000. b. How many different ways are there to compute the product of n matrices? c. Design a dynamic programming algorithm for ﬁnding an optimal order of multiplying n matrices.
8.4 Warshall’s and Floyd’s Algorithms In this section, we look at two wellknown algorithms: Warshall’s algorithm for computing the transitive closure of a directed graph and Floyd’s algorithm for the allpairs shortestpaths problem. These algorithms are based on essentially the same idea: exploit a relationship between a problem and its simpler rather than smaller version. Warshall and Floyd published their algorithms without mentioning dynamic programming. Nevertheless, the algorithms certainly have a dynamic programming ﬂavor and have come to be considered applications of this technique.
Warshall’s Algorithm Recall that the adjacency matrix A = {aij } of a directed graph is the boolean matrix that has 1 in its ith row and j th column if and only if there is a directed edge from the ith vertex to the j th vertex. We may also be interested in a matrix containing the information about the existence of directed paths of arbitrary lengths between vertices of a given graph. Such a matrix, called the transitive closure of the digraph, would allow us to determine in constant time whether the j th vertex is reachable from the ith vertex. Here are a few application examples. When a value in a spreadsheet cell is changed, the spreadsheet software must know all the other cells affected by the change. If the spreadsheet is modeled by a digraph whose vertices represent the spreadsheet cells and edges indicate cell dependencies, the transitive closure will provide such information. In software engineering, transitive closure can be used for investigating data ﬂow and control ﬂow dependencies as well as for inheritance testing of objectoriented software. In electronic engineering, it is used for redundancy identiﬁcation and test generation for digital circuits. DEFINITION The transitive closure of a directed graph with n vertices can be deﬁned as the n × n boolean matrix T = {tij }, in which the element in the ith row and the j th column is 1 if there exists a nontrivial path (i.e., directed path of a positive length) from the ith vertex to the j th vertex; otherwise, tij is 0. An example of a digraph, its adjacency matrix, and its transitive closure is given in Figure 8.11. We can generate the transitive closure of a digraph with the help of depthﬁrst search or breadthﬁrst search. Performing either traversal starting at the ith
8.4
a
b A=
c
d
(a)
305
Warshall’s and Floyd’s Algorithms
a b c d
a 0 0 0 1
b 1 0 0 0
c 0 0 0 1
d 0 1 0 0
a b T= c d
(b)
a 1 1 0 1
b 1 1 0 1
c 1 1 0 1
d 1 1 0 1
(c)
FIGURE 8.11 (a) Digraph. (b) Its adjacency matrix. (c) Its transitive closure.
vertex gives the information about the vertices reachable from it and hence the columns that contain 1’s in the ith row of the transitive closure. Thus, doing such a traversal for every vertex as a starting point yields the transitive closure in its entirety. Since this method traverses the same digraph several times, we should hope that a better algorithm can be found. Indeed, such an algorithm exists. It is called Warshall’s algorithm after Stephen Warshall, who discovered it [War62]. It is convenient to assume that the digraph’s vertices and hence the rows and columns of the adjacency matrix are numbered from 1 to n. Warshall’s algorithm constructs the transitive closure through a series of n × n boolean matrices: R (0), . . . , R (k−1), R (k), . . . R (n).
(8.9)
Each of these matrices provides certain information about directed paths in the digraph. Speciﬁcally, the element rij(k) in the ith row and j th column of matrix R (k) (i, j = 1, 2, . . . , n, k = 0, 1, . . . , n) is equal to 1 if and only if there exists a directed path of a positive length from the ith vertex to the j th vertex with each intermediate vertex, if any, numbered not higher than k. Thus, the series starts with R (0), which does not allow any intermediate vertices in its paths; hence, R (0) is nothing other than the adjacency matrix of the digraph. (Recall that the adjacency matrix contains the information about oneedge paths, i.e., paths with no intermediate vertices.) R (1) contains the information about paths that can use the ﬁrst vertex as intermediate; thus, with more freedom, so to speak, it may contain more 1’s than R (0). In general, each subsequent matrix in series (8.9) has one more vertex to use as intermediate for its paths than its predecessor and hence may, but does not have to, contain more 1’s. The last matrix in the series, R (n), reﬂects paths that can use all n vertices of the digraph as intermediate and hence is nothing other than the digraph’s transitive closure. The central point of the algorithm is that we can compute all the elements of each matrix R (k) from its immediate predecessor R (k−1) in series (8.9). Let rij(k), the element in the ith row and j th column of matrix R (k), be equal to 1. This means that there exists a path from the ith vertex vi to the j th vertex vj with each intermediate vertex numbered not higher than k: vi , a list of intermediate vertices each numbered not higher than k, vj . (8.10)
306
Dynamic Programming j
R (k – 1) = k i
k
1 ↑ 0 →
1
j
R (k) = k
1
i
1
k
1
FIGURE 8.12 Rule for changing zeros in Warshall’s algorithm.
Two situations regarding this path are possible. In the ﬁrst, the list of its intermediate vertices does not contain the kth vertex. Then this path from vi to vj has intermediate vertices numbered not higher than k − 1, and therefore rij(k−1) is equal to 1 as well. The second possibility is that path (8.10) does contain the kth vertex vk among the intermediate vertices. Without loss of generality, we may assume that vk occurs only once in that list. (If it is not the case, we can create a new path from vi to vj with this property by simply eliminating all the vertices between the ﬁrst and last occurrences of vk in it.) With this caveat, path (8.10) can be rewritten as follows: vi , vertices numbered ≤ k − 1, vk , vertices numbered ≤ k − 1, vj . The ﬁrst part of this representation means that there exists a path from vi to vk with (k−1) each intermediate vertex numbered not higher than k − 1 (hence, rik = 1), and the second part means that there exists a path from vk to vj with each intermediate (k−1) vertex numbered not higher than k − 1 (hence, rkj = 1).
What we have just proved is that if rij(k) = 1, then either rij(k−1) = 1 or both (k−1) (k−1) rik = 1 and rkj = 1. It is easy to see that the converse of this assertion is also true. Thus, we have the following formula for generating the elements of matrix R (k) from the elements of matrix R (k−1): ) ( (k−1) (k−1) . (8.11) rik and rkj rij(k) = rij(k−1) or
Formula (8.11) is at the heart of Warshall’s algorithm. This formula implies the following rule for generating elements of matrix R (k) from elements of matrix R (k−1), which is particularly convenient for applying Warshall’s algorithm by hand: If an element rij is 1 in R (k−1), it remains 1 in R (k). If an element rij is 0 in R (k−1), it has to be changed to 1 in R (k) if and only if the element in its row i and column k and the element in its column j and row k are both 1’s in R (k−1). This rule is illustrated in Figure 8.12. As an example, the application of Warshall’s algorithm to the digraph in Figure 8.11 is shown in Figure 8.13.
8.4
a
b
c
d
Warshall’s and Floyd’s Algorithms
307
a b R (0) = c d
a 0 0 0 1
b 1 0 0 0
c 0 0 0 1
d 0 1 0 0
1’s reflect the existence of paths with no intermediate vertices (R (0) is just the adjacency matrix); boxed row and column are used for getting R (1).
a b R (1) = c d
a 0 0 0 1
b 1 0 0 1
c 0 0 0 1
d 0 1 0 0
1’s reflect the existence of paths with intermediate vertices numbered not higher than 1, i.e., just vertex a (note a new path from d to b); boxed row and column are used for getting R (2).
a b (2) R = c d
a 0 0 0 1
b 1 0 0 1
c 0 0 0 1
d 1 1 0 1
1’s reflect the existence of paths with intermediate vertices numbered not higher than 2, i.e., a and b (note two new paths); boxed row and column are used for getting R (3).
a b (3) R = c d
a 0 0 0 1
b 1 0 0 1
c 0 0 0 1
d 1 1 0 1
1’s reflect the existence of paths with intermediate vertices numbered not higher than 3, i.e., a, b, and c (no new paths); boxed row and column are used for getting R (4).
a b R (4) = c d
a 1 1 0 1
b 1 1 0 1
c 1 1 0 1
d 1 1 0 1
1’s reflect the existence of paths with intermediate vertices numbered not higher than 4, i.e., a, b, c, and d (note five new paths).
FIGURE 8.13 Application of Warshall’s algorithm to the digraph shown. New 1’s are in bold.
Here is pseudocode of Warshall’s algorithm. ALGORITHM Warshall(A[1..n, 1..n]) //Implements Warshall’s algorithm for computing the transitive closure //Input: The adjacency matrix A of a digraph with n vertices //Output: The transitive closure of the digraph R (0) ← A for k ← 1 to n do for i ← 1 to n do for j ← 1 to n do R (k)[i, j ] ← R (k−1)[i, j ] or (R (k−1)[i, k] and R (k−1)[k, j ]) (n) return R Several observations need to be made about Warshall’s algorithm. First, it is remarkably succinct, is it not? Still, its time efﬁciency is only (n3). In fact, for sparse graphs represented by their adjacency lists, the traversalbased algorithm
308
Dynamic Programming 2
a 3
c
6
b 7
1
(a)
d
a b W = c d
a b 0 ∞ 2 0 ∞ 7 6 ∞
c 3 ∞ 0 ∞
d ∞ ∞ 1 0
a b D = c d
a b c 0 10 3 2 0 5 7 7 0 6 16 9
(b)
d 4 6 1 0
(c)
FIGURE 8.14 (a) Digraph. (b) Its weight matrix. (c) Its distance matrix.
mentioned at the beginning of this section has a better asymptotic efﬁciency than Warshall’s algorithm (why?). We can speed up the above implementation of Warshall’s algorithm for some inputs by restructuring its innermost loop (see Problem 4 in this section’s exercises). Another way to make the algorithm run faster is to treat matrix rows as bit strings and employ the bitwise or operation available in most modern computer languages. As to the space efﬁciency of Warshall’s algorithm, the situation is similar to that of computing a Fibonacci number and some other dynamic programming algorithms. Although we used separate matrices for recording intermediate results of the algorithm, this is, in fact, unnecessary. Problem 3 in this section’s exercises asks you to ﬁnd a way of avoiding this wasteful use of the computer memory. Finally, we shall see below how the underlying idea of Warshall’s algorithm can be applied to the more general problem of ﬁnding lengths of shortest paths in weighted graphs.
Floyd’s Algorithm for the AllPairs ShortestPaths Problem Given a weighted connected graph (undirected or directed), the allpairs shortestpaths problem asks to ﬁnd the distances—i.e., the lengths of the shortest paths— from each vertex to all other vertices. This is one of several variations of the problem involving shortest paths in graphs. Because of its important applications to communications, transportation networks, and operations research, it has been thoroughly studied over the years. Among recent applications of the allpairs shortestpath problem is precomputing distances for motion planning in computer games. It is convenient to record the lengths of shortest paths in an n × n matrix D called the distance matrix: the element dij in the ith row and the j th column of this matrix indicates the length of the shortest path from the ith vertex to the j th vertex. For an example, see Figure 8.14. We can generate the distance matrix with an algorithm that is very similar to Warshall’s algorithm. It is called Floyd’s algorithm after its coinventor Robert W. Floyd.1 It is applicable to both undirected and directed weighted graphs provided 1.
Floyd explicitly referenced Warshall’s paper in presenting his algorithm [Flo62]. Three years earlier, Bernard Roy published essentially the same algorithm in the proceedings of the French Academy of Sciences [Roy59].
8.4
Warshall’s and Floyd’s Algorithms
309
that they do not contain a cycle of a negative length. (The distance between any two vertices in such a cycle can be made arbitrarily small by repeating the cycle enough times.) The algorithm can be enhanced to ﬁnd not only the lengths of the shortest paths for all vertex pairs but also the shortest paths themselves (Problem 10 in this section’s exercises). Floyd’s algorithm computes the distance matrix of a weighted graph with n vertices through a series of n × n matrices: D (0), . . . , D (k−1), D (k), . . . , D (n).
(8.12)
Each of these matrices contains the lengths of shortest paths with certain constraints on the paths considered for the matrix in question. Speciﬁcally, the element dij(k) in the ith row and the j th column of matrix D (k) (i, j = 1, 2, . . . , n, k = 0, 1, . . . , n) is equal to the length of the shortest path among all paths from the ith vertex to the j th vertex with each intermediate vertex, if any, numbered not higher than k. In particular, the series starts with D (0), which does not allow any intermediate vertices in its paths; hence, D (0) is simply the weight matrix of the graph. The last matrix in the series, D (n), contains the lengths of the shortest paths among all paths that can use all n vertices as intermediate and hence is nothing other than the distance matrix being sought. As in Warshall’s algorithm, we can compute all the elements of each matrix D (k) from its immediate predecessor D (k−1) in series (8.12). Let dij(k) be the element in the ith row and the j th column of matrix D (k). This means that dij(k) is equal to the length of the shortest path among all paths from the ith vertex vi to the j th vertex vj with their intermediate vertices numbered not higher than k: vi , a list of intermediate vertices each numbered not higher than k, vj . (8.13) We can partition all such paths into two disjoint subsets: those that do not use the kth vertex vk as intermediate and those that do. Since the paths of the ﬁrst subset have their intermediate vertices numbered not higher than k − 1, the shortest of them is, by deﬁnition of our matrices, of length dij(k−1). What is the length of the shortest path in the second subset? If the graph does not contain a cycle of a negative length, we can limit our attention only to the paths in the second subset that use vertex vk as their intermediate vertex exactly once (because visiting vk more than once can only increase the path’s length). All such paths have the following form: vi , vertices numbered ≤ k − 1, vk , vertices numbered ≤ k − 1, vj . In other words, each of the paths is made up of a path from vi to vk with each intermediate vertex numbered not higher than k − 1 and a path from vk to vj with each intermediate vertex numbered not higher than k − 1. The situation is depicted symbolically in Figure 8.15. Since the length of the shortest path from vi to vk among the paths that use (k−1) intermediate vertices numbered not higher than k − 1 is equal to dik and the length of the shortest path from vk to vj among the paths that use intermediate
310
Dynamic Programming (k –1)
d ij
vj
vi
(k –1)
(k –1)
dik
dkj
vk
FIGURE 8.15 Underlying idea of Floyd’s algorithm.
(k−1) vertices numbered not higher than k − 1 is equal to dkj , the length of the shortest (k−1) (k−1) path among the paths that use the kth vertex is equal to dik + dkj . Taking into account the lengths of the shortest paths in both subsets leads to the following recurrence: (k−1) (k−1) dij(k) = min{dij(k−1), dik + dkj }
for k ≥ 1, dij(0) = wij .
(8.14)
To put it another way, the element in row i and column j of the current distance matrix D (k−1) is replaced by the sum of the elements in the same row i and the column k and in the same column j and the row k if and only if the latter sum is smaller than its current value. The application of Floyd’s algorithm to the graph in Figure 8.14 is illustrated in Figure 8.16. Here is pseudocode of Floyd’s algorithm. It takes advantage of the fact that the next matrix in sequence (8.12) can be written over its predecessor. ALGORITHM Floyd(W [1..n, 1..n]) //Implements Floyd’s algorithm for the allpairs shortestpaths problem //Input: The weight matrix W of a graph with no negativelength cycle //Output: The distance matrix of the shortest paths’ lengths D ← W //is not necessary if W can be overwritten for k ← 1 to n do for i ← 1 to n do for j ← 1 to n do D[i, j ] ← min{D[i, j ], D[i, k] + D[k, j ]} return D Obviously, the time efﬁciency of Floyd’s algorithm is cubic—as is the time efﬁciency of Warshall’s algorithm. In the next chapter, we examine Dijkstra’s algorithm—another method for ﬁnding shortest paths.
8.4 2
a 3 c
6
b 7
1
d
Warshall’s and Floyd’s Algorithms
311
a b D (0) = c d
a 0 2 ∞ 6
b ∞ 0 7 ∞
c 3 ∞ 0 ∞
d ∞ ∞ 1 0
a b D (1) = c d
a 0 2 ∞ 6
b ∞ 0 7 ∞
c 3 5 0 9
d ∞ ∞ 1 0
Lengths of the shortest paths with intermediate vertices numbered not higher than 1, i.e., just a (note two new shortest paths from b to c and from d to c ).
a b D (2) = c d
a 0 2 9 6
b ∞ 0 7 ∞
c 3 5 0 9
d ∞ ∞ 1 0
Lengths of the shortest paths with intermediate vertices numbered not higher than 2, i.e., a and b (note a new shortest path from c to a).
a b D (3) = c d
a 0 2 9 6
b 10 0 7 16
c 3 5 0 9
d 4 6 1 0
a b D (4) = c d
a 0 2 7 6
b c 10 3 0 5 7 0 16 9
d 4 6 1 0
Lengths of the shortest paths with no intermediate vertices (D (0) is simply the weight matrix).
Lengths of the shortest paths with intermediate vertices numbered not higher than 3, i.e., a, b, and c (note four new shortest paths from a to b, from a to d, from b to d, and from d to b). Lengths of the shortest paths with intermediate vertices numbered not higher than 4, i.e., a, b, c, and d (note a new shortest path from c to a).
FIGURE 8.16 Application of Floyd’s algorithm to the digraph shown. Updated elements are shown in bold.
Exercises 8.4 1. Apply Warshall’s algorithm to ﬁnd the transitive closure of the digraph deﬁned by the following adjacency matrix: ⎤ ⎡ 0 1 0 0 ⎢0 0 1 0⎥ ⎥ ⎢ ⎣0 0 0 1⎦ 0 0 0 0 2. a. Prove that the time efﬁciency of Warshall’s algorithm is cubic. b. Explain why the time efﬁciency class of Warshall’s algorithm is inferior to that of the traversalbased algorithm for sparse graphs represented by their adjacency lists.
312
Dynamic Programming
3. Explain how to implement Warshall’s algorithm without using extra memory for storing elements of the algorithm’s intermediate matrices. 4. Explain how to restructure the innermost loop of the algorithm Warshall to make it run faster at least on some inputs. 5. Rewrite pseudocode of Warshall’s algorithm assuming that the matrix rows are represented by bit strings on which the bitwise or operation can be performed. 6. a. Explain how Warshall’s algorithm can be used to determine whether a given digraph is a dag (directed acyclic graph). Is it a good algorithm for this problem? b. Is it a good idea to apply Warshall’s algorithm to ﬁnd the transitive closure of an undirected graph? 7. Solve the allpairs shortestpath problem for the digraph with the following weight matrix: ⎤ ⎡ 0 2 ∞ 1 8 ⎢ 6 0 3 2 ∞⎥ ⎥ ⎢ ⎢∞ ∞ 0 4 ∞⎥ ⎥ ⎢ ⎣∞ ∞ 2 0 3 ⎦ 3 ∞ ∞ ∞ 0 8. Prove that the next matrix in sequence (8.12) of Floyd’s algorithm can be written over its predecessor. 9. Give an example of a graph or a digraph with negative weights for which Floyd’s algorithm does not yield the correct result. 10. Enhance Floyd’s algorithm so that shortest paths themselves, not just their lengths, can be found. 11. Jack Straws In the game of Jack Straws, a number of plastic or wooden “straws” are dumped on the table and players try to remove them one by one without disturbing the other straws. Here, we are only concerned with whether various pairs of straws are connected by a path of touching straws. Given a list of the endpoints for n > 1 straws (as if they were dumped on a large piece of graph paper), determine all the pairs of straws that are connected. Note that touching is connecting, but also that two straws can be connected indirectly via other connected straws. [1994 EastCentral Regionals of the ACM International Collegiate Programming Contest]
SUMMARY Dynamic programming is a technique for solving problems with overlapping subproblems. Typically, these subproblems arise from a recurrence relating a solution to a given problem with solutions to its smaller subproblems of the
Summary
313
same type. Dynamic programming suggests solving each smaller subproblem once and recording the results in a table from which a solution to the original problem can be then obtained. Applicability of dynamic programming to an optimization problem requires the problem to satisfy the principle of optimality: an optimal solution to any of its instances must be made up of optimal solutions to its subinstances. Among many other problems, the changemaking problem with arbitrary coin denominations can be solved by dynamic programming. Solving a knapsack problem by a dynamic programming algorithm exempliﬁes an application of this technique to difﬁcult problems of combinatorial optimization. The memory function technique seeks to combine the strengths of the topdown and bottomup approaches to solving problems with overlapping subproblems. It does this by solving, in the topdown fashion but only once, just the necessary subproblems of a given problem and recording their solutions in a table. Dynamic programming can be used for constructing an optimal binary search tree for a given set of keys and known probabilities of searching for them. Warshall’s algorithm for ﬁnding the transitive closure and Floyd’s algorithm for the allpairs shortestpaths problem are based on the idea that can be interpreted as an application of the dynamic programming technique.
This page intentionally left blank
9 Greedy Technique Greed, for lack of a better word, is good! Greed is right! Greed works! —Michael Douglas, US actor in the role of Gordon Gecko, in the ﬁlm Wall Street, 1987
L
et us revisit the changemaking problem faced, at least subconsciously, by millions of cashiers all over the world: give change for a speciﬁc amount n with the least number of coins of the denominations d1 > d2 > . . . > dm used in that locale. (Here, unlike Section 8.1, we assume that the denominations are ordered in decreasing order.) For example, the widely used coin denominations in the United States are d1 = 25 (quarter), d2 = 10 (dime), d3 = 5 (nickel), and d4 = 1 (penny). How would you give change with coins of these denominations of, say, 48 cents? If you came up with the answer 1 quarter, 2 dimes, and 3 pennies, you followed— consciously or not—a logical strategy of making a sequence of best choices among the currently available alternatives. Indeed, in the ﬁrst step, you could have given one coin of any of the four denominations. “Greedy” thinking leads to giving one quarter because it reduces the remaining amount the most, namely, to 23 cents. In the second step, you had the same coins at your disposal, but you could not give a quarter, because it would have violated the problem’s constraints. So your best selection in this step was one dime, reducing the remaining amount to 13 cents. Giving one more dime left you with 3 cents to be given with three pennies. Is this solution to the instance of the changemaking problem optimal? Yes, it is. In fact, one can prove that the greedy algorithm yields an optimal solution for every positive integer amount with these coin denominations. At the same time, it is easy to give an example of coin denominations that do not yield an optimal solution for some amounts—e.g., d1 = 25, d2 = 10, d3 = 1 and n = 30. The approach applied in the opening paragraph to the changemaking problem is called greedy. Computer scientists consider it a general design technique despite the fact that it is applicable to optimization problems only. The greedy approach suggests constructing a solution through a sequence of steps, each expanding a partially constructed solution obtained so far, until a complete solution 315
316
Greedy Technique
to the problem is reached. On each step—and this is the central point of this technique—the choice made must be: feasible, i.e., it has to satisfy the problem’s constraints locally optimal, i.e., it has to be the best local choice among all feasible choices available on that step irrevocable, i.e., once made, it cannot be changed on subsequent steps of the algorithm These requirements explain the technique’s name: on each step, it suggests a “greedy” grab of the best alternative available in the hope that a sequence of locally optimal choices will yield a (globally) optimal solution to the entire problem. We refrain from a philosophical discussion of whether greed is good or bad. (If you have not seen the movie from which the chapter’s epigraph is taken, its hero did not end up well.) From our algorithmic perspective, the question is whether such a greedy strategy works or not. As we shall see, there are problems for which a sequence of locally optimal choices does yield an optimal solution for every instance of the problem in question. However, there are others for which this is not the case; for such problems, a greedy algorithm can still be of value if we are interested in or have to be satisﬁed with an approximate solution. In the ﬁrst two sections of the chapter, we discuss two classic algorithms for the minimum spanning tree problem: Prim’s algorithm and Kruskal’s algorithm. What is remarkable about these algorithms is the fact that they solve the same problem by applying the greedy approach in two different ways, and both of them always yield an optimal solution. In Section 9.3, we introduce another classic algorithm— Dijkstra’s algorithm for the shortestpath problem in a weighted graph. Section 9.4 is devoted to Huffman trees and their principal application, Huffman codes—an important data compression method that can be interpreted as an application of the greedy technique. Finally, a few examples of approximation algorithms based on the greedy approach are discussed in Section 12.3. As a rule, greedy algorithms are both intuitively appealing and simple. Given an optimization problem, it is usually easy to ﬁgure out how to proceed in a greedy manner, possibly after considering a few small instances of the problem. What is usually more difﬁcult is to prove that a greedy algorithm yields an optimal solution (when it does). One of the common ways to do this is illustrated by the proof given in Section 9.1: using mathematical induction, we show that a partially constructed solution obtained by the greedy algorithm on each iteration can be extended to an optimal solution to the problem. The second way to prove optimality of a greedy algorithm is to show that on each step it does at least as well as any other algorithm could in advancing toward the problem’s goal. Consider, as an example, the following problem: ﬁnd the minimum number of moves needed for a chess knight to go from one corner of a 100 × 100 board to the diagonally opposite corner. (The knight’s moves are Lshaped jumps: two squares horizontally or vertically followed by one square in
Greedy Technique
317
the perpendicular direction.) A greedy solution is clear here: jump as close to the goal as possible on each move. Thus, if its start and ﬁnish squares are (1,1) and (100, 100), respectively, a sequence of 66 moves such as (1, 1) − (3, 2) − (4, 4) − . . . − (97, 97) − (99, 98) − (100, 100) solves the problem. (The number k of twomove advances can be obtained from the equation 1 + 3k = 100.) Why is this a minimummove solution? Because if we measure the distance to the goal by the Manhattan distance, which is the sum of the difference between the row numbers and the difference between the column numbers of two squares in question, the greedy algorithm decreases it by 3 on each move—the best the knight can do. The third way is simply to show that the ﬁnal result obtained by a greedy algorithm is optimal based on the algorithm’s output rather than the way it operates. As an example, consider the problem of placing the maximum number of chips on an 8 × 8 board so that no two chips are placed on the same or adjacent— vertically, horizontally, or diagonally—squares. To follow the prescription of the greedy strategy, we should place each new chip so as to leave as many available squares as possible for next chips. For example, starting with the upper left corner of the board, we will be able to place 16 chips as shown in Figure 9.1a. Why is this solution optimal? To see why, partition the board into sixteen 4 × 4 squares as shown in Figure 9.1b. Obviously, it is impossible to place more than one chip in each of these squares, which implies that the total number of nonadjacent chips on the board cannot exceed 16. As a ﬁnal comment, we should mention that a rather sophisticated theory has been developed behind the greedy technique, which is based on the abstract combinatorial structure called “matroid.” An interested reader can check such books as [Cor09] as well as a variety of Internet resources on the subject.
FIGURE 9.1 (a) Placement of 16 chips on nonadjacent squares. (b) Partition of the board proving impossibility of placing more than 16 chips.
318
Greedy Technique
a
1
b
1
a
2
5
c
3 graph
b 2
d
c
3 w(T1) = 6
a
1
b
5
d
c
a 5
3 w(T2) = 9
d
1
b 2
c
d w(T3) = 8
FIGURE 9.2 Graph and its spanning trees, with T1 being the minimum spanning tree.
9.1 Prim’s Algorithm The following problem arises naturally in many practical situations: given n points, connect them in the cheapest possible way so that there will be a path between every pair of points. It has direct applications to the design of all kinds of networks— including communication, computer, transportation, and electrical—by providing the cheapest way to achieve connectivity. It identiﬁes clusters of points in data sets. It has been used for classiﬁcation purposes in archeology, biology, sociology, and other sciences. It is also helpful for constructing approximate solutions to more difﬁcult problems such the traveling salesman problem (see Section 12.3). We can represent the points given by vertices of a graph, possible connections by the graph’s edges, and the connection costs by the edge weights. Then the question can be posed as the minimum spanning tree problem, deﬁned formally as follows. DEFINITION A spanning tree of an undirected connected graph is its connected acyclic subgraph (i.e., a tree) that contains all the vertices of the graph. If such a graph has weights assigned to its edges, a minimum spanning tree is its spanning tree of the smallest weight, where the weight of a tree is deﬁned as the sum of the weights on all its edges. The minimum spanning tree problem is the problem of ﬁnding a minimum spanning tree for a given weighted connected graph. Figure 9.2 presents a simple example illustrating these notions. If we were to try constructing a minimum spanning tree by exhaustive search, we would face two serious obstacles. First, the number of spanning trees grows exponentially with the graph size (at least for dense graphs). Second, generating all spanning trees for a given graph is not easy; in fact, it is more difﬁcult than ﬁnding a minimum spanning tree for a weighted graph by using one of several efﬁcient algorithms available for this problem. In this section, we outline Prim’s algorithm, which goes back to at least 19571 [Pri57].
1.
Robert Prim rediscovered the algorithm published 27 years earlier by the Czech mathematician ˇ Jarn´ık in a Czech journal. Vojtech
9.1
Prim’s Algorithm
319
Prim’s algorithm constructs a minimum spanning tree through a sequence of expanding subtrees. The initial subtree in such a sequence consists of a single vertex selected arbitrarily from the set V of the graph’s vertices. On each iteration, the algorithm expands the current tree in the greedy manner by simply attaching to it the nearest vertex not in that tree. (By the nearest vertex, we mean a vertex not in the tree connected to a vertex in the tree by an edge of the smallest weight. Ties can be broken arbitrarily.) The algorithm stops after all the graph’s vertices have been included in the tree being constructed. Since the algorithm expands a tree by exactly one vertex on each of its iterations, the total number of such iterations is n − 1, where n is the number of vertices in the graph. The tree generated by the algorithm is obtained as the set of edges used for the tree expansions. Here is pseudocode of this algorithm. ALGORITHM
Prim(G)
//Prim’s algorithm for constructing a minimum spanning tree //Input: A weighted connected graph G = V , E //Output: ET , the set of edges composing a minimum spanning tree of G VT ← {v0} //the set of tree vertices can be initialized with any vertex ET ← ∅ for i ← 1 to V  − 1 do ﬁnd a minimumweight edge e∗ = (v ∗, u∗) among all the edges (v, u) such that v is in VT and u is in V − VT VT ← VT ∪ {u∗} ET ← ET ∪ {e∗} return ET The nature of Prim’s algorithm makes it necessary to provide each vertex not in the current tree with the information about the shortest edge connecting the vertex to a tree vertex. We can provide such information by attaching two labels to a vertex: the name of the nearest tree vertex and the length (the weight) of the corresponding edge. Vertices that are not adjacent to any of the tree vertices can be given the ∞ label indicating their “inﬁnite” distance to the tree vertices and a null label for the name of the nearest tree vertex. (Alternatively, we can split the vertices that are not in the tree into two sets, the “fringe” and the “unseen.” The fringe contains only the vertices that are not in the tree but are adjacent to at least one tree vertex. These are the candidates from which the next tree vertex is selected. The unseen vertices are all the other vertices of the graph, called “unseen” because they are yet to be affected by the algorithm.) * With+ such labels, ﬁnding the next vertex to be added to the current tree T = VT , ET becomes a simple task of ﬁnding a vertex with the smallest distance label in the set V − VT . Ties can be broken arbitrarily. After we have identiﬁed a vertex u∗ to be added to the tree, we need to perform two operations:
320
Greedy Technique
Move u∗ from the set V − VT to the set of tree vertices VT . For each remaining vertex u in V − VT that is connected to u∗ by a shorter edge than the u’s current distance label, update its labels by u∗ and the weight of the edge between u∗ and u, respectively.2 Figure 9.3 demonstrates the application of Prim’s algorithm to a speciﬁc graph. Does Prim’s algorithm always yield a minimum spanning tree? The answer to this question is yes. Let us prove by induction that each of the subtrees Ti , i = 0, . . . , n − 1, generated by Prim’s algorithm is a part (i.e., a subgraph) of some minimum spanning tree. (This immediately implies, of course, that the last tree in the sequence, Tn−1, is a minimum spanning tree itself because it contains all n vertices of the graph.) The basis of the induction is trivial, since T0 consists of a single vertex and hence must be a part of any minimum spanning tree. For the inductive step, let us assume that Ti−1 is part of some minimum spanning tree T . We need to prove that Ti , generated from Ti−1 by Prim’s algorithm, is also a part of a minimum spanning tree. We prove this by contradiction by assuming that no minimum spanning tree of the graph can contain Ti . Let ei = (v, u) be the minimum weight edge from a vertex in Ti−1 to a vertex not in Ti−1 used by Prim’s algorithm to expand Ti−1 to Ti . By our assumption, ei cannot belong to any minimum spanning tree, including T . Therefore, if we add ei to T , a cycle must be formed (Figure 9.4). In addition to edge ei = (v, u), this cycle must contain another edge (v , u ) connecting a vertex v ∈ Ti−1 to a vertex u that is not in Ti−1. (It is possible that v coincides with v or u coincides with u but not both.) If we now delete the edge (v , u ) from this cycle, we will obtain another spanning tree of the entire graph whose weight is less than or equal to the weight of T since the weight of ei is less than or equal to the weight of (v , u ). Hence, this spanning tree is a minimum spanning tree, which contradicts the assumption that no minimum spanning tree contains Ti . This completes the correctness proof of Prim’s algorithm. How efﬁcient is Prim’s algorithm? The answer depends on the data structures chosen for the graph itself and for the priority queue of the set V − VT whose vertex priorities are the distances to the nearest tree vertices. (You may want to take another look at the example in Figure 9.3 to see that the set V − VT indeed operates as a priority queue.) In particular, if a graph is represented by its weight matrix and the priority queue is implemented as an unordered array, the algorithm’s running time will be in (V 2 ). Indeed, on each of the V  − 1 iterations, the array implementing the priority queue is traversed to ﬁnd and delete the minimum and then to update, if necessary, the priorities of the remaining vertices. We can also implement the priority queue as a minheap. A minheap is a mirror image of the heap structure discussed in Section 6.4. (In fact, it can be implemented by constructing a heap after negating all the key values given.) Namely, a minheap is a complete binary tree in which every element is less than or equal 2.
If the implementation with the fringe/unseen split is pursued, all the unseen vertices adjacent to u∗ must also be moved to the fringe.
1
b
3 a
c
4 5
6
4 5
f
d
2 6
Tree vertices a(−, −)
e
8
Remaining vertices
Illustration
b(a, 3) c(−, ∞) d(−, ∞) e(a, 6) f(a, 5)
1
b
3 5
a
c
4
6
4 5
f
d
2 6
b(a, 3)
c(b, 1) d(−, ∞) e(a, 6) f(b, 4)
1
b
3
c
4 5
a
8
e
6
4 5
f
d
2 6
c(b, 1)
d(c, 6) e(a, 6) f(b, 4)
1
b
3
c
4 5
a
8
e
6
4 5
f
d
2 6
f(b, 4)
d(f, 5) e(f, 2)
1
b
3
c
4 5
a
8
e
6
4 5
f
d
2 6
e(f, 2)
d(f, 5)
1
b
3 a
8
e
c
4 5
6
4 5
f
d
2 6
e
8
d(f, 5) FIGURE 9.3 Application of Prim’s algorithm. The parenthesized labels of a vertex in the middle column indicate the nearest tree vertex and edge weight; selected vertices and edges are shown in bold.
322
Greedy Technique
u′
v′
v
ei
u
Ti –1
FIGURE 9.4 Correctness proof of Prim’s algorithm.
to its children. All the principal properties of heaps remain valid for minheaps, with some obvious modiﬁcations. For example, the root of a minheap contains the smallest rather than the largest element. Deletion of the smallest element from and insertion of a new element into a minheap of size n are O(log n) operations, and so is the operation of changing an element’s priority (see Problem 15 in this section’s exercises). If a graph is represented by its adjacency lists and the priority queue is implemented as a minheap, the running time of the algorithm is in O(E log V ). This is because the algorithm performs V  − 1 deletions of the smallest element and makes E veriﬁcations and, possibly, changes of an element’s priority in a minheap of size not exceeding V . Each of these operations, as noted earlier, is a O(log V ) operation. Hence, the running time of this implementation of Prim’s algorithm is in (V  − 1 + E)O(log V ) = O(E log V ) because, in a connected graph, V  − 1 ≤ E. In the next section, you will ﬁnd another greedy algorithm for the minimum spanning tree problem, which is “greedy” in a manner different from that of Prim’s algorithm.
Exercises 9.1 1. Write pseudocode of the greedy algorithm for the changemaking problem, with an amount n and coin denominations d1 > d2 > . . . > dm as its input. What is the time efﬁciency class of your algorithm? 2. Design a greedy algorithm for the assignment problem (see Section 3.4). Does your greedy algorithm always yield an optimal solution? 3. Job scheduling Consider the problem of scheduling n jobs of known durations t1, t2 , . . . , tn for execution by a single processor. The jobs can be executed in any order, one job at a time. You want to ﬁnd a schedule that minimizes
9.1
Prim’s Algorithm
323
the total time spent by all the jobs in the system. (The time spent by one job in the system is the sum of the time spent by this job in waiting plus the time spent on its execution.) Design a greedy algorithm for this problem. Does the greedy algorithm always yield an optimal solution? 4. Compatible intervals Given n open intervals (a1, b1), (a2 , b2 ), . . . , (an, bn) on the real line, each representing start and end times of some activity requiring the same resource, the task is to ﬁnd the largest number of these intervals so that no two of them overlap. Investigate the three greedy algorithms based on a. earliest start ﬁrst. b. shortest duration ﬁrst. c. earliest ﬁnish ﬁrst. For each of the three algorithms, either prove that the algorithm always yields an optimal solution or give a counterexample showing this not to be the case. 5. Bridge crossing revisited Consider the generalization of the bridge crossing puzzle (Problem 2 in Exercises 1.2) in which we have n > 1 people whose bridge crossing times are t1, t2 , . . . , tn. All the other conditions of the problem remain the same: at most two people at a time can cross the bridge (and they move with the speed of the slower of the two) and they must carry with them the only ﬂashlight the group has. Design a greedy algorithm for this problem and ﬁnd how long it will take to cross the bridge by using this algorithm. Does your algorithm yield a minimum crossing time for every instance of the problem? If it does—prove it; if it does not—ﬁnd an instance with the smallest number of people for which this happens. 6. Averaging down There are n > 1 identical vessels, one of them with W pints of water and the others empty. You are allowed to perform the following operation: take two of the vessels and split the total amount of water in them equally between them. The object is to achieve a minimum amount of water in the vessel containing all the water in the initial set up by a sequence of such operations. What is the best way to do this? 7. Rumor spreading There are n people, each in possession of a different rumor. They want to share all the rumors with each other by sending electronic messages. Assume that a sender includes all the rumors he or she knows at the time the message is sent and that a message may only have one addressee. Design a greedy algorithm that always yields the minimum number of messages they need to send to guarantee that every one of them gets all the rumors. 8. Bachet’s problem of weights Find an optimal set of n weights {w1, w2 , . . . , wn} so that it would be possible to weigh on a balance scale any integer load in the largest possible range from 1 to W , provided a. weights can be put only on the free cup of the scale.
324
Greedy Technique
b. weights can be put on both cups of the scale. 9. a. Apply Prim’s algorithm to the following graph. Include in the priority queue all the vertices not already in the tree. 5
a
2 4
7 c
e
b
3
6
5 d
4
b. Apply Prim’s algorithm to the following graph. Include in the priority queue only the fringe vertices (the vertices not in the current tree which are adjacent to at least one tree vertex). 3
a
5
4 2
c
b
1
d
3
g
k
5 3
i
7
6
f
4 6
h
2
e
5
4
6
3
5 8
j
9
l
10. The notion of a minimum spanning tree is applicable to a connected weighted graph. Do we have to check a graph’s connectivity before applying Prim’s algorithm, or can the algorithm do it by itself? 11. Does Prim’s algorithm always work correctly on graphs with negative edge weights? 12. Let T be a minimum spanning tree of graph G obtained by Prim’s algorithm. Let Gnew be a graph obtained by adding to G a new vertex and some edges, with weights, connecting the new vertex to some vertices in G. Can we construct a minimum spanning tree of Gnew by adding one of the new edges to T ? If you answer yes, explain how; if you answer no, explain why not. 13. How can one use Prim’s algorithm to ﬁnd a spanning tree of a connected graph with no weights on its edges? Is it a good algorithm for this problem? 14. Prove that any weighted connected graph with distinct weights has exactly one minimum spanning tree. 15. Outline an efﬁcient algorithm for changing an element’s value in a minheap. What is the time efﬁciency of your algorithm?
9.2 Kruskal’s Algorithm
325
9.2 Kruskal’s Algorithm In the previous section, we considered the greedy algorithm that “grows” a minimum spanning tree through a greedy inclusion of the nearest vertex to the vertices already in the tree. Remarkably, there is another greedy algorithm for the minimum spanning tree problem that also always yields an optimal solution. It is named Kruskal’s algorithm after Joseph Kruskal, who discovered this algorithm when he was a secondyear graduate student [Kru56]. Kruskal’s algorithm looks at a minimum spanning tree of a weighted connected graph G = V , E as an acyclic subgraph with V  − 1 edges for which the sum of the edge weights is the smallest. (It is not difﬁcult to prove that such a subgraph must be a tree.) Consequently, the algorithm constructs a minimum spanning tree as an expanding sequence of subgraphs that are always acyclic but are not necessarily connected on the intermediate stages of the algorithm. The algorithm begins by sorting the graph’s edges in nondecreasing order of their weights. Then, starting with the empty subgraph, it scans this sorted list, adding the next edge on the list to the current subgraph if such an inclusion does not create a cycle and simply skipping the edge otherwise. ALGORITHM Kruskal(G) //Kruskal’s algorithm for constructing a minimum spanning tree //Input: A weighted connected graph G = V , E //Output: ET , the set of edges composing a minimum spanning tree of G sort E in nondecreasing order of the edge weights w(ei1) ≤ . . . ≤ w(eiE ) //initialize the set of tree edges and its size ET ← ∅; ecounter ← 0 k←0 //initialize the number of processed edges while ecounter < V  − 1 do k←k+1 if ET ∪ {eik } is acyclic ET ← ET ∪ {eik }; ecounter ← ecounter + 1 return ET The correctness of Kruskal’s algorithm can be proved by repeating the essential steps of the proof of Prim’s algorithm given in the previous section. The fact that ET is actually a tree in Prim’s algorithm but generally just an acyclic subgraph in Kruskal’s algorithm turns out to be an obstacle that can be overcome. Figure 9.5 demonstrates the application of Kruskal’s algorithm to the same graph we used for illustrating Prim’s algorithm in Section 9.1. As you trace the algorithm’s operations, note the disconnectedness of some of the intermediate subgraphs. Applying Prim’s and Kruskal’s algorithms to the same small graph by hand may create the impression that the latter is simpler than the former. This impression is wrong because, on each of its iterations, Kruskal’s algorithm has to check whether the addition of the next edge to the edges already selected would create a
1
b
3 a
c
4 5
6
4 5
f
d
2 6
Tree edges
e
8
Sorted list of edges
Illustration
bc ef ab bf cf af df ae cd de 1 2 3 4 4 5 5 6 6 8
1
b
3 5
a
c
4
6
4 5
f
d
2 6
bc 1
bc ef ab bf cf af df ae cd de 1 2 3 4 4 5 5 6 6 8
1
b
3
c
4 5
a
8
e
6
4 5
f
d
2 6
ef 2
bc ef ab bf cf af df ae cd de 1 2 3 4 4 5 5 6 6 8
1
b
3
c
4 5
a
8
e
6
4 5
f
d
2 6
ab 3
bc ef ab bf cf af df ae cd de 1 2 3 4 4 5 5 6 6 8
1
b
3
c
4 5
a
8
e
6
4 5
f
d
2 6
bf 4
bc ef ab bf cf af df ae cd de 1 2 3 4 4 5 5 6 6 8
1
b
3 a
8
e
c
4 5
6
4 5
f
d
2 6
e
8
df 5 FIGURE 9.5 Application of Kruskal’s algorithm. Selected edges are shown in bold.
327
9.2 Kruskal’s Algorithm
v
v
u
u
(a)
(b)
FIGURE 9.6 New edge connecting two vertices may (a) or may not (b) create a cycle.
cycle. It is not difﬁcult to see that a new cycle is created if and only if the new edge connects two vertices already connected by a path, i.e., if and only if the two vertices belong to the same connected component (Figure 9.6). Note also that each connected component of a subgraph generated by Kruskal’s algorithm is a tree because it has no cycles. In view of these observations, it is convenient to use a slightly different interpretation of Kruskal’s algorithm. We can consider the algorithm’s operations as a progression through a series of forests containing all the vertices of a given graph and some of its edges. The initial forest consists of V  trivial trees, each comprising a single vertex of the graph. The ﬁnal forest consists of a single tree, which is a minimum spanning tree of the graph. On each iteration, the algorithm takes the next edge (u, v) from the sorted list of the graph’s edges, ﬁnds the trees containing the vertices u and v, and, if these trees are not the same, unites them in a larger tree by adding the edge (u, v). Fortunately, there are efﬁcient algorithms for doing so, including the crucial check for whether two vertices belong to the same tree. They are called unionﬁnd algorithms. We discuss them in the following subsection. With an efﬁcient unionﬁnd algorithm, the running time of Kruskal’s algorithm will be dominated by the time needed for sorting the edge weights of a given graph. Hence, with an efﬁcient sorting algorithm, the time efﬁciency of Kruskal’s algorithm will be in O(E log E).
Disjoint Subsets and UnionFind Algorithms Kruskal’s algorithm is one of a number of applications that require a dynamic partition of some n element set S into a collection of disjoint subsets S1, S2 , . . . , Sk . After being initialized as a collection of n oneelement subsets, each containing a different element of S, the collection is subjected to a sequence of intermixed union and ﬁnd operations. (Note that the number of union operations in any such sequence must be bounded above by n − 1 because each union increases a subset’s size at least by 1 and there are only n elements in the entire set S.) Thus, we are
328
Greedy Technique
dealing here with an abstract data type of a collection of disjoint subsets of a ﬁnite set with the following operations: makeset(x) creates a oneelement set {x}. It is assumed that this operation can be applied to each of the elements of set S only once. ﬁnd(x) returns a subset containing x. union(x, y) constructs the union of the disjoint subsets Sx and Sy containing x and y, respectively, and adds it to the collection to replace Sx and Sy , which are deleted from it. For example, let S = {1, 2, 3, 4, 5, 6}. Then makeset(i) creates the set {i} and applying this operation six times initializes the structure to the collection of six singleton sets: {1}, {2}, {3}, {4}, {5}, {6}. Performing union(1, 4) and union(5, 2) yields {1, 4}, {5, 2}, {3}, {6}, and, if followed by union(4, 5) and then by union(3, 6), we end up with the disjoint subsets {1, 4, 5, 2}, {3, 6}. Most implementations of this abstract data type use one element from each of the disjoint subsets in a collection as that subset’s representative. Some implementations do not impose any speciﬁc constraints on such a representative; others do so by requiring, say, the smallest element of each subset to be used as the subset’s representative. Also, it is usually assumed that set elements are (or can be mapped into) integers. There are two principal alternatives for implementing this data structure. The ﬁrst one, called the quick ﬁnd, optimizes the time efﬁciency of the ﬁnd operation; the second one, called the quick union, optimizes the union operation. The quick ﬁnd uses an array indexed by the elements of the underlying set S; the array’s values indicate the representatives of the subsets containing those elements. Each subset is implemented as a linked list whose header contains the pointers to the ﬁrst and last elements of the list along with the number of elements in the list (see Figure 9.7 for an example). Under this scheme, the implementation of makeset(x) requires assigning the corresponding element in the representative array to x and initializing the corresponding linked list to a single node with the x value. The time efﬁciency of this operation is obviously in (1), and hence the initialization of n singleton subsets is in (n). The efﬁciency of ﬁnd(x) is also in (1): all we need to do is to retrieve the x’s representative in the representative array. Executing union(x, y) takes longer. A straightforward solution would simply append the y’s list to the end of the x’s list, update the information about their representative for all the elements in the
329
9.2 Kruskal’s Algorithm
subset representatives element index
representative
1 2 3 4 5 6
1 1 3 1 1 3
size last first list 1
4
list 2
0
list 3
2
list 4
0
null null
list 5
0
null null
list 6
0
null null
1
4
3
6
5
2
null
null null
null
FIGURE 9.7 Linkedlist representation of subsets {1, 4, 5, 2} and {3, 6} obtained by quick ﬁnd after performing union(1, 4), union(5, 2), union(4, 5), and union(3, 6). The lists of size 0 are considered deleted from the collection.
y list, and then delete the y’s list from the collection. It is easy to verify, however, that with this algorithm the sequence of union operations union(2, 1), union(3, 2), . . . , union(i + 1, i), . . . , union(n, n − 1) runs in (n2 ) time, which is slow compared with several known alternatives. A simple way to improve the overall efﬁciency of a sequence of union operations is to always append the shorter of the two lists to the longer one, with ties broken arbitrarily. Of course, the size of each list is assumed to be available by, say, storing the number of elements in the list’s header. This modiﬁcation is called the
330
Greedy Technique
1 4
1
3
5
3
2
2
6
(a)
(b)
5
6
4
FIGURE 9.8 (a) Forest representation of subsets {1, 4, 5, 2} and {3, 6} used by quick union. (b) Result of union(5, 6).
union by size. Though it does not improve the worstcase efﬁciency of a single application of the union operation (it is still in (n)), the worstcase running time of any legitimate sequence of unionbysize operations turns out to be in O(n log n).3 Here is a proof of this assertion. Let ai be an element of set S whose disjoint subsets we manipulate, and let Ai be the number of times ai ’s representative is updated in a sequence of unionbysize operations. How large can Ai get if set S has n elements? Each time ai ’s representative is updated, ai must be in a smaller subset involved in computing the union whose size will be at least twice as large as the size of the subset containing ai . Hence, when ai ’s representative is updated for the ﬁrst time, the resulting set will have at least two elements; when it is updated for the second time, the resulting set will have at least four elements; and, in general, if it is updated Ai times, the resulting set will have at least 2Ai elements. Since the entire set S has n elements, 2Ai ≤ n and hence Ai ≤ log2 n. Therefore, the total number of possible updates of the representatives for all n elements in S will not exceed n log2 n. Thus, for union by size, the time efﬁciency of a sequence of at most n − 1 unions and m ﬁnds is in O(n log n + m). The quick union—the second principal alternative for implementing disjoint subsets—represents each subset by a rooted tree. The nodes of the tree contain the subset’s elements (one per node), with the root’s element considered the subset’s representative; the tree’s edges are directed from children to their parents (Figure 9.8). In addition, a mapping of the set elements to their tree nodes— implemented, say, as an array of pointers—is maintained. This mapping is not shown in Figure 9.8 for the sake of simplicity. For this implementation, makeset(x) requires the creation of a singlenode tree, which is a (1) operation; hence, the initialization of n singleton subsets is in (n). A union(x, y) is implemented by attaching the root of the y’s tree to the root of the x’s tree (and deleting the y’s tree from the collection by making the pointer to its root null). The time efﬁciency of this operation is clearly (1). A ﬁnd(x) is
3.
This is a speciﬁc example of the usefulness of the amortized efﬁciency we mentioned back in Chapter 2.
331
9.2 Kruskal’s Algorithm
x T1
T1 T2
T2
x
T3
T4
T3 T4
FIGURE 9.9 Path compression.
performed by following the pointer chain from the node containing x to the tree’s root whose element is returned as the subset’s representative. Accordingly, the time efﬁciency of a single ﬁnd operation is in O(n) because a tree representing a subset can degenerate into a linked list with n nodes. This time bound can be improved. The straightforward way for doing so is to always perform a union operation by attaching a smaller tree to the root of a larger one, with ties broken arbitrarily. The size of a tree can be measured either by the number of nodes (this version is called union by size) or by its height (this version is called union by rank). Of course, these options require storing, for each node of the tree, either the number of node descendants or the height of the subtree rooted at that node, respectively. One can easily prove that in either case the height of the tree will be logarithmic, making it possible to execute each ﬁnd in O(log n) time. Thus, for quick union, the time efﬁciency of a sequence of at most n − 1 unions and m ﬁnds is in O(n + m log n). In fact, an even better efﬁciency can be obtained by combining either variety of quick union with path compression. This modiﬁcation makes every node encountered during the execution of a ﬁnd operation point to the tree’s root (Figure 9.9). According to a quite sophisticated analysis that goes beyond the level of this book (see [Tar84]), this and similar techniques improve the efﬁciency of a sequence of at most n − 1 unions and m ﬁnds to only slightly worse than linear.
Exercises 9.2 1. Apply Kruskal’s algorithm to ﬁnd a minimum spanning tree of the following graphs. a. 1 b
c
5
6 3
a
6
4 d
2
e
332
Greedy Technique
b.
3
a
5
4 2
c
b
1
d
4 3
k
5 3
i
7
6
f
4 6
h
2
e
5
g
6
3
5 8
j
9
l
2. Indicate whether the following statements are true or false: a. If e is a minimumweight edge in a connected weighted graph, it must be among edges of at least one minimum spanning tree of the graph. b. If e is a minimumweight edge in a connected weighted graph, it must be among edges of each minimum spanning tree of the graph. c. If edge weights of a connected weighted graph are all distinct, the graph must have exactly one minimum spanning tree. d. If edge weights of a connected weighted graph are not all distinct, the graph must have more than one minimum spanning tree. 3. What changes, if any, need to be made in algorithm Kruskal to make it ﬁnd a minimum spanning forest for an arbitrary graph? (A minimum spanning forest is a forest whose trees are minimum spanning trees of the graph’s connected components.) 4. Does Kruskal’s algorithm work correctly on graphs that have negative edge weights? 5. Design an algorithm for ﬁnding a maximum spanning tree—a spanning tree with the largest possible edge weight—of a weighted connected graph. 6. Rewrite pseudocode of Kruskal’s algorithm in terms of the operations of the disjoint subsets’ ADT. 7. Prove the correctness of Kruskal’s algorithm. 8. Prove that the time efﬁciency of ﬁnd(x) is in O(log n) for the unionbysize version of quick union. 9. Find at least two Web sites with animations of Kruskal’s and Prim’s algorithms. Discuss their merits and demerits. 10. Design and conduct an experiment to empirically compare the efﬁciencies of Prim’s and Kruskal’s algorithms on random graphs of different sizes and densities.
9.3
Dijkstra’s Algorithm
333
11. Steiner tree Four villages are located at the vertices of a unit square in the Euclidean plane. You are asked to connect them by the shortest network of roads so that there is a path between every pair of the villages along those roads. Find such a network. 12. Write a program generating a random maze based on a. Prim’s algorithm. b. Kruskal’s algorithm.
9.3 Dijkstra’s Algorithm In this section, we consider the singlesource shortestpaths problem: for a given vertex called the source in a weighted connected graph, ﬁnd shortest paths to all its other vertices. It is important to stress that we are not interested here in a single shortest path that starts at the source and visits all the other vertices. This would have been a much more difﬁcult problem (actually, a version of the traveling salesman problem introduced in Section 3.4 and discussed again later in the book). The singlesource shortestpaths problem asks for a family of paths, each leading from the source to a different vertex in the graph, though some paths may, of course, have edges in common. A variety of practical applications of the shortestpaths problem have made the problem a very popular object of study. The obvious but probably most widely used applications are transportation planning and packet routing in communication networks, including the Internet. Multitudes of less obvious applications include ﬁnding shortest paths in social networks, speech recognition, document formatting, robotics, compilers, and airline crew scheduling. In the world of entertainment, one can mention pathﬁnding in video games and ﬁnding best solutions to puzzles using their statespace graphs (see Section 6.6 for a very simple example of the latter). There are several wellknown algorithms for ﬁnding shortest paths, including Floyd’s algorithm for the more general allpairs shortestpaths problem discussed in Chapter 8. Here, we consider the bestknown algorithm for the singlesource shortestpaths problem, called Dijkstra’s algorithm.4 This algorithm is applicable to undirected and directed graphs with nonnegative weights only. Since in most applications this condition is satisﬁed, the limitation has not impaired the popularity of Dijkstra’s algorithm. Dijkstra’s algorithm ﬁnds the shortest paths to a graph’s vertices in order of their distance from a given source. First, it ﬁnds the shortest path from the source 4.
Edsger W. Dijkstra (1930–2002), a noted Dutch pioneer of the science and industry of computing, discovered this algorithm in the mid1950s. Dijkstra said about his algorithm: “This was the ﬁrst graph problem I ever posed myself and solved. The amazing thing was that I didn’t publish it. It was not amazing at the time. At the time, algorithms were hardly considered a scientiﬁc topic.”
334
Greedy Technique
u* v*
v0
FIGURE 9.10 Idea of Dijkstra’s algorithm. The subtree of the shortest paths already found is shown in bold. The next nearest to the source v0 vertex, u∗, is selected by comparing the lengths of the subtree’s paths increased by the distances to vertices adjacent to the subtree’s vertices.
to a vertex nearest to it, then to a second nearest, and so on. In general, before its ith iteration commences, the algorithm has already identiﬁed the shortest paths to i − 1 other vertices nearest to the source. These vertices, the source, and the edges of the shortest paths leading to them from the source form a subtree Ti of the given graph (Figure 9.10). Since all the edge weights are nonnegative, the next vertex nearest to the source can be found among the vertices adjacent to the vertices of Ti . The set of vertices adjacent to the vertices in Ti can be referred to as “fringe vertices”; they are the candidates from which Dijkstra’s algorithm selects the next vertex nearest to the source. (Actually, all the other vertices can be treated as fringe vertices connected to tree vertices by edges of inﬁnitely large weights.) To identify the ith nearest vertex, the algorithm computes, for every fringe vertex u, the sum of the distance to the nearest tree vertex v (given by the weight of the edge (v, u)) and the length dv of the shortest path from the source to v (previously determined by the algorithm) and then selects the vertex with the smallest such sum. The fact that it sufﬁces to compare the lengths of such special paths is the central insight of Dijkstra’s algorithm. To facilitate the algorithm’s operations, we label each vertex with two labels. The numeric label d indicates the length of the shortest path from the source to this vertex found by the algorithm so far; when a vertex is added to the tree, d indicates the length of the shortest path from the source to that vertex. The other label indicates the name of the nexttolast vertex on such a path, i.e., the parent of the vertex in the tree being constructed. (It can be left unspeciﬁed for the source s and vertices that are adjacent to none of the current tree vertices.) With such labeling, ﬁnding the next nearest vertex u∗ becomes a simple task of ﬁnding a fringe vertex with the smallest d value. Ties can be broken arbitrarily. After we have identiﬁed a vertex u∗ to be added to the tree, we need to perform two operations:
9.3
Dijkstra’s Algorithm
335
Move u∗ from the fringe to the set of tree vertices. For each remaining fringe vertex u that is connected to u∗ by an edge of weight w(u∗, u) such that du∗ + w(u∗, u) < du, update the labels of u by u∗ and du∗ + w(u∗, u), respectively. Figure 9.11 demonstrates the application of Dijkstra’s algorithm to a speciﬁc graph. The labeling and mechanics of Dijkstra’s algorithm are quite similar to those used by Prim’s algorithm (see Section 9.1). Both of them construct an expanding subtree of vertices by selecting the next vertex from the priority queue of the remaining vertices. It is important not to mix them up, however. They solve different problems and therefore operate with priorities computed in a different manner: Dijkstra’s algorithm compares path lengths and therefore must add edge weights, while Prim’s algorithm compares the edge weights as given. Now we can give pseudocode of Dijkstra’s algorithm. It is spelled out— in more detail than Prim’s algorithm was in Section 9.1—in terms of explicit operations on two sets of labeled vertices: the set VT of vertices for which a shortest path has already been found and the priority queue Q of the fringe vertices. (Note that in the following pseudocode, VT contains a given source vertex and the fringe contains the vertices adjacent to it after iteration 0 is completed.) ALGORITHM
Dijkstra(G, s)
//Dijkstra’s algorithm for singlesource shortest paths //Input: A weighted connected graph G = V , E with nonnegative weights // and its vertex s //Output: The length dv of a shortest path from s to v // and its penultimate vertex pv for every vertex v in V Initialize(Q) //initialize priority queue to empty for every vertex v in V dv ← ∞; pv ← null Insert(Q, v, dv ) //initialize vertex priority in the priority queue ds ← 0; Decrease(Q, s, ds ) //update priority of s with ds VT ← ∅ for i ← 0 to V  − 1 do u∗ ← DeleteMin(Q) //delete the minimum priority element VT ← VT ∪ {u∗} for every vertex u in V − VT that is adjacent to u∗ do if du∗ + w(u∗, u) < du du ← du∗ + w(u∗, u); pu ← u∗ Decrease(Q, u, du) The time efﬁciency of Dijkstra’s algorithm depends on the data structures used for implementing the priority queue and for representing an input graph itself. For the reasons explained in the analysis of Prim’s algorithm in Section 9.1, it is
336
Greedy Technique 4
b 3
a
Tree vertices a(−, 0)
2
c
d
7
6
5
e
4
Remaining vertices
Illustration
b(a, 3) c(−, ∞) d(a, 7) e(−, ∞)
a
b(a, 3)
2
c(b, 3 + 4) d(b, 3 + 2) e(−, ∞)
2
c(b, 7) e(d, 5 + 4) 3 a
c(b, 7)
3 a
c
c
6
5 d
e
4
4
7
6
5
2
e
4
d
b
6
5
2 7
e(d, 9)
c
4
b
e
4
d
7
6
5
4
b
a
c
d
7
3
d(b, 5)
4
b
3
4
e
e(d, 9)
The shortest paths (identiﬁed by following nonnumeric labels backward from a destination vertex in the left column to the source) and their lengths (given by numeric labels of the tree vertices) are as follows: from a to b :
a−b
of length 3
from a to d :
a−b−d
of length 5
from a to c :
a−b−c
of length 7
from a to e :
a−b−d −e
of length 9
FIGURE 9.11 Application of Dijkstra’s algorithm. The next closest vertex is shown in bold.
9.3
Dijkstra’s Algorithm
337
in (V 2 ) for graphs represented by their weight matrix and the priority queue implemented as an unordered array. For graphs represented by their adjacency lists and the priority queue implemented as a minheap, it is in O(E log V ). A still better upper bound can be achieved for both Prim’s and Dijkstra’s algorithms if the priority queue is implemented using a sophisticated data structure called the Fibonacci heap (e.g., [Cor09]). However, its complexity and a considerable overhead make such an improvement primarily of theoretical value.
Exercises 9.3 1. Explain what adjustments if any need to be made in Dijkstra’s algorithm and/or in an underlying graph to solve the following problems. a. Solve the singlesource shortestpaths problem for directed weighted graphs. b. Find a shortest path between two given vertices of a weighted graph or digraph. (This variation is called the singlepair shortestpath problem.) c. Find the shortest paths to a given vertex from each other vertex of a weighted graph or digraph. (This variation is called the singledestination shortestpaths problem.) d. Solve the singlesource shortestpaths problem in a graph with nonnegative numbers assigned to its vertices (and the length of a path deﬁned as the sum of the vertex numbers on the path). 2. Solve the following instances of the singlesource shortestpaths problem with vertex a as the source: a. 4 b
c
3
6 2
a
d
7
b.
3
a
5
b
4 3
k
5 3
i
7
6
f
4 6
h
2
e
5
g
6
3 1
d
e
4
4 2
c
5
5 8
l
j
9
338
Greedy Technique
3. Give a counterexample that shows that Dijkstra’s algorithm may not work for a weighted connected graph with negative weights. 4. Let T be a tree constructed by Dijkstra’s algorithm in the process of solving the singlesource shortestpaths problem for a weighted connected graph G. a. True or false: T is a spanning tree of G? b. True or false: T is a minimum spanning tree of G? 5. Write pseudocode for a simpler version of Dijkstra’s algorithm that ﬁnds only the distances (i.e., the lengths of shortest paths but not shortest paths themselves) from a given vertex to all other vertices of a graph represented by its weight matrix. 6. Prove the correctness of Dijkstra’s algorithm for graphs with positive weights. 7. Design a lineartime algorithm for solving the singlesource shortestpaths problem for dags (directed acyclic graphs) represented by their adjacency lists. 8. Explain how the minimumsum descent problem (Problem 8 in Exercises 8.1) can be solved by Dijkstra’s algorithm. 9. Shortestpath modeling Assume you have a model of a weighted connected graph made of balls (representing the vertices) connected by strings of appropriate lengths (representing the edges). a. Describe how you can solve the singlepair shortestpath problem with this model. b. Describe how you can solve the singlesource shortestpaths problem with this model. 10. Revisit the exercise from Section 1.3 about determining the best route for a subway passenger to take from one designated station to another in a welldeveloped subway system like those in Washington, DC, or London, UK. Write a program for this task.
9.4 Huffman Trees and Codes Suppose we have to encode a text that comprises symbols from some nsymbol alphabet by assigning to each of the text’s symbols some sequence of bits called the codeword. For example, we can use a ﬁxedlength encoding that assigns to each symbol a bit string of the same length m (m ≥ log2 n). This is exactly what the standard ASCII code does. One way of getting a coding scheme that yields a shorter bit string on the average is based on the old idea of assigning shorter codewords to more frequent symbols and longer codewords to less frequent symbols. This idea was used, in particular, in the telegraph code invented in the mid19th century by Samuel Morse. In that code, frequent letters such as e (.) and a (.−) are assigned short sequences of dots and dashes while infrequent letters such as q (− − .−) and z (− − ..) have longer ones.
339
9.4 Huffman Trees and Codes
Variablelength encoding, which assigns codewords of different lengths to different symbols, introduces a problem that ﬁxedlength encoding does not have. Namely, how can we tell how many bits of an encoded text represent the ﬁrst (or, more generally, the ith) symbol? To avoid this complication, we can limit ourselves to the socalled preﬁxfree (or simply preﬁx) codes. In a preﬁx code, no codeword is a preﬁx of a codeword of another symbol. Hence, with such an encoding, we can simply scan a bit string until we get the ﬁrst group of bits that is a codeword for some symbol, replace these bits by this symbol, and repeat this operation until the bit string’s end is reached. If we want to create a binary preﬁx code for some alphabet, it is natural to associate the alphabet’s symbols with leaves of a binary tree in which all the left edges are labeled by 0 and all the right edges are labeled by 1. The codeword of a symbol can then be obtained by recording the labels on the simple path from the root to the symbol’s leaf. Since there is no simple path to a leaf that continues to another leaf, no codeword can be a preﬁx of another codeword; hence, any such tree yields a preﬁx code. Among the many trees that can be constructed in this manner for a given alphabet with known frequencies of the symbol occurrences, how can we construct a tree that would assign shorter bit strings to highfrequency symbols and longer ones to lowfrequency symbols? It can be done by the following greedy algorithm, invented by David Huffman while he was a graduate student at MIT [Huf52]. Huffman’s algorithm Step 1 Initialize n onenode trees and label them with the symbols of the alphabet given. Record the frequency of each symbol in its tree’s root to indicate the tree’s weight. (More generally, the weight of a tree will be equal to the sum of the frequencies in the tree’s leaves.) Step 2 Repeat the following operation until a single tree is obtained. Find two trees with the smallest weight (ties can be broken arbitrarily, but see Problem 2 in this section’s exercises). Make them the left and right subtree of a new tree and record the sum of their weights in the root of the new tree as its weight. A tree constructed by the above algorithm is called a Huffman tree. It deﬁnes—in the manner described above—a Huffman code. EXAMPLE Consider the ﬁvesymbol alphabet {A, B, C, D, _} with the following occurrence frequencies in a text made up of these symbols: symbol frequency
A
B
C
D
_
0.35
0.1
0.2
0.2
0.15
The Huffman tree construction for this input is shown in Figure 9.12.
340
Greedy Technique
0.1 B
0.15 _
0.2 C
0.2 D
0.2 C
0.2 D
0.35 A
0.25 0.1 B
0.15 _
0.35 A
0.25 0.1 B
0.35 A
0.4
0.15 _
0.2 C
0.4 0.2 C
0.2 D
0.6 0.2 D
0.35 A
0.25 0.1 B
0.15 _ 1.0 1
0 0.4
0.6
0
1
0
0.2 C
0.2 D
1 0.35 A
0.25 0
1
0.1 B
0.15 _
FIGURE 9.12 Example of constructing a Huffman coding tree.
The resulting codewords are as follows: symbol frequency codeword
A
B
C
D
_
0.35 11
0.1 100
0.2 00
0.2 01
0.15 101
9.4 Huffman Trees and Codes
341
Hence, DAD is encoded as 011101, and 10011011011101 is decoded as BAD_AD. With the occurrence frequencies given and the codeword lengths obtained, the average number of bits per symbol in this code is 2 . 0.35 + 3 . 0.1 + 2 . 0.2 + 2 . 0.2 + 3 . 0.15 = 2.25. Had we used a ﬁxedlength encoding for the same alphabet, we would have to use at least 3 bits per each symbol. Thus, for this toy example, Huffman’s code achieves the compression ratio—a standard measure of a compression algorithm’s effectiveness—of (3 − 2.25)/3 . 100% = 25%. In other words, Huffman’s encoding of the text will use 25% less memory than its ﬁxedlength encoding. (Extensive experiments with Huffman codes have shown that the compression ratio for this scheme typically falls between 20% and 80%, depending on the characteristics of the text being compressed.) Huffman’s encoding is one of the most important ﬁlecompression methods. In addition to its simplicity and versatility, it yields an optimal, i.e., minimallength, encoding (provided the frequencies of symbol occurrences are independent and known in advance). The simplest version of Huffman compression calls, in fact, for a preliminary scanning of a given text to count the frequencies of symbol occurrences in it. Then these frequencies are used to construct a Huffman coding tree and encode the text as described above. This scheme makes it necessary, however, to include the coding table into the encoded text to make its decoding possible. This drawback can be overcome by using dynamic Huffman encoding, in which the coding tree is updated each time a new symbol is read from the source text. Further, modern alternatives such as LempelZiv algorithms (e.g., [Say05]) assign codewords not to individual symbols but to strings of symbols, allowing them to achieve better and more robust compressions in many applications. It is important to note that applications of Huffman’s algorithm are not limited to data compression. Suppose we have n positive numbers w1, w2 , . . . , wn that have to be assigned to n leaves ofa binary tree, one per node. If we deﬁne the weighted path length as the sum ni=1 li wi , where li is the length of the simple path from the root to the ith leaf, how can we construct a binary tree with minimum weighted path length? It is this more general problem that Huffman’s algorithm actually solves. (For the coding application, li and wi are the length of the codeword and the frequency of the ith symbol, respectively.) This problem arises in many situations involving decision making. Consider, for example, the game of guessing a chosen object from n possibilities (say, an integer between 1 and n) by asking questions answerable by yes or no. Different strategies for playing this game can be modeled by decision trees5 such as those depicted in Figure 9.13 for n = 4. The length of the simple path from the root to a leaf in such a tree is equal to the number of questions needed to get to the chosen number represented by the leaf. If number i is chosen with probability pi , the sum 5.
Decision trees are discussed in more detail in Section 11.2.
342
Greedy Technique
no
n >2
n >1
n=4
no
yes
yes n=4
n =3
n >3
no
yes
no
yes
n=1
n=2
n=3
n= 4
no
yes n =3
n=2
no
yes
n=1
n=2
FIGURE 9.13 Two decision trees for guessing an integer between 1 and 4.
n
i=1 li pi , where li is the length of the path from the root to the ith leaf, indicates the average number of questions needed to “guess” the chosen number with a game strategy represented by its decision tree. If each of the numbers is chosen with the same probability of 1/n, the best strategy is to successively eliminate half (or almost half) the candidates as binary search does. This may not be the case for arbitrary pi ’s, however. For example, if n = 4 and p1 = 0.1, p2 = 0.2, p3 = 0.3, and p4 = 0.4, the minimum weighted path tree is the rightmost one in Figure 9.13. Thus, we need Huffman’s algorithm to solve this problem in its general case. Note that this is the second time we are encountering the problem of constructing an optimal binary tree. In Section 8.3, we discussed the problem of constructing an optimal binary search tree with positive numbers (the search probabilities) assigned to every node of the tree. In this section, given numbers are assigned just to leaves. The latter problem turns out to be easier: it can be solved by the greedy algorithm, whereas the former is solved by the more complicated dynamic programming algorithm.
Exercises 9.4 1. a. Construct a Huffman code for the following data: symbol frequency
A
B
C
D
_
0.4
0.1
0.2
0.15
0.15
b. Encode ABACABAD using the code of question (a). c. Decode 100010111001010 using the code of question (a). 2. For data transmission purposes, it is often desirable to have a code with a minimum variance of the codeword lengths (among codes of the same average length). Compute the average and variance of the codeword length in two
343
9.4 Huffman Trees and Codes
Huffman codes that result from a different tie breaking during a Huffman code construction for the following data: symbol probability
A
B
C
D
E
0.1
0.1
0.2
0.2
0.4
3. Indicate whether each of the following properties is true for every Huffman code. a. The codewords of the two least frequent symbols have the same length. b. The codeword’s length of a more frequent symbol is always smaller than or equal to the codeword’s length of a less frequent one. 4. What is the maximal length of a codeword possible in a Huffman encoding of an alphabet of n symbols? 5. a. Write pseudocode of the Huffmantree construction algorithm. b. What is the time efﬁciency class of the algorithm for constructing a Huffman tree as a function of the alphabet size? 6. Show that a Huffman tree can be constructed in linear time if the alphabet symbols are given in a sorted order of their frequencies. 7. Given a Huffman coding tree, which algorithm would you use to get the codewords for all the symbols? What is its timeefﬁciency class as a function of the alphabet size? 8. Explain how one can generate a Huffman code without an explicit generation of a Huffman coding tree. 9. a. Write a program that constructs a Huffman code for a given English text and encode it. b. Write a program for decoding of an English text which has been encoded with a Huffman code. c. Experiment with your encoding program to ﬁnd a range of typical compression ratios for Huffman’s encoding of English texts of, say, 1000 words. d. Experiment with your encoding program to ﬁnd out how sensitive the compression ratios are to using standard estimates of frequencies instead of actual frequencies of symbol occurrences in English texts. 10. Card guessing Design a strategy that minimizes the expected number of questions asked in the following game [Gar94]. You have a deck of cards that consists of one ace of spades, two deuces of spades, three threes, and on up to nine nines, making 45 cards in all. Someone draws a card from the shufﬂed deck, which you have to identify by asking questions answerable with yes or no.
344
Greedy Technique
SUMMARY The greedy technique suggests constructing a solution to an optimization problem through a sequence of steps, each expanding a partially constructed solution obtained so far, until a complete solution to the problem is reached. On each step, the choice made must be feasible, locally optimal, and irrevocable. Prim’s algorithm is a greedy algorithm for constructing a minimum spanning tree of a weighted connected graph. It works by attaching to a previously constructed subtree a vertex closest to the vertices already in the tree. Kruskal’s algorithm is another greedy algorithm for the minimum spanning tree problem. It constructs a minimum spanning tree by selecting edges in nondecreasing order of their weights provided that the inclusion does not create a cycle. Checking the latter condition efﬁciently requires an application of one of the socalled unionﬁnd algorithms. Dijkstra’s algorithm solves the singlesource shortestpath problem of ﬁnding shortest paths from a given vertex (the source) to all the other vertices of a weighted graph or digraph. It works as Prim’s algorithm but compares path lengths rather than edge lengths. Dijkstra’s algorithm always yields a correct solution for a graph with nonnegative weights. A Huffman tree is a binary tree that minimizes the weighted path length from the root to the leaves of predeﬁned weights. The most important application of Huffman trees is Huffman codes. A Huffman code is an optimal preﬁxfree variablelength encoding scheme that assigns bit strings to symbols based on their frequencies in a given text. This is accomplished by a greedy construction of a binary tree whose leaves represent the alphabet symbols and whose edges are labeled with 0’s and 1’s.
10 Iterative Improvement The most successful men in the end are those whose success is the result of steady accretion. —Alexander Graham Bell (1835–1910)
T
he greedy strategy, considered in the preceding chapter, constructs a solution to an optimization problem piece by piece, always adding a locally optimal piece to a partially constructed solution. In this chapter, we discuss a different approach to designing algorithms for optimization problems. It starts with some feasible solution (a solution that satisﬁes all the constraints of the problem) and proceeds to improve it by repeated applications of some simple step. This step typically involves a small, localized change yielding a feasible solution with an improved value of the objective function. When no such change improves the value of the objective function, the algorithm returns the last feasible solution as optimal and stops. There can be several obstacles to the successful implementation of this idea. First, we need an initial feasible solution. For some problems, we can always start with a trivial solution or use an approximate solution obtained by some other (e.g., greedy) algorithm. But for others, ﬁnding an initial solution may require as much effort as solving the problem after a feasible solution has been identiﬁed. Second, it is not always clear what changes should be allowed in a feasible solution so that we can check efﬁciently whether the current solution is locally optimal and, if not, replace it with a better one. Third—and this is the most fundamental difﬁculty— is an issue of local versus global extremum (maximum or minimum). Think about the problem of ﬁnding the highest point in a hilly area with no map on a foggy day. A logical thing to do would be to start walking “up the hill” from the point you are at until it becomes impossible to do so because no direction would lead up. You will have reached a local highest point, but because of a limited feasibility, there will be no simple way to tell whether the point is the highest (global maximum you are after) in the entire area. Fortunately, there are important problems that can be solved by iterativeimprovement algorithms. The most important of them is linear programming. 345
346
Iterative Improvement
We have already encountered this topic in Section 6.6. Here, in Section 10.1, we introduce the simplex method, the classic algorithm for linear programming. Discovered by the U.S. mathematician George B. Dantzig in 1947, this algorithm has proved to be one of the most consequential achievements in the history of algorithmics. In Section 10.2, we consider the important problem of maximizing the amount of ﬂow that can be sent through a network with links of limited capacities. This problem is a special case of linear programming. However, its special structure makes it possible to solve the problem by algorithms that are more efﬁcient than the simplex method. We outline the classic iterativeimprovement algorithm for this problem, discovered by the American mathematicians L. R. Ford, Jr., and D. R. Fulkerson in the 1950s. The last two sections of the chapter deal with bipartite matching. This is the problem of ﬁnding an optimal pairing of elements taken from two disjoint sets. Examples include matching workers and jobs, high school graduates and colleges, and men and women for marriage. Section 10.3 deals with the problem of maximizing the number of matched pairs; Section 10.4 is concerned with the matching stability. We also discuss several iterativeimprovement algorithms in Section 12.3, where we consider approximation algorithms for the traveling salesman and knapsack problems. Other examples of iterativeimprovement algorithms can be found in the algorithms textbook by Moret and Shapiro [Mor91], books on continuous and discrete optimization (e.g., [Nem89]), and the literature on heuristic search (e.g., [Mic10]).
10.1 The Simplex Method We have already encountered linear programming (see Section 6.6)—the general problem of optimizing a linear function of several variables subject to a set of linear constraints: c1x1 + . . . + cnxn subject to ai1x1 + . . . + ainxn ≤ (or ≥ or =) bi
maximize (or minimize)
x1 ≥ 0, . . . , xn ≥ 0.
for i = 1, . . . , m (10.1)
We mentioned there that many important practical problems can be modeled as instances of linear programming. Two researchers, L. V. Kantorovich of the former Soviet Union and the DutchAmerican T. C. Koopmans, were even awarded the Nobel Prize in 1975 for their contributions to linear programming theory and its applications to economics. Apparently because there is no Nobel Prize in mathematics, the Royal Swedish Academy of Sciences failed to honor the U.S. mathematician G. B. Dantzig, who is universally recognized as the father of linear
10.1
The Simplex Method
347
programming in its modern form and the inventor of the simplex method, the classic algorithm for solving such problems.1
Geometric Interpretation of Linear Programming Before we introduce a general method for solving linear programming problems, let us consider a small example, which will help us to see the fundamental properties of such problems. EXAMPLE 1 Consider the following linear programming problem in two variables: maximize 3x + 5y subject to
x+ y≤4 x + 3y ≤ 6
(10.2)
x ≥ 0, y ≥ 0. By deﬁnition, a feasible solution to this problem is any point (x, y) that satisﬁes all the constraints of the problem; the problem’s feasible region is the set of all its feasible points. It is instructive to sketch the feasible region in the Cartesian plane. Recall that any equation ax + by = c, where coefﬁcients a and b are not both equal to zero, deﬁnes a straight line. Such a line divides the plane into two halfplanes: for all the points in one of them, ax + by < c, while for all the points in the other, ax + by > c. (It is easy to determine which of the two halfplanes is which: take any point (x0, y0) not on the line ax + by = c and check which of the two inequalities hold, ax0 + by0 > c or ax0 + by0 < c.) In particular, the set of points deﬁned by inequality x + y ≤ 4 comprises the points on and below the line x + y = 4, and the set of points deﬁned by inequality x + 3y ≤ 6 comprises the points on and below the line x + 3y = 6. Since the points of the feasible region must satisfy all the constraints of the problem, the feasible region is obtained by the intersection of these two halfplanes and the ﬁrst quadrant of the Cartesian plane deﬁned by the nonnegativity constraints x ≥ 0, y ≥ 0 (see Figure 10.1). Thus, the feasible region for problem (10.2) is the convex polygon with the vertices (0, 0), (4, 0), (0, 2), and (3, 1). (The last point, which is the point of intersection of the lines x + y = 4 and x + 3y = 6, is obtained by solving the system of these two linear equations.) Our task is to ﬁnd an optimal solution, a point in the feasible region with the largest value of the objective function z = 3x + 5y. Are there feasible solutions for which the value of the objective function equals, say, 20? The points (x, y) for which the objective function z = 3x + 5y is equal to 20 form the line 3x + 5y = 20. Since this line does not have common points 1.
George B. Dantzig (1914–2005) has received many honors, including the National Medal of Science presented by the president of the United States in 1976. The citation states that the National Medal was awarded “for inventing linear programming and discovering methods that led to widescale scientiﬁc and technical applications to important problems in logistics, scheduling, and network optimization, and to the use of computers in making efﬁcient use of the mathematical theory.”
348
Iterative Improvement y
x + 3y = 6 (0, 2) (3, 1)
(0, 0)
x
(4, 0)
x+y=4
FIGURE 10.1 Feasible region of problem (10.2).
with the feasible region—see Figure 10.2—the answer to the posed question is no. On the other hand, there are inﬁnitely many feasible points for which the objective function is equal to, say, 10: they are the intersection points of the line 3x + 5y = 10 with the feasible region. Note that the lines 3x + 5y = 20 and 3x + 5y = 10 have the same slope, as would any line deﬁned by equation 3x + 5y = z where z is some constant. Such lines are called level lines of the objective function. Thus, our problem can be restated as ﬁnding the largest value of the parameter z for which the level line 3x + 5y = z has a common point with the feasible region. We can ﬁnd this line either by shifting, say, the line 3x + 5y = 20 southwest (without changing its slope!) toward the feasible region until it hits the region for the ﬁrst time or by shifting, say, the line 3x + 5y = 10 northeast until it hits the feasible region for the last time. Either way, it will happen at the point (3, 1) with the corresponding z value 3 . 3 + 5 . 1 = 14. This means that the optimal solution to the linear programming problem in question is x = 3, y = 1, with the maximal value of the objective function equal to 14. Note that if we had to maximize z = 3x + 3y as the objective function in problem (10.2), the level line 3x + 3y = z for the largest value of z would coincide with the boundary line segment that has the same slope as the level lines (draw this line in Figure 10.2). Consequently, all the points of the line segment between vertices (3, 1) and (4, 0), including the vertices themselves, would be optimal solutions, yielding, of course, the same maximal value of the objective function.
10.1
349
The Simplex Method
y
(0, 2) (3, 1)
(0, 0)
(4, 0) 3x + 5y = 20
x
3x + 5y = 14 3x + 5y = 10
FIGURE 10.2 Solving a twodimensional linear programming problem geometrically.
Does every linear programming problem have an optimal solution that can be found at a vertex of its feasible region? Without appropriate qualiﬁcations, the answer to this question is no. To begin with, the feasible region of a linear programming problem can be empty. For example, if the constraints include two contradictory requirements, such as x + y ≤ 1 and x + y ≥ 2, there can be no points in the problem’s feasible region. Linear programming problems with the empty feasible region are called infeasible. Obviously, infeasible problems do not have optimal solutions. Another complication may arise if the problem’s feasible region is unbounded, as the following example demonstrates. EXAMPLE 2 If we reverse the inequalities in problem (10.2) to x + y ≥ 4 and x + 3y ≥ 6, the feasible region of the new problem will become unbounded (see Figure 10.3). If the feasible region of a linear programming problem is unbounded, its objective function may or may not attain a ﬁnite optimal value on it. For example, the problem of maximizing z = 3x + 5y subject to the constraints x + y ≥ 4, x + 3y ≥ 6, x ≥ 0, y ≥ 0 has no optimal solution, because there are points in the feasible region making 3x + 5y as large as we wish. Such problems are called unbounded. On the other hand, the problem of minimizing z = 3x + 5y subject to the same constraints has an optimal solution (which?).
350
Iterative Improvement y
(0, 4)
(3, 1) (0, 0)
(6, 0)
x 3x + 5y = 24 3x + 5y = 20 3x + 5y = 14
FIGURE 10.3 Unbounded feasible region of a linear programming problem with constraints x + y ≥ 4, x + 3y ≥ 6, x ≥ 0, y ≥ 0, and three level lines of the function 3x + 5y.
Fortunately, the most important features of the examples we considered above hold for problems with more than two variables. In particular, a feasible region of a typical linear programming problem is in many ways similar to convex polygons in the twodimensional Cartesian plane. Speciﬁcally, it always has a ﬁnite number of vertices, which mathematicians prefer to call extreme points (see Section 3.3). Furthermore, an optimal solution to a linear programming problem can be found at one of the extreme points of its feasible region. We reiterate these properties in the following theorem. THEOREM (Extreme Point Theorem) Any linear programming problem with a nonempty bounded feasible region has an optimal solution; moreover, an optimal solution can always be found at an extreme point of the problem’s feasible region.2 This theorem implies that to solve a linear programming problem, at least in the case of a bounded feasible region, we can ignore all but a ﬁnite number of 2.
Except for some degenerate instances (such as maximizing z = x + y subject to x + y = 1), if a linear programming problem with an unbounded feasible region has an optimal solution, it can also be found at an extreme point of the feasible region.
10.1
The Simplex Method
351
points in its feasible region. In principle, we can solve such a problem by computing the value of the objective function at each extreme point and selecting the one with the best value. There are two major obstacles to implementing this plan, however. The ﬁrst lies in the need for a mechanism for generating the extreme points of the feasible region. As we are going to see below, a rather straightforward algebraic procedure for this task has been discovered. The second obstacle lies in the number of extreme points a typical feasible region has. Here, the news is bad: the number of extreme points is known to grow exponentially with the size of the problem. This makes the exhaustive inspection of extreme points unrealistic for most linear programming problems of nontrivial sizes. Fortunately, it turns out that there exists an algorithm that typically inspects only a small fraction of the extreme points of the feasible region before reaching an optimal one. This famous algorithm is called the simplex method. The idea of this algorithm can be described in geometric terms as follows. Start by identifying an extreme point of the feasible region. Then check whether one can get an improved value of the objective function by going to an adjacent extreme point. If it is not the case, the current point is optimal—stop; if it is the case, proceed to an adjacent extreme point with an improved value of the objective function. After a ﬁnite number of steps, the algorithm will either reach an extreme point where an optimal solution occurs or determine that no optimal solution exists.
An Outline of the Simplex Method Our task now is to “translate” the geometric description of the simplex method into the more algorithmically precise language of algebra. To begin with, before we can apply the simplex method to a linear programming problem, it has to be represented in a special form called the standard form. The standard form has the following requirements: It must be a maximization problem. All the constraints (except the nonnegativity constraints) must be in the form of linear equations with nonnegative righthand sides. All the variables must be required to be nonnegative. Thus, the general linear programming problem in standard form with m constraints and n unknowns (n ≥ m) is maximize subject to
c1x1 + . . . + cnxn ai1x1 + . . . + ainxn = bi , where bi ≥ 0 for i = 1, 2, . . . , m (10.3) x1 ≥ 0, . . . , xn ≥ 0.
It can also be written in compact matrix notations: maximize subject to
cx Ax = b x ≥ 0,
352
Iterative Improvement
where
⎤ ⎡ x1 ⎢ x2 ⎥ ⎥ ⎣ c = [c1 c2 . . . cn], x = ⎢ ⎣ ... ⎦ , A = ⎡
xn
a11 a12 .. .. . . am1 am2
... ...
⎡ ⎤ a1n ⎢ .. ⎦ , b=⎢ . ⎣ amn
⎤ b1 b2 ⎥ . .. ⎥ . ⎦ bm
Any linear programming problem can be transformed into an equivalent problem in standard form. If an objective function needs to be minimized, it can be replaced by the equivalent problem of maximizing the same objective function with all its coefﬁcients cj replaced by −cj , j = 1, 2, . . . , n (see Section 6.6 for a more general discussion of such transformations). If a constraint is given as an inequality, it can be replaced by an equivalent equation by adding a slack variable representing the difference between the two sides of the original inequality. For example, the two inequalities of problem (10.2) can be transformed, respectively, into the following equations: x + y + u = 4 where u ≥ 0
and
x + 3y + v = 6 where v ≥ 0.
Finally, in most linear programming problems, the variables are required to be nonnegative to begin with because they represent some physical quantities. If this is not the case in an initial statement of a problem, an unconstrained variable xj can be replaced by the difference between two new nonnegative variables:
xj = xj − xj , xj ≥ 0, xj ≥ 0. Thus, problem (10.2) in standard form is the following linear programming problem in four variables: maximize subject to
3x + 5y + 0u + 0v x+ y+ u x + 3y +
=4 + v=6
(10.4)
x, y, u, v ≥ 0. It is easy to see that if we ﬁnd an optimal solution (x ∗, y ∗, u∗, v ∗) to problem (10.4), we can obtain an optimal solution to problem (10.2) by simply ignoring its last two coordinates. The principal advantage of the standard form lies in the simple mechanism it provides for identifying extreme points of the feasible region. To do this for problem (10.4), for example, we need to set two of the four variables in the constraint equations to zero to get a system of two linear equations in two unknowns and solve this system. For the general case of a problem with m equations in n unknowns (n ≥ m), n − m variables need to be set to zero to get a system of m equations in m unknowns. If the system obtained has a unique solution—as any nondegenerate system of linear equations with the number of equations equal to the number of unknowns does—we have a basic solution; its coordinates set to zero before solving the system are called nonbasic, and its coordinates obtained by solving the system are called basic. (This terminology comes from linear algebra.
10.1
The Simplex Method
353
Speciﬁcally, we can rewrite the system of constraint equations of (10.4) as x
1 1 1 0 4 +y +u +v = . 1 3 0 1 6
A basis in the twodimensional vector space is composed of any two vectors that are not proportional to each other; once a basis is chosen, any vector can be uniquely expressed as a sum of multiples of the basis vectors. Basic and nonbasic variables indicate which of the given vectors are, respectively, included and excluded in a particular basis choice.) If all the coordinates of a basic solution are nonnegative, the basic solution is called a basic feasible solution. For example, if we set to zero variables x and y and solve the resulting system for u and v, we obtain the basic feasible solution (0, 0, 4, 6); if we set to zero variables x and u and solve the resulting system for y and v, we obtain the basic solution (0, 4, 0, −6), which is not feasible. The importance of basic feasible solutions lies in the onetoone correspondence between them and the extreme points of the feasible region. For example, (0, 0, 4, 6) is an extreme point of the feasible region of problem (10.4) (with the point (0, 0) in Figure 10.1 being its projection on the x, y plane). Incidentally, (0, 0, 4, 6) is a natural starting point for the simplex method’s application to this problem. As mentioned above, the simplex method progresses through a series of adjacent extreme points (basic feasible solutions) with increasing values of the objective function. Each such point can be represented by a simplex tableau, a table storing the information about the basic feasible solution corresponding to the extreme point. For example, the simplex tableau for (0, 0, 4, 6) of problem (10.4) is presented below: x
y
u
v
u
1
1
1
0
4
v
1
3
0
1
6
–3
–5
0
0
0
(10.5)
In general, a simplex tableau for a linear programming problem in standard form with n unknowns and m linear equality constraints (n ≥ m) has m + 1 rows and n + 1 columns. Each of the ﬁrst m rows of the table contains the coefﬁcients of a corresponding constraint equation, with the last column’s entry containing the equation’s righthand side. The columns, except the last one, are labeled by the names of the variables. The rows are labeled by the basic variables of the basic feasible solution the tableau represents; the values of the basic variables of this
354
Iterative Improvement
solution are in the last column. Also note that the columns labeled by the basic variables form the m × m identity matrix. The last row of a simplex tableau is called the objective row. It is initialized by the coefﬁcients of the objective function with their signs reversed (in the ﬁrst n columns) and the value of the objective function at the initial point (in the last column). On subsequent iterations, the objective row is transformed the same way as all the other rows. The objective row is used by the simplex method to check whether the current tableau represents an optimal solution: it does if all the entries in the objective row—except, possibly, the one in the last column—are nonnegative. If this is not the case, any of the negative entries indicates a nonbasic variable that can become basic in the next tableau. For example, according to this criterion, the basic feasible solution (0, 0, 4, 6) represented by tableau (10.5) is not optimal. The negative value in the xcolumn signals the fact that we can increase the value of the objective function z = 3x + 5y + 0u + 0v by increasing the value of the xcoordinate in the current basic feasible solution (0, 0, 4, 6). Indeed, since the coefﬁcient for x in the objective function is positive, the larger the x value, the larger the value of this function. Of course, we will need to “compensate” an increase in x by adjusting the values of the basic variables u and v so that the new point is still feasible. For this to be the case, both conditions x + u = 4 where u ≥ 0 x + v = 6 where v ≥ 0 must be satisﬁed, which means that x ≤ min{4, 6} = 4. Note that if we increase the value of x from 0 to 4, the largest amount possible, we will ﬁnd ourselves at the point (4, 0, 0, 2), an adjacent to (0, 0, 4, 6) extreme point of the feasible region, with z = 12. Similarly, the negative value in the ycolumn of the objective row signals the fact that we can also increase the value of the objective function by increasing the value of the ycoordinate in the initial basic feasible solution (0, 0, 4, 6). This requires y+u=4
where u ≥ 0
3y + v = 6 where v ≥ 0, which means that 4 6 y ≤ min{ , } = 2. 1 3 If we increase the value of y from 0 to 2, the largest amount possible, we will ﬁnd ourselves at the point (0, 2, 2, 0), another adjacent to (0, 0, 4, 6) extreme point, with z = 10. If there are several negative entries in the objective row, a commonly used rule is to select the most negative one, i.e., the negative number with the largest
10.1
The Simplex Method
355
absolute value. This rule is motivated by the observation that such a choice yields the largest increase in the objective function’s value per unit of change in a variable’s value. (In our example, an increase in the xvalue from 0 to 1 at (0, 0, 4, 6) changes the value of z = 3x + 5y + 0u + 0v from 0 to 3, while an increase in the yvalue from 0 to 1 at (0, 0, 4, 6) changes z from 0 to 5.) Note, however, that the feasibility constraints impose different limits on how much each of the variables may increase. In our example, in particular, the choice of the yvariable over the xvariable leads to a smaller increase in the value of the objective function. Still, we will employ this commonly used rule and select variable y as we continue with our example. A new basic variable is called the entering variable, while its column is referred to as the pivot column; we mark the pivot column by ↑ . Now we will explain how to choose a departing variable, i.e., a basic variable to become nonbasic in the next tableau. (The total number of basic variables in any basic solution must be equal to m, the number of the equality constraints.) As we saw above, to get to an adjacent extreme point with a larger value of the objective function, we need to increase the entering variable by the largest amount possible to make one of the old basic variables zero while preserving the nonnegativity of all the others. We can translate this observation into the following rule for choosing a departing variable in a simplex tableau: for each positive entry in the pivot column, compute the θ ratio by dividing the row’s last entry by the entry in the pivot column. For the example of tableau (10.5), these θratios are θu =
4 = 4, 1
θv =
6 = 2. 3
The row with the smallest θratio determines the departing variable, i.e., the variable to become nonbasic. Ties may be broken arbitrarily. For our example, it is variable v. We mark the row of the departing variable, called the pivot row, by ←− ←−−− and denote it row. Note that if there are no positive entries in the pivot column, no θratio can be computed, which indicates that the problem is unbounded and the algorithm stops. Finally, the following steps need to be taken to transform a current tableau into the next one. (This transformation, called pivoting, is similar to the principal step of the GaussJordan elimination algorithm for solving systems of linear equations—see Problem 8 in Exercises 6.2.) First, divide all the entries of the pivot ←−−− row by the pivot, its entry in the pivot column, to obtain rownew . For tableau (10.5), we obtain ←−−−
rownew :
1 3
1 0
1 3
2.
Then, replace each of the other rows, including the objective row, by the difference ←−−−
row − c . rownew , where c is the row’s entry in the pivot column. For tableau (10.5), this yields
356
Iterative Improvement
2 3 4 ←−−− row 3 − (−5) . rownew : − 3 ←−−−
row 1 − 1 . rownew :
0
1
0
0
−
1 3 5 3
2, 10.
Thus, the simplex method transforms tableau (10.5) into the following tableau: x
y
u
v
u
2 3
0
1
– 13
2
y
1 3
1
0
1 3
2
– 43
0
0
5 3
10
(10.6)
Tableau (10.6) represents the basic feasible solution (0, 2, 2, 0) with an increased value of the objective function, which is equal to 10. It is not optimal, however (why?). The next iteration—do it yourself as a good exercise!—yields tableau (10.7): x
y
u
v
x
1
0
3 2
– 12
3
y
0
1
– 12
1 2
1
0
0
2
1
14
(10.7)
This tableau represents the basic feasible solution (3, 1, 0, 0). It is optimal because all the entries in the objective row of tableau (10.7) are nonnegative. The maximal value of the objective function is equal to 14, the last entry in the objective row. Let us summarize the steps of the simplex method. Summary of the simplex method Step 0 Initialization Present a given linear programming problem in standard form and set up an initial tableau with nonnegative entries in the rightmost column and m other columns composing the m × m identity matrix. (Entries in the objective row are to be disregarded in verifying these requirements.) These m columns deﬁne the basic variables of the initial basic feasible solution, used as the labels of the tableau’s rows. Step 1 Optimality test If all the entries in the objective row (except, possibly, the one in the rightmost column, which represents the value of the
10.1
The Simplex Method
357
objective function) are nonnegative—stop: the tableau represents an optimal solution whose basic variables’ values are in the rightmost column and the remaining, nonbasic variables’ values are zeros. Step 2 Finding the entering variable Select a negative entry from among the ﬁrst n elements of the objective row. (A commonly used rule is to select the negative entry with the largest absolute value, with ties broken arbitrarily.) Mark its column to indicate the entering variable and the pivot column. Step 3 Finding the departing variable For each positive entry in the pivot column, calculate the θratio by dividing that row’s entry in the rightmost column by its entry in the pivot column. (If all the entries in the pivot column are negative or zero, the problem is unbounded—stop.) Find the row with the smallest θratio (ties may be broken arbitrarily), and mark this row to indicate the departing variable and the pivot row. Step 4 Forming the next tableau Divide all the entries in the pivot row by its entry in the pivot column. Subtract from each of the other rows, including the objective row, the new pivot row multiplied by the entry in the pivot column of the row in question. (This will make all the entries in the pivot column 0’s except for 1 in the pivot row.) Replace the label of the pivot row by the variable’s name of the pivot column and go back to Step 1.
Further Notes on the Simplex Method Formal proofs of validity of the simplex method steps can be found in books devoted to a detailed discussion of linear programming (e.g., [Dan63]). A few important remarks about the method still need to be made, however. Generally speaking, an iteration of the simplex method leads to an extreme point of the problem’s feasible region with a greater value of the objective function. In degenerate cases, which arise when one or more basic variables are equal to zero, the simplex method can only guarantee that the value of the objective function at the new extreme point is greater than or equal to its value at the previous point. In turn, this opens the door to the possibility not only that the objective function’s values “stall” for several iterations in a row but that the algorithm might cycle back to a previously considered point and hence never terminate. The latter phenomenon is called cycling. Although it rarely if ever happens in practice, speciﬁc examples of problems where cycling does occur have been constructed. A simple modiﬁcation of Steps 2 and 3 of the simplex method, called Bland’s rule, eliminates even the theoretical possibility of cycling. Assuming that the variables are denoted by a subscripted letter (e.g., x1, x2 , . . . , xn), this rule can be stated as follows: Step 2 modiﬁed Among the columns with a negative entry in the objective row, select the column with the smallest subscript. Step 3 modiﬁed Resolve a tie among the smallest θratios by selecting the row labeled by the basic variable with the smallest subscript.
358
Iterative Improvement
Another caveat deals with the assumptions made in Step 0. They are automatically satisﬁed if a problem is given in the form where all the constraints imposed on nonnegative variables are inequalities ai1x1 + . . . + ainxn ≤ bi with bi ≥ 0 for i = 1, 2, . . . , m. Indeed, by adding a nonnegative slack variable xn+i into the ith constraint, we obtain the equality ai1x1 + . . . + ainxn + xn+i = bi , and all the requirements imposed on an initial tableau of the simplex method are satisﬁed for the obvious basic feasible solution x1 = . . . = xn = 0, xn+1 = . . . = xn+m = 1. But if a problem is not given in such a form, ﬁnding an initial basic feasible solution may present a nontrivial obstacle. Moreover, for problems with an empty feasible region, no initial basic feasible solution exists, and we need an algorithmic way to identify such problems. One of the ways to address these issues is to use an extension to the classic simplex method called the twophase simplex method (see, e.g., [Kol95]). In a nutshell, this method adds a set of artiﬁcial variables to the equality constraints of a given problem so that the new problem has an obvious basic feasible solution. It then solves the linear programming problem of minimizing the sum of the artiﬁcial variables by the simplex method. The optimal solution to this problem either yields an initial tableau for the original problem or indicates that the feasible region of the original problem is empty. How efﬁcient is the simplex method? Since the algorithm progresses through a sequence of adjacent points of a feasible region, one should probably expect bad news because the number of extreme points is known to grow exponentially with the problem size. Indeed, the worstcase efﬁciency of the simplex method has been shown to be exponential as well. Fortunately, more than half a century of practical experience with the algorithm has shown that the number of iterations in a typical application ranges between m and 3m, with the number of operations per iteration proportional to mn, where m and n are the numbers of equality constraints and variables, respectively. Since its discovery in 1947, the simplex method has been a subject of intensive study by many researchers. Some of them have worked on improvements to the original algorithm and details of its efﬁcient implementation. As a result of these efforts, programs implementing the simplex method have been polished to the point that very large problems with hundreds of thousands of constraints and variables can be solved in a routine manner. In fact, such programs have evolved into sophisticated software packages. These packages enable the user to enter a problem’s constraints and obtain a solution in a userfriendly form. They also provide tools for investigating important properties of the solution, such as its sensitivity to changes in the input data. Such investigations are very important for many applications, including those in economics. At the other end of the spectrum, linear programming problems of a moderate size can nowadays be solved on a desktop using a standard spreadsheet facility or by taking advantage of specialized software available on the Internet. Researchers have also tried to ﬁnd algorithms for solving linear programming problems with polynomialtime efﬁciency in the worst case. An important milestone in the history of such algorithms was the proof by L. G. Khachian [Kha79] showing that the ellipsoid method can solve any linear programming problem in
10.1
The Simplex Method
359
polynomial time. Although the ellipsoid method was much slower than the simplex method in practice, its better worstcase efﬁciency encouraged a search for alternatives to the simplex method. In 1984, Narendra Karmarkar published an algorithm that not only had a polynomial worstcase efﬁciency but also was competitive with the simplex method in empirical tests as well. Although we are not going to discuss Karmarkar’s algorithm [Kar84] here, it is worth pointing out that it is also based on the iterativeimprovement idea. However, Karmarkar’s algorithm generates a sequence of feasible solutions that lie within the feasible region rather than going through a sequence of adjacent extreme points as the simplex method does. Such algorithms are called interiorpoint methods (see, e.g., [Arb93]).
Exercises 10.1 1. Consider the following version of the post ofﬁce location problem (Problem 3 in Exercises 3.3): Given n integers x1, x2 , . . . , xn representing coordinates of n villages located along a straight road, ﬁnd a location for a post ofﬁce that minimizes the average distance between the villages. The post ofﬁce may be, but is not required to be, located at one of the villages. Devise an iterativeimprovement algorithm for this problem. Is this an efﬁcient way to solve this problem? 2. Solve the following linear programming problems geometrically. a. maximize 3x + y subject to −x + y ≤ 1 2x + y ≤ 4 x ≥ 0, y ≥ 0 b.
maximize x + 2y subject to 4x ≥ y y ≤3+x x ≥ 0, y ≥ 0
3. Consider the linear programming problem minimize c1x + c2 y subject to
x+ y≥4 x + 3y ≥ 6 x ≥ 0, y ≥ 0
where c1 and c2 are some real numbers not both equal to zero. a. Give an example of the coefﬁcient values c1 and c2 for which the problem has a unique optimal solution.
360
Iterative Improvement
b. Give an example of the coefﬁcient values c1 and c2 for which the problem has inﬁnitely many optimal solutions. c. Give an example of the coefﬁcient values c1 and c2 for which the problem does not have an optimal solution. 4. Would the solution to problem (10.2) be different if its inequality constraints were strict, i.e., x + y < 4 and x + 3y < 6, respectively? 5. Trace the simplex method on a. the problem of Exercise 2a. b. the problem of Exercise 2b. 6. Trace the simplex method on the problem of Example 1 in Section 6.6 a. by hand. b. by using one of the implementations available on the Internet. 7. Determine how many iterations the simplex method needs to solve the problem maximize
n
xj
j =1
subject to 0 ≤ xj ≤ bj ,
where bj > 0 for j = 1, 2, . . . , n.
8. Can we apply the simplex method to solve the knapsack problem (see Example 2 in Section 6.6)? If you answer yes, indicate whether it is a good algorithm for the problem in question; if you answer no, explain why not. 9. Prove that no linear programming problem can have exactly k ≥ 1 optimal solutions unless k = 1. 10. If a linear programming problem maximize
n
cj xj
j =1
subject to
n
aij xj ≤ bi
for i = 1, 2, . . . , m
j =1
x1, x2 , . . . , xn ≥ 0 is considered as primal, then its dual is deﬁned as the linear programming problem minimize
subject to
m
i=1 m
bi yi aij yi ≥ cj
for j = 1, 2, . . . , n
i=1
y1, y2 , . . . , ym ≥ 0.
10.2
The MaximumFlow Problem
361
a. Express the primal and dual problems in matrix notations. b. Find the dual of the linear programming problem maximize x1 + 4x2 − x3 subject to x1 + x2 + x3 ≤ 6 x1 − x2 − 2x3 ≤ 2 x1, x2 , x3 ≥ 0. c. Solve the primal and dual problems and compare the optimal values of their objective functions.
10.2 The MaximumFlow Problem In this section, we consider the important problem of maximizing the ﬂow of a material through a transportation network (pipeline system, communication system, electrical distribution system, and so on). We will assume that the transportation network in question can be represented by a connected weighted digraph with n vertices numbered from 1 to n and a set of edges E, with the following properties: It contains exactly one vertex with no entering edges; this vertex is called the source and assumed to be numbered 1. It contains exactly one vertex with no leaving edges; this vertex is called the sink and assumed to be numbered n. The weight uij of each directed edge (i, j ) is a positive integer, called the edge capacity. (This number represents the upper bound on the amount of the material that can be sent from i to j through a link represented by this edge.) A digraph satisfying these properties is called a ﬂow network or simply a network.3 A small instance of a network is given in Figure 10.4. It is assumed that the source and the sink are the only source and destination of the material, respectively; all the other vertices can serve only as points where a ﬂow can be redirected without consuming or adding any amount of the material. In other words, the total amount of the material entering an intermediate vertex must be equal to the total amount of the material leaving the vertex. This condition is called the ﬂowconservation requirement. If we denote the amount sent through edge (i, j ) by xij , then for any intermediate vertex i, the ﬂowconservation requirement can be expressed by the following equality constraint:
xj i = xij for i = 2, 3, . . . , n − 1, (10.8) j : (j,i)∈E
3.
j : (i,j )∈E
In a slightly more general model, one can consider a network with several sources and sinks and allow capacities uij to be inﬁnitely large.
362
Iterative Improvement
5 3 2
1
4 5
2
3
3
2
6
1 4
FIGURE 10.4 Example of a network graph. The vertex numbers are vertex “names”; the edge numbers are edge capacities.
where the sums in the left and righthand sides express the total inﬂow and outﬂow entering and leaving vertex i, respectively. Since no amount of the material can change by going through intermediate vertices of the network, it stands to reason that the total amount of the material leaving the source must end up at the sink. (This observation can also be derived formally from equalities (10.8), a task you will be asked to do in the exercises.) Thus, we have the following equality:
x1j = xj n. (10.9) j : (j,n)∈E
j : (1,j )∈E
This quantity, the total outﬂow from the source—or, equivalently, the total inﬂow into the sink—is called the value of the ﬂow. We denote it by v. It is this quantity that we will want to maximize over all possible ﬂows in a network. Thus, a (feasible) ﬂow is an assignment of real numbers xij to edges (i, j ) of a given network that satisfy ﬂowconservation constraints (10.8) and the capacity constraints 0 ≤ xij ≤ uij
for every edge (i, j ) ∈ E.
(10.10)
The maximumﬂow problem can be stated formally as the following optimization problem:
x1j maximize v = j : (1,j )∈E
subject to
xj i −
xij = 0
for i = 2, 3, . . . , n − 1 (10.11)
j : (j,i)∈E
j : (i,j )∈E
0 ≤ xij ≤ uij
for every edge (i, j ) ∈ E.
We can solve linear programming problem (10.11) by the simplex method or by another algorithm for general linear programming problems (see Section 10.1). However, the special structure of problem (10.11) can be exploited to design faster algorithms. In particular, it is quite natural to employ the iterativeimprovement
10.2
The MaximumFlow Problem
363
idea as follows. We can always start with the zero ﬂow (i.e., set xij = 0 for every edge (i, j ) in the network). Then, on each iteration, we can try to ﬁnd a path from source to sink along which some additional ﬂow can be sent. Such a path is called ﬂow augmenting. If a ﬂowaugmenting path is found, we adjust the ﬂow along the edges of this path to get a ﬂow of an increased value and try to ﬁnd an augmenting path for the new ﬂow. If no ﬂowaugmenting path can be found, we conclude that the current ﬂow is optimal. This general template for solving the maximumﬂow problem is called the augmentingpath method, also known as the FordFulkerson method after L. R. Ford, Jr., and D. R. Fulkerson, who discovered it (see [For57]). An actual implementation of the augmenting path idea is, however, not quite straightforward. To see this, let us consider the network in Figure 10.4. We start with the zero ﬂow shown in Figure 10.5a. (In that ﬁgure, the zero amounts sent through each edge are separated from the edge capacities by the slashes; we will use this notation in the other examples as well.) It is natural to search for a ﬂowaugmenting path from source to sink by following directed edges (i, j ) for which the current ﬂow xij is less than the edge capacity uij . Among several possibilities, let us assume that we identify the augmenting path 1→2→3→6 ﬁrst. We can increase the ﬂow along this path by a maximum of 2 units, which is the smallest unused capacity of its edges. The new ﬂow is shown in Figure 10.5b. This is as far as our simpleminded idea about ﬂowaugmenting paths will be able to take us. Unfortunately, the ﬂow shown in Figure 10.5b is not optimal: its value can still be increased along the path 1→4→3←2→5→6 by increasing the ﬂow by 1 on edges (1, 4), (4, 3), (2, 5), and (5, 6) and decreasing it by 1 on edge (2, 3). The ﬂow obtained as the result of this augmentation is shown in Figure 10.5c. It is indeed maximal. (Can you tell why?) Thus, to ﬁnd a ﬂowaugmenting path for a ﬂow x, we need to consider paths from source to sink in the underlying undirected graph in which any two consecutive vertices i, j are either i.
connected by a directed edge from i to j with some positive unused capacity rij = uij − xij (so that we can increase the ﬂow through that edge by up to rij units), or ii. connected by a directed edge from j to i with some positive ﬂow xj i (so that we can decrease the ﬂow through that edge by up to xj i units). Edges of the ﬁrst kind are called forward edges because their tail is listed before their head in the vertex list 1 → . . . i → j . . . → n deﬁning the path; edges of the second kind are called backward edges because their tail is listed after their head in the path list 1 → . . . i ← j . . . → n. To illustrate, for the path 1→4→3←2→5→6 of the last example, (1, 4), (4, 3), (2, 5), and (5, 6) are the forward edges, and (3, 2) is the backward edge. For a given ﬂowaugmenting path, let r be the minimum of all the unused capacities rij of its forward edges and all the ﬂows xj i of its backward edges. It is easy to see that if we increase the current ﬂow by r on each forward edge and decrease it by this amount on each backward edge, we will obtain a feasible
364
Iterative Improvement
5 0 /3 0 /2
1
0/4
0/5
2
0/3
0/2
3
6
0 /1 4 (a) 5 0 /3
2 /2
1
0/4
2/5
2
0/3
2/2
3
6
0 /1 4 (b) 5 1/3
2 /2
1
1/4
1/5
2
1/3
2/2
3
6
1/1 4 (c)
FIGURE 10.5 Illustration of the augmentingpath method. Flowaugmenting paths are shown in bold. The ﬂow amounts and edge capacities are indicated by the numbers before and after the slash, respectively.
ﬂow whose value is r units greater than the value of its predecessor. Indeed, let i be an intermediate vertex on a ﬂowaugmenting path. There are four possible combinations of forward and backward edges incident to vertex i: +r
+r
−→ i −→,
+r
−r
−→ i ←−,
−r
+r
←− i −→,
−r
−r
←− i ←− .
10.2
The MaximumFlow Problem
365
For each of them, the ﬂowconservation requirement for vertex i will still hold after the ﬂow adjustments indicated above the edge arrows. Further, since r is the minimum among all the positive unused capacities on the forward edges and all the positive ﬂows on the backward edges of the ﬂowaugmenting path, the new ﬂow will satisfy the capacity constraints as well. Finally, adding r to the ﬂow on the ﬁrst edge of the augmenting path will increase the value of the ﬂow by r. Under the assumption that all the edge capacities are integers, r will be a positive integer too. Hence, the ﬂow value increases at least by 1 on each iteration of the augmentingpath method. Since the value of a maximum ﬂow is bounded above (e.g., by the sum of the capacities of the source edges), the augmentingpath method has to stop after a ﬁnite number of iterations.4 Surprisingly, the ﬁnal ﬂow always turns out to be maximal, irrespective of a sequence of augmenting paths. This remarkable result stems from the proof of the MaxFlow MinCut Theorem (see, e.g., [For62]), which we replicate later in this section. The augmentingpath method—as described above in its general form—does not indicate a speciﬁc way for generating ﬂowaugmenting paths. A bad sequence of such paths may, however, have a dramatic impact on the method’s efﬁciency. Consider, for example, the network in Figure 10.6a, in which U stands for some large positive integer. If we augment the zero ﬂow along the path 1→2→3→4, we shall obtain the ﬂow of value 1 shown in Figure 10.6b. Augmenting that ﬂow along the path 1→3←2→4 will increase the ﬂow value to 2 (Figure 10.6c). If we continue selecting this pair of ﬂowaugmenting paths, we will need a total of 2U iterations to reach the maximum ﬂow of value 2U (Figure 10.6d). Of course, we can obtain the maximum ﬂow in just two iterations by augmenting the initial zero ﬂow along the path 1→2→4 followed by augmenting the new ﬂow along the path 1→3→4. The dramatic difference between 2U and 2 iterations makes the point. Fortunately, there are several ways to generate ﬂowaugmenting paths efﬁciently and avoid the degradation in performance illustrated by the previous example. The simplest of them uses breadthﬁrst search to generate augmenting paths with the least number of edges (see Section 3.5). This version of the augmentingpath method, called shortestaugmentingpath or ﬁrstlabeledﬁrstscanned algorithm, was suggested by J. Edmonds and R. M. Karp [Edm72]. The labeling refers to marking a new (unlabeled) vertex with two labels. The ﬁrst label indicates the amount of additional ﬂow that can be brought from the source to the vertex being labeled. The second label is the name of the vertex from which the vertex being labeled was reached. (It can be left undeﬁned for the source.) It is also convenient to add the + or − sign to the second label to indicate whether the vertex was reached via a forward or backward edge, respectively. The source can be always labeled with ∞, −. For the other vertices, the labels are computed as follows. 4.
If capacity upper bounds are irrational numbers, the augmentingpath method may not terminate (see, e.g., [Chv83, pp. 387–388], for a cleverly devised example demonstrating such a situation). This limitation is only of theoretical interest because we cannot store irrational numbers in a computer, and rational numbers can be transformed into integers by changing the capacity measurement unit.
366
Iterative Improvement
2
2
0/U
0/U 0/1
1 0/U
1/U 4
0/U
0/U 1/1
1 0/U
1/U
3
3
(a)
(b)
2
2
1/U
1/U 0/1
1 1/U
U /U 4
1/U
4
U /U 0/1
1
U /U
4
U /U
3
3
(c)
(d)
FIGURE 10.6 Efﬁciency degradation of the augmentingpath method.
If unlabeled vertex j is connected to the front vertex i of the traversal queue by a directed edge from i to j with positive unused capacity rij = uij − xij , then vertex j is labeled with lj , i +, where lj = min{li , rij }. If unlabeled vertex j is connected to the front vertex i of the traversal queue by a directed edge from j to i with positive ﬂow xj i , then vertex j is labeled with lj , i −, where lj = min{li , xj i }. If this labelingenhanced traversal ends up labeling the sink, the current ﬂow can be augmented by the amount indicated by the sink’s ﬁrst label. The augmentation is performed along the augmenting path traced by following the vertex second labels from sink to source: the current ﬂow quantities are increased on the forward edges and decreased on the backward edges of this path. If, on the other hand, the sink remains unlabeled after the traversal queue becomes empty, the algorithm returns the current ﬂow as maximum and stops. ALGORITHM ShortestAugmentingPath(G) //Implements the shortestaugmentingpath algorithm //Input: A network with single source 1, single sink n, and // positive integer capacities uij on its edges (i, j ) //Output: A maximum ﬂow x assign xij = 0 to every edge (i, j ) in the network label the source with ∞, − and add the source to the empty queue Q
10.2
The MaximumFlow Problem
367
while not Empty(Q) do i ← Front(Q); Dequeue(Q) for every edge from i to j do //forward edges if j is unlabeled rij ← uij − xij if rij > 0 lj ← min{li , rij }; label j with lj , i + Enqueue(Q, j ) for every edge from j to i do //backward edges if j is unlabeled if xj i > 0 lj ← min{li , xj i }; label j with lj , i − Enqueue(Q, j ) if the sink has been labeled //augment along the augmenting path found j ← n //start at the sink and move backwards using second labels while j = 1 //the source hasn’t been reached if the second label of vertex j is i + xij ← xij + ln else //the second label of vertex j is i − xj i ← xj i − ln j ← i; i ← the vertex indicated by i’s second label erase all vertex labels except the ones of the source reinitialize Q with the source return x //the current ﬂow is maximum An application of this algorithm to the network in Figure 10.4 is illustrated in Figure 10.7. The optimality of a ﬁnal ﬂow obtained by the augmentingpath method stems from a theorem that relates network ﬂows to network cuts. A cut induced by partitioning vertices of a network into some subset X containing the source and ¯ the complement of X, containing the sink is the set of all the edges with a tail X, ¯ We denote a cut C(X, X) ¯ or simply C. For example, for the in X and a head in X. network in Figure 10.4: ¯ = {(1, 2), (1, 4)}; if X = {1} and hence X¯ = {2, 3, 4, 5, 6}, C(X, X) ¯ = {(3, 6), (5, 6)}; if X = {1, 2, 3, 4, 5} and hence X¯ = {6}, C(X, X) ¯ = {(2, 3), (2, 5), (4, 3)}. if X = {1, 2, 4} and hence X¯ = {3, 5, 6}, C(X, X) The name “cut” stems from the following property: if all the edges of a cut were deleted from the network, there would be no directed path from source to ¯ be a cut. Consider a directed path from source to sink. If sink. Indeed, let C(X, X) vi is the ﬁrst vertex of that path which belongs to X¯ (the set of such vertices is not
368
Iterative Improvement 2, 2+ 5
5 0/3 1
0/2
2
0/4
0/5
0/3
3
0/2
0/3 6
∞, – 1
0/1
0/2
2, 1+ 2
0/5
0/3
4
0/4
Augment the flow by 2 (the sink’s first label) along the path 1 → 2 → 3 → 6. 1, 2+ 5 0/4
0/3 2
2/5
0/3
3
2/2
0/3 6
∞, – 1
0/1
2/2
1, 3– 2
0/4
2/5
0/3
4
2/2
2
1/3
1/1 4
Queue: 1 4 ↑ ↑
3
6 1, 5+
Augment the flow by 1 (the sink’s first label) along the path 1 → 4 → 3 ← 2 → 5 → 6. 5 1/4
1/5
2/2
0/1
5 1/3
3 1, 4+
4 3, 1+
Queue: 1 4 3 2 5 6 ↑ ↑ ↑ ↑ ↑
1
6 2, 3+
0/1
5
2/2
0/2
4 3, 1+
Queue: 1 2 4 3 5 6 ↑ ↑ ↑ ↑
1
3 2, 2+
2/2
1/4
1/3 6
∞, – 1
2/2
2
1/5
3
2/2
6
1/1
1/3 4 2, 1+
No augmenting path (the sink is unlabeled); the current flow is maximal.
FIGURE 10.7 Illustration of the shortestaugmentingpath algorithm. The diagrams on the left show the current ﬂow before the next iteration begins; the diagrams on the right show the results of the vertex labeling on that iteration, the augmenting path found (in bold), and the ﬂow before its augmentation. Vertices deleted from the queue are indicated by the ↑ symbol.
10.2
The MaximumFlow Problem
369
empty, because it contains the sink), then vi is not the source and its immediate predecessor vi−1 on that path belongs to X. Hence, the edge from vi−1 to vi must ¯ This proves the property in question. be an element of the cut C(X, X). ¯ denoted c(X, X), ¯ is deﬁned as the sum of The capacity of a cut C(X, X), capacities of the edges that compose the cut. For the three examples of cuts given above, the capacities are equal to 5, 6, and 9, respectively. Since the number of different cuts in a network is nonempty and ﬁnite (why?), there always exists a minimum cut, i.e., a cut with the smallest capacity. (What is a minimum cut in the network of Figure 10.4?) The following theorem establishes an important relationship between the notions of maximum ﬂow and minimum cut. THEOREM (MaxFlow MinCut Theorem) The value of a maximum ﬂow in a network is equal to the capacity of its minimum cut. ¯ be a cut of PROOF First, let x be a feasible ﬂow of value v and let C(X, X) capacity c in the same network. Consider the ﬂow across this cut deﬁned as the difference between the sum of the ﬂows on the edges from X to X¯ and the sum of the ﬂows on the edges from X¯ to X. It is intuitively clear and can be formally derived from the equations expressing the ﬂowconservation requirement and the deﬁnition of the ﬂow value (Problem 6b in this section’s exercises) that the ﬂow ¯ is equal to v, the value of the ﬂow: across the cut C(X, X)
v= xij − xj i . (10.12) i∈X, j ∈X¯
¯ i∈X j ∈X,
Since the second sum is nonnegative and the ﬂow xij on any edge (i, j ) cannot exceed the edge capacity uij , equality (10.12) implies that
xij ≤ uij , v≤ i∈X, j ∈X¯
i∈X, j ∈X¯
i.e., v ≤ c.
(10.13)
Thus, the value of any feasible ﬂow in a network cannot exceed the capacity of any cut in that network. Let v ∗ be the value of a ﬁnal ﬂow x ∗ obtained by the augmentingpath method. If we now ﬁnd a cut whose capacity is equal to v ∗, we will have to conclude, in view of inequality (10.13), that (i) the value v ∗ of the ﬁnal ﬂow is maximal among all feasible ﬂows, (ii) the cut’s capacity is minimal among all cuts in the network, and (iii) the maximumﬂow value is equal to the minimumcut capacity. To ﬁnd such a cut, consider the set of vertices X ∗ that can be reached from the source by following an undirected path composed of forward edges with positive unused capacities (with respect to the ﬁnal ﬂow x ∗) and backward edges with positive ﬂows on them. This set contains the source but does not contain the sink: if it did, we would have an augmenting path for the ﬂow x ∗, which would
370
Iterative Improvement
contradict the assumption that the ﬂow x ∗ is ﬁnal. Consider the cut C(X ∗, X ∗). By the deﬁnition of set X ∗, each edge (i, j ) from X ∗ to X ∗ has zero unused capacity, i.e., xij∗ = uij , and each edge (j, i) from X ∗ to X ∗ has the zero ﬂow on it (otherwise, j would be in X ∗). Applying equality (10.12) to the ﬁnal ﬂow x ∗ and the set X ∗ deﬁned above, we obtain v∗ =
xij∗ −
i∈X ∗ , j ∈X ∗
j ∈X ∗ , i∈X ∗
xj∗i =
uij − 0 = c(X ∗, X ∗),
i∈X ∗ , j ∈X ∗
which proves the theorem. The proof outlined above accomplishes more than proving the equality of the maximumﬂow value and the minimumcut capacity. It also implies that when the augmentingpath method terminates, it yields both a maximum ﬂow and a minimum cut. If labeling of the kind utilized in the shortestaugmentingpath algorithm is used, a minimum cut is formed by the edges from the labeled to unlabeled vertices on the last iteration of the method. Finally, the proof implies that all such edges must be full (i.e., the ﬂows must be equal to the edge capacities), and all the edges from unlabeled vertices to labeled, if any, must be empty (i.e., have zero ﬂows on them). In particular, for the network in Figure 10.7, the algorithm ﬁnds the cut {(1, 2), (4, 3)} of minimum capacity 3, both edges of which are full as required. Edmonds and Karp proved in their paper [Edm72] that the number of augmenting paths needed by the shortestaugmentingpath algorithm never exceeds nm/2, where n and m are the number of vertices and edges, respectively. Since the time required to ﬁnd a shortest augmenting path by breadthﬁrst search is in O(n + m) = O(m) for networks represented by their adjacency lists, the time efﬁciency of the shortestaugmentingpath algorithm is in O(nm2 ). More efﬁcient algorithms for the maximumﬂow problem are known (see the monograph [Ahu93], as well as appropriate chapters in such books as [Cor09] and [Kle06]). Some of them implement the augmentingpath idea in a more efﬁcient manner. Others are based on the concept of preﬂows. A preﬂow is a ﬂow that satisﬁes the capacity constraints but not the ﬂowconservation requirement. Any vertex is allowed to have more ﬂow entering the vertex than leaving it. A preﬂowpush algorithm moves the excess ﬂow toward the sink until the ﬂowconservation requirement is reestablished for all intermediate vertices of the network. Faster algorithms of this kind have worstcase efﬁciency close to O(nm). Note that preﬂowpush algorithms fall outside the iterativeimprovement paradigm because they do not generate a sequence of improving solutions that satisfy all the constraints of the problem. To conclude this section, it is worth pointing out that although the initial interest in studying network ﬂows was caused by transportation applications, this model has also proved to be useful for many other areas. We discuss one of them in the next section.
10.2
The MaximumFlow Problem
371
Exercises 10.2 1. Since maximumﬂow algorithms require processing edges in both directions, it is convenient to modify the adjacency matrix representation of a network as follows. If there is a directed edge from vertex i to vertex j of capacity uij , then the element in the ith row and the j th column is set to uij , and the element in the j th row and the ith column is set to −uij ; if there is no edge between vertices i and j , both these elements are set to zero. Outline a simple algorithm for identifying a source and a sink in a network presented by such a matrix and indicate its time efﬁciency. 2. Apply the shortestaugmenting path algorithm to ﬁnd a maximum ﬂow and a minimum cut in the following networks. a. 5 2 1
2
6
5
4 3
b.
4 4
7
8
3
2
4
2
1
1
4 7
6
3
6
4 2
5
5
3. a. Does the maximumﬂow problem always have a unique solution? Would your answer be different for networks with different capacities on all their edges? b. Answer the same questions for the minimumcut problem of ﬁnding a cut of the smallest capacity in a given network. 4. a. Explain how the maximumﬂow problem for a network with several sources and sinks can be transformed into the same problem for a network with a single source and a single sink. b. Some networks have capacity constraints on the ﬂow amounts that can ﬂow through their intermediate vertices. Explain how the maximumﬂow problem for such a network can be transformed to the maximumﬂow problem for a network with edge capacity constraints only. 5. Consider a network that is a rooted tree, with the root as its source, the leaves as its sinks, and all the edges directed along the paths from the root to the leaves. Design an efﬁcient algorithm for ﬁnding a maximum ﬂow in such a network. What is the time efﬁciency of your algorithm? 6. a. Prove equality (10.9).
372
Iterative Improvement
b. Prove that for any ﬂow in a network and any cut in it, the value of the ﬂow is equal to the ﬂow across the cut (see equality (10.12)). Explain the relationship between this property and equality (10.9). 7. a. Express the maximumﬂow problem for the network in Figure 10.4 as a linear programming problem. b. Solve this linear programming problem by the simplex method. 8. As an alternative to the shortestaugmentingpath algorithm, Edmonds and Karp [Edm72] suggested the maximumcapacityaugmentingpath algorithm, in which a ﬂow is augmented along the path that increases the ﬂow by the largest amount. Implement both these algorithms in the language of your choice and perform an empirical investigation of their relative efﬁciency. 9. Write a report on a more advanced maximumﬂow algorithm such as (i) Dinitz’s algorithm, (ii) Karzanov’s algorithm, (iii) MalhotraKamarMaheshwari algorithm, or (iv) GoldbergTarjan algorithm. 10. Dining problem Several families go out to dinner together. To increase their social interaction, they would like to sit at tables so that no two members of the same family are at the same table. Show how to ﬁnd a seating arrangement that meets this objective (or prove that no such arrangement exists) by using a maximumﬂow problem. Assume that the dinner contingent has p families and that the ith family has ai members. Also assume that q tables are available and the j th table has a seating capacity of bj . [Ahu93]
10.3 Maximum Matching in Bipartite Graphs In many situations we are faced with a problem of pairing elements of two sets. The traditional example is boys and girls for a dance, but you can easily think of more serious applications. It is convenient to represent elements of two given sets by vertices of a graph, with edges between vertices that can be paired. A matching in a graph is a subset of its edges with the property that no two edges share a vertex. A maximum matching—more precisely, a maximum cardinality matching—is a matching with the largest number of edges. (What is it for the graph in Figure 10.8? Is it unique?) The maximummatching problem is the problem of ﬁnding a maximum matching in a given graph. For an arbitrary graph, this is a rather difﬁcult problem. It was solved in 1965 by Jack Edmonds [Edm65]. (See [Gal86] for a good survey and more recent references.) We limit our discussion in this section to the simpler case of bipartite graphs. In a bipartite graph, all the vertices can be partitioned into two disjoint sets V and U , not necessarily of the same size, so that every edge connects a vertex in one of these sets to a vertex in the other set. In other words, a graph is bipartite if its vertices can be colored in two colors so that every edge has its vertices colored in different colors; such graphs are also said to be 2colorable. The graph in Figure 10.8 is bipartite. It is not difﬁcult to prove that a graph is bipartite if and only if it does not have a cycle of an odd length. We will assume for the rest of this section that
10.3 Maximum Matching in Bipartite Graphs
V
1
2
3
4
U
5
6
7
8
373
FIGURE 10.8 Example of a bipartite graph.
the vertex set of a given bipartite graph has been already partitioned into sets V and U as required by the deﬁnition (see Problem 8 in Exercises 3.5). Let us apply the iterativeimprovement technique to the maximumcardinalitymatching problem. Let M be a matching in a bipartite graph G = V , U, E . How can we improve it, i.e., ﬁnd a new matching with more edges? Obviously, if every vertex in either V or U is matched (has a mate), i.e., serves as an endpoint of an edge in M, this cannot be done and M is a maximum matching. Therefore, to have a chance at improving the current matching, both V and U must contain unmatched (also called free) vertices, i.e., vertices that are not incident to any edge in M. For example, for the matching Ma = {(4, 8), (5, 9)} in the graph in Figure 10.9a, vertices 1, 2, 3, 6, 7, and 10 are free, and vertices 4, 5, 8, and 9 are matched. Another obvious observation is that we can immediately increase a current matching by adding an edge between two free vertices. For example, adding (1, 6) to the matching Ma = {(4, 8), (5, 9)} in the graph in Figure 10.9a yields a larger matching Mb = {(1, 6), (4, 8), (5, 9)} (Figure 10.9b). Let us now try to ﬁnd a matching larger than Mb by matching vertex 2. The only way to do this would be to include the edge (2, 6) in a new matching. This inclusion requires removal of (1, 6), which can be compensated by inclusion of (1, 7) in the new matching. This new matching Mc = {(1, 7), (2, 6), (4, 8), (5, 9)} is shown in Figure 10.9c. In general, we increase the size of a current matching M by constructing a simple path from a free vertex in V to a free vertex in U whose edges are alternately in E − M and in M. That is, the ﬁrst edge of the path does not belong to M, the second one does, and so on, until the last edge that does not belong to M. Such a path is called augmenting with respect to the matching M. For example, the path 2, 6, 1, 7 is an augmenting path with respect to the matching Mb in Figure 10.9b. Since the length of an augmenting path is always odd, adding to the matching M the path’s edges in the oddnumbered positions and deleting from it the path’s edges in the evennumbered positions yields a matching with one more edge than in M. Such a matching adjustment is called augmentation. Thus, in Figure 10.9, the matching Mb was obtained by augmentation of the matching Ma along the augmenting path 1, 6, and the matching Mc was obtained by augmentation of the matching Mb along the augmenting path 2, 6, 1, 7. Moving further, 3, 8, 4, 9, 5, 10 is an augmenting path for the matching Mc (Figure 10.9c). After adding to Mc the edges (3, 8), (4, 9), and (5, 10) and deleting (4, 8) and (5, 9), we obtain the matching Md = {(1, 7), (2, 6), (3, 8), (4, 9), (5, 10)} shown in Figure 10.9d. The
374
Iterative Improvement
V
1
2
3
4
5
U
6
7
8
9
10
(a)
Augmenting path: 1, 6 1
2
3
4
5
6
7
8
9
10
(b)
Augmenting path: 2, 6, 1, 7 1
2
3
4
5
6
7
8
9
10
(c)
Augmenting path: 3, 8, 4, 9, 5, 10 1
2
3
4
5
6
7
8
9
10
(d)
Maximum matching FIGURE 10.9 Augmenting paths and matching augmentations.
10.3 Maximum Matching in Bipartite Graphs
375
matching Md is not only a maximum matching but also perfect, i.e., a matching that matches all the vertices of the graph. Before we discuss an algorithm for ﬁnding an augmenting path, let us settle the issue of what nonexistence of such a path means. According to the theorem discovered by the French mathematician Claude Berge, it means the current matching is maximal. THEOREM A matching M is a maximum matching if and only if there exists no augmenting path with respect to M. PROOF If an augmenting path with respect to a matching M exists, then the size of the matching can be increased by augmentation. Let us prove the more difﬁcult part: if no augmenting path with respect to a matching M exists, then the matching is a maximum matching. Assume that, on the contrary, this is not the case for a certain matching M in a graph G. Let M ∗ be a maximum matching in G; by our assumption, the number of edges in M ∗ is at least one more than the number of edges in M, i.e., M ∗ > M. Consider the edges in the symmetric difference M ⊕ M ∗ = (M − M ∗) ∪ (M ∗ − M), the set of all the edges that are either in M or in M ∗ but not in both. Note that M ∗ − M > M − M ∗ because M ∗ > M by assumption. Let G be the subgraph of G made up of all the edges in M ⊕ M ∗ and their endpoints. By deﬁnition of a matching, any vertex in G ⊆ G can be incident to no more than one edge in M and no more than one edge in M ∗. Hence, each of the vertices in G has degree 2 or less, and therefore every connected component of G is either a path or an evenlength cycle of alternating edges from M − M ∗ and M ∗ − M. Since M ∗ − M > M − M ∗ and the number of edges from M − M ∗ and M ∗ − M is the same for any evenlength cycle of alternating edges in G , there must exist at least one path of alternating edges that starts and ends with an edge from M ∗ − M. Hence, this is an augmenting path for the matching M, which contradicts the assumption that no such path exists. Our discussion of augmenting paths leads to the following general method for constructing a maximum matching in a bipartite graph. Start with some initial matching (e.g., the empty set). Find an augmenting path and augment the current matching along this path. When no augmenting path can be found, terminate the algorithm and return the last matching, which is maximum. We now give a speciﬁc algorithm implementing this general template. We will search for an augmenting path for a matching M by a BFSlike traversal of the graph that starts simultaneously at all the free vertices in one of the sets V and U, say, V . (It would be logical to select the smaller of the two vertex sets, but we will ignore this observation in the pseudocode below.) Recall that an augmenting path, if it exists, is an oddlength path that connects a free vertex in V with a free vertex in U and which, unless it consists of a single edge, “zigs” from a vertex in V to another vertex’ mate in U , then “zags” back to V along the uniquely deﬁned edge from M, and so on until a free vertex in U is reached. (Draw augmenting paths for the matchings in Figure 10.9, for example.) Hence, any candidate to be such a
376
Iterative Improvement
path must have its edges alternate in the pattern just described. This motivates the following rules for labeling vertices during the BFSlike traversal of the graph. Case 1 (the queue’s front vertex w is in V ) If u is a free vertex adjacent to w, it is used as the other endpoint of an augmenting path; so the labeling stops and augmentation of the matching commences. The augmenting path in question is obtained by moving backward along the vertex labels (see below) to alternately add and delete its edges to and from the current matching. If u is not free and connected to w by an edge not in M, label u with w unless it has been already labeled. Case 2 (the front vertex w is in U ) In this case, w must be matched and we label its mate in V with w. Here is pseudocode of the algorithm in its entirety. ALGORITHM MaximumBipartiteMatching(G) //Finds a maximum matching in a bipartite graph by a BFSlike traversal //Input: A bipartite graph G = V , U, E //Output: A maximumcardinality matching M in the input graph initialize set M of edges with some valid matching (e.g., the empty set) initialize queue Q with all the free vertices in V (in any order) while not Empty(Q) do w ← Front(Q); Dequeue(Q) if w ∈ V for every vertex u adjacent to w do if u is free //augment M ← M ∪ (w, u) v←w while v is labeled do u ← vertex indicated by v’s label; M ← M − (v, u) v ← vertex indicated by u’s label; M ← M ∪ (v, u) remove all vertex labels reinitialize Q with all free vertices in V break //exit the for loop else //u is matched if (w, u) ∈ M and u is unlabeled label u with w Enqueue(Q, u) else //w ∈ U (and matched) label the mate v of w with w Enqueue(Q, v) return M //current matching is maximum
377
10.3 Maximum Matching in Bipartite Graphs
An application of this algorithm to the matching in Figure 10.9a is shown in Figure 10.10. Note that the algorithm ﬁnds a maximum matching that differs from the one in Figure 10.9d.
V
1
2
3
4
5
1
2
3
4
5
U
6
7
8
9
10
6
7
8
9
10
1
Queue: 1 2 3
Queue: 1 2 3 ↑ Augment from 6 6
8
1
2
3
4
5
1
2
6
7
8
9
10
6 2
Queue: 2 3
3
4
5
7
8
9
10
1
3
Queue: 2 3 6 8 1 4 ↑ ↑ ↑ ↑ ↑ Augment from 7 6
1
2
3
4
5
1
2
6
7
8
9
10
6
7
8 3
3
Queue: 3
4
5
8
9
10
3
4
4
Queue: 3 6 8 2 4 9 ↑ ↑ ↑ ↑ ↑ Augment from 10
1
2
3
4
5
6
7
8
9
10
Queue: empty ⇒ maximum matching
FIGURE 10.10 Application of the maximumcardinalitymatching algorithm. The left column shows a current matching and initialized queue at the next iteration’s start; the right column shows the vertex labeling generated by the algorithm before augmentation is performed. Matching edges are shown in bold. Vertex labels indicate the vertices from which the labeling is done. The discovered endpoint of an augmenting path is shaded and labeled for clarity. Vertices deleted from the queue are indicated by ↑.
378
Iterative Improvement
How efﬁcient is the maximummatching algorithm? Each iteration except the last one matches two previously free vertices—one from each of the sets V and U. Therefore, the total number of iterations cannot exceed n/2 + 1, where n = V  + U  is the number of vertices in the graph. The time spent on each iteration is in O(n + m), where m = E is the number of edges in the graph. (This assumes that the information about the status of each vertex—free or matched and the vertex’ mate if the latter—can be retrieved in constant time, e.g., by storing it in an array.) Hence, the time efﬁciency of the algorithm is in O(n(n + m)). √ Hopcroft and Karp [Hop73] showed how the efﬁciency can be improved to O( n(n + m)) by combining several iterations into a single stage to maximize the number of edges added to the matching with one search. We were concerned in this section with matching the largest possible number of vertex pairs in a bipartite graph. Some applications may require taking into account the quality or cost of matching different pairs. For example, workers may execute jobs with different efﬁciencies, or girls may have different preferences for their potential dance partners. It is natural to model such situations by bipartite graphs with weights assigned to their edges. This leads to the problem of maximizing the sum of the weights on edges connecting matched pairs of vertices. This problem is called maximumweight matching. We encountered it under a different name—the assignment problem—in Section 3.4. There are several sophisticated algorithms for this problem, which are much more efﬁcient than exhaustive search (see, e.g., [Pap82], [Gal86], [Ahu93]). We have to leave them outside of our discussion, however, because of their complexity, especially for general graphs.
Exercises 10.3 1. For each matching shown below in bold, ﬁnd an augmentation or explain why no augmentation exists. a.
b.
1
2
5
3
6
4
7
1
2
3
4
5
6
7
8
2. Apply the maximummatching algorithm to the following bipartite graph: 1
2
3
4
5
6
10.3 Maximum Matching in Bipartite Graphs
379
3. a. What is the largest and what is the smallest possible cardinality of a matching in a bipartite graph G = V , U, E with n vertices in each vertex set V and U and at least n edges? b. What is the largest and what is the smallest number of distinct solutions the maximumcardinalitymatching problem can have for a bipartite graph G = V , U, E with n vertices in each vertex set V and U and at least n edges? 4. a. Hall’s Marriage Theorem asserts that a bipartite graph G = V , U, E has a matching that matches all vertices of the set V if and only if for each subset S ⊆ V , R(S) ≥ S where R(S) is the set of all vertices adjacent to a vertex in S. Check this property for the following graph with (i) V = {1, 2, 3, 4} and (ii) V = {5, 6, 7}. 1
2
5
3
6
4
7
b. You have to devise an algorithm that returns yes if there is a matching in a bipartite graph G = V , U, E that matches all vertices in V and returns no otherwise. Would you base your algorithm on checking the condition of Hall’s Marriage Theorem? 5. Suppose there are ﬁve committees A, B, C, D, and E composed of six persons a, b, c, d, e, and f as follows: committee A’s members are b and e; committee B’s members are b, d, and e; committee C’s members are a, c, d, e, and f ; committee D’s members are b, d, and e; committee E’s members are b and e. Is there a system of distinct representatives, i.e., is it possible to select a representative from each committee so that all the selected persons are distinct? 6. Show how the maximumcardinalitymatching problem for a bipartite graph can be reduced to the maximumﬂow problem discussed in Section 10.2. 7. Consider the following greedy algorithm for ﬁnding a maximum matching in a bipartite graph G = V , U, E . Sort all the vertices in nondecreasing order of their degrees. Scan this sorted list to add to the current matching (initially empty) the edge from the list’s free vertex to an adjacent free vertex of the lowest degree. If the list’s vertex is matched or if there are no adjacent free vertices for it, the vertex is simply skipped. Does this algorithm always produce a maximum matching in a bipartite graph? 8. Design a lineartime algorithm for ﬁnding a maximum matching in a tree. 9. Implement the maximummatching algorithm of this section in the language of your choice. Experiment with its performance on bipartite graphs with n vertices in each of the vertex sets and randomly generated edges (in both
380
Iterative Improvement
dense and sparse modes) to compare the observed running time with the algorithm’s theoretical efﬁciency. 10. Domino puzzle A domino is a 2 × 1 tile that can be oriented either horizontally or vertically. A tiling of a given board composed of 1 × 1 squares is covering it with dominoes exactly and without overlap. Is it possible to tile with dominoes an 8 × 8 board without two unit squares at its diagonally opposite corners?
10.4 The Stable Marriage Problem In this section, we consider an interesting version of bipartite matching called the stable marriage problem. Consider a set Y = {m1, m2 , . . . , mn} of n men and a set X = {w1, w2 , . . . , wn} of n women. Each man has a preference list ordering the women as potential marriage partners with no ties allowed. Similarly, each woman has a preference list of the men, also with no ties. Examples of these two sets of lists are given in Figures 10.11a and 10.11b. The same information can also be presented by an n × n ranking matrix (see Figure 10.11c). The rows and columns of the matrix represent the men and women of the two sets, respectively. A cell in row m and column w contains two rankings: the ﬁrst is the position (ranking) of w in the m’s preference list; the second is the position (ranking) of m in the w’s preference list. For example, the pair 3, 1 in Jim’s row and Ann’s column in the matrix in Figure 10.11c indicates that Ann is Jim’s third choice while Jim is Ann’s ﬁrst. Which of these two ways to represent such information is better depends on the task at hand. For example, it is easier to specify a match of the sets’ elements by using the ranking matrix, whereas the preference lists might be a more efﬁcient data structure for implementing a matching algorithm. A marriage matching M is a set of n (m, w) pairs whose members are selected from disjoint nelement sets Y and X in a oneone fashion, i.e., each man m from Y is paired with exactly one woman w from X and vice versa. (If we represent Y and X as vertices of a complete bipartite graph with edges connecting possible marriage partners, then a marriage matching is a perfect matching in such a graph.)
men’s preferences
women’s preferences
1st 2nd Lea Ann Lea Sue Sue Lea (a)
1st 2nd Jim Tom Tom Bob Jim Tom (b)
Bob: Jim: Tom:
3rd Sue Ann Ann
Ann: Lea: Sue:
3rd Bob Jim Bob
ranking matrix Bob Jim Tom
Ann 2,3 3,1 3,2 (c)
Lea 1,2 1,3 2,1
Sue 3,3 2,1 1,2
FIGURE 10.11 Data for an instance of the stable marriage problem. (a) Men’s preference lists; (b) women’s preference lists. (c) Ranking matrix (with the boxed cells composing an unstable matching).
10.4
The Stable Marriage Problem
381
A pair (m, w), where m ∈ Y, w ∈ X, is said to be a blocking pair for a marriage matching M if man m and woman w are not matched in M but they prefer each other to their mates in M. For example, (Bob, Lea) is a blocking pair for the marriage matching M = {(Bob, Ann), (Jim, Lea), (Tom, Sue)} (Figure 10.11c) because they are not matched in M while Bob prefers Lea to Ann and Lea prefers Bob to Jim. A marriage matching M is called stable if there is no blocking pair for it; otherwise, M is called unstable. According to this deﬁnition, the marriage matching in Figure 10.11c is unstable because Bob and Lea can drop their designated mates to join in a union they both prefer. The stable marriage problem is to ﬁnd a stable marriage matching for men’s and women’s given preferences. Surprisingly, this problem always has a solution. (Can you ﬁnd it for the instance in Figure 10.11?) It can be found by the following algorithm. Stable marriage algorithm Input: A set of n men and a set of n women along with rankings of the women by each man and rankings of the men by each woman with no ties allowed in the rankings Output: A stable marriage matching Step 0 Start with all the men and women being free. Step 1 While there are free men, arbitrarily select one of them and do the following: Proposal The selected free man m proposes to w, the next woman on his preference list (who is the highestranked woman who has not rejected him before). Response If w is free, she accepts the proposal to be matched with m. If she is not free, she compares m with her current mate. If she prefers m to him, she accepts m’s proposal, making her former mate free; otherwise, she simply rejects m’s proposal, leaving m free. Step 2 Return the set of n matched pairs. Before we analyze this algorithm, it is useful to trace it on some input. Such an example is presented in Figure 10.12. Let us discuss properties of the stable marriage algorithm. THEOREM The stable marriage algorithm terminates after no more than n2 iterations with a stable marriage output. PROOF The algorithm starts with n men having the total of n2 women on their ranking lists. On each iteration, one man makes a proposal to a woman. This reduces the total number of women to whom the men can still propose in the future because no man proposes to the same woman more than once. Hence, the algorithm must stop after no more than n2 iterations.
382
Iterative Improvement
Free men: Bob, Jim, Tom
Bob Jim Tom
Ann 2, 3 3, 1 3, 2
Lea 1,2 1, 3 2, 1
Sue 3, 3 2, 1 1, 2
Free men: Jim, Tom
Bob Jim Tom
Ann 2, 3 3, 1 3, 2
Lea Sue 1,2 3, 3 1, 3 2, 1 2, 1 1, 2
Free men: Jim, Tom
Bob Jim Tom
Ann 2, 3 3, 1 3, 2
Lea 1,2 1, 3 2, 1
Sue 3, 3 2,1 1, 2
Free men: Tom
Bob Jim Tom
Ann 2, 3 3, 1 3, 2
Lea 1,2 1, 3 2, 1
Sue 3, 3 2,1 1, 2
Free men: Tom
Bob Jim Tom
Ann 2, 3 3, 1 3, 2
Lea 1, 2 1, 3 2,1
Sue 3, 3 2,1 1, 2
Free men: Bob
Bob Jim Tom
Ann 2,3 3, 1 3, 2
Lea 1, 2 1, 3 2,1
Sue 3, 3 2,1 1, 2
Bob proposed to Lea Lea accepted
Jim proposed to Lea Lea rejected
Jim proposed to Sue Sue accepted
Tom proposed to Sue Sue rejected
Tom proposed to Lea Lea replaced Bob with Tom
Bob proposed to Ann Ann accepted
FIGURE 10.12 Application of the stable marriage algorithm. An accepted proposal is indicated by a boxed cell; a rejected proposal is shown by an underlined cell.
Let us now prove that the ﬁnal matching M is a stable marriage matching. Since the algorithm stops after all the n men are oneone matched to the n women, the only thing that needs to be proved is the stability of M. Suppose, on the contrary, that M is unstable. Then there exists a blocking pair of a man m and a woman w who are unmatched in M and such that both m and w prefer each other to the persons they are matched with in M. Since m proposes to every woman on his ranking list in decreasing order of preference and w precedes m’s match in M, m must have proposed to w on some iteration. Whether w refused m’s proposal or accepted it but replaced him on a subsequent iteration with a higherranked match, w’s mate in M must be higher on w’s preference list than m because the rankings of the men matched to a given woman may only improve on each iteration of the algorithm. This contradicts the assumption that w prefers m to her ﬁnal match in M. The stable marriage algorithm has a notable shortcoming. It is not “gender neutral.” In the form presented above, it favors men’s preferences over women’s
10.4
The Stable Marriage Problem
383
preferences. We can easily see this by tracing the algorithm on the following instance of the problem: woman 1 1, 2 2, 1
man 1 man 2
woman 2 2, 1 1, 2
The algorithm obviously yields the stable matching M = {(man 1, woman 1), (man 2, woman 2)}. In this matching, both men are matched to their ﬁrst choices, which is not the case for the women. One can prove that the algorithm always yields a stable matching that is manoptimal: it assigns to each man the highestranked woman possible under any stable marriage. Of course, this gender bias can be reversed, but not eliminated, by reversing the roles played by men and women in the algorithm, i.e., by making women propose and men accept or reject their proposals. There is another important corollary to the fact that the stable marriage algorithm always yields a genderoptimal stable matching. It is easy to prove that a man (woman)optimal matching is unique for a given set of participant preferences. Therefore the algorithm’s output does not depend on the order in which the free men (women) make their proposals. Consequently, we can use any data structure we might prefer—e.g., a queue or a stack—for representing this set with no impact on the algorithm’s outcome. The notion of the stable matching as well as the algorithm discussed above was introduced by D. Gale and L. S. Shapley in the paper titled “College Admissions and the Stability of Marriage” [Gal62]. I do not know which of the two applications mentioned in the title you would consider more important. The point is that stability is a matching property that can be desirable in a variety of applications. For example, it has been used for many years in the United States for matching medicalschool graduates with hospitals for residency training. For a brief history of this application and an indepth discussion of the stable marriage problem and its extensions, see the monograph by Gusﬁeld and Irwing [Gus89].
Exercises 10.4 1. Consider an instance of the stable marriage problem given by the following ranking matrix: α β γ
A B C 1, 3 2, 2 3, 1 3, 1 1, 3 2, 2 2, 2 3, 1 1, 3
For each of its marriage matchings, indicate whether it is stable or not. For the unstable matchings, specify a blocking pair. For the stable matchings, indicate whether they are manoptimal, womanoptimal, or neither. (Assume that the Greek and Roman letters denote the men and women, respectively.)
384
Iterative Improvement
2. Design a simple algorithm for checking whether a given marriage matching is stable and determine its time efﬁciency class. 3. Find a stable marriage matching for the instance given in Problem 1 by applying the stable marriage algorithm a. in its menproposing version. b. in its womenproposing version. 4. Find a stable marriage matching for the instance deﬁned by the following ranking matrix: α β γ δ
A B C D 1, 3 2, 3 3, 2 4, 3 1, 4 4, 1 3, 4 2, 2 2, 2 1, 4 3, 3 4, 1 4, 1 2, 2 3, 1 1, 4
5. Determine the timeefﬁciency class of the stable marriage algorithm a. in the worst case. b. in the best case. 6. Prove that a manoptimal stable marriage set is always unique. Is it also true for a womanoptimal stable marriage matching? 7. Prove that in the manoptimal stable matching, each woman has the worst partner that she can have in any stable marriage matching. 8. Implement the stablemarriage algorithm given in Section 10.4 so that its running time is in O(n2 ). Run an experiment to ascertain its averagecase efﬁciency. 9. Write a report on the college admission problem (residentshospitals assignment) that generalizes the stable marriage problem in that a college can accept “proposals” from more than one applicant. 10. Consider the problem of the roommates, which is related to but more difﬁcult than the stable marriage problem: “An even number of boys wish to divide up into pairs of roommates. A set of pairings is called stable if under it there are no two boys who are not roommates and who prefer each other to their actual roommates.” [Gal62] Give an instance of this problem that does not have a stable pairing.
SUMMARY The iterativeimprovement technique involves ﬁnding a solution to an optimization problem by generating a sequence of feasible solutions with improving values of the problem’s objective function. Each subsequent solution in such a sequence typically involves a small, localized change in the previous feasible solution. When no such change improves the value of the
Summary
385
objective function, the algorithm returns the last feasible solution as optimal and stops. Important problems that can be solved exactly by iterativeimprovement algorithms include linear programming, maximizing the ﬂow in a network, and matching the maximum possible number of vertices in a graph. The simplex method is the classic method for solving the general linear programming problem. It works by generating a sequence of adjacent extreme points of the problem’s feasible region with improving values of the objective function. The maximumﬂow problem asks to ﬁnd the maximum ﬂow possible in a network, a weighted directed graph with a source and a sink. The FordFulkerson method is a classic template for solving the maximumﬂow problem by the iterativeimprovement approach. The shortestaugmentingpath method implements this idea by labeling network vertices in the breadthﬁrst search manner. The FordFulkerson method also ﬁnds a minimum cut in a given network. A maximum cardinality matching is the largest subset of edges in a graph such that no two edges share the same vertex. For a bipartite graph, it can be found by a sequence of augmentations of previously obtained matchings. The stable marriage problem is to ﬁnd a stable matching for elements of two nelement sets based on given matching preferences. This problem always has a solution that can be found by the GaleShapley algorithm.
This page intentionally left blank
11 Limitations of Algorithm Power Intellect distinguishes between the possible and the impossible; reason distinguishes between the sensible and the senseless. Even the possible can be senseless. —Max Born (1882–1970), My Life and My Views, 1968
I
n the preceding chapters of this book, we encountered dozens of algorithms for solving a variety of different problems. A fair assessment of algorithms as problemsolving tools is inescapable: they are very powerful instruments, especially when they are executed by modern computers. But the power of algorithms is not unlimited, and its limits are the subject of this chapter. As we shall see, some problems cannot be solved by any algorithm. Other problems can be solved algorithmically but not in polynomial time. And even when a problem can be solved in polynomial time by some algorithms, there are usually lower bounds on their efﬁciency. We start, in Section 11.1, with methods for obtaining lower bounds, which are estimates on a minimum amount of work needed to solve a problem. In general, obtaining a nontrivial lower bound even for a simplesounding problem is a very difﬁcult task. As opposed to ascertaining the efﬁciency of a particular algorithm, the task here is to establish a limit on the efﬁciency of any algorithm, known or unknown. This also necessitates a careful description of the operations such algorithms are allowed to perform. If we fail to deﬁne carefully the “rules of the game,” so to speak, our claims may end up in the large dustbin of impossibilityrelated statements as, for example, the one made by the celebrated British physicist Lord Kelvin in 1895: “Heavierthanair ﬂying machines are impossible.” Section 11.2 discusses decision trees. This technique allows us, among other applications, to establish lower bounds on the efﬁciency of comparisonbased algorithms for sorting and for searching in sorted arrays. As a result, we will be able to answer such questions as whether it is possible to invent a faster sorting algorithm than mergesort and whether binary search is the fastest algorithm for searching in a sorted array. (What does your intuition tell you the answers to these questions will turn out to be?) Incidentally, decision trees are also a great vehicle 387
388
Limitations of Algorithm Power
for directing us to a solution of some puzzles, such as the coinweighing problem discussed in Section 4.4. Section 11.3 deals with the question of intractability: which problems can and cannot be solved in polynomial time. This welldeveloped area of theoretical computer science is called computational complexity theory. We present the basic elements of this theory and discuss informally such fundamental notions as P , NP, and NPcomplete problems, including the most important unresolved question of theoretical computer science about the relationship between P and NP problems. The last section of this chapter deals with numerical analysis. This branch of computer science concerns algorithms for solving problems of “continuous” mathematics—solving equations and systems of equations, evaluating such functions as sin x and ln x, computing integrals, and so on. The nature of such problems imposes two types of limitations. First, most cannot be solved exactly. Second, solving them even approximately requires dealing with numbers that can be represented in a digital computer with only a limited level of precision. Manipulating approximate numbers without proper care can lead to very inaccurate results. We will see that even solving a basic quadratic equation on a computer poses signiﬁcant difﬁculties that require a modiﬁcation of the canonical formula for the equation’s roots.
11.1 LowerBound Arguments We can look at the efﬁciency of an algorithm two ways. We can establish its asymptotic efﬁciency class (say, for the worst case) and see where this class stands with respect to the hierarchy of efﬁciency classes outlined in Section 2.2. For example, selection sort, whose efﬁciency is quadratic, is a reasonably fast algorithm, whereas the algorithm for the Tower of Hanoi problem is very slow because its efﬁciency is exponential. We can argue, however, that this comparison is akin to the proverbial comparison of apples to oranges because these two algorithms solve different problems. The alternative and possibly “fairer” approach is to ask how efﬁcient a particular algorithm is with respect to other algorithms for the same problem. Seen in this light, selection sort has to be considered slow because there are O(n log n) sorting algorithms; the Tower of Hanoi algorithm, on the other hand, turns out to be the fastest possible for the problem it solves. When we want to ascertain the efﬁciency of an algorithm with respect to other algorithms for the same problem, it is desirable to know the best possible efﬁciency any algorithm solving the problem may have. Knowing such a lower bound can tell us how much improvement we can hope to achieve in our quest for a better algorithm for the problem in question. If such a bound is tight, i.e., we already know an algorithm in the same efﬁciency class as the lower bound, we can hope for a constantfactor improvement at best. If there is a gap between the efﬁciency of the fastest algorithm and the best lower bound known, the door for possible improvement remains open: either a faster algorithm matching the lower bound could exist or a better lower bound could be proved.
11.1
LowerBound Arguments
389
In this section, we present several methods for establishing lower bounds and illustrate them with speciﬁc examples. As we did in analyzing the efﬁciency of speciﬁc algorithms in the preceding chapters, we should distinguish between a lowerbound class and a minimum number of times a particular operation needs to be executed. As a rule, the second problem is more difﬁcult than the ﬁrst. For example, we can immediately conclude that any algorithm for ﬁnding the median of n numbers must be in (n) (why?), but it is not simple at all to prove that any comparisonbased algorithm for this problem must do at least 3(n − 1)/2 comparisons in the worst case (for odd n).
Trivial Lower Bounds The simplest method of obtaining a lowerbound class is based on counting the number of items in the problem’s input that must be processed and the number of output items that need to be produced. Since any algorithm must at least “read” all the items it needs to process and “write” all its outputs, such a count yields a trivial lower bound. For example, any algorithm for generating all permutations of n distinct items must be in (n!) because the size of the output is n!. And this bound is tight because good algorithms for generating permutations spend a constant time on each of them except the initial one (see Section 4.3). As another example, consider the problem of evaluating a polynomial of degree n p(x) = anx n + an−1x n−1 + . . . + a0 at a given point x, given its coefﬁcients an, an−1, . . . , a0. It is easy to see that all the coefﬁcients have to be processed by any polynomialevaluation algorithm. Indeed, if it were not the case, we could change the value of an unprocessed coefﬁcient, which would change the value of the polynomial at a nonzero point x. This means that any such algorithm must be in (n). This lower bound is tight because both the righttoleft evaluation algorithm (Problem 2 in Exercises 6.5) and Horner’s rule (Section 6.5) are both linear. In a similar vein, a trivial lower bound for computing the product of two n × n matrices is (n2 ) because any such algorithm has to process 2n2 elements in the input matrices and generate n2 elements of the product. It is still unknown, however, whether this bound is tight. Trivial lower bounds are often too low to be useful. For example, the trivial bound for the traveling salesman problem is (n2 ), because its input is n(n − 1)/2 intercity distances and its output is a list of n + 1 cities making up an optimal tour. But this bound is all but useless because there is no known algorithm with the running time being a polynomial function of any degree. There is another obstacle to deriving a meaningful lower bound by this method. It lies in determining which part of an input must be processed by any algorithm solving the problem in question. For example, searching for an element of a given value in a sorted array does not require processing all its elements (why?). As another example, consider the problem of determining connectivity of
390
Limitations of Algorithm Power
an undirected graph deﬁned by its adjacency matrix. It is plausible to expect that any such algorithm would have to check the existence of each of the n(n − 1)/2 potential edges, but the proof of this fact is not trivial.
InformationTheoretic Arguments While the approach outlined above takes into account the size of a problem’s output, the informationtheoretical approach seeks to establish a lower bound based on the amount of information it has to produce. Consider, as an example, the wellknown game of deducing a positive integer between 1 and n selected by somebody by asking that person questions with yes/no answers. The amount of uncertainty that any algorithm solving this problem has to resolve can be measured by log2 n, the number of bits needed to specify a particular number among the n possibilities. We can think of each question (or, to be more accurate, an answer to each question) as yielding at most 1 bit of information about the algorithm’s output, i.e., the selected number. Consequently, any such algorithm will need at least log2 n such steps before it can determine its output in the worst case. The approach we just exploited is called the informationtheoretic argument because of its connection to information theory. It has proved to be quite useful for ﬁnding the socalled informationtheoretic lower bounds for many problems involving comparisons, including sorting and searching. Its underlying idea can be realized much more precisely through the mechanism of decision trees. Because of the importance of this technique, we discuss it separately and in more detail in Section 11.2.
Adversary Arguments Let us revisit the same game of “guessing” a number used to introduce the idea of an informationtheoretic argument. We can prove that any algorithm that solves this problem must ask at least log2 n questions in its worst case by playing the role of a hostile adversary who wants to make an algorithm ask as many questions as possible. The adversary starts by considering each of the numbers between 1 and n as being potentially selected. (This is cheating, of course, as far as the game is concerned, but not as a way to prove our assertion.) After each question, the adversary gives an answer that leaves him with the largest set of numbers consistent with this and all the previously given answers. This strategy leaves him with at least onehalf of the numbers he had before his last answer. If an algorithm stops before the size of the set is reduced to 1, the adversary can exhibit a number that could be a legitimate input the algorithm failed to identify. It is a simple technical matter now to show that one needs log2 n iterations to shrink an nelement set to a oneelement set by halving and rounding up the size of the remaining set. Hence, at least log2 n questions need to be asked by any algorithm in the worst case. This example illustrates the adversary method for establishing lower bounds. It is based on following the logic of a malevolent but honest adversary: the malev
11.1
LowerBound Arguments
391
olence makes him push the algorithm down the most timeconsuming path, and his honesty forces him to stay consistent with the choices already made. A lower bound is then obtained by measuring the amount of work needed to shrink a set of potential inputs to a single input along the most timeconsuming path. As another example, consider the problem of merging two sorted lists of size n a1 < a2 < . . . < an
and
b1 < b2 < . . . < bn
into a single sorted list of size 2n. For simplicity, we assume that all the a’s and b’s are distinct, which gives the problem a unique solution. We encountered this problem when discussing mergesort in Section 5.1. Recall that we did merging by repeatedly comparing the ﬁrst elements in the remaining lists and outputting the smaller among them. The number of key comparisons in the worst case for this algorithm for merging is 2n − 1. Is there an algorithm that can do merging faster? The answer turns out to be no. Knuth [KnuIII, p. 198] quotes the following adversary method for proving that 2n − 1 is a lower bound on the number of key comparisons made by any comparisonbased algorithm for this problem. The adversary will employ the following rule: reply true to the comparison ai < bj if and only if i < j. This will force any correct merging algorithm to produce the only combined list consistent with this rule: b1 < a1 < b2 < a2 < . . . < bn < an. To produce this combined list, any correct algorithm will have to explicitly compare 2n − 1 adjacent pairs of its elements, i.e., b1 to a1, a1 to b2 , and so on. If one of these comparisons has not been made, e.g., a1 has not been compared to b2 , we can transpose these keys to get b1 < b2 < a1 < a2 < . . . < bn < an, which is consistent with all the comparisons made but cannot be distinguished from the correct conﬁguration given above. Hence, 2n − 1 is, indeed, a lower bound for the number of key comparisons needed for any merging algorithm.
Problem Reduction We have already encountered the problemreduction approach in Section 6.6. There, we discussed getting an algorithm for problem P by reducing it to another problem Q solvable with a known algorithm. A similar reduction idea can be used for ﬁnding a lower bound. To show that problem P is at least as hard as another problem Q with a known lower bound, we need to reduce Q to P (not P to Q!). In other words, we should show that an arbitrary instance of problem Q can be transformed (in a reasonably efﬁcient fashion) to an instance of problem P , so any algorithm solving P would solve Q as well. Then a lower bound for Q will be a lower bound for P . Table 11.1 lists several important problems that are often used for this purpose.
392
Limitations of Algorithm Power TABLE 11.1 Problems often used for establishing lower bounds by problem reduction
Problem sorting searching in a sorted array element uniqueness problem multiplication of ndigit integers multiplication of n × n matrices
Lower bound
Tightness
(n log n) (log n) (n log n) (n) (n2 )
yes yes yes unknown unknown
We will establish the lower bounds for sorting and searching in the next section. The element uniqueness problem asks whether there are duplicates among n given numbers. (We encountered this problem in Sections 2.3 and 6.1.) The proof of the lower bound for this seemingly simple problem is based on a very sophisticated mathematical analysis that is well beyond the scope of this book (see, e.g., [Pre85] for a rather elementary exposition). As to the last two algebraic problems in Table 11.1, the lower bounds quoted are trivial, but whether they can be improved remains unknown. As an example of establishing a lower bound by reduction, let us consider the Euclidean minimum spanning tree problem: given n points in the Cartesian plane, construct a tree of minimum total length whose vertices are the given points. As a problem with a known lower bound, we use the element uniqueness problem. We can transform any set x1, x2 , . . . , xn of n real numbers into a set of n points in the Cartesian plane by simply adding 0 as the points’ y coordinate: (x1, 0), (x2 , 0), . . . , (xn, 0). Let T be a minimum spanning tree found for this set of points. Since T must contain a shortest edge, checking whether T contains a zerolength edge will answer the question about uniqueness of the given numbers. This reduction implies that (n log n) is a lower bound for the Euclidean minimum spanning tree problem, too. Since the ﬁnal results about the complexity of many problems are not known, the reduction technique is often used to compare the relative complexity of problems. For example, the formulas x.y=
(x + y)2 − (x − y)2 4
and
x2 = x . x
show that the problems of computing the product of two ndigit integers and squaring an ndigit integer belong to the same complexity class, despite the latter being seemingly simpler than the former. There are several similar results for matrix operations. For example, multiplying two symmetric matrices turns out to be in the same complexity class as multiplying two arbitrary square matrices. This result is based on the observation that not only is the former problem a special case of the latter one, but also that
11.1
LowerBound Arguments
393
we can reduce the problem of multiplying two arbitrary square matrices of order n, say, A and B, to the problem of multiplying two symmetric matrices 0 A 0 BT X= and Y = , AT 0 B 0 where AT and B T are the transpose matrices of A and B (i.e., AT [i, j ] = A[j, i] and B T [i, j ] = B[j, i]), respectively, and 0 stands for the n × n matrix whose elements are all zeros. Indeed, 0 A 0 BT AB 0 XY = , = AT 0 B 0 0 AT B T from which the needed product AB can be easily extracted. (True, we will have to multiply matrices twice the original size, but this is just a minor technical complication with no impact on the complexity classes.) Though such results are interesting, we will encounter even more important applications of the reduction approach to comparing problem complexity in Section 11.3.
Exercises 11.1 1. Prove that any algorithm solving the alternatingdisk puzzle (Problem 14 in Exercises 3.1) must make at least n(n + 1)/2 moves to solve it. Is this lower bound tight? 2. Prove that the classic recursive algorithm for the Tower of Hanoi puzzle (Section 2.4) makes the minimum number of disk moves needed to solve the problem. 3. Find a trivial lowerbound class for each of the following problems and indicate, if you can, whether this bound is tight. a. ﬁnding the largest element in an array b. checking completeness of a graph represented by its adjacency matrix c. generating all the subsets of an nelement set d. determining whether n given real numbers are all distinct 4. Consider the problem of identifying a lighter fake coin among n identicallooking coins with the help of a balance scale. Can we use the same informationtheoretic argument as the one in the text for the number of questions in the guessing game to conclude that any algorithm for identifying the fake will need at least log2 n weighings in the worst case? 5. Prove that any comparisonbased algorithm for ﬁnding the largest element of an nelement set of real numbers must make n − 1 comparisons in the worst case.
394
Limitations of Algorithm Power
6. Find a tight lower bound for sorting an array by exchanging its adjacent elements. 7. Give an adversaryargument proof that the time efﬁciency of any algorithm that checks connectivity of a graph with n vertices is in (n2 ), provided the only operation allowed for an algorithm is to inquire about the presence of an edge between two vertices of the graph. Is this lower bound tight? 8. What is the minimum number of comparisons needed for a comparisonbased sorting algorithm to merge any two sorted lists of sizes n and n + 1 elements, respectively? Prove the validity of your answer. 9. Find the product of matrices A and B through a transformation to a product of two symmetric matrices if 1 −1 0 1 A= and B = . 2 3 −1 2 10. a. Can one use this section’s formulas that indicate the complexity equivalence of multiplication and squaring of integers to show the complexity equivalence of multiplication and squaring of square matrices? b. Show that multiplication of two matrices of order n can be reduced to squaring a matrix of order 2n. 11. Find a tight lowerbound class for the problem of ﬁnding two closest numbers among n real numbers x1, x2 , . . . , xn. 12. Find a tight lowerbound class for the number placement problem (Problem 9 in Exercises 6.1).
11.2 Decision Trees Many important algorithms, especially those for sorting and searching, work by comparing items of their inputs. We can study the performance of such algorithms with a device called a decision tree. As an example, Figure 11.1 presents a decision tree of an algorithm for ﬁnding a minimum of three numbers. Each internal node of a binary decision tree represents a key comparison indicated in the node, e.g., k < k . The node’s left subtree contains the information about subsequent comparisons made if k < k , and its right subtree does the same for the case of k > k . (For the sake of simplicity, we assume throughout this section that all input items are distinct.) Each leaf represents a possible outcome of the algorithm’s run on some input of size n. Note that the number of leaves can be greater than the number of outcomes because, for some algorithms, the same outcome can be arrived at through a different chain of comparisons. (This happens to be the case for the decision tree in Figure 11.1.) An important point is that the number of leaves must be at least as large as the number of possible outcomes. The algorithm’s work on a particular input of size n can be traced by a path from the root to a leaf in its decision tree, and the number of comparisons made by the algorithm on such
11.2
yes
yes
a
a 1, and therefore
logb n
(bd /a)j = (bd /a)
j =1
(bd /a)logb n − 1 ∈ ((bd /a)logb n). (bd /a) − 1
Hence, in this case, logb n
T (n) = nlogb a [T (1) +
(bd /a)j ] ∈ nlogb a ((bd /a)logb n)
j =1
= (n
logb a
(b /a)logb n) = (a logb n(bd /a)logb n) d
d
= (bd logb n) = (blogb n ) = (nd ). If a > bd , then bd /a < 1, and therefore logb n
(bd /a)j = (bd /a)
j =1
(bd /a)logb n − 1 ∈ (1). (bd /a) − 1
Hence, in this case, logb n
T (n) = nlogb a [T (1) +
(bd /a)j ] ∈ (nlogb a ).
j =1
If a = bd , then bd /a = 1, and therefore logb n
T (n) = nlogb a [T (1) +
(bd /a)j ] = nlogb a [T (1) + logb n]
j =1 d
∈ (nlogb a logb n) = (nlogb b logb n) = (nd logb n). Since f (n) = nd is a smooth function for any d ≥ 0, a reference to Theorem 4 completes the proof. Theorem 5 provides a very convenient tool for a quick efﬁciency analysis of divideandconquer and decreasebyaconstantfactor algorithms. You can ﬁnd examples of such applications throughout the book.
This page intentionally left blank
References
[Ade62]
AdelsonVelsky, G.M. and Landis, E.M. An algorithm for organization of information. Soviet Mathematics Doklady, vol. 3, 1962, 1259–1263.
[Adl94]
Adleman, L.M. Molecular computation of solutions to combinatorial problems. Science, vol. 266, 1994, 1021–1024.
[Agr04]
Agrawal, M., Kayal, N., and Saxena, N. PRIMES is in P. Annals of Mathematics, vol. 160, no. 2, 2004, 781–793.
[Aho74]
Aho, A.V., Hopcroft, J.E., and Ullman, J.D. The Design and Analysis of Computer Algorithms. AddisonWesley, 1974.
[Aho83]
Aho, A.V., Hopcroft, J.E., and Ullman, J.D. Data Structures and Algorithms. AddisonWesley, 1983.
[Ahu93]
Ahuja, R.K., Magnanti, T.L., and Orlin, J.B. Network Flows: Theory, Algorithms, and Applications. Prentice Hall, 1993.
[App]
´ Applegate, D.L., Bixby, R.E., Chvatal, V., and Cook, W.J. The Traveling Salesman Problems. www.tsp.gatech.edu/index.html.
[App07]
´ Applegate, D.L., Bixby, R.E., Chvatal, V., and Cook, W.J. The Traveling Salesman Problem: A Computational Study. Princeton University Press, 2007.
[Arb93]
Arbel, A. Exploring InteriorPoint Linear Programming: Algorithms and Software (Foundations of Computing). MIT Press, 1993.
[Aro09]
Arora, S. and Barak, B. Computational Complexity: A Modern Approach. Cambridge University Press, 2009.
[Ata09]
Atallah, M.J. and Blanton, M., eds. Algorithms and Theory of Computation Handbook, 2nd ed. (twovolume set), Chapman and Hall/CRC, 2009.
[Avi07]
Avidan, S. and Shamir, A. Seam carving for contentaware image resizing. ACM Transactions on Graphics, vol. 26, no. 3, article 10, July 2007, 9 pages. 493
494
References
[Azi10]
Aziz, A. and Prakash, A. Algorithms for Interviews. algorithmsforinterviews.com, 2010.
[Baa00]
Baase, S. and Van Gelder, A. Computer Algorithms: Introduction to Design and Analysis, 3rd ed. AddisonWesley, 2000.
[Bae81]
Baecker, R. (with assistance of D. Sherman) Sorting out Sorting. 30minute color sound ﬁlm. Dynamic Graphics Project, University of Toronto, 1981. video.google.com/videoplay?docid=3970523862559774879#docid=4110947752111188923.
[Bae98]
Baecker, R. Sorting out sorting: a case study of software visualization for teaching computer science. In Software Visualization: Programming as a Multimedia Experience, edited by J. Stasko, J. Domingue, M.C. Brown, and B.A. Price. MIT Press, 1998, 369–381.
[BaY95]
BaezaYates, R.A. Teaching algorithms. ACM SIGACT News, vol. 26, no. 4, Dec. 1995, 51–59.
[Bay72]
Bayer, R. and McGreight, E.M. Organization and maintenance of large ordered indices. Acta Informatica, vol. 1, no. 3, 1972, 173–189.
[Bel09]
Bell, J., and Stevens, B. A survey of known results and research areas for nqueens. Discrete Mathematics, vol. 309, issue 1, Jan. 2009, 1–31.
[Bel57]
Bellman, R.E. Dynamic Programming. Princeton University Press, 1957.
[Ben00]
Bentley, J. Programming Pearls, 2nd ed. AddisonWesley, 2000.
[Ben90]
Bentley, J.L. Experiments on traveling salesman heuristics. In Proceedings of the First Annual ACMSIAM Symposium on Discrete Algorithms, 1990, 91–99.
[Ben93]
Bentley, J.L. and McIlroy, M.D. Engineering a sort function. Software—Practice and Experience, vol. 23, no. 11, 1993, 1249–1265.
[Ber03]
Berlekamp, E.R., Conway, J.H., and Guy, R.K. Winning Ways for Your Mathematical Plays, 2nd ed., volumes 1–4. A K Peters, 2003.
[Ber00]
Berlinski, D. The Advent of the Algorithm: The Idea That Rules the World. Harcourt, 2000.
[Ber05]
Berman, K.A. and Paul, J.L. Algorithms: Sequential, Parallel, and Distributed. Course Technology, 2005.
[Ber01]
Berstekas, D.P. Dynamic Programming and Optimal Control: 2nd Edition (Volumes 1 and 2). Athena Scientiﬁc, 2001.
[Blo73]
Bloom, M., Floyd, R.W., Pratt, V., Rivest, R.L., and Tarjan, R.E. Time bounds for selection. Journal of Computer and System Sciences, vol. 7, no. 4, 1973, 448–461.
[Bog]
Bogomolny, A. Interactive Mathematics Miscellany and Puzzles. www.cuttheknot.org.
References
495
[Boy77]
Boyer, R.S. and Moore, J.S. A fast string searching algorithm. Communications of the ACM, vol. 21, no. 10, 1977, 762–772.
[Bra96]
Brassard, G. and Bratley, P. Fundamentals of Algorithmics. Prentice Hall, 1996.
[Car79]
Carmony, L. Odd pie ﬁghts. Mathematics Teacher, vol. 72, no. 1, 1979, 61–64.
[Cha98]
Chabert, JeanLuc, ed. A History of Algorithms: From the Pebble to the Microchip. Translated by Chris Weeks. Springer, 1998.
[Cha00]
Chandler, J.P. Patent protection of computer programs. Minnesota Intellectual Property Review, vol. 1, no. 1, 2000, 33–46.
[Chr76]
Christoﬁdes, N. WorstCase Analysis of a New Heuristic for the Traveling Salesman Problem. Technical Report, GSIA, CarnegieMellon University, 1976.
[Chv83]
´ Chvatal, V. Linear Programming. W.H. Freeman, 1983.
[Com79]
Comer, D. The ubiquitous Btree. ACM Computing Surveys, vol. 11, no. 2, 1979, 121–137.
[Coo71]
Cook, S.A. The complexity of theoremproving procedures. In Proceeding of the Third Annual ACM Symposium on the Theory of Computing, 1971, 151–158.
[Coo87]
Coopersmith, D. and Winograd, S. Matrix multiplication via arithmetic progressions. In Proceedings of Nineteenth Annual ACM Symposium on the Theory of Computing, 1987, 1–6.
[Cor09]
Cormen, T.H., Leiserson, C.E., Rivest, R.L., and Stein, C. Introduction to Algorithms, 3rd ed. MIT Press, 2009.
[Cra07]
Crack, T.F. Heard on the Street: Quantitative Questions from Wall Street Job Interviews, 10th ed., selfpublished, 2007.
[Dan63]
Dantzig, G.B. Linear Programming and Extensions. Princeton University Press, 1963.
[deB10]
de Berg, M., Cheong, O., van Kreveld, M., and Overmars, M. Computational Geometry: Algorithms and Applications, 3rd ed. Springer, 2010.
[Dew93]
Dewdney, A.K. The (New) Turing Omnibus. Computer Science Press, 1993.
[Dij59]
Dijkstra, E.W. A note on two problems in connection with graphs. Numerische Mathematik, vol. 1, 1959, 269–271.
[Dij76]
Dijkstra, E.W. A Discipline of Programming. PrenticeHall, 1976.
[Dud70]
Dudley, U. The ﬁrst recreational mathematics book. Journal of Recreational Mathematics, 1970, 164–169.
496
References
[Eas10]
Easley, D., and Kleinberg, J. Networks, Crowds, and Markets: Reasoning About a Highly Connected World. Cambridge University Press, 2010.
[Edm65]
Edmonds, J. Paths, trees, and ﬂowers. Canadian Journal of Mathematics, vol. 17, 1965, 449–467.
[Edm72]
Edmonds, J. and Karp, R.M. Theoretical improvements in algorithmic efﬁciency for network ﬂow problems. Journal of the ACM, vol. 19, no. 2, 1972, 248–264.
[Flo62]
Floyd, R.W. Algorithm 97: shortest path. Communications of the ACM, vol. 5, no. 6, 1962, 345.
[For57]
Ford, L.R., Jr., and Fulkerson, D.R. A simple algorithm for ﬁnding maximal network ﬂows and an application to the Hitchcock problem. Canadian Journal of Mathematics, vol. 9, no. 2, 1957, 210–218.
[For62]
Ford, L.R., Jr., and Fulkerson, D.R. Flows in Networks. Princeton University Press, 1962.
[For68]
Forsythe, G.E. What to do till the computer scientist comes. American Mathematical Monthly, vol. 75, no. 5, 1968, 454–462.
[For69]
Forsythe, G.E. Solving a quadratic equation on a computer. In The Mathematical Sciences, edited by COSRIMS and George Boehm, MIT Press, 1969, 138–152.
[Gal62]
Gale, D. and Shapley, L.S. College admissions and the stability of marriage. American Mathematical Monthly, vol. 69, Jan. 1962, 9–15.
[Gal86]
Galil, Z. Efﬁcient algorithms for ﬁnding maximum matching in graphs. Computing Surveys, vol. 18, no. 1, March 1986, 23–38.
[Gar99]
Gardiner, A. Mathematical Puzzling. Dover, 1999.
[Gar78]
Gardner, M. aha! Insight. Scientiﬁc American/W.H. Freeman, 1978.
[Gar88]
Gardner, M. Hexaﬂexagons and Other Mathematical Diversions: The First Scientiﬁc American Book of Puzzles and Games. University of Chicago Press, 1988.
[Gar94]
Gardner, M. My Best Mathematical and Logic Puzzles. Dover, 1994.
[Gar79]
Garey, M.R. and Johnson, D.S. Computers and Intractability: A Guide to the Theory of NPCompleteness. W.H. Freeman, 1979.
[Ger03]
Gerald, C.F. and Wheatley, P.O. Applied Numerical Analysis, 7th ed. AddisonWesley, 2003.
[Gin04]
Ginat, D. Embedding instructive assertions in program design. In Proceedings of ITiCSE’04, June 28–30, 2004, Leeds, UK, 62–66.
[Gol94]
Golomb, S.W. Polyominoes: Puzzles, Patterns, Problems, and Packings. Revised and expanded second edition. Princeton University Press, 1994.
References
497
[Gon91]
Gonnet, G.H. and BaezaYates, R. Handbook of Algorithms and Data Structures in Pascal and C, 2nd ed. AddisonWesley, 1991.
[Goo02]
Goodrich, M.T. and Tamassia, R. Algorithm Design: Foundations, Analysis, and Internet Examples. John Wiley & Sons, 2002.
[Gra94]
Graham, R.L., Knuth, D.E., and Patashnik, O. Concrete Mathematics: A Foundation for Computer Science, 2nd ed. AddisonWesley, 1994.
[Gre07]
Green, D.H. and Knuth, D.E. Mathematics for Analysis of Algo¨ rithms, 3rd edition. Birkhauser, 2007.
[Gri81]
Gries, D. The Science of Programming. Springer, 1981.
[Gus89]
Gusﬁeld, D. and Irwing, R.W. The Stable Marriage Problem: Structure and Algorithms. MIT Press, 1989.
[Gut07]
Gutin, G. and Punnen, A.P., eds. Traveling Salesman Problem and Its Variations. Springer, 2007.
[Har92]
Harel, D. Algorithmics: The Spirit of Computing, 2nd ed. AddisonWesley, 1992.
[Har65]
Hartmanis, J. and Stearns, R.E. On the computational complexity of algorithms. Transactions of the American Mathematical Society, vol. 117, May 1965, 285–306.
[Hea63]
Heap, B.R. Permutations by interchanges. Computer Journal, vol. 6, 1963, 293–294.
[Het10]
Hetland, M.L. Python Algorithms: Mastering Basic Algorithms in the Python Language. Apress, 2010.
[Hig93]
Higham, N.J. The accuracy of ﬂoating point summation. SIAM Journal on Scientiﬁc Computing, vol. 14, no. 4, July 1993, 783–799.
[Hoa96]
Hoare, C.A.R. Quicksort. In Great Papers in Computer Science, Phillip Laplante, ed. West Publishing Company, 1996, 31–39.
[Hoc97]
Hochbaum, D.S., ed. Approximation Algorithms for NPHard Problems. PWS Publishing, 1997.
[Hop87]
Hopcroft, J.E. Computer science: the emergence of a discipline. Communications of the ACM, vol. 30, no. 3, March 1987, 198–202.
[Hop73]
Hopcroft, J.E. and Karp, R.M. An n5/2 algorithm for maximum matchings in bipartite graphs. SIAM Journal on Computing, vol. 2, 1973, 225–231.
[Hor07]
Horowitz, E., Sahni, S., and Rajasekaran, S. Computer Algorithms, 2nd ed. Silicon Press, 2007.
[Hor80]
Horspool, R.N. Practical fast searching in strings. Software—Practice and Experience, vol. 10, 1980, 501–506.
[Hu02]
Hu, T.C. and Shing, M.T. Combinatorial Algorithms: Enlarged Second edition. Dover, 2002.
498
References
[Huf52]
Huffman, D.A. A method for the construction of minimum redundancy codes. Proceedings of the IRE, vol. 40, 1952, 1098–1101.
[Joh07a]
Johnson, D.S. and McGeoch, L.A. Experimental analysis of heuristics for the STSP. In The Traveling Salesman Problem and Its Variations, edited by G. Gutin and A.P. Punnen, Springer, 2007, 369–443.
[Joh07b]
Johnson, D.S., Gutin, G., McGeoch, L.A., Yeo, A., Zhang, W., and Zverovitch, A. Experimental analysis of heuristics for the ATSP. In The Traveling Salesman Problem and Its Variations, edited by G. Gutin and A.P. Punnen, Springer, 2007, 445–487.
[Joh04]
Johnsonbaugh, R. and Schaefer, M. Algorithms. Pearson Education, 2004.
[Kar84]
Karmarkar, N. A new polynomialtime algorithm for linear programming. Combinatorica, vol. 4, no. 4, 1984, 373–395.
[Kar72]
Karp, R.M. Reducibility among combinatorial problems. In Complexity of Computer Communications, edited by R.E. Miller and J.W. Thatcher. Plenum Press, 1972, 85–103.
[Kar86]
Karp, R.M. Combinatorics, complexity, and randomness. Communications of the ACM, vol. 29, no. 2, Feb. 1986, 89–109.
[Kar91]
Karp, R.M. An introduction to randomized algorithms. Discrete Applied Mathematics, vol. 34, Nov. 1991, 165–201.
[Kel04]
Kellerer, H., Pferschy, U., and Pisinger, D. Knapsack Problems. Springer, 2004.
[Ker99]
Kernighan, B.W. and Pike. R. The Practice of Programming. AddisonWesley, 1999.
[Kha79]
Khachian, L.G. A polynomial algorithm in linear programming. Soviet Mathematics Doklady, vol. 20, 1979, 191–194. ´ Algorithm Design. Pearson, 2006. Kleinberg, J. and Tardos, E.
[Kle06] [Knu75]
Knuth, D.E. Estimating the efﬁciency of backtrack programs. Mathematics of Computation, vol. 29, Jan. 1975, 121–136.
[Knu76]
Knuth, D.E. Big omicron and big omega and big theta. ACM SIGACT News, vol. 8, no. 2, 1976, 18–23.
[Knu96]
Knuth, D.E. Selected Papers on Computer Science. CSLI Publications and Cambridge University Press, 1996.
[KnuI]
Knuth, D.E. The Art of Computer Programming, Volume 1: Fundamental Algorithms, 3rd ed. AddisonWesley, 1997.
[KnuII]
Knuth, D.E. The Art of Computer Programming, Volume 2: Seminumerical Algorithms, 3rd ed. AddisonWesley, 1998.
[KnuIII]
Knuth, D.E. The Art of Computer Programming, Volume 3: Sorting and Searching, 2nd ed. AddisonWesley, 1998.
References
499
[KnuIV]
Knuth, D.E. The Art of Computer Programming, Volume 4A, Combinatorial Algorithms, Part 1. AddisonWesley, 2011.
[Knu77]
Knuth, D.E., Morris, Jr., J.H., and Pratt, V.R. Fast pattern matching in strings. SIAM Journal on Computing, vol. 5, no. 2, 1977, 323–350.
[Kol95]
Kolman, B. and Beck, R.E. Elementary Linear Programming with Applications, 2nd ed. Academic Press, 1995.
[Kor92]
Kordemsky, B.A. The Moscow Puzzles. Dover, 1992.
[Kor05]
Kordemsky, B.A. Mathematical Charmers. Oniks, 2005 (in Russian).
[Kru56]
Kruskal, J.B. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society, vol. 7, 1956, 48–50.
[Laa10]
Laakmann, G. Cracking the Coding Interview, 4th ed. CareerCup, 2010.
[Law85]
Lawler, E.L., Lenstra, J.K., Rinnooy Kan, A.H.G., and Shmoys, D.B., eds. The Traveling Salesman Problem. John Wiley, 1985.
[Lev73]
Levin, L.A. Universal sorting problems. Problemy Peredachi Informatsii, vol. 9, no. 3, 1973, 115–116 (in Russian). English translation in Problems of Information Transmission, vol. 9, 265–266.
[Lev99]
Levitin, A. Do we teach the right algorithm design techniques? In Proceedings of SIGCSE’99, New Orleans, LA, 1999, 179–183.
[Lev11]
Levitin, A. and Levitin, M. Algorithmic Puzzles. Oxford University Press, 2011.
[Lin73]
Lin, S. and Kernighan, B.W. An effective heuristic algorithm for the travelingsalesman problem. Operations Research, vol. 21, 1973, 498–516.
[Man89]
Manber, U. Introduction to Algorithms: A Creative Approach. AddisonWesley, 1989.
[Mar90]
Martello, S. and Toth, P. Knapsack Problems: Algorithms and Computer Implementations. John Wiley, 1990.
[Mic10]
Michalewicz, Z. and Fogel, D.B. How to Solve It: Modern Heuristics, second, revised and extended edition. Springer, 2010.
[Mil05]
Miller, R. and Boxer, L. Algorithms Sequential and Parallel: A Uniﬁed Approach, 2nd ed. Charles River Media, 2005.
[Mor91]
Moret, B.M.E. and Shapiro, H.D. Algorithms from P to NP. Volume I: Design and Efﬁciency. Benjamin Cummings, 1991.
[Mot95]
Motwani, R. and Raghavan, P. Randomized Algorithms. Cambridge University Press, 1995.
[Nea09]
Neapolitan, R. and Naimipour, K. Foundations of Algorithms, Fourth Edition. Jones and Bartlett, 2009.
500
References
[Nem89]
Nemhauser, G.L., Rinnooy Kan, A.H.G., and Todd, M.J., eds. Optimization. NorthHolland, Amsterdam, 1989.
[OCo98]
O’Connor, J.J. and Robertson, E.F. The MacTutor History of Mathematics archive, June 1998, wwwhistory.mcs.standrews.ac.uk/history/ Mathematicians/Abel.html.
[Ong84]
Ong, H.L. and Moore, J.B. Worstcase analysis of two traveling salesman heuristics. Operations Research Letters, vol. 2, 1984, 273– 277.
[ORo98]
O’Rourke, J. Computational Geometry in C, 2nd ed. Cambridge University Press, 1998.
[Ove80]
Overmars, M.H. and van Leeuwen, J. Further comments on Bykat’s convex hull algorithm. Information Processing Letters, vol. 10, no. 4/5, 1980, 209–212.
[Pan78]
Pan, V.Y. Strassen’s algorithm is not optimal. Proceedings of Nineteenth Annual IEEE Symposium on the Foundations of Computer Science, 1978, 166–176.
[Pap82]
Papadimitriou, C.H. and Steiglitz, K. Combinatorial Optimization: Algorithms and Complexity. PrenticeHall, 1982.
[Par95]
Parberry, I. Problems on Algorithms. PrenticeHall, 1995.
[Pol57]
´ Polya, G. How to Solve It: A New Aspect of Mathematical Method, 2nd ed. Doubleday, 1957.
[Pre85]
Preparata, F.P. and Shamos, M.I. Computational Geometry: An Introduction. Springer, 1985.
[Pri57]
Prim, R.C. Shortest connection networks and some generalizations. Bell System Technical Journal, vol. 36, no. 1, 1957, 1389–1401.
[Pur04]
Purdom, P.W., Jr., and Brown, C. The Analysis of Algorithms. Oxford University Press, 2004.
[Raw91]
Rawlins, G.J.E. Compared to What? An Introduction to the Analysis of Algorithms. Computer Science Press, 1991.
[Rei77]
Reingold, E.M., Nievergelt, J., and Deo, N. Combinatorial Algorithms: Theory and Practice. PrenticeHall, 1977.
[Riv78]
Rivest, R.L., Shamir, A., and Adleman, L.M. A method for obtaining digital signatures and publickey cryptosystems. Communications of the ACM, vol. 21, no. 2, Feb. 1978, 120–126.
[Ros07]
Rosen, K. Discreet Mathematics and Its Applications, 6th ed., McGrawHill, 2007.
[Ros77]
Rosenkrantz, D.J., Stearns, R.E., and Lewis, P.M. An analysis of several heuristics for the traveling salesman problem. SIAM Journal of Computing, vol. 6, 1977, 563–581.
References
501
[Roy59]
´ Comptes rendus de l’Acad´emie des Roy, B. Transitivite´ et connexite. Sciences, vol. 249, 216–218, 1959.
[Sah75]
Sahni, S. Approximation algorithms for the 0/1 knapsack problem. Journal of the ACM, vol. 22, no. 1, Jan. 1975, 115–124.
[Sah76]
Sahni, S. and Gonzalez, T. Pcomplete approximation problems. Journal of the ACM, vol. 23, no. 3, July 1976, 555–565.
[Say05]
Sayood, K. Introduction to Data Compression, 3rd ed. Morgan Kaufmann Publishers, 2005.
[Sed02]
Sedgewick, R. Algorithms in C/C++/Java, Parts 1–5: Fundamentals, Data Structures, Sorting, Searching, and Graph Algorithms, 3rd ed. AddisonWesley Professional, 2002.
[Sed96]
Sedgewick, R. and Flajolet, P. An Introduction to the Analysis of Algorithms. AddisonWesley Professional, 1996.
[Sed11]
Sedgewick, R. and Wayne, K. Algorithms, Fourth Edition. Pearson Education, 2011.
[Sha07]
Shaffer, C.A., Cooper, M., and Edwards, S.H. Algorithm visualization: a report on the state of the ﬁeld. ACM SIGCSE Bulletin, vol. 39, no. 1, March 2007, 150–154.
[Sha98]
Shasha, D. and Lazere, C. Out of Their Minds: The Lives and Discoveries of 15 Great Computer Scientists. Copernicus, 1998.
[She59]
Shell, D.L. A highspeed sorting procedure. Communications of the ACM, vol. 2, no. 7, July 1959, 30–32.
[Sho94]
Shor, P.W. Algorithms for quantum computation: discrete algorithms and factoring. Proceedings 35th Annual Symposium on Foundations of Computer Science (Shaﬁ Goldwasser, ed.). IEEE Computer Society Press, 1994, 124–134.
[Sip05]
Sipser, M. Introduction to the Theory of Computation, 2nd ed. Course Technology, 2005.
[Ski10]
Skiena, S.S. Algorithm Design Manual, 2nd ed. Springer, 2010.
[Str69]
Strassen, V. Gaussian elimination is not optimal. Numerische Mathematik, vol. 13, no. 4, 1969, 354–356.
[Tar83]
Tarjan, R.E. Data Structures and Network Algorithms. Society for Industrial and Applied Mathematics, 1983.
[Tar85]
Tarjan, R.E. Amortized computational complexity. SIAM Journal on Algebraic and Discrete Methods, vol. 6, no. 2, Apr. 1985, 306–318.
[Tar87]
Tarjan, R.E. Algorithm design. Communications of the ACM, vol. 30, no. 3, March 1987, 204–212.
[Tar84]
Tarjan, R.E. and van Leeuwen, J. Worstcase analysis of set union algorithms. Journal of the ACM, vol. 31, no. 2, Apr. 1984, 245–281.
502
References
[War62]
Warshall, S. A theorem on boolean matrices. Journal of the ACM, vol. 9, no. 1, Jan. 1962, 11–12.
[Wei77]
Weide, B. A survey of analysis techniques for discrete algorithms. Computing Surveys, vol. 9, no. 4, 1977, 291–313. Williams, J.W.J. Algorithm 232 (heapsort). Communications of the ACM, vol. 7, no. 6, 1964, 347–348. Wirth, N. Algorithms + Data Structures = Programs. PrenticeHall, Englewood Cliffs, NJ, 1976. Yanofsky, N.S. and Mannucci, M.A. Quantum Computing for Computer Scientists. Cambridge University Press, 2008. Yao, F. Speedup in dynamic programming. SIAM Journal on Algebraic and Discrete Methods, vol. 3, no. 4, 1982, 532–540.
[Wil64] [Wir76] [Yan08] [Yao82]
Hints to Exercises
CHAPTER 1 Exercises 1.1 1. It is probably faster to do this by searching the Web, but your library should be able to help, too. 2. One can ﬁnd arguments supporting either view. There is a wellestablished principle pertinent to the matter, though: scientiﬁc facts or mathematical expressions of them are not patentable. (Why do you think this is the case?) But should this preclude granting patents for all algorithms? 3. You may assume that you are writing your algorithms for a human rather than a machine. Still, make sure that your descriptions do not contain obvious ambiguities. Knuth provides an interesting comparison between cooking recipes and algorithms [KnuI, p. 6]. 4. There is a quite √ straightforward algorithm for this problem based on the deﬁnition of n. 5. Try to design an algorithm that always makes less than mn comparisons. 6. a. Just follow Euclid’s algorithm as described in the text. b. Compare the number of divisions made by the two algorithms. 7. Prove that if d divides both m and n (i.e., m = sd and n = td for some positive integers s and t), then it also divides both n and r = m mod n and vice versa. Use the formula m = qn + r (0 ≤ r < n) and the fact that if d divides two integers u and v, it also divides u + v and u − v (why?). 8. Perform one iteration of the algorithm for two arbitrarily chosen integers m < n. 9. The answer to part (a) can be given immediately, the answer to part (b) can be given by checking the algorithm’s performance on all pairs 1 < m < n ≤ 10. 503
504
Hints to Exercises
10. a. Use the equality gcd(m, n) = gcd(m − n, n)
for m ≥ n > 0.
b. The key is to ﬁgure out the total number of distinct integers that can be written on the board, starting with an initial pair m, n where m > n ≥ 1. You should exploit a connection of this question to the question of part (a). Considering small examples, especially those with n = 1 and n = 2, should help, too. 11. Of course, for some coefﬁcients, the equation will have no solutions. 12. Tracing the algorithm by hand for, say, n = 10 and studying its outcome should help answering both questions.
Exercises 1.2 1. The farmer would have to make several trips across the river, starting with the only one possible. 2. Unlike the Old World puzzle of Problem 1, the ﬁrst move solving this puzzle is not obvious. 3. The principal issue here is a possible ambiguity. 4. Your algorithm should work correctly for all possible values of the coefﬁcients, including zeros. 5. You almost certainly learned this algorithm in one of your introductory programming courses. If this assumption is not true, you have a choice between designing such an algorithm on your own or looking it up. 6. You may need to make a ﬁeld trip to refresh your memory. 7. Question (a) is difﬁcult, though the answer to it—discovered in the 1760s by the German mathematician Johann Lambert—is wellknown. By comparison, question (b) is incomparably simpler. 8. You probably know two or more different algorithms for sorting an array of numbers. 9. You can: decrease the number of times the inner loop is executed, make that loop run faster (at least for some inputs), or, more signiﬁcantly, design a faster algorithm from scratch.
Exercises 1.3 1. Trace the algorithm on the input given. Use the deﬁnitions of stability and being inplace that were introduced in the section. 2. If you do not recall any searching algorithms, you should design a simple searching algorithm (without succumbing to the temptation to ﬁnd one in the latter chapters of the book).
Chapter 1
505
3. This algorithm is introduced later in the book, but you should have no trouble designing it on your own. 4. If you have not encountered this problem in your previous courses, you may look up the answers on the Web or in a discrete structures textbook. The answers are, in fact, surprisingly simple. 5. No efﬁcient algorithm for solving this problem for an arbitrary graph is known. This particular graph does have Hamiltonian circuits that are not difﬁcult to ﬁnd. (You need to ﬁnd just one of them.) 6. a. Put yourself (mentally) in a passenger’s place and ask yourself what criterion for the “best” route you would use. Then think of people that may have different needs. b. The representation of the problem by a graph is straightforward. Give some thoughts, though, to stations where trains can be changed. 7. a. What are tours in the traveling salesman problem? b. It would be natural to consider vertices colored the same color as elements of the same subset. 8. Create a graph whose vertices represent the map’s regions. You will have to decide on the edges on your own. 9. Assume that the circumference in question exists and ﬁnd its center ﬁrst. Also, do not forget to give a special answer for n ≤ 2. 10. Be careful not to miss some special cases of the problem.
Exercises 1.4 1. a. Take advantage of the fact that the array is not sorted. b. We used this trick in implementing one of the algorithms in Section 1.1. 2. a. For a sorted array, there is a spectacularly efﬁcient algorithm you almost certainly have heard about. b. Unsuccessful searches can be made faster. 3. a. Push(x) puts x on the top of the stack; pop deletes the item from the top of the stack. b. Enqueue(x) adds x to the rear of the queue; dequeue deletes the item from the front of the queue. 4. Just use the deﬁnitions of the graph properties in question and data structures involved. 5. There are two wellknown algorithms that can solve this problem. The ﬁrst uses a stack; the second uses a queue. Although these algorithms are discussed later in the book, do not miss this chance to discover them by yourself! 6. The inequality h ≤ n − 1 follows immediately from the height’s deﬁnition. The lower bound inequality follows from the inequality 2h+1 − 1 ≥ n, which can be
506
Hints to Exercises
proved by considering the largest number of vertices a binary tree of height h can have. 7. You need to indicate how each of the three operations of the priority queue will be implemented. 8. Because of insertions and deletions, using an array of the dictionary’s elements (sorted or unsorted) is not the best implementation possible. 9. You need to know about postﬁx notation in order to answer one of these questions. (If you are not familiar with it, ﬁnd the information on the Internet.) 10. There are several algorithms for this problem. Keep in mind that the words may contain multiple occurrences of the same letter.
CHAPTER 2 Exercises 2.1 1. The questions are indeed as straightforward as they appear, though some of them may have alternative answers. Also, keep in mind the caveat about measuring an integer’s size. 2. a. The sum of two matrices is deﬁned as the matrix whose elements are the sums of the corresponding elements of the matrices given. b. Matrix multiplication requires two operations: multiplication and addition. Which of the two would you consider basic and why? 3. Will the algorithm’s efﬁciency vary on different inputs of the same size? 4. a. Gloves are not socks: they can be righthanded and lefthanded. b. You have only two qualitatively different outcomes possible. Find the number of ways to get each of the two. 5. a. First, prove that if a positive decimal integer n has b digits in its binary representation, then 2b−1 ≤ n < 2b . Then, take binary logarithms of the terms in these inequalities. b. The proof is similar to the proof of formula (2.1). c. The formulas will be the same, with just one small adjustment to account for the different radix. d. How can we switch from one logarithm base to another? 6. Insert a veriﬁcation of whether the problem is already solved. 7. A similar question was investigated in the section. 8. Use either the difference between or the ratio of f (4n) and f (n), whichever is more convenient for getting a compact answer. If it is possible, try to get an answer that does not depend on n.
Chapter 2
507
9. If necessary, simplify the functions in question to single out terms deﬁning their orders of growth to within a constant multiple. (We discuss formal methods for answering such questions in the next section; however, the questions can be answered without knowledge of such methods.) 10. a. Use the formula ni=0 2i = 2n+1 − 1. b. Use the formula for the sum of the ﬁrst n odd numbers or the formula for the sum of arithmetic progression.
Exercises 2.2 1. Use the corresponding counts of the algorithm’s basic operation (see Section 2.1) and the deﬁnitions of O, , and . 2. Establish the order of growth of n(n + 1)/2 ﬁrst and then use the informal deﬁnitions of O, , and . (Similar examples were given in the section.) 3. Simplify the functions given to single out the terms deﬁning their orders of growth. 4. a. Check carefully the pertinent deﬁnitions. b. Compute the ratio limits of every pair of consecutive functions on the list. 5. First, simplify some of the functions. Then, use the list of functions in Table 2.2 to “anchor” each of the functions given. Prove their ﬁnal placement by computing appropriate limits. 6. a. You can prove this assertion either by computing an appropriate limit or by applying mathematical induction. b. Compute limn→∞ a1n/a2n. 7. Prove the correctness of (a), (b), and (c) by using the appropriate deﬁnitions; construct a counterexample for (d) (e.g., by constructing two functions behaving differently for odd and even values of their arguments). 8. The proof of part (a) is similar to the one given for the theorem’s assertion in Section 2.2. Of course, different inequalities need to be used to bound the sum from below. 9. Follow the analysis plan used in the text when the algorithm was mentioned for the ﬁrst time. 10. You may use straightforward algorithms for all the four questions asked. Use the O notation for the time efﬁciency class of one of them, and the notation for the three others. 11. The problem can be solved in two weighings. 12. You should walk intermittently left and right from your initial position until the door is reached.
508
Hints to Exercises
Exercises 2.3 1. Use the common summation formulas and rules listed in Appendix A. You may need to perform some simple algebraic operations before applying them. 2. Find a sum among those in Appendix A that looks similar to the sum in question and try to transform the latter to the former. Note that you do not have to get a closedform expression for a sum before establishing its order of growth. 3. Just follow the formulas in question. 4. a. Tracing the algorithm to get its output for a few small values of n (e.g., n = 1, 2, and 3) should help if you need it. b. We faced the same question for the examples discussed in this section. One of them is particularly pertinent here. c. Follow the plan outlined in the section. d. As a function of n, the answer should follow immediately from your answer to part (c). You may also want to give an answer as a function of the number of bits in the n’s representation (why?). e. Have you not encountered this sum somewhere? 5. a. Tracing the algorithm to get its output for a few small values of n (e.g., n = 1, 2, and 3) should help if you need it. b. We faced the same question for the examples discussed in the section. One of them is particularly pertinent here. c. You can either follow the section’s plan by setting up and computing a sum or answer the question directly. (Try to do both.) d. Your answer will immediately follow from the answer to part (c). e. Does the algorithm always have to make two comparisons on each iteration? This idea can be developed further to get a more signiﬁcant improvement than the obvious one—try to do it for a fourelement array and then generalize the insight. But can we hope to ﬁnd an algorithm with a better than linear efﬁciency? 6. a. Elements A[i, j ] and A[j, i] are symmetric with respect to the main diagonal of the matrix. b. There is just one candidate here. c. You may investigate the worst case only. d. Your answer will immediately follow from the answer to part (c). e. Compare the problem the algorithm solves with the way it does this. 7. Computing a sum of n numbers can be done with n − 1 additions. How many does the algorithm make in computing each element of the product matrix? 8. Set up a sum for the number of times all the doors are toggled and ﬁnd its asymptotic order of growth by using some formulas from Appendix A.
Chapter 2
509
9. For the general step of the proof by induction, use the formula n+1
i=1
i=
n
i + (n + 1).
i=1
The young Gauss computed the sum 1 + 2 + . . . + 99 + 100 by noticing that it can be computed as the sum of 50 pairs, each with the same sum. 10. There are at least two different ways to solve this problem, which comes from a collection of Wall Street interview questions. 11. a. Setting up a sum should pose no difﬁculties. Using the standard summation formulas and rules will require more effort than in the previous examples, however. b. Optimize the algorithm’s innermost loop. 12. Set up a sum for the number of squares after n iterations of the algorithm and then simplify it to get a closedform answer. 13. To derive a formula expressing the total number of digits as a function of the number of pages n, where 1 ≤ n ≤ 1000, it is convenient to partition the function’s domain into several natural intervals.
Exercises 2.4 1. Each of these recurrences can be solved by the method of backward substitutions. 2. The recurrence relation in question is almost identical to the recurrence relation for the number of multiplications, which was set up and solved in the section. 3. a. The question is similar to that about the efﬁciency of the recursive algorithm for computing n!. b. Write pseudocode for the nonrecursive algorithm and determine its efﬁciency. 4. a. Note that you are asked here about a recurrence for the function’s values, not about a recurrence for the number of times its operation is executed. Just follow the pseudocode to set it up. It is easier to solve this recurrence by forward substitutions (see Appendix B). b. This question is very similar to one we have already discussed. c. You may want to include the subtractions needed to decrease n. 5. a. Use the formula for the number of disk moves derived in the section. b. Solve the problem for three disks to investigate the number of moves made by each of the disks. Then generalize the observations and prove their validity for the general case of n disks.
510
Hints to Exercises
6. The required algorithm and the method of its analysis are similar to those of the classic version of the puzzle. Because of the additional constraint, more than two smaller instances of the puzzle need to be solved here. 7. a. Consider separately the cases of even and odd values of n and show that for both of them log2 n satisﬁes the recurrence relation and its initial condition. b. Just follow the algorithm’s pseudocode. 8. a. Use the formula 2n = 2n−1 + 2n−1 without simplifying it; do not forget to provide a condition for stopping your recursive calls. b. A similar algorithm was investigated in the section. c. A similar question was investigated in the section. d. A bad efﬁciency class of an algorithm by itself does not mean that the algorithm is bad. For example, the classic algorithm for the Tower of Hanoi puzzle is optimal despite its exponentialtime efﬁciency. Therefore, a claim that a particular algorithm is not good requires a reference to a better one. 9. a. Tracing the algorithm for n = 1 and n = 2 should help. b. It is very similar to one of the examples discussed in the section. 10. Get the basic operation count either by solving a recurrence relation or by computing directly the number of the adjacency matrix elements the algorithm checks in the worst case. 11. a. Use the deﬁnition’s formula to get the recurrence relation for the number of multiplications made by the algorithm. b. Investigate the righthand side of the recurrence relation. Computing the ﬁrst few values of M(n) may be helpful, too. 12. You might want to use the neighborhood’s symmetry to obtain a simple formula for the number of squares added to the neighborhood on the nth iteration of the algorithm. 13. The minimum amount of time needed to fry three hamburgers is smaller than 4 minutes. 14. Solve ﬁrst a simpler version in which a celebrity must be present.
Exercises 2.5 1. Use a search engine. 2. Set up an equation expressing the number of rabbits after n months in terms of the number of rabbits in some previous months. 3. There are several ways to solve this problem. The most elegant of them makes it possible to put the problem in this section. 4. Writing down the ﬁrst, say, ten Fibonacci numbers makes the pattern obvious.
Chapter 3
511
5. It is easier to substitute φ n and φˆ n into the recurrence equation separately. Why will this sufﬁce? 6. Use an approximate formula for F (n) to ﬁnd the smallest values of n to exceed the numbers given. 7. Set up the recurrence relations for C(n) and Z(n), with appropriate initial conditions, of course. 8. All the information needed on each iteration of the algorithm is the values of the last two consecutive Fibonacci numbers. Modify the algorithm F ib(n) to take advantage of this fact. 9. Prove it by mathematical induction. 10. Consider ﬁrst a small example such as computing gcd(13, 8). 11. Take advantage of the special nature of the rectangle’s dimensions. 12. The last k digits of an integer N can be obtained by computing N mod 10k . Performing all operations of your algorithms modulo 10k (see Appendix A) will enable you to circumvent the exponential growth of the Fibonacci numbers. Also note that Section 2.6 is devoted to a general discussion of the empirical analysis of algorithms.
Exercises 2.6 1. Does it return a correct comparison count for every array of size 2? 2. Debug your comparison counting and random input generating for small array sizes ﬁrst. 3. On a reasonably fast desktop, you may well get zero time, at least for smaller sizes in your sample. Section 2.6 mentions a trick for overcoming this difﬁculty. 4. Check how fast the count values grow with doubling the input size. 5. A similar question was discussed in the section. 6. Compare the values of the functions lg lg n and lg n for n = 2k . 7. Insert the division counter in a program implementing the algorithm and run it for the input pairs in the range indicated. 8. Get the empirical data for random values of n in a range of, say, between 102 and 104 or 105 and plot the data obtained. (You may want to use different scales for the axes of your coordinate system.)
CHAPTER 3 Exercises 3.1 1. a. Think of algorithms that have impressed you with their efﬁciency and/or sophistication. Neither characteristic is indicative of a bruteforce algorithm.
512
Hints to Exercises
b. Surprisingly, it is not a very easy question to answer. Mathematical problems (including those you’ve studied in your secondary school and college courses) are a good source of such examples. 2. a. The ﬁrst question was all but answered in the section. Expressing the answer as a function of the number of bits can be done by using the formula relating the two metrics. b. How can we compute (ab) mod m? 3. It helps to have done the exercises in question. 4. a. The most straightforward algorithm, which is based on substituting x0 into the formula, is quadratic. b. Analyzing what unnecessary computations the quadratic algorithm does should lead you to a better (linear) algorithm. c. How many coefﬁcients does a polynomial of degree n have? Can one compute its value at an arbitrary point without processing all of them? 5. For each of the three network topologies, what properties of the matrix should the algorithm check? 6. The answer to four of the questions is yes. 7. a. Just apply the bruteforce thinking to the problem in question. b. The problem can be solved in one weighing. 8. Just trace the algorithm on the input given. (It was done for another input in the section.) 9. Although the majority of elementary sorting algorithms are stable, do not rush with your answer. A general remark about stability made in Section 1.3, where the notion of stability is introduced, could be helpful, too. 10. Generally speaking, implementing an algorithm for a linked list poses problems if the algorithm requires accessing the list’s elements not in sequential order. 11. Just trace the algorithm on the input given. (See an example in the section.) 12. a. A list is sorted if and only if all its adjacent elements are in a correct order. Why? b. Add a boolean ﬂag to register the presence or absence of switches. c. Identify worstcase inputs ﬁrst. 13. Can bubblesort change the order of two equal elements in its input? 14. Thinking about the puzzle as a sortinglike problem may or may not lead you to the most simple and efﬁcient solution.
Exercises 3.2 1. Modify the analysis of the algorithm’s version in Section 2.1. 2. As a function of p, what kind of function is Cavg ?
Chapter 3
513
3. Solve a simpler problem with a single gadget ﬁrst. Then design a better than linear algorithm for the problem with two gadgets. 4. The content of this quote from Mahatma Gandhi is more thought provoking than this drill. 5. For each input, one iteration of the algorithm yields all the information you need to answer the question. 6. It will sufﬁce to limit your search for an example to binary texts and patterns. 7. The answer, surprisingly, is yes. 8. a. For a given occurrence of A in the text, what are the substrings you need to count? b. For a given occurrence of B in the text, what are the substrings you need to count? 9. You may use either bit strings or a naturallanguage text for the visualization program. It would be a good idea to implement, as an option, a search for all occurrences of a given pattern in a given text. 10. Test your program thoroughly. Be especially careful about the possibility of words read diagonally with wrapping around the table’s border. 11. A (very) bruteforce algorithm can simply shoot at adjacent feasible cells starting at, say, one of the corners of the board. Can you suggest a better strategy? (You can investigate relative efﬁciencies of different strategies by making two programs implementing them play each other.) Is your strategy better than the one that shoots at randomly generated cells of the opponent’s board?
Exercises 3.3 1. You may want to consider two versions of the answer: without taking into account the comparison and assignments in the algorithm’s innermost loop and with them. 2. Sorting n real numbers can be done in O(n log n) time. 3. a. Solving the problem for n = 2 and n = 3 should lead you to the critical insight. b. Where would you put the post ofﬁce if it did not have to be at one of the village locations? 4. a. Check requirements (i)–(iii) by using basic properties of absolute values. b. For the Manhattan distance, the points in question are deﬁned by the equation x − 0 + y − 0 = 1. You can start by sketching the points in the positive quadrant of the coordinate system (i.e., the points for which x, y ≥ 0) and then sketch the rest by using the symmetries. c. The assertion is false. You can choose, say, p1(0, 0) and p2 (1, 0) and ﬁnd p3 to complete a counterexample.
514
Hints to Exercises
5. a. Prove that the Hamming distance does satisfy the three axioms of a distance metric. b. Your answer should include two parameters. 6. True; prove it by mathematical induction. 7. Your answer should be a function of two parameters: n and k. A special case of this problem (for k = 2) is solved in the text. 8. Review the examples given in the section. 9. Some of the extreme points of a convex hull are easier to ﬁnd than others. 10. If there are other points of a given set on the straight line through pi and pj , which of all these points need to be preserved for further processing? 11. Your program should work for any set of n distinct points, including sets with many collinear points. 12. a. The set of points satisfying inequality ax + by ≤ c is the halfplane of the points on one side of the straight line ax + by = c, including all the points on the line itself. Sketch such a halfplane for each of the inequalities and ﬁnd their intersection. b. The extreme points are the vertices of the polygon obtained in part (a). c. Compute and compare the values of the objective function at the extreme points.
Exercises 3.4 1. a. Identify the algorithm’s basic operation and count the number of times it will be executed. b. For each of the time amounts given, ﬁnd the largest value of n for which this limit won’t be exceeded. 2. How different is the traveling salesman problem from the problem of ﬁnding a Hamiltonian circuit? 3. Your algorithm should check the wellknown condition that is both necessary and sufﬁcient for the existence of an Eulerian circuit in a connected graph. 4. Generate the remaining 4! − 6 = 18 possible assignments, compute their costs, and ﬁnd the one with the minimal cost. 5. Make the size of your counterexample as small as possible. 6. Rephrase the problem so that the sum of elements in one subset, rather than two, needs to be checked on each try of a possible partition. 7. Follow the deﬁnitions of a clique and of an exhaustivesearch algorithm. 8. Try all possible orderings of the elements given. 9. Use common formulas of elementary combinatorics. 10. a. Add all the elements in the magic square in two different ways. b. What combinatorial objects do you have to generate here?
Chapter 4
515
11. a. For testing, you may use alphametic collections available on the Internet. b. Given the absence of electronic computers in 1924, you must refrain here from using the Internet.
Exercises 3.5 1. a. Use the deﬁnitions of the adjacency matrix and adjacency lists given in Section 1.4. b. Perform the DFS traversal the same way it is done for another graph in the text (see Figure 3.10). 2. Compare the efﬁciency classes of the two versions of DFS for sparse graphs. 3. a. What is the number of such trees equal to? b. Answer this question for connected graphs ﬁrst. 4. Perform the BFS traversal the same way it is done in the text (see Figure 3.11). 5. You may use the fact that the level of a vertex in a BFS tree indicates the number of edges in the shortest (minimumedge) path from the root to that vertex. 6. a. What property of a BFS forest indicates a cycle’s presence? (The answer is similar to the one for a DFS forest.) b. The answer is no. Find two examples supporting this answer. 7. Given the fact that both traversals can reach a new vertex if and only if it is adjacent to one of the previously visited vertices, which vertices will be visited by the time either traversal halts (i.e., its stack or queue becomes empty)? 8. Use a DFS forest and a BFS forest for parts (a) and (b), respectively. 9. Use either DFS or BFS. 10. a. Follow the instructions of the problem’s statement. b. Trying both traversals should lead you to a correct answer very fast. 11. You can apply BFS without an explicit sketch of a graph representing the states of the puzzle.
CHAPTER 4 Exercises 4.1 1. Solve the problem for n = 1. 2. You may consider pouring soda from a ﬁlled glass into an empty glass as one move. 3. It’s easier to use the bottomup approach. 4. Use the fact that all the subsets of an nelement set S = {a1, . . . , an} can be divided into two groups: those that contain an and those that do not. 5. The answer is no.
516
Hints to Exercises
6. Use the same idea that underlies insertion sort. 7. Trace the algorithm as we did in the text for another input (see Figure 4.4). 8. a. The sentinel should stop the smallest element from moving beyond the ﬁrst position in the array. b. Repeat the analysis performed in the text for the sentinel version. 9. Recall that one can access elements of a singly linked list only sequentially. 10. Compare the running times of the algorithm’s inner loop. 11. a. Answering the questions for an array of three elements should lead to the general answers. b. Assume for simplicity that all elements are distinct and that inserting A[i] in each of the i + 1 possible positions among its predecessors is equally likely. Analyze the sentinel version of the algorithm ﬁrst. 12. a. Note that it’s more convenient to sort sublists in parallel, i.e., compare A[0] with A[hi ], then A[1] with A[1 + hi ], and so on. b. Recall that, generally speaking, sorting algorithms that can exchange elements far apart are not stable.
Exercises 4.2 1. Trace the algorithm as it is done in the text for another digraph (see Figure 4.7). 2. a. You need to prove two assertions: (i) if a digraph has a directed cycle, then the topological sorting problem does not have a solution; (ii) if a digraph has no directed cycles, then the problem has a solution. b. Consider an extreme type of a digraph. 3. a. How does it relate to the time efﬁciency of DFS? b. Do you know the length of the list to be generated by the algorithm? Where should you put, say, the ﬁrst vertex being popped off a DFS traversal stack for the vertex to be in its ﬁnal position? 4. Try to do this for a small example or two. 5. Trace the algorithm on the instances given as it is done in the section (see Figure 4.8). 6. a. Use a proof by contradiction. b. If you have difﬁculty answering the question, consider an example of a digraph with a vertex with no incoming edges and write down its adjacency matrix. c. The answer follows from the deﬁnitions of the source and adjacency lists. 7. For each vertex, store the number of edges entering the vertex in the remaining subgraph. Maintain a queue of the source vertices. 9. a. Trace the algorithm on the input given by following the steps of the algorithm as indicated.
Chapter 4
517
b. Determine the efﬁciency for each of the three principal steps of the algorithm and then determine the overall efﬁciency. Of course, the answers depend on whether a digraph is represented by its adjacency matrix or by its adjacency lists. 10. Take advantage of topological sorting and the graph’s symmetry.
Exercises 4.3 1. Use standard formulas for the numbers of these combinatorial objects. For the sake of simplicity, you may assume that generating one combinatorial object takes the same time as, say, one assignment. 2. We traced the algorithms on smaller instances in the section. 3. See an outline of this algorithm in the section. 4. a. Trace the algorithm for n = 2; take advantage of this trace in tracing the algorithm for n = 3 and then use the latter for n = 4. b. Show that the algorithm generates n! permutations and that all of them are distinct. Use mathematical induction. c. Set up a recurrence relation for the number of swaps made by the algorithm. Find its solution and the solution’s order of growth. You may need the formula: e ≈ ni=0 i!1 for large values of n. 5. We traced both algorithms on smaller instances in the section. 6. Tricks become boring after they have been given away. 7. This is not a difﬁcult exercise because of the obvious way of getting bit strings of length n from bit strings of length n − 1. 8. You may still mimic the binary addition without using it explicitly. 9. Just trace the algorithms for n = 4. 10. There are several decreaseand–conquer algorithms for this problem. They are more subtle than one might expect. Generating combinations in a predeﬁned order (increasing, decreasing, lexicographic) helps with both a design and a correctness proof. The following simple property is very helpful in that regard. Assuming with n−i no loss of generality that the underlying set is {1, 2, . . . , n}, there are k−1 ksubsets whose smallest element is i, i = 1, 2, . . . , n − k + 1. 11. Represent the disk movements by ﬂipping bits in a binary ntuple. 12. Thinking about the switches as bits of a bit string could be helpful but not necessary.
Exercises 4.4 1. Take care of the length of the longest piece present.
518
Hints to Exercises
2. If the instance of size n is to compute log2 n, what is the instance of size n/2? What is the relationship between the two? 3. For part (a), take advantage of the formula that gives the immediate answer. The most efﬁcient prop for answering questions (b)–(d) is a binary search tree that mirrors the algorithm’s operations in searching for an arbitrary search key. 4. Estimate the ratio of the average number of key comparisons made by sequential search to the average number made by binary search in successful searches. 5. How would you reach the middle element in a linked list? 6. a. Use the comparison K ≤ A[m] where m ← (l + r)/2 until l = r. Then check whether the search is successful or not. b. The analysis is almost identical to that of the text’s version of binary search. 7. Number the pictures and use this numbering in your questions. 8. The algorithm is quite similar to binary search, of course. In the worst case, how many key comparisons does it make on each iteration and what fraction of the array remains to be processed? 9. Start by comparing the middle element A[m] with m + 1. 10. It is obvious how one needs to proceed if n mod 3 = 0 or n mod 3 = 1; it is somewhat less so if n mod 3 = 2. 11. a. Trace the algorithm for the numbers given as it is done in the text for another input (see Figure 4.14b). b. How many iterations does the algorithm perform? 12. You may implement the algorithm either recursively or nonrecursively. 13. The fastest way to answer the question is to use the formula that exploits the binary representation of n, which is mentioned at the end of the section. 14. Use the binary representation of n. 15. a. Use forward substitutions (see Appendix B) into the recurrence equations given in the text. b. On observing the pattern in the ﬁrst 15 values of n obtained in part (a), express it analytically. Then prove its validity by mathematical induction. c. Start with the binary representation of n and translate into binary the formula for J (n) obtained in part (b).
Exercises 4.5 1. a. The answer follows immediately from the formula underlying Euclid’s algorithm. b. Let r = m mod n. Investigate two cases of r’s value relative to n’s value.
Chapter 5
519
2. Trace the algorithm on the input given, as is done in the section for another input. 3. The nonrecursive version of the algorithm was applied to a particular instance in the section’s example. 4. Write an equation of the straight line through the points (l, A[l]) and (r, A[r]) and ﬁnd the x coordinate of the point on this line whose y coordinate is v. 5. Construct an array for which interpolation search decreases the remaining subarray by one element on each iteration. 6. a. Solve the inequality log2 log2 n + 1 > 6. log log n
b. Compute limn→∞ log n . Note that to within a constant multiple, one can consider the logarithms to be natural, i.e., base e. 7. a. The deﬁnition of the binary search tree suggests such an algorithm. b. What is the worstcase input for your algorithm? How many key comparisons does it make on such an input? 8. a. Consider separately three cases, (i) the key’s node is a leaf, (ii) the key’s node has one child, (iii) the key’s node has two children. b. Assume that you know a location of the key to be deleted. 9. Starting at an arbitrary vertex of the graph, traverse a sequence of its untraversed edges until either all the edges are traversed or no untraversed edge is available. 10. Follow the plan used in the section for analyzing the normal version of the game. 11. Play several rounds of the game on the graph paper to become comfortable with the problem. Considering special cases of the spoiled square’s location should help you to solve it. 12. Do yourself a favor: try to design an algorithm on your own. It does not have to be optimal, but it should be reasonably efﬁcient. 13. Start by comparing the search number with the last element in the ﬁrst row.
CHAPTER 5 Exercises 5.1 1. In more than one respect, this question is similar to the divideandconquer computation of the sum of n numbers. 2. Unlike Problem 1, a divideandconquer algorithm for this problem can be more efﬁcient by a constant factor than the bruteforce algorithm. 3. How would you compute a 8 by solving two exponentiation problems of size 4? How about a 9?
520
Hints to Exercises
4. Look at the notations used in the theorem’s statement. 5. Apply the Master Theorem. 6. Trace the algorithm as it was done for another input in the section. 7. How can mergesort reverse a relative ordering of two elements? 8. a. Use backward substitutions, as usual. b. What inputs minimize the number of key comparisons made by mergesort? How many comparisons are made by mergesort on such inputs during the merging stage? c. Do not forget to include key moves made both before the split and during the merging. 9. Modify mergesort to solve the problem. 11. A divideandconquer algorithm works by reducing a problem’s instance to several smaller instances of the same problem.
Exercises 5.2 1. We traced the algorithm on another instance in the section. 2. Use the rules for stopping the scans. 3. The deﬁnition of stability of a sorting algorithm was given in Section 1.3. Your example does not have to be large. 4. Trace the algorithm to see on which inputs index i gets out of bounds. 5. Study what the section’s version of quicksort does on such arrays. You should base your answers on the number of key comparisons, of course. 6. Where will splits occur on the inputs in question? 7. a. Computing the ratio n2 /(n log2 n) for n = 106 is incorrect. b. Think the bestcase and worstcase inputs. 8. Use the partition idea. 9. a. You may want to ﬁrst solve the twocolor ﬂag problem, i.e., rearrange efﬁciently an array of R’s and B’s. (A similar problem is Problem 8 in this section’s exercises.) b. Extend the deﬁnition of a partition. 11. Use the partition idea.
Exercises 5.3 1. The problem is almost identical to the one discussed in the section. 2. Trace the algorithm on a small input. 3. This can be done by an algorithm discussed in an earlier chapter of the book. 4. Use strong induction on the number of internal nodes.
Chapter 5
521
5. This is a standard exercise that you have probably done in your data structures course. With the traversal deﬁnitions given at the end of the section, you should be able to trace them even if you have never encountered these algorithms before. 6. Your pseudocode can simply mirror the traversal deﬁnition. 7. If you do not know the answer to this important question, you may want to check the results of the traversals on a small binary search tree. For a proof, answer this question: What can be said about two nodes with keys k1 and k2 if k1 < k2 ? 8. Find the root’s label of the binary tree ﬁrst, and then identify the labels of the nodes in its left and right subtrees. 9. Use strong induction on the number of internal nodes. 11. Breaking the chocolate bar can be represented by a binary tree.
Exercises 5.4 1. You might want to answer the question for n = 2 ﬁrst and then generalize it. 2. Trace the algorithm on the input given. You will have to use it again in order to compute the products of twodigit numbers as well. 3. a. Take logarithms of both sides of the equality. b. What did we use the closedform formula for? 4. a. How do we multiply by powers of 10? b. Try to repeat the argument for, say, 98 ∗ 76. 5. Counting the number of onedigit additions made by the penandpencil algorithm in multiplying, say, two fourdigit numbers, should help answer the general question. 6. Check the formulas by simple algebraic manipulations. 7. Trace Strassen’s algorithm on the input given. (It takes some work, but it would have been much more of it if you were asked to stop the recursion when n = 1.) It is a good idea to check your answer by multiplying the matrices by the bruteforce (deﬁnitionbased) algorithm, too. 8. Use the method of backward substitutions to solve the recurrence given in the text. 9. The recurrence for the number of multiplications in Pan’s algorithm is similar to that for Strassen’s algorithm. Use the Master Theorem to ﬁnd the order of growth of its solution.
Exercises 5.5 1. a. How many points need to be considered in the combiningsolutions stage of the algorithm?
522
Hints to Exercises
b. Design a simpler algorithm in the same efﬁciency class. 2. Divide the rectangle in Figure 5.7b into eight congruent rectangles and show that each of these rectangles can contain no more than one point of interest. 3. Recall (see Section 5.1) that the number of comparisons made by mergesort in the worst case is Cworst (n) = n log2 n − n + 1 (for n = 2k ). You may use just the highestorder term of this formula in the recurrence you need to set up. 6. The answer to part (a) comes directly from a textbook on plane geometry. 7. Use the formula relating the value of a determinant with the area of a triangle. 8. It must be in (n), of course. (Why?) 9. Design a sequence of n points for which the algorithm decreases the problem’s size just by 1 on each of its recursive calls. 11. Apply an idea used in this section to construct a decagon with its vertices at ten given points. 12. The path cannot cross inside the fenced area, but it can go along the fence.
CHAPTER 6 Exercises 6.1 1. This problem is similar to one of the examples in the section. 2. a. Compare every element in one set with all the elements in the other. b. In fact, you can use presorting in three different ways: sort elements of just one of the sets, sort elements of each of the sets separately, and sort elements of the two sets together. 3. a. How do we ﬁnd the smallest and largest elements in a sorted list? b. The bruteforce algorithm and the divideandconquer algorithm are both linear. 4. Use the known results about the averagecase comparison numbers of the algorithms in this question. 5. a. The problem is similar to one of the preceding problems in these exercises. b. How would you solve this problem if the student information were written on index cards? Better yet, think how somebody else, who has never taken a course on algorithms but possesses a good dose of common sense, would solve this problem. 6. a. Many problems of this kind have exceptions for one particular conﬁguration of points. As to the question about a solution’s uniqueness, you can get the answer by considering a few small “random” instances of the problem. b. Construct a polygon for a few small “random” instances of the problem. Try to construct polygons in some systematic fashion.
Chapter 6
523
7. It helps to think about real numbers as ordered points on the real line. Considering the special case of s = 0, with a given array containing both negative and positive numbers, might be helpful, too. 8. After sorting the ai ’s and bi ’s, the problem can be solved in linear time. 9. Start by sorting the number list given. 10. a. Sort the points in nondecreasing order of their x coordinates and then scan them right to left. b. Think of choice problems with two desirable characteristics to take into account. 11. Use the presorting idea twice.
Exercises 6.2 1. Trace the algorithm as we did in solving another system in the section. 2. a. Use the Gaussian elimination results as explained in the text. b. It is one of the varieties of the transformandconquer technique. Which one? 3. To ﬁnd the inverse, you can either solve the system with three simultaneous righthand side vectors representing the columns of the 3 × 3 identity matrix or use the LU decomposition of the system’s coefﬁcient matrix found in Problem 2. 4. Though the ﬁnal answer is correct, its derivation contains an error you have to ﬁnd. 5. Pseudocode of this algorithm is quite straightforward. If you are in doubt, see the section’s example tracing the algorithm. The order of growth of the algorithm’s running time can be found by following the standard plan for the analysis of nonrecursive algorithms. 6. Estimate the ratio of the algorithm running times by using the approximate formulas for the number of divisions and the number of multiplications in both algorithms. 7. a. This is a “normal” case: one of the two equations should not be proportional to the other. b. The coefﬁcients of one equation should be the same or proportional to the corresponding coefﬁcients of the other equation, whereas the righthand sides should not. c. The two equations should be either the same or proportional to each other (including the righthand sides). 8. a. Manipulate the matrix rows above a pivot row the same way the rows below the pivot row are changed. b. Are the GaussJordan method and Gaussian elimination based on the same algorithm design technique or on different ones?
524
Hints to Exercises
c. Derive a formula for the number of multiplications in the GaussJordan method in the same manner this was done for Gaussian elimination in the section. 9. How long will it take to compute the determinant compared to the time needed to apply Gaussian elimination to the system? 10. a. Apply Cramer’s rule to the system given. b. How many distinct determinants are there in the Cramer’s rule formulas? 11. a. If xij is the number of times the panel in the ith row and j th column needs to be toggled in a solution, what can be said about xij ? After you answer this question, show that the binary matrix representing an initial state of the board can be represented as a linear combination (in modulo2 arithmetic) of n2 binary matrices each representing the effect of toggling an individual panel. b. Set up a system of four equations in four unknowns (see part (a)) and solve it by Gaussian elimination, performing all operations in modulo2 arithmetic. c. If you believe that a system of nine equations in nine unknowns is too large to solve by hand, write a program to solve the problem.
Exercises 6.3 1. Use the deﬁnition of AVL trees. Do not forget that an AVL tree is a special case of a binary search tree. 2. For both questions, it is easier to construct the required trees bottom up, i.e., for smaller values of n ﬁrst. 3. The single Lrotation and the double RLrotation are the mirror images of the single Rrotation and the double LRrotation, whose diagrams can be found in the section. 4. Insert the keys one after another doing appropriate rotations the way it was done in the section’s example. 5. a. An efﬁcient algorithm immediately follows from the deﬁnition of the binary search tree of which the AVL tree is a special case. b. The correct answer is opposite to the one that immediately comes to mind. 7. a. Trace the algorithm for the input given (see Figure 6.8 for an example). b. Keep in mind that the number of key comparisons made in searching for a key in a 23 tree depends not only on its node’s depth but also on whether the key is the ﬁrst or second one in the node. 8. False; ﬁnd a simple counterexample. 9. Where will the smallest and largest keys be located?
Chapter 6
525
Exercises 6.4 1. a. Trace the algorithm outlined in the text on the input given. b. Trace the algorithm outlined in the text on the input given. c. A mathematical fact may not be established by checking its validity on a single example. 2. For a heap represented by an array, only the parental dominance requirement needs to be checked. 3. a. What structure does a complete tree of height h with the largest number of nodes have? What about a complete tree with the smallest number of nodes? b. Use the results established in part (a). 4. First, express the righthand side as a function of h. prove the obtained Then, equality by either using the formula for the sum i2i given in Appendix A or by mathematical induction on h. 5. a. Where in a heap should one look for its smallest element? b. Deleting an arbitrary element of a heap can be done by generalizing the algorithm for deleting its root. 6. Fill in a table with the time efﬁciency classes of efﬁcient implementations of the three operations: ﬁnding the largest element, ﬁnding and deleting the largest element, and adding a new element. 7. Trace the algorithm on the inputs given (see Figure 6.14 for an example). 8. As a rule, sorting algorithms that can exchange farapart elements are not stable. 9. One can claim that the answers are different for the two principal representations of a heap. 10. This algorithm is less efﬁcient than heapsort because it uses the array rather than the heap to implement the priority queue. 12. Pick the spaghetti rods up in a bundle and place them end down (i.e., vertically) onto a tabletop.
Exercises 6.5 1. Set up sums and simplify them by using the standard formulas and rules for sum manipulation. Do not forget to include the multiplications outside the inner loop. 2. Take advantage of the fact that the value of x i can be easily computed from the previously computed x i−1. 3. a. Use the formulas for the number of multiplications (and additions) for both algorithms. b. Does Horner’s rule use any extra memory?
526
Hints to Exercises
4. Apply Horner’s rule to the instance given the same way it is applied to another one in the section. 5. Compute p(2) where p(x) = x 8 + x 7 + x 5 + x 2 + 1. 6. If you implement the algorithm for long division by x − c efﬁciently, the answer might surprise you. 7. a. Trace the lefttoright binary exponentiation algorithm on the instance given the same way it is done for another instance in the section. b. The answer is yes: the algorithm can be extended to work for the zero exponent as well. How? 8. Trace the righttoleft binary exponentiation algorithm on the instance given the same way it is done for another instance in the section. 9. Compute and use the binary digits of n “on the ﬂy.” 10. Use a formula for the sum of the terms of this special kind of a polynomial. 11. Compare the number of operations needed to implement the task in question. 12. Although there exists exactly one such polynomial, there are several different ways to represent it. You may want to generalize Lagrange’s interpolation formula for n = 2: p(x) = y1
x − x2 x − x1 + y2 x 1 − x2 x2 − x1
Exercises 6.6 1. a. Use the rules for computing lcm(m, n) and gcd(m, n) from the prime factors of m and n. b. The answer immediately follows from the formula for computing lcm (m, n). 2. Use a relationship between minimization and maximization problems. 3. Prove the assertion by induction on k. 4. a. Base your algorithm on the following observation: a graph contains a cycle of length 3 if and only if it has two adjacent vertices i and j that are also connected by a path of length 2. b. Do not jump to a conclusion in answering this question. 5. An easier solution is to reduce the problem to another one with a known algorithm. Since we did not discuss many geometric algorithms in the book, it should not be difﬁcult to ﬁgure out to which one this problem needs to be reduced. 6. Express this problem as a maximization problem of a function in one variable. 7. Introduce doubleindexed variables xij to indicate an assignment of the ith person to the j th job.
Chapter 7
527
8. Take advantage of the speciﬁc features of this instance to reduce the problem to one with fewer variables. 9. Create a new graph. 10. Solve ﬁrst the onedimensional version of the post ofﬁce location problem (Problem 3(a) in Exercises 3.3). 11. a. Create a statespace graph for the problem as it is done for the rivercrossing puzzle in the section. b. Create a statespace graph for the problem. c. Look at the state obtained after the ﬁrst six river crossings in the solution to part (b). 12. The problem can be solved by reduction to a wellknown problem about a graph traversal.
CHAPTER 7 Exercises 7.1 1. Yes, it is possible. How? 2. Check the algorithm’s pseudocode to see what it does upon encountering equal values. 3. Trace the algorithm on the input given (see Figure 7.2 for an example). 4. Check whether the algorithm can reverse a relative ordering of equal elements. 5. Where will A[i] be in the sorted array? 6. Take advantage of the standard traversals of such trees. 7. a. Follow the deﬁnitions of the arrays B and C in the description of the method. b. Find, say, B[C[3]] for the example in part (a). 8. Start by ﬁnding the target positions for all the statures. 9. a. Use linked lists to hold nonzero elements of the matrices. b. Represent each of the given polynomials by a linked list with nodes containing exponent i and coefﬁcient ai for each nonzero term ai x i . 10. You may use a search of the literature/Internet to answer this question.
Exercises 7.2 1. Trace the algorithm in the same way it is done in the section for another instance of the stringmatching problem. 2. A special alphabet notwithstanding, this application is not different than applications to naturallanguage strings.
528
Hints to Exercises
3. For each pattern, ﬁll in its shift table and then determine the number of character comparisons (both successful and unsuccessful) on each trial and the total number of trials. 4. Find an example of a binary string of length m and a binary string of length n (n ≥ m) so that Horspool’s algorithm makes a. the largest possible number of character comparisons before making the smallest possible shift. b. the smallest possible number of character comparisons. 5. It is logical to try a worstcase input for Horspool’s algorithm. 6. Can the algorithm shift the pattern by more than one position without the possibility of missing another matching substring? 7. For each pattern, ﬁll in the two shift tables and then determine the number of character comparisons (both successful and unsuccessful) on each trial and the total number of trials. 8. Check the description of the BoyerMoore algorithm. 9. Check the descriptions of the algorithms. 11. a. A bruteforce algorithm ﬁts the bill here. b. Enhance the input before a search.
Exercises 7.3 1. Apply the open hashing (separate chaining) scheme to the input given, as is done in the text for another input (see Figure 7.5). Then compute the largest number and average number of comparisons for successful searches in the constructed table. 2. Apply the closed hashing (open addressing) scheme to the input given as it is done in the text for another input (see Figure 7.6). Then compute the largest number and average number of comparisons for successful searches in the constructed table. 3. How many different addresses can such a hash function produce? Would it distribute keys evenly? 4. The question is quite similar to computing the probability of having the same result in n throws of a fair die. 5. Find the probability that n people have different birthdays. As to the hashing connection, what hashing phenomenon deals with coincidences? 6. a. There is no need to insert a new key at the end of the linked list it is hashed to. b. Which operations are faster in a sorted linked list and why? For sorting, do we have to copy all elements in the nonempty lists in an array and then apply a general purpose sorting algorithm, or is there a way to take advantage of the sorted order in each of the nonempty linked lists?
Chapter 8
529
7. A direct application of hashing solves the problem. 8. Consider this question as a minireview: the answers are in Section 7.3 for hashing and in the appropriate sections of the book for the others. Of course, you should use the best algorithms available. 9. If you need to refresh your memory, check the book’s table of contents.
Exercises 7.4 1. Thinking about searching for information should lead to a variety of examples. 2. a. Use the standard rules of sum manipulation and, in particular, the geometric series formula. b. You will need to take logarithms base m/2 in your derivation. 3. Find this value from the inequality in the text that provides the upperbound of the Btree’s height. 4. Follow the insertion algorithm outlined in the section. 5. The algorithm is suggested by the deﬁnition of the Btree. 6. a. Just follow the description of the algorithm given in the statement of the problem. Note that a new key is always inserted in a leaf and that full nodes are always split on the way down, even though the leaf for the new key may have a room for it. b. Can a split of a full node cause a cascade of splits through the chain of its ancestors? Can we get a taller search tree than necessary?
CHAPTER 8 Exercises 8.1 1. Compare the deﬁnitions of the two techniques. 2. Use the table generated by the dynamic programming algorithm in solving the problem’s instance in Example 1 of the section. 3. a. The analysis is similar to that of the topdown recursive computation of the nth Fibonacci number in Section 2.5. b. Set up and solve a recurrence for the number of candidate solutions that need to be processed by the exhaustive search algorithm. 4. Apply the dynamic programming algorithm to the instance given as it is done in Example 2 of the section. Note that there are two optimal coin combinations here. 5. Adjust formula (8.5) for inadmissible cells and their immediate neighbors. 6. The problem is similar to the changemaking problem discussed in the section.
530
Hints to Exercises
7. a. Relate the number of the rook’s shortest paths to the square in the ith row and the j th column of the chessboard to the numbers of the shortest paths to the adjacent squares. b. Consider one shortest path as 14 consecutive moves to adjacent squares. 8. One can solve the problem in quadratic time. 9. Use a wellknown formula from elementary combinatorics relating C(n, k) to smaller binomial coefﬁcients. 10. a. Topologically sort the dag’s vertices ﬁrst. b. Create a dag with n + 1 vertices: one vertex to start and the others to represent the coins given. 11. Let F (i, j ) be the order of the largest allzero submatrix of a given matrix with its low right corner at (i, j ). Set up a recurrence relating F (i, j ) to F (i − 1, j ), F (i, j − 1), and F (i − 1, j − 1). 12. a. In the situation where teams A and B need i and j games, respectively, to win the series, consider the result of team A winning the game and the result of team A losing the game. b. Set up a table with ﬁve rows (0 ≤ i ≤ 4) and ﬁve columns (0 ≤ j ≤ 4) and ﬁll it by using the recurrence derived in part (a). c. Your pseudocode should be guided by the recurrence set up in part (a). The efﬁciency answers follow immediately from the table’s size and the time spent on computing each of its entries.
Exercises 8.2 1. a. Use formulas (8.6)–(8.7) to ﬁll in the appropriate table, as is done for another instance of the problem in the section. b., c. What would the equality of the two terms in max{F (i − 1, j ), vi + F (i − 1, j − wi )} mean? 2. a. Write pseudocode to ﬁll the table in Figure 8.4 (say, row by row) by using formulas (8.6)–(8.7). b. An algorithm for identifying an optimal subset is outlined in the section via an example. 3. How many values does the algorithm compute? How long does it take to compute one value? How many table cells need to be traversed to identify the composition of an optimal subset? 4. Use the deﬁnition of F (i, j ) to check whether it is always true that a. F (i, j − 1) ≤ F (i, j ) for 1 ≤ j ≤ W. b. F (i − 1, j ) ≤ F (i, j ) for 1 ≤ i ≤ n. 5. The problem is similar to one of the problems discussed in Section 8.1.
Chapter 8
531
6. Trace the calls of the function MemoryKnapsack(i, j ) on the instance in question. (An application to another instance can be found in the section.) 7. The algorithm applies formula (8.6) to ﬁll some of the table’s cells. Why can we still assert that its efﬁciencies are in (nW )? 8. One of the reasons deals with the time efﬁciency; the other deals with the space efﬁciency. 9. You may want to include algorithm visualizations in your report.
Exercises 8.3 1. Continue applying formula (8.8) as prescribed by the algorithm. 2. a. The algorithm’s time efﬁciency can be investigated by following the standard plan of analyzing the time efﬁciency of a nonrecursive algorithm. b. How much space do the two tables generated by the algorithm use? 3. k = R[1, n] indicates that the root of an optimal tree is the kth key in the list of ordered keys a1, . . . , an. The roots of its left and right subtrees are speciﬁed by R[1, k − 1] and R[k + 1, n], respectively. 4. Use a spacefortime tradeoff. 5. If the assertion were true, would we not have a simpler algorithm for constructing an optimal binary search tree? 6. The structure of the tree should simply minimize the average depth of its nodes. Do not forget to indicate a way to distribute the keys among the nodes of the tree. 7. a. Since there is a onetoone correspondence between binary search trees for a given set of n orderable keys and binary trees with n nodes (why?), you can count the latter. Consider all the possibilities of partitioning the nodes between the left and right subtrees. b. Compute the values in question using the two formulas. c. Use the formula for the nth Catalan number and Stirling’s formula for n!. 8. Change the bounds of the innermost loop of algorithm OptimalBST by exploiting the monotonicity of the root table mentioned at the end of the section. 9. Assume that a1, . . . , an are distinct keys ordered from the smallest to the largest, p1, . . . , pn are the probabilities of searching for them, and q0, q1, . . . , qn are probabilities of unsuccessful searches for keys in intervals (−∞, a1), (a1, a2 ), . . . , (an, ∞), respectively; (p1 + . . . + pn) + (q0 + . . . + qn) = 1. Set up a recurrence relation similar to recurrence (8.8) for the expected number of key comparisons that takes into account both successful and unsuccessful searches. 10. See the memory function solution for the knapsack problem in Section 8.2. 11. a. It is easier to ﬁnd a general formula for the number of multiplications needed for computing (A1 . A2 ) . A3 and A1 . (A2 . A3) for matrices A1 with
532
Hints to Exercises
dimensions d0 × d1, A2 with dimensions d1 × d2 , and A3 with dimensions d2 × d3 and then choose some speciﬁc values for the dimensions to get a required example. b. You can get the answer