Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Manual Testers

Description

How to Find and Fix the Killer Software Bugs that Evade Conventional Testing

In Exploratory Software Testing, renowned software testing expert James Whittaker reveals the real causes of today's most serious, well-hidden software bugs--and introduces powerful new exploratory techniques for finding and correcting them.

Drawing on nearly two decades of experience working at the cutting edge of testing with Google, Microsoft, and other top software organizations, Whittaker introduces innovative new processes for manual testing that are repeatable, prescriptive, teachable, and extremely effective. Whittaker defines both in-the-small techniques for individual testers and in-the-large techniques to supercharge test teams. He also introduces a hybrid strategy for injecting exploratory concepts into traditional scripted testing. You'll learn when to use each, and how to use them all successfully.

Concise, entertaining, and actionable, this book introduces robust techniques that have been used extensively by real testers on shipping software, illuminating their actual experiences with these techniques, and the results they've achieved. Writing for testers, QA specialists, developers, program managers, and architects alike, Whittaker answers crucial questions such as:

* Why do some bugs remain invisible to automated testing--and how can I uncover them?

* What techniques will help me consistently discover and eliminate show stopper bugs?

* How do I make manual testing more effective--and less boring and unpleasant?

* What's the most effective high-level test strategy for each project?

* Which inputs should I test when I can't test them all?

* Which test cases will provide the best feature coverage?

* How can I get better results by combining exploratory testing with traditional script or scenario-based testing?

* How do I reflect feedback from the development process, such as code changes?

CONTENTS:

Foreword by Alan Page xv

Preface xvii

Chapter 1 The Case for Software Quality 1

The Magic of Software 1

The Failure of Software 4

Conclusion 9

Exercises 9

Chapter 2 The Case for Manual Testing 11

The Origin of Software Bugs 11

Preventing and Detecting Bugs 12

Manual Testing 14

Conclusion 19

Exercises 20

Chapter 3 Exploratory Testing in the Small 21

So You Want to Test Software? 21

Testing Is About Varying Things 23

User Input 23

    What You Need to Know About User Input 24

    How to Test User Input 25

State 32

    What You Need to Know About Software State 32

    How to Test Software State 33

Code Paths 35

User Data 36

Environment 36

Conclusion 37

Exercises 38

Chapter 4 Exploratory Testing in the Large 39

Exploring Software 39

The Tourist Metaphor 41

Touring Tests 43

    Tours of the Business District 45

    Tours Through the Historical District 51

    Tours Through the Entertainment District 52

    Tours Through the Tourist District 55

    Tours Through the Hotel District 58

    Tours Through the Seedy District 60

Putting the Tours to Use 62

Conclusion 63

Exercises 64

Chapter 5 Hybrid Exploratory Testing Techniques 65

Scenarios and Exploration 65

Applying Scenario-Based Exploratory Testing 67

Introducing Variation Through Scenario Operators 68

    Inserting Steps 68

    Removing Steps 69

    Replacing Steps 70

    Repeating Steps 70

    Data Substitution 70

    Environment Substitution 71

Introducing Variation Through Tours 72

    The Money Tour 73

    The Landmark Tour 73

    The Intellectual Tour 73

    The Back Alley Tour 73

    The Obsessive-Compulsive Tour 73

    The All-Nighter Tour 74

    The Saboteur 74

    The Collector's Tour 74

    The Supermodel Tour 74

    The Supporting Actor Tour 74

    The Rained-Out Tour 75

    The Tour-Crasher Tour 75

Conclusion 75

Exercises 76

Chapter 6 Exploratory Testing in Practice 77

The Touring Test 77

Touring the Dynamics AX Client 78

    Useful Tours for Exploration 79

    The Collector's Tour and Bugs as Souvenirs 81

    Tour Tips 84

Using Tours to Find Bugs 86

    Testing a Test Case Management Solution 86

    The Rained-Out Tour 87

    The Saboteur 88

    The FedEx Tour 89

    The TOGOF Tour 90

The Practice of Tours in Windows Mobile Devices 90

    My Approach/Philosophy to Testing 91

    Interesting Bugs Found Using Tours 92

    Example of the Saboteur 94

    Example of the Supermodel Tour 94

The Practice of Tours in Windows Media Player 97

    Windows Media Player 97

    The Garbage Collector's Tour 97

    The Supermodel Tour 100

    The Intellectual Tour 100

    The Intellectual Tour: Boundary Subtour 102

    The Parking Lot Tour and the Practice of Tours in Visual Studio Team System Test Edition 103

Tours in Sprints 103

Parking Lot Tour 105

Test Planning and Managing with Tours 106

Defining the Landscape 106

Planning with Tours 107

Letting the Tours Run 109

Analysis of Tour Results 109

Making the Call: Milestone/Release 110

    In Practice 110

Conclusion 111

Exercises 111

Chapter 7 Touring and Testing's Primary Pain Points 113

The Five Pain Points of Software Testing 113

Aimlessness 114

    Define What Needs to Be Tested 115

    Determine When to Test 115

    Determine How to Test 116

Repetitiveness 116

    Know What Testing Has Already Occurred 117

    Understand When to Inject Variation 117

Transiency 118

Monotony 119

Memorylessness 120

Conclusion 121

Exercises 122

Chapter 8 The Future of Software Testing 123

Welcome to the Future 123

The Heads-Up Display for Testers 124

Testipedia 126

    Test Case Reuse 127

    Test Atoms and Test Molecules 128

Virtualization of Test Assets 129

Visualization 129

Testing in the Future 132

Post-Release Testing 134

Conclusion 134

Exercises 135

Appendix A Building a Successful Career in Testing 137

How Did You Get into Testing? 137

Back to the Future 138

The Ascent 139

The Summit 140

The Descent 142

Appendix B A Selection of JW's Professorial Blog 143

Teach Me Something 143

Software's Ten Commandments 143

    1. Thou Shalt Pummel Thine App with Multitudes of Input 145

    2. Thou Shalt Covet Thy Neighbor's Apps 145

    3. Thou Shalt Seek Thee Out the Wise Oracle 146

    4. Thou Shalt Not Worship Irreproducible Failures 146

    5. Thou Shalt Honor Thy Model and Automation 146

    6. Thou Shalt Hold Thy Developers Sins Against Them 147

    7. Thou Shalt Revel in App Murder (Celebrate the BSOD) 147

    8. Thou Shalt Keep Holy the Sabbath (Release) 148

    9. Thou Shalt Covet Thy Developer's Source Code 148

Testing Error Code 149

Will the Real Professional Testers Please Step Forward 151

    The Common Denominators I Found Are (In No Particular Order) 152

    My Advice Can Be Summarized as Follows 53

Strike Three, Time for a New Batter 154

    Formal Methods 154

    Tools 155

    Process Improvement 156

    The Fourth Proposal 156

Software Testing as an Art, a Craft and a Discipline 157

Restoring Respect to the Software Industry 160

    The Well-Intentioned but Off-Target Past 160

    Moving On to Better Ideas 161

    A Process for Analyzing Security Holes and Quality Problems 161

Appendix C An Annotated Transcript of JW's Microsoft Blog 165

Into the Blogoshere 165

July 2008 166

    Before We Begin 166

    PEST (Pub Exploration and Software Testing) 167

    Measuring Testers 168

    Prevention Versus Cure (Part 1) 169

    Users and Johns 170

    Ode to the Manual Tester 171

    Prevention Versus Cure (Part 2) 173

    Hail Europe! 174

    The Poetry of Testing 175

    Prevention Versus Cure (Part 3) 176

    Back to Testing 177

August 2008 178

    Prevention Versus Cure (Part 4) 179

    If Microsoft Is So Good at Testing, Why Does Your Software Still Suck? 180

    Prevention Versus Cure (Part 5) 183

    Freestyle Exploratory Testing 183

    Scenario-Based Exploratory Testing 183

    Strategy-Based Exploratory Testing 184

    Feedback-Based Exploratory Testing 184

    The Future of Testing (Part 1) 184

    The Future of Testing (Part 2) 186

September 2008 188

    On Certification 188

    The Future of Testing (Part 3) 189

    The Future of Testing (Part 4) 191

    The Future of Testing (Part 5) 192

October 2008 193

    The Future of Testing (Part 6) 194

    The Future of Testing (Part 7) 195

    The Future of Testing (Part 8) 196

    Speaking of Google 198

    Manual Versus Automated Testing Again 198

November 2008 199

    Software Tester Wanted 200

    Keeping Testers in Test 200

December 2008 201

    Google Versus Microsoft and the Dev:Test Ratio Debate 201

January 2009 202

    The Zune Issue 203

    Exploratory Testing Explained 204

    Test Case Reuse 205

    More About Test Case Reuse 206

    I'm Back 207

    Of Moles and Tainted Peanuts 208

Index 211