Software Build Systems: Principles and Experience

Description

This book represents a thorough and extensive treatment of the software build process including the choices, benefits, and challenges of a well designed build process. I recommend it not only to all software build engineers but to all software developers since a well designed build process is key to an effective software development process.

 - Kevin Bodie, Director Software Development, Pitney Bowes Inc.

An excellent and detailed explanation of build systems, an important but often overlooked part of software development projects. The discussion of productivity as related to build systems is, alone, well worth the time spent reading this book.

 - John M. Pantone, Objectech Corporation, VP, IT Educator and Course Developer

Peter Smith provides an interesting and accessible look into the world of software build systems, distilling years of experience and covering virtually every type of tool in the build engineer's toolbox. Well organized, well written, and very thorough; I would recommend this book to anyone with a build system under their responsibility.

 - Jeff Overbey, Project Co-Lead, Photran

Software Build Systems teaches how to think about building software. It surveys the tools and techniques for building software products and the ways things go wrong. This book will appeal to those new to build systems as well as experienced build system engineers.

 - Monte Davidoff, Software Development Consultant, Alluvial Software, Inc.

Inadequate build systems can dramatically impact developer productivity. Bad dependencies, false compile errors, failed software images, slow compilation, and time-wasting manual processes are just some of the byproducts of a subpar build system. In Software Build Systems, software productivity expert Peter Smith shows you how to implement build systems that overcome all these problems, so you can deliver reliable software more rapidly, at lower cost.

Smith explains the core principles underlying highly efficient build systems, surveying both system features and usage scenarios. Next, he encapsulates years of experience in creating and maintaining diverse build systems - helping you make well-informed choices about tools and practices, and avoid common traps and pitfalls. Throughout, he shares a wide range of practical examples and lessons from multiple environments, including Java, C++, C, and C#. Coverage includes

* Mastering build system concepts, including source trees, build tools, and compilation tools

* Comparing five leading build tools: GNU Make, Ant, SCons, CMake, and the Eclipse IDE's integrated build features

* Ensuring accurate dependency checking and efficient incremental compilation

* Using metadata to assist debugging, profiling, and source code documentation

* Packaging software for installation on your target machine

* Best practices for managing complex version-control systems, build machines, and compilation tools

If you're a developer, this book will illuminate the issues involved in building and maintaining the build system that's best for your team. If you're a manager, you'll discover how to evaluate your team's build system and improve its effectiveness. And if you're a build guru, you'll learn how to optimize the performance and scalability of your build system, no matter how demanding your requirements are.

CONTENTS:

PREFACE xxi

Why Do Build Systems Become Complex? xxiii

The True Cost of a Build System xxiv

The Focus of This Book xxvii

Who Should Read This Book? xxviii

How This Book Is Organized xxix

Summary xxxi

ACKNOWLEDGMENTS xxxiii

ABOUT THE AUTHOR xxxv

PART I THE BASICS 1

Chapter 1 BUILD SYSTEM OVERVIEW 3

What Is a Build System? 3

Compiled Languages 4

Interpreted Languages 6

Web-Based Applications 6

Unit Testing 7

Static Analysis 8

Documentation Generation 9

Components of a Build System 10

Version-Control Tools 10

Source and Object Trees 11

Compilation Tools and Build Tools 13

Build Machines 14

Release Packaging and Target Machines 15

The Build Process and Build Description 16

How a Build System Is Used 18

Build-Management Tools 19

Build System Quality 21

Summary 21

Chapter 2 A MAKE-BASED BUILD SYSTEM 23

Calculator Example 23

Creating a Simple Makefile 26

Simplifying the Makefile 28

Additional Build Targets 29

Using a Framework 31

Summary 33

Chapter 3 THE RUNTIME VIEW OF A PROGRAM 35

Executable Programs 36

Native Machine Code 36

Monolithic System Images 37

Full Program Interpretation 38

Interpreted Byte Codes 39

Libraries 40

Static Linking 41

Dynamic Linking 42

Configuration and Data Files 43

Distributed Programs 44

Summary 46

Chapter 4 FILE TYPES AND COMPILATION TOOLS 47

C/C++ 48

Compilation Tools 49

Source Files 50

Assembly Language Files 52

Object Files 53

Executable Programs 56

Static Libraries 57

Dynamic Libraries 58

C++ Compilation 59

Java 60

Compilation Tools 61

Source Files 62

Object Files 63

Executable Programs 65

Libraries 67

C# 68

Compilation Tools 68

Source Files 69

Executable Programs 71

Libraries 74

Other File Types 76

UML-Based Code Generation 77

Graphic Images 79

XML Configuration Files 81

Internationalization and Resource Bundles 81

Summary 82

Chapter 5 SUBTARGETS AND BUILD VARIANTS 83

Building Subtargets 84

Building Different Editions of the Software 86

Specifying the Build Variant 87

Varying the Code 90

Building Different Target Architectures 94

Multiple Compilers 94

Platform-Specific Files/Functions 95

Multiple Object Trees 96

Summary 98

PART II THE BUILD TOOLS 99

Chapter 6 MAKE 107

The GNU Make Programming Language 108

Makefile Rules to Construct the Dependency Graph 109

Makefile Rule Types 110

Makefile Variables 112

Built-In Variables and Rules 114

Data Structures and Functions 116

Understanding Program Flow 119

Further Reading 122

Real-World Build System Scenarios 123

Scenario 1: Source Code in a Single Directory 123

Scenario 2(a): Source Code in Multiple Directories 125

Scenario 2(b): Recursive Make over Multiple Directories 126

Scenario 2(c): Inclusive Make over Multiple Directories 130

Scenario 3: Defining New Compilation Tools 137

Scenario 4: Building with Multiple Variants 138

Scenario 5: Cleaning a Build Tree 140

Scenario 6: Debugging Incorrect Builds 142

Praise and Criticism 144

Praise 144

Criticism 146

Evaluation 148

Similar Tools 149

BerkeleyMake 149

NMake 150

ElectricAccelerator and SparkBuild 151

Summary 153

Chapter 7 ANT 155

The Ant Programming Language 156

A Little More Than Hello World 157

Defining and Using Targets 158

Ant's Flow of Control 161

Defining Properties 162

Built-In and Optional Tasks 164

Selecting Multiple Files and Directories 168

Conditions 170

Extending the Ant Language 172

Further Reading 173

Real-World Build System Scenarios 174

Scenario 1: Source Code in a Single Directory 174

Scenario 2(a): Source Code in Multiple Directories 175

Scenario 2(b): Many Directories, with Multiple build.xml Files 175

Scenario 3: Defining New Compilation Tools 179

Scenario 4: Building with Multiple Variants 183

Scenario 5: Cleaning a Build Tree 188

Scenario 6: Debugging Incorrect Builds 188

Praise and Criticism 191

Praise 191

Criticism 191

Evaluation 193

Similar Tools 193

NAnt 194

MSBuild 194

Summary 196

Chapter 8 SCons 197

The SCons Programming Language 198

The Python Programming Language 199

Simple Compiling 202

Managing Build Environments 206

Program Flow and Dependency Analysis 210

Deciding When to Rebuild 212

Extending the Language 214

Other Interesting Features 218

Further Reading 219

Real-World Build System Scenarios 219

Scenario 1: Source Code in a Single Directory 219

Scenario 2(a): Source Code in Multiple Directories 219

Scenario 2(b): Multiple SConstruct Files 220

Scenario 3: Defining New Compilation Tools 222

Scenario 4: Building with Multiple Variants 224

Scenario 5: Cleaning a Build Tree 226

Scenario 6: Debugging Incorrect Builds 226

Praise and Criticism 229

Praise 230

Criticism 231

Evaluation 231

Similar Tools 232

Cons 232

Rake 233

Summary 235

Chapter 9 CMAKE 237

The CMake Programming Language 238

CMake Language Basics 239

Building Executable Programs and Libraries 240

Control Flow 243

Cross-Platform Support 246

Generating a Native Build System 248

Other Interesting Features and Further Reading 254

Real-World Build System Scenarios 255

Scenario 1: Source Code in a Single Directory 255

Scenario 2: Source Code in Multiple Directories 256

Scenario 3: Defining New Compilation Tools 257

Scenario 4: Building with Multiple Variants 259

Scenario 5: Cleaning a Build Tree 260

Scenario 6: Debugging Incorrect Builds 260

Praise and Criticism 261

Praise 261

Criticism 262

Evaluation 262

Similar Build Tools 263

Automake 263

Qmake 264

Summary 264

Chapter 10 ECLIPSE 267

The Eclipse Concepts and GUI 268

Creating Projects 269

Building a Project 276

Running a Project 282

Using the Internal Project Model 285

Other Build Features 286

Further Reading 288

Real-World Build System Scenarios 288

Scenario 1: Source Code in a Single Directory 288

Scenario 2: Source Code in Multiple Directories 290

Scenario 3: Defining New Compilation Tools 291

Scenario 4: Building with Multiple Variants 292

Scenario 5: Cleaning a Build Tree 295

Scenario 6: Debugging Incorrect Builds 296

Praise and Criticism 296

Praise 297

Criticism 297

Evaluation 298

Similar Build Tools 299

CDT for Eclipse, C/C++ Development Tooling 299

Summary 301

PART III ADVANCED TOPICS 303

Chapter 11 DEPENDENCIES 305

The Dependency Graph 307

Incremental Compilation 307

Full, Incremental, and Subtarget Builds 308

The Problem with Bad Dependencies 310

Problem: Missing Dependencies Causing a Runtime Error 310

Problem: Missing Dependencies Causing a Compile Error 311

Problem: Unwanted Dependencies Causing Excess Rebuilding 312

Problem: Unwanted Dependencies Causing Failed Dependency Analysis 312

Problem: Circular Dependencies 313

Problem: Implicit Sequencing As a Substitute for Dependencies 314

Problem: The Clean Target Doesn't Clean Everything 315

Step 1: Computing the Dependency Graph 315

Gathering Exact Dependencies 316

Caching the Dependency Graph 319

Updating the Cached Dependency Graph 320

Step 2: Determining Which Files Are Out-of-Date 324

Time Stamp-Based Methods 324

Checksum-Based Methods 326

Flag Comparison 328

Advanced Methods 329

Step 3: Sequencing the Compilation Steps 330

Summary 333

Chapter 12 BUILDING WITH METADATA 335

Debugging Support 336

Profiling Support 338

Coverage Support 340

Source Code Documentation 341

Unit Testing 344

Static Analysis 348

Adding Metadata to a Build System 349

Summary 350

Chapter 13 SOFTWARE PACKAGING AND INSTALLATION 351

Archive Files 352

Packaging Scripts 353

Other Archive Formats 356

Improvements 356

Package-Management Tools 359

The RPM Package Manager Format 360

The rpmbuild Process 361

An Example RPM Spec File 363

Creating the RPM File from the Spec File 369

Installing the RPM Example 371

Custom-Built GUI Installation Tools 373

The Nullsoft Scriptable Install System (NSIS) 374

The Installer Script 376

Defining the Pages 379

The License Page 380

Directory Selection 381

The Main Component 381

The Optional Components 383

Defining a Custom Page 385

The Installation Page and the Uninstaller 387

Summary 388

Chapter 14 VERSION MANAGEMENT 391

What Should Be Version-Controlled 392

Build Description Files 393

References to Tools 395

Large Binary Files 400

Source Tree Configurations 401

What Should Not Be in the Source Tree 402

Generated Files in the Source Tree 402

Generated Files Under Version Control 404

Build-Management Scripts 405

Version Numbering 406

Version-Numbering Systems 406

Coordinating and Updating the Version Number 407

Storing and Retrieving the Version Number 410

Summary 411

Chapter 15 BUILD MACHINES 413

Native and Cross-Compilation 414

Native Compilation 414

Cross-Compilation 415

Hybrid Environments 416

Centralized Development Environments 416

Why Build Machines Differ 418

Managing Multiple Build Machines 421

Open-Source Development Environments 424

GNU Autoconf 428

The High-Level Workflow 428

An Autoconf Example 430

Running autoheader and autoconf 434

Running the configure Script on the Build Machine 435

Using the Configuration Information 437

Summary 438

Chapter 16 TOOL MANAGEMENT 441

Rules for Managing Tools 442

Tool Rule #1: Take Notes 442

Tool Rule #2: Use Version Control for the Source Code 443

Tool Rule #3: Periodically Upgrade Tools 444

Tool Rule #4: Use Version Control for the Tool Binaries 445

Breaking the Rules 448

Writing Your Own Compilation Tools 449

Custom-Written Tools with Lex and Yacc 450

Summary 453

PART IV SCALING UP 455

Chapter 17 REDUCING COMPLEXITY FOR END USERS 457

Build Frameworks 458

Developer-Facing Portion of the Build Description 459

Framework Portion of the Build Description 460

Convention over Configuration 461

Maven: An Example Build Tool 462

Reasons to Avoid Supporting Multiple Variants 463

You'll Have More Variants to Test 463

Source Code Becomes Messy 465

Build Times Can Increase 465

Higher Disk Space Requirements 466

Various Ways to Reduce Complexity 466

Use a Modern Build Tool 466

Automatically Detect Dependencies 467

Keep Generated Files out of the Source Tree 467

Ensure That Cleaning a Build Tree Works Correctly 468

Abort the Build After the First Error 468

Provide Meaningful Error Messages 470

Validate Input Parameters 470

Don't Overengineer Build Scripts 471

Avoid Using Cryptic Language Features 471

Don't Use Environment Variables to Control the Build Process 472

Ensure That Release and Debug Builds Are Similar 473

Display the Exact Command Being Executed 474

Version-Control References to Tools 475

Version-Control the Build Instructions 475

Automatically Detect Changes in Compilation Flags 475

Don't Invoke the Version-Control Tool from the Build System 476

Use Continuous Integration as Often as Possible 476

Standardize on a Single Type of Build Machine 477

Standardize on a Single Compiler 477

Avoid Littering Code with #ifdefs 477

Use Meaningful Symbol Names 478

Remove Stale Code 478

Don't Duplicate Source Files 479

Use a Consistent Build System 480

Scheduling and Staffing Build System Changes 480

Summary 482

Chapter 18 MANAGING BUILD SIZE 485

The Problem with Monolithic Builds 486

Component-Based Software 488

Advantages of Using Components 491

What Exactly Is a Component? 493

Integrating Components into a Single Product 498

People and Process Management 502

Development Team Structure 503

Component Line-Up Management 505

Managing the Component Cache 507

Coordinating New Software Features 509

Apache Ivy 512

Chapter 19 FASTER BUILDS 515

Measuring Build System Performance 516

Measuring Performance in the Start-Up Phase 516

Measuring Performance in the Compilation Phase 526

Performance-Measurement Tools 531

Fixing the Problem: Improving Performance 534

Build Avoidance: Eliminating Unnecessary Rebuilds 535

Object File Caching 536

Smart Dependencies 539

Other Build-Avoidance Techniques 544

Parallelism 545

Build Clusters/Clouds 546

Parallel Build Tools 546

Limitations of Scalability 547

Reducing Disk Usage 548

Summary 551

REFERENCES 553

INDEX 559