1 Introduction

15

1.1 Culture

16

1.2 Technology

17

1.3 My Path to DevOps and to This Book

17

1.4 Target Audience

19

1.5 The Structure of the Book

20

1.6 Feedback

20

1.7 Thank You!

21

2 What Is DevOps?

23

2.1 DevOps: The Big Picture

23

2.1.1 CALMS

25

2.1.2 The Three Ways

28

2.1.3 Conclusion on the Three Ways and the CALMS Model

35

2.2 Misunderstandings about DevOps

36

2.2.1 Too Strong a Focus on Automation

36

2.2.2 With DevOps, but without Tests!

37

2.2.3 Incorrect Understanding of Team Structuring

38

2.2.4 Not Tearing Down All the Walls

39

2.2.5 Tools over Processes over People

40

2.2.6 One to One Copies of Working Methods from Other Companies

41

2.3 The DevOps Software Development Lifecycle

42

2.4 Summary

45

3 The Example Company

47

3.1 nicely-dressed.com

48

3.2 The Development Model

48

3.3 The Business Team: Requirements Analysis

49

3.4 The Architecture Team: Design of the Application

50

3.5 The Development Teams

50

3.5.1 The Development Process

51

3.5.2 Integrations with Obstacles

51

3.6 The Quality Assurance Team

52

3.7 The Operations Team: The Ops in DevOps

53

3.7.1 Manual Build of the Project

55

3.7.2 Deployment with Obstacles

55

3.7.3 Alarm from the Monitoring System

56

3.8 The Infrastructure Team

56

3.9 The Security Team

57

3.10 Summary

58

4 Project Management and Planning

61

4.1 The First Step: The Agile Mindset

61

4.2 Project Management for Everyone?

64

4.2.1 Jira Can Do (Almost) Everything

65

4.2.2 People over Processes

66

4.2.3 Good Project Management outside of Jira

67

4.2.4 More than Just a Project Management Tool

69

4.2.5 Project Management at nicely-dressed.com

71

4.3 Summary

72

5 Collaboration when Coding

73

5.1 Typical Problems with Managing the Source Code

73

5.1.1 Organization of the Code

74

5.1.2 Isolation for Supposed Security Reasons

75

5.1.3 Long Development Times Hindering Quick Security Fixes

76

5.1.4 Development Workflow without a Proper Structure

77

5.1.5 Big Bang Integrations

77

5.1.6 Code Reviews Could Help

78

5.1.7 Technical Debt

79

5.1.8 High Learning Curve due to Lack of Documentation

80

5.2 Improve the Organization of the Code

81

5.3 There Is No Way around Git

83

5.3.1 Git Solutions at a Glance

83

5.3.2 Development Workflows with Git

84

5.3.3 Source Code Management at nicely-dressed.com

90

5.4 Code Reviews and Pair Programming

91

5.4.1 Code Reviews

91

5.4.2 Simplify Code Reviews

97

5.4.3 Pair Programming

99

5.5 Inner Sourcing: Sharing Code within the Company

102

5.5.1 Open Source

103

5.5.2 The Path to Inner Sourcing

104

5.5.3 Advantages of Inner Sourcing

106

5.5.4 Monorepositories

108

5.6 Summary

111

6 Continuous Integration and the Build Process

113

6.1 Typical Problems in the Build Process

113

6.1.1 Onboarding with Stumbling Blocks

114

6.1.2 Build Difficulties Due to Infrequent Integrations

116

6.1.3 Only a Few Tests

117

6.1.4 A Build Server behind Closed Doors

118

6.2 Modern Build Management

119

6.3 Continuous Integration

122

6.4 The Continuous Integration Server and the Pipelines

126

6.4.1 The Basic Structure of a Pipeline

126

6.4.2 Scaling and Reproducibility

128

6.4.3 Declarative Pipelines versus Scripted Pipelines

130

6.5 Efficient Pipeline Authoring

132

6.5.1 Avoid Central Pipelines

132

6.5.2 Provide Pipeline Building Blocks

133

6.5.3 Create Visibility

134

6.6 Overview of Continuous Integration Servers

134

6.6.1 Jenkins

136

6.6.2 GitLab CI/CD

142

6.6.3 GitHub Actions

146

6.6.4 Other Continuous Integration Servers and Tools

150

6.6.5 Continuous Integration at nicely-dressed.com

151

6.7 Summary

152

7 Quality Assurance

153

7.1 Typical Problems with Testing

153

7.1.1 Teams in Their Silos

154

7.1.2 Different Understandings of Requirements

155

7.1.3 The Number of Bugs as a Metric

156

7.1.4 Is It Fixed Yet?

157

7.2 Testing as Part of the DevOps Process

158

7.2.1 Tests in the Build Pipeline

160

7.2.2 Different Tests for Different Tasks

162

7.2.3 Automate Tests

164

7.2.4 Test-Driven Development

169

7.3 Summary

171

8 Continuous Delivery and Deployment

173

8.1 Typical Release Management Problems

173

8.1.1 Separate Handling of Changes and Documentation

174

8.1.2 Lengthy Release Process

175

8.1.3 Automations That Are Not Worthwhile

176

8.1.4 Hostilities between the Teams

176

8.1.5 Deployment on Production Systems with Obstacles

177

8.1.6 Conclusion

178

8.2 Implementing Continuous Delivery and Deployment

179

8.2.1 Bringing Development and Operations Together

179

8.2.2 QA, Staging, and Production Environments

185

8.2.3 Deployment on Fridays

190

8.3 Build Management for Deployments

191

8.3.1 The Question of Version Numbers

192

8.3.2 Packaging

193

8.3.3 Containerization

194

8.3.4 Container Registry and Package Registry

196

8.4 Rollbacks, Canaries, and Feature Flags

198

8.4.1 Rollbacks

199

8.4.2 Step-by-Step Activation Using Blue-Green and Canary Deployments

200

8.4.3 Feature Flags

203

8.5 Deployment Targets

205

8.5.1 Orchestrating Deployments with Kubernetes

205

8.5.2 Orchestrating Deployments at nicely-dressed.com

219

8.6 Summary

219

9 Operating the Service

221

9.1 Typical Problems with Operating Services

221

9.1.1 Lengthy Infrastructure Planning

222

9.1.2 Hardware Exchange with Obstacles

222

9.1.3 Unfavorable Server Utilization

223

9.1.4 Common Outages during the Night

224

9.2 Breaking Up the Highly Coupled Infrastructure Architecture

226

9.2.1 Cattle, not Pets

226

9.2.2 Abstracting the Infrastructure

228

9.2.3 Containers for Faster Deployments

231

9.3 Cloud Computing

233

9.3.1 What Is the Cloud?

233

9.3.2 Cloud Models

234

9.3.3 Service Models

235

9.3.4 Cloud Native

240

9.3.5 The Cloud at nicely-dressed.com

242

9.4 Stronger Collaboration between Development and Operations

243

9.4.1 Everyone Should Be Ready

243

9.4.2 Blameless Post-Mortems

245

9.4.3 Communication Solutions and ChatOps

247

9.5 Configuration Management: Everything as Code

249

9.5.1 Infrastructure as Code with Terraform

250

9.5.2 Ansible versus Puppet

253

9.6 Chaos Engineering

258

9.6.1 Making Systems Fail

258

9.6.2 Chaos Engineering without Chaos, but with a Plan

260

9.7 Reliability Engineering

262

9.7.1 Site Reliability Engineering

263

9.7.2 Database Reliability Engineering

264

9.8 Summary

265

10 From Monitoring to Observability

267

10.1 No Visibility at nicely-dressed.com

268

10.1.1 Service Outages Happen Every Day

268

10.1.2 Performance, Performance!

272

10.1.3 Logs

274

10.2 With Insight Comes Foresight

275

10.2.1 Observability Engineering

276

10.2.2 Insights into Processes with Tracing

278

10.2.3 A/B Tests

279

10.2.4 Business Monitoring

281

10.3 Tools for Monitoring, Observability, and Tracing

282

10.3.1 Monitor Systems with Icinga and Nagios

283

10.3.2 Monitoring with Metrics and Time Series Databases

284

10.3.3 Data Visualization with Grafana

292

10.3.4 Error Tracking

294

10.3.5 Distributed Tracing

295

10.3.6 Logging

296

10.3.7 Service Meshes

297

10.3.8 Observability Platforms

298

10.3.9 Monitoring and Observability at nicely-dressed.com

300

10.4 Availability

300

10.4.1 Service Level Agreements

301

10.4.2 Service Level Objectives

302

10.4.3 Service Level Indicators

303

10.4.4 Error Budgets

303

10.5 Summary

304

11 Security and Compliance

307

11.1 Security Disrupts the Agile Waterfall

308

11.2 DevOps with a Separate Security Team

310

11.2.1 To Deploy or Not to Deploy?

311

11.2.2 The Search for Undocumented Dependencies

312

11.2.3 Frustration and Blocking

313

11.3 DevSecOps: Building Security into DevOps

314

11.3.1 The DevSecOps Team Structure

315

11.3.2 Shift Left: Find Errors Earlier

316

11.3.3 Inner Sourcing Ensures Formal Security

317

11.3.4 Security as an Integral Part of the Development Process

318

11.3.5 Dealing with Mistakes

319

11.4 Tools for Higher Security

321

11.4.1 Dashboards and Reporting

321

11.4.2 Pull and Merge Requests

322

11.4.3 Security Scanners in Detail

322

11.5 Supply Chain Security

329

11.5.1 Attacks on the Supply Chain

330

11.5.2 Software Bill of Materials

332

11.5.3 Security of the Build and Deployment Server

332

11.5.4 Secure User Accounts

333

11.5.5 No Code Is Good Code

334

11.5.6 Security at nicely-dressed.com

335

11.6 Compliance

335

11.6.1 Define Compliance Guidelines

336

11.6.2 Manual Compliance

338

11.6.3 Fully Automated Compliance

339

11.6.4 Compliance at nicely-dressed.com

345

11.7 Summary

345

12 Successfully Implementing the DevOps Transformation

347

12.1 Introducing a DevOps Culture

347

12.1.1 Bottom-Up or Top-Down?

348

12.1.2 First Steps in the DevOps Transformation

350

12.2 Making DevOps Success Measurable with DORA Metrics

362

12.2.1 DORA Metric 1: Deployment Frequency

363

12.2.2 DORA Metric 2: Lead Time

364

12.2.3 DORA Metric 3: Change Failure Rate

365

12.2.4 DORA Metric 4: Time to Restore Service

366

12.2.5 DORA Metric 5: Operational Performance and Reliability

366

12.2.6 Findings from the State of DevOps Report

367

12.3 Value Stream Mapping

368

12.3.1 The Value Stream of a Pizzeria

369

12.3.2 The Value Stream at nicely-dressed.com

370

12.3.3 Implementation of Value Stream Mapping

371

12.4 Summary

375

13 DevOps Platforms

377

13.1 Toolchain Complexity

378

13.1.1 Step 0: Toolchains Grow Historically

378

13.1.2 Step 1: Multiple Toolchains to Increase Maintainability

379

13.1.3 Step 2: Standardized Tools, but Still with a Lot of Duct Tape

381

13.1.4 Step 3: DevOps Platforms

382

13.2 DevOps Platforms at a Glance

384

13.2.1 GitLab

384

13.2.2 GitHub

385

13.2.3 Azure DevOps

385

13.2.4 Atlassian

386

13.2.5 Other Platforms

386

13.3 Summary

387

14 Beyond Culture and Tools

389

14.1 The Role of AI in DevOps

389

14.1.1 Making Work Easier with AI-Supported Code Generation?

390

14.1.2 More Code Leads to Higher Review Requirements!

392

14.1.3 AI-Supporting Features

393

14.1.4 Data Protection and Privacy

394

14.1.5 It’s the Overall Concept That Counts!

395

14.2 DataOps, MLOps, and AIOps

397

14.2.1 DataOps

397

14.2.2 MLOps

398

14.2.3 AIOps

398

14.3 DevOps as a Job

399

14.3.1 The Question of DevOps Engineers

399

14.3.2 Soft Skills

401

14.3.3 The Technical DevOps Learning Path

402

14.4 Summary

410