Book Notes: “The Phoenix project: A Novel about IT, DevOps, and helping Your Business Win”

From so many days, I have not posted anything since I was almost like a lost but now wait is over. Over the last weekend and this weekend, I managed to read through a really interesting book and that is “The Phoenix project: A Novel about IT, DevOps, and helping Your Business Win” by Gene Kim,Kevin Behr and George spafford.

The book almost falls in line with the way Eliyahu Goldratt’s “The Goal” but the twist here is it is applied to IT industry. The book is written with fast paced plot of a sinking organization which almost has chaotic IT and business relationships and how it overcome the obstacles and continually improves with the help of laid out principles (Three ways) in the book.

I have read “The Goal” way before I was able to digest the material/principles laid out and always struggled to correlate it with IT industry but ‘Phoenix Project’ does excellent job here. For my future reference & time saving purpose, I had put down below notes which are taken from the book.

• When there are Chaos. Start prioritizing and estimating work. While doing so, you cannot stay away from fighting fires.
• Knowing is better than not knowing things.
• Stay Focused at wider goals.
• WIP is a silent killer. Therefore, one of the most critical mechanisms in the management of any plant is job and materials release. Without it , you can’t control WIP. WIP= work in progress.
• Apparently event undisciplined mobs can get lucky too.
• Read Air Traffic Control Books, Highlight on ubiquitous terminology used between air traffic controller and pilots of aircrafts. Signify accidents of plane crashes that happens. – Useful for DDD.
• Whiteboards, paper, physical movements and physical presence *engages * people and increases their *involvement* in their projects thus increasing success rate of projects.
• As a consultant, my goal is to observe and seek to understand.
• Project is getting delayed, all tasks by Dev team are marked as completed and QAs are still finding twice as many broken features/defects as are getting fixed. – Classic situation of badly run project.
• Processes are supposed to protect people from Distractions and help them deliver their core objectives.
• “I think” this might have happened or the I think the bugs are because of such and such thing. Such statements are sure signs of the problems that goes un noticed. It shows that we are flying without compass (data) and map (direction).
• Interesting term J ! FUBAR = Fucked up beyond all recognition.
• Management gut check from my team.
• There are four types of work in IT Operations: 1. Business Projects 2. Internal IT projects 3. Changes and maintenance work 4. Unplanned work
• Prioritization will help till one point but we need to identify what our constraint (real bottleneck in entire flow of operations) is and guard it from unscheduled work as well as keep it busy on top priority work.
• Focus on Work centers. A work center is made up of a man, machine, methods and measures.
• After chaos and constraints are figured out, work on single most important item/project which is required for survival.
• Once you get some success around, plan for all projects which does not involve constraints/ constrained work centers.
• Improving daily work is even more important than doing daily work.
• Ensure that we’re continually putting tension into the system, so that we’re continually reinforcing habits and improving something. Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful. This is called as improvement kata.
• Repetition creates habits and habits enable mastery.
• Our goal is to maximize flow.
• You win when you protect the organization from putting the meaningless work into the IT system. You win even more when you can take meaningless work out of the IT system.
• Avoid scoping errors.
• Create work centers and lanes of work.
• Understand Upstream and Downstream processes?
• Color coding of cards:
  • Purple cards for changes supporting one of the top five business projects; otherwise, they are yellow.
  • The Green cards are for internal IT improvement projects. [Give 20% of cycle time to these]
  • Pink cards are blocked tasks that are needs to be reviewed twice a day.

Make sure that there is a right balance of purple and green cards in work

• Improving something anywhere not at the constraint is an illusion.
• How to prioritize projects?
  • Do they increase the flow of project work through IT organization?
  • Do they increase operational stability or decrease the time required to detect and recover from outages or security breaches?
  • Do they increase specified constraint’s capacity?
• Projects that decrease your organizations/major project’s throughput, swamp the most constrained resource in organization, decrease scalability, availability , survivability, sustainability, security, supportability should be prioritized on low or entirely discarded, if possible.
• Managing the IT operations production schedule is one of the job for IT Operations top management.
• Wait Time = % of resource busy / % of resource time idle
• Wait time depend upon resource utilization.I.e. if a resource is 90% busy then wait time is 90% / 10% = 9 units of time i.e. 9 hours.

• Create Constant feedback loops from IT operations back to development, designing quality into the product at the early stages.
• You might have deployed an amazing technology (virtualization/cloud), but because you haven’t changed the way you work, you haven’t actually diminished the limitation.
• The flow of work goes in one direction only: forward.
• Takt time=Cycle time needed in order to keep up with customer demand. If any operation in the flow of work takes longer than the takt time, you will not be able to keep up with customer demand. So in IT, if your deployment time or environment setup time is greater than cycle time you will have a problem.
• DevOps is more and more important and their unified goal is to serve business goals. So, instead of fighting with each other, they need to be more collaborative.
• Read book: Continuous Delivery by Jez Humble and Dave Farley.
• Business agility is not about just raw speed. It’s about how good you are at detecting and responding changes in the market and being able to take larger and more calculated risks. It’s about continual experimentation.
• Read about Scott cook’s experiments in Intuit.
• The way to beat competition is out-experiment them.
• Features are always a gamble. Only ten percent will get the desired benefits. So the faster you can get those features to market and test them. Incidentally, you also pay back the business faster for the use of capital, which means the business starts making money faster.
• For above reason, you need to target ten or more deploys per day in production environment.
• Value Stream Mapping is quite useful tool for discovering activities that are adding value and those which are waste.
• BIGGEST LEARNING FOR ME: DESIGN YOUR SYSTEMS FOR IT OPERATIONS!! Build as many possible feature knobs and controls with which we can switch on and switch off the features. Learn Dark launches, canary releases as soon as possible.
• To routinely improve things, inject large faults in the system. It’s been followed in Apple mac OS and Netflix as well. These projects are called as “Simian Army Chaos Monkey”. Read more on these experiments and improvements. This creates culture that reinforces the value of taking risks and learning from failures and the need for repetition and practice to create mastery.
• IT is not merely a department, it is pervasive like electricity.
• In order to survive, the business and IT can’t make decisions exclusive of each other.

This has been a good read in so many months probably years. I look forward to read further through number of books such as Toyota Production System, DevOps cookbook, all lean literature and practice Improvement kata’s.