During 111 Agile Warsaw meetup Mary Poppendieck had a talk on on Impossible Problems. Impossible Problems sound cool and challenging, so I’ve decided to use the title to share my thoughts after listening to the talk. Full video relation recorded despite initial problems is available elsewhere.
Impossible problems often result form scale and innovation, fortunately they can be solved by scale and innovation. Taking IT infrastructure as an example: Each component has a certain probability of failure, growing infrastructure size (more components) makes failure inevitable. We can try to mitigate it by making perfect components. Unfortunately perfection costs and in reality nothing is perfect. So it is much more practical to embrace imperfection of individual components and introduce redundancy. Problem resulting form scale is then solved by scaling up the environment, thus creating space for safe failure. Big system resiliency is about dealing with failure not eliminating it. Attempts to eliminate all failures may result in building huge system risk, which one day will outbreak and result in catastrophic breakdown. The rule is valid outside IT too. Forest fires are good examples here: Small ones are natural, help to clean dry undergrowth and build security for ecosystem. Eliminating small fires results in entire forest prone to rapid spread of fire with catastrophic results no one can contain. Dealing with failure not avoiding is at all cost is the key to success. Failure is inherent to innovation and progress, one has to embrace it and use as learning opportunity it in order to score the final win. Societies with higher tolerance for failing innovate more and have better prosperity. Since Europe was once innovation area, failure tolerance must have been higher. Nowadays a fear to loose hides behind strive to perfection. Nothing ventured nothing gained.
Knowledge sharing helps
In 2003 Google published a paper titled The Google File System, it described file system used for large distributed data intensive applications. The system used commodity components to provide high performance and fault tolerance, actually it was used to support Google core business. The company decided to share concept instead of making it a trade secret. Perhaps they believed sharing knowledge would not hurt them because they were already a winner of search engine wars well protected by scale of operations. Sharing theory supporting their operations helped Google to build research scale e.g. academia running without company funding. Results may help to solve future problems with Google infrastructure – scale of research solving infrastructure scalability problems. In 2004 Google published MapReduce paper, some people believe it initiated current Big Data frenzy. Again Google shared some knowledge on how the process huge amount of data to produce search engine results. This time sharing created tremendous business potential: Google is believed to save all search results, perhaps has the biggest set of data that can be processed to extract whatever information you desire. Google can use Big Data developments to process own data and sell results. Another company following knowledge sharing concept is Amazon. in 2007 they have presented Dynamo concept essential for NoSQL development. Amazon never shared their implementation (Amazon DynamoDB) of Dynamo, but still gets a a justified credit for presenting the idea and benefits from its evolution.
Excessive communication hinders
Amazon Jeff Bezos is credited for developing a two pizzas team concept: If you can’t feed a team with two pizzas, it’s too large. Depending on pizza size and appetite the perfect team would be 5 to 8 people. Rationale behind this requirement is simple: Number of communication channels in n size team is n*(n-1)/2:
Team size Communication channels 5 10 6 15 7 21 8 28 10 45
Team is a group of people with common goal, mutual trust and mutually agreed way to reach the goal. Growing number of communication channels will block team development. Excessive communication is a team killer, so team goal has to match practical team size. Small teams were not invented by Amazon, they are common in military. In ancient Roman legion the smallest military unit was contubernium – a team of 8 soldiers. The group shared a tent, was responsible for logistics and was fighting together. They were rewarded or punished together as a unit. In modern armies we have 8-12 people squads subdivided into 4 people fire teams. Each fire team is capable of autonomous operations as part of a larger unit and has a leader. US Army studies show team motivation to fight results from desire to avoid failing team buddies rater than supporting other abstract ideas (like freedom and democracy). Military two pizzas team is a bit smaller than IT because they do more physical exercises so need to eat more… Basic concept is the same: Create a small self sufficient team, who can successfully execute assigned task. Team leader provides expertise, serves as a link to other teams and is accountable to organization. Inside the team leader authority comes from personal quality not power given by organization. Two pizzas team leader is not a pure management overhead, he is both leader and team resource. Some companies introduce span of control (SOC) defined as employees to managers ratio. I remember from Hewlett-Packard SOC targets growing from 8 to 15 and then to +25. This was not helping organization. Team size is determined by human ability to handle communication and organization structure must be aware of it.
Smaller responds faster
At Amazon team scaling approach hand an impact on service architecture. Traditional architecture was build in layers like:
- Business Process
Amazon found a singe central database not practical for their rapidly growing business. They have decided to implement a federation of services with a (two pizzas) team responsible for each micro service. Service teams in order to be self sufficient need infrastructure. Chris Pinkham responsible for Amazon Infrastructure proposed Amazon Elastic Compute Cloud (EC2) to satisfy the need. It eliminated infrastructure constraints and soon become a world standard for cloud computing. Horizontal scaling of services helps to accommodate changing needs faster and at lower cost. Services can be developed independently by small autonomous teams. Organization decides what the service shall do and team decides how.
Invention and innovation
By definition Innovation is a process of translating an idea or invention into a good or service that creates value or for which customers will pay. The process is not obvious, Xerox PARC founded in 1970 invented many modern computing items, but only some of them Xerox turned into innovations, laser printer was the most notable one. Other inventions found innovators outside Xerox, with little benefit for company funding their inception. Those who are willing to innovate must focus on people, look for distorted demand or constrained supply, chase impossible problems and be prepared for many failures while trying to solve them. Fortunately innovation is optional, so is survival.