2010/02/26

Working Overtime is a Killer of 24 Hours Based Agile

To be successful in any project we have to look at it as a whole, especially from the system point of view. According to the definition " A system consists of interdependent and interacting parts joined by a purpose. A system is not just the sum of its parts-it is the product of their interactions ."

It is very important for every project leader to try to move one step back when looking at the project, when trying to analyze and to solve its problems. I have found a beautiful way of modeling of those interactions together with the set of fantastic patterns (structures) of common problems named Systems Thinking.

I was able to observe a lot of Systems Thinking's patterns in my current project. A good example could be the well-known problem of keeping project in schedule. Even if in my current project we are extensively using agile, our project managers' first reaction to this problem was to extend the working time. What a bullshit. I'm not generally against longer working time but that should be used in case of an exception, not as a rule.

Nevertheless our project managers did it and this short term decision had (counterintuitive) a very negative influence on the project. Let me explain this using the Systems Thinking, and one of its patterns called "Fixes that fail" using Causal Loop Diagram (CLD):

As you can see due to the schedule gap we have run an action working overtime. In the short term it causes that our current estimated schedule had decreased thus reducing the schedule gap. That was exactly that, what the project managers expected and wanted to achieve. But what they have overseen is the second loop which shows that the action working overtime causes the fatigue of the developers, which causes less quality of the code being developed, thus the increase of the number of an open issues, thus increase of the current estimated schedule and finally the schedule gap. It was simply a "shot own leg" solution.

Secondly in a 24 development project as I have, this decision has killed the project rhythm based on so called Home-Handover-Development-Handover-Home (shortly 2HD) flow, which is very important for the time-boxed agile development approach, we have chosen (based on Scrum). Before extending the working time all developers were happy, because they have noticed that time-boxed development (8 hrs. per day) had the advantage of getting task faster done, due to the implicit pressure of the upcoming daily handover. They had simply no possibility to work longer on the task because it has been transferred to the other team, which had a very positive influence on the productivity and effectiveness. As you can see it was the second shot in the own legs.

The right solution to this problem is not to extend the working time, but to reduce the scope. The flexibility in changing scope is one of keys to the successful development projects. Based on many studies we can assume that over 50% of functionality of any software is not used or is used rarely, thus it builds a perfect candidate for reducing scope. I always admit that the flexible scope based on prioritized user stories should be a part of every software development contract (but it is a story for other discussion, I will discuss this in one of my next posts).

So, working overtime is the killer. Simply avoid it.

It is almost always cheaper to build less, but build it stable, rather than to build lots of stuff and then have to do panic hot-fixes (by Henrik Kniberg)

---

2010/02/15

Inventory vs. Partially Done Work vs. Unused Code

I like the idea from LSD to eliminate waste in the project. LSD defines waste as anything or process that does not add value to a product. There are seven main wastes defined by lean for manufacturing. LSD tries to map them to its own definition but not all of the mappings are in my opinion right. Here the short summary of wastes based on LSD and my opinion.

The Seven Wastes of Manufacturing The Seven Wastes of Software Development by LSD The Seven Wastes of Software Development by Boczek.com
Inventory Partially Done Work Unused Code
Extra Processing Extra Processes Extra Processes
Overproduction Extra Features Overdesign
Transportation Task Switching Context Switching
Waiting Waiting Poor Synchronization
Motion Motion Poor Communication
Defects Defects Defects

Management Activities Management Activities

So let't talk about first one: partially done work and unused code.

Partially Done Work Waste

LSD defines "partially done work"-waste as a counterpart to the "inventory"-waste in the manufacturing industry. I don't agree with this definition. In an agile project "partially done work" does not exist. Iterations are time-boxed and short enough that you have always a good ability to close or continue tasks without leaving something in an "open" or "partially done" state for eternity. But we have other type of "inventory"-waste, which is much more difficult to localize: "unused code"-waste.

Unused Code Waste

In our project I have decided to use DDD. To support this I have built a very, very light initial framework based on component object model principles that means with a lot of interfaces and base classes. An additional untypical constraint we had, was that we were not allowed to use any O/R mapper, thus the whole infrastructure layer (using DDD terminology) had to be written manually (every SQL statement for CRUDs and so on) from the scratch. With help of an O/R mapper you would get almost all CRUDs out of the box.

I am an architect and I love frameworks. I love these hours deliberating about what would be good to build at the beginning of the project to reuse this in the future, to make the life of a normal developer easier. Yeah... It so simple to forget "You aren't gonna need it" (or YAGNI for short). As you probably know YAGNI is the principle in extreme programming, that programmers should not add functionality until it is necessary.

My decision was pretty hard for the whole team: "We will follow YAGNI as close as possible! We will not produce any "unused code"". It was especially difficult for the developers to accept this, in case of building infrastructure layer. Our interfaces have defined CRUDs for every entity and it was very unusual even for me to provide (depending on the business flow) e.g. only an "U" without implementing other "CRD" (exactly speaking, we do have implemented them but with only one line of code throwing Not-Implemented-Exception). But we have done this consequently in the whole application.

And it was a great decision. As we know, agile supports role-less teams, so if someone got Not-Implemented-Exception it was pretty clear, he had to implement lacking functionality and did it without problems. The only small challenge we had was regarding to unit testing. Developers love to test "U" using at the beginning "C". We have solved it providing specialized set of test data, which is always a better solution (separation of functionality being tested) comparing to usage of CRUDs to provide test data dynamically.

So we haven't produced any unused code in our application. We haven't produced any waste! It rocks, hammer hard.

2010/02/14

Need for Project Leader

I had recently an interesting discussion with a very experienced project manager that is involved in my current project. He is a typical "old date" project manager with titles like PMP and so on, but this is his first agile project. Our project will be finished within few weeks so we have started to collect information based on "lessons learned" principle. As I'm involved in this project in many different roles that start from architect (responsible for architecture), project manager (responsible for planning and scheduling) to the lead developer and tester (thus almost in the first line trench) at the end, I had an extraordinary possibility to look at my project from very different views.

I have tried to explain him the differences between the role of project manager as he had in a classic (e.g. waterfall driven) projects and the role of project manager in an agile project. My main concern was regarding to the problem of non-existing detailed description of a role I've got in project, resulting in wrong positioning of myself in the organizational structure and some conflicting areas with the project management.

There are many differences between classical and agile project management but one of the main is in the way the work is distributed and assigned to the team members. In a classic project, a project manager defines, dispatches, and monitors the job of the developers. As many studies have shown (and my experience too) that the maximum productivity of this supervised way of working is about 60-70%. Agile projects in contrasts, with their self-organizing teams can achieve up to 95% of the productivity (at the moment, please don't ask how to measure that ;) which is a tremendous result in case of every IT project.

But let's go back to my colleague. His question was actually, what a project manager shall do in case of an agile project. If the agile team is responsible for all the stuff normally done by the project manager, like estimating, scheduling etc. what shall he do? I have found he had the problem with this sudden conclusion; an agile project doesn't need any classic project management. No Pert and Gantt. Ok, not so quickly.

But anyway, especially one of his (humorous) sentences has made me cold: "Let's remove this anxious democracy from the process (self-organizing, feedback etc.). Then we will get a normal project, won't we?" ;).

In my opinion there is still lack of understanding and acceptance for need of project leader role (which actually I have in my project), especially within project management community; even if in the manufacturing industry such role exists since many, many years already (with different names like system developer, chief engineer, solution architect etc.…).

Additionally it is important to state that there is no place for the pure software architects in the IT project world. As a project leader you will have to provide much more capabilities in different areas as you could think ;). Based on my experience and some queues I have read already I've defined the project leader's core task list as follows:

1. Gets, collects and distributes deep domain knowledge

2. Provides extensive experience in project management and technology

3. Creates and maintains design and architecture of the system

4. Creates and maintains FBS (feature brake down structure), estimates and schedules.

5. Acts as focal point of communication

6. Defends vision

7. Empowers, shields and synchronizes project teams

8. Arbitrates tradeoffs

9. Assures external and internal quality

10. Sees the Whole to achieve business success

As you can see this list contains some points that in the classic project management belong to the project managers.

Cool recapitulation of differences between project leaders and project managers provides LSD: project managers arecoped with complexity (plan and budget, organize and staff, track and control) but project leaders are coped with change (set direction, align people, enable motivation).

But as I have stated already what is really important is the positioning of the project leader within his organization and customer projects. It very critical that the project leaders have the same position and should benefit from the same esteem as project managers do. Project leaders should have an access to all information project managers get and be involved in all communication loops defined for the management within project. Only then a project leader can act effectively.

So back again to the question what is the role of a project manager in an agile project? Do we need them? Yes, we do.

The key task of a project manager in an agile project in my opinion is to eliminate waste from everything around and outside of the project core, means design-development-test-delivery area. Waste means anything or process that does not add value to a product.

The key task of the project leader in contrary is to remove waste from everything inside project core.

Only if project manager and project leader are playing together, an agile project can be closed successfully.

---

2010/02/07

And Agile for All, Second Shot

After pushing my post "And Agile for All", I immediately received a great feedback (thank you Joachim) saying "Watch out, that's not the nation!". I tried to understand where exactly the problem was, that the agile didn't suit to all cultures. Is it a problem of the culture (as I stated) or maybe a problem of agile, though? But agile is a process of developing software and as a well-defined method should work regardless to the context it is used in…

And I have found the answer here: "Agile development is more culture than process" by Jeff Patton. It was really great to be able to match my questions and concerns to the sentences in that post. I would recommend reading this article carefully to anyone fascinated by agile. It's great.

But, what is very valuable for me, is the Jeff's conclusion regarding to "so what?":

- Underscore agile values that motivate practice

- Identify organization values that compete with agile values

- Be sensitive to culture shock

So, now I think I will have to give them (other cultures) a chance for the second shot. And for myself too :).

---

2010/02/05

Well-Defined Domain Model is the Key to Success

One of the challenges in agile style of development is to delay making decisions that are crucial for the whole system. The key tool to achieve this, are iterations that allow us, based on the continuous communication, to collect feedback and to provide options as a reaction to the change. But there are some high-level decisions that you have to make at the beginning of the development. "High-level" means in this context, those decisions that would cause a very high cost if they would have to be changed.

One of the tools I'm using to amplify feedback from the customer is the domain driven design (DDD). It allows us to model the domain of the customer using a domain specific language and that makes the customer happy. We are using the language of the customer, thus he can understand us clearly, our questions and concerns and provide appropriate and detailed feedback. It works perfectly.

The problem that we have is to provide a well-defined Domain Model at a very early stage of development. Without clearly defined domain objects and their relations you are not able to define actions on them. You will get very quickly stuck. The second problem that can occur, are mistakes in your model. I have faced that in my current project. Our Domain Model has defined one domain object that was too generic, thus actually it didn't directly exist in our domain. It's like defining a domain object animal in case of the horse racing domain. The domain specific language of the customer doesn't contain the word animal, it uses the word horse instead and all actions of the domain are defined relating to domain objects like schedule horse race (and not schedule animal race). As a result of that mistake, a lot of business logic has hiked to the application/service layer, causing violation of separation of concerns principle.

The key concept to provide a well-defined domain model is to understand as much as possible from the domain. Invest as much time as you can to learn the domain using all possible instruments (interviews, brainstorming etc.). You can try to delay the decision regarding to your domain model, providing options, which means creating more than one domain model. But at the end you have to make the decision (if you don't do this, the decision will be made for you, which is the worst case). And please do not forget that it is very difficult to change or refactor the model later. It is also very difficult to build the flexibility into it. Especially try to avoid generalizing your domain objects to achieve that. They have to stay specific for the domain, otherwise you will lose the domain model purpose: to model the real domain.

The domain model is like a skeleton of your system. The whole body of the system "hangs" on it. And it is very painful if you have to break it. A well-defined domain model is the key to success.

---

2010/02/03

Don't Distribute Your Teams, Just Follow the Stream

Agile development is about learning. And to learn you need the information. So in agile everything is focusing on gathering more information, because only if you get it you can react and adjust. So we try to amplify communication between the customer, management and team (using iterations etc.), within team internally (using daily meetings, retrospectives etc.) and even between product and team (using continuous integration, testing etc.). The main task is to gather as much knowledge as possible.

Distributed teams add another level of complexity to this communication process, thus they are a natural enemy of agile. One of the principles of agile is "to get them together", means that the project team is "living" and working together using one separated project space. In case of distributed teams it is very difficult to achieve. You can in fact try to use modern communication technologies to make the team working "as close" as possible. You can even schedule more meetings or social events. But it is not same comparing to the situation in you can see your colleagues in front of you and communicate with them as oft as you need it. We are just humans.

The problem is even more challenging in case of the feature (domain) based development (I'm explicitly avoiding the word "driven" here) with the collective code ownership. In this situation developers from different teams are working closely on the same piece of software so communication and feedback are crucial topics for the successful synchronization of their work (this is a subject I found very fascinating - how to execute project planning and synchronize work of teams with collective code ownership - surely I will provide a separate blog post regarding only to that stuff, stay tuned).

But the biggest problem occurs when you have teams spread across different countries, cultures and time zones. This situation comes into game because of the frequently returning management idea of developing software using "Follow the Sun" or "Round-the-clock Programming" models in three or more locations around the globe. 24 hours a day, non-stop. To be faster to market, to be cheaper.

I have now an opportunity to work in such global project and I have to say that the quality of communication is one of the main issues that we have. And it is almost killing one. Many very important aspects of agile simply do not work. Due to the distance and time zones we are not able to "get the team together". The verbal communication has to be exchanged by written one. It works but is not agile anymore, because we are losing the possibility to get immediate feedback so we are not able to amplify our learning. Our distributed teams are living in the different and separated worlds so the possibility of conflicts on different areas (technical, personal etc.) is much bigger and the way to solve them much harder. I have to say I'm not a fan of distributed teams.

But I like the FTS model, though and all around it. So, what to do?

I would say, let's change slightly the name from "Follow the Sun" into "Follow the Stream"!

The idea is to eliminate waste from the FTS model, and in my opinion, those communication obstacles are simply wastes! Let's get rid of the time zones, culture differences etc. We have to keep and concentrate on the core of FTS though, exactly: on the stream of work. So my suggestion is to work using 24 hours model but only at one location. Simple. Yes, we have to use shift working to achieve this goal but I'm pretty sure we will create much less waste comparing to the distributed teams.

Martin Fowler has already specified it in his First Law of Distributed Object Design: Don't distribute your objects. My First Law of Distributed Teams states similar: Don't distribute your teams, just follow the stream!

---

2010/02/01

And Agile for All

That is one of the most interesting thoughts I have ever had regarding to the agile way of developing software. As I am a big fan of agile, I'm always saying simply: And Agile for All… But no, not for All.

The history of agile is settled very deeply in the culture, thus the country and people that have created it. It was developed by us, us from Europe, US and all countries around us (with all respect to them). It was developed with all principles that are important for this way of thinking, living and finally in doing projects. Democracy, freedom, revolution, experimentation and riot also. That are all good and bad things, that allow us do think independently, allow us to be creative, allow us to be agile.

I have now an opportunity to work on an agile project in far-far East, exactly in Japan. The far-far East is simply not agile. And there is nothing wrong or bad about it. It is just the history of that nation that is different comparing to ours.

I'm currently in Tokyo and it is snowing. It is an untypical weather for this small dot on the world map. But anyway the weather forecast has spoken about bad weather coming for about one week ago. They were even able to say that the heaviest snowfall will come at 3am at the morning next day. Yes, 3pm., exactly. Can you imagine such exact forecast in Europe? I don't think so. Ok, it is nothing about high end forecast "made in Japan", it is just simply an island here and they are able to observe the neighborhood and the ocean to provide such a perfect forecast. It is simply not agile, it is simply a snowfall, hmm… like waterfall, you know what I mean.

Additionally this nation is educated without agility in its life. Children go to school wearing strictly defined clothes and have to follow strictly defined rules of the school life. There is no graffiti on the walls; there are no broken things around, any trash on the streets. Everything is "almost perfect" and boring. It is uncommon to give a tip in a restaurant, so you are not able to judge the service. There is no contact between individuals in the metro because even if he or she would be worth of it, they are sleeping or playing around with all those modern electronic stuff you can buy here everywhere (mobile phones, Sony Portable, Nintendo DSi etc.). Even when eating, at the table, they get a dish set with predefined order of plates on it. Rice bowl to the left, the miso soup bowl to the right (or backwards, I don't know exactly). Try to change the order, it will be funny.

There is no place for agility here. No place for abstraction, revolution or simply own initiative. And consequently, in an agile project try to push something through, like self-organizing teams, real feedback and so on…

And Agile for All. Here? Forget it.

---