Question

what are the obstacles that might inhibit the collection and evaluation of Agile process metrics and production analysis matrices?

Answer 1

Agile projects come with a set of challenges and problems that are different from those faced by projects following a traditional methodology. This paper covers a selection of considerations for addressing the challenges, failures, and problems that occur in agile projects. Based on an Internet search, just under 50 challenges were identified in introducing agile methodologies into an organization or working with agile projects. The problems include issues with (1) communicating; (2) managing day-to-day operational problems; (3) gaining buy-in from management, customers, and team members; (4) changing culture and mindset; and (5) gaining experience and making it work. Of course, some of the issues and challenges are unique and occur due to differences and idiosyncrasies in the organization or the project. Nevertheless, this paper covers 10 actions that can be used to avoid these common issues or problems or to mitigate their impact. This paper includes extracts from the book “Going Agile Project Management Practices,” Internet research, and reference materials from other industry experts.

State of Agile

Agile project management methodologies for software development have been around since the 1990s. There are thousands of organizations using them and hundreds of thousands of trained agile coaches. The Agile Manifesto (The Agile Alliance), published in 2001, represented a seminal point at which the software community acknowledged that requirements evolve, and cannot be fully pre-defined. Today, there are several software development methodologies, frameworks, and processes that embody the Agile Manifesto's values and principles [for example, Scrum, Lean, Kanban, Feature Driven Development (FDD), Extreme Programming (XP), Crystal, and Dynamic Systems Development Methodology (DSDM)].

According to surveys from VersionOne (2013), Scott Ambler (2012), Microsoft (Begel & Nagappan, 2006), and Dan Rico (2008), projects using agile methodologies deliver sooner, are more flexible for change, and produce higher-quality software. Teams and stakeholders perceive greater satisfaction due to improved communications, smoother collaborations, and increased flexibility. Furthermore, agile projects deliver business results faster than transitional methodologies and deliver high benefit-to-costs ratios. In Dan Rico's report, traditional methodologies delivered higher on the benefits-to-cost ratio than agile. Thus, there is no definitive proof that the return on investment from agile projects is higher than that from traditional projects. However, on average agile projects report achieving the expected business benefit more often than traditional projects.

Agile Failures, Challenges and Issues

The Scott Ambler survey defines success as a solution being delivered and meeting its success criteria within the acceptable range defined by the organization, and failure as the project never delivering a solution. Specifically, Turner and Zolin make similar observations on success in their paper Forecasting Success on Large Projects: Developing Reliable Scale to Predict Multiple Perspectives by Multiple Stakeholders Over Multiple Time Frames. They state that “project success is measured not just by completion of the scope of work to time, cost, and quality, but also by performance of the projects outputs, outcomes, and impacts” (Turner & Zolin, 2012, p. 87). For our discussion on failure, we shorten the definition to “failure is never delivering a solution.”

The Ambler report concluded that agile projects do not fail more than other projects. They succeed at the same level as other iterative methodologies. However, agile projects face a set of challenges and problems related to applying a different approach to project management. According to VersionOne, the top three reasons for agile project failure are:

• Inadequate experience with agile methods
• Little understanding of the required broader organizational change
• Company philosophy or culture at odds with agile values

Table 1 provides an overview of an Internet and document search on the problems, challenges, and issues with agile project management approaches. These challenges can be grouped into four areas and contain just under 50 situations faced introducing or using agile methodologies. According to the VersionOne survey, organizations that have gone agile see some of these challenges as barriers to the further agile adoption.

Communicating, changing culture and mindset	Buy-in from management, customers, and team members	Day-to-day operational problems	Experience Making it Work
Communication problems Too narrow communication bandwidth Communication between development and product owner Communication between development and quality assurance Company culture Lack of culture transition Organization change Mindset	Management buy-in Only developers engaged Big visible charts are intrusive Insufficient training Team does not like showing work Unwillingness of team Customer not signing off Customer fear Customer won't commit to agile External pressure Getting customer into loop Getting stakeholders to agree Departmental fragmentation Lack of management awareness No single product owner authority Management not changing behavior	Back loading documentation Back loading testing Decreased visibility of project progress Effort estimations Good requirements development Integration with other systems Interruptions Lack of decisions on architecture Lack of time to fix failed tests Large projects Project complexity Too big backlog Too old backlog Too many meetings Too many open issues Too many unplanned tasks Getting test automated Regression testing	Budgeting Giving-up too early Inexperienced Product Owners Inexperienced ScrumMasters New to agile Process not understood Reinvent agile methodology Scaling to large projects

Table 1 ? Agile challenges and problems

Avoiding failure

There are several books, including Going Agile Project Management Practices, which describe methodologies, processes, examples, and recommended actions for going agile in a way that should promote success. Nevertheless, the issues, challenges, and problems are usually unique and occur due to differences and idiosyncrasies in the organization, the people, the execution of the practices, or other factors. For this reason, there is no one cookbook or guide that can eliminate all the issues that may be faced in agile projects. The following are some of the actions that should be considered to avoid, eliminate, or mitigate failure or issues in agile projects.

Selecting and Customizing the Right Methodology

The Agile Manifesto (www.agilemanifesto.org ) does not prescribe any specific methodology; it provides a set of values and principles on which agile methodologies are based. The structure, support, advantages, and disadvantages of the different agile methodologies need to be weighed against one another to select the best option given the characteristics of the organization, customer, and project. After gaining experience with a methodology through its use, it should be customized to fit the organization and project situation.

Problems with the cultural transition, management buy-in and only engaging the developers can be avoided or at least mitigated by selecting and customizing a methodology that fits the company and the project.

Selecting a Methodology

For organizations and projects, where experience can be used to plan a course of action with a good degree of certainty for a positive outcome, a traditional methodology may be more appropriate than an agile methodology. In this case, the plans can be developed up-front and then designed, developed, and tested without much variance.

Agile methodologies are effective when the product details cannot be defined or agreed in advance with any degree of accuracy. This situation calls for the collaborative environment between the user and the developer. Agile methodologies are suited for a dynamic and changing environment.

Each agile method defines its own processes or techniques for realizing the core principles for agile methods. Scrum, Feature Driven Development, Dynamic Systems Development Methodology (DSDM), Extreme Programming and Crystal advocate iterative development and incremental release of software development. Lean and Kanban are continuous processes. According to the VersionOne survey, Scrum (or a Scrum Hybrid) is the most popular agile methodology. It is used by two-thirds of survey respondents. It is a lightweight methodology with a few practices, clearly defined roles, and easy practices to implement. Furthermore, there is widespread availability of Scrum professional training and certification.

While the popularity and availability of the methodology is important, organizational acceptance and project success is based on additional factors such as the project characteristics, organizational culture, and customer availability. Table 2 gives an overview of selected criteria and the suitability of agile and traditional methodologies.

Criteria	Description	Agile Fit	Traditional Fit
Skill and experience of the team.	The degree of experience and skill the team has working with the technology and the subject matter. The fewer experienced people, the more the senior people will have to mentor and coach the junior people. The team will be less productive than if it had all experienced people.	Highly experience	Low experience
Stability of requirements.	Degree to which the requirements are likely to change in the two to four week period. Projects that require frequent feedback cycles, e.g., innovative projects. Projects where the customer only knows what they want once they see it.	Low stability	High stability
Customer availability.	The availability and frequency of interactions with the customer. .The more the customer can be involved in the day-to-day project activities, the better the communication, and the less the need for intermediate documents	High availability	Low availability
Company culture.	The degree to which the company culture can tolerate and adjust to delegating decisions-making, allowing negotiations on delivered features, agreeing to a certain amount of uncertainty on exactly what will be delivered.	Low management control	High management control

Table 2 ? Organization and project characteristics

An agile methodology should be selected based on its suitability for the organization and its projects. Cockburn, Boehm, and Turner (2001) suggest that agile methodologies have to be modified to fit the available team and the company culture. However, much caution should be given to adapting the methodology practices before gaining experience. In fact, Alistair Cockburn suggests, it is better to select a methodology, stick as close to the formal definition as possible, and after gaining experience adapt it (2001).

Use a traditional methodology with agile practices

If one of the agile methodologies does not fit, then an option is to add agile practices to a traditional methodology to reach the desired level of agility. The agile team members should decide the practices that they will use based upon the team size, geographical distribution of team members, and the need of the project.

Use hybrid agile approaches

The type of task board used to track the progress of features and tasks as well as the method for coordinating daily activities can be adjusted to address challenges with too many meetings when applying agile practices.

The project teams should optimize the task board to suit the way that they need to coordinate tasks and people and consider the best method for reducing or limiting the coordination costs for activities to schedule, coordinate, or assign people to tasks. For example, using a Kanban board, a co-located team may be able to review the board in stand-up meetings three times a week instead of having daily stand-ups. In such a scenario, a team with seven people would reduce its coordination costs by 40%. That translates into fewer meetings and more time for value-add activities.

Thus, when applicable the project teams should combine methodology practices.

Use Different Problem-Solving Techniques

According to the Agile Manifesto (2001), project managers should “deliver working software frequently, from a couple of weeks to a couple of months, with a preference for the shorter timescale.” However, in practice, the iteration duration should be established considering how often the team requires feedback, how long it takes to deliver a reasonable scope, and how available the stakeholders are to attend the review sessions. If the duration is consistently too short to deliver working features, then change it. Alternatively, if the features are too big for the iteration length, break the features into smaller units.

There are wide selections of techniques and practices that can be used to identify and resolve problems that arise in the day-to-day operation of the agile projects. For example, a Boundary Authority Role Task (BART) can be used to identify problems with team dynamics and the pain snake can be used to determine the root cause of interruptions.

• A BART analysis evaluates four dimensions of team dynamics: boundary, authority, role, and task analysis. Agile coach Dan Mezick (Mezick, 2009) applied BART to Scrum.
• A pain snake is a technique to track issues using post-its on the wall. The concept was created by Kevin E. Schlabach (Schlabach, 2008). It was first used to identify what kinds of activities were interrupting a team's work. As an interruption occurred, the team member would write down what the interruption was and the lost time and post it on the wall at the end of the snake. The snake was monitored for repetitive patterns.

Educate and Align with the Management Team

Going agile requires executive, senior management, and middle management awareness and buy-in that something will change in the project management practices. They need to understand the benefits of the change as well as the details of how the change will affect operational aspects of the business. Furthermore, they need to understand what will be expected from them and what should change in their behavior.

Many cultural and communication problems can be avoided or at least mitigated by aligning with all levels of management before adopting an agile methodology.

The business case for going agile should be defined and agreed to by the management. The desire to change has to be demonstrated with a clear business justification for going through the actions and the cost associated with the transition. If management is not willing to sponsor a transformation to an agile methodology, then perhaps agile practices can be applied to a traditional methodology or a pilot project should be undertaken to gain management buy-in.

The following topics should be addressed to gain management alignment in advance of adopting an agile methodology.

• Their personal time is needed to gain awareness and understanding of agile practices and what they mean to the whole organization, including performance planning and monitoring.
• Their commitment is needed to enforce agreed decisions (even in difficult times). For example, this might include sticking with agreed timebox duration even when a customer or they themselves would like to have a specific feature included.
• An investment budget to facilitate reorganization and resource (or people) realignment to:
  • Budget for training and external coaches.
  • Redesigning and outfitting office space and adding electronic communication tools.
  • Creating cross-functional rollout teams or a Project Management Office (PMO).
  • Increasing the travel and living expenses for meetings, events, and workshops for distributed teams.
  • Budget for investing in test automation and other practices to support the short timebox durations.
  • Modification and changes to project management metrics and reports.
• Their support to transfer the agile practices to other departments and functions outside of the development team, including:
  • Human resources for changes to performance evaluations.
  • Facilitates and Information Technology for team spaces as well as collaboration methods for distributed teams.
  • Marketing for changes to product release cycles.
  • Finance for capitalizing projects with incremental delivery.
  • Procurement for changing contracting methods and practices.
  • PMO for adjusting the project reports, measures, and metrics to those that can be provided by an agile project.
• Their commitment to stick with the practices for (at least) a defined time even when things appear to be failing. Specifically, the first one to two iterations in a project may under-deliver with regards to features expected; according to experience, this is normal. Management commitment is needed to ensure that transformation continues until the performance stabilizes to an acceptable level.
• Finally, middle management commitment to use people in a different way than in the past, including
  • Assignment of tasks to the project as a team and not to individual team members.
  • Group-wide estimation practices.
  • Prioritization of the requirements (and subsequently delivered features) at multiple intervals.

Add Experience

Many day-to-day operation problems can be avoided or mitigated by having team members with experience in agile projects. One of the assumptions with agile methodologies is that the teams are experienced in their subject matter and in agile practices.

Share internal people

The options for gaining that experience in agile practices include:

• One agile team is split to make a second team.
• Add inexperienced team members to an on-going project team to gain experience.
• Place experienced team members from one team in another team to coach them on a part-time basis.
• Establish a cross-functional rollout team (or project management office).

Hire external consultants

Experienced people bring practical knowledge from other situations and environments that can be helpful in avoiding pitfalls, in recommending tips and techniques for executing the project, and in coaching or supporting individual team members. While having many team members with experience is ideal, the Agile Coach, Product Owner, and Agile Tester are three roles where experience is most appreciated, as those are the topics where the most issues surface.

Agile Coach

The Agile Coach is a servant-leader that is responsible for the success of the project by supporting the Product Owner and team in applying the agile practices. The Agile Coach also resolves issues for the team and attempts to limit outside interference in the team's work.

Experienced Agile Coaches have a selection of tools, techniques, and methods on educating the organization on agile practices and supporting the team and the Product Owner in identifying and resolving problems and conflicts. They can help to avoid issues such as identifying the root cause for interruptions or too many unplanned tasks. They can support the group through the stages of development from unwilling to willing team members and from inexperienced to confident team members. They can also facilitate moving a newly-formed team from a group of the individual team members into a self-organizing team.

Product Owner

The Product Owner is a single person that is accountable for setting priorities for the project team and defining the product features to deliver the maximum business value. This role requires time for grooming the backlog, attending planning and review sessions, and supporting the team throughout iteration.

Experienced Product Owners can help align the stakeholders regarding required product features as well as gain their acceptance of the delivered product. They can also manage the backlog, ensuring it is up-to-date and has an appropriate number of features included.

The Product Owner role should be formalized, by giving it the formal title. In addition, there should be investments in Product Owner training to ensure they understand the responsibilities of the role.

Agile Tester

An Agile Tester is “a professional tester who embraces change, collaborates well with both technical and business people, and understands the concept of using tests to document requirements and drive development” (Crispin & Gregory, 2009, p. 19). They are responsible for working with the product owner, customer, users, and other sources to define feature functionality and acceptance criteria. The acceptance criteria are used to determine when a feature is complete.

Experienced Agile Testers can help to avoid delaying testing to the very end of the iteration (back-loading testing) and they can help to automate testing. Furthermore, experienced Agile Testers are useful at every stage of the product development life cycle. For example, they can:

• gather examples and ask questions to clarify user stories;
• pair with developers while they write code or automated tests; and
• use exploratory testing techniques to offer timely feedback.

Conduct Training

For the Scrum Methodology, several training courses provide skills and knowledge necessary to be Agile Coach (i.e., ScrumMaster) or the Product Owner. Furthermore, there are training courses offered for Agile Planning and Estimation and Story Writing. These training courses offer foundational knowledge that will be needed in the project. The Product Owner has authority over the team to direct the work they perform by his or her influence on the backlog. Consequently, it is one of the key roles where a training investment is important.

Further to any external training courses, every team member should attend a training session that describes the agile practices, processes, and role expectations for the selected methodology.

Run a Pilot

A pilot project can be used to try agile practices in a controlled environment. A project with characteristics that will ensure success should be selected as the pilot. The pilot offers an opportunity to gain experience with agile practices, to involve many different types of people (e.g., agile enthusiasts and skeptics), and to demonstrate the expected benefits from going agile. Since the environment is controlled and the pilot should be evaluated upon completion, it also offers the opportunity to know if agile projects will fail or succeed in the organization.

Run an Initial Planning Iteration (Sprint Zero) or Workshop

Some agile methodologies (Scrum and extreme programming) start the project with chartering process. That means the activities to initiate the project should have already occurred. An initial release planning (sprint zero) may be used for working with the Product Owner to perform some initial planning, including defining a high level-architecture.

The initial release planning can be used to:

• produce and estimate the initial product backlog;
• break down requirements into user stories to produce the product backlog;
• identify, analyze, and quantify the risks;
• analyze dependencies between the product backlog elements, the stakeholders, and the risks;
• produce the product roadmap; and
• produce a high-level architectural roadmap.

Depending on the scope of the project, the initial release planning could be conducted as an iteration (sprint zero) or a series of workshops at the start of the first iteration.

Use Innovation Games

Agile methodologies promote a participatory design, where the customers or users become part of the team designing the behavior of their software. User stories facilitate participatory design, as the users are required to provide content for the stories.

Innovation games are a technique for engaging the customer and user in collecting and/or prioritizing user stories. They can help overcome the challenge of getting the customers in the loop. Innovation games are fun ways to collaborate with the customer or user to understand their needs and requirements. The book Innovation Games: Creating Breakthrough Products through Collaborative Play describes 12 innovation games (Hohmann, 2007):

• The Apprentice —the team members use the system the way a customer would so that they understand the challenges and view of the customer.
• Buy-a-Feature — the stakeholders are given a budget, they are shown features with their estimated price, and the stakeholder has to decide which features to purchase on the fixed budget.
• Bang-for-the-Buck — the stakeholders collaborate to prioritize the product backlog based upon value and risk. The goal is to have the stakeholders prioritize what is important and valuable.
• Design the Product Box — the stakeholders identify the features that are most important to the consumer.
• Prune the Product Tree — the stakeholders identify how the product should evolve and grow by aligning features to the base and large or small limbs of a tree.

Manage the Budget with Functional Contingency

Flexible change control is one of the principles of the Agile Manifesto: “welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage” (The Agile Alliance, 2001). Therefore, the scope remains flexible throughout the project. The project duration and costs may be fixed.

The product backlog, along with feature prioritization, change business rules, and iteration planning are used to maintain the project within budget.

The product backlog contains the features that should be implemented into the product. Feature prioritization places the most valuable features of the scope at the top of the backlog. Features should be developed based on their priority. The quantity of features delivered is determined by the team's velocity - the rate at which a feature can go from concept to completion. The prioritization supports a project in which the most valuable features are delivered with the highest priority.

Figure 1 ? Budget management

A functional contingency can be established based upon the feature prioritization.

• DSDM suggests that the contingency is the effort to implement any features beyond the “must have” features and that the “must have” features should represent 60% of the total project effort. In this scenario, the contingency is 40 percent.
• Based on the Pareto principle, 20 percent of the product features will have the biggest effects. This is supported by a DuPont study (Johnson, 2002) that reported that 25% of product features are needed and the remainder is “nice-to-have.”
• Other prioritization methodology (e.g., Kano, Story Maps) also have a way of marketing the “must have” or Minimally Marketable Features (MMF).

By agreeing the functional contingency amongst all parties and delivering features according to their priority, the method for maintaining the project within budget is to continuously evaluate and prioritize features that can be delivered with the remaining funds.

Change business rules are established to allow the features to be added, deleted, or moved in product backlog. In the Scrum process, the change business rules allow for exchanging one feature for another feature of the same size to maintain project constraints on cost and time. Exchanging features is important to managing expectation on the available capacity to deliver features within the agreed project constraints. Alternatively, the Kanban process allocates time to change requests based on demand. Works in Process limits are set for change requests based upon this time allocation.

Should the need arise to add features to the product backlog that would require additional budget or iterations, the Product Owner should agree to the project extension with the project sponsors. Such a process should be considered as extending the scope, and should be managed in a structured fashion.

Scaling

One of the key areas of concern with agile projects is scaling to large projects. The agile methodologies have different ways of scaling, but agile methods have been successfully deployed used in large projects, with distributed teams, and in large organizations. Like any large project (with any type of methodology), large agile projects are challenging. Nevertheless, it is possible to use agile practices in large projects and with distributed teams.

There are four themes to manage when setting up large agile projects: people, product, project, and process. Table 3 includes a sample of the topics that should be considered for scaling agile projects.

People Create a hierarchy of Product Owners Establish methods for working with multiple independent teams Deploy infrastructure and practices for distributed teams Establish a cross-functional PMO Initiate communities of practices	Product Create different views of the product backlog Establish quality assurance practices to evaluate product coverage
Project Formulate ground rules to support coordinating across teams Synchronize iterations Establish forums or meetings for cross team issue resolution (e.g., Scrum of Scrum) (Ideally) Establish a common estimation scale Establish risk management practices for fixing release dates Establish a management reporting structure	Process Coordinate release kick-off and planning sessions Formulate strategies for connecting related task boards Align iteration planning sessions Use rolling look-ahead planning to identify dependencies Develop processes for aligning integration testing Establish product-level as well as team-level product reviews Hold product-level and team-level retrospectives

Table 3 ? Considerations for scaling agile

Run a Retrospective

Implementing agile methodologies to a new team can be very taxing for the team members. For example, an iteration cycle of two to three weeks with all the work needed to go from a concept to a fully tested solution can seem hectic for the first few iterations. While there are many ideas and proven techniques listed for as a reference, keep in mind that the strongest message will be sent out from the team (this is true for traditional or agile projects). Send the message that the management team cares, that they are open-minded and that would appreciate two-way communication. Managing two-way communications with the team members will not only reduce resistance, it will also increase participation and buy-in from the team.

When issues arise with the team, stakeholders, or customers, hold a retrospective and tune the processes. Review often what is working well and worth repeating, what could use optimizing, and what should never be repeated. There is no cookbook for going agile. Options have to be tried and then process adapted as you go.

Closing

There is more to “going agile” than adopting an agile methodology for a project. You and your organization should consider carefully why you need to be agile, how agile you really need to be, and what kind of projects need to be agile. After answering these questions, you can start discussion and planning to decide the right agile methodology. If adopting an agile methodology is not right for all projects or for the organization, then individual agile practices should be adopted in projects to reach a desirable level of agility. Day-to-day operational problems will occur. Having team members with experience and management buy-in can help management mitigate the negative impact of any issues, problems, or challenges.