Operations First Delivery

I've been involved with many projects that attempt to increase their delivery cadence after it has been deployed to production. This is met with severe resistance for multiple reasons; the manual testing effort takes too long, the environment is unstable, or it could result in downtime and customer impact. Since the business isn't asking for more frequent deliveries how can we justify the expense?

These reasons feel similar to the the resistance I receive when introducing test driven development. This got me thinking of an idea I've coined as "Operations First Delivery". The ideas is that the first delivery of a product should be a deployment to the production environment. This may seem ridiculous since nobody is using a non-existent product. I see that as a risk-free opportunity. This is the ideal time to release to production.

If we start every product were every changeset is deployed to production, we will start with Continuous Delivery. We will learn how to do zero-downtime deployments when it's appropriate. We will learn how to monitor our systems and resolve issues that generate support phone alerts. We will learn to evaluate and reduce risks in order to continuously deliver. This learning will be incremental and iterative from the beginning of the product. 

Daily Tally

Have you attended (or participated) in this daily stand-up meeting?

Q. What did I do yesterday?
A. I worked on (pointing) this card yesterday. 

Q. What am I doing today?
A. I'll continue working on this card today. 

Q. What's in my way?
A. Nothing is in my way.

If this continues for an entire iteration the team looses credibility and trust with themselves and other stakeholders and teams. One technique my teams use to provide visibility to potential risks is a "Daily Tally" on every user story/defect card.

5 Point User Story Card with Daily Tallies
Black - Developers
Orange - Testers
Red & Highlighter - Back Flow

Every day we place a tally mark on the card. Developers and testers have unique color markers and we tally from the the left to the right of the card to represent a timeline. The colors tell us how long we've been working on a card and how the time has been split between the developers and testers.

We also add a red tally when a card is returned to the developers from the testers. A red tally indicates that we missed an acceptance test scenario or introduced a defect.

The team uses the information from the daily tally in two primary ways:

1. Retrospective Item - A red tally mark shows when we have an opportunity to improve by reviewing the causes for the back flow and determine if we want to prevent it in the future.
2. Pushing help - When the number of tally marks equal the card's point estimate it triggers the team  to ask "How can we help?". This is not a failure of an individual or the team. It is a mechanism that makes the team aware to potential risks as early as possible.

We still use more traditional tracking methods, but we've found this technique effective for spotting trouble early and providing examples (with data) for retrospective meetings.

Are all your changes included in the next release?

I'd like to share one technique I've used with teams to help error-proof their branching to verify the next release will include all their changes. This technique only applies to the "One Branch" strategy since a single code line will always contain all the changes.

Error-proof Branching Strategies

• One Branch (2 code lines) - main (of default) and release (or stable)
• No Branches (1 code line) - main (or default)

The main error that occurs with branches is that developers forget to apply their changes to all branches. For example, a hot fix needs to be made to the release code line and the development code line to makes it in the next release. Like most errors, the developers aren't malicious and telling them to "do a better job next time" won't improve their behavior.

One highly effective technique I use with teams is to run a script once a day to validate that all changes made to a release branch have been applied to the main (or default) code line. Since this happens daily, any merge conflicts are minimized.

In this example using Mercurial, this batch script is called from a Jenkins job that runs once a day. If a commit exists on the release branch that hasn't been merged to the default branch it will fail the build. I hope you find it useful.

Branching Strategies

I've stumbled across many creative and interesting branching strategies during my career. I'm going to categorize the branching strategies in to two categories: Error-prone and Error-proof.

Error-prone Branching Strategies

• Branch per release per environment based on previous release
• Branch per release
• Branch from a previous release
• Branch per environment

Error-proof Branching Strategies

• One Branch (2 code lines) - main (of default) and release (or stable)
• No Branches (1 code line) - main (or default)

My Mercurial Workflow (using Bookmarks)

The "master" branch...er...bookmark

I create a bookmark that always points to the latest changeset from the shared repository. I should call it "origin/master", but have been able to manage with a single 'master' bookmark. I update to the master bookmark before making changes or pulling changes,

Any changesets that I want to push are ancestors of the 'master' bookmark. I put new commits on 'master' and defer the decision to create a branch when I have changes that need to be shared at different times. For example, when you start working on a new feature, but have to go back and fix a quick defect.

Daily Flow:

hg update master
(edit, commit, edit, commit, edit, commit)

** determine I need to move previous commits to a feature branch
hg bookmark feature1

** move the 'master' bookmark back to the last shareable/pushable changeset (rev 1234)
hg bookmark -f master -r 1234
(edit, commit, edit, commit, edit, commit)

** Quit working on the non-sharable commits push the sharable changes
hg update master
hg pull --rebase
hg push -r master

** Time goes by with dozens of changesets by other....
hg pull --update (--update will move the 'master' bookmark forward)
hg update feature1 (move my working directory to my feature1 branch)
hg rebase -d master (rebase my feature1 changeset on to the latest changes)

** when these are ready to be shared
hg bookmark master -f (move the master bookmark to the changesets we want to share)
hg push -r master

Pushing (The secret sauce)

If you have local changesets on bookmarked branches that haven't been merged in to master and you attempt a push you get the "abort: push creates new remote head" error.

Include the -r (or --rev) option to only push master changesets.

-> hg push -r master
pushing to origin
searching for changes
1 changesets found
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files

Tips for Bookmarks

Keep an eye on the active bookmark. The active bookmark is denoted with an asterisk (*). When you commit, the active bookmark will update to your new commit. This is most important when you're pulling in other changes. Switching the active bookmark is done by updating the bookmark (hg update feature_1). This can be done even when the bookmarks are point to the same revision.

-> hg bookmark
   feature_1                 3097:6757f51d7d1a
 * master                    3101:0b0834be4897

Testing Boundaries

Many projects follow the same pattern for software testing. These projects test the entire system by running manual test cases through the user interface. When the number of manual test cases grow beyond the capacity of the current staff they introduce automation tools to execute the manual test cases. This testing strategy leads to discussions about data management, test maintenance cost and ever increasing execution times.

While there are ways to improve the exist testing process, ultimately I think the strategy is wrong. More specifically the test boundaries are wrong. We can't effectively test a system using end-to-end tests exclusively. This is an example of an alternative testing strategy.

Imagine we have a system with three Modules (A, B, C).

For the sake of simplicity lets assume that each module has 10 paths. Every path in Module A depends on every path in Module B and every path in Module B depends on every path in Module C.

If we test the entire system across all the modules, we need 1000 tests to cover all the paths through the system.

We can drastically reduce the testing effort by splitting our test boundaries.

When we unit test each module in isolation we'll need 10 tests per module, 1 for each path in the module. However these 30 tests don't cover the interactions between the modules. We need 200 integration tests between the modules. 100 tests to validate A and B plus 100 tests to cover B and C.

So if we change our test boundaries we can get the same test coverage for a system using 200 + 30 = 230 test cases.

The most powerful benefit of smaller test boundaries is the ability to quickly localize failures. If there is a single path in Module B that has a defect and we have a full system boundary we will have 100 failing test cases. This large number of failing tests makes it difficult to track down a single defect.

If we split our testing boundaries up, we have a single unit test failure for Module B and 10 broken integration tests between Module A and Module B and 10 broken tests between Module B and Module C. 21 failing tests is still a lot, but our single failing unit test will tell us the exact location of the defect and we can quickly discover the defect.

        

I realize this is a simple example, but I hope it illustrates the effects and impact of using smaller testing boundaries.

Observations from a 2 day Hack-a-thon

Last week I had the opportunity to participate in a two day "hack-a-thon". This is an event where engineers get to work on any interesting project and present their results to a panel of judges. The judges determine which projects are the most innovative and are awarded and promoted across the organization.

In the past I've been skeptical of these types of events. Most of the proposed project are engineering improvements that should be worked on as part of their project deliverables and the majority of the projects for this event would fall into that category. However, I think the event gave the teams authority and permission to work on these projects and was impressed by the results the teams produced in two (more like 1.5 days to allow for demos).

This lead me to ask a couple questions. One to the teams and one to the leaders/managers. To the team, I asked: How were the teams able to deliver such impressive results in 2 days? Some of the team's reasons included:

• Improved collaboration
• Uninterrupted/focused work
• Interesting projects
• Competition with other teams
• Scoping the work to two days
• Food and snacks

The reason that I think contributed the most to the success of the projects is scoping the work to the time available. Most teams had two to three options for what they envisioned for their demo. They delivered the simplest demo first then worked on the next small improvement. Most demos were ready during the first day.

I saw another impressive behavior...The teams followed up on impediments instantly and would not take "no" or "tomorrow" as an answer. I don't know if I would call it Drive, Responsibility or Ownership, but was very happy to see it.

On Monday, I'll follow up with manager and leaders on how to replicate these outcomes on our real projects.

Encapsulate Your Constants

Does your application have a "Constants" file? It's a class with a list of constants used across the application? It's a large file with a list of Strings that don't have any relationship to one another and has a bunch of comments and carriage returns to "organize" the file.Move them.

Exposing a public constant allows developers to access state from all areas of the application. => Limit global variables and global constants. Move them into the class that is responsible for using the constant.

Constant files grow over time => Don't introduce the constants file pattern. Your constants are related to some domain concept. Are they Roles? Are they keys for a cache or session storage? Move them to the correct class and make them private.

Constants are an implementation detail => Don't force you teammates to find the correct key name to access some session variable. Create a method to get and set the value. The storage provider and index is an implementation detail and should be wrapped in a method that knows the storage key and provides appropriate casting.

Do yourself and your team a favor and move the constants into the classes that need them.

The Five Year Rule

I have a rule that has guided my career since college. It has worked well for me and provided me a simple way to evaluate career opportunities.

When people hear my rule for the first time, their initial reaction is to laugh. After further reflection they realize it contains truths about professional career development.

The Five Year Rule

There are two reasons to have the same job for five (or more) years:

1. You love your job.
2. Another company wouldn't hire you.

There are a bunch of reasons to love your job, but make sure the reason you love your job isn't because you don't want to think about the second option.

WARNING: Do not attempt to apply this rule to personal relationships!

Mobile Blogging

Two enhancements to the blog.

1. Mobile Theme - It looks ok on a phone
2. Mobile Posting - Just setup my phone to post directly to my blog. This could be a bad idea.

Leader or Manager

As a Leader or a Manager we make decisions based on all of the available information at the time and the impact to stakeholders.

I'm going to over simplify the difference between a Leader and a Manager by observing a single aspect in the decision making process.

Managers make decisions based on the direction of a superior.

Leaders make decisions based on what why believe is best for their team.

I have held both roles in my career. The tricky part is knowing when to be one over the other...yes there are times for both.

How not to paint a fence

This summer I spent a weekend staining the fence in my back yard with the kind help of my mother and father. Manual labor frees the mind to think and I started thinking about the application of the Theory of Constraints and Lean to the task of painting a fence...yes, I was in the sun too long.

The Work

My fence is a typical 4 ft cedar fence. It has 4x4 posts every 6 to 8 feet. Connecting the posts are a pair of 2x4s. One 6 inches above the ground the other is level with the post. The fencing is made of cedar planks that are drilled into the spanning 2x4s with a couple inch gap between each plank.
There are two types of work when painting a fence. One type is the large brush (or painting pad) work to cover the the large wood planks. The second type is the trim work. The trim work on the outside of the fence is painless. A couple inches between each plank and the edges of each plank. The inside of the fence is another story. The trim work inside the fence includes the two 2x4s that span between the posts and the sections of the plank above and below those spanners.

Estimating

Before we started, I decided to time how long it took us to do the length of the outside of one side of the fence. I would use this sample duration to get a rough estimate on how long it would take to completed the entire fence. We’ll see how that worked out later...

The Flow

My father and I both started by worked on the large plank work. It took 30-45 minutes for my father and I to finish painting the large planks of the first side of the fence, however my poor mother was still finishing the trim work from the first 5 foot section. My Theory of Constraints brain kicked in and I started to think about the 5 Focusing Steps.

The Theory of Constraints (ToC) defines the 5 Focusing Steps:

1. Identify the constraint 
2. Exploit the constraint 
3. Subordinate everything else to the constraint 
4. Elevate the constraint 
5. Repeat 

It was obvious the trim work was the constraint in our system (step 1). When ToC talks about “Exploiting the constraint” it means to squeeze all the capacity out of the constraint that you can. Don’t allow the constraint to do anything activity that isn’t the constrained activity. In this case my mom was fully exploited. She wasn’t being distracted by the children or asked to grab drinks. She was just painting trim. My mom was fully exploited...from a ToC standpoint.

What we should have done to subordinate everything else to trim work (step 3) is to have the large brush work only paint area where the trim work was done, but I knew it would be difficult to convince my dad. Having everyone do trim work wouldn’t “feel” like we were getting much done. I chose a compromise. I had a quick talk with my father to develop a new plan. He would continue to use the painting pad, but he should do as much of the trim work as possible. I would switch to a 2” brush and help my mother with the trim work.

The Big Backside

The outside of a fence is much easier to paint since it is mainly large panels. The structure inside the fence contains the top and bottom 2x4s that span between the posts. This side of the fence requires more trim work. The top, bottom and side of each of the 2x4s plus the sections above and below the spanners that aren’t large enough to get with the large paint pad.

While working on the outside of fence, my father painted the front of the panel, the top and each of the half inch sides. On the backside of the fence, he was able to get the top and middle sections, but not the bottom section. He also did the spanners, but knew the “trimmers” would finish up the work.

This division of labor caused a few problems:

• Buildup of Work in Progress (WIP) 
• Invalid estimates based on the duration of unfinished work 
• Quality 

Splitting the task into specializations of large brush work and trimming we created a large amount of work in progress (WIP). Nothing was “done”. Each section of fence required somebody to go back and “rework” the unfinished plank and trim.

We could visualize how much work was left, but we needed to calculate the duration. We need to know that we had enough daylight to complete the work. Basing our duration estimates on how long it took us to partially complete a fixed length of fence didn’t give me much confidence.

The final issue that was created by the division of labor was a lack of quality focus. Since my dad knew that I would be following him to do the detail trim work, he wasn’t too concerned about completing each plank. He was concerned about applying paint to the fence planks as fast as possible.

While I was doing the large brush work, I had the same motivation. Getting the largest amount of square footage covered with paint meant the job was closer to being done. However, the job’s duration wasn’t strictly a function of square footage. It was a function of the duration to complete the trim work plus the duration to complete the large brush work.

Ultimately we completed painting the fence. I would like to say that we had enough daylight left on Sunday to enjoy an ice cold lemonade on the deck and enjoy our work. I can’t. The sun was setting and we rushed to get everything done.

Next Time...

Next time I will start with a quality focus and limit the work in progress be removing the division of labor. We will have a two person team working each side of a section of fence. Both the trim and large brush work will be completed before moving to the next section. This simple change will allow us to estimate the remaining duration, reduce rework and improve quality.

Continuous Feedback

The Agile Manifesto states that we value “Individuals and interactions over processes and tools”; however we rarely discuss methods to improve the individuals on the team or their inter-personal soft skills. We need more practices that help develop the people on the team to increase the performance of the team. Improve the people and the process will take care if itself. For this reason, I want to introduce a practice I named “Continuous Feedback”.

Continuous Feedback is a practice where the team provides praise and criticisms to other team members to allow them to understand how their performance or behavior is being perceived by other members of the team.

Differentiation
The idea is based on Jack Welch’s concept of “Differentiation”. His concept is to split the organization into 3 segments: the top 20%, the middle 70% and the bottom 10%.

The top 20% of the workforce are the stars of your organization. Lavish them with rewards and bonuses. They are smart, they get things done and they are your leaders. Reward, challenge and develop them or else someone else will!

The middle 70% will require the most attention. Do not diminish the importance of the middle 70%. They are the heart and soul of your organization. The top performers in this segment will need to be developed to move into the top 20%. The people on the lower end of the 70% will need to know where they stand. People react differently when challenged. Some will “step up” others will “step down”. Either way, it is each person’s responsibility to make those decisions based on honest feedback and by knowing exactly where they stand.

Finally, cut the bottom 10% out. By cutting out, I’m not talking about firing people unexpectedly. Let these people know where they stand. By knowing they are in the bottom 10% they may leave on there own. Nobody like to be were they are not wanted. Inversely, they might know where they stand and by understanding exactly where they stand they may be able to move into the 70% or higher!

I know the idea of eliminating the bottom 10% sounds harsh, but most companies/teams never purge the weakest performers and this builds resentment within the other team members.

We all know who these people are and we never provide feedback to let them know where they stand. It’s easier to avoid the situation than to confront them. Nobody wants to be “mean”. However, when layoffs happen they are surprised to find out they are first to be let go. Isn’t it fairer to let the weak performers know exactly where they stand? If you didn’t provide that feedback, aren’t you part of reason they were let go?

Over time, cutting the bottom 10% becomes extremely difficult. Think about it. You can probably think of a few people on your team or in your department you wouldn’t mind if they were “released”. Now image if you had to go through a couple more rounds of cuts. The bottom 10% quickly becomes very qualified people! Your middle 70% doesn’t consist of people “hiding out” in the organization, waiting to collect a pension, those people are long gone. Your organization’s middle 70% are the best people in the industry!

The Game
At the end of every iteration we hold a Retrospective meeting we, as a team, we discuss what went well, what we could improve on and how we could make the process better. As a part of this meeting, each member of the team must select the people who they believe to be in the bottom 10%. On our team 10% is one person. We do not publicly discuss why they were chosen. But they are encouraged to meet with the person to get feedback during the next iteration. After the exchange of the bottom 10%, each person must select the top 20% and during the meeting explain why they picked them as a top performer.

We publicly select the top 20% and the bottom 10%, and publicly state why top performers are in the top 20%. We allow each member to privately receive feedback from members so they can improve and know where they stand within the team. Giving and receiving constructive criticism are important soft skills that now get practiced regularly.

My Windows (with git) RVM

I was disappointed to learn that RVM (Ruby Version Manager) doesn't work on Windows, however I wasn't surprised. Windows and Ruby don't always play well with each other.

I'm also a fan of Homebrew on my Mac. Homebrew is a command line package manager that downloads applications and creates symlinks so they are always on your path.

I hate working with Windows environment variable since they don't always reload without logging off and back in again.

For all of these reasons I started thinking about how I could configure my laptop to use multiple versions of Ruby without hacking my PATH environment variable.

My solution was to create a C:\Ruby directory and initialized a git repository within that directory.

I downloaded the 1.9.1 7-Zip package from rubyinstaller.org and extracted it into C:\Ruby. Then I added/committed all the files to a branch named 'v191'.

Finally I updated my PATH to include the C:\Ruby\bin directory.

To install a different version of Ruby I created a new branch named "v192", delete all the files and committed the deletion of the files. Here are the commands:


git checkout -b v192


rm -rf *


git add . -u && git commit -m "Cleared directory for next version"

After I have a clean C:\Ruby directory, I extract the new version to C:\Ruby and add/commit them to the git repository.

After installing I can do a git checkout {branchname} to switch my version of Ruby.

Migrating the blog

I don't know if any reads this anymore, but I am moving my blog from http://blogs.giffordconsulting.com/ to http://blog.giffordconsulting.com. Hopefully your reader didn't pick up all these posts as "new". They are not "new". They are, in fact, very, very, very old.

I had lofty aspirations when I started blogging back in 2005: 

1. I would blog frequent
2. I would develop a community of bloggers under the giffordconsulting.com domain.
3. I would contribute to the SubText blogging software project

I didn't do any of those things...I did join Twitter!

The new blog is hosted in the cloud:

1. I don't have to pay anything
2. Upgrade or fix software bugs
3. Worry about bandwidth when I get on Hacker News (Is that another lofty goal?)

Testing Carriage Returns in FitNesse

FitNesse doesn't allow carriage returns in test fixture table. Normally, I replace carriage returns with a comma (or some other sentinel value) and pass it back to FitNesse. I noticed a posting on the FitNesse email group that recommended defining a variable and then using the variable in the table.

An Example:

!define multiLineAddress {Gifford Consulting
Attn: Tim Gifford
123 Main Street
Des Moines, IA 50266}

!|Check Address|
|Address|
|${multiLineAddress}|

Microsoft OPEN.NET

Late last week we learned that Microsoft is de-funding the Iron projects (IronRuby, IronPython). Many have suggested that the community should take over the projects, but without access to the DLR, CLR and the compilers success would be limited.

However if Microsoft opened up the DLR, CLR and compilers for what I'm going to call "Open .NET", we in the community could make some interesting things happen on .NET.

I know it sounds crazy to open source the .NET platform, but I don't understand what competitive advantage Microsoft maintain by keeping it closed.

I want to mention a couple advantages they would gain by opening it up.

Allowing the community to create libraries and offer bug patches to the CLR, DLR and compiler we (the community) would feel involvement in the platform. This involvement would translate to more community created libraries and solutions.

Microsoft wouldn't have to create tools and libraries that already exist (ala MsTest) and focus their efforts on expanding the platform.

I'm not proposing that all products and innovation that is created by Microsoft shall be Open. They can build new technology within and around the open platform and release those changes to the community core projects after the product announcements.

Create native extensions for Windows that increased performance and scalability. Create products that only run on IIS or SharePoint. Microsoft's current "enterprise" customers would continue to use the proprietary and supported versions of the compilers and runtime.

Without an "Open.NET" Platform, projects like IronRuby and IronPython are not feasible outside of Microsoft. This innovation while secondary to Microsoft's corporate strategy is important to many of us in the community.

Scheduled Restart

I haven't been blogging for a while and have missed it. Twitter allows for quick quips and lunch/bar conversations allow for a deep introspection but I've been missing a venue to formalize my thoughts and to improve my writing ability.

Topics I will cover:

• Lean Software Development
• Test Driven Development
• Automated Acceptance Testing
• Technology Reviews
• Book Reviews
• Experience Reports
• ...and random posts when I need more that 140 characters.

Team Foundation Server forces teams toward open source solutions

Source control is the lowest common denominator with software development. All (...ok...most) software development teams have a system in place to manage the version of their software source code.
Within the Microsoft eco-system there are 2 options provided by Microsoft:

• Microsoft Source Safe
• Microsoft Team Foundation Server

SourceSafe has a bad reputation so within the last couple years, teams that develop on the Microsoft stack have been forced to limp along with Source Safe or upgrade to Team Foundation Server.
I don't want any one to get the impression that I don't like TFS; because I do. I like the integrated lifecycle, powerful reporting, task and project management tools. I like these things when they are needed. However, most mid-market software development shops don't need (or can't afford) all those tools.
But what if you want the tools? Have you ever tried to determine the pricing for a Team Systems rollout? Can somebody tell me how much it will cost to roll out TFS for a development team of 10 developers? I've tried to do the pricing, but can't navigate the licensing.
So what should you do if you need a source control tool?
a.) Spend a couple days putting together a proposal to submit to your boss, who will submit it to her boss which will ultimately be rejected for "Budgeting Reasons".
b.) Do you stick with Source Safe? Not because you want to or think it's the best solution but because it's the only other alternative from Microsoft?
c.) Or do you install a Subversion Repository (free + time) on an existing server (free), install TortoiseSVN (free + time) on the developer workstations and hold a day of training with the development staff (time)?

Agile defined with one photo

I got this photo today at work when a co-worker recommend a couple alternative strategies. I love it and had to share!
 
From: http://www.michaelsalamon.com/?p=20