Managing Software Releases Effectively

 Software release process varies from organization to organization depending on the kind of the business and development team. When the team working on a software product that is geographically dispersed, the software releases throw more challenges. The process should not even be thought without a global release manager (GRM) or a release co-coordinator or a central release management tool. A solid build/release infrastructure has to be in place.

What's the difference between build and release?

Let’s clarify certain terminologies before proceeding. We often use the terminologies build and release. Sometimes we use these terminologies interchangeably. However there is some difference between build and release.  According to me builds are the deliverables that are used by developer themselves and testing teams internally and releases are something that we hand it over to our end customers.

How do we perform the builds?

Each programmer can build the project source code he/she is working on and continue the work, however this is bad practice. A programmer's code may work well because its not yet merged with every other programmer's code. So as a best practice the builds must be performed by a system that is updated with the latest source code of the entire project.

For effective builds, the build process needs to be automated and continuous improvement tools such as Hudson, CruiseControl etc should be used. I prefer and suggest Hudson which is easy to use. The builds can be categorized into different types as Daily Builds, Weekly Builds, Force Build etc and these builds are triggered by Hudson running on dedicated computer (we call it build machine).  Daily builds ensure that there are no compilation errors.Daily builds are also referred as nightly builds since they are triggered during the night.  Apart from daily builds, the best practice is to initiate builds every time when there is CM change (source file check-ins into repository) and build machine configured to send mails to all team members if there is a build failure. We can configure Hudson to send build failure emails to the commit author due to which the build was failed.  We can enable this kind of builds once the source code is stabilized and every team member are required to be cautious about the code stability. The problem is that if a particular commit by a team member caused a build failure, the other members in the team can’t proceed with their work unless the build failure is corrected and hence it’s important to know what change and by whom the build was failed and same should be corrected by the member as early as possible.To exploit the real advantage of having daily builds is to run automated test cases after the build, by seeing the test execution reports we can better understand how well the code changes are being integrated. 

Managing Software dependencies 

Some software products have dependencies on other software may be developed by the same company or a third party company. It depends on the software product design. Most of the product development is to develop the reusable software pieces. We called them core assets and some may call them as reusable library (in the forms of dll files or java jar files etc). Core assets are software components which are tested, stabilized and ready to use. Core assets can be compared to car parts. To manufacture a car, first the car spare parts are made and then they are assembled together. There can be changes in core assets from one release to another and since a core asset may depend on another core asset, all dependent assets must also be rebuilt along the asset where the change was made. The build machine should be configured to handle these dependencies automatically. Imagine there are 100's of reusable software components and if you are required to modify at least in 10 components, its tedious job to manually identify the inter-dependencies and make releases.

If the product is internet based, the product can be programmed to download all the required versions of dependencies from the web during product installation. 

Packaging(Bundling) the software into an executable 

Any software consists of multiple files such as jar files, dependent software component files, resource files, icons, text files etc. All  these files have to be packaged into a single executable file called Installer. InstallShield is an example. This makes it easier for our customers to install the software product. See how to make executable installer using Java. The packaging process has to be automated, we can have a packaging script that knows what files to package into an installation file. Its important to make it very simple for the customer to install the software. Special care has to be taken when customer is upgrading to new version of the same product. Installation script must be thoroughly reviewed and must not pose any issue after product up gradation. 

Releasing the Software product to customers 

There are various channels as to how a product reaches the customers. It could be contractors, marketing people or through advertisements over the web. The internet media has made it simpler for the customer to access the new product these days.The new product release can be published in the companies websites giving download links for the customers. We did it using Hudson server itself, the company contractors can download the software from build PC and then distribute it to the end customers. I mean it depends on the marketing model of the company.

Certain questions must be asked and answered before releasing the product to the market. I've tried to summarize few, there could several such questions to be answered.

1. Have all the test cases covered and passed? - Test coverage is very crucial
2. All the deployment scenarios have been tested? - Its very important to understand the user deployment scenarios. Not all the customers run our product in power PCs and speed network connectivity.
3. Are there any critical issues still open? All critical issues must be resolved, certain marginal issues can go as open issues.
4. Is the software release bulletin been prepared? - The Software Release Bulletin (SRB) has information on what are the various product deliverable, open issues (if any), installation instructions etc
5. Has the intent of releasing this version of the product  been documented? - This is required because a there will be several product releases and over the time, one needs to know why this version of  product release was made.
6. Is the user guide complete and covers all the functionalities of the product?  - This is very important if you are launching an entirely new product. User guide(manual) is one aspect, its also important to have some training sessions with the customers about the new product.

In Summary

Software release processes must be given more attention because these are the processes that are almost ignored and also since software releases management directly impacts our customers. Its advisable to automate the software release activities and standardize them as much as possible since manual activities are always error prone. Also its good to write something called Build Verifier that will verify the if the build/release was done properly, doing it manually will still be error prone. Build verifiers can  check if the required versions of project files/dependent files  have been bundled/packaged etc.

A Brief Overview of Software Product Development

There are various methodologies like Agile, Srum, and Water fall etc for software product development. Some methods have proven to be effective. Different organizations have different strategy for product development. In this post I’ll give the heads up on activities involved in developing typical software product and some best practices to perform these activities effectively.

A new product evolves from marketing need. So a product starts from Market Requirement Document (MRD) which is typically written by marketing team. It can also be written by Business Analysts, varies based on the business model of the organization. It’s very much essential for every stakeholder to understand the market need for the product because each product has something called Market Window and needs to be delivered at planned date. If market window is crossed, the product may not sell in the market and will be a failure. 

From MRD, a Product Requirement Specifications (PRS) document is written by marketing team or requirements owner. In certain organizations there will be a requirements owner who owns requirements. Any modifications/addition of new requirements will be entertained only by this person.  This gives control over managing the requirements.  For effective requirements management, the requirements are logged into project management tools like Jama contour. 

From PRS a detailed Software Requirement Specifications (SRS) document is written. This might require meeting the key customers who are going to use the product. VOC (Voice of customer) is mandatory to better understand the requirements from end users perspective. One lesson learnt is that VOC must capture the product deployment scenarios. This is very important; we should know how & where the product will be used before even start understanding the detailed product requirements. 

Understanding the requirements

Once the software requirements are base lined, there will be series of requirements walkthrough sessions where the requirements are reviewed/discussed with the team members. It is crucial for all team members to understand and acknowledge the requirements. Asking all the questions on requirements and getting them clarified with requirements owner or the customers is crucial for successful product delivery. Imagine a situation where all the customer requirements written are addressed but one particular customer requirement is completely missed because it was never written down and no one in the team is aware of it! This can happen because of poor VOC. In order for a team member to design and write code with zero defects, there has to be granular requirement analysis. It might so happen that for some abstract requirements there could be implicit requirements. All implicit requirements should also be captured as software requirements. 

Converting requirements to Design

The design can start by writing use cases.  You might be required to do prototyping before taking up some designs. From requirements document, you can start identifying the classes from nouns and verbs in requirements to get started (Just a method). Creating a good design comes by practice and experience, there are no fixed rules for designing software that all should just follow. Quality design requires experience. Typically UML (Unified Modeling Language) is used for creating the design and there are tools to capture the design. The UML design tools such as Enterprise Architect, Rational Rose etc are used for designing. There are many UMLdesign tools, select the one that suits your requirements. There are different types of design diagrams, architectural diagram, deployment diagrams, class diagrams, sequence diagrams, activity diagrams etc. An understanding of what these design diagrams depict is required for designing. The key point in achieving the design excellence is by reviews. The entire design has to be reviewed by software architects to find design defects if any.  Before you say the design is complete, you need to ask certain questions like, does the design address all functional and non-functional (like performance related) requirements? Are all the failure modes captured in the design? etc.  And also it is important to have treatability from requirements to design. All requirements must be mapped to some design tags. 

Implementation

Some UML design tools have capability of creating skeleton code from the design. This can save some coding time.  Keeping the design as reference, the coding will happen. For any language there are coding guidelines which all developers have to follow. Let all classes and methods contain the design tags for treatability purposes. Ensure that the code will handle all the exceptions. Once coding is completed, there will be code reviews. There are static code analyzers for reviewing the code, these analyzers help programmers  know what rules have been violated during coding. Ex: CheckStyle, FindBugs etc.  

  

Unit testing the code (White Box Testing)

Each developer will test the code written by them. Unit testing can be automated using JUnit for testing java code. During unit testing, it’s important to collect code coverage reports to understand if all the code written by you is covered during unit testing. The code coverage tool I used was Emma. As a good practice, it’s very important to log unit test defects into defecting tracking tools like (JIRA, DevTrack, Bugzilla etc). It’s required understand ourselves as to what is the defect count in each phases of software development.  Defects can be in any phases, it can be requirements collection, design & implementation. Its good practice to maintain code coverage reports for your reference and unit test case records

Developer Integration Testing (DIT)

Once each individual finishes the unit testing, a DIT build is made for integration testing the product. This is when the interfaces get tested. I’m not sure if collecting the code coverage during DIT is a good practice or not, but we did collect the coverage. Coverage data from all the members involved integration testing is merged and a final code coverage report is generated and kept as a reference. Certain DIT test cases can be automated (This depends on the kind of the project) and test cases that can’t be automated are captured in a DIT test case document and DIT test results are entered against each test case. DIT is where developers still have control over the product because the build is not handed over to testing team yet.

Product Assurance Testing (PAT)

Product Assurance Testing is done by testing team (Black Box testing). A PAT release is handed over to testing team after completing DIT and fixing DIT issues if any. Before starting the actual PAT, the testing team would perform different tests like Acceptance Testing, Smoke testing to find if PAT can be continued or not. Testing team will a Test plan and PAT test case document. The testing team logs the bugs into defect tracking tool. There will be multiple PAT cycles planned based on the test design.

System Integration Testing (SIT)

SIT is system level testing performed by testing team. If the product being tested has to work with other systems (product), it’s tested in this phase.

Performance and Capacity Testing (PCT)

If the product has non-functional requirements like performance and capacity testing, and then testing team perform a separate test cycle called PCT. Performance constraints will be captured as requirements. Ex: User should be able download a file with 10 seconds. Capacity testing involves load/stress testing. Ex: Server must handle 500 requests at the same time. The key factor to meet performance and capacity requirements is to start addressing them right from design, coding. Faster code helps to meet. If performance requirements are crucial in the product, then having experienced developers for design and coding is important equally.

Beta Release

The beta release is done by the end customers and there will be beta feedback which will be addressed in the final release. Beta testing is performed by identified key customers.

Final Release

The product is made available in the market for the customers. No product is bug free. The final product release will follow a Software Release Bulletin that will have open issues. User guide will also follow. The marketing team will have training programs for the customers as to how to get familiar and use the product.

Conclusion

In all the above phases, the activity that leads to quality product is review. The result of root cause analysis for most of the issues reported was poor or no reviews and improper impacts testing. The best practice is to have reviews for all the activities starting from Software Requirements to change requests. The reviewer has to have domain knowledge and a checklist with him/her to make sure all the review items have been covered.Based on the code change, the impacts have to be identified and tested. This will make sure the code change done by you does not break any other functionality. It is general tendency to have more bugs being reported during DIT than UT. As a best practice defects must be caught in early phases of the product development like, requirements, design etc.

Source Code Configuration Management Best Practices - Software Development

Configuration Management(CM) is another important aspect in Software Development Life Cycle. Here I'm presenting the best practices, lessons learnt & tips while dealing with configuration management taking Tortoise SVN as SVN client Configuration Management Tool. I've categorized these practices as General, Branching & Tagging, Merging & Builds.The guidelines are more of SVN Subversion Dos and Donts. I've shared some 25 tips & tricks in Source Code Management.

GENERAL PRACTICES

1. Check in often: Do not keep checkouts for more than 3 days. The sooner you check-in your changes the sooner you can realize your changes have been well integrated with other changes and are working fine
2. Commit logical changesets: Do not check-in multiple changes that serve different purposes made on the same file at once.  It makes maintenance difficult
3. Do not ever check-in the code without getting it reviewed. Worst case the peer review. You never know the code that's works fine on your system can affect or break any other functionalities. This might cause other developers to pause their work. The review must capture the impact along with other review checklists
4. Do not copy an SVN folder from one location to another and try check-in files from copied location, the files will get committed in the orinal location! Instead use SVN export option available if you want to copy a SVN folder to a different location (Make sure the copied folder does not show SVN icon)
5. Never overwrite a file into SVN repository and commit it, instead merge two files, verify the merge and commit
6. Do not work outside of managed workspaces. SCM system can only track work in progress when it takes place within managed workspaces. The work done outside managed workspace(Ex: on a local copy) is not the work done because that work can any time be last due reasons like disk crash etc
7. Do not share the workspace (where developers build, test, debug). It will be confusing and the work can not be tracked and it's against SCM rule
8. Write Meaningful Commit Messages. The message must be to the point and describing what was changed and possibly why.
9. Use a common and planned editor for viewing the source files. All project members must use the same editor for editing the files. This is because editors can insert/modify certain characters in the source files and such file some times may not be comparable by file compare softwares or SVN diff may not indicate the correct changes made
10. Stay in sync with latest changes: As a developer, the quality of your work depends on how well it meshes with other peoples' work. So update your workspace on regular basis.
11. Before you claim that your files have been checked-in, invoke SVN "Check for modifications" option and verify that any file still needs to be checked-in. This is necessary because there is a chance that we may forget to check-in some files and it gets discovered during the smoke testing the build. Imagine the time taken to rebuild and retest for this mistake!

BRANCHING/TAGGING

1.  Do not branch until unless required, since we'll have to merge the branch to trunk and merging is time consuming and error prone
2. Propagate early : When any changes need to be propagated from branch to trunk, do it sooner than later.Postponed change propagation can result in complex file merges.
3. Always use Head Revision option for tagging the source code folders before the release. It ensures that always the latest copy from main is tagged irrespective of whether you have taken update or not
4. Tag the source code, documents before any release is made. We may want to go back to the source code or any project artifacts for a particular release made at any point of time
5. Never check-in any changes in the tag. Tags must never change. Some CM Tools ensure this. Tag is just a symbolic name for a particular revision. SVN calls it as "cheap copies". For SVN, best way to ensure this is to have commit hook on tag.
6. If a new changes are to be done on the top of a tagged source base, create a new branch and do required changes there and make the release. More importantly merge the changes made in the  branch to the trunk.

MERGING

1. Track merges manually. While committing the result of a merge, write a descriptive log message that clearly explains what was merged.(Ex: Merged revisions 3490:4120 of /branches/example branch to /trunk and purpose of the merge)
2. Preview Merges : Merging results in local modifications, preview all the merges(It's better if the  owner of the change once verifies the merge in case owner is not taking up merge activity)
3. Be cautious about the situation where two sets of changes cannot be separated out. Consider this case; you have local changes to a file called file1 and you are performing merge to file1 itself, if you feel your merge was not done properly and you want to revert the merge and if you revert your local changes to file1 before merge had taken place will be lost!
4. Get right person to do the merge. The person must be best prepared for resolving conflicts.  Changes can be merged by (a) the owner of the target files, (b) the person who made the original changes, or (c) someone else. Either (a) or (b) will do a better job than (c)

BUILDS

1. Check -in all the source files before the build is taken. People who are working on the project files must acknolwegde to the build focal that modified files and their dependent files have been checked-in. 
2. Build often:(Sanity builds) End-to-end builds with regression testing reveal integration problems introduced by check-ins
3. Keep build logs and build outputs for future reference
4. Run the test cases identified to make sure there is no build error.
5. Automate everything in the build process that can be automated. There is always a chance for a mistake in doing the things manually.