Rigor and Formality

Of all the engineering disciplines, Software Engineering seems to have the least amount of guiding scientific principles and universally accepted methods. It remains as much a psychological / sociological practice as it is technological. The reason for this stems from the fact that Software Engineering is still primarily a creative activity as opposed to a mathematical (i.e. formal) process.

Programming versus Engineering

It should be noted at this point that this is not a comparison of software programming to software engineering; these are two fundamentally different scopes of software practices. Software programming normally involves one or two developers fulfilling all the roles of architect, designer, coder, tester and maintainer to deploy a relatively specialized system with a narrow scope of deployment. Many popular methodologies exist to support this style of development including Agile and Extreme programming. Software Engineering involves a team (or multiple teams) of specialists working toward the creation of a complex system supporting the needs of a diverse set of users which is to be deployed in multiple configurations all executed in a predictable manner. Software Engineering environments produce multiple complex systems in a predictable manner.

Software programming is a more ad hoc practice suitable for specific types of projects such as prototypes, utilities or clearly (i.e. narrowly) scoped projects. Even the largest organizations harbor software programming environments particularly in the maintenance and evolution of established or legacy systems.

Software Engineering environments are found anywhere there are multiple stakeholders and technical resources must coordinate their efforts to achieve their goals. It is a more formal and scientific endeavor which can only be successfully performed with the formality and rigor of the other engineering disciplines such as physical and chemical engineering.

Guiding Creativity

Because software development is a creative activity, there is an inherent tendency toward informal ad hoc techniques in software specification, design and coding. While such development practices can, statistically speaking, be successful in some large, complex projects, the need for rigor and formality will become apparent over time. Eventually the teams members begin to burn-out, the projects become too large to properly manage with an informal style, deliverables begin to slip, requirements are skipped or misinterpreted, code debt is incurred and overall product quality suffers. Predictably executing on large-scale projects requires significant effort.

Small software development environments which start out with one or two programmers can often utilize software programming practices for the organizations initial projects. Once that organization grows, however, those same practices begin to generate more problems than they solve. While it may seem that those practices are speeding the development, the quality of that product begins to suffer. Coordination between the increased numbers of participants and stakeholders tends to overwhelm the team and less time is spent productively. Communicating all the needs and expectations becomes difficult and the variables of each project increase greatly making success more difficult to reproduce and failure more difficult to avoid. Code debt and software defects continues to increase until the code base becomes unmanageable. It begins to cost more to maintain and evolve the code and ultimately the margins begin to shrink as software productivity declines. This is when software programming must evolve into Software Engineering.

Software development is a creative activity but is must be practiced systematically and with discipline. Rigor is a necessary complement to creativity that increases confidence in the results of development activities. Creativity often leads to imprecision and inaccuracy and software development can tolerate neither. The informal approach to software development is contrary to good Software Engineering practice.

The Need For Rules

Software Engineering involves many people of differing skill-sets, goals and interests. Without set rules, each participant imposes his/her own interest on the project. When problems occur, they become difficult to resolve and often result in unproductive conflicts. Software development needs a set of rules which allow participants to divide the workload without losing track of the work to be performed.

The rigor of a software development process is the level of discipline the process exhibits often through governance; it is the rules which direct how the process is executed. With rigor, a process can carry on smoothly without hindrances but without it projects invariably stray into problems resulting in unreliable products, high costs, missed schedules and even project failure. Rigor helps to produce products with higher reliability, greater quality while controlling costs and meeting expectations. Of equal importance is rigor enables repeatability and allows teams to avoid problems experienced in past projects.

The cost of activities early in the projects life seems high at that point in time but is insignificant when compared to the costs incurred in later phases. Time spent in clarifying requirements is minuscule when compared to the cost of re-designing a product to accommodate a forgotten requirement. Consider building a system which is required to encrypt all data in transit and then trying to retrofit such encryption requirements into a existing system a year after it was released. All software development projects can benefit from some set of rules to follow to ensure important requirements are not missed or undervalued. Some level of rigor must be applied in requirements gathering and analysis to ensure success and to avoid future costs of issue remediation.

Setting and following process rules allows all participants to operate in a coordinated manner.

Formality

There are varying degrees of rigor from completely ad hoc to the highest level of formality. A formal practice is where software systems can be verified by mathematical laws. Automated testing and error removals are not only possible but one of the benefits of adopting formal rigor in software development. There is a branch of software engineering known as Formal Methods which researches the potential of applying formality to software development. While is is well debated in many classrooms, conferences and coffeehouses, practical application of formal methods has been limited.

An excellent example of formality can be found at NASA in the development of software for space flight. The Goddard Space Flight Center (GSFC) software process has produced products with near zero defects. It is a testament to the level of rigor exercised by GSPC software engineers.

Summary

Rigor one of the elements which separates software programming from Software Engineering. While software development is a creative process, it benefits from some level of rigor which governs how that creativity is applied. Some programming projects complete just fine with a low level of rigor, but to consistently execute software development with predictable and high quality results, software engineers use rigor to guide their activities.

Continue reading...