class: center, middle .center[![Silver Bullet](silver-bullet.png)] # No Silver Bullet Shahid Beheshti University
[S. M. Masoud Sadrnezhaad](https://www.linkedin.com/in/smmsadrnezh/) --- class: center, middle [//]: # (CHAPTER 1) # Software Complexity ### \- Software Crisis and Software Werewolf ### \- Essence and Accidents ### \- Complexity, Conformity, Changeability, Invisibility ### \- Past Breakthroughs Solved Accidental Difficulties ### \- Hopes for the Silver ### \- Computing Capabilities vs. Human Capabilities --- # Software Crisis * **Difficulty** of writing **useful and efficient** computer programs in the required **time** * Due to the **rapid increases in computer power** and the **complexity of the problems** that could now be tackled * With the **increase in the complexity** of the software, many **software problems arose** because existing methods were inadequate. * First NATO Software Engineering Conference in 1968 at Garmisch, Germany * Edsger Dijkstra's 1972 ACM Turing Award Lecture makes reference to this same problem: (Edsger Dijkstra, The Humble Programmer (EWD340), Communications of the ACM) >> Machines have become several orders of magnitude more powerful! To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem. --- # Software Crisis (Contd.) * The crisis manifested itself in several ways: * Projects running **over-budget** * Projects running **over-time** * Software was very **inefficient** * Software was of **low quality** * Software often did **not meet requirements** * Projects were unmanageable and code **difficult to maintain** * Software was **never delivered** * The main cause is that **improvements in computing power** had outpaced the ability of programmers to effectively utilize those capabilities. * Various **processes and methodologies** have been developed over the last few decades to **improve software quality management** such as procedural programming and object-oriented programming. * However software projects that are **large, complicated, poorly specified, and involve unfamiliar aspects**, are still **vulnerable to large, unanticipated problems**. --- # Mythical Man-Month .center[![Silver Bullet](silver-bullet.png)] * Book written by Fred Brooks * Published in 1975 * Essays about software engineering and project management ## Still relevant today? * Brooks Law * No Silver Bullet --- .center[![Moore's law](moores.png)] --- # No Silver Bullet > “There is no single development, in either technology or > management technique, which by > itself promises even one order-of-magnitude > improvement within a decade in productivity, in > reliability, in simplicity” (1986). .center[![Silver](silver.png)] * Silver bullet: a way to defeat werewolves. * Generally any straightforward solution perceived to have extreme effectiveness. * The paper compares software to hardware: * The anomality is not that software progress is so slow, but that computer hardware progress is so fast. * Frederick P. Brooks made the argument that there is no silver bullet that can kill the werewolf software projects. --- # The Software Werewolf * Software is like a werewolf - it looks normal until the moon comes out and it turns into a monster * Missed deadlines * Blown budgets * Buggy software * We want the silver bullet to kill the monster * Something to make software costs drop as rapidly as computer hardware costs do. --- # Essence and Accidents * Following Aristotle, difficulties are either essential or accidental. * **Essential:** constituting or being part of the essense of something; inherent. * **Accidental:** Of or relating to a property, factor, or attribute that is not essential * Note: This does not mean by chance. * The **essence** is what the software does and the **accidents** are the technology by which the software does the essence or by which the software is developed. * E.g : In a calculator doing calculation is essence and calculation algorithm used to do that is accidents. --- # Essential vs. Accidental Difficulties * Essential: a characteristic of software * Difficulties inherent in the nature of software * Data sets, relationships among them, algorithms, and function invocation * Inherent properties of the essence of modern software system: * Complexity, Conformity, Changeability, Invisibility * Accidental: a problem in today’s production methods * Difficulties related to the actual production process. --- # Essential vs. Accidental Difficulties (Contd.) * Essence vs. accident * We can get rid of accidental difficulties in developing software * Getting rid of these accidental difficulties will increase productivity * For example using a high level programming language instead of assembly language programming * The difficulty we remove by replacing assembly language with a high-level programming language is not an essential difficulty of software development, * It is an accidental difficulty brought by inadequacy of assembly language for programming --- # Inherent Difficulties in Software * Brooks argues that software development is inherently difficult >> The hard part of building software is the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. >> >> Even if we remove all accidental difficulties which arise during the translation of this conceptual construct (design) to a representation (implementation), still at its essence software development is difficult * Software has the following properties in its essence: * Complexity * Conformity * Changeability * Invisibility * Since these properties are not accidental representing software in different forms do not effect them. --- # Difficulties in Essence of Software ## complexity * Complexity: due to interaction of components, number of possible states grows much faster than lines of code. * No two parts are alike * Huge number of states * Complexity grows nonlinearly with size * Can’t know the whole domain, process, or system * Unlike other disciplines, we can’t abstract away the complexity because it is essential * Math, physics: complexities ignored in the models were not the essential properties of the phenomena. * Introduces technical and managerial problems leading to unreliability --- # Consequences of Complexity * Technical Problems: * communication among team members * enumerating (much less understanding) of all possible states of the program * Management problems: * conceptual integrity is hard to achieve * learning curve: personnel turnover becomes disaster --- # Consequences of Complexity (Contd.) * Communication overhead → cost overruns, schedule delays * Large number of states → unreliability * overview is hard * hard to find and control all the loose ends. * creates a tremendous learning and understanding burden * Complex function → poor usability * Complex structure → poor maintainability --- # Difficulties in Essence of Software ## Conformity * Conformity: must interface with existing systems. * Conformity comes from the fact that all new software must conform to the way things where done in the past, because it is hard to change everyone. * Main reason software must conform is that it is new on the scene. * Software is required to conform to its * Operating environment * Hardware * Often “last kid on block” * Perceived as most conformable --- # Conformity (Contd.) * Most of the time software systems have to interface with an existing system * Even for a new system, the perception is that, it is easier to make software interfaces conform to other parts of the system. * We need not to change the whole existing system because of new software. --- # Difficulties in Essence of Software ## Changeability * Changeability: must model changing real world, increase functionality, run on new hardware. * Change originates with * New applications, users, machines, standards, laws * Hardware problems * Software is easy to change, unlike hardware * Example: Once an Intel processor goes to the production line, the cost of replacing it is enormous (Pentium bug cost half billion dollars) * If a Microsoft product has a bug, the cost of replacing it is negligible. * Just put the new download on a webpage and ask users to update their software. --- # Changeability (Contd.) * Being able to change something after it has been constructed is a new concept. * Software must be able to change, and the complexity of the systems becomes harder to follow. * All successful software gets changed. * Successful software also survives beyond the normal life of the machine vehicle for which it is first written for. * Software is easier to change than hardware * Changes during development * Changes after deployment * New features * Software lives longer than hardware * Note: People underestimate difficulties of change. --- # Changeability is not an Advantage * Although it sounds like, finally, software has an advantage over hardware * the effect of changeability is that there is more pressure on changing the software * Since software is easy to change software gets changed frequently and deviates from the initial design * adding new features * supporting new hardware * Conformity and Changeability are two of the reasons why reusability is not very successful in software systems * Conformity and Changeability make it difficult to develop component based software, components keep changing. --- # Difficulties in Essence of Software ## Invisibility * Invisibility: cannot visualize all aspects at once. * Software has no 3-D way on which it can be laid out. * Software has data flow charts, time sequence graphs, and other types of things which are hard to understand. * Communication becomes hard because each person may see it a different way. * No obvious representation * E.g., familiar geometric shapes --- # Invisibility (Contd.) * The code is invisible and unvisualizable * Software reality not embedded in space: no ready geometric representation. * Structure is terribly complex and hidden. * Software does not have one map or graph, it will have multiple maps and graphs. * There’s only the external input/output view. --- # Invisibility and Visualization tools * Visualization tools for computer aided design are very helpful to computer engineers * Software tools that show the layout of the circuit (which has a two-dimensional geometric shape) makes it much easier to design a chip * Visualization tools for software are not as successful * There is nothing physical to visualize, it is hard to see an abstract concept * There is no physical distance among software components that can be used in mapping software to a visual representation * UML and similar languages are making progress in this area. --- # Summary * According to Brooks, there are essential difficulties in software development which prevents significant improvements in software engineering: * Complexity * Conformity * Changeability * Invisibility * He argues that an order of magnitude improvement in software productivity cannot be achieved using a single technology due to these essential difficulties --- # Past Breakthroughs Solved Accidental Difficulties * High-Level Languages * Time-Sharing * Integrated programming environments - Use of libraries, unified file formats, pipes, filters etc --- # High-level Languages * The most powerful thing to happen to software development ever. * It allows that some types on data structures, operation, classes etc, to be used by any machine. * Programmers don’t have to develop software for individual machines. * Powerful stroke for software productivity, reliability, & simplicity * Removed the low level complexities * Benefits of further improvement is limited. --- # Time-Sharing * Most observers credit the time-sharing improvement to better programmers, and better programming languages. * Programs run faster because of the high-level languages. * Do not have to accommodate for machine language anymore. * Increased turnaround time and productivity of programmer * Improvement is not as much as that of high-level language * System response time is no longer an issue --- # Unified Programming Environments * Difficulties of using programs together * Providing integrated libraries, unified file formats, and piles and filters * Each new tool could be applied to any programs by using the standard formats --- # Is There Any Hope for Silver Bullet? Yes! .center[![Silver](silver.png)] > “A disciplined, consistent effort to develop, propagate, and exploit these innovations should indeed yield an order-of-magnitude improvement. >> There is no royal road, but there is a road.” * Hopes for the Silver * OOP * AI (expert systems) * “Automatic” programming * Graphical programming * Program verification * Better workstations, environments and tools * Buy vs. Build * Requirements refinement and rapid prototyping * Incremental development * Great designers --- # Hopes for the Silver (Contd.) * OOP * Hierarchical * Data hiding Helps in design, but do not solve design complexity Problem * AI (expert systems) * May be very useful. * “Automatic” programming: generation of a program from problem specification * Used successfully for very specific tasks (differential equations,…) * Hard to imagine having a general solution --- # Hopes for the Silver (Contd.) * Graphical programming * No hope, for software is difficult to visualize * Program verification * Might reduce the program-testing load, not eliminate it * A lot of work * Can establish that a program meet its specification. But the hardest part is to get such complete and consistent specification! * Better workstations, environments and tools * are welcomed, but magical enhancements cannot be expected. --- # Hopes for the Silver (Contd.) * Buy vs. Build * Discusses the process of wide-spread use of software “today” compared to 60-s, adopting procedures to existing software. * We can buy a software and can distribute to others as there is no replica fee instead of building for each user. * Requirements refinement and rapid prototyping * “The hardest single part of building a software system is deciding precisely what to build” * Must show product to customer to get complete specification * Need for iterative feedback --- # Hopes for the Silver (Contd.) * Incremental development * Grow systems, don’t build them * Easy backtracking * Early prototypes * Great designers * “The difference between the great and the average approach an order of magnitude” * Gives hints as to how to grow great designers * Good design can be taught; great design cannot * Bottom line: Nurture great designers. --- # Some Methods for Dealing with Essential Difficulties * Complexity * Breakdown of system into manageable modules * Breakdown of development process into phases, tasks, and subtasks * Team programming * Changeability and Conformity * Change management * Configuration management * Invisibility * use of multiple models, for many views of system. --- # Future Work – Not part of research paper * In 2004 Daniel M. Berry again focussed on this topic that why there is no silver bullet. * His paper tries to get to the root of why any given new programming technique has not improved productivity very much. ## Bottom line (by Berry) >> I no longer get excited over any new language, development model, method, tool, or environment that is supposed to improve programming >> The most important work is addressing requirements, changes, and the psychology and sociology of programming. --- # Methodology * Method of those days (Brook’s Time): * discover some requirements, * code a little, * discover more requirements, * code a little more, * etc, until the coding was done; * test the whole thing, * discover bugs or new requirements, * code some more, etc. * Nowadays, we follow more systematic methods. However, the basic feelings have not changed. --- # Still Relevant? ## "Old" Software Engineering Articles .center[![Human Side](computing-human-side.png)] Software engineering is more about humans that about computers. --- # Conclusion * No one advance will give a 10x improvement * All the accidental difficulties can be solved. * Even if we remove all accidental difficulties still at its essence software development is difficult. * There is still a hope for silver for solving software related problems and thus for the betterment of later issues. * The real problem of software engineering is dealing with ever-changing requirements. * No model, method, artifact, or tool offered to date has succeeded to overcome this problem. * We are making incremental progress, and that is still all we can hope for. --- # References .center[![Brooks Berry](refs.png)] * Brooks, F., & Kugler, H. J. (1987). No silver bullet (pp. 1-14). April. * Berry, D. M. (2002, October). The inevitable pain of software development: Why there is no silver bullet. In International Workshop On Radical Innovations Of Software and Systems Engineering in the Future (pp. 50-74). Springer, Berlin, Heidelberg. --- class: center, middle .center[![Silver Bullet](silver-bullet.png)] # Thank you. Any questions?