class: center, middle .center[] # Software Testing Metraj Company<br/> Instructor: [S. M. Masoud Sadrnezhaad](https://twitter.com/smmsadrnezh) --- # Some Basic Terms - **Software** is more than just a program code. A program is an executable code, which serves some computational purpose. Software is considered to be **collection** of executable programming code, associated libraries and documentations. Software, when made for a specific requirement is called software product. - **Software engineering** is an engineering branch associated with development of software product using well-defined scientific principles, methods and procedures. The outcome of software engineering is an efficient and reliable software product. - In Software engineering and systems engineering, a **functional requirement** defines a function of a system or its component. A function is described as a set of inputs, the behavior, and outputs. - In systems engineering and requirements engineering, a **non-functional requirement** is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors. They are contrasted with functional requirements that define specific behavior or functions. ??? * Security * Logging * Performance * Configurability * Flexibility * Robustness * Reliability -> Mean Time Between/To Failures * Maintainability * Portability * Scalability --- # Need of Software Engineering The need of software engineering arises because of higher rate of change in user requirements and environment on which the software is working. * **Large software-** It is easier to build a wall than to a house or building, likewise, as the size of software become large engineering has to step to give it a scientific process. * **Scalability-** If the software process were not based on scientific and engineering concepts, it would be easier to re-create new software than to scale an existing one. * **Cost-** As hardware industry has shown its skills and huge manufacturing has lower down he price of computer and electronic hardware. But the cost of software remains high if proper process is not adapted. * **Dynamic Nature-** The always growing and adapting nature of software hugely depends upon the environment in which user works. If the nature of software is always changing, new enhancements need to be done in the existing one. This is where software engineering plays a good role. * **Quality Management-** Better process of software development provides better and quality software product. --- # Software Development Life Cycle .center[] --- # Software Development Life Cycle (Contd) #### Communication This is the first step where the user initiates the request for a desired software product. He contacts the service provider and tries to negotiate the terms. He submits his request to the service providing organization in writing. #### Requirement Gathering The team holds discussions with various stakeholders from problem domain and tries to bring out as much information as possible on their requirements. #### Feasibility Study After requirement gathering, the team comes up with a rough plan of software process. At this step the team analyzes if a software can be made to fulfill all requirements of the user and if there is any possibility of software being no more useful. ??? Communication RFP Requirement Gathering * Surveys -> Organization may conduct surveys among various stakeholders by querying about their expectation * Questionnaires * already has some software to perform certain operation * Domain Analysis -> expert people in the domain * Brainstorming * Prototyping * Observation Feasibility Study / Risk / Test * Technical * Legal * Operational -> fits in with the existing business environment * Schedule * Financial * Human Factor --- # Software Development Life Cycle (Contd) #### System Analysis The present system is investigated and its specifications documented. They should contain our understanding of HOW the present system works and WHAT it does. The project team analyzes the scope of the project and plans the schedule and resources accordingly. #### Software Design Next step is to bring down whole knowledge of requirements and analysis on the desk and design the software product. The inputs from users and information gathered in requirement gathering phase are the inputs of this step. The output of this step comes in the form of two designs; logical design and physical design. #### Coding This step is also known as programming phase. The implementation of software design starts in terms of writing program code in the suitable programming language and developing error-free executable programs efficiently. --- # Software Development Life Cycle (Contd) #### Testing Testing, verification and validation of the system just built. #### Integration Software may need to be integrated with the libraries, databases and other program(s). #### Implementation This means installing the software on user machines. At times, software needs post-installation configurations at user end. ??? 1 - Verification addresses the concern: "Are you building it right?" - Validation addresses the concern: "Are you building the right thing?" 2 - VerificationEnsures that the software system meets all the functionality. - Validation Ensures that the functionalities meet the intended behavior. 3 - Verification takes place first and includes the checking for documentation, code, etc. - Validation occurs after verification and mainly involves the checking of the overall product. 4 - Verification Done by developers. - Validation Done by testers. 5 - Verification It has static activities, as it includes collecting reviews, walkthroughs, and inspections to verify a software. - Validation It has dynamic activities, as it includes executing the software against the requirements. 6 - Verification It is an objective process and no subjective decision should be needed to verify a software. - Validation It is a subjective process and involves subjective decisions on how well a software works. --- # Software Development Life Cycle (Contd) #### Operation and Maintenance If required, the users are trained on, or aided with the documentation on how to operate the software and how to keep the software operational. The software is maintained timely by updating the code according to the changes taking place in user end environment or technology. #### Disposition As time elapses, the software may decline on the performance front. It may go completely obsolete or may need intense upgradation. ??? Corrective Maintenance - This includes modifications and updations done in order to correct or fix problems, which are either discovered by user or concluded by user error reports. Adaptive Maintenance - This includes modifications and updations applied to keep the software product up-to date and tuned to the ever changing world of technology and business environment. Perfective Maintenance - This includes modifications and updates done in order to keep the software usable over long period of time. It includes new features, new user requirements for refining the software and improve its reliability and performance. Preventive Maintenance - This includes modifications and updations to prevent future problems of the software. It aims to attend problems, which are not significant at this moment but may cause serious issues in future. Software Development Paradigm The software development paradigm helps developer to select a strategy to develop the software. Waterfall model is the simplest model of software development paradigm. It says the all the phases of SDLC will function one after another in linear manner. That is, when the first phase is finished then only the second phase will start and so on. --- # Software Testing - Testing is the **process of evaluating a system or its component(s)** with the intent to find whether it **satisfies the specified requirements or not**. - Testing is **executing a system** in order to **identify any gaps, errors, or missing requirements** in contrary to the **actual requirements**. ??? software testing, its types, methods, levels, and other related terminologies. --- # Who does Testing? (Contd) - It **depends on the process** and the associated **stakeholders** of the project(s). - In the IT industry, large companies have **a team** with responsibilities **to evaluate** the developed software in context of the given requirements. - Moreover, **developers** also conduct testing which is called **Unit Testing**. - In most cases, the following professionals are involved in testing a system within their respective capacities − - Software Tester, Software Quality Assurance Engineer, QA Analyst, etc. - Software Developer - Project Lead/Manager - End User ??? - Different companies have different designations for people who test the software on the basis of their experience and knowledge such as Software Tester, Software Quality Assurance Engineer, QA Analyst, etc. - It is not possible to test the software at any time during its cycle. The next two sections state when testing should be started and when to end it during the SDLC. --- # When to Start Testing? - An **early start** to testing **reduces the cost and time** to **rework** and **produce error-free software** that is delivered to the client. - However in Software Development Life Cycle (SDLC), testing can be **started from the Requirements Gathering phase** and **continued till the deployment of the software**. - It also **depends on the development model** that is being used. - For example, in the **Waterfall model**, formal testing is conducted in the **testing phase** - But in the **incremental model**, testing is performed at the end of **every increment/iteration** and the **whole application is tested at the end**. --- # When to Start Testing? (Contd) - Testing is done in **different forms** at **every phase of SDLC** - During the **requirement gathering phase**, the analysis and verification of **requirements** are also considered as testing. - **Reviewing the design** in the **design phase** with the intent to improve the design is also considered as testing. - Testing performed by **a developer** on **completion of the code** is also categorized as testing. --- # When to Stop Testing? - Testing **is a never-ending process** and **no one can claim** that a software is **100% tested**. - The following aspects are to be considered for stopping the testing process - Testing Deadlines - Completion of test case execution - Completion of **functional and code coverage** to a certain point - **Bug rate falls** below a certain level and **no high-priority bugs** are identified - Management decision --- # Myth Given below are some of the most common myths about software testing. - **Myth 1: Testing is Too Expensive** - Reality − There is a saying, **pay less for testing during software development or pay more for maintenance or correction later**. - **Early** testing **saves both time and cost** in many aspects. - However **reducing the cost without testing** may result in **improper design** of a software application rendering the product useless. - **Myth 2: Testing is Time-Consuming** - Reality − **During the SDLC phases**, testing is **never a time-consuming process**. - However diagnosing and fixing the **errors identified during proper testing is a time-consuming but productive activity**. --- # Myth (Contd) - **Myth 3: Only Fully Developed Products are Tested** - Reality − No doubt, testing **depends on the source code** - But reviewing requirements and developing test cases **is independent from the developed code**. - However **iterative or incremental** approach as a development life cycle model may **reduce the dependency** of testing **on the fully developed software**. - **Myth 4: Complete Testing is Possible** - Reality − It becomes an issue **when a client or tester thinks** that complete testing is possible. - Occurrence of complete testing is never possible. - There might be **some scenarios** that are **never executed** by the test team or the client during the software development life cycle and **may be executed once** the project **has been deployed**. --- # Myth (Contd) - **Myth 5: A Tested Software is Bug-Free** - Reality − This is a very **common myth** that the clients, project managers, and the management team believes in. - **No one can claim** with absolute certainty that a software application is **100% bug-free** - Even if a tester with **superb testing skills** has tested the application. - **Myth 6: Missed Defects are due to Testers** - Reality − It is not a correct approach to **blame testers for bugs** that remain in the application even after testing has been performed. - This myth **relates to Time, Cost, and Requirements changing Constraints**. - However the **test strategy** may also result in **bugs being missed by the testing team**. --- # Myth (Contd) - **Myth 7: Testers are Responsible for Quality of Product** - Reality − It is a very **common misinterpretation** that **only testers or the testing team** should be **responsible** for **product quality**. - Testers’ responsibilities include the **identification of bugs to the stakeholders** and then it is **their decision whether they will fix the bug or release the software**. - Releasing the software **at the time** puts **more pressure** on the **testers**, as **they will be blamed for any error**. --- # Myth (Contd) - **Myth 8: Test Automation should be used wherever possible to Reduce Time** - Reality − **Yes**, it is true that Test Automation reduces the testing time, but **it is not possible to start test automation at any time during software development**. - Test automaton should be started **when the software has been manually tested** and is **stable to some extent**. - Moreover, test automation **can never be used if requirements keep changing**. --- # Myth (Contd) - **Myth 9: Anyone can Test a Software Application** - Reality − People **outside the IT industry** think and even believe that **anyone can test** a software and testing is not a creative job. - However testers know very well that this is a myth. - Thinking **alternative scenarios**, try to **crash a software with the intent to explore potential bugs** is not possible for the person who developed it. - **Myth 10: A Tester's only Task is to Find Bugs** - Reality − **Finding bugs** in a software is the **task of the testers**, but at the same time, they are **domain experts** of the particular software. - Developers are only **responsible** for the **specific component or area** that is **assigned to them** - But testers **understand the overall workings of the software**, what the **dependencies** are, and the **impacts of one module on another module**. --- # Types of Testing ## Manual Testing - Manual testing includes testing a software manually, i.e., **without using any automated tool** or any script. - In this type, the tester takes over the **role of an end-user** and tests the software to **identify any unexpected behavior or bug**. - Testers **use test plans, test cases, or test scenarios** to test a software to ensure the **completeness** of testing. - Manual testing also includes **exploratory testing**, as testers **explore the software** to identify errors in it. --- # Types of Testing (Contd) ## Automation Testing - Automation testing, which is also known as **Test Automation**, is when the **tester writes scripts** and **uses another software** to test the product. - This process **involves automation of a manual process**. - Automation Testing is used to **re-run** the test **scenarios** that were performed **manually, quickly, and repeatedly**. - Apart from regression testing, automation testing is **also used** to test the application from **load, performance, and stress** point of view. - It **increases the test coverage, improves accuracy, and saves time and money** in comparison to manual testing. ??? Feeling of elapsed time. --- # Types of Testing (Contd) ## What to Automate? - It is **not possible** to **automate everything** in a software. - The areas at which a user can make transactions such as the login form or registration forms, - Any area where **large number of users can access the software simultaneously should be automated**. - Furthermore, **all GUI items, connections with databases, field validations**, etc. can be efficiently tested by automating the manual process. --- # Types of Testing (Contd) ## When to Automate? - Test Automation should be used by considering the **following aspects of a software** - **Large** and **critical** projects - Projects that **require** testing the **same areas frequently** - **Requirements not changing** frequently - Accessing the application for **load and performance** with **many virtual users** - **Stable** software with respect to **manual testing** --- # Types of Testing (Contd) ## How to Automate? - The **process** that can be used to automate the testing process - Identifying **areas within a software** for automation - Selection of **appropriate tool** for test automation - Writing **test scripts** - Development of **test suits** - **Execution** of scripts - Create result **reports** - Identify any **potential bug or performance issues** --- # Testing Methods ## Black-Box Testing - The technique of testing **without having any knowledge of the interior workings** of the application. - The tester is oblivious to the **system architecture** and does **not have access to the source code**. - Typically, while performing a black-box test, a tester will interact with the system's user interface by **providing inputs and examining outputs** without knowing how and where the inputs are worked upon. ??? oblivious = bi tavajoh --- # Testing Methods (Contd) ## White-Box Testing - **Detailed** investigation of **internal logic** and **structure of the code**. - White-box testing is also called **glass testing** or **open-box testing**. - In order to perform white-box testing on an application, a tester **needs to know the internal workings of the code**. - The tester needs to **have a look inside the source code** and find out **which unit/chunk of the code** is behaving inappropriately. --- # Testing Methods (Contd) ## Grey-Box Testing - Technique to test the application with having a **limited knowledge of the internal** workings of an application. - **Mastering the domain of a system** always gives the tester an edge over someone with limited domain knowledge. - In grey-box testing, the tester **has access to design documents and the database**. - Having this knowledge, a **tester can prepare better test data** and **test scenarios** while making a **test plan**. ??? - The test plan is a document that lists all the activities in a QA project, schedules them, defines the scope of the project, roles & responsibilities, risks, entry & exit criteria, test objective and anything else that you can think of. - The test plan is as I like to call a ‘super document’ that lists everything there is to know and need. - Test strategy is also a deliverable and also a document at that. - Test strategy outlines the testing approach and everything else that surrounds it. - Test strategy is only a subset of the test plan. - Example: Test plan gives the information of who is going to test at what time. For example, Module 1 is going to be tested by “X tester”. If tester Y replaces X for some reason, the test plan has to be updated. - On the contrary, a test strategy is going to have details like – “Individual modules are to be tested by test team members. “ In this case, it does not matter who is testing it- so it’s generic and the change in the team member does not have to be updated, keeping it static. - Test scenario is mostly a one-line definition of “What” we are going to test with respect to a certain feature. - Test suit is the list of all the test cases that have to be executed as a part of a test cycle or a regression phase - There is no logical grouping based on functionality. The order in which the constituent test cases get executed may or may not be important. --- # Testing Levels - There are **different levels** during the process of testing. - Levels of testing **include different methodologies** that can be used while conducting software testing. - The **main levels** of software testing are - **Functional** Testing - **Non-functional** Testing --- # Functional Testing - This is a type of **black-box** testing that is based on the **specifications of the software that is to be tested**. - The application is tested by **providing input** and then the **results** are **examined** that need to conform to the **functionality** it was **intended** for. - Functional testing of a software is conducted on a **complete, integrated system** to evaluate the system's compliance with its specified **requirements**. ??? compliance = entebaagh --- # Functional Testing (Contd) - There are **five steps** that are involved while testing an application **for functionality**. Steps | Description ------------- | ------------- I | The **determination of the functionality** that the intended application is meant to perform. II | The **creation of test data** based on the specifications of the application. III | The **output** based on the test data and the specifications of the application. IV | The writing of **test scenarios and the execution of test cases**. V | The **comparison of actual and expected results** based on the executed test cases. - An **effective testing practice** will make sure that the organization maintains the **strictest of standards** when it comes to **software quality**. --- # Unit Testing - This type of testing is performed **by developers before** the setup is **handed over to the testing team** to formally execute the test cases. - **Unit testing** is performed by the respective **developers** on the individual units of **source code assigned** areas. - The **developers** use **test data** that is **different** from the test data of the **quality assurance team**. - The goal of **unit testing** is to **isolate each part** of the program and show that individual parts are correct in terms of **requirements and functionality**. --- # Limitations of Unit Testing - Testing **cannot catch** each and every bug in an application. - It is impossible to evaluate **every execution path** in every software application. - There is a **limit to the number of scenarios and test data** that a developer can use to verify a source code. - After having exhausted all the options, there is **no choice but to stop unit testing** and **merge the code segment** with other units. --- # Integration Testing - Integration testing is defined as the testing of **combined parts of an application** to determine if they function correctly. - Integration testing can be done in **two ways: Bottom-up integration testing and Top-down integration testing**. - In a comprehensive software development environment, **bottom-up** testing is usually done **first, followed by top-down** testing. - The **process conclude** with **multiple tests** of the **complete application**, preferably in scenarios designed to mimic actual situations. ??? concludes = be payan miresad mimic = taghlid --- # System Testing - System testing tests the system **as a whole**. - Once all the **components are integrated**, the application as a whole is tested rigorously to see that it meets the specified **Quality Standards**. - This type of testing is performed by a **specialized testing team**. - System testing is **important** because of the following reasons − - System testing is the **first step** in the Software Development Life Cycle, where the application is tested **as a whole**. - The application is tested thoroughly to verify that it **meets the functional and technical** specifications. - The application is tested in an environment that is **very close to the production environment** where the application will be deployed. - System testing enables us to test, verify, and validate both the **business requirements as well as the application architecture**. ??? rigorously = daghigh, shadid --- # Regression Testing - Whenever a **change** in a software application is made, it is quite possible that **other areas within the application have been affected by this change**. - Regression testing is performed to verify that a **fixed bug hasn't resulted in another functionality or business rule violation**. - The intent of regression testing is to **ensure that a change, such as a bug fix should not result in another fault** being uncovered in the application. --- # Regression Testing (Contd) - Regression testing is **important** because of the following reasons − - **Minimize the gaps in testing** when an application with changes made has to be tested. - Testing the **new changes** to verify that the changes made did **not affect any other area** of the application. - **Mitigates risks** when regression testing is performed on the application. - Test **coverage is increased** without compromising timelines. - Increase **speed to market** the product. ??? Mitigates = kaahesh - Solve Problems, Not Symptoms - Root cause analysis - http://www.spartina.com/items/7812-solve-problems-not-symptoms - https://fa.wikipedia.org/wiki/%D8%AA%D8%AD%D9%84%DB%8C%D9%84_%D8%B1%DB%8C%D8%B4%D9%87%E2%80%8C%D8%A7%DB%8C_%D9%88%D9%82%D8%A7%DB%8C%D8%B9 - eg - You go into the doctor with a headache. You are having some trouble with your vision. You are running a fever. - If these are the things you tell your doctor, and she is symptom oriented, she might say: Here is an aspirin. Wear these glasses. Put this cold cloth over your forehead. - Problem solved? Heck no! --- # Acceptance Testing - This is arguably the **most important type** of testing. - As it is conducted by the **Quality Assurance Team** who will gauge whether the application meets the **intended specifications** and satisfies the **client’s requirement**. - Acceptance Testing is the **fourth and last level** of software testing performed **after System Testing** and **before** making the system **available for actual use**. .center[] ??? gauge = andaaze giri --- # Acceptance Testing (Contd) - Acceptance tests are **not only intended to point out** simple spelling mistakes, cosmetic errors, or interface gaps. - But also to point out **any bugs in the application** that will result in **system crashes or major errors** in the application. - By performing acceptance tests on an application, the testing team will reduce **how the application will perform in production**. - There are also **legal and contractual requirements** for **acceptance of the system**. - **Internal Acceptance Testing** (Also known as Alpha Testing) is performed by **members of the organization** that developed the software but who are **not directly involved** in the project (Development or Testing). - **External Acceptance Testing** (Also known as Beta Testing) is performed by people who are **not employees** of the organization that developed the software. --- # Alpha Testing - Unit testing, integration testing and system testing when **combined together** is known as alpha testing. - During this phase, the following aspects will be tested in the application − - Spelling Mistakes - Broken Links - Cloudy Directions - The Application will be tested on machines with the **lowest specification** to **test loading times** and any **latency problems**. ??? gauge = andaze giri --- # Beta Testing - This test is performed **after alpha testing** has been successfully performed. - In beta testing, a **sample of the intended audience** tests the application. - Beta testing is also known as **pre-release testing**. - Beta test versions of software are **ideally** distributed to a **wide audience on the Web**, **partly** to give the program a **"real-world" test** and **partly** to provide a **preview of the next release**. --- # Beta Testing (Contd) - In this phase, the audience will be testing the following − - Users will **install, run** the application and **send their feedback** to the project team. - **Typographical errors**, confusing **application flow**, and even **crashes**. - Getting the **feedback**, the project team can **fix the problems before releasing** the software **to the actual users**. - The **more issues you fix** that solve real user problems, the **higher the quality** of your application will be. - Having a higher-quality application when you release it to the general public will increase **customer satisfaction**. --- # Fitment of testing levels in the SDLC .center[] --- # Non-Functional Testing - This section is based upon testing an application from its **non-functional attributes**. - Non-functional testing involves testing a software from the requirements which are **nonfunctional in nature** but important such as **performance, security, user interface, etc**. --- # Performance Testing - It is mostly used to **identify any bottlenecks or performance issues** rather than finding bugs in a software. - There are different causes that contribute in **lowering the performance** of a software − - Network delay - Client-side processing - Database transaction processing - Load balancing between servers - Data rendering --- # Performance Testing (Contd) - Performance testing is considered as one of the **important** and mandatory testing type in terms of the following aspects − - Speed (i.e. Response Time, data rendering and accessing) - Capacity - Stability - Scalability - Performance testing can be either **qualitative or quantitative** and can be divided into **different sub-types** such as **Load testing and Stress testing**. --- # Load Testing - It is a process of **testing the behavior** of a software by applying **maximum load** in terms of **software accessing** and **manipulating large input data**. - It can be done at **both normal and peak load conditions**. - This type of testing identifies the **maximum capacity** of software and its behavior at **peak time**. - Most of the time, load testing is performed with the help of **automated tools** such as Load Runner, AppLoader, IBM Rational Performance Tester, Apache JMeter, Silk Performer, Visual Studio Load Test, etc. - **Virtual users (VUsers)** are defined in the automated testing tool and the **script is executed** to **verify the load testing** for the software. - The **number of users** can be increased or decreased **concurrently or incrementally** based upon the requirements. --- # Soak testing - Also known as **Endurance Testing**. - This falls under **load testing**. - Soak testing involves testing a system with a **typical production load**, over a **continuous availability period**, to validate system behavior under **production use**. - In software testing, a system may behave exactly **as expected** when **tested for one hour**. - However, when it is **tested for three hours**, problems such as **memory leaks** cause the system to **fail or behave unexpectedly**. - Soak tests are used primarily to check the **reaction of a subject** under test under a **possible simulated environment** for a **given duration** and for a **given threshold**. - In **electronics**, soak testing may involve testing a system up to or above its maximum ratings for a **long period of time**. Some companies may soak test a product for a period of **many months**, while also applying **external stresses** such as **elevated temperatures**. ??? Soak = khis khordan --- # Stress Testing - Stress testing includes testing the **behavior of a software under abnormal conditions**. - For example, it may include taking away some resources or applying a load **beyond the actual load limit**. - The aim of stress testing is to test the software by **applying the load** to the system and **taking over the resources** used by the software to **identify the breaking point**. - This testing can be performed by testing different scenarios such as − - Shutdown or restart of network ports randomly - Turning the database on or off - Running different processes that consume resources such as CPU, memory, server, etc. --- # Usability Testing - Usability testing is a **black-box** technique and is used to identify any error(s) and improvements in the software by **observing the users** through their **usage and operation**. - Usability can be defined in terms of five factors, i.e. **efficiency of use, learn-ability, memory-ability, errors/safety, and satisfaction**. - Usability is the **quality requirement** that can be **measured as the outcome of interactions with a computer system**. - This requirement can be fulfilled and the **end-user will be satisfied** if the intended goals are achieved **effectively** with the use of **proper resources**. - **User-friendly** system should fulfill the following five goals, i.e., **easy to Learn, easy to remember, efficient to use, satisfactory to use, and easy to understand**. - **Rather than** showing users a rough **draft** and **asking, "Do you understand this?"**, usability testing involves **watching people** trying to use something for its intended purpose. ??? - HCI - user-centered - Usability = External Quality. End-user. - learn-ability = learning-curve - efficiency = eg. peyvast - eg. blocking points in users navigation - eg. apply forms - Systematic observation under Controlled conditions - to determine how well people can use the product - eg. When testing instructions for **assembling a toy**, the test subjects should be given the instructions and a box of parts and, - rather than being asked to **comment on the parts and materials**, they are asked to **put the toy together**. - Instruction phrasing, illustration quality, and the toy's design all affect the assembly process. - A/B Testing is a randomized experiment with two variants, A and B. - As the name implies, two versions (A and B) are compared, which are identical except for one variation that might affect a user's behavior. --- # Accessibility Testing - A type of Software Testing performed to ensure that the application being tested is **usable by people with disabilities** like **hearing, color blindness, old age** and other disadvantaged groups. - It is a subset of **Usability Testing**. - People with disabilities **use assistive technology** which helps them in operating a software product. - **Examples** of such software are: - **Speech Recognition** Software - It will convert the **spoken word to text** , which serves as input to the computer. - **Screen reader** Software - Used to **read out the text** that is displayed on the screen - **Screen Magnification** Software- Used to **enlarge the monitor** and make reading easy for vision-impaired users. - **Special keyboard** made for the users for **easy typing** who have motor control difficulties ??? - motor control = controle harekati - Which Disabilities to Support? - How to do Accessibility Testing? - Accessibility Testing Tools --- # Accessibility Testing (Contd) - **Reason 1:** Cater to **market for Disabled People**. - About **20% of the population has disability issues. - **1 in 10 people** have a sever disability - Disabilities include **blindness, deaf, handicapped,** or any disorders in the body. - A software product can cater to this **big market**, if it's **made disabled friendly**. - Accessibility issues in software **can be resolved** if Accessibility Testing is made part of normal software testing life cycle. - **Reason 2:** Abide by Accessibility Legislations - **Reason 3:** Avoid Potential Law Suits ??? cater = tahiye handicapped = malooliyat --- # UI vs Usability Testing - UI testing involves testing the **Graphical User Interface** of the Software. - UI testing ensures that the **GUI functions** according to the **requirements and tested in terms of color, alignment, size, and other properties**. - Usability testing ensures a good and **user-friendly** GUI that can be **easily** handled. - UI testing can be considered as a **sub-part of usability** testing. --- # Security Testing - Security testing involves testing a software in order to **identify any flaws and gaps** from security and **vulnerability** point of view. - Listed below are the **main aspects** that security testing should ensure - Confidentiality - Integrity - Authentication - Availability - Authorization - Non-repudiation - Software is secure against known and unknown **vulnerabilities** - **Software data** is secure ??? - vaalnerebility - confidentiality is a set of rules that limits access to information - integrity is the assurance that the information is trustworthy and accurate - availability is a guarantee of reliable access to the information by authorized people. - integrity = yekparchegi, mahdoodiyat ha rooye dade ha maanand daamane va ... - Authentication confirms your **identity** to grant **access to the system**. - Authorization determines whether you are authorized to access the **resources**. - Non-repudiation = eg. a secure area may use a **key card** access system. Here, non-repudiation would be violated if key cards were **shared** or if **lost** and **stolen** cards were **not immediately reported**. - Non-repudiation = eg. the owner of a computer account must not allow others to use it, such as by **giving away their password**, and a **policy** should be implemented to enforce this. --- # Security Testing (Contd) - Listed below are the **main aspects** that security testing should ensure (Contd) - Software is according to all **security regulations** - **Input** checking and validation - **SQL insertion** attacks - **Injection** flaws - **Session** management issues - **Cross-site scripting** attacks - **Buffer overflows** vulnerabilities - **Directory traversal** attacks ??? - regulations = ayin naameha - XSS -> JS - XSS = Use website as a mean to attack its **users**. - Cross-site scripting (XSS) is a type of computer security vulnerability typically found in **web applications**. - XSS enables attackers to inject client-side scripts into web pages viewed by other users. - eg. rm -rf / - Buffer overflows => too memory ye chizi benevisi ke length nadare - Directory traversal => Access files in unintended directories. very common. - Directory traversal => http://example.com/../../etc/passwd ... --- # Portability Testing - Portability testing includes testing a software with the aim to ensure its **reusability** and that it can be **moved** from another software as well. - Following are the **strategies** that can be used for portability testing − - Transferring an installed software **from one computer to another**. - Building executable to run the software on **different platforms**. - Portability testing can be considered as one of the **sub-parts of system testing** - As this testing type includes **overall** testing of a software with respect to its usage over **different environments**. - Computer **hardware, operating systems, and browsers** are the major focus of portability testing. --- # Portability Testing (Contd) - Some of the **pre-conditions** for portability testing are as follows − - Software should be **designed and coded**, keeping in mind the **portability requirements**. - **Unit testing** has been performed on the associated components. - **Integration testing** has been performed. - **Test environment** has been **established**. ??? Test environment / Production environment --- # Compliance Testing - Also known as **conformance testing, regulation testing, standards testing**. - A type of testing to determine the **compliance** of a system with **internal or external standards**. - Compliance Testing is performed to maintain and **validate the compliant** state **for the life of the software**. - **Internal standards** could be standards set by the **company itself**. For example, a web application development company might set the standard that **all web pages must be responsive**. - **External standards** could be standards set **outside of the company**. For example, Health Insurance Portability and Accountability Act (HIPAA) has set **regulations for the healthcare industry**. - Every industry has a **regulatory and compliance board** that **protects the end users**. ??? compliant = saazgar conformance = entebagh compliance board = heyate daavari retained = hefz shavand --- # Compliance Testing (Contd) - Compliance testing could also be **done by an external organization**. - This normally results in some sort of **compliance certification**. - Checklists for Compliance Testing: - **Professionals**, who are knowledgeable and experienced, who understand the compliance must **be retained**. - Understanding the **risks and impacts of being non-compliant** - **Document the processes** and follow them - Perform an **internal audit** and follow with an **action plan to fix** the issues ??? audit = momayezi --- # Recovery testing - Recovery testing is the activity of testing how well an application is able to **recover from crashes, hardware failures and other similar problems**. - Recovery testing is the **forced failure** of the software in a variety of ways to verify that **recovery is properly performed**. - **Type or extent of recovery** is specified in the **requirement specifications**. --- # Recovery testing (Contd) - **Examples** of recovery testing: - While an **application is running**, suddenly **restart** the computer, and afterwards check the validness of the application's **data integrity**. - While an application is **receiving data from a network**, **unplug the connecting cable**. - After some time, **plug the cable back** in and analyze the application's ability to continue **receiving data** from the **point at which the network connection disappeared**. - **Restart** the system while a browser **has a definite number of sessions**. - Afterwards, check that the **browser** is able to **recover all of them**. --- # Volume Testing - Testing a software application with a **certain amount of data**. - This amount can, in generic terms, be the **database size** or it could also be the **size of an interface file** that is the subject of volume testing. - For example, if you want to volume test your application with a specific database size, you will **expand your database to that size** and **then test the application's performance on it**. - Another example could be when there is a requirement for your application to **interact with an interface file** (could be any file such as .dat, .xml); this interaction could be **reading and/or writing on to/from the file**. - You will **create a sample file** of the **size you want** and then **test the application's functionality** with that file in order **to test the performance**. --- # Scalability testing - Determine the **user limit** for the web application and **ensure end user experience, under a high load, is not compromised**. - One example is if a web page **can be accessed** in a **timely fashion** with a **limited delay in response**. - Another goal is to check if the server can cope i.e. **Will the server crash if it is under a heavy load?** - Dependent on the application that is being tested, different parameters are tested. - If a webpage is being tested, the **highest possible number** of **simultaneous users** would be tested. ??? compromised = mosaalehe --- # Scalability testing - When creating a new application, it is **difficult to accurately predict** the **number of users** in 1, 2 or even 5 years. - Although an estimate can be made, it is not a **definite number**. - An issue with an **increasing number of users** is that it can create **new areas of failure**. - For example, if you have **100,000 new visitors**, it’s not just **access to the application** that could be a problem; you might also experience **issues with the database** where you need to store all the data of these new customers. --- # Scalability testing ## Increment loads - When creating a scalability test, it is important to **scale up in increments**. - These **steps** can be split into **small, medium and high loads**. - We must scale up in increments as **each stage** tests a **different aspect**. - Small loads ensure the system functions as it should on a **basic level**. - Medium loads test the system can function **at its expected level**. - High loads test the system can **cope with a high load**. --- # Scalability testing ## Test environment - The **environment** should be **constant** throughout testing in order to provide **accurate and reliable results**. - If the testing is a **success**, we should see a **proportional change in performance**. - For example, if we **double the users** on the system, we **should see** a **drop in performance of 50%**. - Alternatively, if **measuring system statistics** such as **memory or CPU usage over time**, this may have a different **graph that is not proportional** as users are not being plotted on either axis. --- # Scalability testing ## Unproportional outcome .center[] --- # Documentation Testing - Any **written or pictorial** information describing, defining, specifying, reporting, or certifying activities, requirements, procedures, or results. - Documentation is **as important** to a product’s success as the **product itself**. - If the documentation is **poor, non-existent, or wrong**, it **reflects on the quality of the product** and the vendor. - As per the IEEE Documentation describing **plans** for, or **results** of, the **testing of a system or component** - Types include test case specification, test incident report, test log, test plan, test procedure, test report. - The documentation can be tested in a number of **different ways** to many different **degrees of complexity**. - These range from running the documents through a **spelling and grammar checking device**, to **manually reviewing** the documentation to remove **any ambiguity or inconsistency**. --- # Testing Documentation - Testing documentation involves the documentation of **artifacts** that should be **developed before or during the testing** of Software. - Documentation for software testing helps in **estimating** the **testing effort required, test coverage, requirement tracking/tracing,** etc. - Some of the **commonly used documented** artifacts related to **software testing** such as − - Test Plan - Test Scenario - Test Case - Traceability Matrix --- # Test Plan - A test plan outlines **the strategy** that will be used to **test an application**, the **resources** that will be used, the **test environment** in which testing will be performed, and the **limitations of the testing** and the **schedule of testing activities**. - Typically the **Quality Assurance Team Lead** will be responsible for writing a Test Plan. - A test plan **includes** the following − - **Introduction** to the Test Plan document - **Assumptions** while testing the application - List of **test cases** included in testing the application - List of **features to be tested** - What sort of **approach** to use while testing the software - List of **deliverables** that need **to be tested** - The **resources allocated** for testing the application - Any **risks** involved during the **testing process** - A **schedule** of tasks and **milestones** to be achieved ??? deliverable = tahvil daadani --- # Test Scenario - Test Scenario notifies **what area in the application** will be tested. - Test scenarios are used to ensure that **all process flows** are tested **from end to end**. - A **particular area** of an application can have as little as **one test scenario to a few hundred scenarios**. - Depending on the **magnitude and complexity** of the application. - The terms 'test scenario' and 'test cases' are used **interchangeably**. - However a **test scenario** has **several steps**, whereas a **test case** has a **single step**. - Test scenarios **include several test cases** and the **sequence** that they should be executed. - Apart from this, **each test is dependent** on the output from the **previous test**. ??? Alternative flows -> SRS --- # Test Case - Test cases involve a **set of steps, conditions, and inputs** that can be used while performing testing tasks. - The main intent of this activity is to **ensure whether a software passes or fails** in terms of its **functionality and other aspects**. - There are many types of test cases such as **functional, negative, error, logical test cases, physical test cases, UI test cases**, etc. - Test cases are written to **keep track of the testing coverage** of a software. ??? - Negative = validated against the invalid input data. - Positive = test valid inputs to verify the software is doing what it’s supposed to do. --- # Test Case (Contd) - Generally, there are **no formal templates** that can be used during test case writing. - However, the following components are always available and **included in every test case** − - Test case ID - Product module - Product version - Revision history - Purpose - Assumptions - Pre-conditions - Steps - Expected outcome - Actual outcome - Post-conditions - Many test cases can be **derived from a single test scenario**. - In addition, sometimes **multiple test cases** are written for a single software which are **collectively known as test suites**. ??? - Test Scenario = A document specifying a sequence of actions for the execution of a test. (IEEE) - Test scenario defines a story - Test suite = Series of Test Cases --- # Traceability Matrix - Traceability Matrix (also known as Requirement Traceability Matrix - RTM) is a **table** that is used to **trace the requirements** during the Software Development Life Cycle. - It can be used for **forward tracing** (i.e. from Requirements to Design or Coding) or **backward** (i.e. from Coding to Requirements). - There are many **user-defined templates** for RTM. .center[] --- # Traceability Matrix (Contd) - Each **requirement** in the RTM document is **linked** with its **associated test case** - So that testing can be done **as per** the mentioned requirements. - **Bug ID** is also included and **associated test** linked with its **associated requirements** and **test case**. - The main **goals** for this matrix are − - Make sure the software is developed as per the **mentioned requirements**. - Helps in finding the **root cause of any bug**. - Helps in **tracing the developed documents** during **different phases of SDLC**. --- # Test Pyramid - The test pyramid is a **way of thinking** about different **kinds of automated tests** should be used to create a balanced portfolio. - Its essential point is that you should have **many more low-level UnitTests than high level** BroadStackTests running through a GUI. .center[] ??? sang kaaghaz gheychi --- # Test Pyramid (Contd) - **Test automation** meant tests that **drove an application** through its **user-interface**. - Such **tools** would often provide the **facility to record an interaction** with the application and then allow you to **play back** that interaction, **checking** that the application returned the **same results**. - Such an approach works well **initially**. - It's **easy to record tests**, and the tests can be recorded by people with **no knowledge of programming**. - But this kind of approach quickly runs into trouble, becoming an **ice-cream cone**. - Testing through the UI like this is **slow**, increasing build times. - Often it requires installed **licences** for the **test automation software**, which means it can only be done on **particular machines**. - Usually these cannot easily be run in a **"headless" mode**, monitored by scripts to put in a proper **deployment pipeline**. ??? - drove = drive - ice-cream cone = bastani ghifi - ketabe CD --- # Test Pyramid (Contd) - Most importantly such tests are **very brittle**. - An **enhancement to the system** can easily end up **breaking lots of such tests**, which then have to be **re-recorded**. - You can reduce this problem by **abandoning record-playback tools**, but that makes the **tests harder to write**. - Even with good practices on writing them, **end-to-end tests** are more prone to **non-determinism problems**, which can undermine trust in them. - In short, tests that run end-to-end through the UI are: **brittle, expensive to write, and time consuming to run**. - So the **pyramid argues** that you should do much **more** automated testing through **unit tests** than you should through traditional **GUI based** testing. ??? prone = likely --- # Test Pyramid (Contd) - The pyramid also argues for an **intermediate layer** of tests that act through a **service layer** of an application. - These can provide many of the **advantages of end-to-end tests** but **avoid** many of the **complexities of dealing with UI frameworks**. - In **web applications** this would correspond to testing through an **API layer** while the **top UI part of the pyramid** would correspond to tests using something like **Selenium** or Sahi. - The test pyramid comes up a lot in **Agile testing circles**. - While its core message is sound, there is much more to say about building a **well-balanced** test portfolio. - A common problem is that **teams conflate** the concepts of **end-to-end tests, UI tests, and customer facing tests**. - These are all orthogonal characteristics. - For example a **rich javascript UI** should have most of its **UI behavior tested** with **javascript unit tests** using something like Jasmine. ??? conflate = khalt orthogonal = statistically independent. --- # Test Pyramid (Contd) - **High-level tests** are there as a **second line of test defense**. - If you get a **failure in a high level test**, not just do you have a bug in your functional code, you also have a **missing or incorrect unit test**. - **Before fixing** a bug exposed by a **high level test**, you should replicate the bug with a **unit test**. - Then the **unit test ensures** the **bug stays dead**. ## Variations - Using the software testing **pyramid** as a **visual way** to represent where you should be **focusing your testing effort**, - And often switch between using a **cloud** or an **Eye of Providence** to represent **human manual exploratory testing** at the top of the pyramid that you should use to supplement and test your automated tests. ??? replicate = copy supplement = mokamel --- # Test Pyramid (Contd) ## Variations (Contd) .center[] --- # Test Pyramid (Contd) ## Variations (Contd) .center[] --- # Ice-Cream Cones Anti-pattern - Organizations fall into the trap of creating **‘inverted’ pyramids** of software testing. .center[] --- # Test Coverage - People asking **what value of test coverage** (also called code coverage) they should aim for, or stating their coverage levels with pride. - Such statements **miss the point**. - Test coverage is a useful tool for **finding untested parts** of a codebase. - Test coverage is of little use as a **numeric statement** of **how good** your tests are. - I've heard of places that may say things like "you **can't go into production** with **less than 87% coverage**". - I've heard some people say that you should **use things like TDD** and **must get 100% coverage**. - A wise man once said: > > I expect a high level of coverage. Sometimes managers require one. There's a subtle difference. > > > > ― Brian Marick ??? Brian Marick = author of "The craft of software testing" craft = honar, mahaarat subtle = riz, zarif --- # Test Coverage (Contd) - If you make a **certain level of coverage a target**, people will **try to attain it**. - The trouble is that high coverage numbers are **too easy to reach** with **low quality testing**. - At the most absurd level you have **Assertion Free Testing**. - But even without that you get **lots of tests** looking for things that **rarely go wrong distracting** you from testing the things that **really matter**. - Like most aspects of programming, testing requires **thoughtfulness**. - **TDD** is a very useful, but certainly **not sufficient**, tool to help you get good tests. - If you are **testing thoughtfully** and well, I would expect a coverage percentage in the **upper 80s or 90s**. - I would be **suspicious** of anything **like 100%** - it would smell of someone **writing tests to make the coverage numbers happy**, but not thinking about what they are doing. ??? AssertionFreeTesting = fowler eg. thoughtfulness = andishe --- # Test Coverage (Contd) - The reason, of course, why people focus on coverage numbers is because they **want to know** if they are **testing enough**. - Certainly low coverage numbers, say **below half**, are a **sign of trouble**. - But **high numbers don't necessarily mean much**, and lead to ignorance-promoting dashboards. - **Sufficiency of testing** is much **more complicated** attribute than coverage can answer. - I would say you are doing **enough testing** if the following is true: - You **rarely** get **bugs that escape into production**, and - You are **rarely hesitant to change** some code for fear it **will cause production bugs**. --- # Test Coverage (Contd) - Can you **test too much**? Sure you can. - You are testing too much if you can **remove tests** while still **having enough**. - But this is a **difficult thing to sense**. - One sign you are testing too much is if your tests are **slowing you down**. - If it seems like a **simple change to code** causes **excessively long changes to tests**, that's a sign that there's a **problem with the tests**. - This may not be so much that you are testing too many things, but that you have **duplication in your tests**. --- # Test Coverage (Contd) - Some people think that you **have too many tests** if they take too **long to run**. - I'm **less convinced** by this argument. - You can always **move slow tests to a later stage** in your deployment pipeline, or even **pull them out of the pipeline** and **run them periodically**. - Doing these things will **slow down** the **feedback from those tests**, but that's part of the **trade-off of build times versus test confidence**. - So what is the **value of coverage** analysis again? Well it **helps you find** which bits of your code **aren't being tested**. - It's worth **running coverage tools** every so **often** and **looking at** these bits of **untested code**. - Do they **worry you** that they **aren't being tested**? > > If a part of your test suite is weak in a way that **coverage can detect**, it's likely **also weak** in a way coverage **can't detect**. > > > > ― Brian Marick ??? http://www.developertesting.com/archives/month200705/20070504-000425.html --- # Test Coverage (Contd) ## Testivus On Test Coverage Early one morning, a programmer asked the great master: > > “I am ready to write some unit tests. What code coverage should I aim for?” The great master replied: > > “Don’t worry about coverage, just write some good tests.” The programmer smiled, bowed, and left. ... ??? bowed = tazim. ehteram. --- # Test Coverage (Contd) ## Testivus On Test Coverage (Contd) Later that day, a second programmer asked the same question. The great master pointed at a pot of boiling water and said: > > “How many grains of rice should put in that pot?” The programmer, looking puzzled, replied: > > “How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.” > > “Exactly,” said the great master. The second programmer smiled, bowed, and left. ... --- # Test Coverage (Contd) ## Testivus On Test Coverage (Contd) Toward the end of the day, a third programmer came and asked the same question about code coverage. > > “Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table. The third programmer smiled, bowed, and left. ... After this last reply, a young apprentice approached the great master: > > “Great master, today I overheard you answer the same question about code coverage with three different answers. Why?” The great master stood up from his chair: > > “Come get some fresh tea with me and let’s talk about it.” ... ??? stern = sakht o mohkam pounding = koobid fist = mosht apprentice = shaagerd --- # Test Coverage (Contd) ## Testivus On Test Coverage (Contd) After they filled their cups with smoking hot green tea, the great master began to answer: > > “The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.” > > > > “The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.” ... --- # Test Coverage (Contd) ## Testivus On Test Coverage (Contd) “I see,” said the young apprentice, “but if there is no single simple answer, then why did you answer the third programmer ‘Eighty percent and no less’?” The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down. > > “The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.” The young apprentice and the grizzled great master finished drinking their tea in contemplative silence. ??? grizzled = mast contemplative = andishmandane --- # Test Double - When you're doing unit testing, you're focusing on **one element** of the software **at a time** -hence the common term **unit testing**. - The problem is that to make a **single unit work**, you often **need other units**. - Test Double is a **generic term** for any case where you **replace a production object for testing purposes**. - There are **various kinds** of double that Gerard lists: - **Dummy objects** are **passed** around but **never actually used**. - Usually they are just **used to fill parameter lists**. - **Fake objects** actually have **working implementations**. - But usually take **some shortcut** which makes them **not suitable for production**. --- # Test Double (Contd) - There are **various kinds** of double that Gerard lists: (Contd) - **Stubs** provide canned **answers to calls** made **during the test**. - Usually **not responding** at all to anything **outside** what's programmed in for the **test**. - **Spies** are stubs that also **record some information** based on **how they were called**. - One form of this might be **an email service** that **records how many messages it was sent**. - **Mocks** are **pre-programmed** with expectations which form a specification of the **calls** they are **expected to receive**. - They can **throw an exception** if they **receive a call** they **don't expect** - And are checked **during verification** to ensure they **got all the calls** they were **expecting**. --- # QA in Production - Gathering **operational data** about a system is **common practice**. - particularly **metrics that indicate system load and performance** such as CPU and memory usage. - This data has been used for years to **help** teams who **support** a system **learn when an outage** is happening or imminent. - When things become slow, a **code profiler** might be enabled in order to **determine which part of the system is causing a bottleneck**, for example **a slow-running database query**. - A recent trend that **combines** the **meticulousness of** this **traditional operational monitoring** with a much **broader view of the quality** of a system. - While **operational data** is an **essential** part of **supporting a system**, - It is also valuable to **gather data** that **helps provide a picture** of whether the **system as a whole is behaving as expected**. ??? log aggregation tools imminent = gharibolvoghoo - 20/80 - Microsoft noted that by fixing the top 20% of the most-reported bugs, 80% of the related errors and crashes in a given system would be eliminated. - 20 percent of the code has 80 percent of the errors. Find them, fix them! - 80% of a certain piece of software can be written in 20% of the total allocated time. - Conversely, the hardest 20% of the code takes 80% of the time. meticulousness = daghigh boodan --- # QA in Production (Contd) - **“QA in production”** as an **approach** where teams pay closer **attention to** the **behaviour** of their **production systems** in order to improve the **overall quality** of the **function these systems serve**. - Things **always go wrong in production**, but this **doesn’t** have to be a **bad thing**. - It’s an **opportunity** to **learn about your system** and the **real world** with which it interacts. - With the right **production monitoring tools** and a **good Continuous Delivery pipeline**, you can build a set of **feedback mechanisms** that help you **find out about issues** as they happen and **ship fixes quickly**. - Adopting **production QA** (Quality Assurance) practices can help you gain a **richer understanding of the real issues** your system faces and **learn new ways to improve its quality**. ??? Test Environ -> Done Tools + Pipeline --- # QA in Production (Contd) .center[] --- # QA in Production (Contd) ## Gathering production data - There’s **a lot of data** you could be **gathering about your system**. - I’ve found it very helpful to think about what is **critical to the success of a system** and to let this **guide my efforts**. - When I talk about **success** in this context, I’m referring to the kind of things **that pay the bills**. - It doesn’t matter if your system serves **thousands of requests per second** if it **fails** to provide a **service that your customers pay for or rely on**. --- # QA in Production (Contd) ## Critical success indicators - At Tes, I’ve spent some time working on a system teachers use to apply for jobs. - One metric that our team identified as critical is whether a teacher’s job application actually reaches the school for which they apply to work. - Email is the primary method via which the system notifies schools of new job applications. - We use a third-party service to send emails, which means that this sits outside the boundaries of the system we’re working on. --- # QA in Production (Contd) ## Critical success indicators (Contd) - As we’ve come to learn, many things can go wrong when trying to send an email. - Email addresses can be invalid even when they seem valid, mailboxes get full and people have out-of-office messages which can make it seem like they didn’t receive an email even when they did. - These kinds of real-world complications are almost impossible to predict and hard to test. - What we did instead was to learn from what happened in production. --- # QA in Production (Contd) ## Critical success indicators (Contd) - Here’s a summary of how our understanding of, and reaction to, the quality of the system evolved: - We started counting the number of job applications submitted and the number of emails we sent (by sending simple metrics to a metrics server). - We set up an alert so that we got notified whenever these numbers were out of sync. - We set up another alert that let us know if our email-sending microservice was unable to process a request to send an email. - It turned out that this happened frequently. - Our alerts were firing, but it wasn’t clear why. - We took a look at the logs, but couldn’t find any useful information, so we set about improving our logging. - In particular, we logged more detail about the errors we were receiving from our third-party email provider. --- # QA in Production (Contd) ## Critical success indicators (Contd) - From the logs, we learned about different kinds of responses - some meant the person probably got the email despite the error (e.g. an out-of-office reply), some (like a DNS issue) could be temporary and some errors meant the email would never be delivered. - Some of the documentation and error messages were confusing, so we used examples of unsent emails to speak to the schools to find out which emails did actually get received and which did not. - We noticed that our email-sending code was too strict regarding the errors it got back, so we relaxed it a bit. - (We stopped logging errors when we knew the error response meant the school is likely to have received the mail.) --- # QA in Production (Contd) ## Critical success indicators (Contd) - We noticed that quite a few problems were due to incorrectly captured email addresses. - In these cases, we looked at the logs and talked to our customer services team. - “Hi. We tried to send an email to example@example.com, but it couldn’t go through. Can you please check with the school whether the email address is correct?” - For a while, the customer services team would let us know when an email address had been fixed, and we’d manually trigger an email resend using an endpoint we’d built on our web service. - After a while, we started automatically resending emails when email addresses were updated. - (We listen for updates to job information on a message queue.) --- # QA in Production (Contd) ## Critical success indicators (Contd) - We still had to do a lot of work to email the customer services team about email addresses that didn’t work, so we started sending these emails automatically. - With this automatic healing system in place, we stopped alerting on every failed email, because most of them were being fixed. - Now we only get alerted if a bad email address falls through the cracks and hasn’t been fixed in a certain amount of time. - Another critical usage metric for our job application system at Tes is whether teachers are actually applying for jobs. - Did they hit the "submit" button at the end of the job application form or did something stop them from doing so? It might be that a small CSS or JavaScript library change has caused the "submit" button to display or behave incorrectly in a certain browser. --- # QA in Production (Contd) ## Critical success indicators (Contd) - A dip in how frequently teachers are submitting their applications may be an indication of an issue, so we are notified if something like that happens. - We also keep a look out for HTTP 500 response codes returned by our service. - We really want teachers to be able to apply for jobs, so if anything stops them we want to know. - We’re so pedantic about this that we do whatever we can to always show them a job application form. - If something goes wrong in getting the data required to show the form, we make sure we log the error and then still do our best to let the teacher apply despite the issue. ??? pedantic = mooshekaaf of the opinion = believe --- # QA in Production (Contd) ## Critical success indicators (Contd) - We look at these logs (our alerting system reminds us to) and we deal with whatever has gone wrong. - We’re of the opinion that we’d rather fix bad data or system state than disappoint a potential candidate. ??? Raveshe log nevisi => TOSAN --- # Software Testing Anti-Patterns - Having unit tests without integration tests - Having integration tests without unit tests - Having the wrong kind of tests - Testing the wrong functionality - Testing internal implementation - Paying excessive attention to test coverage - Having flaky or slow tests --- # Software Testing Anti-Patterns (Contd) - Running tests manually - Treating test code as a second class citizen - Not converting production bugs to tests - Treating TDD as a religion - Writing tests without reading documentation first - Giving testing a bad reputation out of ignorance --- # Having unit tests without integration tests - This problem is a **classic** one with **small to medium companies**. - The application that is being developed in the company has **only unit tests** (the **base of the pyramid**) and **nothing else**. - Usually **lack of integration tests** is **caused by** any of the following issues: - The company has **no senior developers**. - The team has only junior developers fresh out of college who have **only seen unit tests** - Integration tests **existed at one point** but were **abandoned** because they caused **more trouble than their worth**. - Unit tests were **much more easy** to maintain and so they **prevailed**. - The **running environment** of the application is very **“challenging” to setup**. - Features are **“tested” in production**. ??? prevailed = ghalabe --- # HUTWIT (Contd) - Every **effective team** should have at least some kind of mentor/champion that can **show good practices to the other members**. - Difficulty in setting up a test environment. - There are indeed **some applications** that are **really hard to test**. - Once I had to work with a set of **REST applications** that actually required **special hardware on their host machine**. - This hardware **existed only in production**, making **integration** tests very **challenging**. - But this is a **corner case**. - For the **run-of-the-mill web** or back-end application that the typical company creates, **setting up a test environment** should be a **non-issue**. - With the appearance of **Virtual Machines** and lately **Containers** this is **more true than ever**. - Basically if you are trying to **test an application** that is **hard to setup**, you need to **fix the setup process first** before dealing with the tests themselves. ??? run-of-the-mill = ordinary --- # HUTWIT (Contd) - But **why are integration tests essential** in the first place? - Basically any **cross-cutting concern** of your application will require integration tests. - With the recent **microservice** craze **integration tests** become even **more important** as you now have **contracts between your own services**. - If those **services** are **developed by other teams**, you need an **automatic way** to verify that **interface contracts** are **not broken**. - This can only be covered with **integration tests**. - To sum up, unless you are creating something **extremely isolated** (e.g. a command line linux utility), you really **need integration tests** to **catch issues not caught by unit tests**. ??? cross-cutting = moteghaate --- # Having integration tests without unit tests - This is the **inverse** of the previous anti-pattern. - This anti-pattern is more common in **large companies** and large **enterprise projects**. - Almost always the history behind this anti-pattern involves **developers** who believe that **unit tests have no real value** and **only integration tests can catch regressions**. - There is a large **majority** of **experienced developers** who consider **unit tests a waste of time**. - Usually **if you probe them with questions**, you will discover that at some point **in the past**, upper **management** had **forced** them to **increase code coverage** forcing them to **write trivial unit tests**. - It is true that **in theory** you could have **only integration tests** in a software project. - But **in practice** this would become **very expensive to test** (both in **developer time** and in **build time**). ??? Regression testing is the process of testing changes to computer programs to make sure that the older programming **still works with the new changes**. Regression testing is a normal part of the program development process and, in larger companies, is done by code testing specialists. --- # HITWUT (Contd) - While **in theory** you could **only have integration** tests, **in practice** - Unit tests are **easier to maintain** - Unit tests can easily replicate **corner cases** and **not-so-frequent scenarios** - Unit tests run **much faster** than integration tests - Broken unit tests are **easier to fix** than broken integration tests - If you only have integration tests, you **waste developer time** and **company money**. - You need **both unit and integration tests** are the same time. They are not mutually exclusive. - There are **several articles** on the internet that **advocate using only one type** of tests. All these articles are **misinformed**. **Sad but true**. ??? mutually exclusive = motenaghez --- # Having the wrong kind of tests - Now that we have seen why we need both kinds of tests (unit and integration) - We need to **decide on how many** tests we need from **each category**. - There is **no hard and fast rule** here, it **depends on your application**. - The important point is that you need to **spend some time** to **understand what type of tests** add the **most value** to your application. - The **test pyramid** is **only a suggestion** on the amount of tests that you should create. - It **assumes** that you are writing a **commercial web application**, but that is **not always the case**. --- # Testing the wrong functionality - **In theory**, getting 100% code coverage in an application is the ultimate goal. - In practice this goal is **not only difficult to achieve** but also it **doesn’t guarantee** a **bug free application**. - There are some cases where indeed it is **possible to test all functionality of your application**. - If you start on a green-field project, if you work in a **small team** that is well behaved and **takes into account** the **effort required for tests**, it is perfectly **fine to write new tests for all new functionality you add** (because the existing code already has tests). - But **not** all developers are **lucky** like this. ??? The Greenfield project means that a work which is not following a prior work. In infrastructure the projects on the unused lands where there is no need to remodel or demolish an existing structure are called Green Field Projects. The projects which are modified or upgraded are called brownfield projects. --- # Testing the wrong functionality (Contd) - In most cases you **inherit** an **existing application** that has a **minimal amount of tests** (or even none!). - If you are part of **a big and established company**, working with **legacy code** is **mostly the rule** rather **than the exception**. - **Ideally** you would have **enough development time** to **write tests** for both **new and existing code** for a legacy application. - This is a **romantic idea** that will probably be **rejected** by the **average project manager** who is mostly **interested on adding new features** rather **then testing/refactoring.** - You have to pick your battles and find a **fine balance** between **adding new functionality (as requested by the business)** and **expanding the existing test suite**. - So **what** do you test? Where do you **focus your efforts**? Several times I have seen developers wasting valuable testing time by **writing “unit tests”** that add **little or no value** to the **overall stability of the application**. --- # Testing the wrong functionality (Contd) - The canonical **example of useless testing** is **trivial tests** that **verify the application data model**. - If you **ask any developer** to **show you the source code** of any application, he/she will probably **open an IDE** or code repository browser and show you the **individual folders**. - This representation is the **physical model of the code**. - It defines the **folders in the filesystem** that **contain the source code**. - While this **hierarchy of folders** is **great for working with the code** itself, unfortunately it **doesn’t define** the **importance** of **each code folder**. - A flat list of code folders implies that **all code components** contained in them are of **equal importance**. - This is **not true** as **different code components** have a **different impact** in the **overall functionality of the application**. ??? canonical = standard --- # Testing the wrong functionality (Contd) - As a **quick example** let’s say that you are writing an eshop application and **two bugs appear in production**: - Customers cannot **check-out their cart** halting all sales - Customers get **wrong recommendations** when they **browse products**. - Even though **both bugs should be fixed**, it is obvious that the **first one** has **higher priority**. - Therefore if you **inherit an eshop application with zero tests**, you should **write new tests** the directly validate the **check-out functionality** rather than the **recommendation engine**. - Despite the **fact** that the **recommendation engine** and the **check-out process** might **exist on sibling folders** in the filesystem, their **importance is different** when it comes to testing. --- # Testing the wrong functionality (Contd) - To generalize this example, if you work for some time in any medium/large application you will soon need to **think about code** using a **different representation** - the **mental model**. .center[] --- # Testing the wrong functionality (Contd) - I am showing here **3 layers of code**, but **depending** on the **size** of your application it **might have more**. These are: - **Critical code -** This is the code that **breaks often**, **gets most of new features** and has a **big impact** on **application users** - **Core code -** This is the code that **breaks sometimes**, **gets few new features** and has **medium impact** on the **application users** - **Other code -** This is code that **rarely changes**, **rarely gets new features** and has **minimal impact** on **application users**. - This mental mode should be your **guiding principle** whenever you **write a new software test**. - **Ask yourself** if the **functionality** you are writing tests for now **belongs to the critical or core categories**. - **If yes**, then **write** a software test. - **If no**, then maybe your development time should better be **spent elsewhere (e.g. in another bug)**. --- # Testing the wrong functionality (Contd) - The concept of having **code with different severity categories** is also great when you need to answer the age old question of **how much code coverage is enough** for an application. - To answer this question you **need to** either **know** the **severity layers** of the application or **ask somebody** that does. - Once you have this information at hand the **answer is obvious**: - Try to **write tests** that work towards **100% coverage of critical code**. - **If** you have already **done this**, then try to write tests that **work towards 100% of core code**. - Trying however to get **100% coverage on total code** is **not recommended**. - The important thing to notice here is that the **critical code** in an application is always a **small subset of the overall code**. - So if in an application **critical code** is **let’s say 20% of the overall code**, then getting just **20% overall code coverage** is a **good first step** for reducing bugs in production. --- # Testing internal implementation - More tests are **always a good thing**. Right? - Wrong! You also need to make sure that the tests are actually **structured in a correct way**. - Having **tests** that are written **in the wrong manner** is bad in two ways. - They **waste** precious **development time** the **first time they are written** - They **waste** even **more time** when they **need to be refactored** (when a new feature is added) - Strictly speaking, **test code is like any other type of code**. - You will need to **refactor it** at some point in order **to improve it** in a gradual way. - But if you find yourself **routinely changing existing tests** just to make them pass **when a new feature is added** then your tests are **not testing what they should be testing**. --- # Testing internal implementation (Contd) - I have seen several companies that started new projects and thinking that they will get it right this time. - They started **writing a big number of tests** to **cover the functionality** of the application. - After a while, a **new feature got added** and several existing **tests needed to change** in order to **make them pass** again. - Then **another new feature** was added and **more tests needed to be updated**. - Soon the amount of **effort spent refactoring/fixing the existing tests** was actually **larger than** the **time needed to implement the feature** itself. - In such situations, **several developers** just **accept defeat**. - They declare **software tests a waste of time** and **abandon completely the existing test suite** in order to **focus fully on new features**. - In some **extreme scenarios** some **changes** might even be **held back** because of the **amount of tests that break**. --- # Testing internal implementation (Contd) - The **problem** here is of course the **bad quality of tests**. - Tests that **need to be refactored** all the time suffer from **tight coupling with the main code**. - Unfortunately, you need some **basic testing experience** to understand **which tests are written in this “wrong” way**. - Having to **change a big number of existing tests** when a **new feature** is introduced shows the **symptom**. - The actual **problem** is that tests were instructed to verify **internal implementation** which is always a **recipe for disaster**. --- # Running tests manually - Depending on your **organization** you might actually have **several types of tests** in place. - **Unit tests**, **Load tests**, User **acceptance tests** are **common categories** of test suites that might be executed before the code goes into production. - Ideally all your tests should run **automatically** without any human intervention. - If that is not possible **at the very least** all tests that deal with **correctness of code (i.e. unit and integration tests)** must run in an **automatic manner**. - This way developers get feedback on the code in the **most timely manner**. - It is very **easy to fix** a feature when the **code is fresh** in your mind and you **haven’t switched context** yet to an unrelated feature. ??? intervention = modakhele timely = be moghe --- # Running tests manually (Contd) - Test feedback loop tests - In the past the most lengthy step of the software lifecycle was the deployment of the application. - With the move into cloud infrastructure where machines can be created on demand (either in the form of VMs or containers) the time to provision a new machine has been reduced to minutes or seconds. - This paradigm shift has caught a lot of companies by surprise as they were not ready to handle daily or even hourly deployments. - Most of the existing practices were centered around lengthy release cycles. - Waiting for a specific time in the release to “pass QA” with manual approval is one of those obsolete practices that is no longer applicable if a company wants to deploy as fast as possible. - Deploying as fast as possible implies that you trust each deployment. - Trusting an automatic deployment requires a high degree of confidence in the code that gets deployed. - While there are several ways of getting this confidence, the first line of defense should be your software tests. - However, having a test suite that can catch regressions quickly is only half part of the equation. - The other half is running the tests automatically (possibly after every commit). - A lot of companies think that they practice continuous delivery and/or deployment. - In reality they don’t. - Practicing true CI/CD means that at any given point in time there is a version of the code that is ready to be deployed. - This means that the candidate release for deployment is already tested. - Therefore having a package version of an application “ready” which has not really “passed QA” is not true CI/CD. - Unfortunately, while most companies have correctly realized that deployments should be automated, because using humans for them is error prone and slow, I still see companies where launching the tests is a semi-manual process. - And when I say semi-manual I mean that even though the test suite itself might be automated, there are human tasks for house-keeping such as preparing the test environment or cleaning up the test data after the tests have finished. - That is an anti-pattern because it is not true automation. - All aspects of testing should be automated. - Automated tests - Having access to VMs or containers means that it is very easy to create various test environments on demand. - Creating a test environment on the fly for an individual pull request should be a standard practice within your organization. - This means that each new feature is tested individually on its own. - A problematic feature (i.e. that causes tests to fail) should not block the release of the rest of the features that need to be deployed at the same time. - An easy way to understand the level of test automation within a company is to watch the QA/Test people in their daily job. - In the ideal case, testers are just creating new tests that are added to an existing test suite. - Testers themselves do not run tests manually. - The test suite is run by the build server. - In summary, testing should be something that happens all the time behind the scenes by the build server. - Developers should learn the result of the test for their individual feature after 5-15 minutes of committing code. - Testers should create new tests and refactor existing ones, instead of actually running tests. --- # References * Page-Jones' book, The Practical Guide to Structured Systems Design * Pressman's text, Software Engineering - A Practitioner's Approach * http://blog.codepipes.com/testing/software-testing-antipatterns.html * https://martinfowler.com/tags/testing.html --- class: center, middle .center[] # Thank you. Any questions?