Mindmap for Test Strategy

A test strategy acts as an outline document that describes the testing approach in the software development cycle. It is created to inform different stakeholders including project managers, testers, and developers about some key issues in the testing process.

A test strategy can be defined at organisation level or project level. If described at organisation level, the strategy is implemented in the projects in the organisation. Test Strategy document is not updated often and defines standards for testing activities and processes.

Below image shows the basic information about test strategy captured in a mindmap.

Please follow below excellent reference materials to understand the concept of test strategy:

• http:/www.satisfice.com/blog/archives/63

• http://www.softwaretestingclub.com/forum/topics/writing-a-test-strategy

• http://www.slideshare.net/monthisrael8/what-is-difference-between-a-test-plan-a-test-strategy

• http://cgi.csc.liv.ac.uk/~coopes/comp208/Test_Strategy_Template_.pdf

• https://strongqa.com/qa-portal/testing-docs-templates/test-strategy

• http://www.ministryoftesting.com/wp-content/uploads/2012/08/Heuristic-Testing-Strategy-Model-Master.png

Six challenges in Software Integration testing:

Integration testing has been changing over the time with the development of web services and change from integrating large application to smaller sets of application. In this article, we will focus on the major challenges in the system integration testing.

1.     Perception for Integration testing

The perception that Integration testing is testing the integration points between two units or system and verifying if the systems are working properly. This can actually cause issues when the actual user s) uses the application. Integration between different systems should not be oversimplified and should follow by an exhaustive testing considering collection of Systems integrated together as an environment focusing on different sets of inputs and corresponding outputs from other system.

2.     Our Individual systems are perfect, but together teamwork is missing.

Let us consider a banking system. There are multiple layers  Individual systems) for Front Office, middle Office and back office. While delivering the system, there may be possibility for systems developed by different parties. The integration testing should be done in a manner that the integration testing suite should be minimal impacted by change in other third party systems. While Creating individual systems, we should keep in mind, possible integrations with other software systems with which the software will interact, so as to minimize efforts and seamless integration of software with other soft wares

3.     Are we following the standards

 Consider transfer of banking instruction across the globe in the form of Swift messages? Suppose different banks have different messaging format, it would have been very difficult to integrate between different bank and virtually impossible to create an interconnected network between systems. Similar to this, we can follow standards of messages in the form of web services with predefined set of request and response messages. Use of Web Services also helps in integrating systems developed by different teams

4.  Complexity in managing Systems Integration

Test includes factor like Application, Platforms, Environments, data and resource availability for different application.

5.  Most of the legacy systems were isolated system with lot of functionality

Integration of new systems with the legacy system or between two different legacy systems requires a lot of changes in both the systems.

6.   In earlier systems, there was relatively small number of large application as compared to new system which focuses on large number of small applications. 

This results in changing approach for Integration testing with time.

Understanding User Acceptance Testing

In software development, the application developed is tested by different teams. Developers perform Unit testing and Integration testing before passing the product to QA for their validations. QA Performs System Testing, SIT and regression testing on the product. Once the QA certifies the product, the Product is passed on to the client/UAT Team for User Acceptance Testi

Client/UAT Team before passing the product to the end users needs to verify if the product delivered to him is the same as requested or a different product is developed. The basis for testing is the requirements of client and product is developed as expected in the requirement. The product is passed on to client once the exit criteria between client and supplier is met which includes conditions of no major issues in the developed product by supplier.

Client on receiving the developed product tests the product internally if it is as per expectation and then passes the same to a subset of end users (beta users) to validate the product from end user point of view. Once it passes both internal customers testing and external customer testing (by beta users and can be called as beta testing), it is released to market to be used by customers or end users.

Entry Criteria for UAT testing is usually the exit criteria for System Testing Phase. Entry criteria for UAT testing includes in general:

• No Major open defects in the product

• Test Execution completed and sign off communication provided by System Testing team

• List of Known issue provided.

• Environment for UAT deployed.

Once Entry Criteria for UAT is met, the UAT team starts testing on the product. UAT can happen in two stages. In First stage, the testing is done by the UAT Team of the client. The focus or key aspect of testing is to validate the product is developed as per the requirement agreed between the stakeholders. To Test the requirements, business scenario needs to be identified and testing should focus on business scenario validations. Once UAT Team feels the product is as per agreed in requirements and there are no major defects in the product, the product can be provided to subset of targeted end users to know their views and feedback on the usability of the products before releasing it officially to the end users. It is important to know either of the two stages i.e. testing by UAT Team of client or subset of targeted end users can be skipped based on the product.

To understand this, beta testing by subset of the end users is very important for a shopping or social networking website. As it is very critical to know the actual end user needs and before releasing it to market. While if the application is an internal tool to be used by client itself, he can skip the beta testing.

Next Question is who the UAT testers are. UAT Testers are usually the subject matter expert with good in-depth knowledge of domain. UAT testers usually focus on real life scenario that can happen. For e.g. for a banking Lending application, it is important to have someone with prior experience in banking and specifically in Lending domain to drive the UAT testing and preparing plan for execution.

Once the UAT team completes User Acceptance testing on the product, it provides sign-off for the product to be released to production/Go-Live. And then it’s party time for all stakeholders involved in the application with finger-crossed for no defect identified in production. 

Signing off this post and publishing to end users of this blog. Feedback/suggestion must welcome.

Understanding System Testing

In this article, we will discuss on System Testing. To Understanding System Testing, we need to understand what a system is. Let us take our self away from software testing and go to Politics. In Politics, when we say System is corrupt, it means our political system, Administrative, and support system is corrupt. If a project comes with requirement to test this system as a whole, We need to test political, administrative and support system are working correctly and we need to validate also the integration between them is also working fine.

So now we have identified system and sub-systems. It is important to identify what all needs to be tested in system, i.e. we need to identify testing types required to test the system. Testing Types needs to be identified based on the requirement in hand.Testing type for a financial site and can be different from an e-commerce website.

While a financial domain will stress more on functional testing – The requirement is completely tested and the calculations and functionality is complete is utmost priority as a small mistake can lead to huge losses and impact the business value. Next priority can be security testing as to avoid loss to customer due to security functionality not matching. Performance Testing can or cannot be priority to the financial project. E.g. for a front –end system (e.g. Market Order Placement Website), Performance testing can be priority so as to ensure the user transaction is completed in time and issues like crashing do not take place, Whereas for a back-end system (where accounting and settlement happens in back-end for the transactions), Performance testing can be lower priority compared to various other testing types.

So now we know which testing needs to be done depends upon the system requirements and on basis of testing type, we identify the tools and resources required for the testing type.

• GUI Testing – Testing the graphical user interface.

• Usability testing – Testing whether the software developed is usable for the end users

• Performance testing – Testing how system performs in terms of responsiveness and stability

• Compatibility testing – Tests the compatibility of software with different environment conditions

• Exception handling – Tests different error conditions are properly handled.

• Load testing- tests the behavior of system under both normal and peak load conditions

• Volume testing – tests the system under varying volume of data.

• Stress testing - Tests robustness of system under stress condition.

• Security testing   - Test for Security features of the application.

• Scalability testing – tests whether the software is scalable.

• Sanity and Smoke testing – initial tests to identify basic tests are working and the software is good to continue further testing.

• Exploratory/ Ad-hoc testing – Performed mostly by experience tester without any proper documentation.

• Regression testing – Performed in the final phase of software testing to identify any defect fixes has not resulted in residual effect in rest of software.

• Installation testing – Testing the installing of software on different environment.

• Accessibility testing - Test the application meets the accessibility requirement

Also based on application type, the testing can vary. E.g.: Web Services testing for SOA Application, Localization testing for same application used across different geographies and so on. All of above testing types are a topic in itself.

Right, Once we are done with System testing, It means the software developer team( and this includes testers too) have developed and tested the application and feels the system is good enough to be delivered to the client once all the open defects are closed, or some minor issues exists which can be shared with client. Once system testing is completed, it is passed to the client to perform acceptance testing on the same to provide itself and release it to the required audience. So next topic will be around acceptance testing. Perhaps next weekend. Will keep you posted.

Understanding Integration Testing with example

In this article we will discuss on integration testing. Stage two in testing after component testing. We start integration testing once the individual components are tested and it is the time to validate if they fit in with each other.

Best example to understand importance of integration testing is the jigsaw puzzle, and you will know the importance far better, if you put in lot of effort in solving the puzzle and due to few incorrect pieces, you failed to complete the picture.

The small pieces of jigsaw puzzle can be considered as the module or component. When all the pieces in the puzzles fit in together perfectly, it creates the image. In software terms, it can be considered as the software system. Integration of different modules is important to fit in the pieces together. Below diagram shows the flow in testing as discussed in above explanation.

So now we have the basic knowledge of Integration testing. Let us discuss on the different standard approach for the Integration testing. Below are the different approaches for integration testing and will try to explain the same in plain language.

• Top down Approach – In this approach, the top modules is tested and integrated first. When we say the top modules are tested first. Two questions come in mind immediately.

• How to test the top module when the individual modules that will be integrated to form the top module are not developed yet?

• Answer to this, we use stubs for flow of data from top level to bottom level modules. Stubs acts as simulator to simulate behavior of lower level module which are not developed or integrated as yet. Expected Output from stub is the same as expected in case module would have been developed and integrated.

• What is the purpose of integration testing using top down approach? 

• Answer:  Using Top down approach allows high level logic and data flow to be tested early in the testing process. Testing the top module earlier helps to flow the logic from system to sub-module so that end result is in focus and information flow from top level to bottom.

 

• Bottom up Approach –If we represent Bottom up and top down approach in a pictorial representation, Bottom up approach is just reverse of Top down approach. In this approach, we test the bottom modules first and move to the top modules. Till the time top modules are ready, we use driver to simulate the system flow.

• Big Bang Approach – In this all the sub-modules are developed first and are integrated once all the module are developed instead of incremental approach as in above examples. Big Bang Approach can be used in case of smaller system. In Large system, the big bang approach probability of failing is very high.

We have discussed on what is integration testing, types of approaches in integration testing. Let us now summarize on integration testing.

• What is integration testing –Testing Integrating and Interaction of different modules or Interfaces in the system.
• Who Performs integration tests – Integration testing should be performed by developers before releasing the system to QA. Testers should also test the interaction between subsystems.
• Bottom up and top down are example of incremental approach to integration testing.

Are there any references for further study? While studying for integration testing and advanced concept on same. I find following two articles interesting. Please read through for further details on the topic

http://www.eecs.yorku.ca/course_archive/2009-10/W/4313/slides/14-IntegrationTesting.pdf

http://www.iai.uni-bonn.de/III/lehre/vorlesungen/SWT/OOSC05/slides/15%20-%20Testing%20(2-2).pdf

Component Testing – First Testing in the Project

The first phase is testing the smallest testing unit developed in the Project. A unit can be defined as an isolated piece of code which can be tested independently.Let us take an example, suppose we have to develop an application for an e-commerce site with following requirement.

• Login into System.
• Home Page is displayed.
• Selecting any product and adding to my cart.
• Paying for the Product.

So In this example, we can divide the project development into following components or units.

• Login Module.
• Home Page Module.
• Transaction Page.
• View Cart Module.
• Payment Module.

Before discussing on different testing type. Let us discuss how the development flow takes place. 

• Development team will develop different modules or units of code.
• The modules should be integrated successfully with rest of the modules
• The System developed should be integrated with third party components or existing components.
• The system should work as per requirement document.

As one of the principle of software testing, defects should be captured as early as possible in the software development life cycle. Since the first step in testing is testing the individual module developed. If the individual module has defect, fixing the same can be painful if captured in later stages of SDLC.Unit testing searches for defects in the testable unit or module as discussed. We need to focus on testing the functional features as well as non-functional characteristics.

Let us take the example of unit testing the login module. A login Page needs to be tested as shown below.

The functional test case for the unit will include:

• Error Message on providing invalid username or Password.
• Error Message for minimum and maximum allowed characters.
• On providing successful username/Password, Home Page is displayed. 
• Since Home Page is not yet created, we will use stubs and drivers to dummy for home Page.

The above are examples of functional test cases for components.

.

Component testing also includes non-functional characteristics including memory leaks, and structural testing for code coverage. Component testing is done mainly by Developers and is mostly white box testing specially for verification of non-functional validation. Defects are fixed as they are found as is mostly development defect uncovered during dev testing itself.

There are various tools and framework available for unit testing. Junit, testNG are example of framework available for unit testing. Another framework is Test Driven Framework, in which test scripts are prepared before coding. The tests are expected to fail before coding phase. Once Coding is done, all the unit tests are expected to pass. In case, test failing post development, code needs to be fixed until all the tests created for component passed.

Summarizing, all we discussed so far.

Component testing is testing the unit or component or module as an independent unit.
Component Testing is the first testing performed in the project.Component testing is performed mainly by developers.Component testing is more of white box testing on the code.Different Unit testing frameworks are available in the project.

Some Excellent reference article for detailed understanding of Component testing are as follows:

http://www.testingstandards.co.uk/Component%20Testing.pdf

http://people.cs.clemson.edu/~johnmc/joop/col3/column3.html

Functional and Non Functional Testing - Difference with Real life example

It is common in testing project to scope of testing in the project is to perform functional as well as non-functional testing in the project. It is important to understand the difference between the two to define strategies for each testing. Let me try to explain this with a simple example in daily life. Suppose I have to test the functional and non-functional requirement of a restaurant.

The functional requirement can be following

When a user enters the restaurant, he will be welcomed by the Staff.  User will place the order; Order will be delivered to User. Once user completes the dinner/lunch, he will pay either by cash or credit card. Staff will say goodbye to user once user exits from the restaurant.

The above is an example of functional requirement in real price, similar in software can be compared to an online e-commerce site.

The functional requirement will be broken down into functional test scenarios which can further be broken down into test cases ensuring all the requirement are covered using RTM (Requirement traceability matrix).

Now coming to the same example, Let us talk about the non-functional requirement in the same example.User will be very much influenced by time it takes from the time he places the order to time order reaches him. This is related to performance and efficiency aspect of requirement.

Similarly there can be security compromise, suppose I pay my bill by credit card and the staff instead of returning the bill with credit card to me, give it to some other customer, and the customer exits with my credit card and spends money from my card. This is an example of Security testing.

Let us take another example, suppose I ordered a dish which needs long cooking, Suppose the food is undercooked. Although I received what I ordered, the food is such that, if I eat it, It will take me one week to recover from its after effect. This is an example of usability testing.

As we see in the example, Non-Functional features (features that were not a part of functional requirement) are very important and needs to be properly tested to ensure the business success.

As in above example, If any one of the performance, security and usability of the restaurant does not meet the customer requirement, There is high probability, the customer will never visit the restaurant again, which will be a business loss to the restaurant.

The same example can be easily reciprocated in software projects. Therefore non-functional testing should be given proper priorities before releasing a project. How much to spend on non functional testing depends on which non-performance parameters are critical.

For e.g in a trading site, security testing should be preference, while in a search engine, e.g. google, the performance, load, stress testing are more important.

Some examples of Non functional includes:

• Usability Testing

• Security Testing

• Performance testing

• Load Testing

• Stress testing

• Scalability Testing

• Data Integrity Testing

• Interoperability Testing

Key Information in a Test Plan

Test Plan is a document providing structural approach describing how testing will be accomplished in the Project. This acts as a contract between testing team and stakeholders on how testing is planned for the project. 

For successful execution of testing in the project, it is very important to do thorough planning of the testing requirement and documenting the same in the form of test plan.

Most of the organisation have defined template for test plan and have slight difference in the structure of the test Plan from each other. What information needs to be captured in the test plan will be discussed in this article in the first section:

• Defining the Scope of testing – The first and one of the important component in test planning is defining the scope of testing and at the same time providing the items which are Out of scope of testing. Scope of testing should be in line with the overall scope of Project. Defining scope early in testing helps in defining the acceptance criteria and agrees on the exit criteria for testing.

• Objective of Testing in the Project should be defined - Objective of testing can be testing the functional requirement or creating an automation suite for regression scope. Also there can be multiple objective of testing with some critical and some secondary objective e.g. : Testing complete functional requirement of application can be primary objective of testing with communicating all known defects and issue with different stakeholder and closure on all of them before release to production can be the secondary objective of testing. All the objectives should be clearly defined in test plan. Each objective should have the priority defined and acceptance criteria should be associated with each test objective.

• Staffing and training needs for the testing team should be mentioned in the test plan.

• Environment and resource requirement required for testing, i.e. Software and hardware requirements should be clearly defined.

• Assumptions acts as probable risks to Project. All the assumptions should be added in test plan and proper sign-off  should be taken from stakeholders to keep them informed and confirmation should be taken from them for the correctness of assumptions made during testing and will not lead to possible risks in future.

• We should define constraints in the test plan. Constraints impact the efficiency and effectiveness of testing. Constraint can include for e.g. staffing, time or cost constraints

To summarize, so far we have discussed the ‘What’ part of testing, i.e. What we will test, what will be our scope, what our assumption, constraints are, what will be the scope of testing. Once we are clear on What, Next question is how to do what.

How Testing will be executed is explained in Testing Methodology. The Key points explained in Testing Methodology include:

• Define testing strategies for different phases and functional and non-functional requirements. This includes:

• Different types of testing required for requirement.

• Approach for each testing type, e.g. Regression, functional, automation or locale testing.

• Risk/issues identification and test data generation for testing types.

• Testing cycle for each testing and testing milestones are set.

• Entry and exit criteria for each testing milestones are described

• Providing defect lifecycle to be followed.
• Defining testing schedule in sync with the milestone defined.

Another Topic to be covered in test plan includes test deliverables and defining the exit criteria for testing completion. Test deliverables should be defined; it can include user guide, test scripts, test data, Test execution results, and Test plan.

Together, we can add matrix for different areas in test plan for easier understanding and better explanation. We will discuss on methodology and test deliverables in details in the next articles on topic Software Testing.

Understanding Agile Methodology and principles

In 2001, At a summit seventeen practitioners of different software methodologies met together and created the agile manifesto. Principles of Agile Manifesto are implemented in various projects for agile development.

The Manifesto for agile development states developing keeping the following values as focus:

·         Individuals and interactions over processes and tools

·         Working software over comprehensive documentation

·         Customer collaboration over contract negotiation

·         Responding to change over following a plan

Agile Methodology thus focuses on close interaction between resources and customers to create working software and responding to changes in software based on feedback between the teams. 

If we go back to Iterative and Incremental model, we can say Agile Methodology as a subset of IID Model. Software Testers works in a collaborative model as a part of Agile teams providing necessary feedback together with reporting defects in the development.

Agile as per dictionary definition means:  able to move quickly and easily and able to think and understand quickly. The Definition defines agile methodology in a nice manner. Using Agile in projects, we can move in quick manner thinking over the problem and understanding it quickly and easily by working in a collaborative manner.

In terms of Software testing, agile methodology is different from previously used methodology, e.g. Waterfall and V-Model, as it allows tester a role early in the development in a collaborative manner, thus uncovering the issues from early analysis phase. Also Testers are integral part of agile teams instead of independent teams. The Collaborative approach separates agile from other SDLC Models, and helps better communication between various stakeholders including customer, managers, developers, testers, and analysts bringing responsiveness to changing requirements and a more efficient product developed since all the stakeholders works as a team with better understanding of the requirements.

Key Principles of Agile Manifesto:

1.  Satisfying the customer through early and continuous delivery of valuable software.

2.  Agile processes harness change for the customer competitive advantage.

3.  Delivering working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

4.  Business people and developers must work together daily throughout the project.

5.  Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

6.  The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

7.  Working software is the primary measure of progress.

8.  Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

9.  Continuous attention to technical excellence and good design enhances agility.

10. Simplicity–the art of maximizing the amount of work not done–is essential.

11. The best architectures, requirements, and designs emerge from self-organizing teams.

12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Reference:  http://agilemanifesto.org/principles.html

Where to Use and Where not to use Agile?

• In case, most of the project in the organization is handled through traditional development Model, it is better to start with a smaller project. The reason we cannot predict the success or failure of the agile is the collaborative nature of the project and this includes developers, management, Customers, and testers. The traditional models do not allow regular interactions between the stakeholders. E.g. we get the requirement from the customers, based on requirements the analyst prepares the design, developers code the design and testers test the code. Thus each of the groups works as individual units with collaboration missing which is a must for Agile.

• Since the individual as well as customer had worked in the traditional model for a longer part of their career, it becomes difficult to switch to agile methodologies, where we need to work collaboratively. So to understand the approach and mindset for Agile methodologies before working on larger project with large number of resources and stakeholder.

• Agile Methodologies should be used in project whose business severity is neither too high nor too low, so that proper importance is provided to the project.

• Agile should not be used on Projects where large documentation is required e.g. Legal or Compliance Projects.

Understand Iterative and Incremental Development Model

In Iterative and Incremental Model, the whole requirement is divided into various sub–requirements or builds. Each build in itself follows waterfall model, i.e. each build passes through requirement, design, implementation and coding. The need for incremental and iterative model arise and is very useful in scenario where requirement changes frequently or we are working on an idea and the detailed requirement may not be clear at the start of development but a high level vision of the end product is available to us.

As the name suggest, this model will consider both iterative and incremental development. Before we discuss this in detail, let us know what iterative and incremental means. Various parts of the system are developed at different time and are integrated into the existing developed work once they are completed. And iterative strategy assigns time for rework and improvement in the part of system developed, thus continuously working in iterations to rework and improve the system adding more features in an iterative manner.

IID Model

To start with this model, we divide the project in a number of modules or stories. Each module/story represents functionality or use case in the product. Stories should be created or planned in such a manner that each story covers a feature of product, has set of entry and acceptance criteria and needs to be of shorter span of time, may be few days to few weeks. 

This is the analysis phase where we have analyzed the various stories from the requirement based on the features expected from the product. With each story having its own story means features  which will have a short span of time to complete.

Now once we have analyzed all the stories in the requirement, how do we react to stories, Do we start working on all stories together in a bang bang manner or do we need to priority to the stories, We will start with the story with most risk and complexity, and before this will complete on other smaller stories on which the most complex and risk story is developed. Once the complex piece is selected, we work on following a small waterfall on the story, 

We start with analysis, then design, coding and then testing the code, but with regular feedback at each of the steps.

The first piece will take a longer time compared to next stories but will provide useful immense information on the feasibility of the project, Knowledge to team on the product and possible bottlenecks in the development of project. It is strictly advised not to set deadlines for the initial few pieces as it will act as guidance to rest of stories on how long it will take to develop another story in the project.

Once we develop another story, we add it to existing product thus increment the functionality added to existing product and at the same time, we do iteration to improve and rework to existing developed piece based on defects, feedback from various stakeholder. 

All the pieces are developed, integrated with existing developed functionality, once the piece is fitted with rest of the developed pieces, we can have feedback and testing on the incremented product so that we can do rework and review the product. The development happens in an incremental and iterative manner unless the completed product is developed.

Where to Use IID Model

• Major functionality and goal of development are well defined, but features and enhancement mayevolve with time.

• IID is very useful in case of development with new technologies to validate a prototype filling preparing the complete.

• High Risk features which may change in future.

Positives in the Model

• An initial working product is developed in the early stages quickly.

• Developing the initial product early helps in getting feedback and defect Identification early in the lifecycle.

• We can work on different stories/pieces at the same time.

• Continuously improving the product by rework and review is possible based on iterative nature of development.

• Risks are identified in a better manner and can be implemented in the remaining project development.

Negatives in the model

• The timelines of the project are not fixed and can change drastically based on requirement change.

• Sometimes incorrect implementation of model can lead of frequent requirement change and estimation of project.

• Initial development which helps in identifying risks in the system required experienced risk analysts as incorrect risk analysis during initial phase may lead to failure of development.

• Management complexity is higher.