SDA India is an online resource for Software, Development,IT, Architecture, Open Source, Mobile, Security, Databases, Delphi, C, OS, Asp, .Net, Php, Xml, Java
Articles
Delivering Quality Applications Through Continuous Integrated Testing (CIT)
In this increasingly pressurised environment, many IT organisations have begun to adopt new ways of structuring development, including agile development, continuous integration and modified waterfall, that seek to make the development, testing and release process more continuous and enable IT to respond faster to the business. They shift the traditional development life cycle by incorporating more activities during the development phase.
Gone are the days when business managers would request an application, and then leave IT to determine how long it would take to build, test and deliver an optimal application back to the business.
Today’s reality is budget constraints and a brutally competitive marketplace. Business managers therefore need applications that can be modified quickly to reflect future market changes. And they need IT to respond more quickly to the business needs for new and updated applications.
While both development and QA have common business goals, they have traditionally used very different tools and processes to meet those business goals. Because of shortened cycle times and increased pressures from business, QA and development are finding themselves thrown together in a more intimate application life-cycle environment. However, without a way to work together, these two teams are unable to improve turnaround times as they lack common tools and process that will enable them to work together more effectively. As a result, things fall between the cracks, schedules are pushed out and the two groups become divided further.
Best Practices for Building Performance from the Start
Developing an effective application performance assurance strategy requires input and collaboration from many disciplines within IT and the business.
To ensure high performance levels, application performance assurance practices must measure and provide feedback on how a given system meets the needs of end users on an ongoing basis. A well-rounded application performance assurance strategy requires integrated performance profiling with load testing. This combination offers an end-to-end approach that builds performance into the application early in the development life cycle.
While no one process or methodology fits every IT environment, there are several best practices to consider for managing application performance.
Best Practice #1: Implement a Disciplined Approach to Gathering Requirements
Designing a business-critical application and planning out the supporting infrastructure is crucial. To start, create a logical flow of how the application will work and, based on the business type and size, understand how much transaction volume an application can handle. Walk through the logic of the application to ensure it is correct, as straightforward as possible, and meets the stated business objectives.
Make sure all the appropriate business and technology stakeholders, including developers, testers, operations specialists, business analysts, web designers and, when appropriate, business executives, are represented in the process of application and infrastructure planning. You need to take into account how web, application and database servers perform individually and as a connected infrastructure. It is also equally important to include multiple disciplines from operations, such as network specialists, database engineers, etc.
Best Practice #2: Begin Proactive Performance Analysis
Identifying and addressing the majority of performance issues prior to an application going live is best because you can catch problems at a stage when they are less expensive and easier to fix.
Once the design of the infrastructure is agreed upon and the initial components are in place, we recommend IT organisations build a prototype of the application that touches all tiers of the infrastructure to test and monitor how individual infrastructure components work together. As soon as an application creates traffic between itself and its servers, you can measure how much the application consumes network resources and the latency using a single transaction in an isolated environment. You can understand how users accessing an application with different connectivity environments are affected. This approach also provides a timely warning of when an application may not support hundreds or thousands of simultaneous users.
Best Practice #3: Couple Performance Profiling with Performance Analysis and Testing
Application performance profiling should be done prior to an application going live and provide response time metrics by transaction type. IT teams should state their expectations for performance based on infrastructure choices, such expected response times for each transaction type and class of connectivity. Results from the first round of performance tests and predictions can serve as points of reference or performance benchmarks to highlight where improvements can be made as the application evolves. The final set of acceptable results establishes the baseline and thresholds for performance monitoring in production.
An important result of early profiling is both business and IT management gain a realistic understanding of achievable performance. With performance measurements, businesses have factual data to manage user expectations and service-level agreements, eliminating performance surprises and increasing the overall awareness of an application’s capabilities.
Best Practice #4: Integrate Root Cause Analysis with Load Testing
An effective Application Performance Assurance solution should provide developers with the information that a performance or scalability problem exists and the data to help developers analyse a problem based on how the applications respond under load.
Sophisticated performance analysis during development collects and analyses data on the amount of time it takes to execute a given function or even a line of code. This makes it easy to examine all performance aspects of a running application.
By running performance analysis in conjunction with load testing, the performance team can provide developers with the information they need to analyse, diagnose and fix the problem, thereby reducing the overall MTTR (mean- time-to-repair) for the application. Critical applications reach production more quickly and perform more reliably, lowering maintenance costs and losses due to downtime.
Best Practice #5: Perform Iterative Performance Testing Throughout the Life Cycle
When conducting scalability tests, IT teams can simplify the problem identification process by individually screening small pieces of the application. Once these pieces have passed their tests, the operations group typically will start tuning server configurations for the application components. These early performance tests will eliminate bigger problems later on when the application and its infrastructure are more complex. They also serve as benchmarks for performance improvements as the application and its supporting infrastructure evolve.
When the entire application is assembled, more rigorous performance tests are required. Start with a small number of virtual users and increase the number gradually to hundreds or thousands of simultaneous users to understand how an application will scale.
Best Practice #6: Provide a Robust Test Data Environment that Simulates Production
Achieving quality testing requires valid data, complete with working relationships between tables and files, so pay particular attention to the data used in large-scale performance tests.
An effective performance assurance solution should include variable parameterisation so the full range of variable test data can be adequately load tested. Consider automatically generating millions of unique names, SSNs and many other values. This allows all aspects of the application, down to the line of SQL code, to be tested against any number of users and data records.
Also, protect your production data during testing. To accomplish this, you need a data privacy workbench that helps you translate, generate, age, encrypt and validate your test data.
Best Practice #7: Combine Pre-production Testing and Production Monitoring for the Strongest Line of Defense
The use of both production monitoring and automated testing tools significantly increases the odds of maintaining acceptable levels of performance and user satisfaction.
The information you gain from monitoring a production application can be used to enhance the realism of ongoing performance tests that occur as the application is updated and modified. Combined, testing and monitoring create a detailed picture of system and application performance.
This approach measures the effect of change and helps teams understand where performance issues originate, and resolve them correctly. The net result is a multi-dimensional view of performance that works to eradicate errors continuously. You know for sure applications perform to meet user expectations and fulfill service level agreements.
Best Practice #8: Provide End-to-end Performance Visibility for Effective Decision-Makin
Making efficient and effective decisions based on real-time information leads to better results and, ultimately, better performance. This real-time information can be provided by a combination of testing and monitoring that gathers and presents a detailed picture of system and application performance, helping IT management make informed decisions regarding resource assignments, deployment and investment decisions and recommendations to the business.
End-to-end performance should be measured in terms that provide value for the business. End-user, service-oriented metrics offer the best-of-breed in today’s IT shops. Managing from the end-user perspective lets IT identify problems long before help desks receive the first call.
By measuring end-user response time and application availability consistently over time, IT can resolve problems more quickly and identify patterns and trends to avoid problems in the future.
Continuous Integrated Testing (CIT): The Glue for Development and QA
Continuous Integrated Testing (CIT) is increasingly being viewed as an effective solution that helps IT organisations successfully marry development and QA to meet business needs.
CIT is an important step in the process of helping IT organisations become more responsive to business needs while meeting ever-tighter development schedules. It cost-effectively increases both the number of testing cycles and the resulting application quality while decreasing the amount of time it takes to find problems and the cost to fix them.
Continuous Integration focuses on integrating the components of an application more frequently than traditional development approaches—from every week to once a day. CIT extends this approach by incorporating functional testing, enabling organisations to maximise the use of testing throughout the development process to increase overall application quality.
CIT can be applied to both agile-based development organisations as well as traditional development methodologies such as waterfall or modified waterfall approaches. It provides both development and QA with common tools to work together, enabling them to more effectively produce quality applications.
In traditional models, application testing is a distinct phase that does not start until after development is completed. With CIT, application testing begins as soon as development starts, resulting in continuous test cycles, and enabling developers to isolate application problems as they are introduced and take corrective action immediately, rather than passing them over the wall to testing, where it may take several cycles to detect the error. This not only speeds the ultimate solution but can also have a positive impact on other components simultaneously being developed.
CIT combines functional testing with developer productivity tools. Developers utilise the testing tools to generate unit test cases, while profiling the application, providing them with method and line-level detail in the areas of code coverage, performance and memory issues. Automating this process and combining it with developer productivity tools enables IT to create a daily, repeatable process that will highlight issues that were introduced into the code that day. This impacts the amount of time for testing which is reduced as many code and application errors are eliminated at their root.
Benefiting from CIT
CIT is that it can be applied to most IT organisations. You don’t need to be a dedicated agile or Extreme Programming shop to benefit from adopting a CIT approach.
Using CIT, any organisation that is interested in bridging the gap between development and QA to compress development and testing can accomplish this objective. It is important for organisations to understand that CIT can apply to many different application development and testing environments, and its widespread relevance can deliver impressive ROI to IT organisations looking for ways to improve overall responsiveness and productivity.
CIT has numerous measurable benefits, including:
1. CIT minimises defects. By streamlining the developer’s workday, CIT leads to an overall reduction in application defects. Using CIT-based tools, developers are regularly testing code they have developed, especially how that code works with the integrated application. Bugs are detected almost immediately and repaired within 24 hours of their creation. This frees up development to focus on writing more code rather than burning cycles trying to hunt down a bug.
2. CIT shortens total testing time. Testing early does not have to mean that the total testing time has increased. In fact, most IT organisations that use a CIT approach find the overall testing time decreases. Defects are found earlier in the project when they are less likely to be compounded.
3. CIT reduces cost over time. An important benefit of the CIT methodology is that it enables developers and testers to identify defects early and correct them immediately, not weeks or months later, when more traditional functional testing would take place.
4. CIT provides integrated tools for development and QA. With CIT, the brick wall that has existed between development and QA is starting to come down. By structuring the view of the application process in a universal way for both teams, through the use of common testing tools, each can gain value from their existing build and testing processes.
CIT includes a number of different tools that developers, QA and IT managers can use to maximise the effectiveness of their work.
Vendors like Compuware help businesses deliver quality applications through solutions that integrate performance profiling and monitoring with load testing to offer an end-to-end approach that builds performance into the application early in the development life cycle. These solutions include TestPartner and DevPartner. The end result is quicker application deployments at lower cost.
Conclusion
As IT managers look for ways to optimise their application life cycle, they need solutions that will help bridge the gap between development and testing, reducing cycles and, as a result, cutting costs and delivering higher quality applications.
By implementing CIT, organisations can be confident that they are moving their applications into production on time and with far fewer defects. With the proper CIT-oriented tools, developers and testers can work together in a team-oriented environment toward a common goal of quality applications delivered when the business needs them. IT managers benefit from the visibility that CIT provides into the development process, enabling them to make more informed decisions.
Gone are the days when business managers would request an application, and then leave IT to determine how long it would take to build, test and deliver an optimal application back to the business.
Today’s reality is budget constraints and a brutally competitive marketplace. Business managers therefore need applications that can be modified quickly to reflect future market changes. And they need IT to respond more quickly to the business needs for new and updated applications.
While both development and QA have common business goals, they have traditionally used very different tools and processes to meet those business goals. Because of shortened cycle times and increased pressures from business, QA and development are finding themselves thrown together in a more intimate application life-cycle environment. However, without a way to work together, these two teams are unable to improve turnaround times as they lack common tools and process that will enable them to work together more effectively. As a result, things fall between the cracks, schedules are pushed out and the two groups become divided further.
Best Practices for Building Performance from the Start
Developing an effective application performance assurance strategy requires input and collaboration from many disciplines within IT and the business.
To ensure high performance levels, application performance assurance practices must measure and provide feedback on how a given system meets the needs of end users on an ongoing basis. A well-rounded application performance assurance strategy requires integrated performance profiling with load testing. This combination offers an end-to-end approach that builds performance into the application early in the development life cycle.
While no one process or methodology fits every IT environment, there are several best practices to consider for managing application performance.
Best Practice #1: Implement a Disciplined Approach to Gathering Requirements
Designing a business-critical application and planning out the supporting infrastructure is crucial. To start, create a logical flow of how the application will work and, based on the business type and size, understand how much transaction volume an application can handle. Walk through the logic of the application to ensure it is correct, as straightforward as possible, and meets the stated business objectives.
Make sure all the appropriate business and technology stakeholders, including developers, testers, operations specialists, business analysts, web designers and, when appropriate, business executives, are represented in the process of application and infrastructure planning. You need to take into account how web, application and database servers perform individually and as a connected infrastructure. It is also equally important to include multiple disciplines from operations, such as network specialists, database engineers, etc.
Best Practice #2: Begin Proactive Performance Analysis
Identifying and addressing the majority of performance issues prior to an application going live is best because you can catch problems at a stage when they are less expensive and easier to fix.
Once the design of the infrastructure is agreed upon and the initial components are in place, we recommend IT organisations build a prototype of the application that touches all tiers of the infrastructure to test and monitor how individual infrastructure components work together. As soon as an application creates traffic between itself and its servers, you can measure how much the application consumes network resources and the latency using a single transaction in an isolated environment. You can understand how users accessing an application with different connectivity environments are affected. This approach also provides a timely warning of when an application may not support hundreds or thousands of simultaneous users.
Best Practice #3: Couple Performance Profiling with Performance Analysis and Testing
Application performance profiling should be done prior to an application going live and provide response time metrics by transaction type. IT teams should state their expectations for performance based on infrastructure choices, such expected response times for each transaction type and class of connectivity. Results from the first round of performance tests and predictions can serve as points of reference or performance benchmarks to highlight where improvements can be made as the application evolves. The final set of acceptable results establishes the baseline and thresholds for performance monitoring in production.
An important result of early profiling is both business and IT management gain a realistic understanding of achievable performance. With performance measurements, businesses have factual data to manage user expectations and service-level agreements, eliminating performance surprises and increasing the overall awareness of an application’s capabilities.
Best Practice #4: Integrate Root Cause Analysis with Load Testing
An effective Application Performance Assurance solution should provide developers with the information that a performance or scalability problem exists and the data to help developers analyse a problem based on how the applications respond under load.
Sophisticated performance analysis during development collects and analyses data on the amount of time it takes to execute a given function or even a line of code. This makes it easy to examine all performance aspects of a running application.
By running performance analysis in conjunction with load testing, the performance team can provide developers with the information they need to analyse, diagnose and fix the problem, thereby reducing the overall MTTR (mean- time-to-repair) for the application. Critical applications reach production more quickly and perform more reliably, lowering maintenance costs and losses due to downtime.
Best Practice #5: Perform Iterative Performance Testing Throughout the Life Cycle
When conducting scalability tests, IT teams can simplify the problem identification process by individually screening small pieces of the application. Once these pieces have passed their tests, the operations group typically will start tuning server configurations for the application components. These early performance tests will eliminate bigger problems later on when the application and its infrastructure are more complex. They also serve as benchmarks for performance improvements as the application and its supporting infrastructure evolve.
When the entire application is assembled, more rigorous performance tests are required. Start with a small number of virtual users and increase the number gradually to hundreds or thousands of simultaneous users to understand how an application will scale.
Best Practice #6: Provide a Robust Test Data Environment that Simulates Production
Achieving quality testing requires valid data, complete with working relationships between tables and files, so pay particular attention to the data used in large-scale performance tests.
An effective performance assurance solution should include variable parameterisation so the full range of variable test data can be adequately load tested. Consider automatically generating millions of unique names, SSNs and many other values. This allows all aspects of the application, down to the line of SQL code, to be tested against any number of users and data records.
Also, protect your production data during testing. To accomplish this, you need a data privacy workbench that helps you translate, generate, age, encrypt and validate your test data.
Best Practice #7: Combine Pre-production Testing and Production Monitoring for the Strongest Line of Defense
The use of both production monitoring and automated testing tools significantly increases the odds of maintaining acceptable levels of performance and user satisfaction.
The information you gain from monitoring a production application can be used to enhance the realism of ongoing performance tests that occur as the application is updated and modified. Combined, testing and monitoring create a detailed picture of system and application performance.
This approach measures the effect of change and helps teams understand where performance issues originate, and resolve them correctly. The net result is a multi-dimensional view of performance that works to eradicate errors continuously. You know for sure applications perform to meet user expectations and fulfill service level agreements.
Best Practice #8: Provide End-to-end Performance Visibility for Effective Decision-Makin
Making efficient and effective decisions based on real-time information leads to better results and, ultimately, better performance. This real-time information can be provided by a combination of testing and monitoring that gathers and presents a detailed picture of system and application performance, helping IT management make informed decisions regarding resource assignments, deployment and investment decisions and recommendations to the business.
End-to-end performance should be measured in terms that provide value for the business. End-user, service-oriented metrics offer the best-of-breed in today’s IT shops. Managing from the end-user perspective lets IT identify problems long before help desks receive the first call.
By measuring end-user response time and application availability consistently over time, IT can resolve problems more quickly and identify patterns and trends to avoid problems in the future.
Continuous Integrated Testing (CIT): The Glue for Development and QA
Continuous Integrated Testing (CIT) is increasingly being viewed as an effective solution that helps IT organisations successfully marry development and QA to meet business needs.
CIT is an important step in the process of helping IT organisations become more responsive to business needs while meeting ever-tighter development schedules. It cost-effectively increases both the number of testing cycles and the resulting application quality while decreasing the amount of time it takes to find problems and the cost to fix them.
Continuous Integration focuses on integrating the components of an application more frequently than traditional development approaches—from every week to once a day. CIT extends this approach by incorporating functional testing, enabling organisations to maximise the use of testing throughout the development process to increase overall application quality.
CIT can be applied to both agile-based development organisations as well as traditional development methodologies such as waterfall or modified waterfall approaches. It provides both development and QA with common tools to work together, enabling them to more effectively produce quality applications.
In traditional models, application testing is a distinct phase that does not start until after development is completed. With CIT, application testing begins as soon as development starts, resulting in continuous test cycles, and enabling developers to isolate application problems as they are introduced and take corrective action immediately, rather than passing them over the wall to testing, where it may take several cycles to detect the error. This not only speeds the ultimate solution but can also have a positive impact on other components simultaneously being developed.
CIT combines functional testing with developer productivity tools. Developers utilise the testing tools to generate unit test cases, while profiling the application, providing them with method and line-level detail in the areas of code coverage, performance and memory issues. Automating this process and combining it with developer productivity tools enables IT to create a daily, repeatable process that will highlight issues that were introduced into the code that day. This impacts the amount of time for testing which is reduced as many code and application errors are eliminated at their root.
Benefiting from CIT
CIT is that it can be applied to most IT organisations. You don’t need to be a dedicated agile or Extreme Programming shop to benefit from adopting a CIT approach.
Using CIT, any organisation that is interested in bridging the gap between development and QA to compress development and testing can accomplish this objective. It is important for organisations to understand that CIT can apply to many different application development and testing environments, and its widespread relevance can deliver impressive ROI to IT organisations looking for ways to improve overall responsiveness and productivity.
CIT has numerous measurable benefits, including:
1. CIT minimises defects. By streamlining the developer’s workday, CIT leads to an overall reduction in application defects. Using CIT-based tools, developers are regularly testing code they have developed, especially how that code works with the integrated application. Bugs are detected almost immediately and repaired within 24 hours of their creation. This frees up development to focus on writing more code rather than burning cycles trying to hunt down a bug.
2. CIT shortens total testing time. Testing early does not have to mean that the total testing time has increased. In fact, most IT organisations that use a CIT approach find the overall testing time decreases. Defects are found earlier in the project when they are less likely to be compounded.
3. CIT reduces cost over time. An important benefit of the CIT methodology is that it enables developers and testers to identify defects early and correct them immediately, not weeks or months later, when more traditional functional testing would take place.
4. CIT provides integrated tools for development and QA. With CIT, the brick wall that has existed between development and QA is starting to come down. By structuring the view of the application process in a universal way for both teams, through the use of common testing tools, each can gain value from their existing build and testing processes.
CIT includes a number of different tools that developers, QA and IT managers can use to maximise the effectiveness of their work.
- Developers: CIT enables developers to hone right in on a defect, rather than spending hours searching for a needle in a haystack. Using CIT, developers can discover defects right away and fix them immediately. Following the next build of the software, the same tests that found the issue can validate that it has been rectified and that the fix did not spawn any other issues. This leads to greater developer productivity by decreasing the time to find and fix bugs. Continuous testing of the application leads to fewer fires for developers to put out, freeing them to focus on their development strengths.
- Quality Assurance: CIT benefits QA by involving testers in the process as soon as development commences. As development progresses, test scripts grow exponentially. Accurate and thorough test assets are passed from development to QA. This enables the QA team to focus on testing the most important areas of the application, thus avoiding wasted time on non-productive tests. In addition, CIT supports automated testing, further expediting the testing process and maximising its effectiveness. Developers and testers work in harmony to expedite the development process and raise the quality of the application.
- IT Managers: CIT provides a common toolkit that both development and QA can use together to increase productivity. It helps move the IT organisation toward more continuous development, even if it does not practice an “agile development” approach. CIT enables IT managers to adopt the benefits of continuous integration while using their current development and testing practices, a benefit that is both cost effective and increases ROI.
Vendors like Compuware help businesses deliver quality applications through solutions that integrate performance profiling and monitoring with load testing to offer an end-to-end approach that builds performance into the application early in the development life cycle. These solutions include TestPartner and DevPartner. The end result is quicker application deployments at lower cost.
Conclusion
As IT managers look for ways to optimise their application life cycle, they need solutions that will help bridge the gap between development and testing, reducing cycles and, as a result, cutting costs and delivering higher quality applications.
By implementing CIT, organisations can be confident that they are moving their applications into production on time and with far fewer defects. With the proper CIT-oriented tools, developers and testers can work together in a team-oriented environment toward a common goal of quality applications delivered when the business needs them. IT managers benefit from the visibility that CIT provides into the development process, enabling them to make more informed decisions.
Jeff Findlay has over twenty years experience as an information technology professional. He specifically worked in the development of architectural software focusing on 3-D modelling. Recently appointed regional champion for Compuware’s Requirements Management solution – OptimalTrace, Jeff joined Compuware eight years ago and has held a number of different positions in the areas of application development, software testing and development process improvement.
Related Links
None
