Chapter 12 – Modeling Application Usage

From Guidance Share

(Difference between revisions)
Jump to: navigation, search
Revision as of 05:07, 29 April 2009 (edit)
GardenTender (Talk | contribs)

← Previous diff
Revision as of 05:09, 29 April 2009 (edit)
GardenTender (Talk | contribs)

Next diff →
Line 41: Line 41:
==Identify the Objectives== ==Identify the Objectives==
-The objectives of creating a workload model typically center on ensuring the realism of a test, or on designing a test to address a specific requirement, goal, or performance-testing objective. (For more information, see [[Chapter 9 – Determine Performance Testing Objectives]] and [[Chapter 10 – Quantify End-User Response Time Goals]].) When identifying the objectives, work with targets that will satisfy the stated business requirements. Consider the following key questions when formulating your objectives:+The objectives of creating a workload model typically center on ensuring the realism of a test, or on designing a test to address a specific requirement, goal, or performance-testing objective. (For more information, see [[Chapter 9 – Determining Performance Testing Objectives]] and [[Chapter 10 – Quantifying End-User Response Time Goals]].) When identifying the objectives, work with targets that will satisfy the stated business requirements. Consider the following key questions when formulating your objectives:
*What is the current or predicted business volume over time? For example, how many orders are typically placed in a given time period, and what other activities — number of searches, browsing, logging, and so on — support order placement? *What is the current or predicted business volume over time? For example, how many orders are typically placed in a given time period, and what other activities — number of searches, browsing, logging, and so on — support order placement?
*How is the business volume expected to grow over time? Your projection should take into account future needs such as business growth, possible mergers, introduction of new products, and so on. *How is the business volume expected to grow over time? Your projection should take into account future needs such as business growth, possible mergers, introduction of new products, and so on.

Revision as of 05:09, 29 April 2009

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Contents

Objectives

  • Learn the difference between concurrent users and user sessions and why this is important when defining input for Web load tests.
  • Learn how to identify individual usage scenarios.
  • Learn about the metrics that will help in developing realistic workload characterizations.
  • Learn how to incorporate individual usage scenarios and their variances into user groups.
  • Learn how to identify and model special considerations when blending groups of users into single models.
  • Learn how to construct realistic workload models for Web applications based on expectations, documentation, observation, log files, and other data available prior to the release of the application to production.


Overview

The most common purpose of Web load tests is to simulate the user’s experience as realistically as possible. For performance testing to yield results that are directly applicable to understanding the performance characteristics of an application in production, the tested workloads must represent a real-world production scenario. To create a reasonably accurate representation of reality, you must understand the business context for the use of the application, expected transaction volumes in various situations, expected user path(s) by volume, and other usage factors. By focusing on groups of users and how they interact with the application, this chapter demonstrates an approach to developing workload models that approximate production usage based on various data sources.


Testing a Web site in such a way that the test can reliably predict performance is often more art than science. As critical as it is to creating load and usage models that will predict performance accurately, the data necessary to create these models is typically not directly available to the individuals who conduct the testing. When it is, it is typically not complete or comprehensive.


While it is certainly true that simulating unrealistic workload models can provide a team with valuable information when conducting performance testing, you can only make accurate predictions about performance in a production environment, or prioritize performance optimizations, when realistic workload models are simulated.


How to Use This Chapter

Use this chapter to understand how to model workload characterization, which can be used for performance testing to simulate production characteristics. To get the most from this chapter:

  • Use the “Approach for Modeling Application Usage” section to get** an overview of the approach for modeling workload characterization and as a quick reference guide for you and your team.
  • Use the various activity sections** to understand the details of the activities, and to find critical explanations of the concepts of user behavior involved in workload modeling.


Approach for Modeling Application Usage

The process of identifying one or more composite application usage profiles for use in performance testing is known as workload modeling. Workload modeling can be accomplished in any number of ways, but to varying degrees the following activities are conducted, either explicitly or implicitly, during virtually all performance-testing projects that are successful in predicting or estimating performance characteristics in a production environment:

  • Identify the objectives.
  • Identify key usage scenarios.
  • Determine navigation paths for key scenarios.
  • Determine individual user data and variances.
  • Determine the relative distribution of scenarios.
  • Identify target load levels.
  • Prepare to implement the model.


These activities are discussed in detail in the following sections.


Identify the Objectives

The objectives of creating a workload model typically center on ensuring the realism of a test, or on designing a test to address a specific requirement, goal, or performance-testing objective. (For more information, see Chapter 9 – Determining Performance Testing Objectives and Chapter 10 – Quantifying End-User Response Time Goals.) When identifying the objectives, work with targets that will satisfy the stated business requirements. Consider the following key questions when formulating your objectives:

  • What is the current or predicted business volume over time? For example, how many orders are typically placed in a given time period, and what other activities — number of searches, browsing, logging, and so on — support order placement?
  • How is the business volume expected to grow over time? Your projection should take into account future needs such as business growth, possible mergers, introduction of new products, and so on.
  • What is the current or predicted peak load level? This projection should reflect activities that support sales and other critical business processes, such as marketing campaigns, newly shipped products, time-sensitive activities such as stock exchange transactions dependent on external markets, and so on.
  • How quickly do you expect peak load levels to be reached? Your prediction should take into consideration unusual surges in business activity — how fast can the organization adjust to the increased demand when such an event happens?
  • How long do the peak load levels continue? That is, how long does the new demand need to be sustained before exhaustion of a resource compromises the service level agreements (SLAs)? For example, an economic announcement may cause the currency-exchange market to experience prolonged activity for two or three days, as opposed to just a few hours.


This information can be gathered from Web server logs, marketing documentation reflecting business requirements, or stakeholders. The following are some of the objectives identified during this process:

  • Ensure that one or more models represent the peak expected load of X orders being processed per hour.
  • Ensure that one or more models represent the difference between “quarterly close-out” period usage patterns and “typical business day” usage patterns.
  • Ensure that one or more models represent business/marketing projections for up to one year into the future.


It is acceptable if these objectives only make sense in the context of the project at this point. The remaining activities will help you fill in the necessary details to achieve the objectives.


Considerations

Consider the following key points when identifying objectives:

  • Throughout the process of creating workload models, remember to share your assumptions and drafts with the team and solicit their feedback.
  • Do not get overly caught up in striving for perfection, and do not fall into the trap of oversimplification. In general, it is a good idea to start executing tests when you have a testable model and then enhance the model incrementally while collecting results.


Determine Key Usage Scenarios

To simulate every possible user task or activity in a performance test is impractical, if not a sheer impossibility. As a result, no matter what method you use to identify key scenarios, you will probably want to apply some limiting heuristic to the number of activities or key scenarios you identify for performance testing. You may find the following limiting heuristics useful:

  • Include contractually obligated usage scenario(s).
  • Include usage scenarios implied or mandated by performance testing goals and objectives.
  • Include most common usage scenario(s).
  • Include business-critical usage scenario(s).
  • Include performance-intensive usage scenario(s).
  • Include usage scenarios of technical concern.
  • Include usage scenarios of stakeholder concern.
  • Include high-visibility usage scenarios.


The following information sources are frequently useful in identifying usage scenarios that fit into the categories above:

  • Requirements and use cases
  • Contracts
  • Marketing material
  • Interviews with stakeholders
  • Information about how similar applications are used
  • Observing and asking questions of beta-testers and prototype users
  • Your own experiences with how similar applications are used


If you have access to Web server logs for a current implementation of the application ? whether it is a production implementation of a previous release, a representative prototype, or a beta release ? you can use data from those logs to validate and/or enhance the data collected using the resources above.


After you have collected a list of what you believe are the key usage scenarios, solicit commentary from the team members. Ask what they think is missing, what they think can be de-prioritized, and, most importantly, why. What does not seem to matter to one person may still be critical to include in the performance test. This is due to potential side effects that activity may have on the system as a whole, and the fact that the individual who suggests that the activity is unimportant may be unaware of the consequences.


Considerations

Consider the following key points when determining key usage scenarios:

  • Whenever you test a Web site with a significant amount of new features/functionality, use interviews. By interviewing the individuals responsible for selling/marketing the new features, you will find out what features/functions will be expected and therefore most likely to be used. By interviewing existing users, you can determine which of the new features/functions they believe they are most likely to use.
  • When testing a pre-production Web site, the best option is to roll out a (stable) beta version to a group of representative users (roughly 10-20 percent the size of the expected user base) and analyze the log files from their usage of the site.
  • Run simple in-house experiments using employees, customers, clients, friends, or family members to determine, for example, natural user paths and the page-viewing time differences between new and returning users. This method is a highly effective method of data collection for Web sites that have never been live, as well as a validation of data collected by using other methods.
  • Remember to ask about usage by various user types, roles, or personas. It is frequently the case that team members will not remember to tell you about the less common user types or roles if you do not explicitly ask.
  • Think about system users and batch processes as well as human end users. For example, there might be a batch process that runs to update the status of orders while users are performing activities in the site. Be sure to account for those processes because they might be consuming resources.
  • For the most part, Web servers are very good at serving text and graphics. Static pages with average-size graphics are probably less critical than dynamic pages, forms, and multimedia pages.
  • Think about nonhuman system users and batch processes as well as end users. For example, there might be a batch process that runs to update the status of orders while users are performing activities on the site. In this situation, you would need to account for those processes because they might be consuming resources.
  • For the most part, Web servers are very effective at serving text and graphics. Static pages with average-size graphics are probably less critical than dynamic pages, forms, and multimedia pages.


Determine Navigation Paths for Key Scenarios

Now that you have a list of key scenarios, the next activity is to determine how individual users actually accomplish the tasks or activities related to those scenarios.


Human beings are unpredictable, and Web sites commonly offer redundant functionality. Even with a relatively small number of users, it is almost certain that real users will not only use every path you think they will to complete a task, but they also will inevitably invent some that you had not planned. Each path a user takes to complete an activity will put a different load on the system. That difference may be trivial, or it may be enormous ? there is no way to be certain until you test it. There are many methods to determine navigation paths, including:

  • Identifying the user paths within your Web application that are expected to have significant performance impact and that accomplish one or more of the identified key scenarios
  • Reading design and/or usage manuals
  • Trying to accomplish the activities yourself
  • Observing others trying to accomplish the activity without instruction


After the application is released for unscripted user acceptance testing, beta testing, or production, you will be able to determine how the majority of users accomplish activities on the system under test by evaluating Web server logs. It is always a good idea to compare your models against reality and make an informed decision about whether to do additional testing based on the similarities and differences found.


Apply the same limiting heuristics to navigation paths as you did when determining which paths you wanted to include in your performance simulation, and share your findings with the team. Ask what they think is missing, what they think can be de-prioritized, and why.


Considerations

Consider the following key points when determining navigation paths for key scenarios:

  • Some users will complete more than one activity during a visit to your site.
  • Some users will complete the same activity more than once per visit.
  • Some users may not actually complete any activities during a visit to your site.
  • Navigation paths are often easiest to capture by using page titles.
  • If page titles do not work or are not intuitive for your application, the navigation path may be easily defined by steps the user takes to complete the activity.
  • First-time users frequently follow a different path to accomplish a task than users experienced with the application. Consider this difference and what percentage of new versus return user navigation paths you should represent in your model.
  • Different users will spend different amounts of time on the site. Some will log out, some will close their browser, and others will leave their session to time out. Take these factors into account when determining or estimating session durations.
  • When discussing navigation paths with your team or others, it is frequently valuable to use visual representations.


Example Visual Representation

Personal tools