MFQ & PPDCS – Test Analysis and Test Design for Large Embedded Software Systems

Background Knowledge

A. Characteristics of large embedded software systems

Nowadays, embedded software makes up a significant proportion of the whole software. For example in Japan, “embedded software engineers make up a large part of the software industry, mainly because there are so many large companies producing electronics, cars etc. [3] ”

Compared with traditional fat client or browser-based desk-top software applications, large embedded software systems have following characteristics:

· Numerous and complicated functions: the size of a typical product can often reach many million lines of code, include dozens of new features for each version, involve both software and hardware functions, both O&M (Operation and Maintenance) and service handling functions, etc.

· A large number of function interactions: since embedded software often runs on a real-time operating system, any functions can be affected at any time by other events, such as by other running functions, by fixed timing tasks, or by some unexpected events (e.g. swapping or resetting of certain hardware).

· Especially stringent quality requirements: besides highly accurate functionality requirement, embedded software has to offer high quality non-functional characteristics as well, which usually includes reliability, scalability, compatibility, flexibility, and robustness, etc.

The quality requirements make the role of testers highly important and the complexity of software systems makes the job of the testers very challenging

B. Problem Investigation of Test Analysis and Design

Generally, after coding, developers should do Low Level Test (LLT), which usually includes UT (Unit Test) and IT (Integration Test), etc. before they deliver the product to testers who will be in charge of HLT (High Level Test such as system test). LLT focuses very much on single functions, while HLT focuses primarily on function interactions and quality characteristics. It is obvious that the answers of the questions of what should be tested in LLT and HLT phases and who should be responsible for each test phase can be different.

The effectiveness of single function test design, whether done by developers or testers, is often poor due to poor understanding of how to apply test design techniques, such as equivalence partitions, boundary values, decision tables, etc., This may be due to insufficient training in these techniques, and even if there exists some kind of training, the training only focuses on one or two techniques at a time, and the examples involved in the training are too simple [4] [5]. All of these factors make it difficult for learners to put test specification techniques into use. The real situation is often that people rely heavily on their experience to do test design, so the test cases set is far from being complete and effective.

Another problem in test design is that function interaction points and non-functional quality characteristics are not well considered by test analysts. Experienced based test design process relies heavily on people’s testing experience and it is not so easy to think of all related function interaction points and quality characteristics to be tested.

C. Contents of This Paper

This paper is trying to guide test analysts to choose suitable test specification techniques quickly, to build test analysis models efficiently, and to arrive at relatively efficient and complete test cases in a systematic and hierarchical way, based on a new method of building models and a test analysis and test design framework for large embedded software system.

The organization of this paper is:

· Section I is about some background information;

· Section II clarifies the conceptions of test analysis and test design;

· Section III proposes the MFQ framework for large embedded software systems;

Section IV proposes the PPDCS method to help guide the selection of test specification techniques

II.Test Analysis and Test Design

A. Different Viewpoints

What is usually called “Test Analysis and Design”, to certain degree, mixes up “Test Analysis” and “Test Design”. Since final test cases are the direct result of test design activity and not the result of test analysis activity, test analysts tend to overlook the importance of test analysis activity.

Mike Smith [4] points out that “People tend to refer to ‘the activity of Test Analysis & Design’. I would rather view this as separate activities of ‘Test Analysis’ & ‘Test Design’ leading to separate work products that may have separate structures for their organization. This better reflects the complex logical to physical relationships that usually exist between requirements and systems as implemented.”Mike Smith thinks “Test Analysis” as “What Issue”, i.e. what are the measures & targets of success for testing? And he thinks “Test Design” as “How Issue”, i.e. how will those measures and targets be achieved?

Another point of view, Torbjorn Ryber simplified testing as “a process continuously asking questions [2]”, like the model in Fig.1. It can be deduced from this model that “Test Analysis” is actually “How Issue – How do I ask questions?” and “Test Design” is actually “What Issue – What questions do I ask?”

In fact, both viewpoints are emphasizing the importance of test analysis activity. Then the question of how to do test analysis comes into play.

B.Test Analysis and Models

In the early primitive testing days, test design process is probably like “requirements/specification à test cases”. That is to say, test analysts arrive at test cases directly from requirements/specification documents, mainly based on testing experience.

Later, people begin to learn designing test cases with the help of certain test specification techniques, such as EP (Equivalence Partitioning), BV (Boundary Values), Decision Tables, etc. These test specification techniques are like tools, means, or ways used by testers to get final test cases. The activities of test analysis and test design are not evidently differentiated from each other. It can be said that test analysis activity and test design activity are going on in parallel. When one is finished, the other is over. Test design process is more like “requirements/specification à test analysis & test design à test cases”.

When software to be tested gets more and more complicated, more test analysis work needs to be done before arriving at the final test cases. For example, for large embedded telecommunication software systems, test analysts have to put much of their effort into learning the test object (the system under test), drawing the whole picture, decomposing the test object into several components so that each component is small enough to be designed, and analyzing each component with different test specification techniques. After all these test analyzing work, test cases design work can begin. Hence, test design process becomes “requirements/specification à test analysis à test design à test cases”.

It is no doubt that a highly creative and experienced tester will do a better test analysis work than normal testers, since “analysis” is an ability which can be obtained through working experience. However, that is not to say that this “analysis ability” can not be obtained or strengthened through certain methods or theory. In fact, the idea of model based testing [1] can be a very good help in improving the quality of test analysis work. Torbjorn Ryber [2] defines models: “Our models can be in the form of tables, flowcharts or other graphs which describe how the system works.” It is just like showing a city with a map; test object can be shown by models. Models can show the inner logical relationship of the test objects, in an abstraction level and in a simplified way. In fact, the process of building models is just the process of test analysis.

It can be proved by experience that there are at least three good points doing test analysis with models:

· Through the process of building models, test analysts are becoming more and more familiar with test objects and many original doubts about test objects can be clarified. During building models, many uncertain problems appear in the requirement description, and test analysts will communicate with software designers continuously to find answers for these problems. Thus, many potential faults are rectified before they really become defects in the production code. This is just the activity of preventing bugs rather than detecting bugs.

· Due to their easily understanding characteristic, models are good media for communication between testers and designers, or between the authors of test design specifications and their reviewers.

· Models are usually easily covered by test cases. By use of graphical knowledge, testers can always find ways to derive test cases from models (graphs or tables or diagrams), and the resulting test coverage for models is often better than that of traditional experience-based test design methods.

=========================================================

The following sections will introduce model based test analysis and test design techniques – MFQ & PPDCS.

III. MFQ – Test Analysis and Test Design Framework

A.Introduction of MFQ

As stated above that large embedded software systems have three evident characteristics: numerous and complicated functions, a great number of function interactions, especially high requirement in quality characteristics. Correspondingly, MFQ test analysis and test design framework for large embedded software systems includes three parts: M – Model based single function test analysis & design, F – Functions interaction test analysis & design, and Q – Quality characteristics test analysis & design.

For each of the three parts, 4-step model based testing analysis and test design method will be used, which will be detailed in part B of this section.

All three parts can be used in any test level (Unit Test, Integrated Test, or System Test), but here are some useful hints:

· In Unit or Component Test level, the first part “M – Model based single function test analysis & design” should be always used. The purpose of this is to make sure that independent single functions are thoroughly tested before they are integrated with other components in higher test level.

· In System Test level, the second part “F – Function interaction test analysis & design” and the third part “Q – Quality characteristics test analysis & design” should be used as thoroughly as possible. This is to guarantee the accuracy of functionalities and quality characteristics for the whole system.

· It depends on the context of projects and people’s technical skills whether or not to apply M and Q in lower test levels. If possible, some quality characteristics can be verified earlier in the project. For example, robustness is a very important aspect of component test.

Figure 2. MFQ framework

· When the software is switched from lower test level to higher test level (generally from developer team to tester team), a delivery test is usually done by testers to verify basic functionalities. So, the first part “M – Model based single function test analysis & design” is also needed in higher test level.

B.4-step Test Analysis & Test Design Procedure

Figure 3. 4-step procedure

MFQ framework uses 4-step test analysis & test design procedure, the details of which can be found in [2]. Here is some simple introduction of it.

Step 1: Build models for test objects.

Good analysis models are the key for success. However, good models are built on the basis of familiarity with test objects. It is especially true for large embedded software systems since firm business background knowledge is forever the foundation for a good test analysis and design work. So the very first step before doing any test analysis is to collect as much information as possible about a test object so that deep product knowledge can be obtained. This information can be all kinds of design documents at hand such as feature design specification, software requirement specification, high level design specification and low level design specification, etc. After having good knowledge of the test object, test analysts can try to select a suitable model to depict the test object. With many popular test specification techniques available such as EP (Equivalence Partitioning), BV (Boundary Values), Decision Tables, Business Process, State Based Testing, etc., test analysts will often find that there are too many choices in building a model and they are quickly getting into the situation of not knowing which one to choose. As is known that multiple models can be used to depict one test object, but often one of them is most suitable for use. The idea of PPDCS trying to solve this problem will be presented in the next section of this paper.

Step 2: Design base test cases (sometimes called logical or abstract test cases) to cover the model.

The so called base test cases refer to rough test cases without specified values for test data in test cases. Different from traditional one-step test cases generation process, the generation of test cases in 4-step procedure needs two-step: first is to design base test cases, second is to add more test data for each base test case so as to create final executable test cases.

The purpose of designing base test cases is to get a better model coverage. Different models have different test coverage methods. In recent years, many researchers studied model based testing in the aspect of test cases auto-generation from existing models according to certain generation rules or algorithms. This paper focuses more on the model building process and focuses on manual-generation of test cases from models because in my experience the work of building models takes most of the time while test cases generation from models is relatively easy and not time-consuming.

Moreover, in this paper, the conception of “model” is extended, i.e. a model is not necessarily being expressed with UML or other certain languages. In fact, a model can be in any form of tables, graphs, trees, etc., as long as it helps us depict test objects clearly. Based on my experience, most test objects can be expressed with simple models. I believe that test specification techniques need not be very complex in most situations. When a difficult-to-learn test specification technique needs to be used before one can start test design process for a certain test object, I would propose that the system design or requirement design of this test object needs a rework.

Step 3: Consider variations of test data to finalize test cases (sometimes called physical or concrete test cases)

If the purpose of step 1 is to ensure requirements coverage with models (using models to cover requirements), and the purpose of step 2 is to ensure models coverage with base test cases (using base test cases to cover models), then the purpose of step 3 is to ensure base test cases coverage with test data (using test data to cover each base test case). Some test data are involved in one base test case, so the first thing to do is to identify test data, and the second thing to do is to consider variations of the test data using EP (Equivalence Partitioning) and BV (Boundary Values) (Note: EP and BV are somehow special because they also can be used in step 1 to build models.). For each variation of test data, design a test case. After step 3, final executable test cases are complete.

Step 4: Advanced Testing

There isn’t any one type of model which can effectively describe all aspects of a test object. Even if the above three steps are used, there are still many other aspects of the test object that need to be tested. E.g. some special restriction relationship of variables in specification is hard to put into models, so a separate test design is needed; the other thing is people’s experience. Test analysts often have their own understandings of test objects which are hard to put into models, so a further test designing process is needed.

Good testing = formal testing + informal testing. The former three steps are formal test design procedures, and the last step “advanced testing” is informal testing procedure. In fact, informal testing does not mean random testing without any rules to follow. There are mature methods for informal testing, such as error based testing [6], exploratory testing [11], etc., which will not be detailed in this paper.

C.M – Model based Single Function Test Analysis & Design

The above “4-step model based test design procedure” is most suitable for single function test design due to the suitable granularity of single function, so this part is represented by “M” (Model).

In applying the 4-step procedure for single function test analysis & design, the very first thing is to divide the test object into different “single functions” (“units” or “components”). The size of a single function can be dozens of lines of code (LOC) to hundreds of LOC, but it is better not to exceed one thousand LOC according to my experience. One single function can correspond to 1 or several components during unit testing, or 1 or several requirements in an SRS (Software Requirement Specification), or one or two use cases. There are no definite rules specifying which documents should be referenced and what requirements or specifications should be corresponding to a single function. Therefore, test analysts often need to collect enough materials to do test analysis to identify suitable single functions to be tested.

In object-oriented programs, single functions may be relatively easy to identify. The responsibilities implemented by the methods (member functions) of an object or a class can be regarded as a single function of the overall system. But in other languages like C, it is not so easy to identify single functions. Generally, a single function possesses two characteristics:

· From a requirements point of view a single function is relatively independent. A software system may be composed of many features; a single feature may decompose into many individual functions.

· From an implementation point of view, the inner logic of a single function is coherent. For example, an object’s behavior may be implemented by many methods (member functions); a method may consist of many steps.

Similar to the INVEST model for User Stories as described by Bill Wake [14], a single function should be independent, testable, valuable (realizing specific function), sizable (context dependent, no fixed range of size), and negotiable (The division of single functions can be adjusted with the work of test analysis and test design going on.).

After identifying single functions from features or system, the next step is to do test analysis and test design for each single function using 4-step procedure. Section IV will describe it in detail.

D.F – Function Interaction Test Analysis & Design

“F” represents function interactions.

After test analysis and design for single functions, how can we deal with the relationship and interactions between single functions? We can use the following 4-step procedure to do function interaction analysis.

Note: the word “feature” will be used in following tables instead of “single function”, since “F” and “Q” parts are frequently used in a feature level.

Step 1: Build models.

· List legacy functions which are related with the functions you are testing (the testing functions). Their relationship can be “Interaction” or “Modified”. “Interaction” means legacy functions and testing functions need to coordinate to do something. “Modified” means legacy functions need to be modified with the introduction of new testing functions.

· List other new functions which are related with the testing functions. Their relationship can be timing relationship (one running before the other, or the two concurrently running), spacing relationship (either using same resources like timers or memories, or communicating with messages, etc.) or any other relationship.

· Put the legacy functions and new functions into the first row of a table, and put the testing functions into the first column.

· Mark the cell if a column and a row have interaction point with “X”.

TABLE I. Function Interaction Table

Step 2: Design base test cases.

For each of the checked cells in the table, we design one or several base test cases – each base test case clearly describes the relationship between the two interacting functions. Table II illustrates this step.

TABLE II. Function Interaction Points – Base Test Cases

Base test case 1FIP (function interaction point) 1 between Fn1_1 and Legacy fn 1 : xxxxxxxxBase test case 2FIP (function interaction point) 2 between Fn1_1 and Legacy fn 1 : xxxxxxxxBase test case 3FIP (function interaction point) 1 between Fn1_2 and Legacy fn 2 : xxxxxxxxBase test case 4FIP (function interaction point) 1 between Fn1_2 and New fn 1 : xxxxxxxx…………

Step 3: Supplement with test data.

Identify variables in each FIP base test case, and use EP and BV to get more test data, completing test cases.

Step 4: Advanced testing.

Try to get more test cases based on tester’s experience. James Bach’s Heuristic testing method [12] can be a very good reference in this step.

E.Q – Quality Characteristics Test Analysis & Design

“Q” stands for quality characteristics.

Besides functionality test analysis and test design, other quality characteristics should also be considered.

Step 1: Build models.

· Choose and define non-functional quality characteristics pertinent to the product to be tested. ISO 9126 [13] has a good definition of quality characteristics and sub characteristics.

· Put all quality characteristics into a row of a table, and put each component, function or feature into the first column.

TABLE III. Quality Characteristics Definition

…………

· Mark the cell if a column and a row have interaction point with “X”, which means a certain quality characteristic is needed to be considered for this component, function or feature. The analyzing granularity is not fixed here. Either component level or function level or even feature level is ok. The example in Tab. IV is based on function level.

TABLE IV. Quality Characteristics Analysis Table

Step 2: Design base test cases.

For each of the check point in the table, design one or several base test cases – describe clearly the quality characteristics to be verified.

TABLE V. Quality Characteristics Interaction Points – Base Test Cases

Base test case 1QCIP (QC interaction point) 1 between Fn1_1 and QC1 : xxxxxxxxBase test case 2QCIP (QC interaction point) 2 between Fn1_2 and QC2 : xxxxxxxxBase test case 3QCIP (QC interaction point) 1 between Fn1_2 and QC3 : xxxxxxxxBase test case 4QCIP (QC interaction point) 1 between Fn1_3 and QC3 : xxxxxxxx…………

Step 3: Supplement with test data.

Recognize variables in each QCIP base test case, and use EP and BV to get more test data, completing test cases.

Step 4: Advanced testing.

Try to get more test cases based on tester’s experience. Again, [12] can be a good help.

The good point of using “tables” as models for “F” and “Q” part is that it is not easy for test analysts to forget some testing points that are related.

This section talks about the MFQ framework. “Tables” are used to draw models for function interactions and quality characteristics test analysis. But for single function test analysis, the types of models can be various. Next section talks about this.

IV.PPDCS – Choose appropriate test techniques to build models

A.Introduction of PPDCS

As is said above that 4-Step procedure will be used for each of the M, F and Q test analysis and test design process. Among them, step 1 is about test analysis and is the most important step. For F and Q parts, step 1 is easy and tables can be used as models. But for M part, step 1 is probably the most difficult step for most test analysts. Facing numerous test design techniques and facing testing objects with different characteristics, testers are often wondering about which technique to choose to build a good model?

The key problem of test design is “What subset of all possible test cases has the highest probability of detecting the most errors? [8]”. Knowing how to select suitable test analysis models is beneficial to solve this problem. This section presents PPDCS method, which is a technique for selection of one or more test techniques.

On one hand, when analyzing different test design techniques, it can be found that most test design techniques can be classified into 5 major categories, each with an obvious characteristic. For example “Business Process” technique is characterized by its dealing with processes characteristic. Usually a process is composed of several steps, and this technique uses activity diagram or flow graph to show the whole process.

On the other hand, when analyzing different test objects (often through requirement specifications), it can also be found that test objects have similar characteristics. For example, a specification about ATM machine can tell a whole story about withdrawing money “from inserting a card, to verifying validity, to checking password, to dealing with transactions, to ejecting card”. So this specification also has the characteristic of dealing with processes.

When the two characteristics match, test analysts can use the specific test design technique to model the specifications. This is just the original thought of PPDCS, in which each letter represents a specific characteristic. The first letter “P” means “Process”, the second letter “P” means “Parameter”, the third letter “D” means “Data”, the fourth letter “C” means “Combination”, and the last letter “S” means “State”. Each of these characteristics and the model based single function test analysis and test design process will be further described in detail in the following parts of this paper.

Figure 4. PPDCS idea

B. P-Process

1) Application Condition

If any hint of “process” characteristic exists in design specifications of test objects, “P-Process” approach can be used to build models.

The features of “process” include:

· Multiple steps compose one process, with the sequential relationship between steps.

· One or more than one actors or triggers are involved.

Design specifications with characteristic of “P-Process” often include a serial of steps or use cases.

2) Application Steps

Step 1: Build models.

Test objects with characteristic of “P-Process” can be modeled with flow graph oractivity diagram, with the help of “Business Process [2]” testing technique.

During building models, test analysts should communicate with product designers frequently. Any possible abnormal branch of the process should be considered. Test analysts should ensure that design specifications have been accurately pictured with models. If the flow chart has already been included in specification documents, test analysts need carefully review it or even redraw it because the process of building a model is very helpful for thorough understanding of the test object. The following flow graph takes the single function “withdrawing money” of ATM machine as an example.

Figure 5. Modelling for ATM single function “Withdrawing money”

Step 2: Design base test cases.

There are two ways to cover the model with base test cases. One way is to design base test cases to cover each path of the flow graph, when the flow graph is simple. The other way is to design base test cases to cover “main flow + important alternative flows”, when the flow graph is rather complex. Each path or flow is represented by one base test case.

The base test cases for the ATM “withdrawing money” single function are listed in the following table.

TABLE VI. example of base test cases

No.Base test casesPriorityATM-UNIT1-BTC001

A0 main flow: withdrawing money successfully.

ATM-UNIT1-BTC002

A0-A1: wrong card.

ATM-UNIT1-BTC003

A0-A2: wrong passward, 3 times.

ATM-UNIT1-BTC004

A0-A3: wrong amount , for once.

Step 3: Supplement with test data.

For each base test case, identify related variables, and add more test data using EP or BV so that the final executable test cases can be obtained.

Step 4 Advanced Testing.

Are there any other processes possible? What else should be considered while not shown in the flow graph?

C.P-Parameter

1) Application Condition

If any hint of “variables or parameters” characteristic exists in design specifications of test objects, “P-Parameter” approach can be used to build models.

The features of “parameter” include:

· No evident processes exist in design specifications, but “multiple parameters” are involved.

· Multiple rules are included in design specifications. Each rule is composed of the combination of different values of different variables.

· The number of parameters involved is limited. Test analysts can easily recognize the logical relationship between parameters.

2) Application Steps

Step 1: Build models.

Test objects with characteristic of “P-Parameter” can be modeled with tables, or tree graphs or coordinate graphs, with the help of “Decision Table, Decision Tree, or Domain testing [2]” test technique.

TABLE VII. Decision Table

RulesParameter 1Parameter 2ResultValue1.1Value2.1Value1.2Value2.21XXXXXXXXXXXX2XXXXXXXXXXXX

Step 2: Design base test cases.

There are two ways to cover the model with base test cases. One way is to satisfy 100% rules coverage, i.e. designing one base test case for each of the rules in the decision table. This method is effective when the decision table is simple (not too many parameters involved). The other way is to convert decisions tables to decision trees, designing one base test case for each leaf of the tree. This method is effective when the decision table is rather complex. One advantage of converting decision tables into decision trees is that some leaked or erroneous requirements can be easily identified during the conversion.

When multiple conditions exist in one decision, ECT [2] (Elementary Comparison Testing, also called MCDC (Modified Condition Decision Coverage)) can be used to generate base test cases.

Step 3: Supplement with test data.

For each base test case, mark the related variables, and add more test data using EP or BV.

Step 4: Advanced Testing.

Supplement with ad hoc test cases based on experience.

D.D-Data

1) Application Condition

If any hint of “data” characteristic exists in design specifications of test objects, “D-Data” approach can be used to build models.

The features of “data” include:

· Each data has its special value range.

· Unlike “parameters”, no obvious “rules” or logical relationships exist between data.

· It is possible that some restrictions exist between the value ranges of different data.

· The number of data involved is limited.

For example, in such a specification “When constructing a building, one room can have at most 4 windows”, two kinds of data can be identified, that is “the number of rooms” and “the number of windows”.

2) Application Steps

Step 1: Build models.

Test objects with characteristic of “D-Data” can be modeled with tables, with the help of “EP (Equivalence Partitioning) or BV (Boundary Values)” [8] test technique.

TABLE VIII. EP & BV Table

DataValues123456

“the number of rooms” Valid Group

1~200XX

“the number of rooms” Invalid Group

0, >200X

“the number of rooms” Valid boundary values

1,200XX

“the number of rooms” Invalid boundary values

0,201X

“the number of windows” Valid Group

0~4XX

“the number of windows” Invalid Group

>4X“the number of windows” Valid boundary values0,4XX“the number of windows”
Invalid boundary values5X

Step 2: Design base test cases.

Design base test cases covering each valid group and invalid group, at the same time considering all kinds of combinations of valid boundary values and invalid boundary values.

Step 3: Supplement with test data.

For each base test case, mark the specified values for each data.

Step 4: Advanced Testing.

Supplement with ad hoc test cases based on experience.

E.C-Combination

1)Application Condition

In the cases of “P-Parameter” and “D-Data”, if the number of parameters or data is too big to be listed manually, “C-Combination” approach can be used.

The features of “combination” include:

· There exist a great number of parameters (or data).

· Each parameter has multiple values.

· It is possible that some logical relationship exists between parameters.

2)Application Steps

Step 1: Build models.

The specification of “D-Data” can be modeled with “Factor-State Tables”, with the help of “Combinatorial Testing” [2] (Pair wise, 3-wise, etc.), or Orthogonal Trail Design[9] test technique.

For example, given four factors as shown in the factor-state table in Table VIX, testing all combinations would require 72 (=2*3*4*3) test cases. We can reduce the number of test cases with a minimal increase in risk by ensuring that each value of each factor is tested at least once with each other value of each other factor. With this approach only 12 test cases are needed.

TABLE IX. Factor-State Table

Factor AFactor BFactor CFactor D1A1B1C1D12A2B2C2D23B3C3D34C4

Step 2: Design base test cases.

There are a lot of tools for pair wise in the website http://www.pairwise.org/tools.asp. Among them, PICT is an easy-to-use tool.

The process of designing base test cases with PICT is a form of functional programming using PICT’s special “language”. Once we have defined the rules we can run this script in a DOS environment. All the base test cases are generated automatically.

For example, if the following rules are defined in the file “model_test.txt”:

Factor A: A1, A2

Factor B: B1, B2, B3

Factor C: C1, C2, C3, C4

Factor D: D1, D2, D3

When we execute the command:

C:\>pict model_test.txt >output.xls

The following 12 base test cases are saved into output.xls automatically:

TABLE X. The result of pair wise method

Factor AFactor BFactor CFactor DA1B1C2D2A2B2C3D2A2B3C1D1A1B2C2D3A1B1C3D1A2B2C4D1A2B2C1D3A1B3C3D3A2B3C2D1A2B1C4D3A1B3C4D2A1B1C1D2

These twelve combinations are all that we need to test to ensure that each value of each factor is tested at least once with each other value of each other factor.

Step 3: Supplement with test data.

For each base test case, define a value for each of the parameters involved. For example, if “A1” represents “>50”, take a value of “60” for it.

Step 4: Advanced Testing.

Supplement with ad hoc test cases based on experience.

F.S-State

1)Application Condition

If any hint of “state” characteristic exists in design specifications of test objects, “S-State” approach can be used to build models.

The features of “state” include:

· The behavior of the test object varies based on its internal state.

· Certain events trigger a change of the test object’s state.

2)Application Steps

Step 1: Build models.

The specification of “S-State” can be modeled with “State graphs”, with the help of theState Based Testing [2] technique.

In a state graph, states can be expressed by nodes and events can be shown by the arcs connecting the nodes.

The steps of modeling are:

· Identify a behavior object from the specifications.

· Determine all possible states for this behavior object.

· For each state, draw a node.

· Identify all possible transitions between states.

· For each transition, determine the event that triggers it.

· For each event, drawing a connection between corresponding nodes.

Fig. 6 illustrates a typical state graph.

Step 2: Design base test cases.

Many methods can be used to design base test cases to cover state graphs. One is Chow’s method [10].

Chow’s 0-switch coverage approach involves defining one base test case for each transition in the state graph. Chow’s 1-switch approach is to design one base test case for every possible transition of each node.

Figure 6. state graph

Table XI lists base test cases for above state graph using Chow’s 0-Switch Coverage approach. Table XII lists the base test cases using Chow’s 1-Switch Coverage approach. Note how the number of base test cases is increased by considering pairs of transitions rather than single transitions.

TABLE XI. Chow’s 0-Switch Coverage

Base test casesEVENTSSTARTEND1aV1V22bV2V33cV3V14dV1V35eV3V2

TABLE XII. Chow’s 1-Switch Coverage

Base test casesSTARTNEXTEND1V3V1V22V3V1V33V1V2V34V3V2V35V2V3V16V1V3V17V2V3V28V1V3V2

Step 3: Supplement with test data.

For each base test case, we identify the variables involved and define test data using EP and BV methods. This results in each abstract test case being expanded into one or more concrete test cases we can execute.

Step 4: Advanced Testing.

Supplement with ad hoc test cases based on experience.

This section talks about the PPDCS method. For single function test analysis and test design, the first thing to do is to identify the number of single functions (or called units or components). For each single function, what test analysts need to do is: building a model using PPDCS method, designing base test cases to cover the model, supplementing with more test data for each base test case, and designing other test cases based on experience.

V.RESULTS

Several pilot projects were conducted in which developers were taught to use MFQ & PPDCS methods in Huawei. Table XIII compares testing result for M part (single function part of MFQ framework) between traditional test design methods and PPDCS method for one feature of a product.

86 test cases were designed by traditional testing methods (mainly experienced based testing methods) by the developer. Because no evident test analysis process existed before test cases design process, 2 components of this feature were neglected by the developer when designing test cases (The step of identifying the number of single functions was not performed at all.). After running of all 86 test cases, only 1 defect was detected by this developer.

For the purpose of comparing testing effectiveness between traditional way of test design method and the new way of PPDCS method, a new test analyst was designated to redo the test design of this feature with PPDCS method. As a result, 102 test cases were designed. All of 4 components were correctly identified and more function points of this feature were covered by test cases. At the end of test analysis and test design process, 5 defects were detected, even though no test case had been executed yet by that time.

TABLE XIII. comparison between different testing methods

ItemsTraditional testing design methodsModel based PPDCS method1. The number of test cases86 testcases;

Less than 10% negative test cases102 testcases;

About 50% negative test cases2.The sufficiency of test casesThe feature actually contained 4 components, but only 2 components were covered by test cases due to the lack of a formal test analysis process. Furthermore, the test cases ony covered basic function points of the components.When building models and doing test analysis, all of 4 components were successfully identified and covered well by test cases.

For each of the components, more function points were covered by test cases.3. Testing techniques appliedMainly experienced based testing techniques were used.Formal testing techniques: EP & BV (D of PPDCS), Pairwise (C of PPDCS), flow graph testing (P of PPDCS);

Informal testing techniques: experienced based testing.4. Testing result1 defect was detected after running all 86 TCs.5 defects were detected before the execution of any test cases.

The following conclusions were drawn from these pilots:

· It was found that developers can quickly learn and do a better job of Unit test analysis and test design with the help of PPDCS, even with little prior test design knowledge.

· Many potential defects are prevented during the process of building models before they enter into production code, while traditionally these defects are detected by testers after the implementation phase of the software. This is because the process of building models is at the same time the process of test analysis, in which design specifications are generally reviewed by test analysts and some potential problems can be identified in an earlier phase.

· Since PPDCS is a black box testing method, when combined with white box testing method (i.e. to supplement with test cases with white box testing techniques such as statement coverage, branch coverage, path coverage, etc.), good test coverage can be achieved.

· Developers can easily incorporate the test analysis process into their software analysis process and this resulted in a much better software design.

VI.Summary

To sum up, the test analysis and test design process for large scale embedded software systems can be:

· In lower level testing such as Unit test, single functions (“M” part) are thoroughly tested. In higher test level such as System Test, function interactions (“F” part) and quality characteristics (“Q” part) should be thoroughly tested. However, this is not the absolute case.

· In each of the M, F, or Q part, 4-step test analysis and test design procedure will be used.

· For the M part, we start by decomposing the test object into features and individual functions. We use the PPDCS framework to help us choose a suitable test specification technique to build a model for each function.

· For the F and Q parts, interaction-point tables are used to build models. James Bach’s heuristic testing strategy model is a useful supplement for this purpose.

Search This Blog

Effective Test strategy-Technique-Methdology for high test coverage