“Functional” Testing – Pure Permutations Approach

Brief:

This article will dive into an essential technique to calculate the exhaustive functional test scenario set of your application and achieve maximum code coverage in a quick time.

Designing functional tests for any application can face many challenges:

• Domain Knowledge
• Business requirements understanding
• Understanding the overall product/requirements can be overwhelming.
• Requirement changes
• User stories come in installments to the QA.

One implicit expectation from a quality engineer is to “test and give a sign-off/go-ahead” on a feature, bug, user story, release or in general, a ticket. Giving the go-ahead is a huge responsibility and, at the same time, drills down to “how much testing you have done!!!”

This becomes easier for smaller tickets and independent items. This decision becomes harder as the software grows and things start to spill off. Management and traceability become more and more difficult.

Bugs are missed even if you follow all processes, health checks, sanity tests, entire regression suite runs, UAT and more.

Why?

We cannot test a software exhaustively. End-to-end exhaustive testing is a myth. All combinations that will occur in a production environment with x number of variable customers and y features of an application can’t be tested in a test lab !!!

However, we can minimize this, at least, by knowing the exhaustive set, not at the production level, but right from designing the test scenarios and building it up from there and selecting the best tests from it to cover as much ground as we can.

Luckily for us, not all modules of software are used exhaustively by the end customer. We have to focus comprehensively on “30% features that are used 80% of the time”. This decision is always based on the usage and criticality of the module under test.

Creating this test scenario set is a challenge and the solution lies in a fundamental combinatorial formula. Let’s start understanding the approach using a basic customized example. Later we will dig into more complex applications.

Example: Calculator

One digit can be entered/processed at a time to keep this simple (it can do 1+1, cannot do 11+2):

• First digit A (0,1,2,3,4…9)
• Operator O (plus, minus, multiplication, division) 
• Second digit B (0,1,2,3,4…9)
• Equals and get the result

Step 1: Find the variables and create an equation.

Variables are any input parameters that affect the end result; there are three variables in our case.

{ A, O, B }

Step 2: Find the different values a variable can take

A: It can take 10 different values say a [0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]

O: It can take 4 different values o [plus, minus, multiplication, division]

B: It can take 10 different values b  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]

Now to test the core feature exhaustively. The total number of scenarios would be a * o * b.

• 10 * 4 * 10 = 400 test scenarios

“That’s it, we need to execute 400 functional test scenarios to make sure that the calculator will give correct answers for all inputs.”

Note: We are doing a subset of functional testing, UI/UX, monkey testing and more are not considered.

Step 3: Scale it by adding more variables.

If it’s a physical calculator or a desktop/web app, we can always add more variables to it.

• If the battery of the physical calculator is low

“Be thoughtful of adding variables. Every variable you add to your equation increases the count of scenarios in multiples of the values it can take”

Example:

If you add variable P;power and let’s say we test for 4 power conditions 

power = [100%, 50%, 5 %, <1%] 

Now, the equation becomes

{P,  A, O, B }

• 4 * 10 * 4 * 10 = 1600 test scenarios !!!

Check the actual test set here.

Step 4: Narrow down using software testing techniques

We will use Boundary Value Analysis and Equivalence Class Partitioning techniques to reduce the set to a practical count of scenarios.

Check the actual test set here.

Step 5: Reduce the permutations by spreading the values

In most of the conditions we have come across, spreading a new variable (low priority/impact) over the existing scenarios is the best way to go for

Example:

Instead of adding the new variable as we did in Step3, we can spread it over the existing 400 scenarios. We can do it randomly or alternatively, assigning the values as 100% charged, 50%, 5% and <1%. So our total scenarios are still 400, but we are now also testing them with power variables.

Check the actual test set here.

Step 6: Decide what to include in our tests.

The above approach will always give you an exhaustive list of test scenarios. How to reduce them will always depend on the criticality of the AUT and the variables.

Example:

Condition 1: 

User1 has developed this as a desktop/web calculator app.

Condition 2:

User2 owns a factory and manufactures physical hardware calculators.

Condition 3:

User3 has developed a desktop/web app. Instead of our basic operators, it has complex scientific operators and is meant to be used in a research facility (please consider important scientific research !!!).

For Condition 1; 

Chances of usage are meager for it, not sure if anyone will ever test it or rather use this app in the first place.

For Condition 2;

Due to the production line’s variance, multiple physical calculators, different components used, different conditions and physical hardware components. User2 might go for a big chunk of the exhaustive test set of ~600 test scenarios on 10 to 20 hardware calculators from a production batch of 1,00,000 calculators.

For Condition 3;

As the purpose of this is to be used for a research facility, the app might be tested for the entire exhaustive set and more as use cases become critical here.

Implement and do more

• This method can be used at the base level to design the test scenarios for a particular page, form or module. The variables here will be input fields on your forms:
  • Example:
    • Testing filters on any shopping site
    • Creation
      • User
        • Guest user
        • Registered user
        • User with discount coupon/promo-code
        • User with points
      • Product creation
        • Different rates
        • Discounts
        • Taxes
        • SKU (size, color…)

• Moving one level up, it can again give you a complete view of the exhaustive test scenario set. The variables here will be the modules you created in the above example.
  • Example:
    • Testing a retail checkout on the same shopping site

Here the variables are 

• Type of user (4)
• Type of products (20) can be more…
• Type of discount applied (2) cart level, product level.
• Taxes (3) tax-a, tax-b, tax-c
• Coupon / promo-code (2) yes, no
• Delivery / Pickup (2)
• COD / Payment (2)
• Payment
  • Wallet, payment gateway (2)