ExtentReport Review(by: Rogelio “CAGS” Caga-anan Jr.)

Comparison between- extentreports-java-2.031.jar

and- extentreports-java-2.4.jar

Pseudocode// verifyIssueWorkflow Testparent = report.startTest("verifyIssueWorkflow","verify that the Issue workflow functions as expected");

// create child report - Login child = report.startTest("Login");

// log child steps resultschild.log(LogStatus.PASS,"type username","OK");child.log(LogStatus.PASS,"type password","OK");child.log(LogStatus.PASS,"click login","OK");child.log(LogStatus.PASS,"verify landing page","OK");

// append this child parent.appendChild(child);

// create child report - Goto Zone child = report.startTest("Goto Zone");

// log child steps resultschild.log(LogStatus.PASS,"select Zone","OK");child.log(LogStatus.PASS,"click login","OK");child.log(LogStatus.PASS,"verify landing page","OK");

// append this child parent.appendChild(child);

// create child report - Select Trade Application child = report.startTest("Select Trade Application");

// log child steps resultschild.log(LogStatus.PASS,"select Application","OK");child.log(LogStatus.PASS,"click OK","OK");child.log(LogStatus.PASS,"verify landing page","OK");

// append this child parent.appendChild(child);

// create child report - Logout child = report.startTest("Logout");

// log child steps resultschild.log(LogStatus.PASS,"click Exit","OK");child.log(LogStatus.PASS,"click Logout","OK");child.log(LogStatus.PASS,"verify landing page","OK");

// append this child parent.appendChild(child);

// based on the logStatus of the child steps, log status as parent test statusparent.log(LogStatus.PASS,"verify Issue Workflow","PASSED");

report.endTest(parent);

// verifyAmendWorkflow Testparent = report.startTest("verifyAmendFlow","verify that the amend workflow functions as expected");

// create child report - Login child = report.startTest("Login");

// log child steps resultschild.log(LogStatus.PASS,"type username","OK");child.log(LogStatus.PASS,"type password","OK");child.log(LogStatus.PASS,"click login","OK");child.log(LogStatus.PASS,"verify landing page","OK");

// append this child parent.appendChild(child);

// create child report - Goto Zone child = report.startTest("Goto Zone");

// log child steps resultschild.log(LogStatus.PASS,"select Zone","OK");child.log(LogStatus.PASS,"click login","OK");child.log(LogStatus.PASS,"verify landing page","OK");

// append this child parent.appendChild(child);

// create child report - Select Trade Application child = report.startTest("Select Trade Application");

// log child steps resultschild.log(LogStatus.PASS,"select Application","OK");child.log(LogStatus.PASS,"click OK","OK");child.log(LogStatus.PASS,"verify landing page","OK");

// append this child parent.appendChild(child);

// create child report - Logout child = report.startTest("Logout");

// log child steps resultschild.log(LogStatus.PASS,"click Exit","OK");child.log(LogStatus.PASS,"click Logout","OK");child.log(LogStatus.PASS,"verify landing page","OK");

// append this child parent.appendChild(child);

// based on the logStatus of the child steps, log status as parent test statusparent.log(LogStatus.PASS,"verify Issue Workflow","PASSED");

report.endTest(parent);

Psuedo code is meant to represent

2 Parent nodes - verifyIssueWorkflowTest - verifyAmendWorkflowTest

Each parent has 4 child nodes and associated steps - Login

- type username- type password- click login- verify landing page

- Goto Zone- select zone- click ok- verify landing page

- Select Trade Application- select Application- click ok- verify landing page

- Logout- click Exit- click Logout- verify landing page

Each Parent node has1 call to startTest()- parent = report.startTest()AND 1 log representing the status of the test- parent.log (LogStatus.PASS,”Pass”)

Each associated step is represented by a log- child.log(LogStatus.PASS,”type username”,”OK”);

Each child node is appended to parent- parent.appendChild(child);

Each child node is represented by a call to startTest- child = report.startTest()

Code Structure using TestNG@Test (priority=1)public void verifyIssueWorkflow(){

parent = report.startTest("verifyIssueWorkflow", "verify that the Issue workflow functions as expected"); // insert child node and logs child = report.startTest(“Login”); child.log(LogStatus.PASS,”type username”,”OK”); : // append child parent.appendChild(child);

// insert more child node and logs as needed // must do a parent.log to show details in summary view if (testStatus){ parent.log(LogStatus.PASS, "verifyIssueWorkflow", "Status: PASSED"); } else { // need to log this to parent if at least 1 test step failed parent.log(LogStatus.FAIL, "verifyIssueWorkflow", "Status: FAILED"); } // close the report report.endTest(parent);

// force fail the test if any test step fails if (!testStatus){ Assert.fail(); }}

TestNG records this as a Test, corresponding to the count in Test View in v2.031

TestNG records this as the status of the Test, corresponding to the parent.log() in v2.031

Test View in v2.04 counts all report.startTest() as a Test in Test View

Test View in v2.04 the status of the parent is based on the last RunStatus as seen in the Quick Test Summary and not the no. of parent = report.startTest() calls

Total test count is based on the number of report.startTest() calls, hence there is always an additional 1 test counted which is the parent = report.startTest() call instead of only counting child = report.startTest() calls

Running the code Using extentreports-java-2.031.jar

1. Quick Test Summary reflects correct status of parent (verifyIssueWorkflow & verifyAmendFlow) based on parent.logStatus()

3. Test view reflects correct test count graph and matches with Quick Test Summary

2. Quick Test Summary reflects correct runStatus, based on the last logStatus of the child node

4. Pass Percentage reflects correct value based on Test View

Reflects 28 steps instead of 26

Reflects 2 Failed steps instead of 1

In Step View, all child nodes are taken as a step, hence the extra 2 steps that could be brought by calling parent = report.startTest() & appendChild() for verifyIssueWorkflow and verifyAmendFlow

Expanding verifyIssueWorkflow

Child Node representing LoginChild logStatus -> Login Steps

Child Node status is taken from last runStatus

There are 13 steps for verifyIssueWorkflowLogin has 4Goto Zone has 3Select Trade Application has 3Logout has 3

Steps View counts 14, including the additional step mentioned above

Status taken from parent.logStatus()

This line is counted as 1 step possible side effect when calling - parent = report.startTest()

Expanding verifyAmendFlowStatus taken from parent.logStatus()

Child Node status is taken from last runStatus

This line is counted as 1 step as a side effect of calling - parent = report.startTest()

For v2.031 & v2.04, it would be more accurate if these are the only items counted as test stepsWhich means there would be:13 steps, 12 passed, 1 failed

Should be the same for verifyIssueWorkflow

There are 13 steps for verifyAmendFlowLogin has 4Goto Zone has 3Select Trade Application has 3Logout has 3

Steps View counts 14, including the additional step mentioned above

Running the code using extentreports-java-2.04.jar

1. In Quick Test Summary, only LogStatus used is reflected

Steps View values is now the same as the Quick Test View.

Still has 28 steps total instead of 26

Test View reflects the no. of child nodes instead of the no. of parent nodes

Expanding the parent nodes – verifyIssueWorkflow and verifyAmendFlow,

Reflected in Test View as 10 tests8 passed, 2 failed

For v2.04, it would be more accurate if only these items are counted as tests.

Which means there would be:8 tests total7 passed1 failed

Count in Steps View for v2.04 is the same as that in v2.031

Comparing TestNG result with ExtentReport using v2.0311. Quick Test Summary matches TestNG @Test status2. Total time reported by ExtentReport and TestNG are similar3. Status of TestNG test class match with the count in Test View4. Additional Run status indicates status of last child.log() status

21

3

4

Comparing TestNG result with ExtentReport using v2.041. RunStatus matches TestNG @Test status2. Total time reported by ExtentReport and TestNG are similar3. Status of TestNG test class does not match with the count in Test

View

1

2

3

Findings• Test View using v2.031 is correct based on the pseudo code provided as it

counts the actual top level test (parent node)• Test View using v2.04 is correct based on definition of a test because

parent node based on the pseudo code represents a test scenario (sequence of tests) rather than a single test

• In both versions, the Steps View does not count the steps based on the pseudo code as an additional step is counted possibly as a side effect of the parent = report.startTest() and the parent.appendChild() methods.

• Changes have been implemented in v2.04– [Feature] Show Child test's step counts in Quick Test Summary #44 – [Feature] Quick Test Summary - Show only Headers which have at least 1 count #45

Conclusions• Both versions, depending on intended reporting schema, can be used

independently. End users would benefit greatly if the reporting schema is configurable– Schema 1 : Parent node is counted as test (v2.031)– Schema 2 : Child node is counted as a test (v2.04)

• Depending on user preferences, user might expect that test framework used (ex. TestNG or Junit) would have the same report results as ExtentReports

• Revisit Test Step count to reflect the actual test steps without adding the additional step that could be brought about by parent = report.startTest() and appendChild() methods