...

AnchorTestingProcessTestingProcess 

Testing

This article covers how to execute and document the results of your tests in the OLETS Jira project. In some cases, the results you need to document will depend upon which of two testing methods you employ.

User Testing is the most common testing method, and refers to the process of testing an issue by hand, linking it to the appropriate OLE issues, and recording a Selenium script.

Selenium Testing is generally only used for Bug/Defect issues, and involves selecting a pre-populated OLETS Test Case and executing the pre-recorded Selenium script to determine success or failureparticular, this article relates to how to test a new Test Case.  If you are retesting a previously failed case, revision of test steps may not be necessary.

Top of Page

...

Testing Process

• User Testing consists of the following steps:
  • Revising the Test Case
    • You will need to revisit the Test case and ensure that the steps, if they exist, adequately match the current functionality of the OLE system.
    • If testing steps do not yet exist, you will need to review the steps necessary to fulfill the purpose statement of the Test Case.
  • Executing the Steps
    • You will need to execute the testing steps by hand. Selenium will need to be running so that it can record your actions.
  • Recording Your ResultsOnce you have determined whether the test should pass or fail, you will need to advance Providing Feedback
    • Pass or fail the Test Case to advance it through the workflow, and comment with your results and any other feedback you would like to include.

...

Selenium Testing

• Selenium Testing consists of the following steps:
  • Executing the Selenium Test Script
    • You will need to open the Test Case and find the Selenium test script in the "Attachments" section. It will be included as an HTML file, with a name like "OLETS-### - Title of Test Case," where ### is the OLETS Test Case ID.
  • Recording Your Results
    • In Selenium testing, your comments will largely focus on whether the script passed or failed. If the script failed, you will need to include additional documentation, such as a screenshot or a copy of the Selenium logif there were complications or a failure, provide a comment describing the results.  In general, a comment on a passed test case is helpful, but not strictly necessary.

Top of Page
Top of Section

Determining Success or Failure

The methods by which you determine success or failure of a given test depend upon the testing procedure used. Below, you will find separate instructions for User Testing and Selenium Testing.

...

The main point on which success or failure hinges during user testing is the fulfillment of the Acceptance Criteria statement, written out as a statement of purpose on the Test Case. The AC statement is meant to give the QA Team and testers a clear, well-defined goal to meet in testing.

There may be some cases in which the function is fulfilled, but not in the exact way that was requested by the Functional Specification. The details are a secondary point, but it is sometimes necessary to fail a Test Case based on the finer details of functionality. If, for example, information needs to be displayed, but is not displayed in a way which is clear and useful to the users, a Test Case might fulfill its stated purpose yet fail on finer details.  If however that information is displayed in a way that is usable and clear, but not as clear as it could be, the Test Case would need to be passed, but with feedback indicating how it could be improved.  This feedback could later become an Enhancement (non-PP) to be reviewed by the Functional Specification teams for possible resolution.

Failure for such issues of nuance is a difficult thing to decide. The QA Team encourages you to think in terms of the big picture: does the software accomplish its task in a reasonably useful way? If so, the best course of action may be is generally to pass the Test Case, but share your insight in a comment so that we can address the issue in a later release of the software.Top of Page
Top of Section
AnchorDocumentSeleniumDocumentSelenium

Selenium Testing

In Selenium testing, success or failure depends upon the outcome of the test script. If the Selenium script finishes and reports 0 failures, the test was successful, and the Test Case should be passed. If the Selenium script reports a failure, the test has failed, and the Test Case should be failed.

There may be times at which a non-essential function causes a test script to fail. There may also be cases in which a test script fails simply because the OLE system was referring to a function in a slightly different way than previously. The QA Team will review all failures, and if a failure in Selenium testing was determined to be the result of a non-essential function or a faulty test script, the script will be revised and you will be asked to re-test the issue.

There are some commands that will suggest that a true failure has occurred during test script execution. If a command that begins with "assert" or "verify" fails, it will be highlighted in dark red in the command window, and an error message will be populated in the log window in a bold red font.

It is also worth noting that warnings may appear frequently in the Selenium log file, especially if you happen to be testing a function that involves pop-up windows or items opening in new tabs. These warnings will appear in red, but they will not be bolded, and they will be prefixed with "warn" rather than "error" in the log window. These are just informational messages, and do not signify an impending failure.

Top of Page
Top of Section
AnchorRecordingRecording

Recording Your Results

This section details a number of methods for providing feedback on the results of your test. The first and most important thing is to advance the Test Case through its workflow by selecting "Test Passed" or "Test Failed" from the menu bar at the top of the screen in OLETS. This will assign the Test Case to the QA Team, notifying us that you've finished testing the issue.

The second most important feedback method to emphasize is use of comments in OLETS. While the QA Team can determine if an issue has been passed or failed by reviewing its workflow, putting this information in a comment will allow you to make your reasons explicit, especially in the case of failure. We always need to know why a Test Case has failed, and in the case of Bug/Defect testing, we may also need to know why it passed.

Top of Page

Advance the Test Case in the Workflow

...