The test campaigns

Here is a typical campaign tree. It immediately shows a number of useful information:
  • the total number of campaigns and campaign sessions
  • the number of campaigns and campaign sessions in each category
  • the number of campaigns and campaign sessions in each folder
  • the status (stopped, paused, running, idle) of each campaign session (indicated by the overlay on the icon)
  • the start and stop date and time of each campaign session

Highslide JS


Once all the tests have been defined and implemented, you have to run them. To do so, you’ll first have to gather all the tests you wish to run in a campaign. For instance, it may be of interest to define a campaign including all the tests of a specific category (or just a subset). A campaign is by definition just an ordered list of tests.

At this stage, you would be able to run a campaign but what happens if you run a campaign several times? of course you want to be able to retrieve results from each runs of a campaign. Here comes the Campaign Session.

You can create from a campaign as many campaign sessions as you want (there is an easy "duplicate" procedure). This allows archiving independently all the results and report of each execution of the tests.

Progress Metrics

In order to follow the progress of the test executions, the campaigns (sessions) includes a progress column showing at a glance the estimated progress of each campaign (session) in the tree.
The progress is also consolidated at the folder/root folder level.


The progress indicator provides the percentage of execution of the tests in the campaign session.




Depending on what is selected in the drop-down menu, the figure on the right-hand side of the progress bar will indicate:
  • Test cases executed: how many test cases still need to be executed
  • Time spent: an estimation of the time necessary to complete the execution


The progress of each session is calculated based on:
  • the number of test cases in the session,
  • the number of test cases executed in the session.





Progress aggregation

For each folder or campaign the progress is calculated by aggregation, based on the progress of its underlying objects.


For instance, the progress of a campaign will be calculated using the formula:




Quality Metrics

In order to have a quantifiable follow-up of the quality, the campaigns (sessions) includes a coverage column showing at a glance the estimated quality of each campaign (session) in the tree.
The quality is also consolidated at the folder/root folder level.


The quality indicator provides the estimation of the percentage of success of the latest executions of the tests functionally covering the SUT. The figure on the right-hand side of the quality progress bar indicates how many links are linking the SUT to its successful/total test cases. If the SUT is linked several times to the same test/test case through different path, it is counted as different links.





For each node of the corresponding trees (folders), quality will be calculated taking into account the share of successful related tests.

The quality indicator will be an integer with possible values from 0 to 100.
  • 0% value means no test linked to the current object is successful.
  • 100% value means that coverage is optimal and all linked tests are successful.


The generic quality formula for a session is:



Quality aggregation

For each folder or campaign the quality is calculated by aggregation, based on the quality of its underlying objects.


For instance, the quality of a campaign will be calculated using the formula:




Searching

By name

All the trees in XStudio include an indexing system allowing to search real-time an entity in the tree by his name. To experiment it by yourself just start entering some text in the Search field of the tree

Highslide JS


To automatically select a searched item in the tree, you can move using the arrows keys within the suggestion list and validate by pressing Enter or just click on the item in the list.

By Id

The data model gives a unique identifier for each entity managed by XStudio. The Ids are on the form:

Template Represents
C_<id> Campaign
CS_<id> Campaign session
SC_<id> Schedule


Ids are also available from XStudio’s GUI (in the Details tab there is an Identifier field that corresponds to the <id> in the template.

To search for an entity from its Id, just type the Id (i.e. SU_123) in the Search Id field and validate.

The root node

Toolbar

Create a folder

A folder can be created only under the root node or another folder node.


Here is the process to create a new folder:
  • select a root node in the tree
  • on the right panel, click on the create folder button
  • enter the name of the folder and submit
  • the folder immediately appears in the tree


The folder nodes

Tabs

Details

The folder details tab allows you to specify the name of the folder


Highslide JS

Attachments

Attachments are handled the same way than any documents in the documents tree.

Toolbar

Create a folder

A folder can be created under another folder node.


Here is the process to create a new folder:
  • select a folder node in the tree
  • on the right panel, click on the create folder button
  • enter the name of the folder and submit
  • the folder immediately appears in the tree


Delete a folder

You can delete a folder by just clicking this button and confirming the deletion.
WARNING: deleting a folder will delete all objects linked to this folder (i.e. underlying folders and campaigns, sessions etc.)


Empty a folder

You can empty a folder by just clicking this button and confirming the emptying (all child nodes will be deleted).
WARNING: emptying a folder will delete all objects linked to its childs (i.e. underlying folders and campaigns, sessions etc.)


Move a folder

You can move a folder by just clicking this button, selecting the destination folder and submitting.


Create a campaign

A campaign can be created only under a folder node.
Here is the process to create a new campaign:
  • in the tree, select a folder (create one if necessary)
  • on the right panel, click on the create campaign icon
  • a dialog box including two tabs is displayed
  • Highslide JS Highslide JS
  • fill the Details tab with the name of the campaign
  • in the Content tab, select all the tests you want to be part of this campaign
  • click on submit
  • the campaign immediately appears in the tree


Create a campaign based on attribute filter

When you have a lot of tests, it may be interesting to preselect all tests having certain attribute values.

To create a campaign based on attribute filter:
  • in the tree, select a folder
  • on the right panel, click on the create campaign icon
  • a dialog box including two tabs is displayed
    Highslide JS Highslide JS
  • fill the Details tab with the name of the campaign
  • in the Content tab
    • click on the preset filter settings button
    • a dialog box is displayed
      • select the root folder and add a AND operator
      • click on the add an expression button
      • a dialog box is displayed
        Highslide JS
      • enter the expression priority < 7 (priority is an integer attribute that you will have created previously and enabled on some tests with specific values. More details are available in the documentation about how to create and associate an attribute to a test)
      • click on submit

      Highslide JS
    • click on submit
    • click on the preset filter button
    • filtering will be performed and the content tab will highlight and pre-select all the tests matching the filter. You can of course unselect or select new tests from this list
    Highslide JS Highslide JS
  • click on submit
  • the campaign immediately appears in the tree


Create a campaign based on SUT inheritance

It may be interesting to preselect all tests linking to an specific SUT.

To create a campaign based on SUT inheritance:
  • in the tree, select a folder
  • on the right panel, click on the create campaign icon
  • a dialog box including two tabs is displayed
    Highslide JS Highslide JS
  • fill the Details tab with the name of the campaign
  • in the Content tab
    • click on the preset import from SUT settings button
    • the Import from SUT settings dialog box is displayed
      • click on expand all button
      • select the SUT
      • click on submit

      Highslide JS
    • click on submit
    • click on the preset import from SUT button
    • Import will be performed and the content tab will highlight and pre-select all the tests matching the import. You can of course unselect or select new tests from this list
    Highslide JS Highslide JS
  • click on submit
  • the campaign immediately appears in the tree

Create a Kalistick campaign

If you have integrated XStudio with Kalistick, you will be able to build automatically your non-regression campaign.
To create a campaign from kalistick data:
  • in the tree, select a folder
  • on the right panel, click on the create Kalistick campaign icon
  • select the file generated by kalistick
  • click on submit
  • the campaign immediately appears in the tree


The campaigns nodes

Tabs

Details

The campaign Details tab allows you to specify the name and the description of the campaign.

Highslide JS

Content

The campaign Content tab allows you to select all the tests you want to be part of this campaign

Order

The campaign Order tab allows you to order all the tests by dependencies or priorities.
Here the process to order the tests in the campaign:
  • in the tree, select the campaign
  • on the right panel, select the Order tab
  • Highslide JS
  • select one or several tests in the list and use the move buttons to position the test(s) wherever you want in the list
  • click on submit



Statistics

A Statistics tab is present on any campaign. This tab shows the progression/regression of this campaign. The information is taken from all the sessions executed belonging to the campaign. All the results are put in a graph so that the evolution (progression or regression) in time is clearly visible. The last session's results are also displayed.
To get campaign statistics:
  • switch to the Tree tab on the left panel
  • in the tree, select a campaign
  • on the right panel, select the Statistics tab
  • for tests, select the Tests tab
  • for test case, select the Test cases tab
The new tab is split in 2 tabbed panes: one that shows the evolution of tests results and another one going deeper at the testcase level:

Highslide JS Highslide JS

Coverage

The Coverage tab will display the coverage details of the selected campaign in terms of requirements and specifications (if enabled).
In addition, some advises are given to improve the coverage of the campaign:
To get campaigns coverage metrics:
  • switch to the Campaigns tab on the left panel
  • in the tree, select the campaign folder
  • on the right panel, select the Coverage tab
  • two tabs By specifications and By requirements are displayed. Each shows some useful information such as:
    • percentage of the coverage
    • for the specifications, the list of ones covered, partially covered and not covered by the campaign.
    • for the requirements, the list of ones covered, partially covered and not covered by the campaign.

Highslide JS Highslide JS

Attachments

Attachments on requirement are handled similarly to the attachments on the folders.

Toolbar

Delete a campaign

You can delete a campaign by just clicking this button and confirming the deletion.
WARNING: deleting a campaign will delete all the items associate with it (i.e. sessions etc.)


Copy a campaign

You can copy a campaign by just clicking this button, selecting the destination folder and submitting.


Move a campaign

You can move a campaign by just clicking this button, selecting the destination folder and submitting.


Create a campaign session

To create a campaign session:
  • in the tree, select a campaign
  • on the right panel, click on the create campaign session button
  • a dialog box including ten tabs is displayed:
    Highslide JS Highslide JS
    Highslide JS Highslide JS
    Highslide JS Highslide JS
    Highslide JS Highslide JS
    Highslide JS Highslide JS
  • fill the Details tab with the name of the session.
  • (opt.) select the Test operator tab and pick the user who will run the session
  • select the Agent tab and pick the agent on which the session will be run. If you wish to run the tests on the local host, leave the default selection (that should already match the local host)
  • (opt.) select the Monitoring agent tab and pick the monitoring agent on which the session will be run
  • (opt.) select the Monitoring servers tab and pick the monitoring servers on which the session will be run
  • (opt.) select the Monitoring features tab and select or create your monitoring configuration
  • select the SUT tab and pick the SUT on which the session will be run
  • select the Configuration tab and pick the configuration you which for each category involved in this campaign session. Once a campaign session is created, it is impossible (by purpose) to change the configurations associated. If no configuration are available:
    • click on the create configuration button
    • a dialog box is displayed
    • enter the name of the configuration
    • fill in all the forms displayed and submit
    • the launchers (xml and jar files) needed for this campaign session MUST be accessible in the bin/launchers folder
  • (opt.) if needed, modify the dynamic attributes values in the Attributes tab
  • (opt.) select the CC Emails tab and check some users to receive a notification when the campaign session will be completed
  • click on submit
  • the campaign session immediately appears in the tree in idle state

Create schedule

When some tests are aimed at being executed completely automatically (using some specific automatic launcher), you can schedule a campaign to be executed on a regular basis. To do this, you need to create a Schedule. Then, at the right time, a campaign session will be automatically generated and started by XStudio.

To create a schedule:
  • in the tree, select a campaign
  • on the right panel, click on the create schedule button
  • a dialog box including nine tabs is displayed
    Highslide JS Highslide JS
    Highslide JS Highslide JS
    Highslide JS Highslide JS
    Highslide JS Highslide JS
    Highslide JS
  • fill the Details tab with the name and the description of the schedule
  • in the Scheduling tab, select the days and the time when you want the sessions to be created and executed. Don’t forget to check the Enabled checkbox. A disabled schedule will not generate any test execution
  • (opt.) select the Test operator tab and pick the user who will (virtually) run the sessions
  • select the Agent tab and pick the agent on which the sessions will be run. This agent needs to have XAgent installed and running
  • select the Monitoring agent tab and pick the agent who will run the sessions
  • select the Monitored servers tab and pick the monitored servers on which the sessions will be run
  • select the SUT tab and pick the SUT on which the sessions will be run
  • select the Configuration tab and pick the configuration you which for each category involved in this schedule. Once a schedule is created, it is impossible (by purpose) to change the configurations associated. If no configuration are available you will need to create one
  • (opt.) select the CC Emails tab and check some users to receive a notification when the campaign sessions will be completed
  • click on submit
  • the schedule immediately appears in the tree
Highslide JS

When the time will come for an agent to execute the tests, the agent will create the campaign session and then execute it. From any station hosting XStudio, after a refresh, the campaign session will appear under the schedule node in the tree.

Create a test plan

It may be interesting to export or to print a report on all the tests belonging to a campaign.

For more information check the process to generate a report chapter from the root node section.

The campaign sessions

Tabs

Details

The campaign session Details tab allows you to specify the name, the report introduction and the report conclusion.


Test operator

The campaign session Test operator tab allows you to specify the user who will run the session.


Agents

The campaign session Agent tab allows you to specify the agent on which the session will be run. If you wish to run the tests on the local host, leave the default selection (that should already match the local host)


Monitored agent

The campaign session Monitoring agent tab allows seeing the agent on which the session will be run.


Monitored servers

The campaign session Monitored servers tab allows seeing the servers on which the session will be run.


Monitored features

The campaign session Monitored features tab allows seeing the monitoring configuration.


SUT

The campaign session SUT tab shows the SUT on which the session will be run


Requirements

The campaign session Requirements tab shows the list of requirements linked to the campaign session with their execution status .


Specifications

The campaign session Specifications tab shows the list of specifications linked to the campaign session.


Content

The campaign session Content tab returns the execution details.
To see the details of the campaign session execution:
  • in the tree, select a campaign session
  • on the right panel, select the Content tab
    Highslide JS
  • the screen is split in three different areas
    • a test tree showing the tests included in the campaign with their results. It shows a number of useful information:
      • the total number of tests that succeeded or failed
        success
        failed
        unknown
      • the number of tests that succeeded or failed in each category
      • the number of tests that succeeded or failed in each folder
    • a testcase sub-tree showing the result of each testcase:
      success
      failed
      not executed
      relative (some tests – i.e. performances - may just return some values that will need to be analyzed by an operator)
    • a message area showing the details of execution of one specific testcase
    • an attachment area including any file the launchers may have uploaded in the database at runtime


NOTE: Test's results are calculated from allunderlying test cases's results using the following algoriothm:
  • if one of the test cases is failed then the test is set as failed
  • if all the test cases are successful then the test is set as successful
  • in all the other cases the test is set as unknown


NOTE: The common process to get all the details of an execution is the following:
  • click on a test
  • the testcase sub-tree is updated showing the associated testcases results
  • click on a testcase
  • the messages area shows the details of each step during the testcase execution
  • the attachments area shows all the files attached to the testcase during its execution


Bugs

This tab shows all the bugs that have been found on the selected campaign session.
Bugs have been found through a failed test and link to this test execution.

Highslide JS

Configuration

The campaign session Configuration tab allows seeing the configuration used for the execution.


Attributes

The campaign session Attributes tab allows seeing the attributes used for the execution.


Highslide JS

Results

This tab allows you to see the campaign session execution.
To see the results of the campaign session execution:
  • in the tree, select a campaign session
  • on the right panel, select the Results tab

    Highslide JS

The screen is split in two columns:
  • the Tests column gives the statistics/results only based on the tests results
  • the Testcases column gives the statistics/results only based on the testcases results

Each column immediately shows a number of useful information:
  • in the header tables:
    • the percentage of success, failure etc. of tests/testcases
    • the coverage of the execution (based on the number of tests/testcases that were not executed)
  • in the pie charts:
    • the number of success, failure etc. of tests/testcases
    • the percentage of success and failure (after removal of the unknown, relative or not executed tests/testcases)


Attachments

Attachments on campaign sessions are handled similarly to the attachments on the folders.

CC Emails

The campaign session CC Emails tab allows you to ensure some users will receive a notification when the campaign session will be completed


Toolbar

Delete a campaign session

You can delete a campaign session by just clicking this button and confirming the deletion.


Copy a campaign session

You can copy a campaign session by just clicking this button, selecting the test operator, the agents, the monitoring agent, the monitored servers, the monitored features, the SUT, the configuration, the attributes etc... and submitting.


Merge campaign sessions

It’s possible to merge several sessions with one instance or several instances.

One instance

To merge several sessions with one instance:
  • in the tree, select a campaign session
  • on the right panel, click on the merge campaign sessions button
  • a dialog box contains three tab is displayed
    Highslide JS Highslide JS
    Highslide JS
  • by default, the Source campaign sessions tab is displayed
  • select sessions you want merged
  • click on Destination folder tab
  • select the destination folder
  • click on options tab
  • select the following option: Merge all sessions in a "one-agent/one-instance" campaign session
  • select the conflict resolution method. Three choices are possible:
    • The most recent result is retained
    • The best result is retained
    • The worst result is retained
  • and submit
  • The merged campaign session is generated in the destination folder with one instance
    Highslide JS


Several instances

To merge several sessions with several instances:
  • in the tree, select a campaign session
  • on the right panel, click on the merge campaign sessions button
  • a dialog box contains three tab is displayed
    Highslide JS Highslide JS
    Highslide JS
  • by default, the Source campaign sessions tab is displayed
  • select sessions you want merged
  • click on Destination folder tab
  • select the destination folder
  • click on options tab
  • select the following option: Add as many instances as necessary in the merged campaign session...
  • and submit
  • The merged campaign session is generated in the destination folder with so many instances as sessions merged
    Highslide JS Highslide JS


Generate residual tests campaign

It's possible to generate tests campaigns automatically from several campaign sessions based on rules defining which tests to include.
For instance, it's possible to selected several campaign sessions and generate a new campaign continaing only the tests that have never been successful.


To do that:
  • in the tree, select the source campaign session(s)
  • on the right panel, click on the generate residual tests campaign button
  • a dialog box containing two tabs is displayed
    Highslide JS Highslide JS
  • select the destination folder and options
  • Click on submit
  • The residual tests campaign is generated in the destination folder


  • Highslide JS

Create report

It may be interesting to export or to print a campaign session report.

For more information check the process to generate a report chapter from the root node section.

Start

To run a campaign session:
  • in the tree, select a campaign session
  • on the right panel, click on the start button
  • the Test campaign details screen immediately appears and will display real time the results of testing
    Highslide JS
  • If the campaign includes tests to be executed manually (the tests are part of one or several categories where you choose manual.jar, simple_manual.jar or tabular_manual.jar as launcher), then you will get additional dialog boxes such as:
    Highslide JS


Pause

You can pause the test campaign session run by just clicking this button.


Stop

You can stop the test campaign session run by just clicking this button.


Campaign session details

You can view the test campaign session details by just clicking this button .

Highslide JS

---------------------------------------------------------------------------------------------------

The campaign schedules

Tabs

Details

The campaign schedule Details tab allows you to specify the name and the description of schedule.


Scheduling

The campaign schedule Scheduling tab allows you to enabled the schedule and setting the date of scheduling.


Highslide JS

Operator

The campaign schedule Operator tab allows you to specify the user who will run the schedule.


Agents

The campaign schedule Agent tab allows you to specify the agent on which the schedule will be run. If you wish to run the tests on the local host, leave the default selection (that should already match the local host)


Monitored agent

The campaign schedule Monitoring agent tab allows seeing the agent on which the schedule will be run.


Monitored servers

The campaign schedule Monitored servers tab allows seeing the servers on which the schedule will be run.


SUT

The campaign schedule SUT tab shows the SUT on which the schedule will be run


Configuration

The campaign schedule Configuration tab allows seeing the configuration used for the execution.


CC Emails

The campaign schedule CC Emails tab allows you to ensure some users will receive a notification when the campaign session will be completed


Toolbar

Delete a campaign schedule

You can delete a campaign schedule by just clicking this button and confirming the deletion.