Smart Analysis

Smart Analysis provides insightful categorizations of test cases, including newly failed, always failing, recovered, and flaky tests. These categorizations aim to streamline debugging processes, improve test reliability, and ensure robust software quality.

Categorizations

Newly Failed ⭕

• Automatically identifies tests that have recently failed after previously passing.
• Flags potential recent changes in code or environment impacting test outcomes.

Always Failing 🔴

• Highlights tests that consistently fail across multiple test executions.
• Indicates persistent issues requiring immediate attention and resolution.

Recurring Error 🟠

• Detects tests that fail with similar error messages from previous runs.
• Pinpoints common causes of recurring errors.

Flaky 🟡

• Detects tests that intermittently alternate between passing and failing states.
• Pinpoints unreliable tests affected by race conditions or environmental dependencies.

Recovered 🟢

• Detects tests that intermittently alternate between passing and failing states.
• Pinpoints unreliable tests affected by race conditions or environmental dependencies.

Configuration

Smart Analysis is enabled by default. You can disable it from the test run configuration.

{
  "api_key": "<api-key>",
  "show_smart_analysis": false,
  "targets": [
    {
      "name": "slack",
      "inputs": {
        "url": "<slack-incoming-webhook-url>"
      }
    }
  ],
  "results": [
    {
      "type": "testng",
      "files": [
        "path/to/testng-results.xml"
      ]
    }
  ]
}