-
Notifications
You must be signed in to change notification settings - Fork 4
Test Strategy
This document describes general approach for Sirius Test Automation Platform Testing. It is the basic document defining what/where/when and how should be tested during current project development.
The testing covers all Sirius components including:
- Server modules
- All client modules
Static testing is done using dedicated static code analysis tools (see appropriate table in the Toolset section).
| Metric | Definition |
|---|---|
| Code style violations | The number of naming, formatting and any other conventions which are designed to keep the code formatted uniformely |
| Code analysis violations | The number of redundant, empty, useless or potentially dangerous code constructions |
| Cyclomatic Complexity Number | Also known as McCabe Metric. Represents the number of potential flows which each specific method may have |
| Non Commenting Source Statements | Represents the number of statements withing the method |
| Metric | Accepted Count | Exceptions |
|---|---|---|
| Code style violations | 0 | Service client generated code |
| Code analysis violations | 0 | Service client generated code |
| Cyclomatic Complexity Number | 10 per method | Service client generated code |
| Non Commenting Source Statements | 100 per method | Service client generated code |
Configuration testing should be applied for all modules separately as well as the entire system. It should be performed on all supported environment instances. Major varying parameters here are runtime entironment versions. They are:
| Technology | Version | Components |
|---|---|---|
| Java | 1.6, 1.7 | All Java Clients, All server modules |
| .NET | 2.0, 3.5, 4.0 | All C# clients, C# Win32 Server |
| Ruby | 1.9 | All Ruby clients |
Additionally, Web modules should be tested against the following browsers:
| Browser | Versions |
|---|---|
| IE | 9.0 |
| Firefox | N/A |
| Chrome | N/A |
| Opera | N/A |
This group of tests is targeted to verify that system keeps working in case of fail. So, major focus is paid to negative cases. Special attention should be paid to the methods which process big volume of data and potentially cause OutOfMemory exceptions.
| Module | Technology | IDE | Build Engine | Code Analysis | Test Engine |
|---|---|---|---|---|---|
| Server | Java, .NET | Eclipse, Visual Studio | Maven, MSBuild | CheckStyle, StyleCop | JUnit, JBehave, NUnit, SpecFlow |
| Java Client | Java | Eclipse | Maven | CheckStyle | JUnit, JBehave |
| Ruby Client | Ruby | Eclipse | Rake | Rubocop | Cucumber |
| C# Client | .NET | Visual Studio | MSBuild | StyleCop | NUnit, SpecFlow |
| Check Type/Language | Java | C# | Ruby |
|---|---|---|---|
| Code style correspondence | Check-Style | StyleCop | Rubocop |
| Code analysis | PMD | FXCop | |
| Cyclomatic Complexity | JavaNCSS |
| Engine Type/Language | Java | C# | Ruby |
|---|---|---|---|
| Build Engine | Maven | MSBuild | Rake |
| Core test engine | JUnit | NUnit | Test/Unit |
| BDD Engine | Cucumber-JVM | SpecFlow | Cucumber |
| Metric | Description | Area of applicability |
|---|---|---|
| Code style violations | 0 | Service client generated code |
| Code analysis violations | 0 | Service client generated code |
| Cyclomatic Complexity Number | 10 per method | Service client generated code |
| Non Commenting Source Statements | 100 per method | Service client generated code |
| Code Coverage | 50% | |
| Conditions coverage | 50% | |
| Features coverage | 100% | |
| The number of failed tests | 0 | Edge cases, Experimental code |
| Metric | Accepted Count | Exceptions |
|---|---|---|
| Code style violations | 0 | Service client generated code |
| Code analysis violations | 0 | Service client generated code |
| Cyclomatic Complexity Number | 10 per method | Service client generated code |
| Non Commenting Source Statements | 100 per method | Service client generated code |
| Code Coverage | 50% | |
| Conditions coverage | 50% | |
| Features coverage | 100% | |
| The number of failed builds | 0 | |
| The number of failed tests | 0 | Edge cases, Experimental code |