| ... | @@ -60,65 +60,65 @@ Each test scenarios are created by following imaginary stories about clients usi |
... | @@ -60,65 +60,65 @@ Each test scenarios are created by following imaginary stories about clients usi |
|
|
* Automated Robot Framework scenarios are again performed in SUT(System under test) or in "Foreman" environments to find any new failures before acceptance(regressio). This is also the smoke test phase.
|
|
* Automated Robot Framework scenarios are again performed in SUT(System under test) or in "Foreman" environments to find any new failures before acceptance(regressio). This is also the smoke test phase.
|
|
|
|
|
|
|
|
#TEST ENVIRONMENT
|
|
#TEST ENVIRONMENT
|
|
|
Testing tools
|
|
## Testing tools
|
|
|
Same scenarios are used for stability tests...
|
|
Same scenarios are used for stability tests...
|
|
|
|
|
|
|
|
Tools used in testing process:
|
|
Tools used in testing process:
|
|
|
|
|
|
|
|
Ubuntu 14.04
|
|
* Ubuntu 14.04
|
|
|
Robot Framework
|
|
* Robot Framework
|
|
|
RIDE (Robot Framework IDE)
|
|
* RIDE (Robot Framework IDE)
|
|
|
Selenium2 library
|
|
* Selenium2 library
|
|
|
Virtualbox + Vagrant (setting up services for testing environment)
|
|
* Virtualbox + Vagrant (setting up services for testing environment)
|
|
|
fMBT
|
|
* fMBT
|
|
|
Jenkins
|
|
* Jenkins
|
|
|
Locust
|
|
* Locust
|
|
|
Grafana
|
|
* Grafana
|
|
|
Jarmo
|
|
* Jarmo
|
|
|
Hardware
|
|
* Hardware
|
|
|
Workstations for local testing
|
|
* Workstations for local testing
|
|
|
|
|
|
|
|
MANAGEMENT
|
|
# MANAGEMENT
|
|
|
Test process
|
|
## Test process
|
|
|
Testing process for different tests:
|
|
Testing process for different tests:
|
|
|
|
|
|
|
|
Manual test cases are executed in TestLink. Tester follows the test case steps and executes the test in SUT.
|
|
* Manual test cases are executed in TestLink. Tester follows the test case steps and executes the test in SUT.
|
|
|
|
|
|
|
|
Testlink is used as an documentation tool for all the tests, including manual tests. Manual tests are executed and reported in Testlink for later inspection. This gives us the control to observe and manage result statistics during the testing phase.
|
|
* Testlink is used as an documentation tool for all the tests, including manual tests. Manual tests are executed and reported in Testlink for later inspection. This gives us the control to observe and manage result statistics during the testing phase.
|
|
|
|
|
|
|
|
Robot Framework with selenium2 library is used for automation tests. Scenarios are created in Robot Framework and ran with different versions of the application.
|
|
* Robot Framework with selenium2 library is used for automation tests. Scenarios are created in Robot Framework and ran with different versions of the application.
|
|
|
|
|
|
|
|
Stability Tests are run on a local client and executed for long period of times. Stats and version tags are connected for later analyzing.
|
|
* Stability Tests are run on a local client and executed for long period of times. Stats and version tags are connected for later analyzing.
|
|
|
|
|
|
|
|
NoiseGenerator is used to create realistic environment for the application. NoiseGenerator creates network delay.
|
|
* NoiseGenerator is used to create realistic environment for the application. NoiseGenerator creates network delay.
|
|
|
|
|
|
|
|
Performance is tested with Foreman and locust.
|
|
* Performance is tested with Foreman and locust.
|
|
|
|
|
|
|
|
Test infrastructure management
|
|
### Test infrastructure management
|
|
|
Test product environment called "Foreman" is created and split on a multiple computers to reduce the load. This test environment is used to run tests during the development continuum. This environment is created for performance testing purpose, it simulates the actual product environment. Build instructions can be found here.
|
|
* Test product environment called "Foreman" is created and split on a multiple computers to reduce the load. This test environment is used to run tests during the development continuum. This environment is created for performance testing purpose, it simulates the actual product environment. Build instructions can be found here.
|
|
|
|
|
|
|
|
We also have an additional test product environment called "SUT(System under test)". This environment runs on one computer, and it cannot simulate the actual product. Functional and stablility tests can be run on this platform. Multiple users can join this platform. Build process can be found here.
|
|
* We also have an additional test product environment called "SUT(System under test)". This environment runs on one computer, and it cannot simulate the actual product. Functional and stablility tests can be run on this platform. Multiple users can join this platform. Build process can be found here.
|
|
|
|
|
|
|
|
Robot Framework with selenium2 library is used for automation tests. Only one user can be on the board because this test environment is runned locally. Test client is installed after cloning the github repository. README file tells everything you need to know.
|
|
* Robot Framework with selenium2 library is used for automation tests. Only one user can be on the board because this test environment is runned locally. Test client is installed after cloning the github repository. README file tells everything you need to know.
|
|
|
|
|
|
|
|
Test product management
|
|
### Test product management
|
|
|
Test environment version is selected by using GitHub tags for version controlling. These can also be found in contriboard's profile -> "about"-dialog. This helps us to determine what version we are testing.
|
|
Test environment version is selected by using GitHub tags for version controlling. These can also be found in contriboard's profile -> "about"-dialog. This helps us to determine what version we are testing.
|
|
|
|
|
|
|
|
Tags can be found under the "branch:master"-button:
|
|
Tags can be found under the "branch:master"-button:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Client
|
|
* Client
|
|
|
API
|
|
* API
|
|
|
IMG
|
|
* IMG
|
|
|
IO
|
|
* IO
|
|
|
Defects procedure / Error management
|
|
* Defects procedure / Error management
|
|
|
All defects and enhancement proposals should be reported using GitHub issue tracking, to address and debug defects. Defects and enhancement proposals are classified by different labels. In addition customer can report defects or propose enhancements using UserVoice feature in the product.
|
|
* All defects and enhancement proposals should be reported using GitHub issue tracking, to address and debug defects. Defects and enhancement proposals are classified by different labels. In addition customer can report defects or propose enhancements using UserVoice feature in the product.
|
|
|
|
|
|
|
|
Milestones
|
|
### Milestones
|
|
|
Challenge factory is using scrum. Every sprint is a milestone for testing.
|
|
Challenge factory is using scrum. Every sprint is a milestone for testing.
|
|
|
|
|
|
|
|
Goal
|
|
## Goal
|
|
|
Our main goal is a successful test management and documentation over the project development phase.
|
|
Our main goal is a successful test management and documentation over the project development phase.
|
|
|
|
|
|
|
|
## Results
|
|
## Results
|
| ... | |
... | |
| ... | | ... | |