Home
This document is an annotated outline for a Software Test Plan, adapted from the IEEE Standard for Software Test Documentation (Std 829-1998).
Version: (4) Date: 05/13/2017
Document History and Distribution
| Revision # | Revision Date | Description of Change | Author |
|---|---|---|---|
| 1 | 5/11/2017 | Created Wiki | Alfredo Guillen |
| 2 | 5/11/2017 | Formatted Wiki to Markdown | Alfredo Guillen |
| 3 | 5/13/2017 | Added onto sections 4,5,8 | Andrew Shin |
| 4 | 5/13/2017 | Added 2.1 | Alfredo Guillen |
| 5 | 5/14/2017 | Added 9 | EJ Reilly |
| 6 | 5/14/2017 | Added 7.2, 7.3 | Andrew Shin |
| 7 | 5/15/2017 | Added Intro & Background and features to be tested | Andrew Shin |
| 8 | 5/16/2017 | Added Test Items | Andrew Shin |
| 9 | 5/16/2017 | Added onto 7.3, 7.5 | EJ Reilly |
1. INTRODUCTION 1
2. TEST ITEMS 2
3. FEATURES TO BE TESTED 3
4. FEATURES NOT TO BE TESTED 3
5. APPROACH 3
6. PASS / FAIL CRITERIA 5
7. TESTING PROCESS 5
8. ENVIRONMENTAL REQUIREMENTS 6
9. CHANGE MANAGEMENT PROCEDURES 7
The Introduction section of the Software Test Plan (STP) provides an overview of the project and the product test strategy, a list of testing deliverables, the plan for development and evolution of the STP, reference material, and agency definitions and acronyms used in the STP.
FreeCol is a turn-based strategy game based on the old game Colonization, and similar to Civilization. The objective of the game is to create an independent nation. The FreeCol team aims to create an Open Source version of Colonization that was released under the GPL
(Describe, at a high level, the scope, approach, resources, and schedule of the testing activities. Provide a concise summary of the test plan objectives, the products to be delivered, major work activities, major work products, major milestones, required resources, and master high-level schedules, budget, and effort requirements.)
The goal is to improve the quality of the application. The way we accomplish this is by first seeing how the game works, and then analyzing the code for particular feature one at a time. We will be approaching this in a 'bottom-up' way, meaning that we will look how small pieces of code will produce a result. We will improve the quality of the application by fixing any bugs or code smells that we can find. By doing this, we will strengthen the code to have it such that the code will is readable and reliable.
Another major goal would be to implement a new feature for the game. A feature that we think will improve the game and make it more enjoyable for the user.
Each member will have the same roles and the goal will be reached through a shared effort of all of the developers.
Testing is the process of analyzing a software item to detect the differences between existing and required conditions and to evaluate the features of the software item.
(This may appear as a specific document (such as a Test Specification), or it may be part of the organization's standard test approach. For each level of testing, there should be a test plan and an appropriate set of deliverables. The test strategy should be clearly defined and the Software Test Plan acts as the high-level test plan. Specific testing activities will have their own test plan. Refer to section 5 of this document for a detailed list of specific test plans.)
We will make use of the documents already available to us for the application, and use an issue tracker so developers will know of bugs or potential areas that need improvement.
(Specify the plans for producing both scheduled and unscheduled updates to the Software Test Plan (change management). Methods for distribution of updates shall be specified along with version control and configuration management requirements must be defined.)
Testing will be performed at several points in the life cycle as the product is constructed. Testing is a very 'dependent' activity. As a result, test planning is a continuing activity performed throughout the system development life cycle. Test plans must be developed for each level of product testing.
(Specify the test items included in the plan. Supply references to the following item documentation:
• Requirements specification:
http://www.freecol.org/about.html
• Design specification:
http://www.freecol.org/documentation/javadoc.html
• Users guide:
http://www.freecol.org/documentation/freecol-user-manual.html
• Operations guide:
http://www.freecol.org/documentation/freecol-user-manual.html
• Installation guide:
http://www.freecol.org/download.html
• Features (availability, response time):
http://www.freecol.org/roadmap.html
• Defect removal procedures, and verification and validation plans:
http://www.freecol.org/how-to-contribute.html
(Outline testing to be performed by the developer for each module being built.)
All developers are solely editing .java files found in packages under the src folder. We'll be ignoring net.sf.freecol.server packages for improvements as testing changes would be costly since they involve network interactions.
(Identify all software features and combinations of software features to be tested. Identify the test design specifications associated with each feature and each combination of features.)
The team will test anything that we create changes or enhancements to.
(Identify all features and specific combinations of features that will not be tested along with the reasons.)
We will not be modifying any of the game's existing networking capabilities. We will also not modify any of the game's models and assets.
(Describe the overall approaches to testing. The approach should be described in sufficient detail to permit identification of the major testing tasks and estimation of the time required to do each task. Identify the types of testing to be performed along with the methods and criteria to be used in performing test activities. Describe the specific methods and procedures for each type of testing. Define the detailed criteria for evaluating the test results.)
(For each level of testing there should be a test plan and the appropriate set of deliverables. Identify the inputs required for each type of test. Specify the source of the input. Also, identify the outputs from each type of testing and specify the purpose and format for each test output. Specify the minimum degree of comprehensiveness desired. Identify the techniques that will be used to judge the comprehensiveness of the testing effort. Specify any additional completion criteria (e.g., error frequency). The techniques to be used to trace requirements should also be specified.)
(Testing conducted to verify the implementation of the design for one software element (e.g., unit, module) or a collection of software elements. Sometimes called unit testing. The purpose of component testing is to ensure that the program logic is complete and correct and ensuring that the component works as designed.)
If a change is made to a method, or a method is added, then we will test these methods through JUnit tests to ensure that functionality will remain as intended.
(Testing conducted in which software elements, hardware elements, or both are combined and tested until the entire system has been integrated. The purpose of integration testing is to ensure that design objectives are met and ensures that the software, as a complete entity, complies with operational requirements. Integration testing is also called System Testing.)
We will ensure that the application performs as it did previously. Along with running all unit tests against the software, we will also run the games on the development computers to see if it will run better or identically to how it was before we made any changes.
(Testing done to ensure that the application operates efficiently and effectively outside the application boundary with all interface systems.)
There is nothing outside the application boundary as this is a standalone application, so other systems do not depend on this application.
(Testing done to ensure that the application systems control and auditability features of the application are functional.)
No changes to how the application implements networking will be made. We can only ensure that the security will perform identically to how the application was before we had started working on it.
(Testing done to ensure that that the application performs to customer expectations (response time, availability, portability, and scalability)).
Since there are no customers that are expecting this application, we can only ensure that the application will perform better or identically to how it was before we had started working on it.
(Testing done to ensure that that applied changes to the application have not adversely affected previously tested functionality.)
Testing will be done through JUnit unit tests. We will run our changes against the previous unit tests that are already existing in the project to see if our new changes fail with these tests.
(Testing conducted to determine whether or not a system satisfies the acceptance criteria and to enable the customer to determine whether or not to accept the system. Acceptance testing ensures that customer requirements' objectives are met and that all components are correctly included in a customer package.)
Since there are specific customer we are releasing this application to, the application will be determined if it will be made available to the general public if performs as intended or identically to how it was before it was worked on by us.
(Testing, done by the customer, using a pre-release version of the product to verify and validate that the system meets business functional requirements. The purpose of beta testing is to detect application faults, failures, and defects.)
New changes will be tested by running the game and playing the game as intended.
(Specify the criteria to be used to determine whether each item has passed or failed testing.)
(Specify the criteria used to suspend all or a portion of the testing activity on test items associated with the plan.)
(Specify the conditions that need to be met to resume testing activities after suspension. Specify the test items that must be repeated when testing is resumed.)
(Specify the conditions that need to be met to approve test results. Define the formal testing approval process.)
(Identify the methods and criteria used in performing test activities. Define the specific methods and procedures for each type of test. Define the detailed criteria for evaluating test results.)
(Identify the deliverable documents from the test process. Test input and output data should be identified as deliverables. Testing report logs, test incident reports, test summary reports, and metrics' reports must be considered testing deliverables.)
Test deliverables will include:
- The test suite
- Test suite documentation (this wiki)
(Identify the set of tasks necessary to prepare for and perform testing activities. Identify all intertask dependencies and any specific skills required.)
Functionality will be properly tested before and after a change has been implemented. Results from testing will be reported in the issue tracker.
_(Identify the groups responsible for managing, designing, preparing, executing, witnessing, checking, and resolving test activities. These groups may include the developers, testers, operations staff, technical support staff, data administration staff, and the user staff.) _
Each developer will be responsible for managing, designing, preparing, executing, witnessing, checking, and resolving all test activities. After implementing any new features/changes, it will be the responsibility of each developer to regression test using the most current unit test suite.
Each developer will also be responsible for updating documentation related to their tests, as necessary.
(Identify the resources allocated for the performance of testing tasks. Identify the organizational elements or individuals responsible for performing testing activities. Assign specific responsibilities. Specify resources by category. If automated tools are to be used in testing, specify the source of the tools, availability, and the usage requirements.)
All developers will be using Eclipse to code and test the application. All developers are responsible for finding bugs and areas where enhancements can be made. Developers will also be expected to document their changes and findings.
(Identify the high level schedule for each testing task. Establish specific milestones for initiating and completing each type of test activity, for the development of a comprehensive plan, for the receipt of each test input, and for the delivery of test output. Estimate the time required to do each test activity.)
The following schedule is tentative and subject to change.
| Target Date | Task |
|---|---|
| 5/3/2017 | Setup Project Repository/Wiki |
| 5/4/2017 | Get Familiarized with Project |
| 5/6/2017 | Identify/Prioritize Test Items |
| 5/7/2017 | Create tests for tests items |
| 5/10/2017 | Execute and refine tests |
| 5/16/2017 | Final touches |
(When planning and scheduling testing activities, it must be recognized that the testing process is iterative based on the testing task dependencies.)
(Specify both the necessary and desired properties of the test environment including the physical characteristics, communications, mode of usage, and testing supplies. Also provide the levels of security required to perform test activities. Identify special test tools needed and other testing needs (space, machine time, and stationary supplies. Identify the source of all needs that is not currently available to the test group.)
FreeCol requires at least 256 MB memory, although some systems slow down badly and require 512MB. FreeCol works best with a screen resolution of at least 1024x768 pixels. It is also possible to play the game with a screen resolution of 1024x600 pixels, although some panels will look a bit cramped. You can play the game with an even smaller screen, but is not supported.
FreeCol is written in Java. In order to run, it requires a Java Virtual Machine. FreeCol should run on any platform on which a Java Virtual Machine compatible with Sun Java 8 or higher is available. FreeCol is known to work with Oracle Java and OpenJDK. FreeCol is known to run on recent versions of Windows, Linux, and Mac OS X.
(Identify the testing environment security and asset protection requirements.)
The code for the project along with any changes made will be hosted on a private Git repository that only developers will have access to.
(Identify the special software tools, techniques, and methodologies employed in the testing efforts. The purpose and use of each tool shall be described. Plans for the acquisition, training, support, and qualification for each tool or technique.)
The primary unit testing tool we will use is JUnit. We will also use Eclipse plugins such as FindBugs, CodePro, JDeodorant, as means to find where bugs can be fixed and enhanced can be made.
(Identify significant constraints on testing such as test item availability, test resource availability, and time constraints. Identify the risks and assumptions associated with testing tasks including schedule, resources, approach and documentation. Specify a contingency plan for each risk factor.)
A constraint for the team would be that team members have limited knowledge of the game and how it functions. Risks would be changing something in the code that could negatively impact the functionality of the game. The deadline of a month after the project was received is another constraint. Tests that were included with the project is also limited or sparse.
(Identify the software test plan change management process. Define the change initiation, change review, and change authorization process.)
Given the size of the team, each team member has the authority to introduce and implement changes as they see fit. The procedure for any changes to the project will be as follows:
1. A GitHub issue is opened for the change.
2. The change is implemented.
3. The GitHub issue is closed.