CMS Pub. 100-01, ch. 7, § 40.3.2

Single Testing Contractor (STC)/Beta Testers

(Rev. 97, Issued: 01-15-16, Effective: 09-21-15, Implementation: 09-21-15)

1. The Shared System Maintainers (SSMs), and the Single Testing Contractor (STC) shall fully

test the quarterly release to ensure it is ready to be elevated to production. For the quarterly

release to be considered fully tested, all the requirements contained within the release must

be tested. Shared System Maintainers (SSMs) and the Single Testing Contractor (STC) must

be able to demonstrate the degree to which each discrete requirement within a CR has been

tested and by which test cases. It is therefore mandatory that the testers maintain traceability

between test cases and the discrete requirements being implemented in the release.

Additionally, for each CR or transmittal under test, the Shared System Maintainers and the

Single Testing Contractor (STC) must ensure that each discrete requirement specified in the

Business Requirements section of any CR/transmittal has been fully tested. The Shared

System Maintainers and the Single Testing Contractor (STC) shall specifically:

• Maintain a repository of test requirements against which all test cases must be traced.

• Prepare and execute a set of test cases that demonstrate the requirements were correctly

implemented for all change requests within the quarterly release.

• Maintain traceability between each test case and the requirement that the case was

designed to test.

2. The Shared System Maintainers and Single Testing Contractor (STC) shall distinguish each

test requirement with a unique Requirement Identifier. The Requirement Identifier must be a

number or qualifier preceded by the CMS CR number and SSM CR number, separated by

dashes. The format of the Requirement Identifier is: [SSM CR No.]-[CMS CR No.]-

[Requirement No.], where:

• SSM CR No. – is a number that identifies a CMS mandate or user change request under

test. Free form text can also be used to identify changes not associated with maintainer

CR numbers, e.g., “Regression” to indicate regression testing. Avoid spaces and use

underscore symbol “_” instead. Dashes not allowed within this number; they are

reserved for separation between the individual numbers in the Requirement Identifier.

• CMS CR No. – is a minimum 4-digit number that identifies the CMS CR associated with

the maintainer CR number. If no CMS CR is associated with a maintainer CR, use

“0000”. Dashes are not allowed. Avoid spaces and use underscore symbol “_” instead.

Dashes not allowed within this number; they are reserved for separation between the

individual numbers in the Requirement Identifier.

• Requirement No. – is a number that uniquely identifies the requirement with the CR. For

any requirement taken from the Business Requirements section of a CMS CR, use the

actual number from the Requirement # column. Do not repeat the CMS CR number.

Dashes not allowed within this number; they are reserved for separation between the

individual numbers in the Requirement Identifier.

Example: Maintainer CR 22522 corresponds to CMS CR 2634. Business Requirement

2.8 was taken directly from CMS CR 2634. The Requirement Identifier would be 22522-

2634-2.8.

3. The Shared System Maintainer and the Single Testing Contractor (STC) shall complete Test

Case specifications that include specific input situations and the expected results associated

with a single test purpose. Each test case specification must include the following:

• A unique Test Case Identifier (which includes a cross-reference to the requirement in

which the case is designed to test);

• The specific objective or purpose of the case;

• Input specifications (i.e., a description of the input situation[s]);

• Output specifications (i.e., a description of the expected results); and

• Intercase dependencies - in instances where the test results of one test case may impact

other test cases, the test case specification must identify the other test case(s) and

describe the relationship(s).

Refer to section 40.3.10, Test Case Specification Standard, for the specific format required to

electronically maintain test cases.

4. All Test Cases must contain a unique Test Case Identifier. The CMS standard for the Test

Case Identifier is the Requirement Identifier, followed by a number that uniquely qualifies

the test case specification, separated by a dash.

The format of the Test Case Identifier is: [Requirement Identifier]-[Test Case Number],

where:

• Requirement Identifier – is the actual identifier of the requirement being testing by the

case.

• Test Case Number – is a number that uniquely qualifies the test case. This is generally a

sequential number. This is necessary since more than one test case is often needed to test

a single requirement. Dashes not allowed within this number; they are reserved for

separation between the individual numbers in the Test Case Identifier.

Example: Two test cases were developed to test the implementation of Requirement

22522-2634-2.8 (see example above). The unique Test Case Identifier for the two test

cases would be 22522-2634-2.8-01 and 22522-2634-2.8-02.

5. The Shared System Maintainers and the Single Testing Contractor (STC) shall document and

execute both positive and negative test cases to ensure the requirements of the release are

correctly implemented.

• Positive test cases are required to ensure that the system is directly fulfilling the

requirements as specified. One or more positive test cases are required for each

requirement. As an example, if a program mandate effects a change for services

beginning on July 1, a positive test case would include service dates in July or later and

validate that the actual mandate was correctly implemented.

• Negative test cases test cases are required to ensure that the system does not perform an

incorrect action. As an example, if a program mandate effects a change for services

beginning on July 1, a negative test case would ensure that implementing the mandate did

not negatively impact claims with service dates prior to July 1. Unlike positive test cases,

a negative test case may not be applicable to every requirement within a CR.

Additionally, although due diligence might necessitate a negative test case, the need may

be mitigated by an existing case in your regression test set.

6. The Shared System Maintainers and the Single Testing Contractor (STC) shall document all

test cases and the actual results for each test case electronically. Each test case and the

associated results must be stored in a test management repository (i.e., Application Lifecycle

Management (ALM) tool) and must at a minimum contain the data elements outlined in the

CMS Test Case specification standard. See subsection 40.3.10 for the Test Case

specification standard.

7. The Shared System Maintainers and the Single Testing Contractor (STC) shall maintain a

test log that provides a record of each test execution. Test Log requirements may be fulfilled

by correctly using the ALM “run” feature (or an equivalent tool or approach) as outlined in

the Quarterly Release Test Management User Guide.

8. The Shared System Maintainers and the Single Testing Contractor (STC) shall execute a full

regression test set on their system for every quarterly release. Each testing entity shall

perform regression testing within their designated testing window as outlined in subsection

40.3.7, Timeframe Requirements.

9. The Shared System Maintainers and the Single Testing Contractor (STC) shall perform

interface testing.

• The Shared System Maintainers and the Single Testing Contractor (STC) shall validate

that all output files are correctly created by their system. The SSMs and STC shall

validate that their system can accept and correctly process all input files.

• The Shared System Maintainers and the Single Testing Contractor (STC) shall perform

interface testing that includes full data exchanges (both ways) between the shared system

and any principal claims processing adjudication or financial system (e.g., the CWF and

HIGLAS respectively). The STC is required to perform data exchanges with HIGLAS

after HIGLAS is implemented at the STC’s data center.

• The Shared System Maintainers and the Single Testing Contractor (STC) shall complete

an integrated system test coordinating the maintenance of test data baselines, such as

beneficiary data, with the CWF Beta tester.