30. Skip to content

30. Purpose and scope of testing

The test plan presented herein describes tasks related to testing the CDOC2 SmartID/MobileID server components: Key Shares Server, Authentication Server and Relying Party Server. In this document the server components are collectively referred to as ‘server’. The test plan does not cover the client-side components of CDOC2; it is solely focused on the server side of the system.

Within the scope of the test plan presented here, server testing serves two main purposes:

  • Functional testing, to verify that the implemented functionality of the server is in line with what is prescribed in the documentation and the server can fulfil the established goals.
  • Load testing, to establish the operation and behaviour of the server under different usage loads and patterns.

30.1 Functional testing

The purpose of functional testing is to verify that the implemented functionality of the server is in line with what is prescribed in the documentation and the server can fulfil the established goals.

Functional testing uses unit tests and automated API tests written by the developers that cover all scenarios described in server use cases and potential API error situations.

30.2 Load testing

Although no explicit requirements have been set out for server capability, it is important to be aware of the capacity limits of the developed software on a specific existing platform. This information is provided by load tests which seek answers to the following questions:

  • What is the system’s response time to users under different system loads?
  • How many users can the system serve simultaneously without exceeding the defined response time?
  • How will the system perform under extreme load conditions (e.g. very large number of simultaneous users or key shares in the database)?
  • How will the system recover after peak load?
  • How will the system perform under conditions of long-term moderate or higher load?

31. Test procedure and results

Test planning is a continuous process, regularly repeated throughout the project after the addition of significant new information. The result of this process is the test plan (the present document).

In the case of the shares server, functional testing is covered by unit tests and API tests run against the server API.

Unit tests are created by the developers and can be used by the development team, as well as in a continuous integration environment. The output of the tests is a human- and machine-readable test report. The source code for the unit tests is maintained along with the server’s source code, following the same principles as the application source code.

Just as the unit tests, the shares server API tests are designed to be implementable as automatic tests to ensure the simplicity of running the tests and facilitate their repeatability. A test report on the API tests run will be generated by the tool used.

Tools for the development of API tests are chosen to ensure the simple maintenance of the developed tests, and the maintainability of their source code along with and following the same principles as the application source code.

Tests used for establishing server capability are also implemented as automatic tests, utilizing a suitable testing tool. The best tool for this purpose will be determined during test development, considering the following requirements:

  • Test development should not be unreasonably complex for the developer.
  • Adjusting the desired load should not require software development skills.
  • The resulting test report should be easily available and understandable.

Note that the scope of load test development only includes the capability of stressing the system and the load tests do not include solutions for monitoring the system during load testing. Tools created for system monitoring must be used for this purpose.

The result of load testing is a test report containing information on the questions covered in the corresponding section of the test plan. The report provides information on the loads applied to the server, as well as the usage of system resources (memory usage, processor load, storage usage) required for servicing queries.

32. Designed test scenarios

Test ideas and scenarios are normally not explicitly covered in a test plan. However, as the CDOC2 servers are applications with limited functionality, there would be little need for a separate document for test scenario management. The present section also describes the scenarios designed to be used in load testing.

32.1 Key Shares Server API functionality tests

Server functionality is tested by emulating the end-user client application utilizing the server API interfaces.

Positive scenarios:

  • Sender successfully transmits a key share to the server (POST_KEYSHARE-POS-01-ONCE)
  • Sender has already transmitted a key share and is retransmitting the key share to the server (POST_KEYSHARE-POS-02-REPEATEDLY)
  • Sender transmits a random byte array not exceeding the defined length to the server as key share material (POST_KEYSHARE-POS-03-RANDOM_CONTENT)
  • Recipient successfully requests a nonce for an existing key share (POST_NONCE-POS-01-CORRECT_SHARE_ID)
  • Recipient successfully requests a session nonce used for session token authentication (POST_SESSION_NONCE-POS-01)
  • Recipient successfully requests a key share (GET_KEYSHARE-POS-01-CORRECT_REQUEST)

Negative scenarios:

  • Sender transmits a key share containing overlength key share material (POST_KEYSHARE-NEG-01-SHARE_TOO_BIG)
  • Recipient requests a key share with an underlength share ID (GET_KEYSHARE-NEG-01-TOO_SHORT_SHARE_ID)
  • Recipient requests a key share with an empty share ID (GET_KEYSHARE-NEG-02-EMPTY_STRING_SHARE_ID)
  • Recipient requests a key share with an overlength share ID (GET_KEYSHARE-NEG-03-TOO_LONG_RANDOM_STRING_SHARE_ID)
  • Recipient requests a key share at the key-shares endpoint with both the share ID and the trailing path separator omitted (GET_KEYSHARE-NEG-04-MISSING_SHARE_ID_AND_URI_SLASH)
  • Recipient requests a key share using a random, unsigned authentication ticket (GET_KEYSHARE-NEG-05-RANDOM_AUTH_TICKET)
  • Recipient requests a key share with a random share ID (GET_KEYSHARE-NEG-06-RANDOM_SHARE_ID)
  • Recipient requests a key share with a valid share ID but the recipient’s identity does not match the key share (GET_KEYSHARE-NEG-07-RECIPIENT_NOT_MATCHING)
  • Recipient requests a key share without authentication headers (GET_KEYSHARE-NEG-08-MISSING_AUTH_HEADERS)
  • Recipient requests a nonce for a non-existing key share (POST_NONCE-NEG-01-RANDOM_SHARE_ID)

32.2 Authentication Server API functionality tests

Server functionality is tested by emulating the end-user client application utilizing the server API interfaces.

Positive scenarios:

  • Recipient successfully starts CDOC2 authentication process with SID (POST_START_AUTH-POS-01-SID)
  • Recipient successfully starts CDOC2 authentication process with MID (POST_START_AUTH-POS-02-MID)
  • Recipient successfully requests completed CDOC2 authentication process status (GET_AUTH_STATUS-POS-01-COMPLETED-ONCE)
  • Recipient successfully requests completed CDOC2 authentication process status repeatedly (GET_AUTH_STATUS-POS-02-COMPLETED-REPEATEDLY)
  • Recipient successfully requests running CDOC2 authentication process status (GET_AUTH_STATUS-POS-03-RUNNING)
  • Recipient successfully requests failed CDOC2 authentication process status (GET_AUTH_STATUS-POS-04-FAILED)
  • Recipient successfully requests failed server info (GET_INFO-POS-01)

Negative scenarios:

  • Recipient starts auth process with identifier missing (POST_START_AUTH-NEG-01-IDENTIFIER_MISSING)
  • Recipient starts auth process with too short identifier (POST_START_AUTH-NEG-02-IDENTIFIER_TOO_SHORT)
  • Recipient starts auth process with too long identifier (POST_START_AUTH-NEG-03-IDENTIFIER_TOO_LONG)
  • Recipient starts auth process with incorrect identifier format (POST_START_AUTH-NEG-04-IDENTIFIER_FORMAT_INCORRECT)
  • Recipient starts auth process with too short mobile nr (POST_START_AUTH-NEG-05-MOBILE_NR_TOO_SHORT)
  • Recipient starts auth process with too long mobile nr (POST_START_AUTH-NEG-05-MOBILE_NR_TOO_LONG)
  • Recipient starts auth process with unknown requested language (POST_START_AUTH-NEG-06-LANGUAGE_UNKNOWN)
  • Recipient requests auth process status for too short UUID (POST_AUTH_STATUS-NEG-01-UUID_TOO_SHORT)
  • Recipient requests auth process status for too long UUID (POST_AUTH_STATUS-NEG-02-UUID_TOO_LONG)
  • Recipient requests auth process status for malformed UUID (POST_AUTH_STATUS-NEG-03-UUID_MALFORMED)
  • Recipient requests auth process status for process UUID not present in database (POST_AUTH_STATUS-NEG-04-AUTH_PROCESS_NOT_FOUND)

32.3 Relying Party Server API functionality tests

Server functionality is tested by emulating the end-user client application utilizing the server API interfaces.

Positive scenarios:

  • Recipient successfully starts SID authentication (POST_SID_AUTHENTICATE-POS-01)
  • Recipient successfully requests SID authentication session (GET_SID_SESSION-POS-01)
  • Recipient successfully starts MID authentication (POST_MID_AUTHENTICATE-POS-01)
  • Recipient successfully requests MID authentication session (GET_MID_SESSION-POS-01)

Negative scenarios:

  • Recipient starts SID authentication without session token (POST_SID_AUTHENTICATE-NEG-01-SESSION_TOKEN_MISSING)
  • Recipient starts SID authentication with malformed session token (POST_SID_AUTHENTICATE-NEG-02-SESSION_TOKEN_MALFORMED)
  • Recipient starts SID authentication without session token certificate (POST_SID_AUTHENTICATE-NEG-03-SESSION_TOKEN_CERT_MISSING)
  • Recipient starts SID authentication with malformed session token certificate (POST_SID_AUTHENTICATE-NEG-04-SESSION_TOKEN_CERT_MALFORMED)
  • Recipient starts SID authentication with session token sub and certificate serial number mismatch (POST_SID_AUTHENTICATE-NEG-05-SESSION_TOKEN_SUB_CERT_SN_MISMATCH)
  • Recipient starts SID authentication withouth request body (POST_SID_AUTHENTICATE-NEG-06-REQUEST_BODY_MISSING)
  • Recipient starts SID authentication withouth semantics identifier (POST_SID_AUTHENTICATE-NEG-07-SEMANTICS_IDENTIFIER_MISSING)
  • Recipient starts SID authentication withouth signature protocol (POST_SID_AUTHENTICATE-NEG-08-SIGNATURE_PROTOCOL_MISSING)
  • Recipient starts SID authentication withouth signature protocol parameters (POST_SID_AUTHENTICATE-NEG-09-SIGNATURE_PROTOCOL_PARAMS_MISSING)
  • Recipient starts SID authentication withouth interactions (POST_SID_AUTHENTICATE-NEG-10-INTERACTIONS_MISSING)
  • Recipient starts SID authentication withouth verification code type (POST_SID_AUTHENTICATE-NEG-11-VC_TYPE_MISSING)
  • Recipient requests SID session without session token (GET_SID_SESSION-NEG-01-SESSION_TOKEN_MISSING)
  • Recipient requests SID session with malformed session token (GET_SID_SESSION-NEG-02-SESSION_TOKEN_MALFORMED)
  • Recipient requests SID session without session token certificate (GET_SID_SESSION-NEG-03-SESSION_TOKEN_CERT_MISSING)
  • Recipient requests SID session with malformed session token certificate (GET_SID_SESSION-NEG-04-SESSION_TOKEN_CERT_MALFORMED)
  • Recipient requests SID session with session token sub and certificate serial number mismatch (GET_SID_SESSION-NEG-05-SESSION_TOKEN_SUB_CERT_SN_MISMATCH)
  • Recipient requests SID session without session id (GET_SID_SESSION-NEG-06-SESSION_ID_MISSING)

  • Recipient requests SID session with malformed session id (GET_SID_SESSION-NEG-07-SESSION_ID_MALFORMED)

  • Recipient starts MID authentication without session token (POST_MID_AUTHENTICATE-NEG-01-SESSION_TOKEN_MISSING)

  • Recipient starts MID authentication with malformed session token (POST_MID_AUTHENTICATE-NEG-02-SESSION_TOKEN_MALFORMED)
  • Recipient starts MID authentication without session token certificate (POST_MID_AUTHENTICATE-NEG-03-SESSION_TOKEN_CERT_MISSING)
  • Recipient starts MID authentication with malformed session token certificate (POST_MID_AUTHENTICATE-NEG-04-SESSION_TOKEN_CERT_MALFORMED)
  • Recipient starts MID authentication with session token sub and certificate serial number mismatch (POST_MID_AUTHENTICATE-NEG-05-SESSION_TOKEN_SUB_CERT_SN_MISMATCH)
  • Recipient starts MID authentication without request body (POST_MID_AUTHENTICATE-NEG-06-REQUEST_BODY_MISSING)
  • Recipient starts MID authentication without phone number (POST_MID_AUTHENTICATE-NEG-07-PHONE_NUMBER_MISSING)
  • Recipient starts MID authentication without national identity number (POST_MID_AUTHENTICATE-NEG-08-NATIONAL_IDENTIDY_NUMBER_MISSING)
  • Recipient starts MID authentication without hash (POST_MID_AUTHENTICATE-NEG-09-HASH_MISSING)
  • Recipient starts MID authentication without hash type (POST_MID_AUTHENTICATE-NEG-10-HASH_TYPE_MISSING)
  • Recipient starts MID authentication without language (POST_MID_AUTHENTICATE-NEG-11-LANGUAGE_MISSING)
  • Recipient starts MID authentication without display text (POST_MID_AUTHENTICATE-NEG-12-DISPLAY_TEXT_MISSING)
  • Recipient starts MID authentication without display text format (POST_MID_AUTHENTICATE-NEG-13-DISPLAY_TEXT_FORMAT_MISSING)
  • Recipient requests MID session without session token (GET_MID_SESSION-NEG-01-SESSION_TOKEN_MISSING)
  • Recipient requests MID session with malformed session token (GET_MID_SESSION-NEG-02-SESSION_TOKEN_MALFORMED)
  • Recipient requests MID session without session token certificate (GET_MID_SESSION-NEG-03-SESSION_TOKEN_CERT_MISSING)
  • Recipient requests MID session with malformed session token certificate (GET_MID_SESSION-NEG-04-SESSION_TOKEN_CERT_MALFORMED)
  • Recipient requests MID session with session token sub and certificate serial number mismatch (GET_MID_SESSION-NEG-05-SESSION_TOKEN_SUB_CERT_SN_MISMATCH)
  • Recipient requests MID session without session id (GET_MID_SESSION-NEG-06-SESSION_ID_MISSING)
  • Recipient requests MID session with malformed session id (GET_MID_SESSION-NEG-07-SESSION_ID_MALFORMED)

33. Server load tests

To receive information about the server’s behaviour under stress, the server must be overloaded with queries designed to be as close as possible to the behavioural patterns of real-life users.

Depending on the design and functionality of the shares server, queries made to the server can be divided into two main groups: transmission of key shares to the server, and key share requests via user authentication. While the transmission of key shares does not require user authentication, requesting a key share requires the request to include an authentication token signed with the recipient's Smart-ID or Mobile-ID certificate; both query groups are served over the same interface, sharing a common database for storing key shares.

Putting a load on the server requires the use of queries for transmitting key shares to the server and requesting key shares from the server.

  • In the case of key share transmission queries, the queries must be functionally successful and result in key shares being saved to the server database.
  • In the case of key share request queries, the key share to be used and the user to be authenticated are selected randomly and the reply returned by the server may be either positive (i.e. contain a key share) or negative (i.e. contain an error code).

The contents of the query are irrelevant for load testing, as the same internal queries and comparisons are required for both positive and negative results.

33.1 Load generation

Both the functional tests and the load tests utilize the Gatling test framework where the desired load on the tested software can be adjusted using the following parameters:

  • start-users-per-second: Number of active users (queries per second) immediately applied at the start of the test
  • increment-users-per-second: Number of active users added per each following test cycle
  • increment-cycles: Number of test cycles used
  • cycle-duration-seconds: Duration of a single test cycle (in seconds)

The duration of a load test depends on the number of test cycles and cycle duration. Number of queries being made to the tested software depends on the initial number of users and the number of users added per each following test cycle. For example, in order to generate a steadily increasing load, the initial number of users can be adjusted to be relatively small, and a larger number of users added per each test cycle.

Using the settings below, the duration of the test will be 600 seconds (10 minutes), the initial query rate at the start of the test is 10 queries per second, and the query rate in the last test cycle is 110 queries per second:

  • start-users-per-second = 10
  • increment-users-per-second = 10
  • increment-cycles = 10
  • cycle-duration-seconds = 60

To generate a steady load, the number of users applied at the start of the test must be adjusted to the desired query rate and the number of users added per test cycle kept minimal.

Using the settings below, the duration of the test will be 600 seconds (10 minutes), the initial query rate at the start of the test is 75 queries per second, and the query rate in the last test cycle is 85 queries per second:

  • start-users-per-second = 10
  • increment-users-per-second = 1
  • increment-cycles = 10
  • cycle-duration-seconds = 60

33.2 Implementation of scenarios in load tests

  • What is the system’s response time to users under different system loads?
  • How many users can the system serve simultaneously without exceeding the defined response time?

Since system operability is directly dependent on the system’s operational environment, answering these questions requires repeatedly running load tests using a variety of loads. To start off, gradually increasing loads can be used to determine potential capacity limits.

  • How will the system perform under extreme load conditions (e.g. very large number of simultaneous users or key shares in the database)?

The goal of this test is to take the system to or above maximum load and monitor the system’s performance under such conditions. Probing for extreme loads should be carried out using a steady load over a long period of time.

  • How will the system recover after peak load?

The goal of this test is to gather information on the ability to recover from peak load. Multiple simultaneous load generators can be used here, one performing a longer, steady-load test and the other a shorter, increasing-load test.

  • How will the system perform under conditions of long-term moderate or higher load?

The system is run for a long period of time under a steady moderate load to detect anomalies or errors that could occur in the long-term operation of the system (small memory leaks etc.). A test involving a steady load of a large number of longer test cycles can be used to seek an answer to this question.