This document describes all the OP tests that the OpenID Connect test suite contains.

1. Test description file

1.1. assert
1.2. desc
1.3. group
1.4. note
1.5. reference
1.6. sequence
1.7. usage
1.8. MTI

2. Tests

2.1. OP-claims-essential
2.2. OP-ClientAuth-Basic-Dynamic
2.3. OP-ClientAuth-Basic-Static
2.4. OP-ClientAuth-SecretPost-Dynamic
2.5. OP-ClientAuth-SecretPost-Static
2.6. OP-Discovery-claims_supported
2.7. OP-Discovery-Config
2.8. OP-Discovery-JWKs
2.9. OP-Discovery-jwks_uri
2.10. OP-display-page
2.11. OP-display-popup
2.12. OP-IDToken-at_hash
2.13. OP-IDToken-C-Signature
2.14. OP-IDToken-c_hash
2.15. OP-IDToken-kid
2.16. OP-IDToken-none
2.17. OP-IDToken-RS256
2.18. OP-nonce-code
2.19. OP-nonce-noncode
2.20. OP-nonce-NoReq-code
2.21. OP-nonce-NoReq-noncode
2.22. OP-OAuth-2nd
2.23. OP-OAuth-2nd-30s
2.24. OP-OAuth-2nd-Revokes
2.25. OP-prompt-login
2.26. OP-prompt-none-LoggedIn
2.27. OP-prompt-none-NotLoggedIn
2.28. OP-redirect_uri-Missing
2.29. OP-redirect_uri-NotReg
2.30. OP-redirect_uri-Query-Added
2.31. OP-redirect_uri-Query-Mismatch
2.32. OP-redirect_uri-Query-OK
2.33. OP-redirect_uri-RegFrag
2.34. OP-Registration-Dynamic
2.35. OP-Registration-Endpoint
2.36. OP-Registration-jwks
2.37. OP-Registration-jwks_uri
2.38. OP-Registration-logo_uri
2.39. OP-Registration-policy_uri
2.40. OP-Registration-Sector-Bad
2.41. OP-Registration-tos_uri
2.42. OP-Req-acr_values
2.43. OP-Req-claims_locales
2.44. OP-Req-id_token_hint
2.45. OP-Req-login_hint
2.46. OP-Req-max_age=1
2.47. OP-Req-max_age=10000
2.48. OP-Req-NotUnderstood
2.49. OP-Req-ui_locales
2.50. OP-request-Unsigned
2.51. OP-request_uri-Sig
2.52. OP-request_uri-Sig-any
2.53. OP-request_uri-Unsigned
2.54. OP-Response-code
2.55. OP-Response-code+id_token
2.56. OP-Response-code+id_token+token
2.57. OP-Response-code+token
2.58. OP-Response-form_post
2.59. OP-Response-form_post-Error
2.60. OP-Response-id_token
2.61. OP-Response-id_token+token
2.62. OP-Response-Missing
2.63. OP-Rotation-OP-Enc
2.64. OP-Rotation-OP-Sig
2.65. OP-Rotation-RP-Sig
2.66. OP-scope-address
2.67. OP-scope-All
2.68. OP-scope-email
2.69. OP-scope-phone
2.70. OP-scope-profile
2.71. OP-UserInfo-Body
2.72. OP-UserInfo-Endpoint
2.73. OP-UserInfo-Header
2.74. OP-UserInfo-RS256
2.75. OP-UserInfo-sig-any

3. Assertions

3.1. asym-signed-userinfo
3.2. auth_time-check
3.3. authn-response-or-error
3.4. bare-keys
3.5. check-http-error-response
3.6. check-http-response
3.7. check-idtoken-nonce
3.8. check-query-part
3.9. claims-check
3.10. is-idtoken-signed
3.11. multiple-sign-on
3.12. new-encryption-keys
3.13. new-signing-keys
3.14. providerinfo-has-claims_supported
3.15. providerinfo-has-jwks_uri
3.16. same-authn
3.17. unsigned-idtoken
3.18. used-acr-value
3.19. verify-authn-response
3.20. verify-base64url
3.21. verify-claims
3.22. verify-error-response
3.23. verify-https-usage
3.24. verify-id_token_signing-algorithm-is-supported
3.25. verify-idtoken-is-signed
3.26. verify-nonce
3.27. verify-op-endpoints-use-https
3.28. verify-op-has-registration-endpoint
3.29. verify-response
3.30. verify-scopes
3.31. verify-signed-idtoken-has-kid

4. In-flow functions

4.1. acr_value
4.2. check_config
4.3. check_support
4.4. claims_locales
4.5. conditional_execution
4.6. expect_exception
4.7. id_token_hint
4.8. login_hint
4.9. multiple_return_uris
4.10. redirect_uri_with_query_component
4.11. redirect_uris_with_fragment
4.12. redirect_uris_with_query_component
4.13. register
4.14. request_in_file
4.15. set_discovery_issuer
4.16. set_essential_arg_claim
4.17. set_expect_error
4.18. set_op_args
4.19. set_redirect_uri
4.20. set_redirect_uris
4.21. set_request_args
4.22. set_response_where
4.23. set_state
4.24. set_uri
4.25. set_webfinger_resource
4.26. static_jwk
4.27. store_sector_redirect_uris
4.28. ui_locales

1. Test description syntax

The test description files are JSON documents. Each of these documents contains a JSON object with a set of keys and values. The possible keys are:

assert
desc
group
note
reference
sequence
usage
MTI

These will be described one by one below.

1.1. assert

Lists the assertions that are performed after a test has completed.

A example is:

    "assert": {
        "verify-response": {
              "response_cls": [
                    "OpenIDSchema"
              ]
        }
    }

The interpretation of this is that the assertion with the identifier

verify-response will be executed with the argument

    response_cls="OpenDISchema"

If a test run reaches the end of the request sequence the assertion checks are run one after the other.

When some assertion checks fails that is registered as an Error, for some other checks a failure is logged as a Warning. And then there are a few where how a failure is interpreted is configurable.

To get the result of the whole test run the results of the assertion checks are added together. Such that if there is one or more Error failures then the result of the test is an Error. If there is no Error failures but at least on Warning failure then the result is a Warning. If no failures are encountered that is logged as a Success.

1.2. desc

A human readable description of what the test is trying to accomplish.

1.3. group

Which group the test belongs to. When the tests are presented to the tester tests belonging to the same group are listed together under the group header.

Present list of used groups:

Response Type
Response Mode
Discovery
Dynamic Client Registration
ID Token
Client Authentication
Access Token
Userinfo Endpoint
claims Request Parameter
display Request Parameter
nonce Request Parameter
prompt Request Parameter
redirect_uri Request Parameter
request Request Parameter
request_uri Request Parameter
scope Request Parameter
Misc Request Parameters
OAuth behaviors
Key Rotation
RP-Initiated Logout
Back-Channel Logout
Front-Channel Logout
Session Management

1.4. note

In some case the test tool needs to pass information to the tester. This is where that information is stored.

1.5. reference

Links to parts of the OpenID Connect and accompanying standards that are tested by the test.

1.6. sequence

The complete flow of a test. This consists of a number of requests.

Before a request is issued the arguments fo the request may be set to specific values. Also before a request is sent a check my be performed to figure out if there is any meaning to send the request. One reason for it not to be so would be that the OP does not support the functionality that is under test.

An simple example of a sequence:

  "sequence": [
    {
      "Webfinger": {
        "set_webfinger_resource": null
      }
    },
    {
      "Discovery": {
        "set_discovery_issuer": null
      }
    }
  ]

This sequence contains 2 requests, the first being a Webfinger request and the second a discovery request. Before the webfinger request is sent, the webfinger resource is specified by the set_webfinger_resource function. Similarly, before the discovery request is sent the issuer is collected using the set_discovery_issuer function.

Note: If webfinger is not supported then that request is not sent.
The same goes for discovery. Which means that running this test when not supporting dynamic discovery is useless.

A slightly more complex sequence:

  "sequence": [
    {
      "Webfinger": {
        "set_webfinger_resource": null
      }
    },
    {
      "Discovery": {
        "set_discovery_issuer": null
      }
    },
    "Registration",
    {
      "AsyncAuthn": {
        "set_response_where": null
      }
    },
    {
      "AccessToken": {
        "conditional_execution": {
          "return_type": [
            "CIT",
            "CI",
            "C",
            "CT"
          ]
        }
      }
    },
    {
      "UserInfo": {
        "set_op_args": {
          "method": "POST",
          "authn_method": "token_in_message_body"
        }
      }
    }
  ]

This starts in the same way as the previous one. Webfinger, provider info discovery and client registration is performed if supported.

Then follows an Authorization, an Access token and finally an Userinfo request.

Note:The Access token request will only be performed if the response_type is one of "code", "code token", "code id_token" or "code id_token token".

Note: The Userinfo request will be formed as described in Section 2.2 of RFC6750

usage

Specifies when the test can be used. This is connected to the test profiles.

Example:

  "usage": {
    "sig": true,
    "register": true,
    "extra": true
  }

This specific test will be included if the tester has specified that extra tests should be used and the OpenID provider that is being tested supports dynamic registration and signature creation/verification.

The complete set of usage demands are:

webfinger: Tests if Webfinger is supported
enc: Tests if encryption and decryption is supported
extra: This test is an extra test. Not necessary to run for basic standard conformance testing
discover: Tests that dynamic discovery is supported
sig: Tests that signature creation and signature validation is supported
none: Tests support for signing_alg=None
return_type: The response_types that are expected to be used during testing
register: Tests is dynamic client registration is supported
form_post: Tests if the form_post response_type is supported

MTI

Mandatory to implement NOT USED

2. Tests

2.1. OP-claims-essential

Claims request with essential name claim

JSON description	OP-claims-essential
In-flow checks	Webfinger set_webfinger_resource Discovery set_discovery_issuer AsyncAuthn set_response_where set_essential_arg_claim AccessToken conditional_execution UserInfo conditional_execution set_op_args
Assertions	verify-claims check-http-response
Group	claims Request Parameter
Return Types	Code IDToken IDToken Token Code IDtoken Code Token Code IDToken Token
Link to specification	https://openid.net/specs/openid-connect-core-1_0.html#IndividualClaimsRequests