SendVerificationCode method is returning Error Code: 16 in production for some users

[minlite]

Description

I'm using Firebase Auth in production. I have Firebase AppCheck enabled and enforced for Authentication.

A small percentage of my users on iOS are seeing the sdk error out with An internal error has occurred, print and inspect the error details for more information. when trying to request a verification code.

I went through our logs in Google Cloud and noticed that the google.cloud.identitytoolkit.v1.AuthenticationService.SendVerificationCode method is failing with error code 16 and message Error code: 16 for them. The logs don't provide any other indication of what error code 16 might be and my search of this error online plus in the firebase emulation tools has not resulted in any answers. Please see the full log with number, ip address, and project details redacted below:

{
  "insertId": "kjdxlhdp2nu",
  "jsonPayload": {
    "status": {
      "code": 16,
      "message": "Error code: 16"
    },
    "request": {
      "phoneNumber": "xxxxxxxxxxxxx",
      "@type": "type.googleapis.com/google.cloud.identitytoolkit.v1.SendVerificationCodeRequest"
    },
    "requestMetadata": {
      "callerIp": "xxxxxxxxxxxx",
      "callerSuppliedUserAgent": "FirebaseAuth.iOS/10.17.0 xxxxxxxxxxxx/7.27 iPhone/16.5.1 hw/iPhone12_3,gzip(gfe),gzip(gfe)"
    },
    "methodName": "google.cloud.identitytoolkit.v1.AuthenticationService.SendVerificationCode",
    "@type": "type.googleapis.com/google.cloud.identitytoolkit.logging.RequestLog",
    "serviceName": "identitytoolkit.googleapis.com"
  },
  "resource": {
    "type": "identitytoolkit_project",
    "labels": {
      "project_id": "xxxxxxxxxxx"
    }
  },
  "timestamp": "2023-11-15T23:25:18.518Z",
  "severity": "ERROR",
  "logName": "projects/xxxxxxxxxxxx/logs/identitytoolkit.googleapis.com%2Frequests",
  "receiveTimestamp": "2023-11-15T23:25:18.876791432Z"
}

As you can see, I am using the latest Firebase iOS SDK (10.17.0) and the user has the latest version of my app.

I doubt this is a clear cut misconfiguration on our side since the majority of users are able to sign in with no issue. I would like if Firebase team can decipher what error code 16 might mean and why the SDK is not providing a more useful error message.

Reproducing the issue

No response

Firebase SDK Version

10.17.0

Xcode Version

15.0

Installation Method

CocoaPods

Firebase Product(s)

App Check, Authentication

Targeted Platforms

iOS

Relevant Log Output

{
  "insertId": "kjdxlhdp2nu",
  "jsonPayload": {
    "status": {
      "code": 16,
      "message": "Error code: 16"
    },
    "request": {
      "phoneNumber": "xxxxxxxxxxxxx",
      "@type": "type.googleapis.com/google.cloud.identitytoolkit.v1.SendVerificationCodeRequest"
    },
    "requestMetadata": {
      "callerIp": "xxxxxxxxxxxx",
      "callerSuppliedUserAgent": "FirebaseAuth.iOS/10.17.0 xxxxxxxxxxxx/7.27 iPhone/16.5.1 hw/iPhone12_3,gzip(gfe),gzip(gfe)"
    },
    "methodName": "google.cloud.identitytoolkit.v1.AuthenticationService.SendVerificationCode",
    "@type": "type.googleapis.com/google.cloud.identitytoolkit.logging.RequestLog",
    "serviceName": "identitytoolkit.googleapis.com"
  },
  "resource": {
    "type": "identitytoolkit_project",
    "labels": {
      "project_id": "xxxxxxxxxxx"
    }
  },
  "timestamp": "2023-11-15T23:25:18.518Z",
  "severity": "ERROR",
  "logName": "projects/xxxxxxxxxxxx/logs/identitytoolkit.googleapis.com%2Frequests",
  "receiveTimestamp": "2023-11-15T23:25:18.876791432Z"
}

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

Replace this line with the contents of your Package.resolved.

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

# Uncomment this line to define a global platform for your project
# platform :ios, '9.0'
# Uncomment this line if you're using Swift
use_frameworks!

platform :ios, '13.0'

target 'XXXXXXXXX' do
    project 'XXXXXXXXX', 'Debug' => :debug, 'Debug.Staging' => :debug, 'Release' => :release, 'Release.Staging' => :release
    pod 'FirebaseMessaging'
    pod 'FirebaseAuth'
    pod 'FirebaseAnalytics'
    pod 'FirebaseAppCheck'
    pod 'FirebaseCrashlytics'
    pod 'FirebaseRemoteConfig'

end

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[morganchen12]

When App Check is enabled, error code 16 indicates a request was rejected because the App Check token was missing or invalid. You can view a more granular breakdown of rejected requests in the App Check monitoring graph on Firebase Console. In this case App Check is working as expected and rejecting bad requests.

cc @weixifan, we should probably document this better since there's no online reference materials for this error code.

[morganchen12]

Closing this issue since it appears to be working as intended, but if you reproduce this error on a device with a valid token, please re-open the issue.

[minlite]

Thank you for the reply @morganchen12!

That was one of my guesses. I checked the monitoring graph and it looks like around 12% of production requests fail due to an invalid AppCheck token. It doesn't seem plausible to me that 12% of my customers are using jailbroken devices or using emulated environments. There has to be an issue with AppCheck token generation in firebase SDK to be causing this high percentage of failures. I expect the natural failure rate to be somewhere close to 1-2%.

I initially thought that #11986 could be the culprit (I personally observed this issue on my device), but from my investigation and the log I included, it is clear that the customers are on 10.17.0 (with the fix merged in) and still facing this issue.

I am going to investigate more and try to produce this on a local device or obtain logs from a customer.

[morganchen12]

@minlite, can you file a support ticket? This way the Firebase team can look at your project directly to see if something's going wrong.

[TheHmmka]

The same situation is here. Part of our audience is failing with Error Code 16...

[minlite]

@minlite, can you file a support ticket? This way the Firebase team can look at your project directly to see if something's going wrong.

I have opened support case 10259958.