Metrics fail to be reported with `INVALID_ARGUMENT` error

[flovouin]

Please make sure you have searched for information in the following guides.

• Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
• Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
• Check our Troubleshooting guide: https://github.com/googleapis/google-cloud-node/blob/main/docs/troubleshooting.md
• Check our FAQ: https://github.com/googleapis/google-cloud-node/blob/main/docs/faq.md
• Check our libraries HOW-TO: https://github.com/googleapis/gax-nodejs/blob/main/client-libraries.md
• Check out our authentication guide: https://github.com/googleapis/google-auth-library-nodejs
• Check out handwritten samples for many of our APIs: https://github.com/GoogleCloudPlatform/nodejs-docs-samples

A screenshot that you have tested with "Try this API".

N/A, I don't understand why this would be required for all bug reports.

Link to the code that reproduces this issue. A link to a public Github Repository or gist with a minimal reproduction.

https://github.com/googleapis/nodejs-spanner

A step-by-step description of how to reproduce the issue, based on the linked reproduction.

Use version 8.2.1, and run the Node.js client within a Cloud Run container (although other execution environments may be affected as well).

A clear and concise description of what the bug is, and what you expected to happen.

When running version 8.2.1 within Cloud Run, I'm getting the following errors :

ERROR: Send TimeSeries failed: 3 INVALID_ARGUMENT: Field resource.labels.project_id had an invalid value of "{{projectId}}": if present, must be the project number or ID in the request name.

Also, both this and the previous (IAM permission-related) error [1] are logged every minute, which generates a lot of useless logs.

[1] When upgrading to the new version of the client, it tries to automatically send the new metrics. Although the Spanner IAM roles are supposed to add the relevant permission, the new monitoring permission must be defined at the project level.
When assigning Spanner roles at the database-level, the monitoring permission is ineffective.
I understand that the Node.js client is not responsible for this permission-related issue. However its default behavior is a problem and generates too many logs in a case that should probably be the default for users (i.e. the client not having the new required permission by default).

This is the second issue (#2373) I post about this feature. Can it be more thoroughly tested, or made opt-in in the mean time?
cc @surbhigarg92

A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **

I expect a new feature to work, and not to log so many errors when it doesn't.

[surbhigarg92]

@flovouin, Thank you for reporting this issue and for your detailed feedback. We apologize for the inconvenience this has caused.

The Client metrics export feature was enabled by default with the intention of helping us provide better support to customers by having these metrics. We designed this feature such that any failures during metric export should not impact your application's critical path. It should only log an error and a warning message, not block any core Spanner functionality.

Regarding the specific concerns you raised:

Excessive Logging: You are correct; logging an error every minute upon failure is not the intended behavior. We have merged a fix for this in Pull Request #2425, which should reduce the log verbosity.

IAM Permissions: We acknowledge that the required monitoring.metricWriter permission needs to be granted at the Project level, not just on the Spanner resource, and that this wasn't clearly documented. We have already identified this gap and working with our documentation team to update the public documentation to make this requirement clear. While we've included necessary permissions within the standard Spanner roles as a baseline, IAM permissions are ultimately controlled by customers within their projects. We are looking into how we can improve this experience.

INVALID_ARGUMENT on Cloud Run: The INVALID_ARGUMENT error you're encountering when running the library on Cloud Run is unexpected. I will attempt to reproduce this issue in a Cloud Run environment to understand the cause and will update this issue with my findings.

Thanks again for your patience and for helping us improve the library.

[flovouin]

Thanks for your reply.

Indeed, the core Spanner client features continued working and I haven't noticed any change on this side. I did however opt out of the built-in metrics using the corresponding environment variable to avoid the error logs in the meantime. Thanks for fixing the amount of logs.

To be honest, knowing about the permission and the roles/monitoring.metricWriter role, it was clear to me that I was going to have to grant it at the project level. But yes, the Spanner documentation and roles are a bit counter intuitive, as it is likely for them to be granted at the instance / database level (especially for a service account with restricted permissions).

Finally, regarding the invalid argument error, while I haven't investigated and don't know enough about the related packages, {{projectId}} does look like the project ID is being fetched while it still has its placeholder value. I don't know why that is, but I can confirm that other Spanner client features (and other GCP packages like Pub/Sub) aren't affected and could correctly resolve the project ID within the Cloud Run service (without the project ID being explicitly specified).