Create a service binding for the sample app with microgateway-coresident plan failed

[phong2tran]

We're in the process of evaluating and adopting Apigee Edge for application API management and I'm tasked with getting Apigee Edge to work apps deployed in Cloud Foundry (CFAR). Currently I'm exploring the Apigee service broker using microgateway-coresident plan.

I got the Apigee service broker (https://github.com/apigee/cloud-foundry-apigee/blob/master/docs/setup-microgateway-coresident.md#step-1-install-the-apigee-service-broker-from-source) and the microgateway buildpack decorator (https://github.com/apigee/cloud-foundry-apigee/blob/master/docs/setup-microgateway-coresident.md#step-2-install-the-decorator) deployed, and now I'm following this guide to deploy the coresident sample app (https://github.com/apigee/cloud-foundry-apigee/tree/master/samples/coresident-sample). However, I ran into this error when binding the service instance to the sample app:

$ cf bind-service apigee-coresident-sample coresident-service -c '{"org":"amer-poc2","env":"test","bearer":"'$(cat ~/.sso-cli/valid_token.dat)'","action":"proxy bind","protocol":"http","target_app_route":"apigee-coresident-sample.bp-paas.otxlab.net","edgemicro_key":"e94ff8e1215dccba78192ad77c719b694e2494afa9105c977f510d19********","edgemicro_secret":"e08c7a4555a8a1a23d6ff31cf107e9c40cf5b97693f27065651fcbd1********","target_app_port":"8081"}'
Binding service coresident-service to app apigee-coresident-sample in org dev / space phongspace as admin...
Service broker error: [E0032] - Error uploading proxy to Apigee
FAILED

When checking the Apigee service broker log, I found this error:

2020-02-07T17:29:40.61-0800 [APP/PROC/WEB/0] OUT {"name":"apigee","hostname":"324e93ac-4b24-45a1-6c89-9502","pid":13,"level":50,"errAt":"at /home/vcap/app/helpers/open_api.js:47:36","errDetails":{"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":8081,"name":"Error","message":"Error downloading file "http://localhost:8081/openApi.json\" \nconnect ECONNREFUSED 127.0.0.1:8081","stack":"Error: Error downloading file "http://localhost:8081/openApi.json\" \nconnect ECONNREFUSED 127.0.0.1:8081\n at ono (/home/vcap/app/node_modules/swagger-parser/node_modules/ono/lib/index.js:67:20)\n at ClientRequest. (/home/vcap/app/node_modules/swagger-parser/node_modules/json-schema-ref-parser/lib/read.js:195:16)\n at ClientRequest.emit (events.js:182:13)\n at Socket.socketErrorListener (_http_client.js:391:9)\n at Socket.emit (events.js:182:13)\n at emitErrorNT (internal/streams/destroy.js:82:8)\n at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)\n at process._tickCallback (internal/process/next_tick.js:63:19) \n\nError: connect ECONNREFUSED 127.0.0.1:8081\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1113:14)"},"msg":"[E0060] - Error getting OpenAPI interface file","time":"2020-02-08T01:29:40.611Z","v":0}

The Apigee service broker is trying to connect to this URL (http://localhost:8081/openApi.json) to download the Open API document and it failed to make the connection. It looked to me that the service broker is trying to connect to the target application (the sample app) to fetch the Open API document using the localhost.

I have a few questions:

1. Why the localhost is being for the target application in this case?
2. How can the Open API document be downloaded from the sample app at the time of service binding if the app is not yet started (cf push --no-start)? The target app will only be started up at the next step (cf v3-push) after the service binding step.

I try with 3.1.0 release (https://github.com/apigee/cloud-foundry-apigee/releases/tag/v3.1.0) and latest from github, but have the same error.

Service Broker Env:

$ cf env apigee-cf-service-broker
Getting env variables for app apigee-cf-service-broker in org dev / space phongspace as admin...
OK

System-Provided:
...
User-Provided:
APIGEE_CONFIGURATIONS: [
{
"org": "amer-poc2",
"env": "test",
"apigee_dashboard_url": "https://enterprise.apigee.com/platform/#/",
"apigee_mgmt_api_url": "https://api.enterprise.apigee.com/v1",
"apigee_proxy_domain": "apigee.net"
}
]
HTTPS_PROXY: http://bp2-prox01-l001.otxlab.net:3128
HTTP_PROXY: http://bp2-prox01-l001.otxlab.net:3128
SECURITY_USER_NAME: apigee
SECURITY_USER_PASSWORD: *****

Coresident Service Instance:

$ cf service coresident-service
Showing info of service coresident-service in org dev / space phongspace as admin...

name: coresident-service
service: apigee-edge
tags:
plan: microgateway-coresident
description: Apigee Edge API Platform
documentation: http://apigee.com/docs/
dashboard: https://enterprise.apigee.com/platform/#/

Showing status of last operation from service coresident-service...

status: update succeeded
message:
started: 2020-02-08T01:16:46Z
updated: 2020-02-08T01:16:46Z

There are no bound apps for this service.

Could you please help to take a look what to see what is the cause of this problem?

Thanks,
Phong

[KyleWiese]

Hi Phong,

Unfortunately, I'm not able to replicate your issue on my end. If I had to guess as to what's causing your error is something to do with your HTTPS/HTTP proxy setup. What the service broker is trying to do is get the open api spec from the swagger parser (more specifically https://github.com/apigee/cloud-foundry-apigee/blob/master/apigee-cf-service-broker/helpers/edge_proxy.js#L161 ---> https://github.com/apigee/cloud-foundry-apigee/blob/master/apigee-cf-service-broker/helpers/open_api.js#L41). This is all within the proxy action flow of the bind and doesn't have anything to do with trying to reach out to your separate cloud foundry application.

One way to get around this is to not do proxy bind on the service binding, but rather just do bind and make the proxy manually as a separate step. I've attached a sample proxy that should look like what you want within your apigee org (you'll want to replace the app name and port with whatever your cloud foundry application is configured as).

edgemicro_cf-sample-issue.apps.apigee-demo.net_rev1_2020_02_11.zip

[phong2tran]

Thank you for your response Kyle! After my Apigee Edge account was given the permission to create API proxies, the service binding operation is now successfully and I can see an API proxy created in the our org/environment. However, I still see the error in the Apigee service broker log:

"Error downloading file "http://localhost:8081/openApi.json\" \nconnect ECONNREFUSED 127.0.0.1:8081"

Based on the code your pointed out (https://github.com/apigee/cloud-foundry-apigee/blob/master/apigee-cf-service-broker/helpers/edge_proxy.js#L16), the generatePolicy() method is expecting a dummyTargetUrl argument, and based on this conditional statement (https://github.com/apigee/cloud-foundry-apigee/blob/master/apigee-cf-service-broker/helpers/edge_proxy.js#L144-L150), it's trying to check on the length of the proxyData.micro_coresident and set the the dummyTargetUrl to the localhost if micro_coresident is set. This might be what is causing the error as we're using the microgateway coresident plan in this case. Could you take a look and help to explain what is happening here?

Thanks,
Phong

[KyleWiese]

Glad you were able to bind the service!

One thing to try (since I'm unable to replicate this on my end) is to update the swaggerParser.parse from https://github.com/apigee/cloud-foundry-apigee/blob/master/apigee-cf-service-broker/helpers/open_api.js#L41 to look something like:

swaggerParser.parse(url, {resolve: {http: false}}, function (err, api, metadata) {

That way the swagger-parser doesn't attempt to resolve the localhost:8081 $ref

[phong2tran]

Hi Kyle,
I still encountered the same error after making the change you suggested in open_api.js.

2020-02-19T17:22:14.23-0800 [APP/PROC/WEB/0] OUT {"name":"apigee","hostname":"abbb7a7a-7f4d-4c1d-5a36-9ec1","pid":14,"level":30,"msg":"Get virtual hosts url: **https://api.enterprise.apigee.com/v1/organizations/amer-poc2/environments/test/virtualhosts**","time":"2020-02-20T01:22:14.236Z","v":0}
2020-02-19T17:22:14.45-0800 [APP/PROC/WEB/0] OUT {"name":"apigee","hostname":"abbb7a7a-7f4d-4c1d-5a36-9ec1","pid":14,"level":50,"errAt":"at /home/vcap/app/helpers/open_api.js:47:36","errDetails":{"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":8081,"name":"Error","message":"Error downloading file "http://localhost:8081/openApi.json\" \nconnect ECONNREFUSED 127.0.0.1:8081","stack":"Error: Error downloading file "http://localhost:8081/openApi.json\" \nconnect ECONNREFUSED 127.0.0.1:8081\n at ono (/home/vcap/deps/0/node_modules/json-schema-ref-parser/node_modules/ono/lib/index.js:61:20)\n at ClientRequest. (/home/vcap/deps/0/node_modules/json-schema-ref-parser/lib/read.js:195:16)\n at ClientRequest.emit (events.js:182:13)\n at Socket.socketErrorListener (_http_client.js:391:9)\n at Socket.emit (events.js:182:13)\n at emitErrorNT (internal/streams/destroy.js:82:8)\n at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)\n at process._tickCallback (internal/process/next_tick.js:63:19) \n\nError: connect ECONNREFUSED 127.0.0.1:8081\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1113:14)"},"msg":"[E0060] - Error getting OpenAPI interface file","time":"2020-02-20T01:22:14.455Z","v":0}

see the attached file for all log entries from the Apigee service broker.
apigee-service-broker-binding-error.txt

There is a call to get virtual hosts (https://github.com/apigee/cloud-foundry-apigee/blob/master/apigee-cf-service-broker/helpers/edge_proxy.js#L139)
from Edge Cloud service in the error. As virtual hosts are used for routing requests to the API proxy by Edge Router (https://docs.apigee.com/api-platform/fundamentals/virtual-hosts), why are virtual hosts needed in the context of Edge Microgateway? I thought when we're using the microgateway, the routing to API proxies (edge microgateway-aware) are not handled through virtual hosts by Edge Router. I'm trying to understand what role of virtual hosts and policies play in creating edge microgateway-aware API proxies? and how would this error affect the functionality of microgateway deployment?

Thanks,
Phong