sdc-imgadm import hangs during triton install

[Smithx10]

While trying to install a development triton on VMWare Workstation I ran into the following issue on the release channel.

import being stuck

    [root@headnode (us-east-1) /usbkey/images]# /opt/smartdc/bin/sdc-imgadm import --skip-owner-check -m /usbkey/images/${MYI}.imgmanifest -f /usbkey/images/${MYI}.imgfile
Imported image c2c31b00-1d60-11e9-9a77-ff9f06554b0f (minimal-64-lts, 18.4.0, state=unactivated)
c2c31b00-1d60-11e9-9a77-ff9f06554b0f                                   [                                                                                                                                                                              ]   0%  64.00KB
    

Log from imgapi0

[2020-03-11T02:05:44.423Z]  INFO: imgapi/35166 on 9ea9427c-f772-4d4c-9e8a-0aae718be2ac:  (req_id=22562256-c070-4ec9-be35-c33bd3aa1fc5)
    evt: {
      "ph": "b",
      "name": "updateimage",
      "req_seq": 1583892344422018
    }
[2020-03-11T02:05:44.424Z]  INFO: imgapi/35166 on 9ea9427c-f772-4d4c-9e8a-0aae718be2ac: AdminImportImage: create it (req_id=22562256-c070-4ec9-be35-c33bd3aa1fc5)
    data: {
      "v": 2,
      "uuid": "c2c31b00-1d60-11e9-9a77-ff9f06554b0f",
      "owner": "930896af-bf8c-48d4-885c-6573a94b1853",
      "name": "minimal-64-lts",
      "version": "18.4.0",
      "state": "active",
      "disabled": false,
      "public": false,
      "published_at": "2019-01-21T09:41:43Z",
      "type": "zone-dataset",
      "os": "smartos",
      "files": [
        {
          "sha1": "3f466deb1a044c73bf47c19eaf4c69f8004e3dc2",
          "size": 28864297,
          "compression": "gzip"
        }
      ],
      "description": "A 64-bit SmartOS image with just bootstrap packages installed. Ideal for users who want the smallest possible image upon which to build.",
      "homepage": "https://docs.joyent.com/images/smartos/minimal",
      "urn": "sdc:sdc:minimal-64-lts:18.4.0",
      "requirements": {
        "min_platform": {
          "7.0": "20141030T081701Z"
        },
        "networks": [
          {
            "name": "net0",
            "description": "public"
          }
        ]
      },
      "tags": {
        "role": "os",
        "group": "minimal-64-lts"
      }
    }
[2020-03-11T02:05:46.324Z]  INFO: imgapi/35166 on 9ea9427c-f772-4d4c-9e8a-0aae718be2ac:  (req_id=22562256-c070-4ec9-be35-c33bd3aa1fc5)
    evt: {
      "ph": "e",
      "name": "updateimage",
      "req_seq": 1583892344422018
    }
[2020-03-11T02:05:46.325Z]  INFO: imgapi/35166 on 9ea9427c-f772-4d4c-9e8a-0aae718be2ac: handled: 200 (req_id=22562256-c070-4ec9-be35-c33bd3aa1fc5, route=updateimage, action=import, audit=true, remoteAddress=10.10.10.2, remotePort=42855, err=false, latency=1903, _audit=true, req.version=*)
    POST /images/c2c31b00-1d60-11e9-9a77-ff9f06554b0f?action=import&skip_owner_check=true HTTP/1.1
    accept: application/json
    content-type: application/json
    user-agent: restify/1.6.0 (ia32-sunos; v8/3.14.5.11; OpenSSL/1.0.1u) node/0.10.48
    accept-version: *
    date: Wed, 11 Mar 2020 02:05:44 GMT
    content-length: 774
    content-md5: K76dKa2B7IphdYE9/VsQtg==
    host: imgapi.us-east-1.sm1thcl0udz.com
    connection: keep-alive

    {}
    --
    HTTP/1.1 200 OK
    etag: d22270f53c49806900e47053d9852eff299c1deb
    content-type: application/json
    content-length: 842
    date: Wed, 11 Mar 2020 02:05:46 GMT
    server: imgapi/4.10.0
    x-request-id: 22562256-c070-4ec9-be35-c33bd3aa1fc5
    x-response-time: 1901
    x-server-name: 9ea9427c-f772-4d4c-9e8a-0aae718be2ac

    {
      "v": 2,
      "uuid": "c2c31b00-1d60-11e9-9a77-ff9f06554b0f",
      "owner": "930896af-bf8c-48d4-885c-6573a94b1853",
      "name": "minimal-64-lts",
      "version": "18.4.0",
      "state": "unactivated",
      "disabled": false,
      "public": false,
      "published_at": "2019-01-21T09:41:43Z",
      "type": "zone-dataset",
      "os": "smartos",
      "files": [],
      "description": "A 64-bit SmartOS image with just bootstrap packages installed. Ideal for users who want the smallest possible image upon which to build.",
      "homepage": "https://docs.joyent.com/images/smartos/minimal",
      "urn": "sdc:sdc:minimal-64-lts:18.4.0",
      "requirements": {
        "min_platform": {
          "7.0": "20141030T081701Z"
        },
        "networks": [
          {
            "name": "net0",
            "description": "public"
          }
        ]
      },
      "tags": {
        "role": "os",
        "group": "minimal-64-lts"
      }
    }
    --
    req.query: {
      "action": "import",
      "skip_owner_check": "true"
    }
    --
    req.timers: {
      "pauseReqStreamEtc": 25,
      "handler-0": 241,
      "bunyan": 95,
      "handler-1": 314,
      "parseQueryString": 84,
      "reqNoAuth": 3,
      "reqValidUuid": 12,
      "reqChannel": 5,
      "resume": 3,
      "readBody": 168,
      "parseBody": 35,
      "apiAdminImportRemoteImage": 9,
      "apiAdminImportImage": 1900165
    }
    [2020-03-11T02:05:46.337Z]  INFO: imgapi/35166 on 9ea9427c-f772-4d4c-9e8a-0aae718be2ac:  (req_id=3e49b8b2-d6f2-4459-9cbb-14521afa70c4)
    evt: {
      "ph": "b",
      "name": "addimagefile",
      "req_seq": 1583892346336019
    }  

[Smithx10]

I think this is something up with VMWare workstation or it's networking, but at the moment the send is going at 75B/s and will import in 91h 7m

[root@headnode (us-east-1) /usbkey/images]# /opt/smartdc/bin/sdc-imgadm import --skip-owner-check -m /usbkey/images/${MYI}.imgmanifest -f /usbkey/images/${MYI}.imgfile
Imported image c2c31b00-1d60-11e9-9a77-ff9f06554b0f (minimal-64-lts, 18.4.0, state=unactivated)
c2c31b00-1d60-11e9-9a77-ff9f06554b0f                                   [                                                                                  c2c31b00-1d60-11e9-9a77-ff9f06554b0f   [==========>                                                                     ]  14%   4.06MB      75B/s 91h 7m

[Smithx10]

I tested the link speed, and everything seems fine, not sure where it's getting stuck at.

[Smithx10]

I just booted COAL on vmware workstation and ran through, no issues. Not sure what was causing the issue. There are a lot of variables that could have led to some strange condition. But COAL is working so this doesn't seem to be an issue with the code .

[Smithx10]

I discovered what allowed this to work vs fail. When using e1000g in Vmware or KVM it works. When using vmxnet3 / virtio it fails. This leads me to believe something in the platform driver may be not working.

[Smithx10]

Here are links to the snoops
headnode: https://gist.github.com/91c88161e4a64875bd797dd16914ddae

imgapi: https://gist.github.com/49e0f6c29cd8285d899d5d94cb638758