Support enrolling newer RTL8720cm/CLIP firmware (protocolVer 4.9): publicKey whitespace fix + generic device fallback

[aaronsb]

Before you read

I wasn't sure how to introduce this into the project, because in order to enroll the machine, I had to broaden some assumptions built into the code that rethink has now - for instance, greedy device enroll vs selective, QA mode in headers, geo regions, crypto pairing, cert storage tics, etc. It's also not finished - I had to cycle the power on the dryer after enrollment was finished to put it in the active state, because I don't know what command to send back to issue a reset. Anyway, it's here for reference, not necessarily to pull in.

What & why

Brings up a current-gen LG appliance — an BDH_D30007_US dryer (DeviceType 202, RTK_RTL8720cm
"CLIP" module, protocolVer 4.9, sw 2.11.263) — fully local on rethink. It now completes SoftAP →
/route → cert → MQTT clip, enrolls, and streams telemetry with zero LG cloud contact (verified
across a full ~50-minute dryer cycle: one MQTT connection, 0 undeploys, ~960 telemetry packets).

Two focused code changes + a docs page. Each commit is self-explaining.

1. rethink-setup: strip whitespace from the setup publicKey (the real blocker)

The hardcoded setup publicKey is a tab-indented template literal, so every base64 line carries a
leading TAB inside the PEM. Older firmware tolerates this; the RTL8720cm CLIP parser is strict —
with the indentation, getDeviceInfo's RSA-encrypt step fails:

encrypt_val: ''
extra: 'POWER_ON|...|encryptRes:ffff'

…so the device can't build a cert request and loops on /route forever — which looks like a /route
bug but isn't. A clean PEM (same key bytes, base64 at column 0) → encryptRes:0, a valid
encrypt_val, and it proceeds straight through /route/certificate → cert → MQTT.

2. ha_bridge: generic raw-capture fallback for unknown device types

Today an unknown modelId is dropped (device type ... unknown → return), so a brand-new appliance
can provision perfectly and still never enroll — and you get no captured data to build a class from.
The fallback enrolls any unknown thinq2 device via a minimal class that completes onboarding (so it
stays connected) and republishes raw packet hex to a diagnostic HA sensor. Adding real support then
becomes a decode exercise against live data instead of guesswork.

3. docs/enrolling-newer-firmware.md

Ties it together and documents the findings that don't need code here: the legacy-TLS profile (your
issues #17/#18), the post-provision power-cycle this firmware needs to settle, and the things that
were NOT the problem (fake OTP, shared publicKey, minimal /route body, svcphase) so nobody
re-chases them.

Deliberately excluded / honest caveats

• No changes to the OTP, the /route body, or TLS defaults here — kept focused. The OTP and the
  minimal /route body work as-is for this firmware; the legacy-TLS profile is your Support newer OpenSSL versions #17/Handshake failed #18.
• Our working setup also ran svcphase: 'OP' (vs the debug-UART 'QA' default). We could not
  isolate whether OP is required for this firmware, so we left your default alone — flagging in
  case it matters for other newer modules.
• The post-provision power-cycle is documented as a manual step; whether a clean software-triggerable
  reset exists is an open question.

What I'd value your input on

• Does "strict PEM parser on newer CLIP firmware" match your experience with other modules, or is it
  specific to RTL8720cm?
• Is there a cloud- or setup-triggerable clean reset that would avoid the manual power-cycle?
• Preferred shape for the generic fallback — always-on, or behind an opt-in flag?

I can share full packet captures + decrypted MQTT clip transcripts (SoftAP setup, /route, cert,
deploy/completeProvisioning/_ack, telemetry). Device did fe8b2ea0-…-3034dbd055fe.

---

This was obviously drafted with Claude code and my edits and steering. Further work I won't continue to disclose that, assume it is co-written or just me replying like a caveman or something.

[anszom]

Good find! It's likely that the whitespace issue might have been the root cause for #58.

I'll need some time to work through the backlog of PRs and issues, so I'll make more comments later.

[anszom]

Is there a cloud- or setup-triggerable clean reset that would avoid the manual power-cycle?

In some old notes I've found mention that a "resetDevice" command does exactly that. You can try it on your module.
That string is also present in the LG app, so it's plausible that they use it - at least for some devices.

[diijkstra]

My device did only one pull

[diijkstra]

This did not occured on my machine, it stayed connected

[diijkstra]

Worked on my device, x-service-phase was set to QA, version 2.11.306 of the modem clip_ble_v1.9.215.

I would suggest checking encrypt_val in provisioning flow and aborting if it is empty string, will be easier to catch failures.

I am not sure about reboot after provisioning, my device did not needed it. logs are at #58 (comment)

[maciejsszmigiero]

The Home Assistant instance does not seem to have much use for such "placeholder" device - why not just leave the packets streaming to clip/message/devices/ and let someone read it from there much like tools/packet-parser.ts does?

This will also prevent polluting HA dashboard or history.

[anszom]

I am not sure about reboot after provisioning, my device did not needed it. logs are at #58 (comment)

Indeed this is the first device that appears to require this. But if it works, then maybe we should simply send the reboot command to every device for maximum compatibility.

[anszom]

The Home Assistant instance does not seem to have much use for such "placeholder" device - why not just leave the packets streaming to clip/message/devices/ and let someone read it from there much like tools/packet-parser.ts does?

Agreed. I don't think mapping the raw packets to HomeAssistant would provide any advantages.

Raw packets can also be observed through the management web UI & its underlying websocket.

[aaronsb]

My intention for tracking raw packets was to collect as much variety of history as possible (over a week or two of operations) which can then be dumped as a full body of data all at once. For me, maintaining multiple endpoints is inconvenient in a home network.

I performed some testing this morning and if x-service-phase is set to QA, enrollment of the dryer fails repeatedly, until set to OP. The newer firmware seems to want qic-qa-* routing and reject common.lgthinq.com (4.9+)

[aaronsb]

Just thinking out loud here - if someone intentionally or unknowingly enrolls a machine that's not supported, I think it would be a much better experience if the machine can at least be captured (a phase one, if you will) and marked as not supported (...yet 😉) (the intention of the code changes) which means that the traces can be exported. As new versions of the backend get opcode translation, then the unrecognized opcodes are marked as supported and data arrives in HA.

It seems like a lot of extra steps to just discard opcodes. Anyway, that's why I marked this as a draft PR with a teach/ header - no reason to pull any of this in if it doesn't fit the architecture you have in mind.

[oirad]

Confirming this fixes enrollment for LG wall-mount AC units with the same firmware generation.

Device info: RAC_0B0001_WW, modem clip_ble_v1.9.223, RTK_RTL8720cm, protocolVer: 4.9, deviceType: 401. Five units total (models EZ09CSNCSJ1, EZ12CSUCA31, EZ12CSNNCSJ1), deployed in Italy — the device itself reports countryCode: DE in the preDeploy payload.

---

Symptom before fix — getDeviceInfo response:

{
  "cmd": "getDeviceInfo",
  "data": {
    "protocolVer": "4.9",
    "deviceType": "401",
    "modelName": "RAC_0B0001_WW",
    "modemVer": "clip_ble_v1.9.223",
    "demandType": "RTK_RTL8720cm",
    "isReg": "N",
    "countryCode": "DE",
    "encrypt_val": "",
    "extra": "POWER_ON|...|encryptRes:ffff",
    "regStep": 0
  }
}

encrypt_val: '' and encryptRes:ffff exactly as described — RSA key exchange failing because the RTL8720cm strict PEM parser rejects the indented base64 block in the template literal.

After applying the fix (rethink-setup.ts lines 76–83, removing leading tabs from the PEM block), provisioning completes cleanly: setCertInfo result: 000, setApInfo result: 000, releaseDev result: 000.

---

Additional finding: devices connect to port 443, not 4433

After provisioning, the device connects back on port 443. tcpdump before setting https_port: 443 in the rethink config:

IP <device-ip>.44590 > <ha-ip>.53:  A? common.lgthinq.com. → <ha-ip>
IP <device-ip>.51735 > <ha-ip>.443: Flags [S]   ← SYN to :443
IP <ha-ip>.443 > <device-ip>.51735: Flags [R.]  ← RST — rethink not listening on 443
IP <device-ip>.51736 > <ha-ip>.443: Flags [S]   ← retry
IP <ha-ip>.443 > <device-ip>.51736: Flags [R.]  ← RST again

DNS resolves correctly, then immediately SYNs to :443. Setting https_port: 443 resolves it.

---

After both fixes — rethink logs confirming device connection:

... HTTPS common.lgthinq.com /route
... HTTPS rethink.lgthinq.com /route/certificate
... HTTPS rethink.lgthinq.com /route/certificate?name=common-server
... HTTPS rethink.lgthinq.com /device/<uuid>/certificate
... HTTPS rethink.lgthinq.com /route/certificate?name=aws-iot
... incoming clip/provisioning/devices/<uuid>  cmd: preDeploy  kind: RAC_0B0001_WW
... incoming lime/devices/<uuid>               cmd: completeProvisioning
... incoming clip/message/devices/<uuid>       cmd: completeProvisioning_ack
... incoming clip/message/devices/<uuid>       cmd: device_packet  data: "000004000000870204A204..."
... incoming clip/message/devices/<uuid>       cmd: device_packet  data: "000004000000870204F304..."

Device streams device_packet telemetry continuously. No power cycle needed — releaseDev alone was sufficient.

Note: the thinq2 device type RAC_0B0001_WW unknown log line is a separate issue (missing device handler) — unrelated to this PR.

[anszom]

@aaronsb @oirad thanks for all the diagnostics, this should hopefully finally resolve #58.

[joneshf]

The Home Assistant instance does not seem to have much use for such "placeholder" device - why not just leave the packets streaming to clip/message/devices/ and let someone read it from there much like tools/packet-parser.ts does?

Agreed. I don't think mapping the raw packets to HomeAssistant would provide any advantages.

Raw packets can also be observed through the management web UI & its underlying websocket.

So I tried to just use the changes from first commit here, and the device starting provisioning, but then went offline after a few minutes. After using the changes from both the first and second commits here, the device has stayed online overnight. Which is to say, I'm not sure how much the reverse engineering you can do without the device staying online for more than a few minutes.

Maybe I did something wrong the first time, or maybe there's something else going on with these RTL8720cm devices where they go offline after a few minutes that the other devices don't do (and so that's why it seems like it needs to be fully provisioned into Home Assistant). But maybe there's a way to finish provisioning enough that the device can communicate with rethink, and stay online indefinitely without also sending data to Home Assistant. Would that be an acceptable compromise?

[anszom]

So I tried to just use the changes from first commit here, and the device starting provisioning, but then went offline after a few minutes. After using the changes from both the first and second commits here, the device has stayed online overnight. Which is to say, I'm not sure how much the reverse engineering you can do without the device staying online for more than a few minutes.

Can you repeat the test? As far as I can tell from reading the sources, the device should not see any difference whether we expose its packets to HA or not.

[joneshf]

Hmm, sorry if this sounds dense, but what exactly are you looking for me to do? I assume the only way to recreate the conditions would be to factory reset the device. Because anything else will be just like the second time I tried to provision it from the changes in this PR, no? Or am I misunderstanding something?

[joneshf]

I actually wonder a Wi-Fi reset is equivalent to the manual power cycle that was mentioned above, and it's happenstance that I did that to get it to stay provisioned. Like, they're not the same thing, but maybe they have the same effect?

[aaronsb]

I actually wonder a Wi-Fi reset is equivalent to the manual power cycle that was mentioned above, and it's happenstance that I did that to get it to stay provisioned. Like, they're not the same thing, but maybe they have the same effect?

This is the behavior my devices had. In my situation I just unplugged the dryer and plugged it back in while it's status was "finishing wifi setup" (and it was reachable on network via icmp)...after the fixes for the cert newline things and changed out of qa mode. Once there, a quick power cycle the dryer came directly up on wifi and began sending mqtt data.

[joneshf]

I actually wonder a Wi-Fi reset is equivalent to the manual power cycle that was mentioned above, and it's happenstance that I did that to get it to stay provisioned. Like, they're not the same thing, but maybe they have the same effect?

This is the behavior my devices had. In my situation I just unplugged the dryer and plugged it back in while it's status was "finishing wifi setup" (and it was reachable on network via icmp)...after the fixes for the cert newline things and changed out of qa mode. Once there, a quick power cycle the dryer came directly up on wifi and began sending mqtt data.

Then maybe that's all I had to do, and using the other commits was a red herring. I.e. it didn't matter that I added the other commits, what mattered that I re-provisioned after the initial provision.

[joneshf]

As far as moving forward with the fix, should the first commit be split out so it can be merged and the RTL8720cm devices can be provisioned again with rethink?

[XanderLuciano]

Confirming that part of this PR fixed my issue attempting to connect my microwave to rethink. I'm just trying to intercept log and reverse engineer things, so getting the device provisioned successfully was my primary blocker so far.

Device type: WMVEL2137
Chipset: RTL8720cm

Part of this PR should get merged because it's actually fixing a critical bug. I support splitting the commit to a simpler narrow scoped change to just get the provisioning working again so others can add devices without issue as well. Right now, adding newer chipsets is completely broken.

[XanderLuciano]

Successfully tested this PR with an LG WMVEL2137 microwave/hood (DeviceType 302, RTL8720cm CLIP, firmware clip_ble_v1.9.210, protocolVer 6, softVer 20230920).

Everything confirmed to work:

1. PublicKey whitespace fix ✅ — Exact same symptoms: tab-indented PEM caused encryptRes:ffff, device looped on /route. Stripping tabs fixed it — device proceeded through cert signing immediately.

2. Generic device fallback ✅ — WMVEL2137 is unknown (thinq2 device type WMVEL2137 unknown — enrolling with generic raw class) and enrolled successfully. Raw hex packets published to HA diagnostic sensor.

3. Hostname in /route response must match DNS — Initially used rethink.lan which the device couldn't resolve. Changed to common.lgthinq.com with DNS forwarder → NUC, device resolved fine.

4. Bridge mode + local-only can coexist — DNS forwarder redirects common.lgthinq.com → NUC for the appliance. NUC uses /etc/hosts override (common.lgthinq.com → real CloudFront IP) so bridge mode can talk to LG's real cloud simultaneously.

Additional findings from the bridge mode handshake:

Beyond the preDeploy/completeProvisioning that local rethink handles, the bridge showed two things needed to exit setup cleanly:

• provisioningType: "deploy" (not "preDeploy") on the completeProvisioning response
• resp_timesync from the cloud (GgYOBRIaAg== — protobuf timestamp)
• Cloud config packet aa1cf0ed114101... (sets monitoring intervals)
• Periodic cloud keepalive aa08f0ef004682bb (~every 20s)

After these, the microwave chimed and exited setup mode.

Device details captured:

• Model: MVEL2033F (sticker) / WMVEL2137 (firmware)
• Device ID: 99b2e9f5-ae2e-1569-980b-3034db792a1e
• WiFi: SpaceLaser, RSSI -63, 4800 baud
• Regional code: aic, Country: US, Timezone: -0800

TLV packets observed (raw hex):

• AA1F4131... — device init/identification (contains "SAA43115104")
• AA6241EC... — 98-byte status/telemetry packet
• AA084100430063bb — config acknowledgment

Thanks for the detailed writeup in docs/enrolling-newer-firmware.md — saved hours of debugging.

[XanderLuciano]

Update: Local-only provisioning now works end-to-end, no bridge mode needed.

After analyzing the bridge mode handshake, I found the device needed provisioningType: "deploy" + resp_timesync before completeProvisioning_ack to exit setup. Three small fixes:

1. cloud/thinq2/provisioning.ts — always respond with "deploy"

- provisioningType: payload.cmd,
+ provisioningType: 'deploy',

The device sends preDeploy, but generateDeployResponse echoed back preDeploy — telling the device "not done." Responding with "deploy" tells it to complete provisioning and enter normal mode.

2. cloud/thinq2/device.ts — handle req_timesync before completeProvisioning_ack

- if (payload.cmd === 'req_timesync' && client.deviceObj && ...)
+ if (payload.cmd === 'req_timesync' && client.deployMsg && ...)

The device sends req_timesync BEFORE completeProvisioning_ack, but the handler checked client.deviceObj which only exists AFTER ack. Changed to check client.deployMsg which exists as soon as preDeploy/deploy is received.

3. timeSyncRequest — publish directly instead of via deviceObj.send()
Replaced client.deviceObj?.send(...) with direct broker publish — works without deviceObj.

Result: factory reset → provision → device completes full handshake locally → chime → exits setup. Bridge mode can then optionally register with LG cloud (already works with the /etc/hosts DNS split trick).

With the publicKey whitespace fix, generic fallback, raw logging, and local handshake, this is now a complete pipeline for bringing up an unsupported RTL8720cm device entirely locally.

[anszom]

I've taken the liberty to split this into three parts:

1. I've merged commit 7c52739. Along with some other smaller changes, this should be a big step towards resolving catch-all issue: device provisioning doesn't work #58. Thanks!
2. I've copied your write-up to the wiki.
3. I'm still not convinced about forwarding raw data to HomeAssistant.

[anszom]

@XanderLuciano

Hostname in /route response must match DNS — Initially used rethink.lan which the device couldn't resolve. Changed to common.lgthinq.com with DNS forwarder → NUC, device resolved fine.

I'd guess that this may not be related to the domain being exactly the same. Official LG servers use various hostnames for various services. My bet is that rethink.lan is not resolvable via DNS (mDNS maybe?), or has a zero TTL.

Bridge mode + local-only can coexist — DNS forwarder redirects common.lgthinq.com → NUC for the appliance. NUC uses /etc/hosts override (common.lgthinq.com → real CloudFront IP) so bridge mode can talk to LG's real cloud simultaneously.

I've tried this approach before, but it's not reliable. CloudFront changes IP assignments without notice, and you end up with a dead entry.

1. cloud/thinq2/provisioning.ts — always respond with "deploy"
2. cloud/thinq2/device.ts — handle req_timesync before completeProvisioning_ack
3. timeSyncRequest — publish directly instead of via deviceObj.send()

I will need to test this - at least with my own AC, to verify that the changes are portable across devices. Can you open a separate issue/PR for this? EDIT: never mind, I've just noticed your PR.

[aaronsb]

I've taken the liberty to split this into three parts:

1. I've merged commit 7c52739. Along with some other smaller changes, this should be a big step towards resolving catch-all issue: device provisioning doesn't work #58. Thanks!
2. I've copied your write-up to the wiki.
3. I'm still not convinced about forwarding raw data to HomeAssistant.

Maybe make it a configuration mode of the server; discovery/learning/verbose mode and a "production" ode where the flag turns off raw forwarding.

Raw mode has been helpful to snoop cycle data on the washer and dryer.

[joneshf]

I've taken the liberty to split this into three parts:

1. I've merged commit 7c52739. Along with some other smaller changes, this should be a big step towards resolving catch-all issue: device provisioning doesn't work #58. Thanks!
2. I've copied your write-up to the wiki.
3. I'm still not convinced about forwarding raw data to HomeAssistant.

Maybe make it a configuration mode of the server; discovery/learning/verbose mode and a "production" ode where the flag turns off raw forwarding.

Raw mode has been helpful to snoop cycle data on the washer and dryer.

Let me put another vote for forwarding the raw data. rethink seems to only show the packets when you have the page open. If you refresh, or otherwise lose the connection to rethink, you lose the packets.

I'm trying to add support for a dishwasher. It sometimes has multiple hour long cycles. I can't sit at the computer watching the rethink server for hours. But I can start a cycle, let the packets forward to Home Assistant, and check them afterward at my leisure.

There's also things like the "refill rinse aid" light that will only come on after so many cycles. I live in a house with other people that run the dishwasher. So, I cannot watch rethink every time someone starts a cycle to figure out when that changes and try to reverse engineer that part of the packet (if it even does show up!). But with the packets going to Home Assistant, all I have to do is wait until I see the light or hear from someone else that they refilled the rinse aid, and then I can check the packets over the timframe to try and reverse engineer it.

I think for certain devices, you probably could use rethink's real time packet logs. But for other devices, not having history makes this reverse engineering process much harder than it needs to be. And i think giving us the option to forward the packets to Home Assistant (so we can choose whether we want the data or not) is beneficial. I understand you don't want to flood folks Home Assistant databases by default. But making it an opt-in "I know what I'm doing" configuration would be very useful.

[joneshf]

And FWIW, I've had this forwarding the raw packets to Home Assistant for the past couple of weeks, and it hasn't ballooned my Home Assistant database any more than normal. I think the data doesn't persist to long-term storage, because I can't see anything older than 7 days. But I don't really know how persistence works in Home Assistant. I just see the (lack of) difference in backup size, and infer that it's not making that much difference. But maybe I'm not looking at the right thing.

[maciejsszmigiero]

To be clear, even if we end up adding an ability to forward raw packets from an (un)known device to a HA sensor I am still against enabling this functionality by default.

Some "device-handler-developer" config file switch would probably be okay though.