Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does deCONZ serial protocol for ConBee/RaspBee Zigbee Coordinator adapters not support joining/pairing via install code and qr code? #20

Open
Hedda opened this issue Feb 12, 2023 · 34 comments

Comments

@Hedda
Copy link

Hedda commented Feb 12, 2023

@manup Reposting from zigpy/zigpy-deconz#214 and Koenkk/zigbee2mqtt#17492 as a public request to you dresden-elektronik's deCONZ (ConBee) developers because the developers of zigpy (used by Home Assistant's ZHA integration + Zigbee plugins for Domoticz and Jeedom) and zigbee-herdsman (Zigbee2MQTT + IoBroker) replied there that this function is not supported in zigpy-deconz or zigbee-herdsman since the function for install code (and qr code) is not a documented in the linked deCONZ serial protocol documentation:

https://github.com/dresden-elektronik/deconz-serial-protocol/blob/master/README.md

UPDATE: The exact same issue also applies to Zigbee2MQTT (Z2M) and IoBroker (which are both based on zigbee-herdsman):

Koenkk/zigbee2mqtt#17492

It started with one user of Home Assistant’s ZHA integration with an ConBee Zigbee Coordinator adapter posted that they could not join/pair a Zigbee 3.0 (ZB3) device that required an "install code", and then some other user replied to that post saying "install code" (and "qr code"?) support is not implemented in the zigpy-deconz radio library for zigpy which the ZHA integration depends on. Is that true?

https://github.com/zigpy/zigpy-deconz/blob/80041b9dbf334e0224b95081fcd49bb2da3e8c5d/zigpy_deconz/zigbee/application.py#L108

For reference that specific Zigbee 3.0 device that require install code to join was the "Bosch Thermostat 2":

https://community.home-assistant.io/t/bosch-thermostat-2/492845/

ZHA (and zigpy) can set an Install Code for Zigbee 3.0 devices if using Silicon Labs EmberZNet or Texas Instruments Z-Stack ZNP:

https://www.home-assistant.io/integrations/zha#services

https://www.silabs.com/documents/public/application-notes/an1089-using-installation-codes-with-zigbee-devices.pdf

https://wiki.st.com/stm32mcu/wiki/Connectivity:Zigbee_Install_Code

Just reposting this issue reported in the Home Assistant community forum:

https://community.home-assistant.io/t/bosch-thermostat-2/492845/10

https://community.home-assistant.io/t/bosch-thermostat-2/492845/11

The first device in question was the Bosch Thermostat II 230 V (BTH-RA / BTH-RM230Z), however, I understand that some people using deCONZ or Zigbee2MQTT have reported the same issue with a few other Bosch-branded Zigbee devices, like Bosch Smoke Alarm II (BSD-2), Bosch Door/window contact II (RBSH-SWD-ZB), and Bosch BWA-1 water sensor, all of which require adding the install code to pair/join. Another device that been mentioned to require install code is the rotary switch Aqara H1 -> Koenkk/zigbee-herdsman#420

PS: I do not have a ConBee/RaspBee Zigbee Coordinator setup myself any longer so can not replicate it to confirm or disconfirm this but think that it would be a good idea if this could be tracked as a feature request because the function is missing the deconz serial protocol documentation and for deconz firmware based Zigbee Coordinator adapters like ConBee/RaspBee but supported by other Zigbee Coordinator adapters like Silicon Labs and Texas Instruments based Zigbee Coordinator adapters via their respective radio library for zigpy.

@martgras
Copy link

I think that you have to set the network parameter 'Link Key' at 0x19
(page 12). However I don't have a Conbee Adapter to test it out

@puddly
Copy link

puddly commented Feb 13, 2023

Unfortunately, Link Key (0x19) is the Trust Center Link Key, which is always ZigBeeAlliance09. There needs to be a specific API to add a link key for a specific IEEE address, to replace the well-known key during transport key exchange.

@Hedda
Copy link
Author

Hedda commented Feb 15, 2023

@ChrisHae Can you tell if deconz adapter/driver in zigbee-herdsman (used by Zigbee2MQTT and IoBroker) supports install codes?

https://github.com/Koenkk/zigbee-herdsman

https://github.com/Koenkk/zigbee-herdsman/tree/master/src/adapter/deconz

@martgras
Copy link

@ChrisHae
Copy link

The Conbee2 fw does not support it yet. But it will come soon. And then it will also be implemented in zigbee herdsman deconz adapter.

@Hedda
Copy link
Author

Hedda commented Feb 16, 2023

The Conbee2 fw does not support it yet. But it will come soon. And then it will also be implemented in zigbee herdsman deconz adapter.

Great to hear! If possible please update the documentation then also ping the zigpy developers too -> zigpy/zigpy-deconz#214

@xam-ps
Copy link

xam-ps commented Mar 17, 2023

Any news on this? Unfortunately the Bosch Thermostat 2 only supports pairing via install code :/

@pdsccode
Copy link

pdsccode commented May 7, 2023

Bumping here. Still waiting for the support of pairing via install code to get my devices up and running. Any info on the current state?

@xam-ps
Copy link

xam-ps commented May 8, 2023

@pdsccode
It seems like it's included in the latest version: https://github.com/dresden-elektronik/deconz-rest-plugin/releases/tag/v2.22.0-beta
But it's still in Beta-Status. Waiting for the version to be finally released.

@pdsccode
Copy link

pdsccode commented May 8, 2023

@xam-ps
Thanks for the info. Didn't see that obviously 😂

@Hedda
Copy link
Author

Hedda commented May 9, 2023

Can anyone confirm or disconfirm if Zigbee install code support is enabled in the current or latest ConBee/RaspBee firmware?

dresden-elektronik/deconz-rest-plugin#6791

dresden-elektronik/deconz-rest-plugin@24c7ff9

https://github.com/dresden-elektronik/deconz-rest-plugin/releases/tag/v2.22.0-beta

If and when it has can you then please also update the deconz-serial-protocol documentation to include all the commands for it?

Developers of Home Assistant's ZHA integration and Zigbee2MQTT need it to implement it in zigpy-deconz and zigbee-herdsman:

zigpy/zigpy-deconz#214

Koenkk/zigbee2mqtt#17492

@SwoopX
Copy link

SwoopX commented May 9, 2023

What makes you think this has anything to do with the device firmware? You can see all the necessary changes to make the install codes work from the referenced PR. To my knowledge, last changes involving the firmware were about a year ago or so and none of the change log entries reference install codes.

Generally, I cannot comment on the serial protocol, since I have no clue about it.

@Hedda
Copy link
Author

Hedda commented May 10, 2023

So you are saying all that is missing is the documentation for the deconz serial protocol?

What makes you think this has anything to do with the device firmware?

That is what ChisHae wrote above:

Originally posted by @ChrisHae in #20 (comment)

The Conbee2 fw does not support it yet. But it will come soon. And then it will also be implemented in zigbee herdsman deconz adapter.

Again, this interface information is needed in the deconz serial protocol documentation in order zigpy and zigbee-herdsman for third parties to be able to communicate with the Zigbee devices via ConBee/RaspBee as Zigbee Coordinator adapters:

https://github.com/dresden-elektronik/deconz-serial-protocol/blob/master/README.md

zigpy is for example used by Home Assistant's ZHA integration, the Zigbee Plugin for Domoticz, and Zigbee Plugin for Jeedom.

https://github.com/zigpy/zigpy (see zigpy-deconz feature request -> zigpy/zigpy-deconz#214 )

zigbee-herdsman is for example used by Zigbee2MQTT and IoBroker:

https://github.com/Koenkk/zigbee-herdsman (see Zigbee2MQTT feature request -> Koenkk/zigbee2mqtt#17492 )

@SwoopX
Copy link

SwoopX commented May 10, 2023

So you are saying all that is missing is the documentation for the deconz serial protocol?

I more or less said the exact opposite: I don't know anything about the deconz serial protocol. Please read my previous reply again.

Other than that, I also cannot comment if for other software any changes on the firmware might be necessary. For deconz itself, the PR documents the necessary changes to make it work. Nothing more, nothing less 🙂

@puddly
Copy link

puddly commented May 11, 2023

We don't use the binary deCONZ library in ZHA or Z2M though and we instead directly interact with the coordinator over the documented serial protocol. The deCONZ object is a higher-level API so while this code is straightforward:

#if DECONZ_LIB_VERSION >= 0x010B00
            QVariantMap m;
            m["mac"] = uniqueid.toULongLong(&ok, 16);
            m["key"] = mmoHash;
            if (ok && mmoHash.size() == 32)
            {
                ok = deCONZ::ApsController::instance()->setParameter(deCONZ::ParamLinkKey, m);
            }
#endif

Writing the hashed link key and a joining device's IEEE address to this specific parameter and then permitting joins still results in the Conbee sending the network key as usual for me. So unless I'm making some simple mistake somewhere, there may be something else happening in the background within the library or a specific sequence in which this parameter write needs to be sent when permitting joins.

Various questions:

  • Does this command need to be sent before permitting joins? After?
  • How do you clear a link key? If you can't, is it cleared automatically after the join window closes? How many can you write at once?
  • What firmware version was this feature introduced in?

And while I could spend the time to reverse engineer deCONZ by intercepting traffic between it and the Conbee, it's a lot nicer to not have to do this.

@manup
Copy link
Member

manup commented May 11, 2023

Hi, the MAC + derived Link Key is indeed what is used also on the serial protocol side.

image

The firmware doesn't know anything about Install Codes itself and the Link Key is derived as shown in above PR. Formerly this was done with a command line tool from the Zigbee Alliance but this was meh.

The Link Key parameter 0x19 is in the firmware since a long time, also for ConBee I, where we did our initial tests for a student diploma with various real devices. Here we also found that Install Codes sometimes are invalid/non standard, e.g. byte order of the checksum was wrong.

But I'd like to put a big warning on using the 0x19 parameter yet, while it does work it could also make the configuration in the firmware messy, e.g. after setting it classical joining of devices might not work any longer until the trust center MAC address + link key is set again. Fixing this is on the bucket list but doesn't have a high priority currently, since there are too many other pressing things on the todo list.

@Hedda
Copy link
Author

Hedda commented Sep 4, 2023

Any updates on this? ...AFAIK, install code is still not supported for ConBee in Home Assistant's ZHA integration or Zigbee2MQTT?

zigpy/zigpy-deconz#214

Koenkk/zigbee2mqtt#17492

Support for commissioning Zigbee 3.0 devices via “Install Code” or “QR Code” via the ‘zha.permit’ service has so far only been implemented for ‘ezsp’ (Silicon Labs EmberZNet) or ‘znp’ (Texas Instruments Z-Stack) radio type adapters in the ZHA integration in Home Assistant (that depends on zigpy-deconz) and Zigbee2MQTT (that depends zigbee-herdsman) . Other radio types are missing support in their respective radio libraries for zigpy or manufacturer’s firmware commands/APIs.

https://www.home-assistant.io/integrations/zha#limitations

@Hedda
Copy link
Author

Hedda commented Oct 13, 2023

So you are saying all that is missing is the documentation for the deconz serial protocol?

I more or less said the exact opposite: I don't know anything about the deconz serial protocol.

Mixed signals(?); as according to @ChrisHae support for using install code is not yet implemented in the deconz serial protocol:

Originally posted by @ChrisHae in Koenkk/zigbee2mqtt#17492 (comment)

It is not yet implemented in the serial protocol. When it is, it can be added to z2mqtt deconz adapter.

Again, deconz firmware for ConBee and RaspBee does not yet seem to implement Zigbee install code support in the deconz serial protocol and therefore ConBee/RaspBee does not currently offer feature-partity with with serial protocols for Texas Instruments Z-Stack and Silicon Labs EmberZNet for Zigbee Gateway host applications such as the ZHA integration in Home Assistant (which is turn depends on the zigpy and zigpy-deconz libraries) and Zigbee2MQTT (which in turn depends on zigbee-herdsman).

Any updates or news on this from dresden-elektronik's developers?

Can you assist with getting Zigbee install code support working in ZHA (zigpy-deconz) and Zigbee2MQTT(zigbee-herdsman)? See:

zigpy/zigpy-deconz#214

and

Koenkk/zigbee2mqtt#17492

@pixartist
Copy link

Can we get an update on this?

@Pioneer42m
Copy link

I would be also very happy this there was a solution for this. Thank you in advance!

@lerra
Copy link

lerra commented Dec 13, 2023

Any update on this?

@finkerle
Copy link

Hi, any updates on this?

@Hedda
Copy link
Author

Hedda commented Mar 28, 2024

Any updates of deCONZ Serial Protocol documentation to be able to support "install code" for ConBee III (ConBee 3) adapter?

https://phoscon.de/en/conbee3

As it stands, deconz "install code" and "qr code" support is not available in Home Assistant's ZHA integration of Zigbee2MQTT:

zigpy/zigpy-deconz#214

Koenkk/zigbee2mqtt#17492

PS: Still listed as a deconz serial protocol related limitation under ZHA docs and until then any ConBee cannot be recommended:

https://www.home-assistant.io/integrations/zha#limitations

"Support for commissioning Zigbee 3.0 devices via “Install Code” or “QR Code” via the ‘zha.permit’ service has so far only been implemented for ‘ezsp’ (Silicon Labs EmberZNet) or ‘znp’ (Texas Instruments) radio type in ZHA. Other radio types are missing support in their respective radio libraries for zigpy or manufacturer’s firmware commands/APIs."

@hksthff
Copy link

hksthff commented Mar 28, 2024

I tested yesterday with Conbee 3 and ZHA without success.

File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1006, in _async_admin_handler await result File "/usr/src/homeassistant/homeassistant/components/zha/websocket_api.py", line 1257, in permit await application_controller.permit_with_link_key( File "/usr/local/lib/python3.12/site-packages/zigpy_deconz/zigbee/application.py", line 115, in permit_with_link_key await self._api.write_parameter( File "/usr/local/lib/python3.12/site-packages/zigpy_deconz/api.py", line 849, in write_parameter await self.send_command( File "/usr/local/lib/python3.12/site-packages/zigpy_deconz/api.py", line 508, in send_command return await self._command(cmd, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy_deconz/api.py", line 589, in _command return await fut ^^^^^^^^^ zigpy_deconz.exception.CommandError: <CommandId.write_parameter: 11>, status: <Status.UNSUPPORTED: 4>

@Hedda
Copy link
Author

Hedda commented Mar 28, 2024

I tested yesterday with Conbee 3 and ZHA without success.

Even if was supported in ConBee 3 firmware it would still have to be implemented in zigpy-deconz -> zigpy/zigpy-deconz#214

And that is not possible until deconz serial protocol documentation is updated to include info about "install code" + "qr code".

Again, deconz serial protocol for ConBee/RaspBee Zigbee Coordinator adapters not support joining/pairing via install code and qr code, or at least it is not yet documented. Until then you can not use ConBee/RaspBee radios with ZHA or Zigbee2MQTT if want to provision/commision via QR-code or install-code:

https://www.home-assistant.io/integrations/zha#limitations

@Primo2966
Copy link

Any news?

@Hedda
Copy link
Author

Hedda commented Sep 9, 2024

@manup Any chance you or someone else from dresden-elektronik could prioritize this and updating the documentation here?

While I can only speak for myself, think I can still say that as it stands right now many of us who are a part of the overall Zigbee2MQTT and ZHA communities can no longer not with good conscience recommend ConBee II or ConBee III to new end-users as they would not be able to add any Zigbee devices that require install code / QR code for provisioning and we have not seen any (public) progress on this for over a year. 😞

This "limitation" with the ConBee/RaspBee firmware and deCONZ Serial Protocol is currently only indirectly mentioned in FAQs:

PS: Imagine though that if support for this can not be added then ConBee will be removed from lists of recommended adapters:

@manup
Copy link
Member

manup commented Sep 11, 2024

Sorry this took a while, I had reworked the document since the former .docx based one was messy to update. The document references are in a bit rough shape but that's for another version. The new version 1.22 has now been uploaded http://deconz.dresden-elektronik.de/raspbian/deCONZ-Serial-Protocol_en_1.22.pdf.

There is a new small section 6.2.3 Zigbee Install Codes which describes the basic usage.

As reference how it works in deCONZ the following sources might be helpful as well:

Calculating the derived Link Key from Zigbee Install Code (I think ZHA and Z2M already have similar code):
https://github.com/dresden-elektronik/deconz-rest-plugin/blob/master/crypto/mmohash.cpp

How the install code is processed by the REST-API and set to the firmware:
https://github.com/dresden-elektronik/deconz-rest-plugin/blob/0e2dc155e85893b439f354cd5d764e414ad0ed0c/rest_devices.cpp#L1090

@Hedda
Copy link
Author

Hedda commented Sep 13, 2024

Sorry this took a while, I had reworked the document since the former .docx based one was messy to update. The document references are in a bit rough shape but that's for another version. The new version 1.22 has now been uploaded http://deconz.dresden-elektronik.de/raspbian/deCONZ-Serial-Protocol_en_1.22.pdf

Nice but that HTTP URL does not work for me

aslo tested HTTPS without any better luck:

@manup
Copy link
Member

manup commented Sep 13, 2024

Are you sure? The link I've posted works here normally 🤔

@Hedda
Copy link
Author

Hedda commented Sep 13, 2024

Are you sure? The link I've posted works here normally 🤔

Yes I get “404 Not Found” error from your Nginx server so should not be on my (client side). See screenshot from Chrome:

image

The “404 Not Found” error is an HTTP status code that indicates the requested resource couldn't be found on the server. In the context of Nginx, this error typically occurs when a client requests a file or page that doesn't exist or isn't properly configured in the Nginx server block or configuration file.

@Hedda
Copy link
Author

Hedda commented Sep 13, 2024

Ah, can get HTTP link to work in other web browsers, but the HTTPS does not work in any webbrowsers, so still issue on your side?

@wu7zh
Copy link

wu7zh commented Sep 13, 2024

http link works for me, https gives errors.
But otherwise awesome that this is now documented!
I really hope it can be implemented into zigbee2mqtt in the near future, need to get the Bosch thermostats on my network before the cold season starts ;)

@littldr
Copy link

littldr commented Oct 12, 2024

Thanks for sharing the updated documentation, @manup!

I took a look at the current permit_with_link_key code used in ZHA (https://github.com/zigpy/zigpy-deconz/blob/dev/zigpy_deconz/zigbee/application.py#L115). If I am not overlooking something, then it already does the same as the code you shared above. It is just a bit more hidden. E.g. the key derivation happens in the zigpy lib (in convert_install_code https://github.com/zigpy/zigpy/blob/dev/zigpy/util.py#L242).

So I gave it a try with a Conbee III (latest firmware 0x26530900), but it seems that the stick returns 0x04 (unsupported) for the 'write parameter' 0x19. 🤔 I would expect 0x05 (Error) if the given payload would be wrong.
Is there a way to get debug logs from the Conbee III directly?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests