Race Condition with CO2 Sensor

[cregini]

After about 6 clean readings using the code below, I am getting an error that Atlas suggests may be due to a race condition:

Thoughts?

[timboring]

When you say, "Atlas suggests" are you talking about Atlas Scientific? If so, would you mind copy/pasting the conversation you had with them into this issue?

How frequently does this happen? After the six readings, does it work for another X readings, then error again? What happens if you increase the processing_delay? Does it result in the same behavior? I vaguely recall seeing a similar issue with a different sensor, and I had to find the right processing_delay value by trial-and-error, unfortunately. That said, 1200 seems reasonable given the sensor docs suggest 300.

Is there a reason you're decoding the bytes? What happens if you do this:

carbon = round(float(carbon.data), 1)

I'm using this library in a project to read from temp, pH and dissolved oxygen sensors, and this is how I'm handling the data being returned from the query() call. But, maybe the CO2 sensor behaves differently from these sensors (I haven't used a CO2 sensor). 🤔

[cregini]

I have been decoding the bytes so that I can pass the string to our Adafruit IO dashboard without the leading 'b'. If there is a better way to do that I'd be happy to apply it. The CO2 sensor is being used in conjunction with just about every sensor they have. They are initiated to take a reading with an ignition switch when needed. All of the root zone sensors appear to work fine, but the CO2 sensor returns only a handful of clean readings before the program stops when the error appears. This seems to be exacerbated by combining it with their HUM and O2 atmospheric sensors. Would replacing the query() call with the write() call followed by time.sleep() and read() change this? Here are the two replies from Jordan @ Atlas Scientific: 1. You have a timing issue. UTF-8 can not handle the number 255. 255 is an error message. Page 42 of the data sheet: https://atlas-scientific.com/files/EZO_CO2_Datasheet.pdf "no data to send" 2. Yes, your delay is long enough. This means you have some type of race condition going on. The definition of race condtion can get quite complex. Besicaly whats going on is (and this is my guess, I do not know for sure). You are asking for a reading, waiting X time, consuming the reading. When you are waiting X time. The reading comes in and all is good. But, something, IDK what, is causing that processing delay to start counting down early then it should. (the code is still moving forward with other things) Its a small difference. But compounds each time. After ~12 readings its now going to fast. What I would do it use the pi as its own diagnostic tool. When you send the R command, start a timer. Have it print the amount of time that has passed from when you sent the R command to when you get a reading. Is is going down by a little bit each time? If so, you know whats up. If not, keep digging. Somthing is messing up your delay clock. New York State Master Science and Engineering Teacher Half Hollow Hills CSD Website: https://www.reginiusscience.com Linktree: https://linktr.ee/Reginius214

…

________________________________ From: Tim Boring ***@***.***> Sent: Tuesday, May 4, 2021 10:20 PM To: timboring/atlas_i2c Cc: Christopher Regini; Author Subject: Re: [timboring/atlas_i2c] Race Condition with CO2 Sensor (#6) CAUTION: This email originated outside of Half Hollow Hills CSD. Please do not click links or attachments unless you recognize the sender and know the content is safe. When you say, "Atlas suggests" are you talking about Atlas Scientific? If so, would you mind copy/pasting the conversation you had with them into this issue? How frequently does this happen? After the six readings, does it work for another X readings, then error again? What happens if you increase the processing_delay? Does it result in the same behavior? I vaguely recall seeing a similar issue with a different sensor, and I had to find the right processing_delay value by trial-and-error, unfortunately. That said, 1200 seems reasonable given the sensor docs suggest 300. Is there a reason you're decoding the bytes? What happens if you do this: carbon = round(float(carbon.data), 1) I'm using this library in a project to read from temp, pH and dissolved oxygen sensors, and this is how I'm handling the data being returned from the query() call. But, maybe the CO2 sensor behaves differently from these sensors (I haven't used a CO2 sensor). 🤔 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHUE7SBR6M7OWNGHJNYFWDLTMCTO7ANCNFSM44DK2CHA>. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe.

[timboring]

I have been decoding the bytes so that I can pass the string to our Adafruit IO dashboard without the leading 'b'. If there is a better way to do that I'd be happy to apply it.

Hmmm, this is interesting. So are you saying that the value returned from the call to the Query() method contains the leading b? That shouldn't be happening, so if it is then there is a bug in the library.

To verify, can you modify your program to print the value of carbon.data and paste it here? In particular, it'd be good to see what that value is when you're hitting the decode error.

I ordered a CO2 sensor from Atlas Scientific last night. Should be here by the weekend, at which point I'll wire it up and do some more debugging/testing on my end.

Also, thanks for posting the response from Atlas Scientific.

[cregini]

Hey Tim, Here is what I have from the output from carbon.data: Thoughts? Chris[cid:a4e27d50-3646-44c0-aebf-938386e19d41] [cid:c66906c0-f199-4056-871b-f2d350651d2d] New York State Master Science and Engineering Teacher Half Hollow Hills CSD Website: https://www.reginiusscience.com Linktree: https://linktr.ee/Reginius214

…

________________________________ From: Tim Boring ***@***.***> Sent: Wednesday, May 5, 2021 8:37 PM To: timboring/atlas_i2c Cc: Christopher Regini; Author Subject: Re: [timboring/atlas_i2c] Race Condition with CO2 Sensor (#6) CAUTION: This email originated outside of Half Hollow Hills CSD. Please do not click links or attachments unless you recognize the sender and know the content is safe. I have been decoding the bytes so that I can pass the string to our Adafruit IO dashboard without the leading 'b'. If there is a better way to do that I'd be happy to apply it. Hmmm, this is interesting. So are you saying that the value returned from the call to the Query() method contains the leading b? That shouldn't be happening, so if it is then there is a bug in the library. To verify, can you modify your program to print the value of carbon.data and paste it here? In particular, it'd be good to see what that value is when you're hitting the decode error. I ordered a CO2 sensor from Atlas Scientific last night. Should be here by the weekend, at which point I'll wire it up and do some more debugging/testing on my end. Also, thanks for posting the response from Atlas Scientific. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHUE7SDJ5DTQOIDMYTDD5DDTMHQDRANCNFSM44DK2CHA>. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe.

[timboring]

So looks like the CO2 sensor just shipped from Atlas Scientific today. That's the slowest turnaround I've seen from them; must be due to Covid. At any rate, once it arrives, I'll get it wired up and see if I can reproduce the same behavior you're seeing. I'll keep you posted.

[cregini]

Sounds good. FYI, I've had no issue with the decoding of the O2, HUM, TEMP, pH, EC, ORP, or DO sensors. CO2 seems to be only one misbehaving at the moment. I am curious to hear your feedback on the leading 'b' in the output as well. You had indicated this may be a bug in the query() call. New York State Master Science and Engineering Teacher Half Hollow Hills CSD Website: https://www.reginiusscience.com Linktree: https://linktr.ee/Reginius214

…

________________________________ From: Tim Boring ***@***.***> Sent: Tuesday, May 11, 2021 4:27:36 PM To: timboring/atlas_i2c Cc: Christopher Regini; Author Subject: Re: [timboring/atlas_i2c] Race Condition with CO2 Sensor (#6) CAUTION: This email originated outside of Half Hollow Hills CSD. Please do not click links or attachments unless you recognize the sender and know the content is safe. So looks like the CO2 sensor just shipped from Atlas Scientific today. That's the slowest turnaround I've seen from them; must be due to Covid. At any rate, once it arrives, I'll get it wired up and see if I can reproduce the same behavior your seeing. I'll keep you posted. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHUE7SEPFPQ3DGWHI6KF4ZLTNGHLRANCNFSM44DK2CHA>. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe.

[timboring]

Ok, the CO2 sensor arrived and I've wired it up. One thing I noticed is that it operated in UART mode by default. The library only supports I2C. Can you confirm which mode your sensor is using?

[cregini]

Def using i2c New York State Master Teacher Science and Engineering Half Hollow Hills CSD Social sites: https://linktr.ee/Reginius214 Web: www.reginiusscience.com

…

________________________________ From: Tim Boring ***@***.***> Sent: Sunday, May 23, 2021 12:23:46 PM To: timboring/atlas_i2c ***@***.***> Cc: reginius214 ***@***.***>; Author ***@***.***> Subject: Re: [timboring/atlas_i2c] Race Condition with CO2 Sensor (#6) CAUTION: This email originated outside of Half Hollow Hills CSD. Please do not click links or attachments unless you recognize the sender and know the content is safe. Ok, the CO2 sensor arrived and I've wired it up. One thing I noticed is that it operated in UART mode by default. The library only supports I2C. Can you confirm which mode your sensor is using? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHUE7SB7GAV4KOYOLJLOEDLTPETZFANCNFSM44DK2CHA>. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe.

[timboring]

Def using i2c

Cool.

So I'm seeing some readings coming back from the sensor that I'm not sure how to interpret. I've sent an inquiry to Atlas Scientific to see if can explain what's happening.

Here's what a "good" raw reading from the sensor looks like:

b'\x01903\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

The first byte of that reading, is 1, which means success, and 903 is CO2 reading. The 00 represents the null terminator.

Then, there are "bad" raw readings that look like:

b'\x01\x1c\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff

The first byte of this reading is also 1, which is supposed to indicate success. Then it's followed 1c, which is decimal 28. Finally, instead of being null-terminated, it's terminated by ff, which is decimal 255. According to the data sheet, 255 is the status code for no data to send, but I'm not sure if that's really what it means in this case given that it's not in the expected position for the status code.

Even in the cases where the sensor seems to be returning bad data, it's intermittent, as you can see here:

>>> for i in range(10):
...   result = a.query("R", processing_delay=1200)
...   print(f"Result status: {result.status_code}")
...   try:
...     print(f"Result data: {int(result.data)}")
...   except ValueError:
...     print(f"Bad data: {result.data}")
... 
Result status: 1
Bad data: b'8\x1c\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
Result status: 1
Result data: 894
Result status: 1
Result data: 894
Result status: 1
Result data: 894
Result status: 1
Result data: 894
Result status: 1
Result data: 894
Result status: 1
Result data: 894
Result status: 1
Result data: 894
Result status: 1
Result data: 894
Result status: 1
Bad data: b'\x1c\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'

For the time being, I'd recommend two things:

1. Read the data in a try/except block, where the except is limited to a ValueError (for example, how it's done in the last code block above). Inside the except block, handle the situation however makes sense for your application (maybe just log it?).

2. Don't use decode on the data. For "good" data, the query() method will convert the data to a string. So you just call float() on it.

Once I hear back from Atlas Scientific I'll update this thread with next steps.

[cregini]

Tim, All makes sense to me. Your try/except recommendation is the same Jordan from Atlas gave to me. While it now keeps my program from crashing there is still a part of my brain that wants/needs to know how to avoid the error. Haven’t encountered it with any other sensor from their suite...and I have them all. For the application I’m using with my students, the sensor reading must be sent as a string to an Adafruit IoT dashboard. From the digging around I’ve done, the decode was the most direct way to ditch the leading ‘b’ and send the reading for visualization in a widget. Is this incorrect? I really appreciate how thoroughly you’ve dug through this. I’m developing a research lab for high school students and you’re library is at the heart of our work. Thanks a ton. Chris New York State Master Teacher Science and Engineering Half Hollow Hills CSD Social sites: https://linktr.ee/Reginius214 Web: www.reginiusscience.com

________________________________ From: Tim Boring ***@***.***> Sent: Sunday, May 23, 2021 4:39:05 PM To: timboring/atlas_i2c ***@***.***> Cc: reginius214 ***@***.***>; Author ***@***.***> Subject: Re: [timboring/atlas_i2c] Race Condition with CO2 Sensor (#6) CAUTION: This email originated outside of Half Hollow Hills CSD. Please do not click links or attachments unless you recognize the sender and know the content is safe. Def using i2c Cool. So I'm seeing some readings coming back from the sensor that I'm not sure how to interpret. I've sent an inquiry to Atlas Scientific to see if can explain what's happening. Here's what a "good" raw reading from the sensor looks like: b'\x01903\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' The first byte of that reading, is 1, which means success, and 903 is CO2 reading. The 00 represents the null terminator. Then, there are "bad" raw readings that look like: b'\x01\x1c\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff The first byte of this reading is also 1, which is supposed to indicate success. Then it's followed 1c, which is decimal 28. Finally, instead of being null-terminated, it's terminated by ff, which is decimal 255. According to the data sheet, 255 is the status code for no data to send, but I'm not sure if that's really what it means in this case given that it's not in the expected position for the status code. Even in the cases where the sensor seems to be returning bad data, it's intermittent, as you can see here:

>> for i in range(10):

... result = a.query("R", processing_delay=1200) ... print(f"Result status: {result.status_code}") ... try: ... print(f"Result data: {int(result.data)}") ... except ValueError: ... print(f"Bad data: {result.data}") ... Result status: 1 Bad data: b'8\x1c\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' Result status: 1 Result data: 894 Result status: 1 Result data: 894 Result status: 1 Result data: 894 Result status: 1 Result data: 894 Result status: 1 Result data: 894 Result status: 1 Result data: 894 Result status: 1 Result data: 894 Result status: 1 Result data: 894 Result status: 1 Bad data: b'\x1c\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' For the time being, I'd recommend two things: 1. Read the data in a try/except block, where the except is limited to a ValueError (for example, how it's done in the last code block above). Inside the except block, handle the situation however makes sense for your application (maybe just log it?). 2. Don't use decode on the data. For "good" data, the query() method will convert the data to a string. So you just call float() on it. Once I hear back from Atlas Scientific I'll update this thread with next steps. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHUE7SDMHOVJT4CXJEFWMITTPFRWTANCNFSM44DK2CHA>. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe. CAUTION: Do not click links or open attachments unless you recognize the sender and know the content/link is safe.

[timboring]

Tim, All makes sense to me. Your try/except recommendation is the same Jordan from Atlas gave to me. While it now keeps my program from crashing there is still a part of my brain that wants/needs to know how to avoid the error. Haven’t encountered it with any other sensor from their suite...and I have them all.

In the current situation, there isn't anything you can do to totally avoid the error--i.e., keep it from happening. You are not doing anything "wrong" to cause the problem. The issue is the data coming back from the CO2 sensor is "weird", so once Atlas Scientific responds to my email we should have a clearer path forward. The most likely path forward is that if Atlas Scientific is able to provide more clarity about this "bad" data, then I'll refactor the relevant bit(s) of the library to handle these situations.

That said, errors happen. All the time. The challenge for the programmer is how to deal with them. Just let the program crash? Catch the error and just log it? Catch the error and retry the failed operation? There's not a one-size-fits-all answer, and how to handle errors is largely dependent on the needs of the programmer and users.

For the time being, your best course of action for working around the situation is to use a try/except block. Not knowing your system, if it were me, I'd just log a message then wait for the next reading.

For the application I’m using with my students, the sensor reading must be sent as a string to an Adafruit IoT dashboard. From the digging around I’ve done, the decode was the most direct way to ditch the leading ‘b’ and send the reading for visualization in a widget. Is this incorrect?

Ah, ok, From the initial screenshot you provided, you were decoding bytes into a string, then converting it to a float, then rounding it. If all you need is a string, then just call decode() on it. In the case of the CO2 sensor, the data returned from the sensor is an integer, so calling float() and round() on it isn't going to have any effect. If, however, you've got a function that is handling these values for all of the sensors, some of which do indeed return floats, then what you're currently doing is fine. That is, as long as you're not trying to call decode() on bad data.

really appreciate how thoroughly you’ve dug through this. I’m developing a research lab for high school students and you’re library is at the heart of our work.

Wow, that's awesome! It makes me happy knowing that you find the library useful. 😄