Framegrabber timeout error

[robinbansal08]

I am running my application over the O3D303 (after cross compilation), and getting such framegrabber timeout errors sometimes (occurs out of sudden at a time when my application was running fine and was able to grab previous frames successfully). I tried frame grabber reset but that didn't help. I have to do camera reboot to get it work again as my application is not able to get the images and keep getting frame grabber timeout errors. My camera is set to software triggered and not free run.

Can you please provide any suggestions related to this issue @graugans

E1029 12:30:41.621506  1317 frame_grabber.cpp:343] Bad ticket: 1001L000000007
1001!
W1029 12:30:41.621618  1317 frame_grabber.cpp:348] Are you software triggering in free run mode?
==> Trigger difference: 2565ms
W1029 12:30:42.615631  1311 frame_grabber.cpp:193] Timeout waiting for image buffer from camera
Timeout waiting for camera!
Restarting FrameGrabber...
I1029 12:30:42.629215  1311 frame_grabber.cpp:72] Camera connection info: ip=192.168.0.19, port=50010
W1029 12:30:42.630647  1317 frame_grabber.cpp:429] Exception: Thread interrupted
I1029 12:30:42.630798  1317 frame_grabber.cpp:432] Framegrabber thread done.
W1029 12:30:43.634541  1311 frame_grabber.cpp:193] Timeout waiting for image buffer from camera
Timeout waiting for camera!
W1029 12:30:43.637190  1319 frame_grabber.cpp:429] Exception: Thread interrupted
I1029 12:30:43.637830  1319 frame_grabber.cpp:432] Framegrabber thread done.

o3d3xx::Camera::Ptr cam_;
o3d3xx::ImageBuffer::Ptr img_;
o3d3xx::FrameGrabber::Ptr fg_;

void init_cam()
{
    cam_ = std::make_shared<o3d3xx::Camera>(camera_IP);
    img_ = std::make_shared<o3d3xx::ImageBuffer>();
    fg_ = std::make_shared<o3d3xx::FrameGrabber>(cam_, o3d3xx::IMG_AMP|o3d3xx::IMG_RDIS);
    cout<<"Cam init done"<<endl;
}

int getImageFromCamera(Mat& image, pcl::PointCloud<pcl::PointXYZI>::Ptr& cloud_xyzi)
{
    fg_->SWTrigger();

    if (! fg_->WaitForFrame(img_.get(), 1000))
    {
        cout << RED<<"Timeout waiting for camera!" << RESET <<std::endl;
        std::cout << "Restarting FrameGrabber..." << std::endl;
        //exit (-1);
        fg_.reset(new o3d3xx::FrameGrabber(cam_, o3d3xx::IMG_AMP|o3d3xx::IMG_RDIS));
        if (! fg_->WaitForFrame(img_.get(), 1000))
        {
            cout << RED<<"Timeout waiting for camera!" << RESET <<std::endl;
            exit (-1);
        }
        return 0;
    }

    pcl::copyPointCloud(*(img_->Cloud()), *cloud_xyzi);
    image = img_->DepthImage();

    return 1;
}

{
    "o3d3xx": {
        "libo3d3xx": "705",
        "Date": "Thu Jun 17 13:02:13 2021",
        "HWInfo": {
            "Connector": "#!02_A300_C40_04246016_008023176",
            "Diagnose": "#!02_D322_C34_04250651_008026824",
            "MiraSerial": "8470-81f7-0e63-0720",
            "MACAddress": "00:02:01:41:3B:B5",
            "Frontend": "#!02_F342_C34_19_00105_008023607",
            "Mainboard": "#!02_M381_C41_03901679_008023690",
            "Illumination": "#!02_I300_001_04201465_008001175"
        },
        "SWVersion": {
            "IFM_Software": "1.71.7444",
            "Main_Application": "O2I_1.20.7428-0-g91c60a74be",
            "IFM_Recovery": "unversioned",
            "Algorithm_Version": "3.0.5",
            "Calibration_Device": "00:02:01:41:3b:b5",
            "Calibration_Version": "1.0.2",
            "Linux": "Linux version 3.14.34-rt31-yocto-standard-00048-gc8112a91aeb2-d",
            "Diagnostic_Controller": "O2I_1.20.7428-0-g91c60a74be"
        },
        "Device": {
            "Name": "New sensor",
            "Description": "",
            "ActiveApplication": "1",
            "PcicTcpPort": "50010",
            "PcicProtocolVersion": "3",
            "IOLogicType": "1",
            "IODebouncing": "true",
            "IOExternApplicationSwitch": "0",
            "SessionTimeout": "30",
            "ServiceReportPassedBuffer": "15",
            "ServiceReportFailedBuffer": "15",
            "ExtrinsicCalibTransX": "0",
            "ExtrinsicCalibTransY": "0",
            "ExtrinsicCalibTransZ": "0",
            "ExtrinsicCalibRotX": "0",
            "ExtrinsicCalibRotY": "0",
            "ExtrinsicCalibRotZ": "0",
            "EvaluationFinishedMinHoldTime": "10",
            "SaveRestoreStatsOnApplSwitch": "true",
            "IPAddressConfig": "0",
            "PasswordActivated": "false",
            "OperatingMode": "1",
            "DeviceType": "1:2",
            "ArticleNumber": "O3D303",
            "ArticleStatus": "AD",
            "UpTime": "25.1308333333333",
            "ImageTimestampReference": "1603975097",
            "TemperatureFront1": "3276.6999999999998",
            "TemperatureFront2": "3276.6999999999998",
            "TemperatureIMX6": "32.812000274658203",
            "TemperatureIllu": "39.200000000000003",
            "PNIODeviceName": "",
            "EthernetFieldBus": "1",
            "EthernetFieldBusEndianness": "0",
            "EnableAcquisitionFinishedPCIC": "false"
        },
        "Net": {
            "MACAddress": "",
            "NetworkSpeed": "0",
            "StaticIPv4Address": "192.168.0.19",
            "StaticIPv4SubNetMask": "255.255.255.0",
            "StaticIPv4Gateway": "192.168.0.201",
            "UseDHCP": "false"
        },
        "Apps": [
            {
                "Name": "Box Detection Application",
                "Description": "",
                "TriggerMode": "2",
                "PcicTcpResultSchema": "{ \"layouter\": \"flexible\", \"format\": { \"dataencoding\": \"ascii\" }, \"elements\": [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"blob\", \"id\": \"normalized_amplitude_image\" }, { \"type\": \"blob\", \"id\": \"distance_image\" }, { \"type\": \"blob\", \"id\": \"x_image\" }, { \"type\": \"blob\", \"id\": \"y_image\" }, { \"type\": \"blob\", \"id\": \"z_image\" }, { \"type\": \"blob\", \"id\": \"confidence_image\" }, { \"type\": \"blob\", \"id\": \"diagnostic_data\" }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "PcicEipResultSchema": "{ \"layouter\": \"flexible\", \"format\": { \"dataencoding\": \"binary\", \"order\": \"big\" }, \"elements\" : [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"records\", \"id\": \"models\", \"elements\": [ { \"type\": \"int16\", \"id\": \"boxFound\" }, { \"type\": \"int16\", \"id\": \"width\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"height\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"length\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"xMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"zMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yawAngle\" }, { \"type\": \"int16\", \"id\": \"qualityLength\" }, { \"type\": \"int16\", \"id\": \"qualityWidth\" }, { \"type\": \"int16\", \"id\": \"qualityHeight\" } ] }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "PcicPnioResultSchema": "{\"layouter\" : \"flexible\", \"format\": { \"dataencoding\": \"binary\", \"order\": \"big\" }, \"elements\" : [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"records\", \"id\": \"models\", \"elements\": [ { \"type\": \"int16\", \"id\": \"boxFound\" }, { \"type\": \"int16\", \"id\": \"width\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"height\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"length\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"xMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"zMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yawAngle\" }, { \"type\": \"int16\", \"id\": \"qualityLength\" }, { \"type\": \"int16\", \"id\": \"qualityWidth\" }, { \"type\": \"int16\", \"id\": \"qualityHeight\" } ] }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "LogicGraph": "{\"IOMap\": {\"OUT1\": \"RFT\",\"OUT2\": \"AQUFIN\"},\"blocks\": {\"B00001\": {\"pos\": {\"x\": 200,\"y\": 200},\"properties\": {},\"type\": \"PIN_EVENT_IMAGE_ACQUISITION_FINISHED\"},\"B00002\": {\"pos\": {\"x\": 200,\"y\": 75},\"properties\": {},\"type\": \"PIN_EVENT_READY_FOR_TRIGGER\"},\"B00003\": {\"pos\": {\"x\": 600,\"y\": 75},\"properties\": {\"pulse_duration\": 0},\"type\": \"DIGITAL_OUT1\"},\"B00005\": {\"pos\": {\"x\": 600,\"y\": 200},\"properties\": {\"pulse_duration\": 0},\"type\": \"DIGITAL_OUT2\"}},\"connectors\": {\"C00000\": {\"dst\": \"B00003\",\"dstEP\": 0,\"src\": \"B00002\",\"srcEP\": 0},\"C00001\": {\"dst\": \"B00005\",\"dstEP\": 0,\"src\": \"B00001\",\"srcEP\": 0}}}",
                "Type": "Camera",
                "Index": "1",
                "Id": "476707713",
                "Imager": {
                    "AutoExposureReferencePointX": "88",
                    "AutoExposureReferencePointY": "66",
                    "AutoExposureReferenceType": "0",
                    "AutoExposureMaxExposureTime": "10000",
                    "AutoExposureReferenceROI": "{\"ROIs\":[{\"id\":0,\"group\":0,\"type\":\"Rect\",\"width\":130,\"height\":100,\"angle\":0,\"center_x\":88,\"center_y\":66}]}",
                    "Channel": "0",
                    "ClippingBottom": "263",
                    "ClippingLeft": "0",
                    "ClippingRight": "351",
                    "ClippingTop": "0",
                    "ContinuousAutoExposure": "false",
                    "EnableAmplitudeCorrection": "true",
                    "EnableFastFrequency": "false",
                    "EnableFilterAmplitudeImage": "false",
                    "EnableFilterDistanceImage": "true",
                    "EnableRectificationAmplitudeImage": "false",
                    "EnableRectificationDistanceImage": "false",
                    "UseSimpleBinning": "false",
                    "ExposureTime": "10000",
                    "ExposureTimeList": "2500;10000",
                    "ExposureTimeRatio": "4",
                    "FrameRate": "5",
                    "MinimumAmplitude": "2",
                    "Resolution": "1",
                    "ClippingCuboid": "{\"XMin\": -3.402823e+38, \"XMax\": 3.402823e+38, \"YMin\": -3.402823e+38, \"YMax\": 3.402823e+38, \"ZMin\": -3.402823e+38, \"ZMax\": 3.402823e+38}",
                    "SpatialFilterType": "1",
                    "SymmetryThreshold": "0",
                    "TemporalFilterType": "0",
                    "ThreeFreqMax2FLineDistPercentage": "80",
                    "ThreeFreqMax3FLineDistPercentage": "80",
                    "TwoFreqMaxLineDistPercentage": "80",
                    "Type": "upto30m_moderate",
                    "MaxAllowedLEDFrameRate": "4.9000000000000004",
                    "SpatialFilter": {
                        "Type": "1",
                        "TypeStr_": "Median Filter",
                        "MaskSize": "1",
                        "MaskSizeStr_": "5x5"
                    },
                    "TemporalFilter": {
                        "Type": "0",
                        "TypeStr_": "Off"
                    }
                }
            }
        ]
    }
}

[robinbansal08]

• gelog > camera_log.txt

T: 13.0102 P: 662 C: WARN S: iom M: Program starting ... 
T: 13.0102 P: 657 C: WARN S: qt_dbus_web_bridge M: Starting ... 
T: 13.2157 P: 662 C: WARN S: iom M: DIAGC_DBUS_EVENT_BOOT_TIMEOUT 
T: 13.2194 P: 662 C: INFO S: iom M: DIAGC_DBUS_EVENT_DIAGC_BOOTED 
T: 13.2263 P: 662 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 13.2267 P: 662 C: WARN S: diagCtrl M: drv_cfg_hw_late_init[231]: calib_dac failed 
T: 13.2270 P: 662 C: INFO S: diagCtrl M: sysboot_hw_evt[139]: successfully init HW : ### 
T: 13.2272 P: 662 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 13.2275 P: 662 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 13.2277 P: 662 C: WARN S: diagCtrl M: Option Byte Loader (OBL) reset  
T: 13.2280 P: 662 C: WARN S: diagCtrl M: Pin reset  
T: 13.2283 P: 662 C: WARN S: diagCtrl M: V1.8 power domain reset  
T: 13.2285 P: 662 C: WARN S: diagCtrl M: POR/PDR reset  
T: 13.2288 P: 662 C: WARN S: diagCtrl M: Software reset  
T: 13.8485 P: 705 C: WARN S: suprge M: suprge: supervision activity starting, Bello the watchdog, is unfused ... 
T: 14.3427 P: 691 C: WARN S: gecco M: Starting ... 
T: 14.3826 P: 691 C: WARN S: gecco M: Build version        :  O2I_1.20.7428-0-g91c60a74be  
T: 14.3833 P: 691 C: WARN S: gecco M: ifm SW version       :  1.71.7444  
T: 14.3838 P: 691 C: WARN S: gecco M: ifm Recovery version :  unversioned  
T: 15.1873 P: 691 C: ERROR S: gecco M: static void ifm::JsonSchemaPropertyReader::createDefaultFromSchema(const QString&, const QString&): initializing key default:time_read_only failed (god error 1010) 
T: 15.3518 P: 705 C: WARN S: suprge M: Registering services for the O3D3XX family of devices 
T: 15.4434 P: 691 C: ERROR S: gecco M: Couldn't open load file. 
T: 15.4468 P: 677 C: INFO S: GECOM M: DBUS connection successful 
T: 15.6388 P: 691 C: WARN S: gecco M: ifm::FieldbusDevice::FieldbusDevice(ifm::SmartCamera&): fieldbus type 1 is not allowed on this device 
T: 15.7513 P: 686 C: WARN S: imager M: Starting ... 
T: 15.7844 P: 691 C: WARN S: gecco M: QDBusConnection: name 'com.ifm.suprge' had owner '' but we thought it was ':1.6' 
T: 15.8116 P: 662 C: INFO S: diagCtrl M: Stop VLED monitoring # 
T: 15.8892 P: 686 C: ERROR S: imager M: void gelfi::SingleMotorError::print() const :  No motor  ( 28 )  
T: 15.8908 P: 686 C: WARN S: imager M: void gelfi::StepperMotor::initSystem(bool, const gelfi::StepperMotor*): No motor is available for this article 
T: 19.2622 P: 662 C: INFO S: diagCtrl M: set_illu_power 1 # 
T: 19.5009 P: 662 C: INFO S: diagCtrl M: Start VLED monitoring # 
T: 19.5061 P: 691 C: WARN S: gecco M: Imager is available again 
T: 19.5087 P: 686 C: WARN S: imager M: LRAE is available again 
T: 20.8215 P: 691 C: WARN S: gecco M: Finished imager configuration 
T: 20.8395 P: 662 C: WARN S: diagCtrl M: First Trigger 
T: 1701.0793 P: 691 C: INFO S: gecco M: This function is not supported in current layout version!# 
T: 1704.1475 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.1489 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.1493 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0 
T: 1704.1499 P: 686 C: WARN S: imager M: void ifm::FrameReevaluator::reevaluateIfPossible(std::function<ifm::ErrorStatus(const QByteArray&)>, int): Cannot re-process the snapshot: Invalid chunk type expected 600 received: 0 
T: 1704.6674 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.6687 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.6691 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0 
T: 1704.6697 P: 686 C: WARN S: imager M: void ifm::FrameReevaluator::reevaluateIfPossible(std::function<ifm::ErrorStatus(const QByteArray&)>, int): Cannot re-process the snapshot: Invalid chunk type expected 600 received: 0 
T: 1704.6719 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.6730 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.6735 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0 
T: 1704.6741 P: 686 C: WARN S: imager M: void ifm::FrameReevaluator::reevaluateIfPossible(std::function<ifm::ErrorStatus(const QByteArray&)>, int): Cannot re-process the snapshot: Invalid chunk type expected 600 received: 0 
T: 1704.6763 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.6776 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.6780 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0 
T: 1704.6786 P: 686 C: WARN S: imager M: void ifm::FrameReevaluator::reevaluateIfPossible(std::function<ifm::ErrorStatus(const QByteArray&)>, int): Cannot re-process the snapshot: Invalid chunk type expected 600 received: 0 
T: 1704.6808 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.6857 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.6864 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0 
T: 1704.6872 P: 686 C: WARN S: imager M: void ifm::FrameReevaluator::reevaluateIfPossible(std::function<ifm::ErrorStatus(const QByteArray&)>, int): Cannot re-process the snapshot: Invalid chunk type expected 600 received: 0 
T: 1704.6908 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.6928 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.6933 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0 
T: 1704.6941 P: 686 C: WARN S: imager M: void ifm::FrameReevaluator::reevaluateIfPossible(std::function<ifm::ErrorStatus(const QByteArray&)>, int): Cannot re-process the snapshot: Invalid chunk type expected 600 received: 0 
T: 1704.6974 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.7002 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.7007 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0 
T: 1704.7038 P: 686 C: WARN S: imager M: void ifm::FrameReevaluator::reevaluateIfPossible(std::function<ifm::ErrorStatus(const QByteArray&)>, int): Cannot re-process the snapshot: Invalid chunk type expected 600 received: 0 
T: 106779.4969 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 106779.4974 P: 662 C: INFO S: diagCtrl M: rx: 246707, tx: 246545, err: 1 
T: 106779.4977 P: 662 C: WARN S: diagCtrl M: host_rcv: invalid pdu (unexpected pl_len) 
T: 106779.4980 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_HOST_SYNC 
T: 106779.4983 P: 662 C: INFO S: diagCtrl M: rx: 246707, tx: 246545, err: 3 
T: 106779.4986 P: 662 C: WARN S: diagCtrl M: host_rcv: invalid pdu (hdr_len) 
T: 706246.3254 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 706246.3257 P: 662 C: INFO S: diagCtrl M: rx: 5148756, tx: 5148594, err: 5 
T: 706246.3261 P: 662 C: WARN S: diagCtrl M: host_write: spi_write(hdr) txcnt: -1 failed 
T: 706246.3264 P: 662 C: WARN S: diagCtrl M: host_do_receive: host_write(write_pdu) failed! 
T: 707785.3324 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 707785.3328 P: 662 C: INFO S: diagCtrl M: rx: 5154500, tx: 5154338, err: 6 
T: 707785.3331 P: 662 C: WARN S: diagCtrl M: host_write: spi_write(hdr) txcnt: -1 failed 
T: 707785.3334 P: 662 C: WARN S: diagCtrl M: host_do_receive: host_write(write_pdu) failed! 
T: 707785.3337 P: 662 C: INFO S: diagCtrl M: CRC pdu (hdr) 
T: 802741.3256 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 802741.3259 P: 662 C: INFO S: diagCtrl M: rx: 5327729, tx: 5327566, err: 7 
T: 802741.3262 P: 662 C: WARN S: diagCtrl M: host_write: spi_write(hdr) txcnt: -1 failed 
T: 802741.3265 P: 662 C: WARN S: diagCtrl M: host_do_receive: host_write(write_pdu) failed! 
T: 844108.3455 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 844108.3459 P: 662 C: INFO S: diagCtrl M: rx: 5464182, tx: 5464019, err: 8 
T: 844108.3462 P: 662 C: WARN S: diagCtrl M: host_write: spi_write(hdr) txcnt: -1 failed 
T: 844108.3465 P: 662 C: WARN S: diagCtrl M: host_do_receive: host_write(write_pdu) failed! 
T: 848076.3323 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 848076.3326 P: 662 C: INFO S: diagCtrl M: rx: 5479019, tx: 5478856, err: 9 
T: 848076.3329 P: 662 C: WARN S: diagCtrl M: host_write: spi_write(hdr) txcnt: -1 failed 
T: 848076.3332 P: 662 C: WARN S: diagCtrl M: host_do_receive: host_write(write_pdu) failed! 
T: 848076.3335 P: 662 C: INFO S: diagCtrl M: CRC pdu (hdr) 
T: 978911.5721 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 978911.5725 P: 662 C: INFO S: diagCtrl M: rx: 5688154, tx: 5687991, err: 10 
T: 978911.5728 P: 662 C: WARN S: diagCtrl M: host_rcv: invalid pdu (unexpected pl_len) 
T: 978911.5731 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_HOST_SYNC 
T: 978911.5734 P: 662 C: INFO S: diagCtrl M: rx: 5688154, tx: 5687991, err: 12 
T: 978911.5736 P: 662 C: WARN S: diagCtrl M: host_rcv: invalid pdu (hdr_len) 
T: 978913.8578 P: 686 C: ERROR S: imager M: ifm::ErrorStatus::ErrorCode ifm::TriggerControl::trigger(): IO manager trigger returned error code 200000, current state is 1 
T: 978913.8599 P: 662 C: WARN S: iom M: DIAGC_DBUS_EVENT_TRG_OVERRUN 
T: 978913.8617 P: 662 C: INFO S: diagCtrl M: main_task[81]: evt_call_funcs returned 6, evt ID: 15 

[graugans]

@robinbansal08, have you tried to upload an image to the camera? This is what I do read from:

T: 1704.1475 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.1489 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.1493 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0 

Those SPI errors we do typically see when there is electromagnetic disturbance.

T: 106779.4969 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 106779.4974 P: 662 C: INFO S: diagCtrl M: rx: 246707, tx: 246545, err: 1 
T: 106779.4977 P: 662 C: WARN S: diagCtrl M: host_rcv: invalid pdu (unexpected pl_len) 
T: 106779.4980 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_HOST_SYNC 
T: 106779.4983 P: 662 C: INFO S: diagCtrl M: rx: 246707, tx: 246545, err: 3 
T: 106779.4986 P: 662 C: WARN S: diagCtrl M: host_rcv: invalid pdu (hdr_len) 
T: 706246.3254 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 

From the log above it seems like the camera detected an situation where it is not able to perform any trigger. This can be caused by over temperature or an issue with the power-supply. Is the log from a situation where you not able to recover the framegrabber?

[robinbansal08]

• I did software updates for many cameras yesterday but I do not remember if I did for this particular camera
• The camera is being triggered at every 2.5 seconds, I am not sure if it can raise the temperature upto that limit
• We have AC/DC power converters with output of fixed 24V DC supply
• Yes the log is from the situation where it was not able to recover the framegrabber

[robinbansal08]

Here is the log from another camera, in which I got same framegrabber timeout error (as explained before) while my application was running fine for several hours and was able to capture previous frames successfully but failed later suddenly with framegrabber issue.

loadConfig: Redis connection problem for connecting via /tmp/redis-cache.sock
loadConfig: Redis connection problem for connecting via /tmp/redis-cache.sock
T: 11.5936 P: 423 C: WARN S: iom M: Program starting ... 
T: 11.5971 P: 419 C: WARN S: qt_dbus_web_bridge M: Starting ... 
T: 11.7942 P: 423 C: INFO S: iom M: DIAGC_DBUS_EVENT_DIAGC_BOOTED 
T: 11.8023 P: 423 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 11.8027 P: 423 C: WARN S: diagCtrl M: drv_cfg_hw_late_init[231]: calib_dac failed 
T: 11.8029 P: 423 C: INFO S: diagCtrl M: sysboot_hw_evt[139]: successfully init HW : ### 
T: 11.8032 P: 423 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 11.8034 P: 423 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 11.8037 P: 423 C: WARN S: diagCtrl M: Pin reset  
T: 12.4094 P: 460 C: WARN S: suprge M: suprge: supervision activity starting, Bello the watchdog, is unfused ... 
T: 12.7586 P: 452 C: WARN S: gecco M: Starting ... 
T: 12.7664 P: 452 C: WARN S: gecco M: Build version        :  O2I_1.20.7428-0-g91c60a74be  
T: 12.7670 P: 452 C: WARN S: gecco M: ifm SW version       :  1.71.7444  
T: 12.7675 P: 452 C: WARN S: gecco M: ifm Recovery version :  unversioned  
T: 13.7110 P: 460 C: WARN S: suprge M: Registering services for the O3D3XX family of devices 
T: 13.7702 P: 452 C: ERROR S: gecco M: static void ifm::JsonSchemaPropertyReader::createDefaultFromSchema(const QString&, const QString&): initializing key default:time_read_only failed (god error 1010) 
T: 14.0034 P: 440 C: INFO S: GECOM M: DBUS connection successful 
T: 14.0881 P: 452 C: WARN S: gecco M: ifm::FieldbusDevice::FieldbusDevice(ifm::SmartCamera&): fieldbus type 1 is not allowed on this device 
T: 14.1437 P: 449 C: WARN S: imager M: Starting ... 
T: 14.2983 P: 423 C: INFO S: diagCtrl M: Stop VLED monitoring # 
T: 14.4189 P: 452 C: WARN S: gecco M: QDBusConnection: name 'com.ifm.suprge' had owner '' but we thought it was ':1.7' 
T: 14.4542 P: 449 C: ERROR S: imager M: void gelfi::SingleMotorError::print() const :  No motor  ( 28 )  
T: 14.4583 P: 449 C: WARN S: imager M: void gelfi::StepperMotor::initSystem(bool, const gelfi::StepperMotor*): No motor is available for this article 
T: 17.8466 P: 423 C: INFO S: diagCtrl M: set_illu_power 1 # 
T: 18.0893 P: 423 C: INFO S: diagCtrl M: Start VLED monitoring # 
T: 18.0943 P: 452 C: WARN S: gecco M: Imager is available again 
T: 18.0988 P: 449 C: WARN S: imager M: LRAE is available again 
T: 19.2069 P: 452 C: WARN S: gecco M: Finished imager configuration 
T: 1524.9848 P: 423 C: WARN S: diagCtrl M: First Trigger 
T: 15724.6401 P: 423 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 15724.6405 P: 423 C: INFO S: diagCtrl M: rx: 34133, tx: 34096, err: 1 
T: 15724.6408 P: 423 C: WARN S: diagCtrl M: host_rcv: invalid pdu (unexpected pl_len) 
T: 15726.9223 P: 449 C: ERROR S: imager M: ifm::ErrorStatus::ErrorCode ifm::TriggerControl::trigger(): IO manager trigger returned error code 200000, current state is 1 
T: 15726.9235 P: 423 C: WARN S: iom M: DIAGC_DBUS_EVENT_TRG_OVERRUN 
T: 15726.9251 P: 423 C: INFO S: diagCtrl M: main_task[81]: evt_call_funcs returned 6, evt ID: 15 

[robinbansal08]

• My application requires CPU within 25-40%

• This error is occurring over all the cameras if I run my application, and keep it running for several hours over them.