Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

REQUIRED process [ouster/os_nodelet_mgr-2] has died! #365

Open
retepik opened this issue Sep 10, 2024 · 10 comments
Open

REQUIRED process [ouster/os_nodelet_mgr-2] has died! #365

retepik opened this issue Sep 10, 2024 · 10 comments
Labels
bug Something isn't working

Comments

@retepik
Copy link

retepik commented Sep 10, 2024

Hi,
my configuration is :
Ubuntu 20.04
Ros : Noetic
Ouster OS-1 128 Rev C.
firmware: 2.5.3

after update to latest ouster-ros driver I get error

REQUIRED process [ouster/os_nodelet_mgr-2] has died! process has died [pid 43842, exit code -11, cmd /opt/ros/noetic/lib/nodelet/nodelet manager __name:=os_nodelet_mgr __log:=/home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/ouster-os_nodelet_mgr-2.log]. log file: /home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/ouster-os_nodelet_mgr-2*.log Initiating shutdown!

Strange think is that I get error only on Release build
catkin_make --cmake-args -DCMAKE_BUILD_TYPE=Release
when I use Debug, or RelWithDebInfo build type, all work.
I try it with fresh default installation of ROS and ouster-ros driver, but with same behavior

my command is :
roslaunch --master-logger-level='debug' -v /home/mapper/Programs/ouster/ouster-ros/src/ouster-ros/launch/sensor.launch sensor_hostname:=169.254.192.176 udp_dest:=192.168.113.195 lidar_port:=7502 imu_port:=7503 replay:=false lidar_mode:=512x10 metadata:=/home/mapper/metadata.json viz:=true image:=true udp_profile_lidar:=LEGACY

output :
`... logging to /home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/roslaunch-EOS-PC-01-43815.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

... loading XML file [/opt/ros/noetic/etc/ros/roscore.xml]
... executing command param [rosversion roslaunch]
Added parameter [/rosversion]
... executing command param [rosversion -d]
Added parameter [/rosdistro]
Added core node of type [rosout/rosout] in namespace [/]
... loading XML file [/home/mapper/Programs/ouster/ouster-ros/src/ouster-ros/launch/sensor.launch]
Added node of type [nodelet/nodelet] in namespace [/ouster/]
Added parameter [/ouster/os_node/sensor_hostname]
Added parameter [/ouster/os_node/udp_dest]
Added parameter [/ouster/os_node/lidar_port]
Added parameter [/ouster/os_node/imu_port]
Added parameter [/ouster/os_node/udp_profile_lidar]
Added parameter [/ouster/os_node/lidar_mode]
Added parameter [/ouster/os_node/timestamp_mode]
Added parameter [/ouster/os_node/metadata]
Added parameter [/ouster/os_node/azimuth_window_start]
Added parameter [/ouster/os_node/azimuth_window_end]
Added parameter [/ouster/os_node/persist_config]
Added parameter [/ouster/os_node/attempt_reconnect]
Added parameter [/ouster/os_node/dormant_period_between_reconnects]
Added parameter [/ouster/os_node/max_failed_reconnect_attempts]
Added node of type [nodelet/nodelet] in namespace [/ouster/]
... loading XML file [/home/mapper/Programs/ouster/ouster-ros/src/ouster-ros/launch/common.launch]
Added parameter [/ouster/os_cloud_node/tf_prefix]
Added parameter [/ouster/os_cloud_node/sensor_frame]
Added parameter [/ouster/os_cloud_node/lidar_frame]
Added parameter [/ouster/os_cloud_node/imu_frame]
Added parameter [/ouster/os_cloud_node/point_cloud_frame]
Added parameter [/ouster/os_cloud_node/timestamp_mode]
Added parameter [/ouster/os_cloud_node/dynamic_transforms_broadcast]
Added parameter [/ouster/os_cloud_node/dynamic_transforms_broadcast_rate]
Added parameter [/ouster/os_cloud_node/proc_mask]
Added parameter [/ouster/os_cloud_node/scan_ring]
Added parameter [/ouster/os_cloud_node/ptp_utc_tai_offset]
Added parameter [/ouster/os_cloud_node/point_type]
Added node of type [nodelet/nodelet] in namespace [/ouster/]
Added parameter [/ouster/img_node/proc_mask]
Added node of type [nodelet/nodelet] in namespace [/ouster/]
Added node of type [rviz/rviz] in namespace [/]
... done importing include file [/home/mapper/Programs/ouster/ouster-ros/src/ouster-ros/launch/common.launch]
started roslaunch server http://EOS-PC-01:35957/

SUMMARY

PARAMETERS

  • /ouster/img_node/proc_mask: IMG|PCL|IMU|SCAN
  • /ouster/os_cloud_node/dynamic_transforms_broadcast: False
  • /ouster/os_cloud_node/dynamic_transforms_broadcast_rate: 1.0
  • /ouster/os_cloud_node/imu_frame: os_imu
  • /ouster/os_cloud_node/lidar_frame: os_lidar
  • /ouster/os_cloud_node/point_cloud_frame:
  • /ouster/os_cloud_node/point_type: original
  • /ouster/os_cloud_node/proc_mask: IMG|PCL|IMU|SCAN
  • /ouster/os_cloud_node/ptp_utc_tai_offset: -37.0
  • /ouster/os_cloud_node/scan_ring: 0
  • /ouster/os_cloud_node/sensor_frame: os_sensor
  • /ouster/os_cloud_node/tf_prefix:
  • /ouster/os_cloud_node/timestamp_mode:
  • /ouster/os_node/attempt_reconnect: False
  • /ouster/os_node/azimuth_window_end: 360000
  • /ouster/os_node/azimuth_window_start: 0
  • /ouster/os_node/dormant_period_between_reconnects: 1.0
  • /ouster/os_node/imu_port: 7503
  • /ouster/os_node/lidar_mode: 512x10
  • /ouster/os_node/lidar_port: 7502
  • /ouster/os_node/max_failed_reconnect_attempts: 2147483647
  • /ouster/os_node/metadata: /home/mapper/meta...
  • /ouster/os_node/persist_config: False
  • /ouster/os_node/sensor_hostname: 169.254.192.176
  • /ouster/os_node/timestamp_mode:
  • /ouster/os_node/udp_dest: 192.168.113.195
  • /ouster/os_node/udp_profile_lidar: LEGACY
  • /rosdistro: noetic
  • /rosversion: 1.16.0

NODES
/
rviz (rviz/rviz)
/ouster/
img_node (nodelet/nodelet)
os_cloud_node (nodelet/nodelet)
os_node (nodelet/nodelet)
os_nodelet_mgr (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [43825]
ROS_MASTER_URI=http://localhost:11311

setting run_id to 3eee6cfc-6f8b-11ef-b8ff-4de79e617463
process[rosout-1]: started with pid [43835]
started core service [/rosout]
process[ouster/os_nodelet_mgr-2]: started with pid [43842]
process[ouster/os_node-3]: started with pid [43843]
process[ouster/os_cloud_node-4]: started with pid [43844]
process[ouster/img_node-5]: started with pid [43845]
process[rviz-6]: started with pid [43847]
[ INFO] [1725982932.289950072]: Loading nodelet /ouster/os_node of type ouster_ros/OusterSensor to manager os_nodelet_mgr with the following remappings:
[ INFO] [1725982932.293050262]: waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] has not been advertised, waiting...
[ INFO] [1725982932.294509733]: Loading nodelet /ouster/os_cloud_node of type ouster_ros/OusterCloud to manager os_nodelet_mgr with the following remappings:
[ INFO] [1725982932.298256232]: waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] has not been advertised, waiting...
[ INFO] [1725982932.299489196]: Loading nodelet /ouster/img_node of type ouster_ros/OusterImage to manager os_nodelet_mgr with the following remappings:
[ INFO] [1725982932.299622864]: Initializing nodelet with 8 worker threads.
[ INFO] [1725982932.314283501]: waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] is now available.
[ INFO] [1725982932.317125156]: OusterImage: node initialized!
[ INFO] [1725982932.319181422]: waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] is now available.
[ INFO] [1725982932.323722392]: reset service created
[ INFO] [1725982932.323961602]: get_metadata service created
[ INFO] [1725982932.324198765]: get_config service created
[ INFO] [1725982932.324426216]: set_config service created
[ INFO] [1725982932.326390974]: Will send UDP data to 192.168.113.195
[ INFO] [1725982932.326405420]: Contacting sensor 169.254.192.176 ...
[ INFO] [1725982932.434273591]: Sensor 169.254.192.176 was configured successfully
[ INFO] [1725982932.434348984]: Starting sensor 169.254.192.176 initialization... Using ports: 7502/7503
[2024-09-10 17:42:12.434] [ouster::sensor] [info] initializing sensor client: 169.254.192.176 expecting lidar port/imu port: 7502/7503 (0 means a random port will be chosen)
[ INFO] [1725982937.387807405]: rviz version 1.14.25
[ INFO] [1725982937.387842245]: compiled against Qt version 5.12.8
[ INFO] [1725982937.387848643]: compiled against OGRE version 1.9.0 (Ghadamon)
[ INFO] [1725982937.394058730]: Forcing OpenGl version 0.
[ INFO] [1725982937.504260036]: Stereo is NOT SUPPORTED
[ INFO] [1725982937.504309830]: OpenGL device: llvmpipe (LLVM 12.0.0, 256 bits)
[ INFO] [1725982937.504320998]: OpenGl version: 4.5 (GLSL 4.5) limited to GLSL 1.4 on Mesa system.
[2024-09-10 17:42:29.069] [ouster::sensor] [info] parsing non-legacy metadata format
[ INFO] [1725982949.071479386]: OusterImage: retrieved new sensor metadata!
[2024-09-10 17:42:29.072] [ouster::sensor] [info] parsing non-legacy metadata format
[ INFO] [1725982949.084645981]: Wrote sensor metadata to /home/mapper/metadata.json
[ INFO] [1725982949.084790939]: ouster client version: 0.10.0+2898060-release
product: OS-1-128, sn: 992038000083, firmware rev: v2.5.3
lidar mode: 512x10, lidar udp profile: LEGACY
[ INFO] [1725982949.128753576]: OusterCloud: nodelet created!
[ INFO] [1725982949.128811387]: OusterCloud: retrieved new sensor metadata!
[2024-09-10 17:42:29.129] [ouster::sensor] [info] parsing non-legacy metadata format
[ WARN] [1725982949.135778085]: point_cloud_frame is set to sensor_frame. If you need to reproject or raytrace the points then use the lidar_frame instead. Refer to: #33 for more details.
[ INFO] [1725982949.139339167]: OusterCloud: using static transforms broadcast
REQUIRED process [ouster/os_nodelet_mgr-2] has died!
process has died [pid 43842, exit code -11, cmd /opt/ros/noetic/lib/nodelet/nodelet manager __name:=os_nodelet_mgr __log:=/home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/ouster-os_nodelet_mgr-2.log].
log file: /home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/ouster-os_nodelet_mgr-2*.log
Initiating shutdown!

[rviz-6] killing on exit
[ouster/img_node-5] killing on exit
[ouster/os_node-3] killing on exit
[ouster/os_cloud_node-4] killing on exit
[ouster/os_nodelet_mgr-2] killing on exit
[ INFO] [1725982949.473568623]: Unloading nodelet /ouster/img_node from manager os_nodelet_mgr
[ INFO] [1725982949.473692499]: Unloading nodelet /ouster/os_node from manager os_nodelet_mgr
[ INFO] [1725982949.473811439]: Unloading nodelet /ouster/os_cloud_node from manager os_nodelet_mgr
[ INFO] [1725982949.476741537]: waitForService: Service [/ouster/os_nodelet_mgr/unload_nodelet] could not connect to host [EOS-PC-01:47311], waiting...
[ WARN] [1725982949.476782891]: Couldn't find service os_nodelet_mgr/unload_nodelet, perhaps the manager is already shut down
[ INFO] [1725982949.477072606]: waitForService: Service [/ouster/os_nodelet_mgr/unload_nodelet] could not connect to host [EOS-PC-01:47311], waiting...
[ WARN] [1725982949.477121257]: Couldn't find service os_nodelet_mgr/unload_nodelet, perhaps the manager is already shut down
[ INFO] [1725982949.477417235]: waitForService: Service [/ouster/os_nodelet_mgr/unload_nodelet] could not connect to host [EOS-PC-01:47311], waiting...
[ WARN] [1725982949.477476105]: Couldn't find service os_nodelet_mgr/unload_nodelet, perhaps the manager is already shut down
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done`

Could you help me with this bug?
Thanks
retepik
@retepik retepik added the bug Something isn't working label Sep 10, 2024
@Samahu
Copy link
Contributor

Samahu commented Sep 10, 2024 via email

@Samahu
Copy link
Contributor

Samahu commented Sep 10, 2024 via email

@retepik
Copy link
Author

retepik commented Sep 11, 2024

Hi Samahu,
I didn't make changes in original code, I used cloned code from git repositary

@yulan0215
Copy link

Hi Samahu, I didn't make changes in original code, I used cloned code from git repositary

Hi, did you have any solution for this? I have the same sensor and same problem, thx.

@yulan0215
Copy link

yulan0215 commented Oct 17, 2024 •
edited
Loading

I also used the very recently version ROS driver and the firmware is:
image
Can you please check if the problem was from firmware or IP?
The error is shown here:
image

image
Thanks for your help!

@Samahu
Copy link
Contributor

Samahu commented Oct 17, 2024

Here are few things to try that can help us narrow down the problem (the log isn't giving sufficient output unfortunately), try each step one by one:

  • First update to the latest version 0.13.1
  • Switch to driver.launch instead of the sensor.launch (it offers the same parameters).
  • Try leaving lidar_port and imu_port values set to zero so the driver would pick random free ports
  • Set proc_mask="IMG" alone or proc_mask="IMU" alone or even leave empty proc_mask="", this should help us find if the bug is a result of one processors.

Finally, you may need to enable GDB (but let's try this after the steps suggested above).

@Samahu
Copy link
Contributor

Samahu commented Oct 17, 2024 •
edited
Loading

Can you please check if the problem was from firmware or IP?
The error is shown here:

@yulan0215 your problem could be different, it is not clear how do you initialized the driver, a text log would be much clearer than an image.

@yulan0215
Copy link

Can you please check if the problem was from firmware or IP?

The error is shown here:

@yulan0215 your problem could be different, it is not clear how do you initialized the driver, a text log works would be much clearer than an image.

Many thanks for your quick reply. Although right now I could not change the Roslauch and I want to mention I also loss the data even there was no error or warning from terminal. I have to mention my ouster LiDAR directly connected to Jetson Xavier NX with Ethernet cable, do u have any idea for that? Many thx.

@Samahu
Copy link
Contributor

Samahu commented Oct 17, 2024

I want to mention I also loss the data even there was no error or warning from terminal. I have to mention my ouster LiDAR directly connected to Jetson Xavier NX with Ethernet cable, do u have any idea for that? Many thx.

This might be a network related issue, try applying the recommend network settings described in https://github.com/ouster-lidar/ouster-ros/blob/master/util/network-configure.bash

@yulan0215
Copy link

yulan0215 commented Oct 19, 2024 •
edited
Loading

I want to mention I also loss the data even there was no error or warning from terminal. I have to mention my ouster LiDAR directly connected to Jetson Xavier NX with Ethernet cable, do u have any idea for that? Many thx.

This might be a network related issue, try applying the recommend network settings described in https://github.com/ouster-lidar/ouster-ros/blob/master/util/network-configure.bash

Hi, image
I have this issue of Ethernet problem. Do you think if increasing the buffer can solve the data loss problem? Thx

And I want to say that I had data loss problem when the system ran long time.

And I want to say that I had lidar data loss problem when the system ran long time.

in addition you can try increasing the receive buffer size

"# sudo ethtool -G <sensor_network_interface> rx 2048"

"## increase the MTU size to 9000 to decrease fragmentation"
"# sudo ifconfig <sensor_network_interface> mtu 9000 "
I used Jetson Xavier NX and do you have any suggestion if I need the configure above(commented) or not? If they are needed, do you have any suggested configuration?Like the 2048 is suitable in my case? Thx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Samahu @retepik @yulan0215