From f39f06f2cc13721d7b2fb15aaea38cbe6a9f96b9 Mon Sep 17 00:00:00 2001 From: minibot-1 <paddy-hofmann@web.de> Date: Fri, 17 Mar 2023 12:43:11 +0000 Subject: [PATCH] docu --- minibot_vision/README.md | 47 +++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/minibot_vision/README.md b/minibot_vision/README.md index ac0f618..fafe6d6 100644 --- a/minibot_vision/README.md +++ b/minibot_vision/README.md @@ -11,11 +11,23 @@ The z value of the pose message is then the detected depth in the camera frame ( The detected signs are cropped to patches and zoomed so that the patch width/height matches our tensorflow requirements. This segmentation depends on a some hyper parameters. Those are loaded based on the input of `config/sign_detector.yaml`. -A tool to adjust them can be started by executing `ROS_NAMESPACE=/$ROBOT/; rosrun minibot_vision SegmentSign.py` and works with following hotkeys: + +A tool to adjust them can be started by executing `rosrun minibot_vision AdjustSegmentSign.py`. +Changes made by the sliders are directly updated at the param server, however, +`SignDetector.py` is only loading new params after an update service call. +The following hotkeys are supported: - p: Toggle patch visualization [default: True] - d: Load default parameters (specified within `config/sign_detector_default.yaml`). This will neither override `config/sign_detector.yaml` nor the ros-params. - s: Save the current parameters persistent to `config/sign_detector.yaml`. This will also upload to ros-param. +The functionality of this tool is also available via ros. +The two hyper-params are on the param server as `sign_detector/canny_param1` and `sign_detector/canny_param2` +and represent param1 and param2 of HoughCircles() at https://docs.opencv.org/3.4/dd/d1a/group__imgproc__feature.html#ga47849c3be0d0406ad3ca45db65a25d2d. +- **crop_sign/enable** (service): enables the functionality of the tool. +- **crop_sign/command** (service): A service to reflect the functionality described above. See `minibot_msgs/srv/segment_sign_command.srv`. +- **crop_sign/result_image/compressed** (topic): A compressed version of the resulting image, + so an image with some information what the sign segmentation is doing. + ## Guideline for optimal Sign Detection The hough transform used to pre-detect the signs works best if the actual traffic signs have the following conditions: - non reflecting signs (eg. paper) @@ -27,7 +39,10 @@ If there are still issues with the segmentation/detection check these points: - the background of the signs is very important during classification ## Sign Detector -This node starts the SegmentSign node together with a tensorflow classification. +This node starts the SegmentSign node together with a tensorflow classification. \ +There is also a separate tool to capture images as training data for the sign classification network. +This will process the output of the realsense camera by SignSegmentation. +We have observed that training the network with a segmented version of the signs leads to a more robust classification. ### Services - **sign_detector/set_model**: With this service call a model trained and uploaded by `teachablemachine.withgoogle.com` is downloaded by the given url. @@ -36,6 +51,14 @@ The new model overwrites the existing model and updates its label on the param s - **sign_detector/set_visualize**: A service to set the `visualize` flag (default: true). This flag controls the image stream on the topic `sign_detector/result_image/compressed`. Check the Topics section for more details. +- **sign_detector/enable**: Enables the classification pipeline for the sign detection (default: enabled). + If disabled visualization is still possible, but the label will be set to -1. + +- **capture_imgs/enable**: Enables the capture images stream (default: disabled). + +- **capture_imgs/toggle_hough_detection**: Enables autofocus for each patch on signs in the camera image (default: enabled). + If this is disabled not a patch of the visible sign will be published but just a patch of the center of the color camera. + ### Topics - **sign_detector/keypoints**: A `vision_msgs.msg.Detection2D` msg will be published. The relevant data is stored in results as `vision_msgs.msg.ObjectHypothesisWithPose`. @@ -50,22 +73,10 @@ This is especially important if tensorflow is performing low to filter results t The ROS tool image_view can be used for visualization: `rosrun image_view image_view image:=/<namespace>/sign_detector/result_image _image_transport:=compressed` (note that the given topic is without the `/compressed` suffix). This is intended to be used while debugging the image classification neural network. -## Capture Images -This is a tool to capture images as training data for the sign classification network. -This will process the output of the realsense camera by SignSegmentation. -We have observed that training the network with a segmented version of the signs leads to a more robust classification. - -You can run it by launching `roslaunch minibot_vision capture_imgs.launch`. -There are two arguments: -- `remote_node`: (default: true) - - if true the node runs on the minibot and the resulting images are published to `capture_images/result_image` as raw and compressed. - - otherwise the node runs in local mode and the script will ask you to type the filename of the resulting images. -Then the images are visualized and can be sampled by holding `r` and are then saved to the before specified filename. -- `save_dir`: (default: "/resources/training_imgs/") only used in local mode. -The save directory relative to the minibot_package for all images (they will be in sub-folders of their filename). - -If the node is in remote mode, you need to call the service `capture_images/enable` to activate the node. +- **capture_imgs/result_image**: The topic where only the cropped images of `capture_imgs` are published. + Only works when `capture_imgs/enable` is set to true. + +- **capture_imgs/result_image/compressed**: A compressed version of `capture_imgs/result_image`. ## TODO - [ ] Crop sign also publish x and y position in camera frame. -- [ ] SignDetector: Publish multiple keypoints (currently only one is published) \ No newline at end of file -- GitLab