@@ -11,11 +11,23 @@ The z value of the pose message is then the detected depth in the camera frame (
The detected signs are cropped to patches and zoomed so that the patch width/height matches our tensorflow requirements.
This segmentation depends on a some hyper parameters.
Those are loaded based on the input of `config/sign_detector.yaml`.
A tool to adjust them can be started by executing `ROS_NAMESPACE=/$ROBOT/; rosrun minibot_vision SegmentSign.py` and works with following hotkeys:
A tool to adjust them can be started by executing `rosrun minibot_vision AdjustSegmentSign.py`.
Changes made by the sliders are directly updated at the param server, however,
`SignDetector.py` is only loading new params after an update service call.
The following hotkeys are supported:
- p: Toggle patch visualization [default: True]
- d: Load default parameters (specified within `config/sign_detector_default.yaml`). This will neither override `config/sign_detector.yaml` nor the ros-params.
- s: Save the current parameters persistent to `config/sign_detector.yaml`. This will also upload to ros-param.
The functionality of this tool is also available via ros.
The two hyper-params are on the param server as `sign_detector/canny_param1` and `sign_detector/canny_param2`
and represent param1 and param2 of HoughCircles() at https://docs.opencv.org/3.4/dd/d1a/group__imgproc__feature.html#ga47849c3be0d0406ad3ca45db65a25d2d.
-**crop_sign/enable** (service): enables the functionality of the tool.
-**crop_sign/command** (service): A service to reflect the functionality described above. See `minibot_msgs/srv/segment_sign_command.srv`.
-**crop_sign/result_image/compressed** (topic): A compressed version of the resulting image,
so an image with some information what the sign segmentation is doing.
## Guideline for optimal Sign Detection
The hough transform used to pre-detect the signs works best if the actual traffic signs have the following conditions:
- non reflecting signs (eg. paper)
...
...
@@ -27,7 +39,10 @@ If there are still issues with the segmentation/detection check these points:
- the background of the signs is very important during classification
## Sign Detector
This node starts the SegmentSign node together with a tensorflow classification.
This node starts the SegmentSign node together with a tensorflow classification. \
There is also a separate tool to capture images as training data for the sign classification network.
This will process the output of the realsense camera by SignSegmentation.
We have observed that training the network with a segmented version of the signs leads to a more robust classification.
### Services
-**sign_detector/set_model**: With this service call a model trained and uploaded by `teachablemachine.withgoogle.com` is downloaded by the given url.
...
...
@@ -36,6 +51,14 @@ The new model overwrites the existing model and updates its label on the param s
-**sign_detector/set_visualize**: A service to set the `visualize` flag (default: true). This flag controls the image stream on the topic `sign_detector/result_image/compressed`.
Check the Topics section for more details.
-**sign_detector/enable**: Enables the classification pipeline for the sign detection (default: enabled).
If disabled visualization is still possible, but the label will be set to -1.
-**capture_imgs/enable**: Enables the capture images stream (default: disabled).
-**capture_imgs/toggle_hough_detection**: Enables autofocus for each patch on signs in the camera image (default: enabled).
If this is disabled not a patch of the visible sign will be published but just a patch of the center of the color camera.
### Topics
-**sign_detector/keypoints**: A `vision_msgs.msg.Detection2D` msg will be published.
The relevant data is stored in results as `vision_msgs.msg.ObjectHypothesisWithPose`.
...
...
@@ -50,22 +73,10 @@ This is especially important if tensorflow is performing low to filter results t
The ROS tool image_view can be used for visualization: `rosrun image_view image_view image:=/<namespace>/sign_detector/result_image _image_transport:=compressed` (note that the given topic is without the `/compressed` suffix).
This is intended to be used while debugging the image classification neural network.
## Capture Images
This is a tool to capture images as training data for the sign classification network.
This will process the output of the realsense camera by SignSegmentation.
We have observed that training the network with a segmented version of the signs leads to a more robust classification.
You can run it by launching `roslaunch minibot_vision capture_imgs.launch`.
There are two arguments:
-`remote_node`: (default: true)
- if true the node runs on the minibot and the resulting images are published to `capture_images/result_image` as raw and compressed.
- otherwise the node runs in local mode and the script will ask you to type the filename of the resulting images.
Then the images are visualized and can be sampled by holding `r` and are then saved to the before specified filename.
-`save_dir`: (default: "/resources/training_imgs/") only used in local mode.
The save directory relative to the minibot_package for all images (they will be in sub-folders of their filename).
If the node is in remote mode, you need to call the service `capture_images/enable` to activate the node.
-**capture_imgs/result_image**: The topic where only the cropped images of `capture_imgs` are published.
Only works when `capture_imgs/enable` is set to true.
-**capture_imgs/result_image/compressed**: A compressed version of `capture_imgs/result_image`.
## TODO
- [ ] Crop sign also publish x and y position in camera frame.
- [ ] SignDetector: Publish multiple keypoints (currently only one is published)
