A Universal Railway Obstacle Detection System based on Semi-supervised Segmentation And Optical Flow (2024)

Qiushi Guo
CSRD
guoqiushi@csrd.cn

Abstract

Detecting obstacles in railway scenarios is both crucial and challenging due to the wide range of obstacle categories and varying ambient conditions such as weather and light. Given the impossibility of encompassing all obstacle categories during the training stage, we address this out-of-distribution (OOD) issue with a semi-supervised segmentation approach guided by optical flow clues. We reformulate the task as a binary segmentation problem instead of the traditional object detection approach. To mitigate data shortages, we generate highly realistic synthetic images using Segment Anything (SAM) and YOLO, eliminating the need for manual annotation to produce abundant pixel-level annotations. Additionally, we leverage optical flow as prior knowledge to train the model effectively. Several experiments are conducted, demonstrating the feasibility and effectiveness of our approach.

1 Introduction

With the rapid advancement of high-speed trains, ensuring the security of railway systems has emerged as a critical public concern. One of the primary challenges is obstacle detection, which plays a crucial role in railway safety. Developing a reliable and scalable obstacle detection system can empower train operators and dispatchers to take preemptive actions and mitigate potential accidents.

Deep learning techniques have been widely adopted across various security domains, including mobile payments[3], disaster detection [9], and fraud detection [6]. This technology exhibits substantial promise in enhancing railway safety through sophisticated obstacle detection capabilities. Significant efforts have recently been devoted to addressing obstacle detection using deep learning methods. Although these approaches have achieved some success, they also exhibit notable disadvantages:

A Universal Railway Obstacle Detection System based on Semi-supervised Segmentation And Optical Flow (1)

•
Fragility to complex ambient conditions
•
Requirement for extensive manual annotations
•
Difficulty in extending to different scenarios

Designing an extendable, annotation-free model with strong generalization ability remains a significant challenge in both industry and academia.

To address the aforementioned issues, we propose a semi-supervised approach guided by optical flow. To mitigate the data shortage problem, we employ SAM [7] and YOLO [8] to generate highly realistic pseudo-images for training. Instead of manually collecting and annotating images pixel by pixel, we prepare two image sets: base images (fewer than 100 background images with only railway areas annotated) and object images. The object images include categories such as pedestrians, animals, and textures. Using SAM and YOLO, we obtain masks for the intended objects in these images. These objects are then pasted onto the base images according to the masks. The entire process are illustrated as Fig. 1 This process simultaneously generates image and mask pairs without manual effort.

2 Related work

2.1 Obstacle Detection in Railyway

Matthias Brucker et al.[2]propose a a shallow netwrok to learn railway segmentation from normal railway images. They explore the controlled inclusion of global information by learning to hallucinate obstacle-free images. Zhang Qiang et al.[12]. conbine segmentation model with the LiDAR in their obstacle detection system; Amine Boussik et al.[1] propose an unsupervised models based on a large set of generated convolutional auto-encoder models to detect obstacles on railway’s track level.

2.2 Segmentation with Optical flow

Laura et al. [10]. demonstrate the effectiveness of jointly optimizing optical flow and video segmentation using an iterative scheme; Volodymyr et al[5]. present an architecture forVideo Object Segmentation that combines memory-basedmatching with motion-guided propagation resulting in stable long-term modeling and strong temporal consistency.

3 Method

The pipeline of our approach is illustrated in Fig.2.Given a set of base images $B$ and target images $T$ , our objective is to identify potential obstacles within specific regions $\eta$ . Unlike traditional detection methods that categorically detect each obstacle, we reformulate the problem as a binary segmentation task. Instead of attempting to detect all potential obstacles, which is impractical, our emphasis is on segmenting the railway area, a region that remains consistent over time compared to obstacles.

To simulate these scenarios effectively, we generate highly realistic pseudo-images using a copy-paste approach. Additionally, to address challenges posed by extreme weather conditions, which can obscure object segmentation, we introduce optical flow to provide prior information guiding the segmentation model. Pseudo images $I_{t}$ and $I_{t+\delta}$ are generated by applying a small shift $\delta$ to the target object, simulating its movement. The output of the optical flow model is incorporated along with pseudo images as input to facilitate accurate predictions.This section will delve into the detailed methodology employed throughout this process.

3.1 Data Acquisition

Base Images are used in our experiments are gathered at our facility in Chengdu, which features a railway spanning over 60 meters and includes simulators for fog and rain conditions. To ensure diversity in our dataset, we capture images under different weather scenarios, specifically rainy, foggy, and sunny conditions Fig. 3. Due to the fixed position of the camera, only one mask is required for annotation purposes. Importantly, the railway areas in the base images are devoid of any potential obstacles. Any obstacles present are generated using a copy-paste method.

A Universal Railway Obstacle Detection System based on Semi-supervised Segmentation And Optical Flow (3)

Object Image dataset comprises three categories: PennFudanPed, Obj365 (part) [11], and DTD [4]. To facilitate fully automated application of our methodology, we proceed under the assumption that no masks are initially available. We focus on selecting categories likely to occur in our scenario, such as animals (e.g., deer, horse, cow) and vehicles (e.g., truck, cart). This ensures our approach is tailored to handle relevant objects effectively.

A Universal Railway Obstacle Detection System based on Semi-supervised Segmentation And Optical Flow (4)

The entire process can be delineated into sequential steps: Initially, object images are fed into the YOLO model, which returns a list of bounding boxes identifying detected targets. These bounding boxes serve as inputs for SAM, which generates segmentation masks to outline the object pixels. Subsequently, these segmented object pixels are integrated into the base images based on the segmentation mask guidance.Here, we elaborate on the detailed methodology.

•
Object Detection with YOLO: Object images are inputted into the YOLO model, specifically trained on Obj365, to detect objects belonging to predefined target categories fitting our scenario.
•
Segmentation with SAM: Bounding boxes from YOLO are used as prompts for SAM to generate segmentation masks. These masks delineate object pixels, facilitating their extraction from the object images.
•
Integration with Base Images: Extracted object pixels are seamlessly integrated into the corresponding regions of base images, aligning with the guidance provided by the segmentation masks.

During the SAM stage, while not every segmentation mask achieves perfection, each contributes to the overall objective of accurately segmenting the railway area rather than focusing on obstacles. To address challenges related to out-of-distribution (OOD) scenarios, we introduce random polygon generation with texture rendering from DTD. Additionally, object resizing and rescaling are applied to enrich image content and bolster model robustness.The rescale follow the equation below:

h=\alpha\ast y+\beta

(1)

w=\frac{h}{H}\ast W

(2)

where h,w and H,W are the shapes of the target obj and original obj, respectively. $\alpha$ and $\beta$ are hyper-parameters to adjust the scale. In our project, we set $\alpha$ to 0.6 and $\beta$ to 30.The value should be varied by the camera’s position and it’s parameters. The final generated samples aredemonstrated as Fig .4

3.2 Optical-Flow

Optical flow is based on the assumption that the intensity of a point in an image remains constant as it moves from one frame to the next.

I(x,y,t)=I(x+\Delta t,y+\Delta t,t+\Delta t)

(3)

In our scenario, we employ RAFT (Recurrent All-Pairs Field Transforms) as our chosen model, which demonstrates robust performance across a wide range of scales from tiny to large. The size of obstacles in our dataset varies, spanning from hundreds of pixels down to less than 50 pixels in size. Utilizing the RAFT model requires two consecutive frames for optical flow estimation. Accordingly, we generate two pseudo images $I_{t}$ and $I_{t+1}$ , where the same target objects are pasted with a slight positional shift $\eta$ .

Motion=\phi(I_{t},I_{t+1})

(4)

I_{t+1}=I_{t}(obj_{x}+\Delta x,obj_{y}+\Delta y)

(5)

We set $\Delta x$ and $\Delta y$ range between 5-10. The motion prediction will be leveraged as priorinformation fused with pseudo image to train the model.

4 Experiments

4.1 Dataset and Evaluation Metrics

Dataset Our training dataset is consist of three parts: $obs\_{person}$ , $obs\_{animals}$ and $obs\_{textures}$ , namely person obstacles, animal obstacles and obstacles generated from texture polygons. The details are described as follow:As for test dataset, we recollect images with various obstacles under different weather conditions in different distance to the camera.

Metrics $mIoU$ is used to evaluate the performance of our model. $mIoU$ refers to the Mean Intersection over union, which is a widely used metric in segmentation task. It can be calculated as follow:

IoU_{i}=\frac{TP_{i}}{TP_{i}+FP_{i}+FN_{i}}

(6)

mIoU=\frac{1}{n}\sum_{i=0}^{n}IoU_{i}

(7)

pixel accuracy is also a metric to evaluate the segmentation models.

Pixel\_{accuracy}=\frac{N\_{corr}}{N\_{total}}

(8)

where $N\_{corr}$ is the number of correctly classified pixels, $N\_{total}$ is the number of total pixels.

Name	Volume	Dis(m)	Category
$obs\_{person}$	4000	0-70	person
$obs\_{animal}$	4000	0-70	cow,horse,deer
$obs\_{texture}$	2000	0-70	see DTD
$val\_{near}$	200	0-20	person,rock,board
$val\_{middle}$	200	20-50	person,rock,board
$val\_{far}$	200	50-70	person,rock,board

4.2 Implementation Details

Our method is implemented using the PyTorch framework and the model is trained on an RTX 4070Ti. We select Jaccard loss as the loss function and AdamW as the optimizer. The batch size is set to 8 and the number of epochs to 20. Data transformations include horizontal flip, coarse dropout, and random brightness contrast adjustments.

4.3 Results

To validate the performance of our approach, we conduct experiments on our three self-collected datasets: val_near, val_mid, and val_far. The details are described in Table 1. The basic training dataset contains 10,000 images (4,000+4,000+2,000). To fully assess the impact of the number of generated images, we increase the dataset size by 10% and 50% in rows 4 and 5.

The results are illustrated in Table2, which show that both RAFT and segmentation-based approaches can effectively segment obstacles in our railway area experiments. Combining RAFT and pseudo-images enhances model performance. As more generated images are added to the training dataset, the model’s performance gradually reaches its limit.

	val_near	val_mid	val_far
yolov5	0.744	0.623	0.457
Raft	0.717	0.636	0.585
DeepLab	0.825	0.817	0.747
DeepLab+Raft	0.843	0.828	0.709
DeepLab+Raft+10%	0.837	0.843	0.724
DeepLab+Raft+50%	0.863	0.851	0.802

4.4 Ablation Study

We conduct ablation experiment to validate the effect of different target objects. The results are demonstrated as Table 3 Comparing the row 1,2,3 with row 4, we can find that each obs dataset contributes to improving the robustness and accuracy of the model.

	obs_person	obs_animal	obs_texture	mIoU
1	✓	✓	✗	0.781
2	✓	✗	✓	0.817
3	✗	✓	✓	0.732
4	✓	✓	✓	0.849

5 Conclusion

This paper introduces a universal segmentation model based on a semi-supervised approach. To address out-of-distribution (OOD) challenges, we generate highly realistic pseudo images instead of relying on manual pixel-level annotations. Additionally, we enhance performance by incorporating optical flow techniques. Experimental results demonstrate satisfactory performance across various potential objects.

References

[1]Amine Boussik, Wael Ben-Messaoud, Smail Niar, and Abdelmalik Taleb-Ahmed.Railway obstacle detection using unsupervised learning: An exploratory study.In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 660–667. IEEE, 2021.
[2]Matthias Brucker, Andrei Cramariuc, Cornelius VonEinem, Roland Siegwart, and Cesar Cadena.Local and global information in obstacle detection on railway tracks.In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9049–9056. IEEE, 2023.
[3]Han Cai, JiLin, Yujun Lin, Zhijian Liu, Haotian Tang, Hanrui Wang, Ligeng Zhu, and Song Han.Enable deep learning on mobile devices: Methods, systems, and applications.ACM Transactions on Design Automation of Electronic Systems (TODAES), 27(3):1–50, 2022.
[4]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.Describing textures in the wild.In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014.
[5]Volodymyr Fedynyak, Yaroslav Romanus, Bohdan Hlovatskyi, Bohdan Sydor, Oles Dobosevych, Igor Babin, and Roman Riazantsev.Devos: Flow-guided deformable transformer for video object segmentation.In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 240–249, 2024.
[6]Qiushi Guo, Yifan Chen, and Shisha Liao.Enhancing mobile privacy and security: A face skin patch-based anti-spoofing approach.In 2023 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pages 52–57. IEEE, 2023.
[7]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, AlexanderC Berg, Wan-Yen Lo, etal.Segment anything.In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
[8]Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.You only look once: Unified, real-time object detection.In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
[9]Cem Sazara, Mecit Cetin, and KhanM Iftekharuddin.Detecting floodwater on roadways from image data with handcrafted features and deep transfer learning.In 2019 IEEE intelligent transportation systems conference (ITSC), pages 804–809. IEEE, 2019.
[10]Laura Sevilla-Lara, Deqing Sun, Varun Jampani, and MichaelJ Black.Optical flow with semantic segmentation and localized layers.In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3889–3898, 2016.
[11]Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.Objects365: A large-scale, high-quality dataset for object detection.In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430–8439, 2019.
[12]Qiang Zhang, Fei Yan, Weina Song, Rui Wang, and Gen Li.Automatic obstacle detection method for the train based on deep learning.Sustainability, 15(2):1184, 2023.