Autonomous Recognition of Collective Motion Behaviours in Robotic Swarms from Video Using a Deep Neural Network

Recognition of swarm behavior is important for two reasons. First, it permits the early detection of adversarial collective motion behaviours such that counter-collective motion can be activated. Second, it permits the monitoring and assessment of own swarms to enable detecting any disruptions to the required behavior. Existing work in this area requires feature-based data provided by an external observer that has access to all the swarm states that could not be available all the time. However, the need for pre-processing swarm data to calculate its features can lead to inefficient behavior recognition. This paper addresses this limitation by using raw video data for swarm behavior recognition. This paper proposes a new framework to autonomously recognize structured collective behavior in swarm robots from camera data First, we present a dataset of both collective motion and random behaviors of Pioneer 3DX robots. Then we formulate the recognition problem as a spatiotemporal learning problem. A recurrent neural network is used as the main building block to evaluate each video representation—Our experimental results showed that this video-based recognition without any data pre-processing results in high accuracy that is comparative to feature-based recognition techniques. The proposed model is able to efficiently recognise robots’ behaviour with 99.8% accuracy. Also, our methodology can distinguish behaviours of real robots with 79% accuracy even with different numbers of agents.