Video Semantic Scenes Detection

The paper considers application of the Transformer architecture to solve the problem of splitting video into semantic scenes. Semantic segmentation is aimed to separate scenes based on storyline for further top-­level analysis. The following problem statement is considered ­- short scenes (shots) taken sequentially by a single camera are extracted with existing algorithms, the method considered in this paper groups the resulting short scenes into final semantic scenes. Short scenes within a semantic scene are united by a storyline, characters, place of action and form a separate part of the narrative. The grouping problem is solved as a binary classification problem of sequence elements. ­For each short scene, it is predicted whether it is the end of the semantic scene. Adapting the Transformer architecture allows usage of longer video context than existing approaches do. In addition, it is possible to use pre­training on unlabeled data (similar to the BERT and VideoBERT) and utilize pre­trained for other tasks weights of architectures based on the Transformer encoder. The paper presents experiments on adapting the Transformer model encoder for the task of video semantic scenes segmentation, experiments with size of the architecture, length of the history, increasing the stability of training, pre­training using masked loss function, using the weights of the existing pre­trained COOT model for video analysis, adding a BNet block from the LGSS model specific to this task. The resulting model is comparable in quality to the existing methods of video segmentation into semantic scenes and surpasses methods that use only place features by Average Precision and mean Intersection over Union metrics on the open MovieNet­-SSeg feature film dataset.