Applying Hierarchical Text Clustering for Building Category Tree on a Classified Website

The rapid growth in the number of user-generated online content has created the demand of efficiently classifying such data. Document clustering algorithms play an important role in providing intuitive navigation and browsing mechanisms by organizing large amounts of information into a small number of meaningful clusters. This paper proposes clustering technique intended to build category tree on a classified website Avito. The paper presents a comprehensive study of document clustering algorithms that build hierarchical solutions and use different criterion functions and merging schemes and proposes the most appropriate method regarding data volume, dimensionality, and noisiness – density-based clustering OPTICS. Algorythm is implemented and optimized with creating statial indexes for data base. In addition several dimensionality methods are introduced to solve the problem of data scalability: TF-IDF matrix prunning, feature extraction with latent-semantic analysis and feature selection with random forest. All techniques are implemented and tested on a text corpus of 15 000 texts. Comparison analysis of different outputs resulted in final clusters' list that have been proposed to business stakeholders.