Data Engineering in Pedestrian Dead Reckoning Sensor Data with Azure Cloud Services

To process sensors data recorded from mobile sensors efficiently with minimal cost and decent speed, after considering all the available options and compare the advantages and disadvantages of all components in a data pipeline, we proposed an optimal solution in this thesis, to build a data pipeline on the public cloud and explain the reasons for each choice we have selected for each component. Although cloud providers have improved their offerings, many researchers still have doubts about all the different tools offered by cloud service providers and many of them have little knowledge on “when to use which and why”. Therefore we provide our comparison table with our size of data, type of data, and our specific requirement in time as constraints in data pre-processing, ML applying stage, and data manipulation afterward. We compare all offers which are specified for parallel programming, and designed to be run on clusters, then give our reason why a specific solution is better than the others. We also provide an implemented example that uses the Batch job as the foundation of the pipeline and with several tools that facilitate creating and managing virtual machines on compute nodes and submitting jobs to these nodes. Service monitoring is added to each stage of the pipeline to capture the possible errors and help to resolve them in real-time. Docker containers and an image container registry are used to create and store an image for a dashboard web app. Finally, we share with the readers what we have learned from this half-year project and share part of our configurations and set up to enhance transparency in our computational data processing.