Conference Article

·2019

On the Use of Hyperparameter Optimization in Big Data Processing Pipelines: A Case Study

Jasser Dhaouadi YTU , Mehmet S. Aktaş YTU , Oya Kalıpsız YTU , Erman Balçık

2019 Innovations in Intelligent Systems and Applications Conference (ASYU)

Abstract

The term "Data Analytics" sparks a wide range of multidisciplinary fields since it requires a high analytical expertise in different domains. Data Analytics applications include many successive steps, such as data collection, outlier detection, missing data imputation, feature selection, clustering analysis, classification model selection and result interpretation. The whole procedure can be seen as a chain of steps, organized in a pipeline manner, where the output of the upper layer is the input of the lower layer. Thus, the tasks of every step depend directly on the results of the previous steps. Moreover, there are alternative algorithmic methods for each step in Data Analytics. Opting for one methodology over the other requires high-skilled data scientists with a huge technical background. The key point of such a decision is optimizing the parameters of every layer in the pipeline. In this study, we develop an automated pipeline with different layers. Every layer contains several methods. We investigate the implementation of a suitable hyperparameter optimization algorithm, which allows the pipeline to be autonomous and select wisely the best algorithm for every layer. We discuss the specifics of the proposed prototype and the details of the used frameworks. We evaluate the prototype with an experimental study. The results are pertinent.

Keywords

Computer science Data mining Pipeline (software) Outlier Cluster analysis Analytics Hyperparameter Pipeline transport Missing data Data analysis Machine learning Artificial intelligence Engineering

Subject Areas

Machine Learning and Data Classification ·Artificial Intelligence ·Physical Sciences
Scientific Computing and Data Management ·Information Systems and Management ·Social Sciences
Gaussian Processes and Bayesian Inference ·Artificial Intelligence ·Physical Sciences

Citations by Year