Title
Sparkmach: A Distributed Data Processing System Based on Automated Machine Learning for Big Data
Date Issued
2019
Access level
restricted access
Resource Type
book part
Publisher(s)
Springer Verlag
Abstract
This work proposes a semi-automated analysis and modeling package for Machine Learning related problems. The library goal is to reduce the steps involved in a traditional data science roadmap. To do so, Sparkmach takes advantage of Machine Learning techniques to build base models for both classification and regression problems. These models include exploratory data analysis, data preprocessing, feature engineering and modeling. The project has its basis in Pymach, a similar library that faces those steps for small and medium-sized datasets (about ten millions of rows and a few columns). Sparkmach central labor is to scale Pymach to overcome big datasets by using Apache Spark distributed computing, a distributed engine for large-scale data processing, that tackle several data science related problems in a cluster environment. Despite the software nature, Sparkmach can be of use for local environments, getting the most benefits from the distributed processing tools. © 2019, Springer Nature Switzerland AG.
Start page
121
End page
128
Volume
898
Number
1
Language
English
Scopus EID
2-s2.0-85063475416
Resource of which it is part
Information Management and Big Data; Communications in Computer and Information Science
ISSN of the container
1865-0929 1865-0937
Conference
5th International Conference on Information Management and Big Data, SIMBig 2018
Sponsor(s)
Acknowledgments. The project would have been impossible without the support of Ciencia Activa and Fondo para la Innovación, la Ciencia y la Tecnología - Innovation, Science and Technology Fund (FINCyT).
Sources of information: Directorio de Producción Científica