Secure, Reproducible And Adaptive Machine Learning In Distributed Systems

Abstract

Distributed machine learning allows multiple workers to focus a same machine learning task collaboratively. Motivated by the explosive growth in scientific and industry technical researches about distributed systems and machine learning, the dissertation explores three main topics of machine learning in distributed systems, includes security, reproducibility and adaptivity. For tackling the challenge of security in distributed machine learning, we present an intrusion-tolerant federated learning architecture namely ReDeF. On the parameter server side, ReDeF utilizes Byzantine fault-tolerant protocol to enhance the reliability of the service. On the worker side, we study a new failure model called fully corrupted workers that capture scenarios that are never considered before. To handle fully corrupted workers, we propose a worker centric approach, allowing workers to collaboratively decide how the global model evolves. To support the approach, we also propose a new data structure called the global model tree, allowing workers to track the evolution of global models. For reproducibility in distributed machine learning with cloud, in order to achieve automate end-to-end execution of analytics and solve the vendor lock-in problem, we propose and develop an open-source toolkit that supports 1) fully automated end-to-end execution and reproduction via a single command, 2) automatic data and configuration storage for each execution, 3) flexible client modes based on user preferences, 4) execution history query, and 5) simple reproduction of existing executions in the same environment or a different environment. Different from traditional analytics that assumes data to be processed are available ahead of time and will not change, stream analytics deals with data that are being generated continuously and data distribution could change (aka concept drift), which will cause prediction/forecasting accuracy to drop over time. One other challenge is to find the best resource provisioning for stream analytics to achieve good overall latency. Based on these, we study how to best leverage edge and cloud resources to achieve better accuracy and latency for RNN-based stream analytics. We propose a novel edge-cloud integrated framework for hybrid stream analytics that support low latency inference on the edge and high capacity training on the cloud. We also study three flexible deployments of our hybrid learning framework. Further, our hybrid learning framework can dynamically combine inference results from an RNN model pre-trained based on historical data and another RNN model re-trained periodically based on the most recent data. Our evaluations and discussions in each chapter show that all the proposed solutions are effective and efficient for solving the security, reproducibility and adaptivity challenges respectively.