Bilkent University
Department of Computer Engineering
CS 690 SEMINAR
Joker: A Stream Processing Engine with Organic Adaptation
Basri Kahveci
PhD Student
Computer Engineering Department
Bilkent University
Stream processing is a paradigm that processes large volumes of continuous data streams in an on-the-fly manner. The high throughput processing requirement of streaming applications makes it necessary to take advantage of multi-processors and multiple machines to achieve scalability. The data flow graph representation of streaming applications eases development of computations and contains a rich set of parallelization opportunities, such as pipeline, task, and data parallelism. However, there are several challenges in applying parallelism to scale streaming applications, while keeping the simplicity of the data flow graph representation. In this study, we present Joker, an elastic stream processing engine that performs organic adaptation, which is the ability of scaling execution of a streaming application in a dynamic, automatic, and transparent manner. First, it proposes a unified operator API that greatly simplifies development of stateless and stateful operators, while at the same time making operator properties visible to the runtime system. Second, it develops a parallelized execution model and accompanying runtime machinery that is able to change the level of parallelization at run-time, transparently to the user code. Last, but not the least, Joker determines a set of metrics to characterize performance of a parallelized execution, identifies bottlenecks, and decides on a parallelization configuration change to resolve the bottlenecks and improve throughput, all at run-time.
DATE: 24 April, 2017, Monday @ 15:40
PLACE: EA-409