The term “Big Data” describes datasets that are either too big or change too fast or both to be processed on a single computer.
Big Data Processing provides an introduction to systems used to process Big Data. The main focus of the course is programming and engineering big data systems; initially, the course explores general programming primitives that span across big data systems and touches upon distributed systems. Then, the course examines in detail the implementation of data analysis algorithms in Spark, in the context of batch processing applications, and Flink, in the context of streaming applications.
The course is also optional for the Minor “Software Design and Application.”
After the end of the course, all students should be able to:
5 ECTS: This means that you need to devote at least 140 hours of study for this course.
Lectures: The course consists of 12 2-hour lectures. You are not required, but you are strongly encouraged, to attend.
Homework: In the homework assignments, you will have to write code or reply to open questions. You will always work in pairs.
Groups: The students are responsible to form pairs and communicate them to the course TAs, by registering them to CPM.
Labs: 4 hours per week, designed to help you work together and get support from teaching assistants.
Teaching Assistants: Teaching assistants will be available during lab hours to provide your with feedback on your assignemnts. Do be active in asking questions, but don’t expect them to provide you with solutions.
The head TA is Yoshi van den Akker. The TA team is managed by Goshia Migut.
You can find the course assignments linked through this page.
For submission, we will use CPM. The course name is TI2736-B: Big Data Processing
The student groups must submit each assignment before 23:59 on the day of the deadline.
Late submission: All submissions must be handed in time, with no exceptions. Any late submission will be discarded and will be graded with 0. In case of provable sickness, please contact the course teacher to arrange a case-specific deadline.
There will be an exam-only resit during Q3/4. You are allowed to transfer your assignment grade to the resit as a whole. This means that you will not be able to re-submit individual assignments. Effectively, you can only resit your written exam.
Lab sessions every Monday morning
You are welcome to join the BDP 2018-2019 Mattermost channel
The course VM: Contains all software (Spark, HDFS, Flink) you may need pre-installed
The course, by design, touches upon various current technologies; as such, there is no single source of truth. The following is an indicative list of resources where more information can be found.
This work is (c) 2017, 2018 - onwards by TU Delft and Georgios Gousios and licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.