Course Introduction

In this course, we will learn

Programming techniques, specifically functional programming, useful for Big Data Systems.
Basic distributed systems theory, applied on databases and file systems
Batch processing
Stream processing

During our course we will learn how to process data with

The course will be using the following 3 programming languages

Week	Date	Topic	Teacher	Assignment (Deadline)
1	14/11	Course introduction, Big and Fast data, Intro to course PLs	GG
1	15/11	The Unix programming environment	GG	Unix (jupyter) (28/11)
2	21/11	Programming for Big Data (1)	GG	Functional programming: Scala (jupyter), Python (jupyter) (4/12)
2	22/11	Programming for Big Data (2)	GG
3	28/11	Distributed Systems	JR
3	29/11	Distributed Databases, Distributed filesystems	JR

Week	Date	Topic	Teacher	Assignment (Deadline)
4	5/12	Spark: RDDs and Pair RDDs	GG	Spark (18/12)
4	6/12	Spark Internals	JR
5	12/12	Spark SQL, Spark use cases: Synonyms with Word2Vec, Recommending bands, Predicting pull request merges	GG
5	13/12	Live Data Processing	GG
6	19/12	Stream processing	GG	Streaming (14/1) (Note: Optional for minor students)
6	20/12	Stream processing systems	GG
7	8/1	Recap	GG
7	9/1	No lecture	GG

60% final exam, open questions + multiple-choice.
40% assignments. 4 assignments, 10% each
- For minor students, 3 assignments (the last is optional), 13% each

There will be a resit, there is no mid-term.

You can transfer your assignment grade to the resit AS A WHOLE. No individual assignment resubmissions!

The course consists of 4 mandatory assignments (3 for minor students)
You always work in pairs: select your team mate!
Grade: \(\frac{\sum_{assign=1}^{4} grade(assign)}{4}\), aka the mean
- No submission? Grade \(\rightarrow\) 0
- Late submission? Grade \(\rightarrow\) 0
Most assignment grades add to \(> 10\)
We will use Jupyter for the first 3 assignments, and Intellij/Scala for the last one.

We will use CPM. The course name is TI2736-B: Big Data Processing

You need to signup to enroll and also declare your pairs
To submit, hit the overview button and select the appropriate assignment
All the assignments have deadlines
Feedback and grading is automatic: the results are available on CPM.
Technical support: ask the Mattermost channel
- If no feedback after 1 hour: DO ASK THE TAs.

Lab sessions every Monday morning
- All TAs will be around
- Work together to solve assignments
- Ask questions about the next assignment
- Ask questions about your grades
You are welcome to join the BDP 2018-2019 Mattermost channel
The course VM: Contains all software (Spark, HDFS, Flink) you may need pre-installed
- Find it here
- Provided only for convenience: no technical support
- Default account username/password: bigdata/bigdata

Q A question with a known answer; this will be revealed, but we should work together towards it!
D A open discussion item; we need to think and discuss.

Freely available on the web, on my homepage (http://gousios.org/teaching.html)
You can print/download them before the lecture and bring them along to make additional notes.
I am looking forward to improve them! If you have suggestions, find errors etc, let me know!