As we have seen during the lecture, the Unix command line is extremely versatile. This comes at a cost: you need to know which tool is suitable for the job or to build it. For this assignment, you will have to use a Unix command line (i.e. in the course's VM, a Mac or Ubuntu for Windows) to accomplish the indicated tasks. Not all tools you will need are readily available in the default installation of Xubuntu that we are using in the course. You need to search online (or implement!) for the appropriate tools and command line options.
Note: To develop and run those commands in Jupyter, you need to install the Bash kernel. Alternatively, you can create a file where you paste all programs/pipelines, print it and upload it to the assignment directory.
T (10 points) Write a pipeline that converts (recursively!) a directory
structure full of .wav files to .mp3s.
T (10 points) Write a program that given a directory of text files, it prints the 10 most common words in those files (across all files).
T (10 points) Write a program that will print all files (recursively!) that where not accessed the last 30 days.
Hint: Use date to format date strings
T (10 points) Write a program that checks whether all the links in this web page work and reports the ones that do not. You can check whether a link works by inspecting the HTTP return header for 404 errors.
T (10 points) Write a program that will create a tar.gz archive out
of a directory of source code, ommiting all files that are binary (i.e. non-text).
Hint: Use file --mime
T (10 point) Implement a case-insensitive spell checker. Given an input file, it should report all words not in the dictionary.
Hint: You can use this dictionary file
For the assignments in this section, we use the same log file and interesting.csv files that we used in the Spark assignment.
Write pipelines to calculate answers to the following questions (they may look familiar :-)):
T (10 points) Count the number of WARNing messages
T (10 points) How many repositories where processed in total? Use the api_client
lines only.
T (10 points) Which client did most HTTP requests?
T (10 points) What is the most active repository?
T (10 points): Which access keys are failing most often?
T (10 points): Which of the interesting repositories has the most failed API calls?