ISIT312/ISIT912 Big Data Management

Assignment 1

This assignment contributes to 5% of the total evaluation in the subject. This assignment consists of 3 tasks. Specification of each task starts from a new page.

It is a requirement that all Laboratory and Assignment tasks in this subject must be solved individually without any cooperation with the other students. If you have any doubts, questions, etc. please consult your lecturer or tutor during lab classes or office hours. Plagiarism will result in a FAIL grade being recorded for that assessment task.

Deliverables

You must submit one PDF file for each task and a Java source file for Task 3.

The PDF files must be up to 5 pages. The answers must be presented in a readable manner that facilitates evaluation. Poor presentation will result in a reduction of marks.

The Java sourcecode must be correct and compilable. The environment for Task 1 and Task 3 is the BigDataVM imported from the BigDataVM-12-07-2019.ova file

Task 1. PutCombine Application (1.5 marks)

This task is based on the sourcecode PutCombine.java, and the data in FilesToBeMerged.zip (available on the Moodle site).

The PutCombine application extends Hadoop’s own functionality. The motivation for this application comes when we want to analyse fragmented files such as logs from web servers. We can copy each file into HDFS, but in general, Hadoop works more effectively with a single large file rather than a number of smaller ones. Besides, for analytics purposes we think of all the data as one big file, even though it spreads over multiple files as an incidental result of the physical infrastructure that creates the data.

One solution is to merge all the files first and then upload the combined file into HDFS. Unfortunately, the file merging will require a lot of disk space in the local machine. It would be much easier if we could merge all the files on the fly as we upload them into HDFS.

What we need is, therefore, a “put-and-combine”-type of operation. Hadoop’s command line utilities include a getmerge command for merging a number of HDFS files before copying them onto the local machine. What we’re looking for is the exact opposite, which is not available in Hadoop’s file utilities. The attached sourcecode of PutCombine is a Java application that fulfils this purpose.

1. Compare PutCombine with the FileSystemPut and FileSystemPutAlt applications in the lecture note and describe the difference. You must link the difference to the sourcecode.

(0.8 mark)

1. Compile PutCombine.java and create a jar file, and use to it to upload and merge the files in FilesToBeMerged.zip (you need to unzip it first). Produce a report in the PDF format that includes all the commands that you enter to a Zeppelin notebook or the Terminal as well as brief explanations of those commands (e.g., the purpose of each command). (0.7 mark)

Task 2. MapReduce Model (2 marks)

Produce the answers to the following questions in a PDF file.

• (1 mark) Suppose there are two sets of letters D1 and D2 stored on two nodes A1 and A2: D1 = {“a”, “b”, “c”, “b”, “c”, “d”, “a”} on Node A1, and
• D2 = {“a”, “a”, “a”, “d”, “d”, “c”} on Node A2.

There is a MapReduce job running to process D1 and D2. The job is a “get average” application, namely, computing the average number of copies for each letter.