Big Data and Hadoop
Instructor: Venkatesh Vinayakarao
Term: Jan - Apr 2020
TA: Suchitra

Welcome to Big Data and Hadoop course! Massive increase in the availability of data has made the storage, management, and analysis extremely challenging. Various tools, technologies and frameworks have surfaced to help address this challenge. Apache Hadoop is one such framework that enables us to handle big data by making distributed computing easier. Concerns such as reliability, distributed file management and distributed processing have been abstracted from us by hadoop. In this course, we shall start with understanding the characteristics of big data and the fundamental concepts of cloud computing. We will explore the hadoop ecosystem. Specifically, we will explore HDFS, Map-Reduce, Pig and NoSQL DB. Our objective is to handle big data effectively and build web applications and RESTful services over cloud. This is an introductory course focused on the breadth of the big data landscape.

Key Learning Objectives

At the end of this course, you should be able to:
  • Understand how distribtued file systems work. Be able to use Hadoop HDFS.
  • Understand distributed processing fundamentals. Code using map-reduce framework and pig scripts.
  • Understand NoSQL DB concepts. Use any one NoSQL DB such as HBase/Hive/MongoDB.

Lecture Schedule

Lecture #TopicReadingsSlides/Material
Part 1: Introduction
1Introduction to Big DataThe Complete Beginner's Guide To Big Data Everyone Can Understand
Basics About Cloud Computing
Lecture 1
Lecture 2
2Hands-On Tutorial: A Tour of Big Data Stack with Cloudera VMCDH OverviewTutorial1
3Distributed File SystemsFiles and Directories
File System
The Hadoop Distributed File System
4Hands-On Tutorial: HDFSExploring the File System
5Distributed Processing with Map-Reduce and PigOverview of Map-Reduce
6Hands-On Tutorial: Map-Reduce and PigMapReduce Tutorial
Pig Tutorial
7Introduction to Design PatternsChapter 1 from Thinking in Patterns
8Big Data Design PatternsMap-Reduce Design Patterns
9NoSQL DBNoSQL Databases by Christof Strauch
NoSQL Explained
10Hands-On Tutorial: HBase/Hive/MongoDBMongoDB CRUD Operations
11Web Application Development and Service Oriented ArchitectureWeb Application Development
12Hands-On Tutorial: Apache Tomcat, JSON, RESTful ServicesBuilding Web Applications with Tomcat
RESTful Services
13Putting it all together, Building Webscale Applications
14Hands-On Tutorial: SolrApache Solr Tutorial
Part 2: Products and Practices - Student Presentations
15Student Presentation: Azure and AWS (or Apache Spark and Google Cloud)
16Student Presentation: Big Data Visualization using Tableau
17Student Presentation: Crawling with Apache Nutch
18Student Presentation: Data Collection with Apache Flume and Sqoop
19Student Presentation: Logging with log4j and Log Aggregation with Apache Flume
20Student Presentation: Graph Databases and Neo4j
21Student Presentation: Stream Processing with Apache Storm
22Student Presentation: Co-ordination with ZooKeeper
23Student Presentation: Workflow Management with OOZIE
24Student Presentation: Publish-Subscribe Messaging Frameworks with Apache Kafka and Kinesis
25Student Presentation: Knowledge Discovery(or Apache Mahout)
Part 3: Advanced
26Research Trends in Big Data
27-31Advanced concepts (if time permits)

InstrumentMax Marks
Mid Exam20%
Final Exam30%
Hands-on Tutorials (2% * (best of) 5 Quizzes)10%
Student Presentations (15% for presentation, 5% (0.5% * 10) for quiz on the presentation topic)20%

This is a group project. Allowed group sizes are 3 and 4. Submit a one page report before the project demonstration.



There is no prescribed text for this course. Readings will be shared during the lectures.

References Optional Readings

If you are not having fun, you are not the best student you can be!