Schedule

8/23

Introduction to the course, ice breaker, and overview of distributed systems

Preliminaries

In-class/Homework

Our main goal this weekend is for everyone to deploy at least three virtual machines on Microsoft Azure Cloud and make sure they can all see each other. Your main task over the weekend is to familarize yourself with their system. They provide free (no credit card needed) accounts for students. I found their interface pretty straightfoward, but I’ve been doing this a long time, so I’m giving you the entire weekend to make sure you can deploy 3 VMs and have them all communicate with each other, and you can access them via ssh. Here are some links to help you along the way.

1. Start by getting your free $100 credit: https://azure.microsoft.com/en-us/free/students/.
2. Make sure you know about ssh private/public keys: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/mac-create-ssh-keys.
3. I called my resource group CSIS604.
4. You’ll need to copy and paste your public key when creating your VM.
5. Make sure you create a Ubuntu 17.10. Other OS versions will of course work, but life will be easier if we are all on the same platform.
6. I went with the cheapest model for the VMs. Our goal is to get experience and not great performance.
7. You’ve got to enable SSH as your public inbound port (22).

What to turn in: A PDF copy of the following to OAKS with the content filled in: https://docs.google.com/presentation/d/15Ory2-5Ak11MPd_W1WYBGvObIYjI2OcfUnNEf-xKh1Q/edit?usp=sharing

8/30

Quiz over preliminaries

Discussion of ssh, keys, and VMs.

Snapshots

In-class/Homework

Our goal this week is to start to put our VMs to work with each other. One the simplest but most powerful ways to do that is using distributed compute frameworks such as http://dask.pydata.org/en/latest/.

Please upload and create a PDF from Google Slides that shows:

1. Dask is properly installed
2. Dask workers and initator can all see each other
3. You have successfully deployed one of the use cases shown online that uses all three VMs.

9/6

Quiz over snapshots

HDFS

In-class/Homework

1. Install HDFS and Hadoop on your 3 node cluster in Azure using one of many install guides available.
2. Using this, demonstrate how Dask can be used with HDFS to compute the word count in a distributed environment.
3. Describe one example that you’d like to use Dask to solve that is not listed on the dask website.

Please upload and create a PDF from Google Slides that shows:

1. HDFS and Hadoop are properly installed.
2. Dask can correctly interact with HDFS to perform a word count example.
3. Your example idea.

9/13

No class. Dr. Anderson at a conference.

9/20

Linux Command Line

Wave Algorithms

Remaining time on fixing HDFS issues.

9/27

Class to start at 7:30

Quiz over Wave Algorithms

Containers and clusters (Docker and Dockerswarm)

• https://docs.docker.com/get-started/
• https://docs.docker.com/engine/swarm/swarm-tutorial/

Hadoop

• https://github.com/big-data-europe/docker-hadoop
• https://docs.docker.com/engine/swarm/

10/4

Garbage Collection

Docker Swarm (and Kubernetes if time)

• https://docs.docker.com/engine/swarm/
• Complete tutorial and deploy a docker swarm that has multiple nodes of nginx.
• Deploy a docker container that has dask. Create two workers and one scheduler in the swarm.

10/11

No class (hurricane)

10/18

Quiz over Garbage Collection

MXNet

MxNet Tutorials

10/25

Client Day (Javascript, HTML, AJAX)

AJAX Tutorial

JQuery

Client-Server Sudoku game

Java Implementation

11/1

Client-Server Sudoku game

Server Day (Python Flask)

11/8

Find a scholarly article (Google Scholar is my suggestion for searching) on a distributed systems topic. It must be an article about distributed systems, and not just one that uses them.

You should have a working Flask application that you can send a curl POST request with JSON content representing a unfinished Sodoku board. Your server must then return a solution in JSON.

In class we will do the following (to be completed for homework):

• Modify the server so you immediately get a reference token back which you can use to query the server for your finished solution. We’ll want dask configured and working for this, so please make sure it is working for you with Docker Swarm. We will NOT have time in class to trouble shoot dask related problems.
• Test with CURL
• Build a quick and dirty JQuery powered front end in place of CURL.

11/15

8-puzzle AI parallel solver Day

11/22

Thanksgiving!

11/29

Presentation day and last day of class