Focuing on

• How to study data engineering?
• How to become a data engineer?
• Let’s look at the big picture!

---

Lectures

• Data Flair - Learn Data Engineering

---

Useful References

• https://github.com/igorbarinov/awesome-data-engineering
• https://github.com/meirwah/awesome-workflow-engines
• https://github.com/manuzhang/awesome-streaming
• Docker vs Kubernetes vs Apache Mesos

---

IaaS Platform

Openstack

• What is OpenStack
• https://www.openstack.org/

Amazone EC2

• Proprietary
• https://aws.amazon.com/ko/ec2/

Google Compute Engine

• Proprietary
• https://cloud.google.com/compute/?hl=ko

Azure IaaS

• Proprietary
• https://azure.microsoft.com/ko-kr/overview/what-is-azure/iaas/

DigitalOcean

• Proprietary
• https://www.digitalocean.com/products/droplets/

---

PaaS Platoform

Cloud Foundry

• https://www.cloudfoundry.org/

Amazon Lamda

• Proprietary
• https://aws.amazon.com/ko/lambda/

Google App Engine

• Proprietary
• https://cloud.google.com/appengine/docs/

Microsoft App Service

• Proprietary
• https://azure.microsoft.com/en-us/services/app-service/?v=18.51

---

Linux OS

CoreOS

• A lightweight Linux operating system designed for clustered deployments
• https://github.com/coreos/

---

Distributed Platform

Openshift

• Container support
• A container application platform that brings Docker and Kubernetes to the enterprise.
• https://github.com/openshift

DC/OS

• Container + Non-container support
• Chronos + Marathon + Aurora, etc. based on Mesos.
• Premium paid model for extra features
• https://dcos.io/

---

Distributed Framework

Mesos

• Support both containerized and non-containerized (e.g. analytics and stateful data services) workloads
• Even can run kubernetes on it
• A distributed systems kernel (Framework)
• http://mesos.apache.org/

---

Container Framework

Docker

• Provide operatiing-system-level virtualization
• https://www.docker.com/

LXC

• Operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel
• https://linuxcontainers.org/ko/lxc/introduction/

---

Container Orchestration

Kubernetes

• Open-source system for automating deployment, scaling, and management of containerized applications.
• https://kubernetes.io/

Marathon

• Apache Mesos Framework for Docker orchestration
• https://mesosphere.github.io/marathon/

Docker Swarm

• Native clustering for Docker. It turns a pool of Docker hosts into a single, virtual host.
• https://github.com/docker/swarm

Amazon ECS

• Amazon Elastic Container Service
• https://aws.amazon.com/ko/ecs/

HashiCorp Nomad

• https://www.nomadproject.io/

Metronome

• Apache Mesos framework for scheduled jobs.
• https://github.com/dcos/metronome

Chronos

• Apache Mesos framework for cron job orchestration.
• https://github.com/mesos/chronos

Aurora

• Apache Mesos framework for long-running services and cron jobs
• http://aurora.apache.org/

---

Data warehouse (Datalake)

Hadoop Ecosystem

• Study
• hadoop
• Monitoring
  • Ambari
• Database
  • HBase
• Processing
  • Map Reduce
  • hive
  • Pig
• Execution Engine
  • Tez
• Resource Manager
  • Yarn
• FileSystem
  • HDFS

Amazon Redshift

• https://aws.amazon.com/ko/redshift/

Snowflake

• https://www.snowflake.com/

---

Security

Kerberos

• https://web.mit.edu/kerberos/

LDAP

• https://tools.ietf.org/rfc/rfc4511.txt

Knox

• https://knox.apache.org/

Ranger

• https://ranger.apache.org/

---

Data Companies

Amazone AWS

• https://aws.amazon.com
• Products

Google Cloud

• https://cloud.google.com/
• Products

Microsoft Azure

• https://azure.microsoft.com
• Products

Alibaba Cloud

• https://www.alibabacloud.com

IBM Cloud

• https://www.ibm.com/cloud/

SAP Cloud

• https://cloudplatform.sap.com/index.html

Oracle Cloud

• https://cloud.oracle.com/home

Red Hat Cloud suite

• https://www.redhat.com/ko/technologies/cloud-computing/cloud-suite

Apache Cloud Stack

• https://cloudstack.apache.org/

Cloudera

• https://www.cloudera.com/

Hotornworks

• https://hortonworks.com/

Databricks

• https://databricks.com/

Teradata

• https://www.teradata.com/

Snowflake

• https://www.snowflake.com/

---

Workflow

Airflow

• Study
• https://airflow.apache.org/

Azkaban

• https://azkaban.github.io/

Oozie

• http://oozie.apache.org/

Luigi

• https://github.com/spotify/luigi

Dkron

• https://dkron.io/

Nomad

• https://www.nomadproject.io/

Data Ingestion

Kafka

• http://kafka.apache.org/

RabbitMQ

• https://www.rabbitmq.com/

Sqoop

• http://sqoop.apache.org/

---

Data Stream Processing

Flink

• https://flink.apache.org/

Spark Streaming

• https://spark.apache.org/streaming/

---

Data Batch Processing

Mapreduce

• http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

Hive

• Study
• https://hive.apache.org/

Spark

• https://spark.apache.org/
• https://www.datacamp.com/community/tutorials/apache-spark-python

Presto

• https://prestodb.io/

Storm

• http://storm.apache.org

Impala

• https://impala.apache.org

Pig

• https://pig.apache.org/

Samza

• http://samza.apache.org

Imply

• Powered by Druid
• https://imply.io/

Mahout

• http://mahout.apache.org/

HAWQ

• http://hawq.apache.org/

Drill

• https://drill.apache.org/

Kylin

• http://kylin.apache.org

---

Databases

MySQL

• https://www.mysql.com/

Mongo DB

• https://docs.mongodb.com/manual/mongo/

ElasticSearch

• https://www.elastic.co/

Redis

• http://redis.io/

HBase

• https://hbase.apache.org/

InfluxDB

• https://github.com/influxdata/influxdb

OpenTSDB

• https://github.com/OpenTSDB/opentsdb

Prometheus

• https://prometheus.io/

Cassandra

• http://cassandra.apache.org/

Riak

• https://github.com/basho/riak

Druid

• http://druid.io/

---

Business Intelligence (Analysis & Visualization)

Hue

• http://gethue.com/

Yanagishima

• https://github.com/yanagishima/yanagishima

Tableau

• https://www.tableau.com/

Kibana

• https://www.elastic.co/products/kibana

Grafana

• https://grafana.com/

Chronograf

• https://github.com/influxdata/chronograf

Superset

• https://superset.incubator.apache.org/

Metabase

• https://www.metabase.com/

redash

• https://redash.io/

Zeppelin

• https://zeppelin.apache.org/

IBM Cognos Analytics

• https://www.ibm.com/products/cognos-analytics

File System

HDFS

• Study
• https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html

---

Data Serialization Formats

ORC

• Study
• https://orc.apache.org/

Avro

• https://avro.apache.org/

JSON

• https://json.org/

BSON

• http://bsonspec.org/

Protocol Buffers

• https://developers.google.com/protocol-buffers/

Thrift

• https://thrift.apache.org/

YAML

• https://yaml.org/

XML

• https://www.w3.org/TR/REC-xml/

---

Log Collector

Flunetd

• https://www.fluentd.org/

Logstash

• https://www.elastic.co/products/logstash

Beats

• https://www.elastic.co/kr/products/beats

StatsD

collectd

dd-agent

Telegraf

logrotate

Graphite

Flume

---

Tracing

Zipkin

• https://zipkin.io/

---

Configuration Management

Ansible

• https://www.ansible.com/

Chef

• https://www.chef.io/chef/

Salt

• https://github.com/saltstack/salt

CFEngine

• https://github.com/cfengine/core

Puppet

• https://github.com/puppetlabs/puppet

Vagrant

• https://www.vagrantup.com/

Web Servers

Nginx

• https://nginx.org/en/
• http://nginx.org/en/docs/beginners_guide.html

Apache

• https://httpd.apache.org/

Tomcat

• https://tomcat.apache.org/

Jetty

• http://www.eclipse.org/jetty/

---

Distributed Synchronization Service

Zookeeper

• https://zookeeper.apache.org/

---

Uncategorized

Data Visualization libraries

• Apache Calcite https://calcite.apache.org/
• d3.js
• chart.js
• Leaflet.js
• Database.js
• Swiper.js
• vue.js
• AOS(animation on scroll)
• https://seaborn.pydata.org/
• https://matplotlib.org/index.html