![Plumbers of Data Science](https://s3-us-west-2.amazonaws.com/anchor-generated-image-bank/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
Plumbers of Data Science
By Andreas Kretz
We talk about interesting Data Engineering trends and topics. I also train Data Engineering in my Data Engineering Academy at LearnDataEngineering.com
![Currently playing episode](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
Plumbers of Data ScienceMar 09, 2018
![#90 Taylor McGrath - The Future of the Modern Data Stack](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo/160885/160885-1581580069740-85f36fdcedb77.jpg)
#90 Taylor McGrath - The Future of the Modern Data Stack
Super happy to have Taylor with me on this stream. She is the VP of Data Labs at Rivery and therefore has a lot of experience with data platforms. We'll talk about the modern data stack and where it's going. I'm excited to hear her experience about the changes that are happening in the data space, and what that means for data engineers & data teams.
![#89 Piyush Sachdeva - Getting Into Google After Eight Rejections from Amazon!](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo/160885/160885-1581580069740-85f36fdcedb77.jpg)
#89 Piyush Sachdeva - Getting Into Google After Eight Rejections from Amazon!
In this video I talk to Piyush who's an engineer at Google and has his own YouTube channel: "Tech Tutorials with Piyush". He's a really good guy and I love how he's dedicated to teaching engineering. We are talking about some awesome topics like:
- Is Linkedin a must for getting a job?
- Tips for recording yourself
- Cloud Engineering vs Data Engineering
- Which Cloud Platform should you choose right now?
- The amazing Google work culture explained
- Everybody should learn how to use Kubernetes
- How getting rejected over and over at Amazon got him into Google
- The hiring process at Google
Have fun!
You can also check this on out on YouTube: https://youtu.be/FZemVaqQcnM
If you want to get into Data Engineering check out my Academy at https://learndataengineering.com
![#88 - Wouter Trappers - How to Realize a Data Strategy Like a Pro!](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#88 - Wouter Trappers - How to Realize a Data Strategy Like a Pro!
I have seen people doing that wrong a few times. Luckily Wouter Trappers who is helping companies as a professional can help. We talked about The steps you need to take from value proposition to dashboards. Wouter is really knowledgeable and it was super fun talking with him and hearing his approach.
![#87 - Dhruba Borthakur - From Hadoop to real time analytics](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#87 - Dhruba Borthakur - From Hadoop to real time analytics
Dhruba Borthakur is CTO at Rockset and a passionate Data Engineer. Before co-founding Rockset he played a big role in development of Hadoop HDFS at Yahoo as well as HBase and RocksDB at Facebook. His current project is the serverless Rockset platform where you can gain real time analytics insight into your data. I tried it out before our talk and really liked it.
![#86 The Ultimate Data Engineering Introduction](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#86 The Ultimate Data Engineering Introduction
The Podcast is back!!!! I promise I am going to keep it up to date this time ;)
In this episode I talk about my newest Data Engineering course. I think it's the ultimate 1 hour 15 minutes introduction to Data Engineering.
There were also a ton of questions from the chat that I answered. Think you really enjoy this.
![#085 Big Data and Data Science Landscape plus trying to read Tweets with Nifi](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#085 Big Data and Data Science Landscape plus trying to read Tweets with Nifi
We are looking into the network communication protocol map. I first saw this like 10 years ago and its awesome.
Then we check out the Big Data and Data Science Landscape image. It shows you all the tools available to do data science, machine learning and data engineering. Which is very helpful if you are researching for tools to use.
Before using the Twitter API you got to create a developer account. So, I show you how I created one. After that I tried to get Nifi to download Tweets but it is not working.
![#084 Behind the scenes: Audio podcast, free transcriptions and GitHub](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#084 Behind the scenes: Audio podcast, free transcriptions and GitHub
Today's podcast is a bit of a behind the scenes.
What it takes to do a audio podcast. How you can get audio to text transcriptions for free.
.Also Github questions on how to work with branches on the Cookbook
![#083 Data Engineering at OLX Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#083 Data Engineering at OLX Case Study
Today a case study about OLX with a guest it was super fun!
Here are the slides Alexeyand I talked about:
https://www.slideshare.net/mobile/AlexeyGrigorev/image-models-infrastructure-at-olx
![#082 Reading Tweets With Apache Nifi & IaaS vs PaaS vs SaaS](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#082 Reading Tweets With Apache Nifi & IaaS vs PaaS vs SaaS
In this episode we install the Nifi docker container and look into how we can extract the twitter data.
We are also talking about the differences between infrastructure as a service, platform as a service and application as a service.
![#081 How to get tweets from the Twitter API](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#081 How to get tweets from the Twitter API
In this episode we look into the Twitter API documentation, which I love by the way.
How can we get old tweets for a certain hashtags and how to get current live tweets for these hashtags.
![#080 How To Find A Job In Germany & Answering Mails](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#080 How To Find A Job In Germany & Answering Mails
Tips on how you find a job in Germany and two super interesting mails.
![#079 Trying to stay true to myself and making the cookbook public on GitHub](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#079 Trying to stay true to myself and making the cookbook public on GitHub
The cookbook my Youtube, it will be for free, forever! Check out the data engineering cookbook on GitHub:
![#078 Cookbook collaboration and updates](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#078 Cookbook collaboration and updates
Updates of the cookbook and how to collaborate on it
![#077 Lambda and Kappa Architecture](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#077 Lambda and Kappa Architecture
In this episode we talk about the lambda architecture with stream and batch processing as well as a alternative the Kappa Architecture that consists only of streaming. Also Data engineer vs data scientist and we discuss Andrew Ng's AI Transformation Playbook
![#076 Cloud vs On Premise How To Decide](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#076 Cloud vs On Premise How To Decide
How do you choose between Cloud vs On-Premise, pros and cons and what you have to think about. Because there are good reasons to not go cloud.
Also thoughts on how to choose between the cloud providers by just comparing instance prices. Otherwise the comparison will drive you insane.
![#075 Creating the Course Structure For My Data Engineering Course](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#075 Creating the Course Structure For My Data Engineering Course
In this episode we go over the ideas I have for the data engineering course structure. It was your chance for you to influence what we put in there.
![#074 Starting My Data Engineering Online Course](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#074 Starting My Data Engineering Online Course
In this video we go over some of the 100+ comments I received on LinkedIn about a data engineering training.
![#073 Data Engineering At LinkedIn Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#073 Data Engineering At LinkedIn Case Study
Let's check out how LinkedIn is processing data
![#072 Data Engineering At Twitter Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#072 Data Engineering At Twitter Case Study
How is Twitter doing Data Engineering? Oh man, they have a lot of cool things to share these tweets.
![#071 Data Engineering At Spotify Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#071 Data Engineering At Spotify Case Study
In this episode we are looking at the data engineering at Spotify, my favorite music streaming service. How do they process all that data?
![#070 The Engineering Culture At Spotify](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#070 The Engineering Culture At Spotify
In this podcast we look at the engineering culture at Spotify, my favorite music streaming service.
The process behind the development of Spotify is really awesome.
![#069 Data Engineering At Pinterest Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#069 Data Engineering At Pinterest Case Study
A look into how Pinterest is doing data engineering.
![#068 A Budget Data Science PC Build](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#068 A Budget Data Science PC Build
Configuring a sub 1000 dollar PC for data engineering and machine learning
Link to the builds:
900$ build: https://pcpartpicker.com/list/22ThcY
1500$ build: https://pcpartpicker.com/list/hXJdV6
![#067 Data Engineering At NASA Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#067 Data Engineering At NASA Case Study
A look into how NASA is doing data engineering.
![#066 How To Do Data Science From A Data Engineers Perspective](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#066 How To Do Data Science From A Data Engineers Perspective
A simple introduction how to do data science in the context of the internet of things.
![#065 Data Engineering At CERN Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#065 Data Engineering At CERN Case Study
A look into how CERN is doing Data Engineering. They get huge amounts of data from the Large Hydron Colider. Let's check it out.
![#064 Data Engineering At Booking.com Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#064 Data Engineering At Booking.com Case Study
A look into how booking.com is doing data engineering.
![#063 Data Engineering At Airbnb Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#063 Data Engineering At Airbnb Case Study
A look into how Airbnb is doing Data Engineering.
![#062 Data Engineering At Netflix Case Study](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#062 Data Engineering At Netflix Case Study
How Netflix is doing Data Engineering using their Keystone platform
![#061 Reworking My Cookbook For Data Engineering](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#061 Reworking My Cookbook For Data Engineering
I decided to rework the cookbook focusing more on case studies and less on explaining tools.
People keep asking me for a path to become a data engineer and, let's be honest, you will never achieve that with just knowledge of the tools.
Finding out how companies do data engineering on their data science platforms is way more useful.
Over the next weeks we will go over each study on my YouTube channel. The stuff we talk about will then go into the cookbook too.
![#060 What Is Hadoop And Is Hadoop Still Relevant In 2019?](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#060 What Is Hadoop And Is Hadoop Still Relevant In 2019?
A Introduction into Hadoop HDFS, YARN and MapReduce.
Yes, Hadoop is still relevant in 2019 even if you look into serverless tools.
![#059 A Look Into The Siemens Mindsphere IoT Platform? | #059](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#059 A Look Into The Siemens Mindsphere IoT Platform? | #059
The Internet of things is a huge deal. There are many platforms available. But, which one is actually good?
Join me on a 50 minute dive into the Siemens Mindsphere online documentation.
I have to say I was super unimpressed by what I found.
Many limitations, unclear architecture and no pricing available?
Not good!
![#058 Guitars And Data Live Stream](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#058 Guitars And Data Live Stream
A stream full of mediocre guitar playing and great Q&A about Hadoop.
![#057 Introducing The Plumbers Medium Publication](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#057 Introducing The Plumbers Medium Publication
I have created a Medium Publication especially for us Plumbers of Data Science who work in Data Engineering and Big Data.
It's called, you guessed it, Plumbers of Data Science.
![#056 NoSQL Key Value Stores Explained With HBase](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#056 NoSQL Key Value Stores Explained With HBase
What is the difference between SQL and NoSQL?
In this episode I show you on the example of HBase how a key/value store works.
![#055 Data Warehouse vs Data Lake](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#055 Data Warehouse vs Data Lake
On this podcast I talk about data warehouses and data lakes.
When do people use which? What are the pros and cons of both?
Architecture examples for both and does it make sense to completely move to a data lake?
![#054 How to Market Yourself in 2019 Student or Professional](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#054 How to Market Yourself in 2019 Student or Professional
In this episode I talk about how you can gain a competitive edge on the job market. It's super simple, you can and should start with it TODAY by putting yourself out there.
![#053 The Data Science Depression Is Coming? What You Can Do](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#053 The Data Science Depression Is Coming? What You Can Do
The Data Science Hype is still strong. Where's the industry going, towards a cliff? Here's what can you do?
![#052 Data Engineering Cookbook Live Stream](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#052 Data Engineering Cookbook Live Stream
In this episode I show you the first version of my data engineering cookbook.
![#051 Five Books To Buy As A Data Engineer & My Book Buying Strategy](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#051 Five Books To Buy As A Data Engineer & My Book Buying Strategy
Getting a book and reading it cover to cover is useless. In this episode I show you my strategy of buying books complimentary to your work. And 5 great books I read over the years that helped me get where I am now.
![#050 Data Engineer Scientist or Analyst Which One Is For You?](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#050 Data Engineer Scientist or Analyst Which One Is For You?
In this podcast we talk about the differences between data scientists, analysts and engineers. Which are the three main data science jobs.
All three super important.
![#049 I Found A REAL Use For Blockchain, At Least I thought So](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#049 I Found A REAL Use For Blockchain, At Least I thought So
After all the BS solutions using Blockchain I thought I finally found one that makes sense. Of all the possibilities it's the EU data protection law GDPR. Well, one problem I overlooked in this podcast is, that it is impossible to delete data after it is in the chain. That's however a rule for GDPR.
So, I was wrong. Again :D
![#048 From Wannabe Data Scientist To Engineer My Journey](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#048 From Wannabe Data Scientist To Engineer My Journey
In this episode Kate Strachnyi interviews me for her humans of data science podcast. We talk about how I found out that I am more into the engineering part of data science.
![#047 The Truth About Data Science Salary For Graduates](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#047 The Truth About Data Science Salary For Graduates
In this episode I show you how much data science graduates are actually payed in Germany.
All over the internet you can find that Data Science salary is over 100k Dollars. Data Engineer or Data Scientist. It's way lower then that.
Then I give you a few really good tips on how to choose the right company to work for. Huge corporation, startup or small company? Here's how to choose.
![#046 How To Use GitHub for LaTeX Version Control](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#046 How To Use GitHub for LaTeX Version Control
In this podcast I am showing you how I use GitHub to write my Data Engineering Cookbook with LaTex.
![#045 Why I Use LaTeX to Write Professionally And You Should Too](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#045 Why I Use LaTeX to Write Professionally And You Should Too
In today's video I show you why I decided to use LaTeX to write my data engineering cookbook.
I used it before for my diploma thesis and I am in love again :)
Here's the link to the cheatsheet:
wch.github.io/latexsheet/latexsheet.pdf
Check out my Patreon for the Data Engineering Cookbook:
bit.ly/PatreonAndreasKretz
Music:
"Day One" by Declan DP
soundcloud.com/declandp
Attribution 3.0 Unported
creativecommons.org/licenses/by/3.0/
![#044 How to Increase Your Chances for Internships or a Full-time Job](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#044 How to Increase Your Chances for Internships or a Full-time Job
Sharing your ideas and knowledge will increase your chances!
Here's how you can do that.
Music:
"Day One" by Declan DP
soundcloud.com/declandp
Attribution 3.0 Unported
creativecommons.org/licenses/by/3.0/
![#041 Agile Development Is Important But Please Don't Do Scrum](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#041 Agile Development Is Important But Please Don't Do Scrum
![#040 Huge Big Data News! Cloudera and Hortonworks Merge](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)
#040 Huge Big Data News! Cloudera and Hortonworks Merge
If you are interested in the whole hadoop ecosystem you need to check out this episode. You won't regret it ;)
![#039 Is ETL Dead For Data Science and Big Data?](https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo400/160885/160885-1581580067805-f753f20a214ae.jpg)