View profile

SF Data Weekly - Facebook's Scribe, Small-data Engineering, DynamoDB vs. Hadoop vs. MongoDB

Revue
 
This week I had an email conversation with an engineer at a large social network, and we were discuss
 
October 15 · Issue #136 · View online
SF Data Weekly
This week I had an email conversation with an engineer at a large social network, and we were discussing how the role of the data engineer is changing.
tl;dr
The “write ETL all day” or “optimize the db for speed” roles are being eaten by hosted services, who can do things once and take advantage of expertise and scale to beat someone trying to do it by hand.
So what does that do to the role of the data engineer? One scenario is commoditization of the skillset, driving salaries down. But I don’t think that’s going to happen.
Instead, I think the role of the data engineer will comprise:

  • control and oversight of data sources
  • guardians of the company’s data architecture
  • building and maintaining models
  • test their code behind the models
  • cost control (because analysts write bad SQL)
  • vendor management (for ETL, workflows, monitoring)

☕️☕️☕️
If you have an opinion on that topic - maybe you want to join our Community Channel on Slack. We’re still in the process of gated on-boarding, so hit reply if you want to be part of the next 20 to join!
☕️☕️☕️

Lars

Our Pick
A Reference Guide for FinTech & Small-data Engineering A Reference Guide for FinTech & Small-data Engineering
Data Pipelines
Scribe: Transporting petabytes per hour - Facebook Engineering Scribe: Transporting petabytes per hour - Facebook Engineering
Samza 1.0: Stream Processing at Massive Scale | LinkedIn Engineering
Data Storage
[sponsored] How Mode and intermix.io Deliver Fast Queries on Amazon Redshift [sponsored] How Mode and intermix.io Deliver Fast Queries on Amazon Redshift
DataLake DataLake
DynamoDB vs. Hadoop vs. MongoDB
Data Analysis
Twitter data analysis in R | R-bloggers Twitter data analysis in R | R-bloggers
Data Visualization
The Best 9 Free and Open Source Data Visualization Software The Best 9 Free and Open Source Data Visualization Software
Data-driven Products
The Complete Kubernetes Collection [Tutorials and Tools] The Complete Kubernetes Collection [Tutorials and Tools]
Data Engineering Jobs
Did you enjoy this issue?
If you don't want these updates anymore, please unsubscribe here
If you were forwarded this newsletter and you like it, you can subscribe here
Powered by Revue
650 California St., San Francisco, CA 94108