SF Data Weekly - LinkedIn's Gobblin, Redshift ETL, Microservice Madness, Data Science at Twitter, Uber Maps

Microservices became a very popular topic in over the last couple of years, but many don’t understand its costs and benefits. This article describes in detail what microservices are, their pros and cons, and key challenges.

Amazon Redshift is a fast, petabyte-scale data warehouse that enables you easily to make data-driven decisions. This post guides you through some of the best practices for ensuring optimal, consistent runtimes for your ETL processes - workload management, ad-hoc ETL, copy data from multiple sources, monitoring…

A tool produced a while ago, but still very useful for understanding different aspects and tools of a data pipeline from data ingestion, to processing to storing. A great place to pick the tools you need.

The extract, transform, load (ETL) processing of data, probably takes the largest portion of time in building machine learning models. This post gives a way to make your ETLs more maintainable and easier to write using a web tool - jinja2.

LinkedIn’s Gobblin evolved from a data ingestion framework for offline big data, to a distributed data integration framework for complete lifecycle management across both streaming and batch environments, by recently adding support for Helix clusters and Kafka streams. It’s actively used by Apple, Paypal, CERN etc.

Yes, mastering a 50-year-old programming language is the key to getting a data science job. These are the reasons why SQL is everywhere, and it’s not seeming to go away any time soon.

Data Science has changed a lot in the last few years at Twitter, and it also turns out to include a pretty big portion of data-related responsibilities, such as: product insights, data pipelines, statistical experimentation, machine learning etc. The author describes all these different aspects through his own experience. 

If you can’t bring your data to the your compute, bring your compute to the your data! The Google Cloud Datalab is a tool built on top of Jupyter notebook allowing easy integration with your BigQuery datasets and Google Cloud storage. Here’s how to get started with it.

This is how Uber crafts maps to visualize millions of data points, monitor road conditions, and advocate for policy change. The core issue to address is building a unified, consistent set of maps across various use cases.

Data visualization of the week - an example of a perfectly done interactive data chart! This time on historical marriage data across the United States.

The author of the post does a Monte Carlo simulation on the daily returns of the USD bitcoin price to try to know what will be its most likely price by the end of 2018.