View profile

SF Data Weekly - SalesLoft's Data Warehouse, Data Science at Faire, Docker & Postgres, CI/CD Explained

Revue
 
The analytics world is heating up. Even more so than before. In the past two weeks, I've had conversa
 
October 8 · Issue #135 · View online
SF Data Weekly
The analytics world is heating up. Even more so than before.
In the past two weeks, I’ve had conversations with a number of companies that have built out sophisticated data warehouse stacks. Across my conversations, I heard three major trends:

  • It’s all about processing operational workloads in the warehouse, not just delivering reports. Reporting is still one use case, but the companies that I’ve talked to have moved their business critical (data) processes into their cloud warehouse. That coincides with the rise of tools like dbt, AtScale and Dataform (the new kid on the block). Lots of companies thinking about shifting their Looker PDTs into their “semantic layer” or “modeling layer”.

  • There’s a new focus on cost management. “Our data infrastructure spend can’t grow faster than the business, and in the last few year it has. That’s not an option anymore.” is what I heard from pretty much every company. Data teams are facing true cost constraints now, which is why understanding at a granular level who are their most expensive users and queries has become so important. With the separation of storage compute, it has become easy to “just spin up a new model or query”, and that can become expensive. “Contain model sprawl” is another topic I heard, also in the light of the new importance of the modeling layer.

  • Data teams have a headcount limit as they are still considered a cost center despite the high value they deliver. When data, query and user volume are all on the rise, yet your team’s headcount is fixed and you can’t add more capacity to your data infrastructure, then the pressure is on to “deliver complete data science freedom” as one engineer put it. So lots of exploratory conversations around tooling for data teams to make data engineers and analysts more productive.
Do any of these three trends apply to your work and your company? Would love to hear about it, hit reply and share your story!
Until next week,
Lars

Our Pick
Gartner Reveals Five Major Trends Shaping the Evolution of Analytics and Business Intelligence
Data Pipelines
[sponsored] How to Optimize Data Pipelines Using Talend and intermix.io on Amazon Redshift [sponsored] How to Optimize Data Pipelines Using Talend and intermix.io on Amazon Redshift
Developing a Data Warehouse in the Cloud for SaaS at SalesLoft
Let me automate that for you - GameChanger Tech Blog Let me automate that for you - GameChanger Tech Blog
A beginner’s guide to Docker — how to create your first Docker application
Data Storage
Top 5 reasons to go for a data lake architecture - JAXenter
Tricks for Postgres and Docker that will make your life easier Tricks for Postgres and Docker that will make your life easier
Data Analysis
Ranking to maximize economic value Ranking to maximize economic value
Three tables every analyst needs
Data Visualization
Interactive Data Visualization with Modern JavaScript and D3
Data-driven Products
CI/CD: Continuous Integration & Delivery Explained - Semaphore CI/CD: Continuous Integration & Delivery Explained - Semaphore
Data Engineering Jobs
Did you enjoy this issue?
If you don't want these updates anymore, please unsubscribe here
If you were forwarded this newsletter and you like it, you can subscribe here
Powered by Revue
650 California St., San Francisco, CA 94108