Setup Airbyte, BigQuery, dbt, Metabase, and everything else you need to run a Modern Data Stack using Terraform.

A Modern Data Stack Architecture (image by author)

What is a Modern Data Stack

A Modern Data Stack (MDS) is a stack of technologies that makes a modern data warehouse perform 10–10000x better than a legacy data warehouse. Ultimately, an MDS saves time, money, and effort. The four pillars of an MDS are a data connector, a cloud data warehouse, a data transformer, and a BI & data exploration tool.

Easy integration is made possible with managed & open-source tools that pre-build hundreds of ready-to-use connectors. What used to take a team of data engineers to build and maintain regularly can now be replaced with a tool for simple use cases. …

The tale of a Data Analyst who evolves into an Analytics Engineer and resources so you can use to be like her.

Image credit

Analytics Engineer is a new position coined (and made possible) by dbt. If a Data Engineer (DE) marries a Data Analyst (DA) and they have a baby girl, that baby girl will be an Analytics Engineer (AE). Well, it does not work that way, but you get the point.

A Guide to staying focus in today’s distracted world—a review of Deep Work by Cal Newport.

Photo by Alexander Hafemann on Unsplash

We are living in a connected world, where the pace of innovation is astonishing. Never before do we have access to so much information and can easily connect to others within seconds. The fourth industrial revolution is here, and things will only get better. Or is it?

Nothing in life is absolute (well, except for Vodka). With more connectivity, we are becoming more distracted than ever. This is more true with knowledge workers than any other profession. …

What to know in terms of security when setting up a data environment in BigQuery.

Photo by Jason Dent on Unsplash

We have all seen and heard about data breaches and the damages, both in terms of financial and reputation, that they can inflict. As we rely more and more on our data to make better decisions, data is becoming a critical asset for any organization. So like any other asset of a company, controlling access to data is essential to protecting your data.

Fortunately, modern data warehouses have great functionalities built-in to control precisely who has access to what. This is often referred to as Identity and access management or IAM. …

Video games don’t deserve the bad raps they often get. Here are the ones that shaped me (literally and figuratively).

Photo by Carl Raw on Unsplash

The first video game ever created was all the way back in 1958. Since then, the global gaming market has grown tremendously and is worth USD 173.70 billion in 2020, according to Mordor Intelligence.

I remember playing my first ever video game back when I was 6. It was one of the most famous DOS games, Prince of Persia. I spent hours navigating the dungeon and the palace in the game. …

How to (and not to) deploy dbt to production.

Photo by Sam Loyd on Unsplash

With the rise of the Modern Data Stack, more and more people use dbt as the main tool for data transformations, aka data modeling. The folks at Fishtown create an amazing dbt Cloud offering that suits the needs of data teams, large and small. With dbt Cloud, any Analyst, seasoned or fresh, can easily start modeling and deploying data transformations pipelines to production.

I highly recommend checking out dbt Cloud since the product does much more than helping you deploy dbt. Firstly, let’s look at how you should not deploy dbt to production.

What not to do

There are many ways to deploy dbt…

My review of the book “How to take smart notes” by Sönke Ahrens and my journey towards building a Personal Knowledge Management system.

A Physical Zettlekasten — Image from Wikipedia

I am a bookworm, and over the years, I have read and logged nearly 200 books. However, I must admit that for many of the books I read, what little remained in my head are some general ideas of what the book is about. Surely, if I want to remind myself about a particular one, I can just read the book’s summary, which is readily available online nowadays. …

My 8 tips for people started using VSCode for the first time or switching from another IDE.

Photo by AltumCode on Unsplash

Getting your way around an IDE is a vital part, whether you are an engineer or an analyst. Watching a senior developer jumping around in Vim or VSCode can be intimidating for people with less familiarity. However, I assure you that there is no magic here. It is all practice and Googling answers when you got stuck.

Most of the tips I shared in this article will greatly improve your productivity. Once you learn some of these, you will have a hard time imagining life without them.

1. The master of all shortcuts

If there is one shortcut you take away from this post, it is…

How to get out of the email hell and take back your precious time.

I don’t need a system… I keep everything in my email. Photo by Eugenio Mazzone on Unsplash

Email is still the core mode of communication in our work and personal life for many of us. When I started with my career, dealing with email is not an issue since I did not get that many emails. However, they started coming in more and more, and I started wearing different hats in my life.

At one point, I started missing important tasks and forget details that I should have remembered. Our brain is terrible at keeping and organizing a lot of information. …

Running 50 commands to generate base models? Writing the same transforms for the 100 times for your base models? This package will streamline this process for you.

Photo by Lenny Kuhne on Unsplash

We love dbt, and we use it for almost all of our consulting projects. There are so many cool pre-made packages in the dbt hub that dramatically improves your productivity. For example, Fivetran has created pre-made starter models for many common sources such as Hubspot, Salesforce, Google Ads, and Facebook Ads. Fishtown Analytics also created some great starters packages, such as dbt-utils, audit-helper, and code-gen.

When starting a project with a new client, we often go through the same steps to set up a new dbt project. After integrating data from sources to a data warehouse, we use the code-gen…

Tuan Nguyen

CTO & Board member @Joon Solutions. Check out my website

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store