and Data warehouse Gitflow pipelines to run it automatically — Do you unit test your data warehouse scripts? I am going to talk about unit tests for complex SQL queries which might consist of multiple operations (actions). I tried it before using BigQuery scripting: SQL Unit Testing in BigQuery? Here is a tutorial. Complete guide for scripting and UDF testing.towardsdatascience.com However, here is another (and probably better) way to do it.

Complete guide using Tensorflow, Airflow scheduler and Docker — Google AI Platform allows advanced model training using various environments. So it is really easy to train your model with just one command like so: gcloud ai-platform jobs submit training ${JOB_NAME} \ --region $REGION \ --scale-tier=CUSTOM \ --job-dir ${BUCKET}/jobs/${JOB_NAME} \ --module-name trainer.task \ --package-path trainer \ --config trainer/config/config_train.json …

Complete Data Studio guide and BigQuery tutorial for Firebase users, Machine Learning enthusiasts and Marketers. All you wanted to know. Data Studio template included. — Have you ever wondered how to reduce user churn and save money spent on user acquisition? This article is about how to count those users who stay in your App in order to understand what makes them stay. Who is this article for? Marketers who have been tasked to create custom user retention dashboards.

Infrastructure as Code for database practitioners — With these two simple Cloudformation templates, we will learn how to deploy Postgres and MySQL Aurora database instances with just one command CLI command. I have optimised template files reducing the number of properties to a minimum so it’s easier to comprehend. I know that Infrastructure as Code might…

Parquet vs ORC vs AVRO vs JSON. Which one to choose and how to use them? — I’m a big fan of data warehouse (DWH) solutions with ELT-designed (Extract-Load-Transform) data pipelines. However, at some point, I faced the requirement to process raw event data in Cloud Storage and had to choose the file format for data files. This is a typical scenario when machine learning engineers are…

Beginners guide: Why unit and integration tests are so important for your data platform — This story is for those who would like to learn how to code and how to run tests, automate CI/CD checks and run them in any environment including locally. Unit testing is an essential must-have skill for machine learning engineers these days. …

How well does it answer your business needs? Dilemma of a choice. — It is easy to get lost with the abundance of data tools available in the market right now. The Internet is filled with opinionated stories (often speculative) about which data tools to use and how to make our Data Stack modern in this particular year. Which data tools are the…

Data modeling, Python, DAGs, Big Data file formats, costs… It covers everything — This is a real-life scenario when I was tasked to create a highly scalable machine learning pipeline with raw event data sent from the mobile application. The story offers a set of advanced techniques that might be useful for interview preparation. Learn how to work with raw data, transform it…

A couple of tricks to speed up Your Data Warehousing — Wildcard tables in BigQuery are considered somewhat legacy but still in use by many services and third-party applications. I will use a small public dataset to show a few tricks In BigQuery scripting table names can’t be used as parameters, at least for now. It makes it extremely difficult to…