and Data warehouse Gitflow pipelines to run it automatically — Do you unit test your data warehouse scripts? I am going to talk about unit tests for complex SQL queries which might consist of multiple operations (actions). I tried it before using BigQuery scripting: SQL Unit Testing in BigQuery? Here is a tutorial. Complete guide for scripting and UDF testing.towardsdatascience.com However, here is another (and probably better) way to do it.

Complete guide using Tensorflow, Airflow scheduler and Docker — Google AI Platform allows advanced model training using various environments. So it is really easy to train your model with just one command like so: gcloud ai-platform jobs submit training ${JOB_NAME} \ --region $REGION \ --scale-tier=CUSTOM \ --job-dir ${BUCKET}/jobs/${JOB_NAME} \ --module-name trainer.task \ --package-path trainer \ --config trainer/config/config_train.json …

Complete Data Studio guide and BigQuery tutorial for Firebase users, Machine Learning enthusiasts and Marketers. All you wanted to know. Data Studio template included. — Have you ever wondered how to reduce user churn and save money spent on user acquisition? This article is about how to count those users who stay in your App in order to understand what makes them stay. Who is this article for? Marketers who have been tasked to create custom user retention dashboards.

CI/CD for data engineers and ML Ops — What is a data environment? Data engineers split infrastructure resources into live and staging to create isolated areas (environments) where they can test ETL services and data pipelines before promoting them to production. Data environment refers to a set of applications and related physical infrastructure resources that enable data storage…

Magic Tricks to Work with Logging, AWS Lambda and Much More — In this story I will write about a few AWS CLI (Command Line Interface) tricks which saved me a lot of time and made my work more efficient. Anything we can do on the AWS website is also possible via the command line. It helps with documentation while writing `readme`…

A few Compelling Reasons to Choose Apache Iceberg for Data Lakes — Apache Iceberg: What is it? Apache Iceberg — is it a new data lake file format? A table format? Why is it so good? Time travel for data lakes? I will try to answer all these questions in this story. Transactionally consistent data lake tables with point-in-time snapshot isolation is…

Data pipeline management done right simplifies deployment and increases the availability and accessibility of data for analytics — What is data orchestration? DataOps teams use Data Pipeline Orchestration as a solution to centralize administration and oversight of end-to-end data pipelines. The process of automating the data pipeline is known as data orchestration. It is important to manage data pipelines right as it affects almost everything, i.e. …

Infrastructure as Code for database practitioners — With these two simple Cloudformation templates, we will learn how to deploy Postgres and MySQL Aurora database instances with just one command CLI command. I have optimised template files reducing the number of properties to a minimum so it’s easier to comprehend. I know that Infrastructure as Code might…

Parquet vs ORC vs AVRO vs JSON. Which one to choose and how to use them? — I’m a big fan of data warehouse (DWH) solutions with ELT-designed (Extract-Load-Transform) data pipelines. However, at some point, I faced the requirement to process raw event data in Cloud Storage and had to choose the file format for data files. This is a typical scenario when machine learning engineers are…