How to run BigQuery jobs programmatically using AWS Lambda and save results to S3 as CSV.

Photo by Filiberto Santillán on Unsplash

Have you heard about Google BigQuery?

Have you ever wondered how to automate things and query it programmatically from AWS using Lambdas?

In this 20 minute tutorial, we’ll walk through building a Node.js Lambda function in AWS to run BigQuery jobs and save the result to AWS S3 bucket as CSV file.

Prerequisites:

• AWS account. If you don’t have it just create it now. It’s Free. How to create AWS account
• AWS cli (Command Line Interface). You will need Python first. Then follow this article to install cli if you don’t have it yet.
• Google developer account.
• Google Command Line Interface. Follow this article to install it: https://cloud.google.com/sdk/ .
• Finally, we’ll need Node.js v8.6 or above to run our core application.

If you don’t have any of these just create it. It’s Free.

Before starting, let’s think about what we need to do here. We will create a lambda function which does the following:

• Authenticates with Google Cloud Platform using service account credentials.
• Our Lambda will create a new query job in BigQuery.