Apr 30, 2023
Hi @c.bugra.alkan,
You would want to create a read stream from your big big data source and then split that super big file into chunks, maybe output to s3 and ingest into BigQuery from there.
Examples how to do so:
1. https://towardsdatascience.com/mysql-data-connector-for-your-data-warehouse-solution-db0d338b782d