https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/Recent content in BigQuery onHugo -- gohugo.ioSun, 11 Dec 2022 19:37:55 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_connection/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_connection/Prerequisite # For the BigQuery Data Sources to run successfully in Gathr applications, the following permissions/policy actions should be assigned in Google Cloud: bigquery.readsessions.create bigquery.readsessions.getData bigquery.tables.get bigquery.tables.getData bigquery.jobs.create bigquery.datasets.create For the BigQuery Emitters to run successfully in Gathr applications, the following permissions/policy actions should be assigned in Google Cloud: bigquery.tables.get bigquery.tables.create bigquery.tables.list bigquery.tables.updateData storage.buckets.create storage.buckets.get storage.buckets.list storage.objects.create storage.objects.delete storage.objects.get storage.objects.list bigquery.datasets.get bigquery.jobs.create Connection Configuration # Configure the BigQuery connection by providing the details for each field as described below:https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_ingestion_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_ingestion_source/BigQuery data source allows you to read data from the BigQuery data warehouse. Data Source Configuration # Fetch From Source/Upload Data File For designing the application, you can either fetch the sample data from the BigQuery source by providing the data source connection details or upload a sample data file in one of the supported formats to see the schema details during the application design. If Upload Data File is selected to fetch sample data, provide the below details.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_etl_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_etl_source/BigQuery data source allows you to read data from the BigQuery data warehouse. Schema Type # See the topic Provide Schema for ETL Source → to know how schema details can be provided for data sources. After providing schema type details, the next step is to configure the data source. Data Source Configuration # Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for GCS Batch earlier.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_incremental_configuration/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_incremental_configuration/Optional Step: You can choose to enable incremental read. This is useful if the need is only to read the objects that have been updated in the source, since the last execution of the application for the given configuration. Provide the details for enabling incremental read as described below: Column: Select a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp, decimal type of values.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_ingestion_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_ingestion_target/The BigQuery emitter allows you to push data in BigQuery data warehouse. Target Configuration # Save as Data Asset: Select checkbox to save the schema as a data asset in Gathr. Data Asset Name: Provide a name for the data asset to be saved. Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for GCS earlier.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_etl_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/bigquery/bigquery_etl_target/The BigQuery emitter allows you to push data in BigQuery data warehouse. Target Configuration # Save as Data Asset: Select checkbox to save the schema as a data asset in Gathr. Data Asset Name: Provide a name for the data asset to be saved. Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for GCS earlier.