https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/Recent content in GCS onHugo -- gohugo.ioSun, 11 Dec 2022 19:37:55 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_connection/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_connection/Prerequisite # For the GCS Data Sources and Emitters to run successfully in Gathr applications, the following permissions/policy actions should be assigned in Google Cloud: storage.buckets.get storage.buckets.create storage.buckets.list storage.objects.create storage.objects.delete storage.objects.get storage.objects.list Connection Configuration # Configure the GCS Batch connection by providing the details for each field as described below: Connection Name: Name of the connection to be created. Scope: Define the connection scope to customize their accessibility. Organization: Accessible to organization users across all Gathr projects for usage in applications.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_ingestion_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_ingestion_source/Gathr provides GCS (Google Cloud Storage) channel. Permissions Required in GCS: Cloud storage permissionshould be there to access and read data from Buckets of GCS data source. Data Source Configuration # Fetch From Source/Upload Data File For designing the application, you can either fetch the sample data from the GCS source by providing the data source connection details or upload a sample data file in one of the supported formats to see the schema details during the application design.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_batch_etl_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_batch_etl_source/Gathr provides GCS (Google Cloud Storage) Batch channel. Permissions Required in GCS # The user should have cloud storage permission to access and read data from Buckets of GCS Batch data source. Schema Type # See the topic Provide Schema for ETL Source → to know how schema details can be provided for data sources. Data Source Configuration # After providing schema type details, the next step is to configure the data source.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_streaming_etl_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_streaming_etl_source/Gathr provides GCS (Google Cloud Storage) Streaming channel. Permissions Required in GCS # The user should have cloud storage permission to access and read data from Buckets of GCS Batch data source. Schema Type # See the topic Provide Schema for ETL Source → to know how schema details can be provided for data sources. Data Source Configuration # After providing schema type details, the next step is to configure the data source.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_incremental_configuration/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_incremental_configuration/Enable Incremental Read: Option to read the objects updated in the source since the last execution of the application for the given configuration. Provide the details for enabling incremental read as described below: Read By: Choose the incremental read condition to be applied and data to be read as per file modification time or column partition. If File Modification Time is chosen, provide the offset value. Offset: Specifies the last modified time of a file.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_ingestion_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_ingestion_target/Gathr provides GCS (Google Cloud Storage) emitter. Target Configuration # Save as Data Asset: Select checkbox to save the schema as a data asset in Gathr. Data Asset Name: Provide a name for the data asset to be saved. Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for GCS earlier. Or create one as explained in the topic - GCS Connection →https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_etl_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/gcs/gcs_etl_target/Gathr provides GCS (Google Cloud Storage) emitter. Target Configuration # Save as Data Asset: Select checkbox to save the schema as a data asset in Gathr. Data Asset Name: Provide a name for the data asset to be saved. Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for GCS earlier. Or create one as explained in the topic - GCS Connection →