https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/Recent content in ADLS onHugo -- gohugo.ioSun, 11 Dec 2022 19:37:55 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_connection/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_connection/ADLS, or Azure Data Lake Storage, is a scalable and secure data lake solution in Microsoft Azure designed for big data analytics and storage needs. Connection Configuration # Configure the fields required to create the connection as explained below. Connection Name # Name of the connection to be created. Scope # Define the connection scope to customize their accessibility. Organization: Accessible to organization users across all Gathr projects for usage in applications.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_ingestion_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_ingestion_source/On a ADLS Batch Channel you will be able to read data from specified ADLS Container. Data Source Configuration # Configure the data source parameters that are explained below. Fetch From Source/Upload Data File # For designing the application, you can either fetch the sample data from the Amazon S3 source by providing the data source connection details or upload a sample data file in one of the supported formats to see the schema details during the application design.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_batch_etl_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_batch_etl_source/On a ADLS Batch Channel you will be able to read data from specified ADLS Container. Schema Type # See the topic Provide Schema for ETL Source → to know how schema details can be provided for data sources. After providing schema type details, the next step is to configure the data source. Data Source Configuration # Configure the data source parameters that are explained below. Connection Name # Connections are the service identifiers.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_streaming_etl_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_streaming_etl_source/ADLS Streaming data source reads the objects from the ADLS Container. Schema Type # See the topic Provide Schema for ETL Source → to know how schema details can be provided for data sources. After providing schema type details, the next step is to configure the data source. Data Source Configuration # Configure the data source parameters that are explained below. Connection Name # Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for ADLS earlier.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_incremental_configuration/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_incremental_configuration/Enable Incremental Read # Option to read the objects updated in the source since the last execution of the application for the given configuration. Provide the details for enabling incremental read as described below: Read By # Choose the incremental read condition to be applied and data to be read as per File Modification Time or Column Partition. If File Modification Time is chosen, provide the offset value. Offset # Specifies the last modified time of a file.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_ingestion_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_ingestion_target/Target Configuration # Configure the target parameters that are explained below. Connection Name # Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for ADLS earlier. Or create one as explained in the topic - ADLS Connection → Container # ADLS container name in which the transformed data should be emitted. Path # Provide Azure Storage/Databricks Mount Point directory path based upon the chosen Access Option.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_etl_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/adls/adls_etl_target/Target Configuration # Configure the target parameters that are explained below. Access Options # 👉 This field appears with a streaming data source. Access Option specifies the method for accessing and interacting with ADLS. Azure Storage enables integration with Azure’s cloud-based storage solution to interact with the file system of ADLS. Connection Name # Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for ADLS earlier.