https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/Recent content in HDFS onHugo -- gohugo.ioSun, 11 Dec 2022 19:37:55 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_connection/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_connection/Connection Configuration # Connection Name: Name of the connection to be created. Scope: Define the connection scope to customize their accessibility. Organization: Accessible to organization users across all Gathr projects for usage in applications. Project: Accessible to organization users limited to projects that are specified by the connection owner for usage in applications. Private: Accessible only to the connection owner for usage in applications. Connection Type: Option to connect securely using SSH tunnel or with a direct connection.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_ingestion_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_ingestion_source/HDFS data source allows you to read data from HDFS storage. An Ingestion application with HDFS data source is supported to run on registered clusters and not on Gathr clusters. To know about how to register a cluster with Gathr by establishing PrivateLink, see Compute Setup → Data Source Configuration # Fetch From Source/Upload Data File For designing the application, you can either fetch the sample data from the HDFS source by providing the data source connection details or upload a sample data file in one of the supported formats to see the schema details during the application design.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_etl_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_etl_source/👉 This is a Batch component. An ETL application with HDFS data source is supported to run on registered clusters and not on Gathr clusters. To know about how to register a cluster with Gathr by establishing PrivateLink, see Compute Setup → Schema Type # See the topic Provide Schema for ETL Source → to know how schema details can be provided for data sources. After providing schema type details, the next step is to configure the data source.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_incremental_configuration/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_incremental_configuration/Enable Incremental Read: You can choose to enable incremental read. This is useful if the need is only to read the objects that have been updated in the source, since the last execution of the application for the given configuration. Provide the details for enabling incremental read as described below: Offset: Specifies last modified time of file - all the files whose last modified time is greater than this value will be read.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_ingestion_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/hdfs_ingestion_target/HDFS target stores data in Hadoop Distributed File System. An Ingestion application with HDFS target is supported to run on registered clusters and not on Gathr clusters. To know about how to register a cluster with Gathr by establishing PrivateLink, see Compute Setup → To configure a HDFS target, provide the HDFS directory path along with the list of fields of schema to be written. These field values get stored in HDFS file(s), in a specified format, inside the provided HDFS directory.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/advanced_hdfs_etl_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/advanced_hdfs_etl_target/Advanced HDFS emitter allows you to add rotation policy to the emitter. An ETL application with Advanced HDFS emitter is supported to run on registered clusters and not on Gathr clusters. To know about how to register a cluster with Gathr by establishing PrivateLink, see Compute Setup → Target Configuration # Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for HDFS earlier.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/native_dfs_receiver_etl_source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/native_dfs_receiver_etl_source/👉 This Data Source enables you to read data from HDFS. This is a streaming component. Schema Type # See the topic Provide Schema for ETL Source → to know how schema details can be provided for data sources. After providing schema type details, the next step is to configure the data source. Data Source Configuration # Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for HDFS earlier.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/native_hdfs_etl_target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/hdfs/native_hdfs_etl_target/HDFS emitter stores data in Hadoop Distributed File System. To configure a Native HDFS emitter, provide the HDFS directory path along with the list of fields of schema to be written. These field values get stored in HDFS file(s), in a specified format, inside the provided HDFS directory. Target Configuration # Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for HDFS earlier.