https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/Recent content in Databricks onHugo -- gohugo.ioSun, 11 Dec 2022 19:37:55 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-connection/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-connection/The Databricks Connection serves as the gateway between your Gathr application and the Databricks platform. It enables seamless communication and interaction with the Databricks Query Endpoint for running SQL queries. 💡 When connected to the Databricks Engine, connection details will not be asked while configuring Databricks channels and emitters. Instead, your Databricks instance session will be used to retrieve the required metadata. Prerequisites # If using SQL warehouse as Query Endpoint, get the connection details as follows:https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-etl-source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-etl-source/The Databricks Data Source in Gathr facilitates data extraction from Databricks for analysis and transformation. It offers an easy configuration process to define extraction parameters. Schema Type # See the topic Provide Schema for ETL Source → to know how schema details can be provided for data sources. After providing schema type details, the next step is to configure the data source. Data Source Configuration # Configure the data source parameters as explained below.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-etl-target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-etl-target/The Databricks Emitter can be customized to send processed data from Gathr to Databricks, enabling a smooth flow of information between the two platforms. Target Configuration # Configure the target parameters as explained below. Fetch From Target/Upload Schema File The data source records needs to be emitted to a Databricks Database target table. In case if the Gathr application has access to a target table in the Databricks Database, choose the option Fetch From Target.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-incremental-configuration/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-incremental-configuration/Enable Incremental Read # You can choose to enable incremental read. This is useful if the need is only to read the objects that have been updated in the source, since the last execution of the application for the given configuration. Provide the details for enabling incremental read as described below: Column # Select a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp, decimal type of values.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-ingestion-source/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-ingestion-source/The Databricks Data Source in Gathr facilitates data extraction from Databricks for analysis and transformation. It offers an easy configuration process to define extraction parameters. Data Source Configuration # Configure the data source parameters as explained below. Fetch From Source/Upload Data File # To design the application, you can either fetch the sample data from the Databricks source by providing the data source connection details or upload a sample data file in one of the supported formats to see the schema details during the application design phase.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-ingestion-target/Mon, 12 Dec 2022 19:50:16 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/components/databricks/databricks-ingestion-target/The Databricks Emitter can be customized to send processed data from Gathr to Databricks, enabling a smooth flow of information between the two platforms. Target Configuration # Configure the target parameters as explained below. Fetch From Target/Upload Schema File # The data source records needs to be emitted to a Databricks Database target table. In case if the Gathr application has access to a target table in the Databricks Database, choose the option Fetch From Target.