For_CDH_Using_Cloudera

Cluster Configuration for CDH Using Cloudera Manager

1. Select Cluster Configuration tile and configure the properties as given below:

Provide values for the following properties:

Field	Description
Cluster Manager	Select the cluster manager as Cloudera Manager.
URL	Provide the Cloudera Manager URL as below
User Name	Username for Cloudera Manager
Password	Password for Cloudera Manager
Cluster Name	Provide the Cloudera Manager cluster name
Enable Kerberos	Enable Kerberos allows Kerberos configuration from cluster manager be configured in StreamAnalytix.

Click Save and configure all managed services that are supported in StreamAnalytix with its progress.

NOTE: Livy configuration is not supported through setup simplification process; please make sure that StreamAnalytix is pointing to correct Livy URL which looks as follows:

Livy URL for CDH/Apache: http://localhost:8998

To install Livy on CDH environment and configure in StreamAnalytix, do as follows:

1. Download apache binary from below url:

http://archive.apache.org/dist/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip

2. Extact zip file using below command:

Cmd: unzip apache-livy-0.6.0-incubating-bin.zip

3. Update livy config file with below parameters.

# What host address to start the server on. By default, Livy will bind to all network interfaces.

livy.server.host = <hostname>

# What port to start the server on.

livy.server.port = 8998

# What spark master Livy sessions should use.

livy.spark.master = yarn

# What spark deploy mode Livy sessions should use.

livy.spark.deploy-mode = cluster

# How long a finished session state should be kept in LivyServer for query.

livy.server.session.state-retain.sec = 30s

# If livy should impersonate the requesting users when creating a new session.

livy.impersonation.enabled = false

# on user request and then livy server classpath automatically.

livy.repl.enable-hive-context =true

4. If we need to enable Kerberos authentication then we need to update below configuration, for this we need to generate keytabs then update the configuration.

# Authentication support for Livy server

# Livy has a built-in SPnego authentication support for HTTP requests with below configurations.

livy.server.auth.type = kerberos

livy.server.auth.kerberos.principal = HTTP/impetus-i0141.impetus.co.in@IMPETUS

livy.server.auth.kerberos.keytab = /etc/security/keytabs/HTTP-impetus-i0141.keytab

livy.server.launch.kerberos.principal = livy@IMPETUS

livy.server.launch.kerberos.keytab = /etc/security/keytabs/livy.service.keytab

5. Start livy using below command

Cmd : <installation_path>/bin/ livy-server start

6. Livy server URL is: http://<hostname>:8998

NOTE: This URL needs to be configured in the StreamAnalytix application under the Processing Engine configuration. livy_config