PostgreSQL User Configuration

Change password of PostgreSQL user by the following command (default password is postgres):

postgres=# alter user <<username>> with password <<newpassword>>;

postgres=# alter user postgres with password ‘scott’;

NOTE: Please keep the new password under quotes.

Create New User (Optional):

In PostgreSQL, the default root user is ‘postgres’, if you want to create a new user with login permission, use the following command:

$ sudo -u postgres psql postgres

postgres=# create role <<new_user_name>> login password '<<new_­password>>';

Email Alert Database

1. In <<installationDir>>/db_dump/pgsql_1.2 , you will find activiti.sql .

2. Create a new database for activiti, you will need to point this database while con­figuring StreamAnalytix application.

3. Import <<installationDir>>/db_dump/pgsql_1.2/activiti.sql

$ psql -U postgres -d <<activiti_db_name>> -h <<pqsql_host>> -f <<installationDir>>/db_dump/pgsql_1.2/activiti.sql

ElasticSearch 5.5.2(Optional)

To install ElasticSearch, follow the steps mentioned below:

1. Download Elasticsearch binary (.tar.gz) version 5.5.2 from the below Url:

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.2.tar.gz

2. Extract the tar.gz using below command:

$ tar -xf elasticsearch-5.5.2.tar.gz -C <<installationDir>>

$ cd <<installationDir>>/<<extractedDir>>

3. Configure the following properties in <<installationDir>>/<<extractedDir>>/conf/elasticsearch.yml file under the extracted folder

Note: - Make sure there is a space at the starting of the line (Just remove # do not remove space)

space before

cluster.name

node.name

path.data

path.logs

4. Elasticsearch nodes join a cluster based on just one property named cluster.name.

For example, if you want to add the node to cluster ‘mass_deployment’, change the value of property ‘cluster.name’ to ‘mass_deployment’ as follows:

cluster.name: mass_deployment

This should be same across all nodes of the cluster. This value will be required while configuring Elasticsearch in StreamAnalytix.

5. The node name should be unique for each ES node in a cluster. This is defined by the ‘node.name’ property.

 For example: If user wants to deploy two nodes for the cluster, the names can be ‘node0’, ‘node1’ and ‘node2’.

• node.name: ‘node0’

This should be unique for each node in the cluster

• node.tag: “node0”

This should be unique for each node and same as node.name. Also, use the conven­tion node0, node1, nodeN.

• path.data: /path/to/data/dir

This defines the directory path of indexed data.

• action.auto_create_index: false

This property restricts auto-creation of indices. The value should be false.

• index.number_of_shards: 2

Sets the number of shards (splits) of an index (5 by default)

• index.number_of_replicas: 0  

Sets the number of replicas (additional copies) of an index (1 by default):

• bootstrap.mlockall: false

Locks the memory for better performance of Elasticsearch.

6. Update conf/elasticsearch.yml with the below details:

index.number_of_shards: 5

index.number_of_replicas: 1

transport.tcp.port: 9300

transport.bind_host: <hostname/IP>

transport.host:<hostname/IP>

network.host: hostname/IP>

http.port:9200

7. (Optional) Specify heap size for Elasticsearch by adding the below line to the file ‘/home/stanalytix/.bashrc’:

export ES_HEAP_SIZE=4g

8. Start the Elasticsearch node by logging in as root user to enable memory locking as follows:

$<<installationDir>>/<<extractedDir>>/bin/elasticsearch -d

9. To enable Elasticsearch plugin, follow the steps mentioned below:

$ cd <<installationDir>>/<<extractedDir>>

$ sudo bin/plugin install mobz/elasticsearch-head

$ sudo service elasticsearch restart #( start | stop | restart | status

10. To access Elasticsearch GUI, open the following URL on a browser:

http://<<HOSTNAME>>:9200/_plugin/head/

Livy Support(HDP)

Livy is packaged with HDP2.6.

Verify LIVY installation by following the below steps:

1. Login into Ambari console.

2. Go to Dashboard and select Spark or Spark2 service from left Pane.

3. For Spark2, 'Livy for Spark2 Server' should be running.

If Livy is not installed, follow these steps to install Livy:’.

Note: When using Livy - StreamAnalytix pipelines should be saved / submitted in the same mode (cluster/ client) that is configured on Livy

 

To verify Livy mode, follow below steps:

1. Login to Ambari.

2. Go to Dashboard and select specific Spark version.

3. Click on 'Config' tab.

4. Now, search 'livy.spark.master' property. If its value is 'yarn-cluster' or 'cluster' then Livy is configured to support ‘Cluster Mode’ otherwise the pipelines must be running on client mode.

 

In client mode, Livy picks up an application binary from local file system. Set the below property which tells Livy to add a local directory into its white list

  livy.file.local-dir-whitelist =/home/sax/StreamAnalytix/lib/

 

Make sure that the local directory (directory where application binaries resides) is mounted on NFS

See - (https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-centos-6)

 

Alternatively, StreamAnalytix can be installed on the node where Livy is deployed in order to support local mode

 

1.Login into Ambari.

2.Go to Hosts and select node on which Livy is to be installed.

3.Hosts and select node on which Livy is to be installed.

Now click on '+Add' button to install 'Livy for Spark2 Server'

To enable Livy support, configure the below StreamAnalytix properties in env-con­fig.yaml file.

$ Webstudio Properties

#job-server,spark-submit, livy

 job.submit.mode: "livy"

livy.url: "http://localhost:8998"

Appendix -1 - StreamAnalytix Pre-req­uisites

Java 1.8 X

Use the following instructions to manually install JDK 1.8:

1. Verify that you have a /usr/java directory. If not, create one:

$ mkdir /usr/java

2. Download the Oracle 64-bit JDK (jdk-8u101-linux-x64.tar.gz) from the Oracle download site. Open a web browser and navigate to http://www.oracle.com/tech­network/java/javase/downloads/jdk8-downloads-2133151.html

3. Copy the downloaded jdk.tar.gz file to the /usr/java directory.

4. Navigate to the /usr/java directory and extract the jdk.tar.gz file.

$ cd /usr/java  tar zxvf jdk-8u101-linux-x64.tar.gz

5. The JDK files will be extracted into a /usr/java/jdk-8u101 directory.

6. Create a symbolic link (symlink) to the JDK:

$ ln -s /usr/java/jdk1.8.0_101 /usr/java/default

$ ln –s /usr/java/jdk1.8.0_101/bin/java /usr/bin/java

7. Set the JAVA_HOME and PATH environment variables.

$ export JAVA_HOME=/usr/java/default

$ export PATH=$JAVA_HOME/bin:$PATH

8. Run below commands to notify system that new java version is ready for use.

sudo update-alternatives --install "/usr/bin/java" "java"

"/usr/java/jdk1.8.0_101/bin/java" 1

 

sudo update-alternatives --install "/usr/bin/javac" "javac"  "/usr/java/jdk1.8.0_101/bin/javac" 1

 

sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/java/jdk1.8.0_101/bin/javaws" 1

9. Verify that Java is installed in your environment by running the following command.

$ java –version

10. You should see output similar to following:

java version "1.8.0_101"

Java(TM) SE Runtime Environment (build 1.8.0_101-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.101-b01, mixed mode)

Appendix-2 Post Deployment Steps