Doris Setup

Instructions to setup Apache Doris in Gathr are provided below.

Prerequisites #

• Vm max map count should be 2000000

• sysctl -w vm.max_map_count=2000000

• Set max number of open file descriptors to a value greater than 60000. Ask your system manager to modify /etc/security/limits.conf and append content like

  * soft nofile 655350
  * hard nofile 655350

• Run ‘ulimit -n 655350’ to take effect on current session.

• Disable swap

1. Temporary (no reboot required)

• run below command.

$ sudo swapoff -a

2. Permanent (reboot required)

– Open file /etc/fstab and comment out line containing word “swap”. Save the file and reboot the machine.

Installation Steps #

• Download binary targz.

wget https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-2.1.6-bin-x64-noavx2.tar.gz

• Extract it.

tar -zxvf apache-doris-2.1.6-bin-x64-noavx2.tar.gz

• Rename folder.

mv apache-doris-2.1.6-bin-x64-noavx2 apache-doris

• Doris has 2 process FE and BE, now first we will configure and start BE. Edit file apache-doris/be/conf/be.conf and add JAVA_HOME at last line of this file.

vi be.conf

JAVA_HOME=/usr/share/openjdk

In same file update these ports if they are already occupied.

# ports for admin, web, heartbeat service 
be_port = 9060
webserver_port = 7040
heartbeat_service_port = 9050
               brpc_port = 8060

• Start BE.

cd apache-doris/be
./bin/start_be.sh --daemon

• Configure doris FE for adding JAVA_HOME

Cd apache-doris/fe/conf

Vi fe.conf

JAVA_HOME=/usr/share/openjdk

In same file update these ports if they are already occupied.

http_port = 7030
rpc_port = 9020
query_port = 9030
              edit_log_port = 9010

• Start FE.

cd apache-doris/fe

bin/start_fe.sh --daemon

• Connect with mysql client to add backend.

$ mysql -uroot -P9030 -h172.26.**.**
 
mysql>  ALTER SYSTEM ADD BACKEND "be_host_ip:heartbeat_service_port";

Example:

Set password for root and admin user.

mysql>  ALTER SYSTEM ADD BACKEND "localhost";
Query OK, 0 rows affected (0.11 sec)
 
mysql> SET PASSWORD FOR 'root' = PASSWORD('root');
Query OK, 0 rows affected (0.03 sec)
 
mysql> SET PASSWORD FOR 'admin' = PASSWORD('admin');
Query OK, 0 rows affected (0.00 sec)

• Create database and table.

Connect to Apache Doris Use admin account to connect to Apache Doris FE.

mysql -uadmin -P9030 -h127.0.0.1 -p

Create database and table

create database demo;

use demo; 
create table mytable
(
    k1 TINYINT,
    k2 DECIMAL(10, 2) DEFAULT "10.05",    
    k3 CHAR(10) COMMENT "string column",    
    k4 INT NOT NULL DEFAULT "1" COMMENT "int column"
) 
COMMENT "my first table"
DISTRIBUTED BY HASH(k1) BUCKETS 1
PROPERTIES ('replication_num' = '1');

• Ingest data

• Save the following example data to the local “data.csv” file:

1,0.14,a1,20 2,1.04,b2,21 3,3.14,c3,22 4,4.35,d4,23

• Load the data from “data.csv” into the newly created table using the Stream Load method.

curl  --location-trusted -u admin:admin_password -T data.csv -H "column_separator:," http://***.0.0.1:8030/api/demo/mytable/_stream_load

-T data.csv: data file name

-u admin:admin_password: admin account and password

localhost: IP and http_port of FE

Once it is executed successfully, a message like the following will be returned:

{                                                     
    "TxnId": 30,                                  
    "Label": "a56d2861-303a-4b50-9907-238fea904363",        
    "Comment": "",                                       
    "TwoPhaseCommit": "false",                           
    "Status": "Success",                                 
    "Message": "OK",                                    
    "NumberTotalRows": 4,                                
    "NumberLoadedRows": 4,                               
    "NumberFilteredRows": 0,                             
    "NumberUnselectedRows": 0,                          
    "LoadBytes": 52,                                     
    "LoadTimeMs": 206,                                    
    "BeginTxnTimeMs": 13,                                
    "StreamLoadPutTimeMs": 141,                           
    "ReadDataTimeMs": 0,                                 
    "WriteDataTimeMs": 7,                                
    "CommitAndPublishTimeMs": 42                         
} 

NumberLoadedRows: the number of rows that have been loaded

NumberTotalRows: the total number of rows to be loaded

Status: “Success” means data has been loaded successfully.

Query data: Execute the following SQL in the MySQL client to query the loaded data:

mysql> select * from mytable;

+------+------+------+------+                                                                                                                                                                                                              
| k1   | k2   | k3   | k4   |                                                                                                                                                                                                              
+------+------+------+------+                                                                                                                                                                                                              
|    1 | 0.14 | a1   |   20 |                                                                                                                                                                                                              
|    2 | 1.04 | b2   |   21 |                                                                                                                                                                                                              
|    3 | 3.14 | c3   |   22 |                                                                                                                                                                                                              
|    4 | 4.35 | d4   |   23 |                                                                                                                                                                                                              
+------+------+------+------+                                                                                                                                                                                                              
4 rows in set (0.01 sec)  

Doris UI 7030 port FE

• Stop Apache Doris
• Stop FE
• Execute the following command under apache-doris/fe to stop FE.

server1:apache-doris/fe doris$ ./bin/stop_fe.sh

• Stop BE

• Execute the following command under apache-doris/be to stop BE.

server1:apache-doris/be doris$ ./bin/stop_be.sh