Graylog Installation on CentOS 7

Graylog Installation on CentOS 7

OpenJDK Installation

yum install java-1.8.0-openjdk

Selinux Settings

Disable Selinux:

# setenforce 0
# vim /etc/sysconfig/selinux
SELINUX=disabled

MongoDB Installation

Create /etc/yum.repos.d/mongodb-org-3.6.repo with the following content:

vim /etc/yum.repos.d/mongodb-org-3.6.repo
[mongodb-org-3.6]
name=MongoDB Repository
baseurl=/repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.6/x86_64/
gpgcheck=0
enabled=1

Now we can install:

yum install mongodb-org

Start and enable the service:

systemctl enable mongod
systemctl start mongod

ElasticSearch Installation

First need to install Elasticsearch GPG-KEY:

rpm --import /packages.elastic.co/GPG-KEY-elasticsearch

Create /etc/yum.repos.d/elasticsearch.repo with the following content:

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=/artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=/artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Now we can start installing it

yum install elasticsearch

ElasticSearch Configuration

vim /etc/elasticsearch/elasticsearch.yml

...
cluster.name: graylog
...

Update daemon and start the service

systemctl daemon-reload

systemctl enable elasticsearch  
systemctl start elasticsearch
  • ElasticSearch check
curl -XGET '/localhost:9200/_cluster/health?pretty=true'

Output must be like this:

{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Finally Graylog Installation

Adding the repository:

yum install /packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.rpm
yum install graylog-server

Configuration

/etc/graylog/server/server.conf 

Change the values according to your system:

password_secret =
root_password_sha2 =
rest_listen_uri = /externalIP:9000/api/
web_listen_uri = /externalIP:9000/
web_endpoint_uri = /graylog.domainname.com

Create the password_secret value with our pwgen tool:

pwgen -s 96 1

Create root_password_sha2 as follows:

echo -n password |sha256sum

Start and enable the service

systemctl enable graylog-server
systemctl start graylog-server

Access the Web Interface

/graylog_server:9000/api/

Have fun 🙂

Graylog Server Email Notification

Graylog Server Email Notification

Graylog Email Notification Settings

To send Graylog e-mail sending over the local postfix, we need to edit the settings as follows.


# vim /etc/graylog/server/server.conf
...
transport_email_enabled = true
transport_email_hostname = 127.0.0.1
transport_email_port = 25
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_subject_prefix = graylog
transport_email_from_email = graylog@domainname.com
  • To correct the stream URL connection, make the following settings and restart the service

# vim /etc/graylog/server/server.conf

transport_email_web_interface_url = /gri.ozguryazilim.com

# service graylog-server restart

Automatic Log Collection On Graylog

Automatic Log Collection On Graylog

Collector-Sidecar Installation on CentOS 7

Installation with Beats Backend

Beat sidecar is included in the package so we go directly to the sidecar installation:

# yum install /github.com/Graylog2/collector-sidecar/releases/download/0.1.6/collector-sidecar-0.1.6-1.x86_64.rpm

Start/enable the service:

# graylog-collector-sidecar -service install
# systemctl enable collector-sidecar
# systemctl start collector-sidecar

Installation with NXLog Backend

# yum install /nxlog.co/system/files/products/files/348/nxlog-ce-2.10.2102-1_rhel7.x86_64.rpm

We need to stop NXLog and delete the default service settings because Sidecar takes control of the start / stop of the service

# service nxlog stop
# chkconfig --del nxlog
# gpasswd -a nxlog root
# chown -R nxlog.nxlog /var/spool/collector-sidecar/nxlog

Sidecar installation:

# yum install /github.com/Graylog2/collector-sidecar/releases/download/0.1.6/collector-sidecar-0.1.6-1.x86_64.rpm

Start/enable the service:

# graylog-collector-sidecar -service install
# systemctl enable collector-sidecar
# systemctl start collector-sidecar

Settings

Sidecar Settings

vim /etc/graylog/collector-sidecar/collector_sidecar.yml
server_url: /graylog_server_ip:9000/api/
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
  - /var/log
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
    - apache
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml
  • After starting the service, sidecar creates /etc/graylog/collector-sidecar/generated/filebeat.yml the moment it finds the settings that overlap with its “tags”. Updates all changes made in Graylog web interface here

  • If you want to change the backend or use NXLog, you need to make changes to enabled = false / enabled = true. Then we can change the default settings of collectors> NXLog> snippet via graylog web interface and we can define it from below interface as well.

Firewall Settings

Let’s open the port 5044 so we can forward the logs we collect

# firewall-cmd --permanent --zone=public --add-port=5044/udp
# firewall-cmd --reload

Graylog Web Interface Settings

  • System > Inputs

Beats Input for Beats Backend

Bind Adress: 0.0.0.0
Port: 5044

If you want to connect with TLS, Enable TLS must be marked. Then, without any additional information, the graylog adds the self-signed certificate to this input.

  • Let’s open global Gelf UDP input for NXLog Backend

  • System > Collectors > Manage configurations

  • Click Create configuration

  • Give a descriptive name

  • Let’s create a new output by clicking on the configuration we have created, let’s give the IP of our graylog server to hosts

Beats Backend

For TLS, the following must also be marked;

Enable TLS support
Insecure TLS connection

NXLog Backend

  • Create an input to collect Apache access logs

Beats Backend

NXLog Backend

  • Update the Apache tag

Let’s control it

# graylog-collector-sidecar

INFO[0000] Using collector-id: 6f7eaddb-0a1e-4d60-b2c7-5f8a0f512523
INFO[0000] No node-id was configured, falling back to hostname
INFO[0000] Fetching configurations tagged by: [apache]   
INFO[0000] Starting signal distributor                  
INFO[0000] [filebeat] Starting (exec driver)            
INFO[0010] [filebeat] Configuration change detected, rewriting configuration file.
INFO[0010] [filebeat] Stopping                          
INFO[0012] [filebeat] Starting (exec driver)

Error detection and debug

# graylog-collector-sidecar -debug -c /etc/graylog/collector-sidecar/collector_sidecar.yml

INFO[0000] Using collector-id: 6f7eaddb-0a1e-4d60-b2c7-5f8a0f512523
INFO[0000] No node-id was configured, falling back to hostname
INFO[0000] Fetching configurations tagged by: [apache]   
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/collector_sidecar.log.%Y%m%d%H%M
DEBU[0000] Add collector backend: filebeat              
INFO[0000] Starting signal distributor                  
DEBU[0000] [signal-processor] (seq=1) handling cmd: restart
..
..

Non-root Access

By default, sidecar uses the root user. To avoid security vulnerabilities:

# useradd -r collector
# chown -R collector /etc/graylog
# chown -R collector /var/cache/graylog
# chown -R collector /var/log/graylog
# setfacl -m u:collector:r /var/log/*

Let’s say to Systemd that Sidecar will start with a non-root user:

# vim /etc/systemd/system/collector-sidecar.service

[Service]
User=collector
Group=collector
# systemctl daemon-reload
# systemctl restart collector-sidecar

If you get a permission error, make sure to replace the files under / var / cache / graylog / collector-sidecar / with chown

PostgreSQL Installation

6788743178

PostgreSQL Installation

( CentOS7 and PostgreSQL-9.6 )

First repository will be installed then PostgreSQL will be installed from this repository

yum install /yum.postgresql.org/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm
yum install postgresql96-server postgresql96-contrib

Initialize the DB:

/usr/pgsql-9.6/bin/postgresql96-setup initdb

Starting and enabling the service:

systemctl start postgresql-9.6
systemctl enable postgresql-9.6

Interface binding:

vim /var/lib/pgsql/9.6/data/postgresql.conf

listen_addresses = '*'
ProxySQL Installation and Configuration

(716) 263-5235

Installation Steps

MySQL Installation

yum install /www.percona.com/downloads/percona-release/redhat/0.1-6/percona-release-0.1-6.noarch.rpm
yum install Percona-Server-server-57 Percona-Server-client-57 Percona-Server-devel-57
yum update
systemctl start mysqld
systemctl enable mysqld
/usr/bin/mysql_secure_installation

A default root password is automatically assigned with MySQL 5.7. The corresponding password can be found by searching the temporary password in the MySQL log.

grep "temporary password" /var/log/mysqld.log

In order to prevent the root password from entering the root user every time, create /root/.my.cnf with the following content:

[client]
user=root
host=localhost
password='password_here'

You can’t touch this, you cant touch this, you cant touch this x3

chmod 0600 /root/.my.cnf

Adding the Repository

yum install /www.percona.com/redir/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm
yum update

Installation

yum install proxysql

Starting and enabling the service

systemctl start proxysql
systemctl enable proxysql

In order to change SQL interface port 6033 of proxysql into MySQL port 3306:

vim /etc/proxysql.cnf
mysql_variables=
{
# ... diğer ayarlar, değiştirmeye gerek yok
interfaces="0.0.0.0:3306;/tmp/proxysql.sock" 
# ... diğer ayarlar, değiştirmeye gerek yok
}

Some security stuff 🙂

firewall-cmd --zone=public --add-port=3306/tcp --permanent
firewall-cmd --reload

Restart the service

systemctl restart proxysql

ProxySQL Configuration

+-------------------------+
|         RUNTIME         |
+-------------------------+
       /|\          |
        |           |
    [1] |       [2] |
        |          \|/
+-------------------------+
|         MEMORY          |
+-------------------------+ _
       /|\          |      |\
        |           |        \
    [3] |       [4] |         \ [5]
        |          \|/         \
+-------------------------+  +-------------------------+
|          DISK           |  |       CONFIG FILE       |
+-------------------------+  +-------------------------+

Connect to ProxySQL through MySQL

mysql -u admin -padmin -h 127.0.0.1 -P6032 --prompt='Admin> '

Clean as my room 😉

Admin> SELECT * FROM mysql_servers;
Empty set (0.00 sec)

Admin> SELECT * from mysql_replication_hostgroups;
Empty set (0.00 sec)

Admin> SELECT * from mysql_query_rules;
Empty set (0.00 sec)

Adding Server Behind ProxySQL

In order to add a server that we want to be behind ProxySQL, we need to add it to the mysql_servers table and transfer it to the working environment. ProxySQL collects servers under groups named hostgroup.

For example, we can add a MySQL server at 192.168.56.101 to ProxySQL as follows.

  • hostgroup0 -> master
  • hostgroup1 -> slaves

INSERT INTO mysql_servers(hostgroup_id, hostname) VALUES (0,'192.168.56.101');

Then to runtime:


LOAD MYSQL SERVERS TO RUNTIME;

Then to disk:


SAVE MYSQL SERVERS TO DISK;

Adding Monitor User

Servers behind ProxySQL need to have a monitor user with read permission on those servers to control connection status and delays. This user has to be the same for all sets.

On ProxySQL server:


UPDATE global_variables SET variable_value='monitor' WHERE variable_name='mysql-monitor_username';
UPDATE global_variables SET variable_value='monitor' WHERE variable_name='mysql-monitor_password';
Admin> LOAD MYSQL VARIABLES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)

Admin> SAVE MYSQL VARIABLES TO DISK;
Query OK, 96 rows affected (0.02 sec)

Try it out:

Admin> SELECT * FROM global_variables WHERE variable_name LIKE 'mysql-monitor_%';
+-----------------------------------------------------+----------------+
| variable_name                                       | variable_value |
+-----------------------------------------------------+----------------+
| mysql-monitor_enabled                               | true           |
| mysql-monitor_connect_timeout                       | 600            |
| mysql-monitor_ping_max_failures                     | 3              |
| mysql-monitor_ping_timeout                          | 1000           |
| mysql-monitor_read_only_max_timeout_count           | 3              |
| mysql-monitor_replication_lag_interval              | 10000          |
| mysql-monitor_replication_lag_timeout               | 1000           |
| mysql-monitor_groupreplication_healthcheck_interval | 5000           |
| mysql-monitor_groupreplication_healthcheck_timeout  | 800            |
| mysql-monitor_replication_lag_use_percona_heartbeat |                |
| mysql-monitor_query_interval                        | 60000          |
| mysql-monitor_query_timeout                         | 100            |
| mysql-monitor_slave_lag_when_null                   | 60             |
| mysql-monitor_wait_timeout                          | true           |
| mysql-monitor_writer_is_also_reader                 | true           |
| mysql-monitor_username                              | monitor        |
| mysql-monitor_password                              | monitor        |
| mysql-monitor_history                               | 600000         |
| mysql-monitor_connect_interval                      | 60000          |
| mysql-monitor_ping_interval                         | 10000          |
| mysql-monitor_read_only_interval                    | 1500           |
| mysql-monitor_read_only_timeout                     | 500            |
+-----------------------------------------------------+----------------+
22 rows in set (0.00 sec)

Lovely…

On mysql servers:


CREATE USER 'monitor'@'%' IDENTIFIED by 'monitor';
GRANT USAGE on *.* TO 'monitor'@'%';

Replication Groups

ProxySQL assigns the read / write hostgroup by the read_only values of the added servers.

Admin> SELECT * FROM monitor.mysql_server_read_only_log ORDER BY time_start_us DESC LIMIT 10;
Empty set (0.00 sec)

Since the default is read_only = 0, first on the slave / slaves:

mysql> set global read_only=1;
Query OK, 0 rows affected (0.00 sec)

Let’s define our replication groups:

Admin> SELECT * FROM mysql_replication_hostgroups;
Empty set (0.00 sec)

Admin> INSERT INTO mysql_replication_hostgroups VALUES (1,2,'cluster1');
Query OK, 1 row affected (0.00 sec)

read_only = 0 ones to hostgroup 1 read_only = 1 ones to be moved to hostgroup 2:

Admin>  SELECT * FROM mysql_replication_hostgroups;
+------------------+------------------+---------+
| writer_hostgroup | reader_hostgroup | comment |
+------------------+------------------+---------+
| 1                | 2                | cluster1|
+------------------+------------------+---------+
1 row in set (0.00 sec)

Let’s check our servers:

Admin> LOAD MYSQL SERVERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)

Admin> SELECT * FROM monitor.mysql_server_read_only_log ORDER BY time_start_us DESC LIMIT 5;
+------------+------+------------------+-----------------+-----------+-------+
| hostname   | port | time_start_us    | success_time_us | read_only | error |
+------------+------+------------------+-----------------+-----------+-------+
| 10.0.0.111 | 3306 | 1540470818742428 | 564             | 0         | NULL  |
| 10.0.0.114 | 3306 | 1540470818717006 | 1185            | 1         | NULL  |
| 10.0.0.114 | 3306 | 1540470817240409 | 587             | 1         | NULL  |
| 10.0.0.111 | 3306 | 1540470817215687 | 615             | 0         | NULL  |
| 10.0.0.111 | 3306 | 1540470815744639 | 689             | 0         | NULL  |
+------------+------+------------------+-----------------+-----------+-------+
5 rows in set (0.00 sec)

Admin> SAVE MYSQL SERVERS TO DISK;
Query OK, 0 rows affected (0.04 sec)

Admin> SAVE MYSQL VARIABLES TO DISK;
Query OK, 97 rows affected (0.01 sec)
Auditd Installation on CentOS 7

nonmunicipal

Auditd Installation

Normally auditd is installed as default. Therefore before going into the next step let’s check if it’s installed or not:

# rpm -qa | grep audit

audit-libs-python-2.8.1-3.el7_5.1.x86_64
audit-2.8.1-3.el7_5.1.x86_64
audit-libs-2.8.1-3.el7_5.1.x86_64

If you see the packages above:

# yum install audit

Starting the service:

# systemctl is-enabled auditd
# systemctl start auditd
# systemctl enable auditd

Configuration File:

/etc/audit/auditd.conf

Managing The Rules

# auditctl -w /etc/passwd -p wa -k passwd_changes
# auditctl -w /etc/group -p wa -k group_changes
# auditctl -w /etc/shadow -p wa -k shadow_changes
# auditctl -w /etc/sudoers -p wa -k sudoers_changes
auditctl -w path_to_file -p permissions -k key_name

r — read access to a file or a directory.
w — write access to a file or a directory.
x — execute access to a file or a directory.
a — change in the file’s or directory’s attribute.
key_name is an optional string that helps you identify which rule or a set of rules generated a particular log entry.

Rules that are exist:

# auditctl -l

-w /etc/ssh/sshd_config -p rwxa -k sshconfigchange
-w /etc/passwd -p wa -k passwd_changes
-w /etc/group -p wa -k group_changes
-w /etc/sudoers -p wa -k sudoers_changes

Searching the Audit Logs

Failed login attempts:

# ausearch -m USER_LOGIN -sv no 

User and time based:

# ausearch -ua kaan -ts yesterday -te now

Search:

# ausearch -k passwd_changes | less

Generating Audit Reports

General summary:

# aureport -x --summary

Executable Summary Report
=================================
total  file
=================================
287  /usr/lib/systemd/systemd
261  /usr/sbin/xtables-multi
246  /usr/sbin/crond
86  /usr/sbin/sshd
66  /usr/sbin/ebtables-restore
50  /usr/bin/kmod
50  /usr/bin/sudo

Failed Summary:

# aureport --failed

Failed Summary Report
======================
Number of changes in configuration: 0
Number of changes to accounts, groups, or roles: 0
Number of logins: 0
Number of failed logins: 0
Number of authentications: 0
Number of failed authentications: 3
Number of users: 1
Number of terminals: 2
Number of host names: 2
Number of executables: 2
Number of commands: 1
Number of files: 0
Number of AVC's: 0
Number of MAC events: 0
Number of failed syscalls: 0
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 0
Number of integrity events: 0
Number of virt events: 0
Number of keys: 0
Number of process IDs: 4
Number of events: 7
Letsencrypt Installation

262-421-4510

Letsencrypt Installation

Installation Steps

Certbot is packaged in EPEL. To use Certbot, you must first enable the EPEL repository.


$ sudo yum install python2-certbot-apache

Certbot has a fairly solid beta-quality Apache plugin, which is supported on many platforms and automates certificate installation.

$ sudo certbot --apache

Certificate Generation

certbot certonly --webroot -w /var/www/linuxinhogwarts.com/ -d linuxinhogwarts.com -d www.linuxinhogwarts.com  

With -d it is possible to add more domain names.

Renew Certificates with Scheduled Task

 
0 0 * * 0 root /usr/bin/certbot renew --quiet  
Audit Log Monitoring with Graylog

Audit Log Monitoring with Graylog

Audit Log Monitoring with Graylog

I assume you are able to send audit logs to graylog already. If you haven’t done it yet, please first read my collector sidecar post.

Rules Creation

System > Pipelines > Manage Rules > Create Rule

Rule Writing 101

  • rule

  • when kind of if

  • then action part

  • has_field self-explained

  • let assign to a variable

Let’s identify audit logs among others:

rule "audit"
    
    when
    
        has_field("msg") AND
        contains(to_string($message.msg),"audit")
    
    then
    
        set_field("is_auditd", true);
    
    end
    

Audit Logs Parsing

rule "Wanna parse?"

when

    has_field("is_auditd")

then

    set_fields(
        fields:
            key_value(
                value: to_string($message.message),
                trim_value_chars: "\""
            ),
            prefix: "auditd_"
        );

end

Let’s get rid of cron logs

rule "Cron jobs go home"

when

 has_field("auditd_terminal") AND
 to_string($message.auditd_terminal)=="cron"

then

  drop_message();

end

Create Pipeline

Stage Oluşturma

  • Since we are starting to run from small to large, we can prioritize it by enumerating it while creating the stage.

Pipiline Stream Connection

Dashboard Creation

  • Dashboards > Create dashboard

h2. Quick Values Plus Widget Installation

|Versiyon 4.0.0 Beta | Graylog versiyon 2.4 | |Versiyon 3.1.0 | Graylog versiyon 2.3 | |Versiyon 2.1.0 | Graylog versiyons 2.2.1, 2.2.2, ve 2.2.3| |Versiyon 1.0.0 | Graylog versiyon 2.1.3|

For Graylog 2.4:

# curl -OL /github.com/billmurrin/graylog-plugin-quickvaluesplus-widget/releases/download/4.0.0-beta/graylog-plugin-quickvaluesplus-widget-4.0.0.jar
  • For Graylog 2.3:
# curl -OL /github.com/billmurrin/graylog-plugin-quickvaluesplus-widget/releases/download/3.1.0/graylog-plugin-quickvaluesplus-widget-3.1.0.jar
  • Installation and service restart
# mv graylog-plugin-quickvaluesplus-widget-3.1.0.jar /usr/share/graylog-server/plugin/

# systemctl restart graylog-server

Then, we need to clear cache by pressing ctrl-f5 several times through the web interface

Adding Logs to Dashboars in Graphs

Add logs to dashboard by using search

  • To edit in Dashboard, we need to start editing with lock / unlock in the top right corner and then save

!quick_values_dashboard1.png! !quick_values_dashboard.png!

h2. Alert Oluşturma

h3. Alerts > Manage conditions > Add new condition

!add_new_condition.png! !add_new_condition2.png! !alert_condition.png!

h3. Alerts > Manage notification > Add new notification

!email_alert.png! !email_alert_ayar.png!

In order for the email to be sent, the server has to be allowed from the configuration file if it is not allowed:

You need to edit the settings to send Graylog e-mail sending over the local postfix. You can make settings by following the related blog:

Graylog Server Email Notification