- ELK란?
- Elasticsearch, Logstash, Kibana의 약자를 딴 스택을 의미하는 약어
- Elasticsearch란?
- LogStash를 통해서 전송받은 데이터 분석 및 변환, 저장 기능을 담당하는 검색/분석 엔진이다.
- 단순한 데이터 검색 및 정보 집계를 넘어 데이터의 추세와 패턴을 발견할 수 있게한다.
- 구축환경
- OS: Ubiuntu 22.04
- CPU: 4v Cpu
- RAM: 16GB
- DISK: 100GB(OS 및 프로그램 설치할 하드) + 100GB(ELK 분석/저장 파일 보관할 하드)
- ETC: Openstack(IAAS)
사전 마운트 작업 (넘기기 가능)
- 필자는 ELK에서 생성되는 분석/저장 파일들을 따로 보관할 예정이여서 하드를 하나 더 마운트시켜 거기로 설정해둘 예정이다.
- 이것은 그 사전작업을 기록한거니 이러한 특수한 경우가 아니라면 넘어가도 된다.
oot@ubuntu:/home/ubuntu# fdisk -l
Disk /dev/loop0: 63.46 MiB, 66547712 bytes, 129976 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop1: 40.86 MiB, 42840064 bytes, 83672 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop2: 111.95 MiB, 117387264 bytes, 229272 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/vda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C62F22EE-6B8C-4F79-846D-A06BA64A39C6
Device Start End Sectors Size Type
/dev/vda1 227328 209715166 209487839 99.9G Linux filesystem
/dev/vda14 2048 10239 8192 4M BIOS boot
/dev/vda15 10240 227327 217088 106M EFI System
Partition table entries are not in disk order.
Disk /dev/vdb: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@ubuntu:/home/ubuntu# fdisk /dev/vd
vda vda1 vda14 vda15 vdb
root@ubuntu:/home/ubuntu# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.37.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xf5d08a84.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-209715199, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-209715199, default 209715199): p
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-209715199, default 209715199):
Created a new partition 1 of type 'Linux' and of size 100 GiB.
Command (m for help): p
Disk /dev/vdb: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xf5d08a84
Device Boot Start End Sectors Size Id Type
/dev/vdb1 2048 209715199 209713152 100G 83 Linux
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
root@ubuntu:/home/ubuntu# mkfs.ext4 /dev/vd
vda vda1 vda14 vda15 vdb vdb1
root@ubuntu:/home/ubuntu# mkfs.ext4 /dev/vd
vda vda1 vda14 vda15 vdb vdb1
root@ubuntu:/home/ubuntu# mkfs.ext4 /dev/vdb1
mke2fs 1.46.5 (30-Dec-2021)
Discarding device blocks: done
Creating filesystem with 26214144 4k blocks and 6553600 inodes
Filesystem UUID: f20f039a-3206-438c-a052-c83ef52cf229
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: done
root@ubuntu:/# blkid
/dev/vdb1: UUID="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX" BLOCK_SIZE="XXXX" TYPE="XXXX" PARTUUID="XXXXXXX-XX"
#마운트할 하드 UUID 확인
root@ubuntu:/# cd mnt/
root@ubuntu:/mnt# ll
total 8
drwxr-xr-x 2 root root 4096 Oct 27 02:33 ./
drwxr-xr-x 19 root root 4096 Feb 1 04:17 ../
root@ubuntu:/mnt# mkdir log_mnt1
#마운트할 폴더 생성
root@ubuntu:/mnt/log_mnt1# vi /etc/fstab
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX /mnt/log_mnt1 ex4 defaults 0 0
#추가
root@ubuntu:/mnt/log_mnt1# mount -a
root@ubuntu:/mnt/log_mnt1# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.6G 1.1M 1.6G 1% /run
/dev/vda1 97G 3.8G 94G 4% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/vda15 105M 6.1M 99M 6% /boot/efi
tmpfs 1.6G 4.0K 1.6G 1% /run/user/1000
/dev/vdb1 98G 24K 93G 1% /mnt/log_mnt1
#완료사전 설치 작업 (nginx 설치, JDK, apt-transport-https 설치)
- nginx를 통해서 80으로 들어올시 바로 서비스 브라우저로 이동하도록 설정할 생각이다.
- JDK는 ELK를 돌릴때 (특히 Logstash 사용시) 필요한 유틸이므로 설치를 해야한다.
apt install default-jdk
#JDK 설치
apt install nginx
#nginx
root@ubuntu:/home/ubuntu# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
#공개키 서명
root@ubuntu:/home/ubuntu# sudo apt-get install apt-transport-https
#https 패키지 접근자 'apt-transport-https' 설치
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
apt-transport-https
0 upgraded, 1 newly installed, 0 to remove and 24 not upgraded.
Need to get 1510 B of archives.
After this operation, 170 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 apt-transport-https all 2.4.11 [1510 B]
Fetched 1510 B in 1s (2766 B/s)
Selecting previously unselected package apt-transport-https.
(Reading database ... 64302 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_2.4.11_all.deb ...
Unpacking apt-transport-https (2.4.11) ...
Setting up apt-transport-https (2.4.11) ...
Scanning processes...
Scanning linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@ubuntu:/home/ubuntu# echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main
#APT 저장소 정의
root@ubuntu:/home/ubuntu# apt-get update
#APT 저장소 최신화
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable InRelease [10.4 kB]
Get:2 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 Packages [82.0 kB]
Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1109 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1325 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [207 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1313 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [267 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1342 kB]
Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [214 kB]
Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [837 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [220 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1042 kB]
Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [160 kB]
Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB]
Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7476 B]
Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [235 kB]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.2 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB]
Fetched 8881 kB in 4s (2216 kB/s)
Reading package lists... Done설치 및 설정 (Elasticsearch)
- Elasticsearch 설치시 주의할점은 yml 파일 설정시 맨밑의 보안 부분인데 기본적으로 설치과정에서는 끄는걸 추천한다. (필자도 초반엔 켰다가 작동 확인이 불가능해 SSL부분만 껏고 추후에 'xpack.security.transport.ssl'와 함께 다시 킬 예정이다)
- yml 설정부분에서 색인된 데이터를 저장하는 경로를 필자는 수정했는데 필요없는 사람은 그부분은 넘어가도 된다.
root@ubuntu:/home/ubuntu# apt-get install elasticsearch
#elasticsearch 설치
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 91 not upgraded.
Need to get 621 MB of archives.
After this operation, 1267 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 elasticsearch amd64 8.12.0 [621 MB]
Fetched 621 MB in 2min 20s (4449 kB/s)
Selecting previously unselected package elasticsearch.
(Reading database ... 64306 files and directories currently installed.)
Preparing to unpack .../elasticsearch_8.12.0_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (8.12.0) ...
Setting up elasticsearch (8.12.0) ...
--------------------------- Security autoconfiguration information ------------------------------
Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.
The generated password for the elastic built-in superuser is : XXXXXXXXXXXXXXXX
If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.
You can complete the following actions at any time:
Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.
Generate an enrollment token for Kibana instances with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.
Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.
-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
sudo systemctl start elasticsearch.service
Scanning processes...
Scanning linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@ubuntu:/mnt/log_mnt1# mkdir elasticsearch
root@ubuntu:/mnt/log_mnt1# ll
total 28
drwxr-xr-x 4 root root 4096 Feb 1 05:53 ./
drwxr-xr-x 3 root root 4096 Feb 1 05:44 ../
drwxr-xr-x 2 root root 4096 Feb 1 05:53 elasticsearch/
drwx------ 2 root root 16384 Feb 1 05:05 lost+found/
#로그 저장할 폴더 생성
root@ubuntu:/mnt/log_mnt1# cd /var/lib/
root@ubuntu:/var/lib# ll | grep elasticsearch
drwxr-s--- 2 elasticsearch elasticsearch 4096 Jan 11 10:12 elasticsearch/
#setGID 권한 확인
root@ubuntu:/mnt/log_mnt1# chmod 2750 elasticsearch/
root@ubuntu:/mnt/log_mnt1# ll
total 28
drwxr-xr-x 4 root root 4096 Feb 1 05:53 ./
drwxr-xr-x 3 root root 4096 Feb 1 05:44 ../
drwxr-s--- 2 elasticsearch elasticsearch 4096 Feb 1 05:53 elasticsearch/
drwx------ 2 root root 16384 Feb 1 05:05 lost+found/
#setGID 권한 부여
vi /etc/elasticsearch/elasticsearch.yml
#elasticsearch 설정파일 선택
...(생략)
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
# 색인된 데이터를 저장하는 경로
>
path.data: /mnt/log_mnt1/elasticsearch
#수정
#...(생략)
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#접근 호스트
#...(생략)
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 05-02-2024 01:43:07
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: false
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["ubuntu"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
wq
#저장
root@ubuntu:/usr/share/elasticsearch# curl http://localhost:9200
Enter host password for user 'elastic':
{
"name" : "ubuntu",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "iAMpG8AsRcyhOARhPzc1Rg",
"version" : {
"number" : "8.12.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "1665f706fd9354802c02146c1e6b5c0fbcddfbc9",
"build_date" : "2024-01-11T10:05:27.953830042Z",
"build_snapshot" : false,
"lucene_version" : "9.9.1",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
#정상 설치 완료 출력 확인계정 등록
root@ubuntu:/home/ubuntu# cd /usr/share/elasticsearch/bin/
root@ubuntu:/usr/share/elasticsearch/bin# ./elasticsearch-setup-passwords auto
#필수 시스템 계정 자동 생성 (내역저장필수)참고 사이트
Installing the Elastic Stack | Elastic Installation and Upgrade Guide [8.12] | Elastic
Installing the Elastic Stackedit When installing the Elastic Stack, you must use the same version across the entire stack. For example, if you are using Elasticsearch 8.12.1, you install Beats 8.12.1, APM Server 8.12.1, Elasticsearch Hadoop 8.12.1, Kibana
www.elastic.co
'구축 > ELK' 카테고리의 다른 글
| [Apache KAFKA] zookeeper, kafka 구축 (0) | 2024.04.01 |
|---|---|
| [ELK] Metricbeat (0) | 2024.03.12 |
| [ELK] Filebeat 설치 (0) | 2024.03.04 |
| [ELK] Logstash 설치 (0) | 2024.02.15 |
| [ELK] Kibana 설치 (0) | 2024.02.13 |