kafka 설치 및 실행
by jsh
1. Docker 이미지 선택
[root@tivlxdkkfs-kafka kafka]# docker search kafka
INDEX NAME DESCRIPTION STARS OFFICIAL AUTOMATED
docker.io docker.io/wurstmeister/kafka Multi-Broker Apache Kafka Image 1352 [OK]
docker.io docker.io/spotify/kafka A simple docker image with both Kafka and ... 411 [OK]
docker.io docker.io/bitnami/kafka Apache Kafka is a distributed streaming pl... 231 [OK]
docker.io docker.io/sheepkiller/kafka-manager kafka-manager 203 [OK]
docker.io docker.io/kafkamanager/kafka-manager Docker image for Kafka manager 118
docker.io docker.io/ches/kafka Apache Kafka. Tagged versions. JMX. Cluste... 117 [OK]
docker.io docker.io/hlebalbau/kafka-manager CMAK (previous known as Kafka Manager) As ... 82 [OK]
docker.io docker.io/landoop/kafka-topics-ui UI for viewing Kafka Topics config and dat... 36 [OK]
docker.io docker.io/johnnypark/kafka-zookeeper Kafka and Zookeeper combined image 22
docker.io docker.io/debezium/kafka Kafka image required when running the Debe... 21 [OK]
docker.io docker.io/solsson/kafka http://kafka.apache.org/documentation.html... 21 [OK]
docker.io docker.io/landoop/kafka-lenses-dev Lenses with Kafka. +Connect +Generators +C... 20 [OK]
docker.io docker.io/danielqsj/kafka-exporter Kafka exporter for Prometheus 18 [OK]
docker.io docker.io/landoop/kafka-connect-ui Web based UI for Kafka Connect. 17 [OK]
docker.io docker.io/digitalwonderland/kafka Latest Kafka - clusterable 15 [OK]
docker.io docker.io/tchiotludo/kafkahq Kafka GUI to view topics, topics data, con... 6 [OK]
docker.io docker.io/solsson/kafka-manager Deprecated in favor of solsson/kafka:cmak 5 [OK]
docker.io docker.io/solsson/kafkacat https://github.com/edenhill/kafkacat/pull/110 5 [OK]
docker.io docker.io/solsson/kafka-prometheus-jmx-exporter For monitoring of Kubernetes Kafka cluster... 4 [OK]
docker.io docker.io/mesosphere/kafka-client Kafka client 3 [OK]
docker.io docker.io/anchorfree/kafka Kafka broker and Zookeeper image 2
docker.io docker.io/digitsy/kafka-magic Kafka Magic images 2
docker.io docker.io/openwhisk/kafkaprovider Apache OpenWhisk event provider service fo... 2 [OK]
docker.io docker.io/zenko/kafka-manager Kafka Manger https://github.com/yahoo/kafk... 2 [OK]
docker.io docker.io/zenreach/kafka-connect Zenreach's Kafka Connect Docker Image 2
- Docker kafka 이미지 중 wurstmeister/kafka 가 가장 인기가 많고 Document도 잘 되어 있음.
2. 사전준비
docker, docker-compose 설치
- docker 설치
[root@tivlxdkkfs-kafka /]# yum install docker -y
- 부팅 시 자동실행 활성화
[root@tivlxdkkfs-kafka /]# systemctl enabled docker.service
- docker 시작
[root@tivlxdkkfs-kafka /]# systemctl start docker.service
- docker-compose 설치
[root@tivlxdkkfs-kafka /]# sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- docker-compose 실행 권한
[root@tivlxdkkfs-kafka /]# sudo chmod +x /usr/local/bin/docker-compose
- 심볼릭 링크가 필요한 경우는 아래 명령어 실행
[root@tivlxdkkfs-kafka /]# sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
- docker-compose 설치 확인
[root@tivlxdkkfs-kafka /]# docker-compose --version
docker-compose version 1.29.1, build c34c88b2
3. docker-compose.yml 파일 작성
[root@tivlxdkkfs-kafka docker]# mkdir kafka
[root@tivlxdkkfs-kafka docker]# cd kafka
[root@tivlxdkkfs-kafka docker]# vi docker-compose.yml
version: '2'
networks:
test:
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
ports:
- "2181:2181"
networks:
- test
restart: always
kafka:
image: wurstmeister/kafka:2.13-2.6.0
container_name: kafka
depends_on:
- zookeeper
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.253.8:9092
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ADVERTISED_HOST_NAME: 192.168.253.8
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
# - /var/run/docker.sock:/var/run/docker.sock
- ./data:/var/lib/kafka/data
ports:
- "9092:9092"
networks:
- test
restart: always
4. 실행 및 테스트
- kafka/zookeeper 실행
[root@tivlxdkkfs-kafka kafka]# docker-compose up
Creating network "kafka_test" with the default driver
Creating zookeeper ... done
Creating kafka ... done
Attaching to zookeeper, kafka
......
- kafka/zookeeper container 확인
[root@tivlxdkkfs-kafka kafka]# docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0ac70cdba1e wurstmeister/kafka:2.13-2.6.0 "start-kafka.sh" 25 seconds ago Up 23 seconds 0.0.0.0:9092->9092/tcp kafka
bf9ce26a04f9 wurstmeister/zookeeper "/bin/sh -c '/usr/..." 27 seconds ago Up 24 seconds 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp zookeeper
[root@tivlxdkkfs-kafka kafka]# docker container exec -it kafka bash
- Topic 생성
bash-4.4# /opt/kafka/bin/kafka-topics.sh --create \
--zookeeper 172.17.0.1:2181 \
--replication-factor 1 \
--partitions 3 \
--topic test01
Created topic test01.
- Topic 확인
bash-4.4# /opt/kafka/bin/kafka-topics.sh --list \
--zookeeper 172.18.0.1:2181
test01
- Topic Message 전송
bash-4.4# /opt/kafka/bin/kafka-console-producer.sh \
--broker-list 172.18.0.1:9092 \
--topic test01
> hello kwic
>
- Topic Message 확인
bash-4.4# /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server 172.18.0.1:9092 \
--topic test01 \
--from-beginning
hello kwic
- Consumer Group Test
bash-4.4# /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server 172.18.0.1:9092 \
--topic test01 -group testgroup \
--from-beginning
hello kwic
- Consumer Group 확인
bash-4.4# /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server 172.18.0.1:9092 \
--list testgroup
testgroup
- 특정 Group 상세 확인
bash-4.4# /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server 172.18.0.1:9092 \
--group testgroup \
--describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
testgroup test01 0 1 1 0 consumer-testgroup-1-2619497e-9b18-4aae-ba55-c5fac2852715 /172.18.0.1 consumer-testgroup-1
testgroup test01 1 1 1 0 consumer-testgroup-1-2619497e-9b18-4aae-ba55-c5fac2852715 /172.18.0.1 consumer-testgroup-1
testgroup test01 2 0 0 0 consumer-testgroup-1-2619497e-9b18-4aae-ba55-c5fac2852715 /172.18.0.1 consumer-testgroup-1
- PARTITION : 0~2까지 partition 3개를 확인 가능
- CURRENT-OFFSET : Consumer offset (BY testgroup)
- LOG-END-OFFSET : 현재 Topic의 마지막 offset
- LAG : Consumer 랙
- CURRENT-OFFSET와 LOG-END-OFFSET의 offset 차이
- Consumer가 가져가지 않은 양의 크기
- LAG의 수치가 클수록 실시간으로 빠르게 처리를 못하고 있다는 반증
- offset reset
bash-4.4# /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server 172.18.0.1:9092 \
--group testgroup \
--topic test01 \
--reset-offsets \
--to-earliest \
--execute
GROUP TOPIC PARTITION NEW-OFFSET
testgroup test01 0 0
testgroup test01 1 0
testgroup test01 2 0
- Current offset 초기화 부분 확인과 그에 따른 LAG 증가 확인 가능
bash-4.4# /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server 172.18.0.1:9092 \
--group testgroup \
--describe
Consumer group 'testgroup' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
testgroup test01 0 0 1 1 - - -
testgroup test01 1 0 2 2 - - -
testgroup test01 2 0 2 2 - - -
- Topic 삭제
bash-4.4# /opt/kafka/bin/kafka-topics.sh \
--delete \
--zookeeper 172.18.0.1:2181 \
--topic test01
Topic test01 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
Subscribe via RSS