建站
咨詢
售后
建站咨詢
新聞中心
這里有您想知道的互聯(lián)網(wǎng)營銷解決方案
CentOS6.5下Kafka安裝部署
系統(tǒng)環(huán)境
| 組件 | 版本 |
|---|---|
| centos | 6.5 64x |
| zookeeper | 3.4.5 |
| kafka | 2.10-0.8.1.1 |
單節(jié)點(diǎn)安裝
下載kafka并且解壓
tar zxvf kafka_2.10-0.8.1.1.tar.gz
cd kafka_2.10-0.8.1.1/啟動kafka默認(rèn)配置
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties創(chuàng)建 topic 名為 “test”
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
#列出 topic
./bin/kafka-topics.sh --list --zookeeper localhost:2181創(chuàng)建客戶端
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test創(chuàng)建消費(fèi)端
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning集群配置
修改server-1.properties中的參數(shù)
cp config/server.properties config/server-1.properties主要修改內(nèi)容
broker.id=0
log.dirs=/home/Hadoop/development/src/kafka_2.10-0.8.1.1/logs
zookeeper.connect=canbot130:2181,canbot131:2181,canbot132:2181修改完成后 copy 到其他節(jié)點(diǎn)
scp -r ./kafka_2.10-0.8.1.1/ hadoop@canbot131:/home/hadoop/development/src/
scp -r ./kafka_2.10-0.8.1.1/ hadoop@canbot132:/home/hadoop/development/src/copy完以后需要修改再修改 server-1.properties 中的 broker.id
broker.id=0 192.169.2.130
broker.id=1 192.169.2.131
broker.id=2 192.169.2.132啟動Kafka
分別在canbot130/1/2三個節(jié)點(diǎn)都啟動
./kafka_2.10-0.8.1.1/bin/kafka-server-start.sh ./kafka_2.10-0.8.1.1/config/server-1.properties &創(chuàng)建集群 Topic
[hadoop@canbot130 kafka_2.10-0.8.1.1]$./bin/kafka-topics.sh --create --zookeeper canbot130:2181 --replication-factor 3 --partitions 1 --topic test提示以下內(nèi)容表示創(chuàng)建Topic 成功
Created topic "test".查看 Topic 列表
[hadoop@canbot130 kafka_2.10-0.8.1.1]$ ./bin/kafka-topics.sh --list --zookeeper canbot130:2181
test
[hadoop@canbot130 kafka_2.10-0.8.1.1]$ 創(chuàng)建生產(chǎn)者
./bin/kafka-console-consumer.sh --zookeeper canbot130:2181 --topic test使用該命令創(chuàng)建生產(chǎn)者,然后將在canbot132節(jié)點(diǎn)上創(chuàng)建 消費(fèi)者,查看消息是否被消費(fèi)
創(chuàng)建消費(fèi)者
在 canbot132 節(jié)點(diǎn)上執(zhí)行
./bin/kafka-console-consumer.sh --zookeeper canbot130:2181 --topic test生產(chǎn)消息==>消費(fèi)消息
在canbot130節(jié)點(diǎn)上的生產(chǎn)者
[hadoop@canbot130 kafka_2.10-0.8.1.1]$ ./bin/kafka-console-producer.sh --broker-list canbot130:9092 --topic test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
"holl"
[2016-05-31 04:27:30,770] INFO Closing socket connection to /192.168.2.130. (kafka.network.Processor)
"hao xiang shi tong bu l haha"
"test kafka"在canbot132節(jié)點(diǎn)上的消費(fèi)者所產(chǎn)生的信息
[hadoop@canbot132 kafka_2.10-0.8.1.1]$ ./bin/kafka-console-consumer.sh --zookeeper canbot130:2181 --topic test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2016-05-31 04:27:22,328] INFO Closing socket connection to /192.168.2.132. (kafka.network.Processor)
holl
hao xiang shi tong bu l haha
test kafka錯誤記錄
錯誤一
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/1. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:205)
at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:57)
at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:44)
at kafka.server.KafkaServer.startup(KafkaServer.scala:103)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)解決方法:該錯誤是由于server.properties 中的broker.id 重復(fù)
網(wǎng)頁題目：CentOS6.5下Kafka安裝部署
網(wǎng)址分享：http://www.5511xx.com/article/djjjsjd.html
