Original link: http://www.nosuchfield.com/2022/06/21/Introduction-and-installation-of-Pulsar/
Introduction
Apache Pulsar is a distributed message queue, which mainly consists of the following three parts.
| components | effect |
|---|---|
| Broker | Responsible for producer and consumer requests as well as message replication and distribution, Broker is stateless and does not store data |
| Zookeeper | Store metadata, cluster configuration, responsible for task coordination and service discovery, etc. |
| Bookkeeper | Message data and persistent storage of cursors data, each storage node of Bookkeeper is called bookie |
The producer sends data to Pulsar, the consumer receives data from Pulsar, and the process of the consumer receiving data is called subscription. Pulsar has four subscription models
| schema name | model |
|---|---|
| exclusive | A subscription can only have one consumer. If multiple consumers use the same subscription to subscribe to a topic, an error will be reported |
| failover | A subscription allows multiple consumers, but only one consumer will work. When the current consumer loses connection, other consumers will go online for consumption |
| shared | Multiple consumers can use the same subscription to subscribe to a topic, and messages are sent to consumers by polling |
| Shared key (key_shared) | Similar to the above, except that the message will be sent to different consumers according to the key |
Broker does not store data, data is stored on Bookkeeper. The topic will be divided into multiple partitions, the partitions will be allocated to different brokers, and the producer and consumer will connect to the partitions on the broker to send and receive data. Pulsar supports multi-level topics, you can set whether to persist and the name of the tenant, namespace and topic
{persistent|non-persistent}://tenant/namespace/topic
Install
We have three nodes 172.19.67.171 , 172.19.67.190 , 172.19.67.202 . First, we download the Pulsar installation package on these three nodes
useradd pulsarsu - pulsarwget https://archive.apache.org/dist/pulsar/pulsar-2.10.0/apache-pulsar-2.10.0-bin.tar.gztar -zxvf apache-pulsar-2.10.0-bin.tar.gzcd apache-pulsar-2.10.0
1. Install the Zookeeper cluster and initialize the Pulsar cluster metadata
Modify the conf/zookeeper.conf configuration file of each node and add the following configuration
server.1=172.19.67.171:2888:3888server.2=172.19.67.190:2888:3888server.3=172.19.67.202:2888:3888
Then, for the serial number of each node, set the corresponding serial number to the myid file in the dataDir directory of the node
mkdir -p data/zookeeperecho 1 > data/zookeeper/myid
After modifying the configuration file, start the Zookeeper service on each node
bin/pulsar-daemon start zookeeper
After starting the cluster, use the following command to write the metadata of Pulsar
bin/pulsar initialize-cluster-metadata \ --cluster pulsar-cluster-1 \ --zookeeper 172.19.67.171:2181 \ --configuration-store 172.19.67.171:2181 \ --web-service-url http://172.19.67.171:8080,172.19.67.190:8080,172.19.67.202:8080 \ --broker-service-url pulsar://172.19.67.171:6650,172.19.67.190:6650,172.19.67.202:6650
The specific meaning is as follows
| parameter | meaning |
|---|---|
| cluster | Pulsar’s cluster name |
| zookeeper | Zookeeper’s address |
| configuration-store | Configure the storage address, use the Zookeeper address |
| web-service-url | The address of the Pulsar cluster web service |
| broker-service-url | The address of the broker service |
2. Install the Bookkeeper cluster
Set the conf/bookkeeper.conf configuration file of all nodes and add the Zookeeper connection information
zkServers=172.19.67.171:2181,172.19.67.190:2181,172.19.67.202:2181
Then start the Bookkeeper service on each node
bin/pulsar-daemon start bookie
Then use the following command to verify the cluster status of Bookkeeper
bin/bookkeeper shell simpletest --ensemble 3 --writeQuorum 3 --ackQuorum 3 --numEntries 3
3. Install Pulsar Brokers
Modify the conf/broker.conf configuration file of all nodes
# 配置pulsar broker连接的zookeeper集群地址zookeeperServers=172.19.67.171:2181,172.19.67.190:2181,172.19.67.202:2181configurationStoreServers=172.19.67.171:2181,172.19.67.190:2181,172.19.67.202:2181# broker数据端口brokerServicePort=6650# broker web服务端口webServicePort=8080# pulsar 集群名字，和前面zookeeper初始化集群元数据时配置的一样clusterName=pulsar-cluster-1# 创建一个ledger时使用的bookie数量managedLedgerDefaultEnsembleSize=2# 每个消息的副本数量managedLedgerDefaultWriteQuorum=2# 完成写操作前等待副本ack的数量managedLedgerDefaultAckQuorum=2
Then start the Pulsar service on each node
bin/pulsar-daemon start broker
Send and consume messages using the client
Modify the conf/client.conf file
webServiceUrl=http://172.19.67.171:8080,172.19.67.190:8080,172.19.67.202:8080brokerServiceUrl=pulsar://172.19.67.171:6650,172.19.67.190:6650,172.19.67.202:6650
Then use the client to consume
bin/pulsar-client consume \ persistent://public/default/pulsar-test \ -n 100 \ -s "consumer-test" \ -t "Exclusive"
Open a new window and use the producer to send messages
bin/pulsar-client produce \ persistent://public/default/pulsar-test \ -n 1 \ -m "hello, this is a test for Pulsar"
refer to
Pulsar introduction and deployment
This article is reprinted from: http://www.nosuchfield.com/2022/06/21/Introduction-and-installation-of-Pulsar/
This site is for inclusion only, and the copyright belongs to the original author.