Introduction and installation of Pulsar

Original link: http://www.nosuchfield.com/2022/06/21/Introduction-and-installation-of-Pulsar/

Introduction

Apache Pulsar is a distributed message queue, which mainly consists of the following three parts.

components effect
Broker Responsible for producer and consumer requests as well as message replication and distribution, Broker is stateless and does not store data
Zookeeper Store metadata, cluster configuration, responsible for task coordination and service discovery, etc.
Bookkeeper Message data and persistent storage of cursors data, each storage node of Bookkeeper is called bookie

The producer sends data to Pulsar, the consumer receives data from Pulsar, and the process of the consumer receiving data is called subscription. Pulsar has four subscription models

schema name model
exclusive A subscription can only have one consumer. If multiple consumers use the same subscription to subscribe to a topic, an error will be reported
failover A subscription allows multiple consumers, but only one consumer will work. When the current consumer loses connection, other consumers will go online for consumption
shared Multiple consumers can use the same subscription to subscribe to a topic, and messages are sent to consumers by polling
Shared key (key_shared) Similar to the above, except that the message will be sent to different consumers according to the key

Broker does not store data, data is stored on Bookkeeper. The topic will be divided into multiple partitions, the partitions will be allocated to different brokers, and the producer and consumer will connect to the partitions on the broker to send and receive data. Pulsar supports multi-level topics, you can set whether to persist and the name of the tenant, namespace and topic

 {persistent|non-persistent}://tenant/namespace/topic

Install

We have three nodes 172.19.67.171 , 172.19.67.190 , 172.19.67.202 . First, we download the Pulsar installation package on these three nodes

 useradd pulsarsu - pulsarwget https://archive.apache.org/dist/pulsar/pulsar-2.10.0/apache-pulsar-2.10.0-bin.tar.gztar -zxvf apache-pulsar-2.10.0-bin.tar.gzcd apache-pulsar-2.10.0

1. Install the Zookeeper cluster and initialize the Pulsar cluster metadata

Modify the conf/zookeeper.conf configuration file of each node and add the following configuration

 server.1=172.19.67.171:2888:3888server.2=172.19.67.190:2888:3888server.3=172.19.67.202:2888:3888

Then, for the serial number of each node, set the corresponding serial number to the myid file in the dataDir directory of the node

 mkdir -p data/zookeeperecho 1 > data/zookeeper/myid

After modifying the configuration file, start the Zookeeper service on each node

 bin/pulsar-daemon start zookeeper

After starting the cluster, use the following command to write the metadata of Pulsar

 bin/pulsar initialize-cluster-metadata \ --cluster pulsar-cluster-1 \ --zookeeper 172.19.67.171:2181 \ --configuration-store 172.19.67.171:2181 \ --web-service-url http://172.19.67.171:8080,172.19.67.190:8080,172.19.67.202:8080 \ --broker-service-url pulsar://172.19.67.171:6650,172.19.67.190:6650,172.19.67.202:6650

The specific meaning is as follows

parameter meaning
cluster Pulsar’s cluster name
zookeeper Zookeeper’s address
configuration-store Configure the storage address, use the Zookeeper address
web-service-url The address of the Pulsar cluster web service
broker-service-url The address of the broker service

2. Install the Bookkeeper cluster

Set the conf/bookkeeper.conf configuration file of all nodes and add the Zookeeper connection information

 zkServers=172.19.67.171:2181,172.19.67.190:2181,172.19.67.202:2181

Then start the Bookkeeper service on each node

 bin/pulsar-daemon start bookie

Then use the following command to verify the cluster status of Bookkeeper

 bin/bookkeeper shell simpletest --ensemble 3 --writeQuorum 3 --ackQuorum 3 --numEntries 3

3. Install Pulsar Brokers

Modify the conf/broker.conf configuration file of all nodes

 # 配置pulsar broker连接的zookeeper集群地址zookeeperServers=172.19.67.171:2181,172.19.67.190:2181,172.19.67.202:2181configurationStoreServers=172.19.67.171:2181,172.19.67.190:2181,172.19.67.202:2181# broker数据端口brokerServicePort=6650# broker web服务端口webServicePort=8080# pulsar 集群名字,和前面zookeeper初始化集群元数据时配置的一样clusterName=pulsar-cluster-1# 创建一个ledger时使用的bookie数量managedLedgerDefaultEnsembleSize=2# 每个消息的副本数量managedLedgerDefaultWriteQuorum=2# 完成写操作前等待副本ack的数量managedLedgerDefaultAckQuorum=2

Then start the Pulsar service on each node

 bin/pulsar-daemon start broker

Send and consume messages using the client

Modify the conf/client.conf file

 webServiceUrl=http://172.19.67.171:8080,172.19.67.190:8080,172.19.67.202:8080brokerServiceUrl=pulsar://172.19.67.171:6650,172.19.67.190:6650,172.19.67.202:6650

Then use the client to consume

 bin/pulsar-client consume \ persistent://public/default/pulsar-test \ -n 100 \ -s "consumer-test" \ -t "Exclusive"

Open a new window and use the producer to send messages

 bin/pulsar-client produce \ persistent://public/default/pulsar-test \ -n 1 \ -m "hello, this is a test for Pulsar"

refer to

Pulsar introduction and deployment

This article is reprinted from: http://www.nosuchfield.com/2022/06/21/Introduction-and-installation-of-Pulsar/
This site is for inclusion only, and the copyright belongs to the original author.

Leave a Comment

Your email address will not be published.