What is RabbitMQ

What is a message queue

Message Queue (MQ), message queue middleware. Many people say: MQ realizes the asynchrony and decoupling of applications by separating the sending and receiving of messages. This gives the intuition that MQ is asynchronous and used for decoupling, but this is only the effect of MQ, not Purpose.

The real purpose of MQ is for communication, shielding the underlying complex communication protocols, and defining a set of simpler communication protocols at the application layer. The communication between two modules in a distributed system is either HTTP or TCP developed by themselves, but these two protocols are actually primitive protocols.

Why doesn’t the message middleware directly use the HTTP protocol?

HTTP protocol is difficult to achieve two-end communication – module A can call B, and B can also actively call A. If you want to do this, both ends must be backed by WebServer, and long-term connections are not supported (HTTP 2.0 libraries cannot be found at all. arrive). TCP is even more primitive, with sticky packets, heartbeats, and private protocols. Just thinking about it makes your scalp tingle.

For a message middleware, its main responsibility is to be responsible for data transmission, storage, distribution, high performance and simplicity are what we pursue, while HTTP request headers and response headers are more complex, including cookies. , data encryption and decryption, window sill, response code and other additional functions, we do not need such complicated functions.

At the same time, in most cases, HTTP is mostly short links. In the actual interaction process, a request to a response is likely to be interrupted. After the interruption, the persistence will not be performed, which will cause the loss of the request. This is not conducive to the business scenario of the message middleware, because the message middleware may be a long-term process of obtaining information. If there is a problem or failure, the data or message must be persisted, etc. The purpose is to ensure the high reliability and reliability of the message and data. Robust operation.

All MQ has to do is build a simple “protocol” on top of these protocols – a producer/consumer model. The “protocol” that MQ brings to me is not a specific communication protocol, but a higher-level communication model. It defines two objects—the one that sends data is called the producer; the one that receives the data is called the consumer. It provides an SDK that allows us to define our own producer and consumer to implement message communication without ignoring the underlying communication protocol.

Usage scenarios for message queues

Asynchronous processing

Scenario description: After users register, they need to send a registration email and a registration SMS. The traditional approach has two serial and parallel ways.

1. Serial mode: After the registration information is successfully written into the database, send the registration email, and then send the registration SMS. After the above three tasks are all completed, return to the client.

2. Parallel mode: After the registration information is successfully written into the database, the registration message will be sent at the same time as the registration email. After the above three tasks are completed, return to the client.

The difference from serial is that parallelism can improve processing time.

Assuming that each of the three business nodes uses 50 milliseconds, without considering other overheads such as the network, the serial time is 150 milliseconds, and the parallel time may be 100 milliseconds.

As described in the above case, the performance (concurrency, throughput, response time) of the traditional system will have bottlenecks. How to solve this problem?

The introduction of message queue will not be necessary for business logic, asynchronous processing.

According to the above convention, the user’s response time is equivalent to the time when the registration information is written into the database, that is, 50 milliseconds. After registering an email, sending a short message and writing it to the message queue, it returns directly, so the speed of writing to the message queue is very fast and can be ignored, so the user’s response time may be 50 milliseconds. So after the architecture change, the throughput of the system increased to 20 QPS per second. It is 3 times higher than serial and 2 times higher than parallel.

Application decoupling

Scenario description: After the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system.

Disadvantages of traditional mode:

1. If the inventory system is inaccessible, the order destocking will fail, causing the order to fail.

2. The order system is coupled with the inventory system.

How to solve the above problems? The scheme after the introduction of the application message queue.

Order system: After the user places an order, the order system completes the persistent processing, writes the message to the message queue, and returns that the user’s order is successful.

Inventory system: Subscribe to the news of the order, and use the pull/push method to obtain the order information. The inventory system performs inventory operations according to the order information.

If: The inventory system cannot be used normally when placing an order. It does not affect the normal order placement, because after the order is placed, the order system writes to the message queue and no longer cares about other subsequent operations. Realize the application decoupling of the order system and the inventory system.

Traffic clipping

Traffic peak shaving is also a common scenario in message queues, and is generally widely used in spike or group grab activities.

Application scenario: spike activity, generally due to excessive traffic, the traffic surges and the application hangs. In order to solve this problem, it is generally necessary to join a message queue at the front end of the application, which can control the number of active people and relieve the high traffic that overwhelms the application in a short period of time .

After the user’s request is received by the server, it is first written to the message queue. If the length of the message queue exceeds the maximum number, the user request will be discarded directly or the error page will be jumped to, and the seckill business will perform subsequent processing according to the request information in the message queue.

AMQP and JMS

MQ is a model of message communication, not a concrete implementation. Now there are two mainstream ways to implement MQ: AMQP and JMS.

Differences and connections between the two:

• JMS defines a unified interface to unify message operations; AMQP is a format for unifying data interaction by specifying protocols, such as RabbitMQ.

• JMS restricts the use of the Java language; AMQP is just a protocol, not an implementation, so it is cross-language, such as RocketMQ.

Introduction to RabbitMQ

RabbitMQ is a message queue developed by the erlang language and implemented based on the AMQP (Advanced Message Queue Protocol) protocol. It is a communication method between applications. Message queues are widely used in distributed system development.

Official address: http://www.rabbitmq.com

Official tutorial: http://www.rabbitmq.com/getstarted.html

RabbitMQ originally originated from the financial system and is used to store and forward messages in a distributed system. It performs well in terms of ease of use, scalability, and high availability. Specific features include:

1. Reliability, RabbitMQ uses some mechanisms to ensure reliability, such as persistence, transmission confirmation, and release confirmation.

2. Flexible Routing routes messages through Exchange before they enter the queue. For typical routing functions, RabbitMQ already provides some built-in Exchange to implement. For more complex routing functions, multiple Exchanges can be bound together, and their own Exchanges can also be implemented through the plug-in mechanism.

3. Message clustering (Clustering), multiple RabbitMQ servers can form a cluster to form a logical Broker.

4. High Availability (Highly Available Queues), the queue can be mirrored on the machines in the cluster, so that the queue is still available when some nodes fail.

5. Multiple protocols (Multi-protocol), RabbitMQ supports multiple message queue protocols, such as STOMP, MQTT and so on.

6. Many Clients, RabbitMQ supports almost all common languages, such as Java, .NET, Ruby, etc.

7. Management UI, RabbitMQ provides an easy-to-use user interface that enables users to monitor and manage many aspects of the message broker.

8. Tracking mechanism (Tracing) If the message is abnormal, RabbitMQ provides a message tracking mechanism, the user can find out what happened.

9. Plug-in mechanism (Plugin System), RabbitMQ provides many plug-ins to be extended in many ways, and you can also write your own plug-ins.

Architecture Model of RabbitMQ

RabbitMQ as a whole is a producer and consumer model, mainly responsible for receiving, storing and forwarding messages. The process of message delivery can be imagined as: when you send a package to the post office, the post office will temporarily store and finally deliver the mail to the recipient through the postman. RabbitMQ is like a system composed of post office, mailbox and postman . In computer terms, the RabbitMQ model is more of a switch model.

Connection

Connection, as a client (whether it is a producer or a consumer), if you want to communicate with RabbitMQ, you must create a TCP connection. Of course, after the connection is established at the same time, the client must also send a “greeting message” to each other I know that we are all AMQP-compliant languages. For example, when you greet others, you usually say “Hello!”, and you usually say “hello!” with foreign beauties.

After you confirm the “language”, it is equivalent to the “authentication” between the client and RabbitMQ. You can create an AMQP channel between you.

Channel

Channels are the channels through which producers/consumers communicate with RabbitMQ. Channel is a virtual connection established on a TCP connection, what does that mean? That is to say, rabbitmq establishes hundreds of channels on a TCP to achieve multiple thread processing. This TCP is shared by multiple threads, each thread corresponds to a channel, and the channel has a unique ID in RabbitMQ, which ensures the privacy of the channel. , which corresponds to the only thread used.

Why not make multiple TCP connections?

Because it is very expensive for the operating system to create and destroy TCP, the system will consume a lot of performance to open up a TCP for each thread, and the creation and destruction of hundreds of TCPs per second will seriously consume the system. So rabbitmq chooses to establish multiple channels (virtual connections established on tcp) to connect to rabbit.

Technically, this is called multiplexing , and it’s useful for multithreaded or asynchronous applications that perform multiple tasks.

Message

A message contains a payload and a tag, the payload refers to the data to be transmitted, the tag describes the payload, and rabbitmq uses it to decide who gets the message, the consumer can only get the payload, and does not know who the producer is.

Producer

The producer, the creator of the message, sends to rabbitmq.

Consumer

Consumers, consumers of messages, connect to rabbitmq, subscribe to queues, consume messages, continuous subscriptions and single subscriptions.

Broker

The proxy service, in short, is the message queue server entity, the default port is 5672.

Exchange

The switch is used to receive the messages sent by the producer, and then send these messages to the queue according to the routing key. There are four main types, which will be introduced later.

Routing key

Routing rules, which the virtual machine uses to determine how to route a particular message, that is, Exchange delivers messages based on this keyword.

Binding

The virtual connection between Exchange and Queue, its function is to bind exchange and queue according to routing rules, Binding can include multiple Routing keys.

Queue

A message queue, used to hold messages until they are sent to consumers. It is the container for the message and the destination for the message. A message can be put into one or more queues. The message has been in the queue, waiting for the consumer to connect to the queue to take it away.

The relationship between exchanges, queues, bindings, routing keys :

The queue is bound to the exchange through the routing key, the producer publishes the message to the exchange, the exchange routes the message to a specific queue according to the bound routing key, and then is received by the consumers who subscribe to this queue.

Virtual Host

A virtual host, representing a collection of exchanges, message queues, and related objects. Virtual hosts are separate server domains that share the same authentication and encryption environment. Each vhost is essentially a mini version of the RabbitMQ server, with its own queues, exchanges, bindings and permission mechanisms. The vhost is the basis of the AMQP concept and must be specified when connecting. The default vhost of RabbitMQ is /, which is accessed by the default user and password guest.

switch type

There are four kinds of direct, fanout, topic, and headers. The headers (almost the same as direct) are not practical and can be ignored.

direct

The routing key matches exactly, the message is delivered to the corresponding queue, and the direct exchange is the default exchange. ExChange will send the message to a Queue that exactly matches the ROUTING_KEY.

fanout

The message is broadcast to the bound queue, no matter what routing key is bound to the queue, the message passes through the exchange, and each queue has a copy.

topic

By processing with * and * wildcards, messages from different origins arrive at the same queue, and the routing key is divided into several identifiers by . * matches 1 identifier after it, # matches one or more identifiers . Such as:

 user.# # can match to user.add user.add.batch user.* # can only match user.add, not user.add.batch  

The text and pictures in this article are from InfoQ