A Preliminary Study of Crooked Neck Scott

Original link: https://blog.2333332.xyz/2022/05/22/2022-05-22-a-guide-to-websocket/

banner: https://www.pixiv.net/artworks/98483067

sequence

The end of the week is approaching in a blink of an eye, and the fact that the diligent group friends are secretly rolling out makes me, a truly rotten person, a little anxious. Some group friends have completed many exams, some group friends have completed many courses, and some group friends have obtained a lot of achievements in ” We Were Here Forever” and fast-passed a lot of gals, you can go on Obviously it has nothing to do with me who skip class every day, which is amazing. One day when I woke up at eight as usual, ( I found that the teacher has already arranged a lesson in the DingTalk group:

But the choice of a topic that I don’t know makes me difficult. At this time, I thought of the almighty group friends:

Since the group of friends can spend less than 40 minutes in the middle of the night from not knowing python to understanding how to write a ws server in python, then I will try it out. As for why the title is Crooked Neck Scott, this is of course also because the above-mentioned group friend, AKA lot ^[1] , personally said that its pronunciation is “WebScott”, and believes that this protocol is the communication between the server and the IoT device. Excellent solution. Although I have some doubts about its pronunciation and usage scenarios, professional people should obey the boss when they do professional things.

Since it is an online course design, it is best to study something a little lower-level. So had to read a stinky and long RFC like Farex’s game library . A Request for Comments (RFC) is a publication in a series – Wikipedia describes it as such. The book also specifically mentions that RFC is “request for comments” ^[2] , which feels strange. Wait until I see RFC 6455 It was found to be very fatal: rfc6455 has a full 70 pages, while the well-known FTP protocol ^[3] has only 68 pages, and the new HTTP/2 ^[4] is only 95 pages. At this time, I have to envy the gold content of Haisheng Level 6 680 . RFC 6455 is currently in the Proposed Standard stage and is a proposed standard.

Looking at the example of the table above, I found that most of the fixed topics are simulated using python. This might be simpler, but it’s not fun. There is a difference between simulation and implementation. In the same way, just find a package and call it to realize the multiplayer flying chess that I have forgotten the rules. It seems that the focus is on the front end. In this blog post, we will try to implement websocket on the server side.

shake hands

client

1.3. Opening Handshake
_This section is non-normative._
The opening handshake is intended to be compatible with HTTP-based server-side software and intermediaries, so that a single port can be used by both HTTP clients talking to that server and WebSocket clients talking to that server. To this end, the WebSocket client’s handshake is an HTTP Upgrade request:

 GET /chat HTTP/1.1
 Host: server.example.com
 Upgrade: websocket
 Connection: Upgrade
 Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
 Origin: http://example.com
 Sec-WebSocket-Protocol: chat, superchat
 Sec-WebSocket-Version: 13

We can see that the client-side handshake request is compatible with the HTTP protocol for port multiplexing.

Then we first build a front-end page to test ^[5] :

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 twenty one
 twenty two
 twenty three
 twenty four
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64

 <!DOCTYPE html >
 < html >

 < head lang = "en" >
 < meta charset = "UTF-8" >
 < title > </ title >
 </ head >

 < body >
 < div >
 < input type = "text" id = "txt" />
 < input type = "button" id = "btn" value = "Submit" onclick = "sendMsg();" />
 < input type = "button" id = "open" value = "Open Connection" onclick = "openConn();" />
 < input type = "button" id = "close" value = "close connection" onclick = "closeConn();" />
 </ div >
 < div id = "content" > </ div >

 < script type = "text/javascript" >
 var socket;
 function openConn ( ) {
 socket = new WebSocket ( "ws://127.0.0.1:8003/chat" );

 socket.onopen = function ( ) {
 /* After the connection to the server is successful, execute automatically */

 var newTag = document . createElement ( 'div' );
 newTag. innerHTML = "[Connection succeeded]" ;
 document . getElementById ( 'content' ). appendChild (newTag);
 };

 socket.onmessage = function ( event ) {
 /* When the server sends data to the client, it will automatically execute */
 var response = event.data ;
 var newTag = document . createElement ( 'div' );
 newTag. innerHTML = response;
 document . getElementById ( 'content' ). appendChild (newTag);
 };

 socket.onclose = function ( event ) {
 /* Automatically execute when the server side actively disconnects */
 socket = undefined ;
 var newTag = document . createElement ( 'div' );
 newTag. innerHTML = "[Close the connection, onclose callback]" ;
 document . getElementById ( 'content' ). appendChild (newTag);
 };
 }

 function sendMsg ( ) {
 var txt = document . getElementById ( 'txt' );
 socket.send ( txt.value );
 txt.value = "" ;
 }

 function closeConn ( ) {
 socket.close ();
 var newTag = document . createElement ( 'div' );
 newTag. innerHTML = "[Close connection]" ;
 document . getElementById ( 'content' ). appendChild (newTag);
 }

    </ script >
 </ body >

 </ html >

If the backend wants to be implemented, it is inseparable from the socket. The book also said ^[6] that the same term socket can mean a variety of different meanings. Here, both the endpoint of the TCP connection and the socket API are used.

 ADDRESS = ( '127.0.0.1' , 8003 )
 BUFSIZE = 4096


 def main ():

 # Create a TCP Socket, AF_INET is an address family for IPv4 protocol; SOCK_STREAM is a Socket type that provides stream-oriented TCP transmission
 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

 sock.bind(ADDRESS)
 sock.listen()

 conn, address = sock.accept() # block waiting for connection
 print ( 'connected by:' , address)
 data = b''
 while True :
 _ = conn.recv(BUFSIZE)
 if not _:
 break
 data += _
 print (data.decode())

Then the front end clicks to open the connection, the browser will automatically send a handshake request, the data is as follows:

 GET /chat HTTP/1.1
 Host: 127.0.0.1:8003
 Connection: Upgrade
 Pragma: no-cache
 Cache-Control: no-cache
 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36
 Upgrade: websocket
 Origin: null
 Sec-WebSocket-Version: 13
 Accept-Encoding: gzip, deflate, br
 Accept-Language: zh-CN,zh;q=0.9,ja;q=0.8,en-US;q=0.7,en;q=0.6
 Sec-WebSocket-Key: f03NpU0iebD9rbIRGj8NPg==
 Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

As you can see, in addition to the regular request headers, there are some unique ws:

Connection: Upgrade #Set the value of the Connection header to “Upgrade” to indicate that this is an upgrade request
Upgrade: websocket # The Upgrade header specifies the upgrade to the ws protocol
Sec-WebSocket-Key # The random number after base64, the subsequent server side needs to use this value to generate Sec-WebSocket-Accept
Sec-WebSocket-Protocol # sub-protocol, optional
Sec-WebSocket-Version # must be 13
Sec-WebSocket-Extensions # optional, specifies the extensions supported by the client

Service-Terminal

When the server receives the handshake request, it should send back a special response that the protocol will change from HTTP to WebSocket. It looks like this:

 HTTP/1.1 101 Switching Protocols
 Upgrade: websocket
 Connection: Upgrade
 Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

How to calculate Sec-WebSocket-Accept , connect the Sec-WebSocket-Key sent by the client with “258EAFA5-E914-47DA-95CA-C5AB0DC85B11” (it’s a “magic string”), encode the result with SHA-1, and then Encode once with base64 and you’re good to go.

 def get_sec_websocket_accept ( key ):
 value = key + '258EAFA5-E914-47DA-95CA-C5AB0DC85B11'
 ac = base64.b64encode(hashlib.sha1(value.encode( 'utf-8' )).digest())
 return ac.decode( 'utf-8' )

Then we can respond to the handshake request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 twenty one
 twenty two
 twenty three
 twenty four
 25
 26
 27
 28
 29
 30
 31
 32
 33

 @classmethod
 def check_headers_and_return_ac ( cls, data: bytes ):
 header_dict = {}
 header = data.decode( 'utf-8' )
 header_list = header.split( '\r\n' )

 assert header_list[ 0 ] == 'GET /chat HTTP/1.1'
 for line in header_list[ 1 :]:
 key, value = line.split( ': ' )
 header_dict[key] = value

 assert header_dict[ 'Upgrade' ] == 'websocket'
 assert header_dict[ 'Connection' ] == 'Upgrade'
 assert header_dict[ 'Sec-WebSocket-Key' ]
 assert header_dict[ 'Sec-WebSocket-Version' ] == '13'
 return cls.get_sec_websocket_accept(header_dict[ 'Sec-WebSocket-Key' ])

 def accept_connection ( self ):
 headers = b''
 while True :
 _ = self.conn.recv( 1 )
 if not _:
 raise socket.error(socket.EBADF, 'Bad file descriptor' )
 headers += _
 if headers.endswith( b'\r\n\r\n' ):
 break
 headers = headers[:- 4 ]
 ac = self.check_headers_and_return_ac(headers)
 ac_headers = "HTTP/1.1 101 Switching Protocols\r\n" \
 "Upgrade:websocket\r\n" \
 "Connection:Upgrade\r\n" \
 f"Sec-WebSocket-Accept: {ac} \r\n\r\n"
 self.conn.sendall(ac_headers.encode( "utf8" ))

At this point, the handshake phase ends.

Data Frame

The structure is as follows:

 0 1 2 3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-------+-+-------------+----------------- --------------+
 |F|R|R|R| opcode|M| Payload len | Extended payload length |
 |I|S|S|S| (4) |A| (7) | (16/64) |
 |N|V|V|V| |S| | (if payload len==126/127) |
 | |1|2|3| |K| | |
 +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - - +
 | Extended payload length continued, if payload len == 127 |
 + - - - - - - - - - - - - - - - +-------------------------------+
 | |Masking-key, if MASK set to 1 |
 +--------------------------------------------+----------------- --------------+
 | Masking-key (continued) | Payload Data |
 +-------------------------------- - - - - - - - - - - - - - - - +
 : Payload Data continued ... :
 + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
 | Payload Data continued ... |
 +------------------------------------------------- --------------+

read frame

When processing data, let’s first consider a problem: how to read data with a length of bufsize (bytes) from the socket ? Like I must come directly to a socket.recv(bufsize) , the pursuit is a simple and rude. But I know that the big guys like Haisheng and Captain Hainan who are deeply involved in quantitative trading, web3 and NFT acquisition will be more delicate:

 def _read_strict ( self, bufsize ):
 remaining = bufsize
 _ bytes = b""
 while remaining:
 _buffer = self.conn.recv(remaining)
 if not _buffer:
 raise socket.error(socket.EBADF, 'Bad file descriptor' )
 _ bytes += _buffer
 remaining = bufsize - len (_ bytes )
 return_bytes _

The above code has many advantages – not only prevents the length of the read data from being smaller than bufsize (which is entirely possible), but also throws an exception when the socket is suddenly closed.

Next we can take a look at Frame according to RFC 5.2:

FIN: 1 bit
Indicates that this is the final fragment in a message. The first fragment MAY also be the final fragment.
RSV1, RSV2, RSV3: 1 bit each
Should always be 0 when no extension is negotiated.
Opcode: 4 bits
Details below.
Mask: 1 bit
Details below.
Payload length: 7 bits, 7+16 bits, or 7+64 bits
The length of the “Payload data”. In bytes. If 0-125, this is the payload length. If 126, the following two bytes are interpreted as a 16-bit unsigned integer which is the payload length. If 127, the following eight bytes are interpreted as a 64-bit unsigned integer that is the payload length. The byte arrangement is (the most significant bit MUST be 0) , which is big endian.
I’m here to add my poor computer basics to understand, so don’t laugh at me like the network engineering tycoon and the civil computer crossover tycoon . The maximum value of an unsigned integer that can be represented by 7 bits is 2^7-1=127 , which corresponds to the above 7+64 bits . The size of 1 byte (byte) is 8 bits, so the above can also be expressed as 7 bits, 7 bits + 2 bytes, or 7 bits + 8 bytes .
Masking-key: 0 or 4 bytes
Details below.
Payload data: Payload length
Cuz no extension has been negotiated , and the length of Payload data is the Payload length in point 5.

Then you can get each field from the byte stream, as long as the corresponding bit is ANDed with 1:

 header_bytes = self._read_strict( 2 )
 b1 = header_bytes[ 0 ]
 fin = b1 >> 7 & 1
 opcode = b1 & 0b1111
 b2 = header_bytes[ 1 ]
 mask = b2 >> 7 & 1
 length = b2 & 0b1111111

 length_data = ""
 if length == 126 :
 length_data = self._read_strict( 2 )
 length = struct.unpack( "!H" , length_data)[ 0 ]
 elif length == 127 :
 length_data = self._read_strict( 8 )
 length = struct.unpack( "!Q" , length_data)[ 0 ]

 The defeat of Saigo Shenglong in the Southwest War marked the end of the Japanese gentry era, and the casualties of the two armies were almost the same. After the war, the samurai in the Satsuma region, which was praised as "the unparalleled western country", almost disappeared. Like the medieval knights in Europe, the samurai were ruthlessly crushed by the wheel of history. The line soldiers and flintlocks smashed the charge of the knights, and the national army and field guns silenced the samurai who were skilled in martial arts. The irony is that Saigo Shenglong himself became the spiritual totem of the Japanese militarism faction and the ruling faction after his death, but the old samurai who followed the failure of Saigo set off a civil rights movement again and again, trying to maintain the collapse of the Meiji Three Heroes after the death. Bad and fragile democracies.

For example, to give an example , after I counted three times with my fingers one by one, the above article of unknown origin has a total of 203 words, and after utf8 encoding, there are 203*3=609 bytes, so length=126 , length_data = b'\x02a' . Using the base-related knowledge we learned in the second grade of primary school, the final length can be expressed as length_data[0] * (16^2) + length_data[1] = 609 . And if length==127 at the beginning, the hard calculated length is length_data[0] * (16^14) + length_data[1] * (16^12) + length_data[2] * (16^10) + length_data[3] * (16^8) + length_data[4] * (16^6) + length_data[5] * (16^4) + length_data[6] * (16^2) + length_data[7] . This is obviously a little troublesome, but fortunately python’s standard library struct ^[7] has built-in unpack() which can be easily converted.

To be continued…

Note that the letter at the beginning is a lowercase L, which is also what the boss said, bloggers note ︎
Xie Xiren, “Computer Networks (8th Edition)”, hereinafter referred to as “Jiwang”, P8 ︎
RFC 959, STD9 ︎
RFC 7540 ︎
The front-end code comes from https://blog.csdn.net/u012324798/article/details/103549249 , with modifications ︎
“Ji Wang” P220 ︎
https://docs.python.org/en-us/3/library/struct.html ︎

This article is reproduced from: https://blog.2333332.xyz/2022/05/22/2022-05-22-a-guide-to-websocket/
This site is for inclusion only, and the copyright belongs to the original author.