foreword
Yesterday, my roommate bought an expensive IP proxy pool for crawlers. I suddenly had an idea next to me. Now all major ISPs/IDCs will assign you an entire IPv6 network segment with a 64-bit prefix. Can we make good use of it? What about this huge IPv6 IP resource?
I couldn’t sleep with this idea in mind, I got up early this morning (9:30) and started researching, and finally succeeded in implementing each request from a separate IPv6 address.
First look at the effect. I put the written program on the server and ran it for a while. The following is the access information from Cloudflare statistics. You can see that the number of unique visitors (independent IPs) has reached an unprecedented number for my small blog. Basically, Is a request for an independent IP, the effect is very good.
tutorial
First of all, you have to have an entire IPv6 subnet route for you. When you encounter a stingy service provider, even if they have a lot of IPv6 resources, they will not give you the use of it. In this case, you have no right. Fortunately, most ISPs/IDCs will give you an entire IPv6 /64 subnet, and some can even apply for a /56 subnet, which increases by several orders of magnitude, so you hardly need to worry about it.
For the convenience of experimentation, I purchased [Vultr’s server] , if you haven’t registered, you can use my [AFF link] .
You can get your IPv6 subnet information by viewing the address of the network interface with the ip a command:
|
|
As you can see, the default IPv6 address given to you is dynamic, which is automatically generated by the SLAAC protocol based on the prefix and the Mac address; there is also an IPv6 address starting with fe80 , which is also an automatically assigned link-local address. It’s nice that IPv6 avoids manual configuration through these stateless address configuration protocols, and is plug-and-play.
In my experiments, the subnet I got was 2001:19f0:6001:48e4::/64 , which is the basis for the following.
binding and routing
After getting the IPv6 subnet, you need to bind it to the local loopback interface, and then add the route.
|
|
In order to be able to bind any IP, we need to enable the ip_nonlocal_bind feature of the kernel:
|
|
NDP
Similar to the role of the ARP protocol in IPv4, the ND protocol needs to be used in IPv6 to discover neighbors and determine available paths. We need to start an ND proxy:
Install ndppd : apt install ndppd
Edit /etc/ndppd.conf file:
|
|
Start ndppd : systemctl start ndppd
verify
Next you can verify it by specifying the egress IP with curl --interface :
|
|
As you can see, requests can be made according to any IP we specify
HTTP proxy
For ease of use, an http proxy server is written in Rust. Each request will go to a random IP under the specified IPv6 subnet, which is a basic demo.
|
|
|
|
Welcome to Star: https://github.com/zu1k/http-proxy-ipv6-pool
broken thoughts
Proxypool project
My roommate wanted to buy a proxy pool, so I went to GitHub to search it. I didn’t expect that the project that I had already stopped updating was ranked first. This project itself is not prepared for reptiles, and ranking so high is a waste of everyone’s feelings.
The second is the proxy pool specially prepared for crawlers. The author is my senior brother. He has a very in-depth research on crawlers and recommends it to everyone.
first thought
In fact, my initial idea was not to directly attach the entire IP segment to the interface. At first, I didn’t know that a network interface can directly attach an entire IP segment. Considering that the number of IPv6 segments is too large, it is obviously unrealistic to attach multiple static IPs to the interface through enumeration, so I was trying to find a way to encapsulate the IP packets myself. Then inject.
I thought of two scenarios for injecting packets:
- It can completely encapsulate IPv6 packets and lower-layer protocols by itself, and directly write data through the raw fd of the network card
I gave up on this solution as soon as I thought of it, because the protocol was too complicated, it was impossible for me to implement it at all
- Create a TUN device, configure the network segment of the TUN device as an IPv6 subnet, and then create a bridge between the TUN device and the real network device.
The source IP is a random IPv6 address under the network segment by TUN injection into the system network stack, forging the illusion that there are many hosts.
Because I have a little understanding of TUN (see my previous article [Modes of Using TUN] ), I naturally have this idea, and I am convinced that it is feasible. The reason why I think this is feasible is because I have done [give each Docker container an independent IP] before, which is also to make full use of the abundant IPv6 resources. Interested students can take a look.
After searching for information, I finally determined that it is not feasible to use TUN, at least use TAP, [because we have to deal with the NDP protocol] , and I have not studied the details behind it.
Fortunately, I found a convenient method in this article after searching for information, which prevented me from falling into these lost paths. Think about it, that is, your knowledge is not enough, and your understanding of the Linux kernel and the many functions it provides is not deep enough. As a result, people don’t know the original functions at all, so they can’t think of a suitable solution.
WAF
I don’t know very much. Does the current WAF limit the current and speed only based on IP? If this is the case, then using this huge IPv6 resource is not easy to bypass the ban.
If it is to directly ban or restrict the entire subnet, will there be a serious manslaughter? After all, not all ISPs are so generous to give an entire IP segment.
Friends who know it are welcome to comment and exchange.
This article is reprinted from: https://zu1k.com/posts/tutorials/http-proxy-ipv6-pool/
For personal collection only, the copyright belongs to the original author
Who doesn’t want a proxy pool of 2^64 IPs? appeared first on Haowen Collection .
This article is reproduced from: https://shoucang.zyzhang.com/%E8%B0%81%E4%B8%8D%E6%83%B3%E8%A6%81-264-%E4%B8%AA-ip-% E7%9A%84%E4%BB%A3%E7%90%86%E6%B1%A0-%EF%BC%9F/
This site is for inclusion only, and the copyright belongs to the original author.