前言
Recent involvement inFollowas well asRSSHubThese two open source projects, some of the development and maintenance work, because it involves a lot of subscription sources with the “battle of wits”, there are some very interesting development practices, so open this new pit series to record.
This is a tutorial on how to use Telegram’s official API and MTProxy to update the TG channel and deploy the full configuration in the past few weeks.
Follow & RSSHub
Follow provides a convenient way to subscribe to feeds, for example, users can subscribe to a channel by typing in the name of the corresponding Telegram channel, so they don’t have to jump to each channel to check it out, which relies on the RSSHub implementation behind it.
RSS (Really Simple Syndication) is an ancient standard for information syndication that allows users to easily subscribe to and get updates from websites through a unified data format. However, with the rise of social media and the mobile Internet, the number of websites that natively support RSS is decreasing.
With the concept of “everything can be RSS”, RSSHub uses the power of the open source community to convert many modern content platforms, including Telegram, YouTube, and podcasts, into the standard RSS format, allowing users to regain control of how they get their information and stay away from algorithmic recommendations.
RSSHub now contains thousands of platforms, for different platforms and categories also have a variety of processing methods, including but not limited to web crawlers, official interfaces, reverse API calls, etc., to cover the vast majority of our daily use of information sources, but also very suitable for platforms as some of the encryption algorithms and anti-crawling mechanism to learn the practice of, for example, the Telegram channel introduced below! RSS feed implementation.
RSSHub Telegram Integration and Deployment
Telegram Web Preview
Since Telegram offers a preview of the channel’s web page, which can be accessed, for example, via thet.me/s/pseudoyulifeThis link is a direct way to view updates from my own channel, so RSSHub has long since implemented the integration of subscribing to Telegram channel updates by crawling content from web pages and converting it to RSS format.
However, many users reported that some channels could not be captured, so I tested it and found that Telegram used a black box mechanism to limit the web preview function of some channels, for example, I have been subscribing to the “Reorx’s Forge” and “Newlearner’s Own Residence” and other channels when using/sWhen you come to visit the page, you will be forced to redirect, indicating that you need to open a client to view the content, so there is no way for us to crawl the content and convert it to RSS for this type of channel.
Telegram APIs
To solve this problem, I went back to Telegram’s official documentation and found that they categorize their APIs as follows:
One of the Telegram APIs has achannels.getMessagesmethod can return messages from a certain channel, which can meet our needs, the specific implementation logic is in the RSSHub code repository — thelib/routes/telegram/channel.ts和lib/routes/telegram/tglibThese are the two parts of the code for those who are interested.
创建 Telegram App
Using the Telegram API requires access to thetelegram core, login via cell phone number.
Click on the API development tools module to create a Telegram Application (see the “Telegram Application” section).Creating your Telegram Application」）。
In the App configuration module, we can get theapi_id、api_hashTwo parameters, record the values, will be used later.
获取 Telegram Session
The process of creating a new client using the Telegram API is quite strict, and you need to authenticate your phone number via SMS to log in. It’s not practical to use it interactively in the code, because we need to create the client and get its session in advance, and then use the api through the session directly.
Pull Scriptpseudoyu/telegram-api-scripts“, then run thenpm i或pnpm iInstall the dependencies and runnpm run start或node index.jsFollow the prompts to enter theapi_id和api_hashand cell phone number (need to be the same as the cell phone number when applying for the Telegram App), get the verification code via SMS or Telegram App and then you can output the command line to get the session.
Note that since we will need to use this session in the RSSHub service, we should try to run the script to get the session on the same server where the RSSHub service is deployed, so that we can avoid some of Telegram’s restrictions on IPs and devices.
Note: If RSSHub is on a Serverless platform or otherwise, IP consistency can also be maintained by additionally configuring MTProxy, as explained later.
(Optional) Use MTProxy to maintain IP consistency
After adding Telegram API logic and configurations to Follow’s RSSHub instance, after a while it reports aAuthKeyDuplicatedErrorerror, and after checking it out a lot of developers have encountered it as well — “AuthKeyDuplicatedError Eror problem #1488“, the guess is that there are some limitations due to the fact that our RSSHub instance is a k8s cluster deployment that calls the Telegram API from a different machine.
I started to look for a solution for this situation and found that Telegram provides a proprietary proxy protocol called MTProxy, which can be deployed to proxy all API requests by deploying an MTProxy Server, which ensures that requests sent from different machines all come from the same IP.
I found it.Dofamin/MTProxy-Docker” This project, after pulling the repository, creates a new.envfile, add this environment variable (the default official code is not maintained much anymore, this version adds some patches):
MTPROTO_REPO_URL=https://github.com/GetPageSpeed/MTProxy
SECRETCustomized values can also be selected.IPIf you don’t fill in the form, it will be passed automatically.curl ifconfig.coGet the local IP, for other environment variables you can check the repository’sREADME.mdThe document modifies itself.
After the configuration is complete, rundocker compose up -dis started, and the proxy service runs on the<IP>:8443Up.
Note: This proxy step can be skipped if RSSHub is deployed as a single instance via Docker, Docker Compose, etc., and you can guarantee that you will run the script to get the session on the same deployment machine.
RSSHub Deployment
After the above service deployment, we obtained the following information:
- Telegram API 的
api_id和api_hash - Telegram 的
sessionstring (computer science) - MTProxy 的
SECRET、IP、PORT
At this point it’s time to start deploying/updating our RSSHub instance, which can be done according to the(computer) fileChoose from different approaches such as Docker, Docker Compose, Serverless, etc.
It is recommended to deploy via Docker Compose, which is easier to maintain and needs to be deployed on thedocker-compose.ymlAdd the following variables to the
environment:
- TELEGRAM_API_ID=<Telegram API ID>
- TELEGRAM_API_HASH=<Telegram API Hash>
- TELEGRAM_PROXY_SECRET=<MTProxy SECRET>
- TELEGRAM_SESSION=<Telegram Session>
- TELEGRAM_PROXY_HOST=<MTProxy IP，如 123.123.123.123>
- TELEGRAM_PROXY_PORT=8443
I maintain it by myself.(architecture) formworkDeployed on the Zeabur platform, the visual deployment is relatively easy to use, and you only need to add a new file after deployment in theVariablesAdd the following variables to the module and restart the service to take effect:
TELEGRAM_API_ID=<Telegram API ID>
TELEGRAM_API_HASH=<Telegram API Hash>
TELEGRAM_PROXY_SECRET=<MTProxy SECRET>
TELEGRAM_SESSION=<Telegram Session>
TELEGRAM_PROXY_HOST=<MTProxy IP，如 123.123.123.123>
TELEGRAM_PROXY_PORT=8443
beta (software)
After the above steps, we have our own RSSHub instance with Telegram-related configuration, and a visit to the home page showing the above page means that the deployment has been successful, and can be accessed via the<RSSHub URL>/telegram/channel/<Channel Name>to subscribe to the Telegram channel.
This can be done, for example, with my public examplehttps://rsshub.pseudoyu.com/telegram/channel/NewlearnerChannelSubscribe to our newsletter.Newlearner’s Own Residence“Channel.
summarize
This article records the development practice and deployment plan of subscribing to some restricted channels via Telegram API in RSSHub, limited by the platform, the program may fail at some point in the future, we will continue to study a more stable solution, you can leave a message or raise an Issue in the RSSHub repository to provide feedback/exchange any problems.