/happy-endless/Self-Rule

Wiki directory

Browse AI-generated documentation for any repository.

Changelog & Updates

Relevant source files

The following files were used as context for generating this wiki page: - [Tool/New.md](https://github.com/happy-endless/Self-Rule/blob/HEAD/Tool/New.md) - [update_providers.py](https://github.com/happy-endless/Self-Rule/blob/HEAD/update_providers.py) - [update_sing_box.py](https://github.com/happy-endless/Self-Rule/blob/HEAD/update_sing_box.py) - [Clash/GeneralClashConfig.yml](https://github.com/happy-endless/Self-Rule/blob/HEAD/Clash/GeneralClashConfig.yml) - [Clash/Providers/ProxyGFWlist.yaml](https://github.com/happy-endless/Self-Rule/blob/HEAD/Clash/Providers/ProxyGFWlist.yaml) - [Tool/Surge转acl工具/shortcuts.xml](https://github.com/happy-endless/Self-Rule/blob/HEAD/Tool/Surge%E8%BD%ACacl%E5%B7%A5%E5%85%B7/shortcuts.xml) - [Tool/Surge转acl工具/acl自定义语言格式文件.xml](https://github.com/happy-endless/Self-Rule/blob/HEAD/Tool/Surge%E8%BD%ACacl%E5%B7%A5%E5%85%B7/acl%E8%87%AA%E5%AE%9A%E4%B9%89%E8%AF%AD%E8%A8%80%E6%A0%BC%E5%BC%8F%E6%96%87%E4%BB%B6.xml) - [Clash/Providers/Ruleset/AI.yaml](https://github.com/happy-endless/Self-Rule/blob/HEAD/Clash/Providers/Ruleset/AI.yaml) - [Clash/Providers/Ruleset/Developer.yaml](https://github.com/happy-endless/Self-Rule/blob/HEAD/Clash/Providers/Ruleset/Developer.yaml)

This document provides an overview of the update history and the automated processes used to maintain the rule sets within the Self-Rule project. The project utilizes a combination of a manual legacy changelog, automated scripts for generating rules for different platforms like Clash and sing-box, and a suite of tools for rule format conversion.

The update mechanism is designed to ensure consistency and maintainability across the various rule formats and applications supported by the repository. This includes generating Clash provider files from master .list files and compiling rulesets for sing-box.

Manual Changelog (Legacy)

A manual changelog, primarily active in 2017, documents the historical evolution of the rule sets. It details optimizations, additions, and fixes to the ACL (Access Control List) rules.

Sources: Tool/New.md

Version/Date	Summary of Changes
3.16 (2017-03-16)	Deduplication, merged China IP segments, improved rules for Baidu, Letv, UC Browser, and added ad filtering for iQiyi, Qidian, and other apps.
2.05 ★★★ (2017-02-18)	Merged mainland China IP addresses, reducing rule count by nearly 1700. Performed deduplication and sorting.
2.01 ★★★ (2017-02-17)	Major refactoring including alphabetical sorting, extensive rule additions, and fixes for regex matching.
1.08 (2017-02-11)	Almost a complete rewrite. Added ad filtering for MangoTV, QQ, JD.com and others. Removed rules for Apple-specific services.
1.01 ★★★ (2017-02-04)	Significant update to improve regex matching speed by reducing backtracking. Merged ad-blocking rules for Sina and QQ.

Sources: Tool/New.md:5-92

Automated Rule Generation and Updates

The project employs Python scripts to automate the generation and conversion of rule files, ensuring that master lists are correctly propagated to formats used by different proxy clients.

Clash Provider Generation (`update_providers.py`)

The update_providers.py script is responsible for converting .list files into Clash-compatible .yaml provider files. It traverses the Clash/ directory, finds all .list files, and generates a corresponding .yaml file in the Clash/Providers/ directory.

The process flow for generating Clash providers is as follows:

Generating architecture diagram...

This diagram illustrates the logic within update_providers.py to process .list files and create .yaml files.

The script also includes a cleanup mechanism that removes any .yaml provider files for which a source .list file no longer exists, keeping the providers synchronized with the master lists.

Sources: update_providers.py

Sing-box Ruleset Generation (`update_sing_box.py`)

A similar automation process exists for sing-box. The update_sing_box.py script converts .list files into compiled .srs rulesets for sing-box.

This script defines a HeadlessRule data structure to represent the various rule types.

@dataclass
class HeadlessRule:
    # ...
    domain: list[str] = None
    domain_suffix: list[str] = None
    domain_keyword: list[str] = None
    domain_regex: list[str] = None
    ip_cidr: list[str] = None
    # ...

Sources: update_sing_box.py:14-33

The conversion process involves parsing each line of a .list file, mapping it to the appropriate field in the HeadlessRule object, and then generating a temporary JSON file. This JSON file is then compiled into the final .srs binary format using the sing-box command-line tool.

The following table shows the mapping from the .list format to the sing-box JSON rule attributes.

`.list` Prefix	`HeadlessRule` Attribute
`DOMAIN`	`domain`
`DOMAIN-SUFFIX`	`domain_suffix`
`DOMAIN-KEYWORD`	`domain_keyword`
`DOMAIN-REGEX`	`domain_regex`
`IP-CIDR` / `IP-CIDR6`	`ip_cidr`
`SOURCE-IP-CIDR`	`source_ip_cidr`
`PROCESS-NAME`	`process_name`

Sources: update_sing_box.py:61-82

The generation flow is depicted below:

Generating architecture diagram...

This diagram shows the process of converting .list files to compiled .srs rulesets for sing-box.

Sources: update_sing_box.py:86-88

Rule Conversion Tooling

The repository includes tools for manual rule format conversion, primarily for use within the Notepad++ editor. These tools facilitate the maintenance of ACL (another proxy format) and Surge rules.

Sources: Tool/Surge转acl工具/

Notepad++ Macros

The shortcuts.xml file defines a series of macros for Notepad++ to automate repetitive text transformations between different rule formats.

Macro Name (Translated)	Function
`网址转acl` (URL to ACL)	Converts a list of domains (e.g., from Surge `DOMAIN-SUFFIX`) into ACL regex format.
`关键字转acl` (Keyword to ACL)	Converts a list of keywords (e.g., from Surge `DOMAIN-KEYWORD`) into ACL regex format.
`ip转acl` (IP to ACL)	Converts a list of IP-CIDR rules into a clean list for ACL.
`acl转surge` (ACL to Surge)	Converts ACL regex rules back into Surge `DOMAIN-SUFFIX` format.
`去掉注释` (Remove Comments)	Strips `//` and `/* ... */` style comments from a file.

Sources: Tool/Surge转acl工具/shortcuts.xml:3-130

ACL Syntax Highlighting

To aid in editing .acl files, a User Defined Language file for Notepad++ is provided. acl自定义语言格式文件.xml defines syntax highlighting for the custom ACL format.

Key elements defined for highlighting include:

Comments: Lines starting with #.
Operators: / and |.
Keywords: Section headers like [proxy_all], [bypass_all], [outbound_block_list], [bypass_list], and [proxy_list].

Sources: Tool/Surge转acl工具/acl自定义语言格式文件.xml:7-22

Core Rule Configuration Files

The updates and conversions result in several key configuration files used by proxy clients.

General Clash Configuration

The Clash/GeneralClashConfig.yml file serves as a template for Clash for Windows. A significant part of this configuration that is subject to updates is the cfw-bypass list. This list contains domains and IP ranges that should bypass the proxy, corresponding to the "direct connect" lists mentioned in the manual changelog.

A snippet of the bypass list:

cfw-bypass:
  # 直连列表
  # MyList
  - "*.13th.tech"
  - "*.423down.com"
  # CN域名直连(中国|公司|网络)
  - "*.cn"
  - "*.xn--fiqs8s"
  # 中国国内常见域名关键词直连
  - "*360buy*"
  - "*alicdn*"
  # Alibaba
  - "*.1688.com"
  - "*.alibaba.com"

Sources: Clash/GeneralClashConfig.yml:218-577

Provider Rule Lists

The automated scripts generate numerous .yaml files in Clash/Providers/. These files are consumed by Clash as remote rule providers. They are categorized for different purposes.

Comprehensive Lists: ProxyGFWlist.yaml and ProxyLite.yaml are large lists containing thousands of rules for proxying traffic.
Categorized Lists: Smaller, more focused lists target specific services.

Ruleset File	Description	Example Rule
`AI.yaml`	Rules for AI platforms like OpenAI and Gemini.	`DOMAIN-SUFFIX,openai.com`
`Developer.yaml`	Rules for common developer websites and services.	`DOMAIN-SUFFIX,stackoverflow.com`
`Wikipedia.yaml`	Rules for Wikipedia and its sister projects.	`DOMAIN-SUFFIX,wikipedia.org`
`ProxyGFWlist.yaml`	A very large collection of rules based on the GFW list.	`DOMAIN-SUFFIX,google.com`

Sources: Clash/Providers/Ruleset/AI.yaml, Clash/Providers/Ruleset/Developer.yaml, Clash/Providers/Ruleset/Wikipedia.yaml, Clash/Providers/ProxyGFWlist.yaml

Summary

The Self-Rule project maintains its rule sets through a dual approach. A legacy manual changelog provides historical context, while a modern, automated workflow using Python scripts ensures that master .list files are consistently converted into up-to-date, platform-specific configurations for Clash and sing-box. This is supplemented by a toolkit for Notepad++ to assist with manual conversions for other formats like ACL and Surge, creating a comprehensive system for rule management and distribution.

Changelog & Updates

Relevant source files

Manual Changelog (Legacy)

A manual changelog, primarily active in 2017, documents the historical evolution of the rule sets. It details optimizations, additions, and fixes to the ACL (Access Control List) rules.

Sources: Tool/New.md

Version/Date	Summary of Changes
3.16 (2017-03-16)	Deduplication, merged China IP segments, improved rules for Baidu, Letv, UC Browser, and added ad filtering for iQiyi, Qidian, and other apps.
2.05 ★★★ (2017-02-18)	Merged mainland China IP addresses, reducing rule count by nearly 1700. Performed deduplication and sorting.
2.01 ★★★ (2017-02-17)	Major refactoring including alphabetical sorting, extensive rule additions, and fixes for regex matching.
1.08 (2017-02-11)	Almost a complete rewrite. Added ad filtering for MangoTV, QQ, JD.com and others. Removed rules for Apple-specific services.
1.01 ★★★ (2017-02-04)	Significant update to improve regex matching speed by reducing backtracking. Merged ad-blocking rules for Sina and QQ.

Sources: Tool/New.md:5-92

Automated Rule Generation and Updates

The project employs Python scripts to automate the generation and conversion of rule files, ensuring that master lists are correctly propagated to formats used by different proxy clients.

Clash Provider Generation (`update_providers.py`)

The process flow for generating Clash providers is as follows:

Generating architecture diagram...

This diagram illustrates the logic within update_providers.py to process .list files and create .yaml files.

The script also includes a cleanup mechanism that removes any .yaml provider files for which a source .list file no longer exists, keeping the providers synchronized with the master lists.

Sources: update_providers.py

Sing-box Ruleset Generation (`update_sing_box.py`)

A similar automation process exists for sing-box. The update_sing_box.py script converts .list files into compiled .srs rulesets for sing-box.

This script defines a HeadlessRule data structure to represent the various rule types.

@dataclass
class HeadlessRule:
    # ...
    domain: list[str] = None
    domain_suffix: list[str] = None
    domain_keyword: list[str] = None
    domain_regex: list[str] = None
    ip_cidr: list[str] = None
    # ...

Sources: update_sing_box.py:14-33

The following table shows the mapping from the .list format to the sing-box JSON rule attributes.

`.list` Prefix	`HeadlessRule` Attribute
`DOMAIN`	`domain`
`DOMAIN-SUFFIX`	`domain_suffix`
`DOMAIN-KEYWORD`	`domain_keyword`
`DOMAIN-REGEX`	`domain_regex`
`IP-CIDR` / `IP-CIDR6`	`ip_cidr`
`SOURCE-IP-CIDR`	`source_ip_cidr`
`PROCESS-NAME`	`process_name`

Sources: update_sing_box.py:61-82

The generation flow is depicted below:

Generating architecture diagram...

This diagram shows the process of converting .list files to compiled .srs rulesets for sing-box.

Sources: update_sing_box.py:86-88

Rule Conversion Tooling

The repository includes tools for manual rule format conversion, primarily for use within the Notepad++ editor. These tools facilitate the maintenance of ACL (another proxy format) and Surge rules.

Sources: Tool/Surge转acl工具/

Notepad++ Macros

The shortcuts.xml file defines a series of macros for Notepad++ to automate repetitive text transformations between different rule formats.

Macro Name (Translated)	Function
`网址转acl` (URL to ACL)	Converts a list of domains (e.g., from Surge `DOMAIN-SUFFIX`) into ACL regex format.
`关键字转acl` (Keyword to ACL)	Converts a list of keywords (e.g., from Surge `DOMAIN-KEYWORD`) into ACL regex format.
`ip转acl` (IP to ACL)	Converts a list of IP-CIDR rules into a clean list for ACL.
`acl转surge` (ACL to Surge)	Converts ACL regex rules back into Surge `DOMAIN-SUFFIX` format.
`去掉注释` (Remove Comments)	Strips `//` and `/* ... */` style comments from a file.

Sources: Tool/Surge转acl工具/shortcuts.xml:3-130

ACL Syntax Highlighting

To aid in editing .acl files, a User Defined Language file for Notepad++ is provided. acl自定义语言格式文件.xml defines syntax highlighting for the custom ACL format.

Key elements defined for highlighting include:

Comments: Lines starting with #.
Operators: / and |.
Keywords: Section headers like [proxy_all], [bypass_all], [outbound_block_list], [bypass_list], and [proxy_list].

Sources: Tool/Surge转acl工具/acl自定义语言格式文件.xml:7-22

Core Rule Configuration Files

The updates and conversions result in several key configuration files used by proxy clients.

General Clash Configuration

A snippet of the bypass list:

cfw-bypass:
  # 直连列表
  # MyList
  - "*.13th.tech"
  - "*.423down.com"
  # CN域名直连(中国|公司|网络)
  - "*.cn"
  - "*.xn--fiqs8s"
  # 中国国内常见域名关键词直连
  - "*360buy*"
  - "*alicdn*"
  # Alibaba
  - "*.1688.com"
  - "*.alibaba.com"

Sources: Clash/GeneralClashConfig.yml:218-577

Provider Rule Lists

The automated scripts generate numerous .yaml files in Clash/Providers/. These files are consumed by Clash as remote rule providers. They are categorized for different purposes.

Comprehensive Lists: ProxyGFWlist.yaml and ProxyLite.yaml are large lists containing thousands of rules for proxying traffic.
Categorized Lists: Smaller, more focused lists target specific services.

Ruleset File	Description	Example Rule
`AI.yaml`	Rules for AI platforms like OpenAI and Gemini.	`DOMAIN-SUFFIX,openai.com`
`Developer.yaml`	Rules for common developer websites and services.	`DOMAIN-SUFFIX,stackoverflow.com`
`Wikipedia.yaml`	Rules for Wikipedia and its sister projects.	`DOMAIN-SUFFIX,wikipedia.org`
`ProxyGFWlist.yaml`	A very large collection of rules based on the GFW list.	`DOMAIN-SUFFIX,google.com`

Sources: Clash/Providers/Ruleset/AI.yaml, Clash/Providers/Ruleset/Developer.yaml, Clash/Providers/Ruleset/Wikipedia.yaml, Clash/Providers/ProxyGFWlist.yaml

Changelog & Updates

Manual Changelog (Legacy)

Automated Rule Generation and Updates

Clash Provider Generation (update_providers.py)

Sing-box Ruleset Generation (update_sing_box.py)

Rule Conversion Tooling

Notepad++ Macros

ACL Syntax Highlighting

Core Rule Configuration Files

General Clash Configuration

Provider Rule Lists

Summary

Changelog & Updates

Manual Changelog (Legacy)

Automated Rule Generation and Updates

Clash Provider Generation (update_providers.py)

Sing-box Ruleset Generation (update_sing_box.py)

Rule Conversion Tooling

Notepad++ Macros

ACL Syntax Highlighting

Core Rule Configuration Files

General Clash Configuration

Provider Rule Lists

Summary

Clash Provider Generation (`update_providers.py`)

Sing-box Ruleset Generation (`update_sing_box.py`)

Clash Provider Generation (`update_providers.py`)

Sing-box Ruleset Generation (`update_sing_box.py`)