Update build-and-release.yml

Update build.sh
Update proxy.js
2024-11-02 16:16:26 +08:00 · 2024-10-10 12:16:19 +08:00 · 2024-10-09 11:37:46 +08:00 · 2024-10-09 11:06:34 +08:00 · 2024-10-08 14:20:41 +08:00 · 2024-10-08 11:12:23 +08:00
9 changed files with 648 additions and 302 deletions
--- a/.github/workflows/build-and-release.yml
+++ b/.github/workflows/build-and-release.yml
@ -4,8 +4,10 @@ on:
  push:
    branches:
      - "master"
-  schedule:
-    - cron: "0 0 * * 0"
+    paths:
+      - '**/*.example'
+      - '**/*.js'
+      - '**/*.sh'

 permissions: write-all

@ -32,4 +34,4 @@ jobs:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        sha: ${{ github.sha }}
      run: |
-        gh release create "$(date '+%Y-%m-%d-')${sha:0:7}" --generate-notes ./proxy.pac
+        gh release create "$(date '+%Y-%m-%d-')${sha:0:7}" --generate-notes ./proxy.pac
--- a/.gitignore
+++ b/.gitignore
@ -1,7 +1,4 @@
 .DS_Store
-china.txt
-china6.txt
-gfwlist.txt
-auto-proxy.txt
-domain-rules-*.txt
+*.txt
+*.tmp
 proxy.pac
--- a/README.md
+++ b/README.md
@ -1,34 +1,269 @@
 # proxy.pac

-这个项目用来生成代理自动配置 `proxy.pac` 文件，可以用于配置浏览器或系统级的代理设置。你可以自定义规则，根据配置文件中指定的域名来路由流量。
+- [English](#proxy.pac)
+- [中文](#中文介绍)
+- [License](#license)

-## 使用方法
+This project is used to generate the proxy auto-configuration `proxy.pac` file, which can be used for configuring browser or system-level proxy settings.

-1. **域名规则配置**
+Supports routing traffic through proxies: You can customize rules to route traffic to different proxy servers based on domain names or IP networks specified in the configuration files.

-   项目包含一些示例配置文件：
+For high-performance matching: Uses a hash table for domain rules, and an IP prefix tree for fast matching of network rules. If there is no domain match, the IP address will be resolved and then matched against network rules.

+## Usage
+
+### 1. **Domain Rule Configuration**
+
+   The project contains some example configuration files:
+
+   - `auto-proxy.txt.example`
   - `domain-rules-blocked.txt.example`
   - `domain-rules-direct.txt.example`
   - `domain-rules-proxy.txt.example`
+   - `ipv4-rules-direct.txt.example`
+   - `ipv6-rules-direct.txt.example`

-   要使用这些文件，去掉 `.example` 扩展名。每个文件代表不同的代理行为：
+   To use these files, remove the `.example` extension. Each file represents a different proxy behavior:

-   - **Blocked**：添加到 `domain-rules-blocked.txt` 中的域名将被阻止访问。
-   - **Direct**：添加到 `domain-rules-direct.txt` 中的域名将绕过代理，直接连接。
-   - **Proxy**：添加到 `domain-rules-proxy.txt` 中的域名将使用默认代理。
+#### a. Auto-Proxy

-   将你的域名添加到合适的文件中，每个域名一行。以 `#` 开头的行被视为注释。例如：
+   - **Auto-Proxy Rules**: Add rules to `auto-proxy.txt` to control how websites are accessed.

+   All files starting with `auto-proxy` and ending with `.txt` will be parsed as Auto-Proxy rules. If you have multiple Auto-Proxy rules, you can save them as multiple files, like `auto-proxy-1.txt`, `auto-proxy-2.txt`, etc.
+
+   **Note**: Currently, the URL matching rules in Auto-Proxy are ignored, and only domain rules are handled.
+
+#### b. Domain and IP Address
+
+   - **Blocked**: Domains added to `domain-rules-blocked.txt` will be blocked from access.
+   - **Direct**:
+     - Domains added to `domain-rules-direct.txt` will bypass the proxy and connect directly.
+     - IPv4 networks (in CIDR format) added to `ipv4-rules-direct.txt` will bypass the proxy and connect directly.
+     - IPv6 networks (in CIDR format) added to `ipv6-rules-direct.txt` will bypass the proxy and connect directly.
+   - **Proxy**: Domains added to `domain-rules-proxy.txt` will use the default proxy.
+
+   Add your domains or IP networks to the appropriate file, one per line. Subdomains will inherit the proxy behavior of their parent domain. You can also add country-level top domains to simplify the configuration. Lines starting with `#` are treated as comments.
+
+   For example:
+
+   Domains added to `domain-rules-direct.txt`, along with their subdomains,  will bypass the proxy and connect directly:
   ```
-   # 直连域名
+   # Direct connect domains
+   cn # All domains ending with .cn will connect directly by default
   google.com
   example.org
   ```

-   你也可以创建自己的自定义规则文件，文件名应遵循 `domain-rules-<rule_name>.txt` 的格式。例如，`domain-rules-companyProxy.txt` 将使该文件中的所有域名使用 `proxy.pac` 中定义的 `companyProxy` 设置。
+   IPv4 Networks added to `ipv4-rules-direct.txt` will bypass the proxy and connect directly:
+   ```
+   # Direct connect IPv4 networks
+   192.168.0.0/16
+   114.114.114.114/32 # To add a specific IP address, append /32 to the IP address
+   ```

-2. **生成 `proxy.pac` 文件**
+   You can also create your own custom rule files, following the format `<domain|ipv4|ipv6>-rules-<rule_name>.txt`. For example, `domain-rules-companyProxy.txt` will make all domains in this file use the `companyProxy` setting defined in `proxy.pac`. `ipv4-rules-block.txt` will block all networks listed in the file.
+
+#### c. Domain Regular Expressions
+
+   - **Domain Regular Expressions**: `domain-regexp.txt` is used to define domain rules based on regular expressions, which allows for flexible matching of similar domains.
+
+   File structure:
+   ```
+   [direct]
+   # host regex ...
+
+   [blocked]
+   # ...
+
+   [proxy]
+   # ...
+   ```
+
+   Each section represents a different proxy behavior, which can be `direct`, `blocked`, `proxy`, or a custom behavior (e.g., `[companyProxy]`).
+
+   - **[direct]**: Domains matched by regular expressions in this section will bypass the proxy and connect directly.
+   - **[blocked]**: Domains matched by regular expressions in this section will be blocked.
+   - **[proxy]**: Domains matched by regular expressions in this section will use the default proxy.
+   - **Custom Behavior**: You can add your own section name, such as `[companyProxy]`, to indicate that domains matching those patterns will use a custom proxy configuration.
+
+   Each line is a regular expression for matching specific domains or their subdomains. Lines starting with `#` are treated as comments. For example:
+   ```
+   [direct]
+   # Direct connection domains
+   ^img-[0-9][0-9].*\.example\.com$
+
+   [blocked]
+   # Blocked domains
+   ^ad-[a-z0-9]\.cdn[0-9]\.example\.com$
+   ```
+
+   Ensure that the regular expressions are valid to avoid affecting normal network access.
+
+### 2. **Generate the `proxy.pac` File**
+
+   Run the script to generate the `proxy.pac` file:
+
+   ```sh
+   ./build.sh
+   ```
+
+   The `proxy.pac` file will be automatically generated in the project root directory.
+
+### 3. **Default Rule Sources**
+
+   The build script [`build.sh`](./build.sh) will, by default, download the following files without overwriting existing files of the same name:
+
+   - `auto-proxy.txt`
+   - `ipv4-rules-direct.txt`
+   - `ipv6-rules-direct.txt`
+
+   If you do not need the Auto-Proxy rules or IP networks rules, you can create empty files with the same name to skip the download.
+
+### 4. **Proxy Configuration**
+
+   The generated `proxy.pac` file uses the following default proxy configurations (note that the default proxy server is `SOCKS5 127.0.0.1:1080`):
+
+   ```javascript
+   var proxyBehaviors = {
+     proxy: "SOCKS5 127.0.0.1:1080", // Default proxy
+     direct: DIRECT,
+     blocked: "PROXY 0.0.0.0:0",
+     "http_proxy": "PROXY 127.0.0.1:3128",
+     "companyProxy": "PROXY 192.168.1.1:8080", // Domains in `domain-rules-companyProxy.txt` will use this proxy setting
+   };
+   ```
+
+   You can modify these values after generating `proxy.pac`, or customize them directly in the original script `proxy.js` to use different default settings. Please adjust these settings according to your environment and requirements.
+
+### 5. **Testing**
+
+   If you have Node.js installed, you can run the following command to test and verify the configuration:
+
+   ```sh
+   node proxy.pac test
+   ```
+
+   The test code is located at the end of the `proxy.pac` file, for example:
+
+   ```javascript
+   assertVisitHostWithProxy("com.google");
+   assertVisitHostWithProxy("domains.google");
+   assertHostWithDefaultAction("www.not-google");
+   assertDirectHost("10.3.4.5");
+   assertDirectHost("114.114.114.114");
+   assertBlockedHost("www.whitehouse.com");
+   ```
+
+## Example
+
+To add a domain that needs to be blocked, simply edit the `domain-rules-blocked.txt` file:
+
+```
+# Blocked domains
+example.com
+ads.example.net
+```
+
+Run `./build.sh` to regenerate the `proxy.pac` file, which will block access to `example.com` and `ads.example.net`.
+
+# 中文介绍
+
+这个项目用来生成代理自动配置 `proxy.pac` 文件，可以用于配置浏览器或系统级的代理设置。
+
+支持路由代理流量：你可以自定义规则，根据配置文件中指定的域名或者 IP 网络段来路由流量到不同的代理服务器。
+
+为了高性能匹配：使用哈希表来匹配域名规则，使用 IP 地址前缀树来快速匹配网络段规则。如果没有匹配的域名规则，会解析 IP 地址后再按网络地址段匹配一次。
+
+## 使用方法
+
+### 1. **域名规则配置**
+
+   项目包含一些示例配置文件：
+
+   - `auto-proxy.txt.example`
+   - `domain-rules-blocked.txt.example`
+   - `domain-rules-direct.txt.example`
+   - `domain-rules-proxy.txt.example`
+   - `ipv4-rules-direct.txt.example`
+   - `ipv6-rules-direct.txt.example`
+   - `domain-regexp.txt.example`
+
+   要使用这些文件，去掉 `.example` 扩展名。每个文件代表不同的代理行为：
+
+#### a. Auto-Proxy
+
+   - **Auto-Proxy 配置的规则**：把规则添加到 `auto-proxy.txt` 中，将会按照规则来访问网站。
+
+   所有以 `auto-proxy` 开头，并且以 `.txt` 结尾的文件都按 Auto-Proxy 规则来解析。如果你有多个 Auto-Proxy 规则，可以保存为多个文件，例如 `auto-proxy-1.txt`、`auto-proxy-2.txt` 等等。
+
+   **注意**：当前，会忽略 Auto-Proxy 中的 URL 的匹配规则，仅仅处理域名规则。
+
+#### b. 域名和IP地址
+
+   - **Blocked**：添加到 `domain-rules-blocked.txt` 中的域名将被阻止访问。
+   - **Direct**：
+     - 添加到 `domain-rules-direct.txt` 中的域名将绕过代理，直接连接。
+     - 添加到 `ipv4-rules-direct.txt` 中的 IPv4 网络段（CIDR格式）将绕过代理，直接连接。
+     - 添加到 `ipv6-rules-direct.txt` 中的 IPv6 网络段（CIDR格式）将绕过代理，直接连接。
+   - **Proxy**：添加到 `domain-rules-proxy.txt` 中的域名将使用默认代理。
+
+   将你的域名或者IP网络段添加到合适的文件中，每个域名一行。子域名会继承父域名的代理行为，可以直接添加国家顶级域，以简化配置。以 `#` 开头的行被视为注释。
+
+   例如：
+
+   文件 domain-rules-direct.txt 中添加的域名及其子域名将会绕过代理直接连接
+   ```
+   # 直连域名
+   cn # 默认所有以 .cn 结尾的域名都是直连
+   google.com
+   example.org
+   ```
+
+   文件 ipv4-rules-direct.txt 中添加的网络段将会绕过代理直接连接
+   ```
+   # 直连的 IPv4 网络段
+   192.168.0.0/16
+   114.114.114.114/32 # 如果要添加一个特定 IP 地址，请追加 /32 到 IP 地址后面
+   ```
+
+   你也可以创建自己的自定义规则文件，文件名应遵循 `<domain|ipv4|ipv6>-rules-<rule_name>.txt` 的格式。例如，`domain-rules-companyProxy.txt` 将使该文件中的所有域名使用 `proxy.pac` 中定义的 `companyProxy` 设置。`ipv4-rules-block.txt` 将不可访问文件中的所有网络段。
+
+#### c. 域名正则表达式
+
+   - **域名正则表达式**：`domain-regexp.txt` 用于灵活地定义基于正则表达式的域名规则。主要用于匹配大量相似的域名规则。
+
+   文件结构如下：
+   ```
+   [direct]
+   # host regex ...
+
+   [blocked]
+   # ...
+
+   [proxy]
+   # ...
+   ```
+   每个分段的名称代表不同的代理行为，可以是 `direct`、`blocked`、`proxy`，或者你自定义的行为（例如 `[companyProxy]`）。
+
+   - **[direct]**：在这一分段中的域名正则表达式将绕过代理，直接连接。
+   - **[blocked]**：在这一分段中的域名正则表达式将被阻止访问。
+   - **[proxy]**：在这一分段中的域名正则表达式将使用默认代理进行连接。
+   - **自定义行为**：你可以新增自己的分段名称，例如 `[companyProxy]`，表示这些匹配的域名将使用自定义代理配置。
+
+   每一行是一个域名的正则表达式，可以通过灵活的正则规则来匹配特定的域名或其子域名。以 `#` 开头的行被视为注释。例如：
+   ```
+   [direct]
+   # 直连的域名
+   ^img-[0-9][0-9].*\.example\.com$
+
+   [blocked]
+   # 阻止访问的域名
+   ^ad-[a-z0-9]\.cdn[0-9]\.example\.com$
+   ```
+
+   确保正则表达式是有效的，以免影响正常的网络访问行为。
+
+### 2. **生成 `proxy.pac` 文件**

   运行脚本生成 `proxy.pac` 文件：

@ -38,7 +273,16 @@

   在项目根目录中会自动生成 `proxy.pac` 文件。

-3. **代理配置**
+### 3. **默认的规则来源**
+   构建脚本 [`build.sh`](./build.sh) 默认会下载以下文件，但不会覆盖已有的同名文件：
+
+   - `auto-proxy.txt`
+   - `ipv4-rules-direct.txt`
+   - `ipv6-rules-direct.txt`
+
+   如果你不需要 Auto-Proxy 的规则或者 IP 网络段的规则，请创建同名的空文件即可忽略下载。
+
+### 4. **代理配置**

   生成的 `proxy.pac` 文件使用以下默认的代理配置（注意默认代理服务器是 `SOCKS5 127.0.0.1:1080`）：

@ -54,7 +298,7 @@

   你可以在生成 `proxy.pac` 后修改这些值，或者直接在原始脚本 `proxy.js` 中进行自定义，以便使用不同的默认设置。请根据实际环境和需求调整这些代理设置。

-4. **测试**
+### 5. **测试**

   如果安装了 Node.js，可以使用以下命令运行测试以验证配置：

--- a/auto-proxy.txt.example
+++ b/auto-proxy.txt.example
@ -0,0 +1 @@
+@@||example.com
--- a/build.sh
+++ b/build.sh
@ -1,5 +1,4 @@
 #!/usr/bin/env bash
-set -euo pipefail

 IFS='.' read -r __major __minor _ <<< "${BASH_VERSION:-0.0.0}"
 if [ "$__major" -lt 4 ] || { [ "$__major" -eq 4 ] && [ "$__minor" -lt 3 ]; }; then
@ -10,42 +9,50 @@ if [ "$__major" -lt 4 ] || { [ "$__major" -eq 4 ] && [ "$__minor" -lt 3 ]; }; th
    exit 1
 fi

+set -euo pipefail
+shopt -s nullglob
+
 cd "$(dirname "$0")"

 declare -A files
-files[gfwlist.txt]=https://raw.githubusercontent.com/gfwlist/gfwlist/master/gfwlist.txt
-files[china.txt]=https://gaoyifan.github.io/china-operator-ip/china.txt
-files[china6.txt]=https://gaoyifan.github.io/china-operator-ip/china6.txt
+files[auto-proxy.txt]=https://raw.githubusercontent.com/gfwlist/gfwlist/master/gfwlist.txt
+files[ipv4-rules-direct.txt]=https://gaoyifan.github.io/china-operator-ip/china.txt
+files[ipv6-rules-direct.txt]=https://gaoyifan.github.io/china-operator-ip/china6.txt

+declare -a files_to_be_deleted=()
 function cleanup() {
    local retval="$?"
-    if [[ -n "${del_file_on_exit-}" && -f "${del_file_on_exit}" ]]; then
-        echo "Remove ${del_file_on_exit}"
-        rm "${del_file_on_exit}"
-        exit "$retval"
-    fi
+    local file
+    for file in "${files_to_be_deleted[@]}"; do
+        if [[ -e "${file}" ]]; then
+            echo "Remove $file"
+            rm -v "${file}"
+        fi
+    done
+    exit "$retval"
 } >&2
 trap cleanup EXIT ERR SIGINT

 for f in "${!files[@]}"; do
-    del_file_on_exit="$f"
    url="${files[$f]}"
-    if [[ ! -f "$f" ]]; then
+    if [[ ! -e "$f" ]]; then
+        files_to_be_deleted=("$f" "${f}.tmp")
        if command -v wget &>/dev/null; then
-            wget -O "$f" "$url"
+            wget -O "${f}.tmp" "$url"
        elif command -v curl &>/dev/null; then
-            curl --output "$f" "$url"
+            curl --output "${f}.tmp" "$url"
        else
            echo "Error: please install wget or curl."
            exit 1
        fi
+        if [[ "$f" = auto-proxy.txt ]]; then
+            base64 -d <"${f}.tmp" > "$f"
+        else
+            mv "${f}.tmp" "$f"
+        fi
    fi
 done
-
-del_file_on_exit=auto-proxy.txt
-[ auto-proxy.txt -nt gfwlist.txt ] || base64 -d <gfwlist.txt >auto-proxy.txt
-
-del_file_on_exit=
+files_to_be_deleted=()

 domain_segments() {
    local domain="$1"
@ -65,18 +72,6 @@ if [[ "$actual" = "$expected" ]]; then
    exit 1
 fi >&2

-ipv6_to_array_format() {
-  while IFS= read -r line; do
-    local ipv6="${line%%/*}"
-    local prefix="${line##*/}"
-
-    local expanded_ipv6=$(expand_ipv6 "$ipv6")
-    local full_hex="${expanded_ipv6//:/}"
-
-    echo "  [0x${full_hex:0:16}n, 0x${full_hex:16:16}n, ${prefix}], // ${line}"
-  done
-}
-
 expand_ipv6() {
  local ipv6="$1"
  local full_ipv6=""
@ -110,9 +105,7 @@ generate_pac() {

    declare -A domain_rules
    local file rule domain
-    for file in *.txt.example; do
-        [ -f "${file%.example}" ] || cp -v "${file}" "${file%.example}"
-    done >&2
+
    for file in domain-rules-*.txt; do
        rule="${file#domain-rules-}"
        rule="${rule%.txt}"
@ -123,62 +116,63 @@ generate_pac() {
            domain_rules["$domain"]="$rule";
        done  < "$file"
    done
-
-    local line item rule parent_rule
-    while read -r line; do
-        echo "$line" >&2
-        rule=
-        case "$line" in
-            (/*)
-                echo "Skip regrex rul: $line"
-                ;;
-            (\!*|\[*)
-                : comment
-                ;;
-            (@@\|\|*)
-                line="${line#@}"
-                ;&
-            (@@\|*)
-                line="${line#@?|}"
-                line="${line#*://}"
-                line="${line%%\%2F*}"
-                domain="${line%%/*}"
-                rule=direct
-                echo "==> direct access: $domain"
-                ;;
-            (\|\|*)
-                line="${line#|}"
-                ;&
-            (\|*)
-                line="${line#|}"
-                line="${line#*://}"
-                ;&
-            ([.a-z0-9]*)
-                line="${line#.}"
-                line="${line%%\%2F*}"
-                domain="${line%%/*}"
-                domain="${domain#*\**.}"
-                rule=proxy
-                echo "==> proxy access: $domain"
-                ;;
-            (*)
-                [[ "$line" =~ ^[[:space:]]*$ ]] ||
-                    echo "Skip: $line"
-                ;;
-        esac >&2
-        [[ -n "${rule}" ]] || continue
-        [[ -z "${domain_rules[$domain]-}" ]] || continue
-        parent_rule=
-        for item in $(domain_segments "$domain"); do
-            if [[ -n "${domain_rules[$item]-}" ]]; then
-                parent_rule="${domain_rules[$item]}"
-                break
+    for file in auto-proxy*.txt; do
+        local line item rule parent_rule
+        while read -r line; do
+            echo "$line" >&2
+            rule=
+            case "$line" in
+                (/*)
+                    echo "Skip regrex rul: $line"
+                    ;;
+                (\!*|\[*)
+                    : comment
+                    ;;
+                (@@\|\|*)
+                    line="${line#@}"
+                    ;&
+                (@@\|*)
+                    line="${line#@?|}"
+                    line="${line#*://}"
+                    line="${line%%\%2F*}"
+                    domain="${line%%/*}"
+                    rule=direct
+                    echo "==> direct access: $domain"
+                    ;;
+                (\|\|*)
+                    line="${line#|}"
+                    ;&
+                (\|*)
+                    line="${line#|}"
+                    line="${line#*://}"
+                    ;&
+                ([.a-z0-9]*)
+                    line="${line#.}"
+                    line="${line%%\%2F*}"
+                    domain="${line%%/*}"
+                    domain="${domain#*\**.}"
+                    rule=proxy
+                    echo "==> proxy access: $domain"
+                    ;;
+                (*)
+                    [[ "$line" =~ ^[[:space:]]*$ ]] ||
+                        echo "Skip: $line"
+                    ;;
+            esac >&2
+            [[ -n "${rule}" ]] || continue
+            [[ -z "${domain_rules[$domain]-}" ]] || continue
+            parent_rule=
+            for item in $(domain_segments "$domain"); do
+                if [[ -n "${domain_rules[$item]-}" ]]; then
+                    parent_rule="${domain_rules[$item]}"
+                    break
+                fi
+            done
+            if [[ -z "$parent_rule" ]] || [[ "$parent_rule" != "$rule" ]]; then
+                domain_rules["$domain"]="$rule"
            fi
-        done
-        if [[ -z "$parent_rule" ]] || [[ "$parent_rule" != "$rule" ]]; then
-            domain_rules["$domain"]="$rule"
-        fi
-    done < <(sed '/URL Keywords/,/^!/d' auto-proxy.txt)
+        done < <(sed '/URL Keywords/,/^!/d' "$file")
+    done

    local domain rule parent_rule
    declare -a segments
@ -196,30 +190,84 @@ generate_pac() {
    done

    sed -n '1,/ begin of ipv4 networks$/p' "$jsfile"
-    cat china.txt |
-        while read -r line; do
-            while IFS=/ read ip prefix; do
+
+    for file in ipv4-rules-*.txt; do
+        rule="${file#ipv?-rules-}"
+        rule="${rule%%.*}"
+        [[ "$rule" = @(blocked|direct|proxy) ]] || rule="\"$rule\""
+        while IFS= read -r line; do
+            line="${line%%#*}"
+            line="${line// }"
+            echo "$rule: $line" >&2
+            while IFS=/ read -r ip prefix; do
                while IFS=. read n1 n2 n3 n4; do
-                    printf "  [0x%02x%02x%02x%02x, %s], // %s\n" "${n1:-0}" "${n2:-0}" "${n3:-0}" "${n4:-0}" "$prefix" "$line"
+                    printf "  [0x%02x%02x%02x%02x, %s, %s], // %s\n" "${n1:-0}" "${n2:-0}" "${n3:-0}" "${n4:-0}" "$prefix" "$rule" "$line"
                done <<< "$ip"
            done <<< "${line}";
-        done
+        done < "$file"
+    done | sort -n
+
    sed -n '/ end of ipv4 networks$/,/ begin of ipv6 networks$/p' "$jsfile"
-    ipv6_to_array_format <china6.txt
+
+    for file in ipv6-rules-*.txt; do
+        rule="${file#ipv?-rules-}"
+        rule="${rule%%.*}"
+        [[ "$rule" = @(blocked|direct|proxy) ]] || rule="\"$rule\""
+        while IFS= read -r line; do
+            line="${line%%#*}"
+            line="${line// }"
+            echo "$rule: $line" >&2
+            local ipv6="${line%%/*}"
+            local prefix="${line##*/}"
+
+            local expanded_ipv6=$(expand_ipv6 "$ipv6")
+            local full_hex="${expanded_ipv6//:/}"
+
+            echo "  [0x${full_hex:0:16}n, 0x${full_hex:16:16}n, ${prefix}, ${rule}], // ${line}"
+        done < "$file"
+    done | sort
+
    sed -n '/ end of ipv6 networks$/,/ begin of proxy rules$/p' "$jsfile"
+
    local domain
    for domain in "${!domain_rules[@]}"; do
        rule="${domain_rules[$domain]}"
        [[ "$rule" = @(blocked|direct|proxy) ]] || rule="\"$rule\""
        printf "  \"%s\": %s,\n" "$domain" "$rule"
-    done | sort
-    sed -n '/ end of proxy rules$/,$p' "$jsfile"
+    done | sort -n
+
+    sed -n '/ end of proxy rules$/,/ begin of regexp rules$/p' "$jsfile"
+
+    for file in domain-regexp*.txt; do
+        rule=""
+        while IFS= read -r line; do
+            line="${line%%#*}"
+            line="${line// }"
+            [[ -n "$line" ]] || continue
+            if [[ "$line" = \[*\] ]]; then
+                rule="${line#[}"
+                rule="${rule%]}"
+                [[ "$rule" = @(blocked|direct|proxy) ]] || rule="\"$rule\""
+            elif [[ -n "$rule" ]]; then
+                printf "  [/%s/, %s],\n" "$line" "$rule"
+            fi
+        done < "$file"
+    done
+
+    sed -n '/ end of regexp rules$/,$p' "$jsfile"
 }

-del_file_on_exit=proxy.pac
-generate_pac "./proxy.js" > proxy.pac
+is_up_to_date=true
+files_to_be_deleted=(proxy.pac)
+for file in "$0" *.js *.txt; do
+    if [ "$file" -nt proxy.pac ]; then
+        is_up_to_date=false
+        break;
+    fi
+done
+"$is_up_to_date" || generate_pac "./proxy.js" > proxy.pac

 if command -v node &>/dev/null; then
    node proxy.pac test
 fi
-del_file_on_exit=
+files_to_be_deleted=()
--- a/domain-regexp.txt.example
+++ b/domain-regexp.txt.example
@ -0,0 +1,8 @@
+[direct]
+^.+-mihayo\.akamaized\.net$
+^a[1-5]\.mzstatic\.com$
+^cdn(-cn)?[1-4]?\.apple-mapkit\.com$
+^cl[1-5]-cdn\.origin-apple\.com\.akadns\.net$
+
+[block]
+^speed\.(coe|open)\.ad\.[a-z]{2,6}\.prod\.hosts\.ooklaserver\.net$
--- a/ipv4-rules-direct.txt.example
+++ b/ipv4-rules-direct.txt.example
@ -0,0 +1,4 @@
+10.0.0.0/8
+100.64.0.0/10
+172.16.0.0/32
+192.168.0.0/16
--- a/ipv6-rules-direct.txt.example
+++ b/ipv6-rules-direct.txt.example
@ -0,0 +1 @@
+2001:db8::/32
--- a/proxy.js
+++ b/proxy.js
@ -10,8 +10,7 @@ var proxyBehaviors = {
  proxy: "SOCKS5 127.0.0.1:1080", // the default proxy
  direct: DIRECT,
  blocked: "PROXY 0.0.0.0:0",
-  "http_proxy": "PROXY 127.0.0.1:3128",
-  "companyProxy": "PROXY 192.168.1.1:8080", // domains list in `domain-rules-companyProxy.txt` will use this proxy setting
+  // "companyProxy": "PROXY 192.168.1.1:8080", // domains list in `domain-rules-companyProxy.txt` will use this proxy setting
 };
 const default_behavior = DIRECT + "; " + proxyBehaviors[proxy];

@ -21,10 +20,6 @@ function isIpAddress(host) {
  return ipv4Pattern.test(host) || ipv6Pattern.test(host);
 }

-function isInDirectAccessNetwork(ip) {
-  return !!findMatchingNetwork(ip, directAccessIPv4Networks, directAccessIPv6Networks);
-}
-
 function ipToNumber(ip) {
  const parts = ip.split('.');
  return parts.reduce((acc, part) => (acc << 8) + parseInt(part, 10), 0);
@ -65,172 +60,21 @@ function twoNumbersToIpv6(high, low) {
  return parts.join(':').replace(/(:0{1,4}){2,}/, '::');
 }

-function findMatchingNetwork(ip, networks4, networks6) {
-  if (ip.includes('.')) { // IPv4
-    const ipNumber = ipToNumber(ip);
-    for (let [network, prefix] of networks4) {
-      mask = subnetMaks32[prefix]
-      if ((ipNumber & mask) === (network & mask)) {
-        return [network, prefix];
-      }
-    }
-  } else { // IPv6
-    const [ipHigh, ipLow] = ipv6ToTwoNumbers(ip);
-    for (let [networkHigh, networkLow, mask] of networks6) {
-      if (mask>64) {
-        maskHigh = 0xffffffffffffffffn;
-        maskLow = subnetMaks64[mask-64];
-      } else {
-        maskHigh = subnetMaks64[mask];
-        maskLow = 0x0000000000000000n;
-      }
-      if (((ipHigh & maskHigh) === (networkHigh & maskHigh)) &&
-        ((ipLow & maskLow) === (networkLow & maskLow))) {
-        return [networkHigh, networkLow, mask];
-      }
-    }
-  }
-  return null;
-}
-
-function printMatchingNetwork(ip, networks4, networks6) {
-  const matchedNetwork = findMatchingNetwork(ip, networks4, networks6);
-  if (matchedNetwork) {
-    if (ip.includes('.')) { // IPv4
-      const [network, prefixLength] = matchedNetwork;
-      return `${numberToIp(network)}/${prefixLength}`;
-    } else { // IPv6
-      const [networkHigh, networkLow, prefixLength] = matchedNetwork;
-      return `${twoNumbersToIpv6(networkHigh, networkLow)}/${prefixLength}`;
-    }
-  } else {
-    return null;
-  }
-}
-
-const subnetMaks32 = [
-  0x00000000, // 0
-  0x80000000, // 1
-  0xc0000000, // 2
-  0xe0000000, // 3
-  0xf0000000, // 4
-  0xf8000000, // 5
-  0xfc000000, // 6
-  0xfe000000, // 7
-  0xff000000, // 8
-  0xff800000, // 9
-  0xffc00000, // 10
-  0xffe00000, // 11
-  0xfff00000, // 12
-  0xfff80000, // 13
-  0xfffc0000, // 14
-  0xfffe0000, // 15
-  0xffff0000, // 16
-  0xffff8000, // 17
-  0xffffc000, // 18
-  0xffffe000, // 19
-  0xfffff000, // 20
-  0xfffff800, // 21
-  0xfffffc00, // 22
-  0xfffffe00, // 23
-  0xffffff00, // 24
-  0xffffff80, // 25
-  0xffffffc0, // 26
-  0xffffffe0, // 27
-  0xfffffff0, // 28
-  0xfffffff8, // 29
-  0xfffffffc, // 30
-  0xfffffffe, // 31
-  0xffffffff, // 32
-];
-
-const subnetMaks64 = [
-  0x0000000000000000n, // 0
-  0x8000000000000000n, // 1
-  0xc000000000000000n, // 2
-  0xe000000000000000n, // 3
-  0xf000000000000000n, // 4
-  0xf800000000000000n, // 5
-  0xfc00000000000000n, // 6
-  0xfe00000000000000n, // 7
-  0xff00000000000000n, // 8
-  0xff80000000000000n, // 9
-  0xffc0000000000000n, // 10
-  0xffe0000000000000n, // 11
-  0xfff0000000000000n, // 12
-  0xfff8000000000000n, // 13
-  0xfffc000000000000n, // 14
-  0xfffe000000000000n, // 15
-  0xffff000000000000n, // 16
-  0xffff800000000000n, // 17
-  0xffffc00000000000n, // 18
-  0xffffe00000000000n, // 19
-  0xfffff00000000000n, // 20
-  0xfffff80000000000n, // 21
-  0xfffffc0000000000n, // 22
-  0xfffffe0000000000n, // 23
-  0xffffff0000000000n, // 24
-  0xffffff8000000000n, // 25
-  0xffffffc000000000n, // 26
-  0xffffffe000000000n, // 27
-  0xfffffff000000000n, // 28
-  0xfffffff800000000n, // 29
-  0xfffffffc00000000n, // 30
-  0xfffffffe00000000n, // 31
-  0xffffffff00000000n, // 32
-  0xffffffff80000000n, // 33
-  0xffffffffc0000000n, // 34
-  0xffffffffe0000000n, // 35
-  0xfffffffff0000000n, // 36
-  0xfffffffff8000000n, // 37
-  0xfffffffffc000000n, // 38
-  0xfffffffffe000000n, // 39
-  0xffffffffff000000n, // 40
-  0xffffffffff800000n, // 41
-  0xffffffffffc00000n, // 42
-  0xffffffffffe00000n, // 43
-  0xfffffffffff00000n, // 44
-  0xfffffffffff80000n, // 45
-  0xfffffffffffc0000n, // 46
-  0xfffffffffffe0000n, // 47
-  0xffffffffffff0000n, // 48
-  0xffffffffffff8000n, // 49
-  0xffffffffffffc000n, // 50
-  0xffffffffffffe000n, // 51
-  0xfffffffffffff000n, // 52
-  0xfffffffffffff800n, // 53
-  0xfffffffffffffc00n, // 54
-  0xfffffffffffffe00n, // 55
-  0xffffffffffffff00n, // 56
-  0xffffffffffffff80n, // 57
-  0xffffffffffffffc0n, // 58
-  0xffffffffffffffe0n, // 59
-  0xfffffffffffffff0n, // 60
-  0xfffffffffffffff8n, // 61
-  0xfffffffffffffffcn, // 62
-  0xfffffffffffffffen, // 63
-  0xffffffffffffffffn, // 64
-];
-
-const directAccessIPv4Networks = [
-  [0xC0A80000, 16], // 192.168.0.0/16
-  [0x0A000000, 8], // 10.0.0.0/8
-  [0xAC100000, 12], // 172.16.0.0/12
-  [0x7F000000, 8], // 127.0.0.0/8 (Loopback)
-  [0xA9FE0000, 16], // 169.254.0.0/16 (Link Local)
-  [0x64400000, 10], // 100.64.0.0/10 (Carrier-grade NAT)
+const ipv4NetworkRules = [
+  [0x7F000000, 8 , direct], // 127.0.0.0/8 (Loopback)
+  [0xA9FE0000, 16, direct], // 169.254.0.0/16 (Link Local)
+  [0x64400000, 10, direct], // 100.64.0.0/10 (Carrier-grade NAT)
  // begin of ipv4 networks
  // end of ipv4 networks
 ];

-const directAccessIPv6Networks = [
-  [0x0000000000000000n, 0x0000000000000000n, 128], // ::/128 (Unspecified Address)
-  [0x0000000000000000n, 0x0000000000000001n, 128], // ::1/128 (Loopback Address)
-  [0x20010db800000000n, 0x0000000000000000n, 32], // 2001:db8::/32 (Documentation Address)
-  [0xfc00000000000000n, 0x0000000000000000n, 7], // fc00::/7 (Unique Local Address)
-  [0xff00000000000000n, 0x0000000000000000n, 8], // ff00::/8 (Multicast Address)
-  [0x2001000000000000n, 0x0000000000000000n, 16], // 2001::/16 (Teredo Address)
-  [0xfe80000000000000n, 0x0000000000000000n, 10], // fe80::/10 (Link-Local Address)
+const ipv6NetworkRules = [
+  [0x0000000000000000n, 0x0000000000000000n, 128, direct], // ::/128 (Unspecified Address)
+  [0x0000000000000000n, 0x0000000000000001n, 128, direct], // ::1/128 (Loopback Address)
+  [0xfc00000000000000n, 0x0000000000000000n, 7  , direct], // fc00::/7 (Unique Local Address)
+  [0xff00000000000000n, 0x0000000000000000n, 8  , direct], // ff00::/8 (Multicast Address)
+  [0x2001000000000000n, 0x0000000000000000n, 16 , direct], // 2001::/16 (Teredo Address)
+  [0xfe80000000000000n, 0x0000000000000000n, 10 , direct], // fe80::/10 (Link-Local Address)
  // begin of ipv6 networks
  // end of ipv6 networks
 ];
@ -244,10 +88,201 @@ const proxyRules = {
  // end of proxy rules
 };

-function FindProxyForURL(url, host) {
+const domainRegexpRules = [
+  [ /^adservice\.google\.([a-z]{2}|com?)(\.[a-z]{2})?$/, blocked], // adservice.google.com.xx
+  // begin of regexp rules
+  // end of regexp rules
+]
+
+class IPv4TrieNode {
+  constructor() {
+    this.children = [null, null]; // 0 and 1
+    this.isEnd = false;
+    this.network = null;
+    this.prefix = null;
+    this.action = direct;
+  }
+}
+
+class IPv4PrefixTrie {
+  constructor() {
+    this.root = new IPv4TrieNode();
+  }
+
+  static buildTrieFromData(data) {
+    const trie = new IPv4PrefixTrie();
+    for (const [network, prefix, action] of data) {
+      let node = trie.root;
+      node = IPv4PrefixTrie._insertBits(node, network, prefix);
+      node.isEnd = true;
+      node.network = network;
+      node.prefix = prefix;
+      node.action = action
+    }
+    return trie;
+  }
+
+  static _insertBits(node, value, bits) {
+    let mask = 0x80000000;
+    for (let i = 0; i < bits; i++) {
+      const bitIndex = ((value & mask) !== 0 ? 1 : 0);
+      mask = mask >>> 1;
+
+      if (!node.children[bitIndex]) {
+        node.children[bitIndex] = new IPv4TrieNode();
+      }
+      node = node.children[bitIndex];
+    }
+    return node;
+  }
+
+  search(ip) {
+    let node = this.root;
+    let lastMatch = null;
+
+    node = IPv4PrefixTrie._searchBits(node, ip, 32, (matchedNode) => {
+      if (matchedNode.isEnd) {
+        lastMatch = matchedNode;
+      }
+    });
+
+    return lastMatch;
+  }
+
+  static _searchBits(node, value, bits, callback) {
+    let mask = 0x80000000;
+    for (let i = 0; i < bits; i++) {
+      const bitIndex = ((value & mask) !== 0 ? 1 : 0);
+      mask = mask >>> 1;
+
+      if (!node.children[bitIndex]) {
+        return null;
+      }
+      node = node.children[bitIndex];
+      callback(node);
+    }
+    return node;
+  }
+}
+
+class IPv6TrieNode {
+  constructor() {
+    this.children = [null, null]; // 0 and 1
+    this.isEnd = false;
+    this.networkHigh = null;
+    this.networkLow = null;
+    this.prefix = null;
+    this.action = direct;
+  }
+}
+
+class IPv6PrefixTrie {
+  constructor() {
+    this.root = new IPv6TrieNode();
+  }
+
+  static buildTrieFromData(data) {
+    const trie = new IPv6PrefixTrie();
+    for (const [networkHigh, networkLow, prefix, action] of data) {
+      let node = trie.root;
+      node = IPv6PrefixTrie._insertBits(node, networkHigh, Math.min(prefix, 64));
+      if (prefix > 64) {
+        node = IPv6PrefixTrie._insertBits(node, networkLow, prefix - 64);
+      }
+      node.isEnd = true;
+      node.networkHigh = networkHigh;
+      node.networkLow = networkLow;
+      node.prefix = prefix;
+      node.action = action;
+    }
+    return trie;
+  }
+
+  static _insertBits(node, value, bits) {
+    let mask = 0x8000000000000000n;
+    for (let i = 0; i < bits; i++) {
+      const bitIndex = ((value & mask) !== 0n ? 1 : 0);
+      mask = mask >> 1n;
+
+      if (!node.children[bitIndex]) {
+        node.children[bitIndex] = new IPv6TrieNode();
+      }
+      node = node.children[bitIndex];
+    }
+    return node;
+  }
+
+  search(ipHigh, ipLow) {
+    let node = this.root;
+    let lastMatch = null;
+
+    node = IPv6PrefixTrie._searchBits(node, ipHigh, 64, (matchedNode) => {
+      if (matchedNode.isEnd) {
+        lastMatch = matchedNode;
+      }
+    });
+
+    if (node) {
+      node = IPv6PrefixTrie._searchBits(node, ipLow, 64, (matchedNode) => {
+        if (matchedNode.isEnd) {
+          lastMatch = matchedNode;
+        }
+      });
+    }
+
+    return lastMatch;
+  }
+
+  static _searchBits(node, value, bits, callback) {
+    let mask = 0x8000000000000000n;
+    for (let i = 0; i < bits; i++) {
+      const bitIndex = ((value & mask) !== 0n ? 1 : 0);
+      mask = mask >> 1n;
+
+      if (!node.children[bitIndex]) {
+        return null;
+      }
+      node = node.children[bitIndex];
+      callback(node);
+    }
+    return node;
+  }
+}
+
+const ipv4Trie = IPv4PrefixTrie.buildTrieFromData(ipv4NetworkRules);
+const ipv6Trie = IPv6PrefixTrie.buildTrieFromData(ipv6NetworkRules);
+
+function findMatchingNetwork(ip, networks4, networks6) {
+  if (ip.includes('.')) { // IPv4
+    const ipNumber = ipToNumber(ip);
+    return ipv4Trie.search(ipNumber);
+  } else { // IPv6
+    const [ipHigh, ipLow] = ipv6ToTwoNumbers(ip);
+    return ipv6Trie.search(ipHigh, ipLow);
+  }
+  return null;
+}
+
+function printMatchingNetwork(ip, networks4, networks6) {
+  const matchedNetwork = findMatchingNetwork(ip, networks4, networks6);
+  if (matchedNetwork) {
+    if (ip.includes('.')) { // IPv4
+      const trie = matchedNetwork;
+      return `${numberToIp(trie.network)}/${trie.prefix}`;
+    } else { // IPv6
+      const trie = matchedNetwork;
+      return `${twoNumbersToIpv6(trie.networkHigh, trie.networkLow)}/${trie.prefix}`;
+    }
+  } else {
+    return null;
+  }
+}
+function FindProxyForURL(_url, _host) {
+  const host = _host;
  if (isIpAddress(host)) {
-    if(isInDirectAccessNetwork(host)) {
-      return DIRECT;
+    const match = findMatchingNetwork(host);
+    if(match) {
+      return proxyBehaviors[match.action] || default_behavior;
    } else {
      var action = proxyRules[host];
      if (action !== undefined) {
@ -257,32 +292,39 @@ function FindProxyForURL(url, host) {
    }
  }

+  const match = domainRegexpRules.find(([regexp, value]) => regexp.test(host) );
+  if(match)
+    return proxyBehaviors[match[1]] || default_behavior;
+
+  var host_segment = host;
  while (true) {
-    var action = proxyRules[host];
+    var action = proxyRules[host_segment];
    if (action !== undefined) {
      return proxyBehaviors[action] || default_behavior;
    }
-    var nextDot = host.indexOf(".");
+    var nextDot = host_segment.indexOf(".");
    if (nextDot === -1) {
      break;
    }
-    host = host.substring(nextDot + 1);
+    host_segment = host_segment.substring(nextDot + 1);
  }
+
  var remote_ip = undefined;
  if(typeof dnsResolveEx == 'function') {
    remote_ip = dnsResolveEx(host);
  } else if(typeof dnsResolve == 'function') {
    remote_ip = dnsResolve(host);
  }
-  if(remote_ip !== undefined && isInDirectAccessNetwork(remote_ip)) {
-    return DIRECT
+  if(remote_ip !== undefined) {
+    const match = findMatchingNetwork(remote_ip);
+    if (match) return proxyBehaviors[match.action] || default_behavior;
  }
  return default_behavior;
 }

 if (typeof process !== 'undefined' && process.argv.includes('test')) {
  function assertNetwork(ip, expected) {
-    const result = printMatchingNetwork(ip, directAccessIPv4Networks, directAccessIPv6Networks);
+    const result = printMatchingNetwork(ip, ipv4NetworkRules, ipv6NetworkRules);
    if (result === expected) {
      console.log(`OK: Test for ${ip} passed.`);
    } else {
@ -316,17 +358,16 @@ if (typeof process !== 'undefined' && process.argv.includes('test')) {
  }

  function runTests() {
-    assertNetwork("192.168.1.10", "192.168.0.0/16");
+    assertNetwork("127.234.168.10", "127.0.0.0/8");
    assertNetwork("1.1.1.1", null);
-    assertNetwork("172.19.1.1", "172.16.0.0/12");
-    assertNetwork("2001:0db8:85a3:0000:0000:8a2e:0370:7334", "2001:db8::/32");
    assertNetwork("fe80::f0:c6b3:c766:9b1e", "fe80::/10");
    assertVisitHostWithProxy("com.google");
    assertVisitHostWithProxy("domains.google");
    assertHostWithDefaultAction("www.not-google");
-    assertDirectHost("10.3.4.5");
+    assertDirectHost("127.3.4.5");
    assertDirectHost("114.114.114.114");
    assertBlockedHost("www.whitehouse.com");
+    assertBlockedHost("adservice.google.com.xx")
  }

  runTests();
Author	SHA1	Message	Date
Chai Feng	b59139d022	Update build-and-release.yml	2024-11-02 16:16:26 +08:00
Chai Feng	244e85c183	Update build.sh	2024-10-10 12:16:19 +08:00
Chai Feng	bd8cc7c25a	Update proxy.js	2024-10-09 11:37:46 +08:00
Chai Feng	0b4e98341d	Add domain regular expressions	2024-10-09 11:06:34 +08:00
Chai Feng	af95f50613	Update README	2024-10-08 14:20:41 +08:00
Chai Feng	cf29ce609a	Update README.md	2024-10-08 11:12:23 +08:00
Chai Feng	7512d30dc3	Trigger the pipeline only when .example, .js, or *.sh files are modified on the master branch	2024-10-08 10:58:31 +08:00
Chai Feng	f4c7d5ff64	Update build.sh	2024-10-08 10:08:09 +08:00
Chai Feng	04bdd59395	Update README.md	2024-10-08 10:02:06 +08:00
Chai Feng	7d686975b9	Quickly search for matching IP network segments, support for any number of proxy configurations.	2024-10-08 09:38:38 +08:00