Building a Cross-City LAN with Docker

January 20, 2017 Docker Docker, VPN 1165 words 6 min read

Overview

At the time, I had servers both at home and at the office and wanted unified management, but the two internal networks were isolated. Traditional port forwarding was too cumbersome, and some services I didn’t want exposed to the public internet. After trying several approaches, I found Docker OpenVPN to be the most hassle-free solution.

I chose Docker OpenVPN over traditional VPN mainly because containerized deployment is extremely convenient, and certificate management is much simpler. The end result was great — I could access various services on the remote internal network via domain names, saving a lot of configuration hassle.

Network Topology

mermaid
graph TD
    classDef gateway fill:#e67e22,stroke:#d35400,color:#fff,stroke-width:2px
    classDef lb fill:#2980b9,stroke:#2471a3,color:#fff,stroke-width:2px
    classDef proxy fill:#5dade2,stroke:#2e86c1,color:#fff,stroke-width:1px
    classDef vpn fill:#5dade2,stroke:#2e86c1,color:#fff,stroke-width:1px
    classDef arm fill:#85c1e9,stroke:#2e86c1,color:#333,stroke-width:1px
    classDef isp fill:#76d7c4,stroke:#1abc9c,color:#333,stroke-width:1px
    classDef user fill:#bdc3c7,stroke:#7f8c8d,color:#333,stroke-width:1px
    
    subgraph overseas["Overseas Service Zone"]
        UserOut@{ shape: rounded, label: "External User" }:::user
        OS@{ shape: rounded, label: "Overseas Service" }:::user
        CDN@{ shape: rounded, label: "CDN" }:::user
        SS1@{ shape: hex, label: "SS Server 1" }:::proxy
        SS2@{ shape: hex, label: "SS Server 2" }:::proxy
        UserOut -->|User Access| OS
        OS -->|User Access| CDN
        CDN -->|User Access| SS2
        OS -->|Shadowsocks| SS1
        OS -->|Shadowsocks| SS2
    end
    
    GFW@{ shape: hex, label: "GFW Firewall" }:::gateway
    LB@{ shape: hex, label: "Load Balancer" }:::lb
    
    subgraph internal["Internal Core Zone"]
        subgraph proxyVpn["Access Proxy & VPN"]
            UserIn@{ shape: rounded, label: "Internal User" }:::user
            Nginx@{ shape: hex, label: "Nginx" }:::proxy
            Haproxy@{ shape: hex, label: "Haproxy" }:::proxy
            OpenVPN@{ shape: hex, label: "OpenVPN" }:::vpn
            UserIn -->|User Access| Nginx
            UserIn -->|User Access| Haproxy
            Nginx <-->|DockerNet| Haproxy
            Haproxy -->|DockerNet| OpenVPN
        end
        subgraph armCluster["Multi-ISP ARM Cluster"]
            ARM1@{ shape: rounded, label: "ARM1" }:::arm --> Telecom@{ shape: hex, label: "Telecom AD" }:::isp
            ARM2@{ shape: rounded, label: "ARM2" }:::arm --> Mobile@{ shape: hex, label: "Mobile AD" }:::isp
            ARM3@{ shape: rounded, label: "ARM3" }:::arm --> Unicom@{ shape: hex, label: "Unicom AD" }:::isp
        end
        Mickey@{ shape: rounded, label: "Mickey" }:::user
    end
    
    SS1 -->|Shadowsocks| GFW
    SS2 -->|Shadowsocks| GFW
    GFW -->|Shadowsocks| LB
    LB -->|SS| ARM1
    LB -->|SS| ARM2
    LB -->|SS| ARM3
    
    SS1 -.->|TCP BBR| GFW
    GFW -.->|BBR| ARM1
    GFW -.->|BBR| ARM2
    GFW -.->|BBR| ARM3
    
    OpenVPN -->|VPN Tunnel| Mickey
    Telecom -->|VPN| Mickey
    Mobile -->|VPN| Mickey
    Unicom -->|VPN| Mickey

The overall architecture is divided into three zones: the overseas service zone traverses the firewall via Shadowsocks, passes through GFW and load balancing before being distributed to the internal ARM server cluster. The ARM boards are connected to three ISP lines — Telecom, Mobile, and Unicom — as egress points. The internal network uses Nginx/Haproxy as proxy access points, and OpenVPN provides VPN tunnels for personal users to access remotely.

Proxy Flow

mermaid
graph TD
    classDef client fill:#27ae60,stroke:#1e8449,color:#fff,stroke-width:2px
    classDef proxy7 fill:#5dade2,stroke:#2471a3,color:#fff,stroke-width:1px
    classDef streamSelect fill:#e67e22,stroke:#d35400,color:#fff,stroke-width:1px
    classDef stream fill:#76d7c4,stroke:#1abc9c,color:#333,stroke-width:1px
    classDef cache fill:#27ae60,stroke:#1e8449,color:#fff,stroke-width:2px
    classDef lb7 fill:#e67e22,stroke:#d35400,color:#fff,stroke-width:1px
    classDef backend fill:#27ae60,stroke:#1e8449,color:#fff,stroke-width:2px
    classDef moduleBox fill:#ecf0f1,stroke:#7f8c8d,stroke-width:1px
    
    subgraph L1_7["Layer 1 Proxy · Layer 7"]
        direction TB
        C@{ shape: rounded, label: "Client" }:::client
        A1@{ shape: hex, label: "Layer 1 Proxy" }:::proxy7
        S1@{ shape: hex, label: "Stream to Layer 1" }:::streamSelect
        Cache@{ shape: cyl, label: "Cache" }:::cache
        C --> A1
        A1 --> S1
        S1 --> Cache
    end
    
    subgraph L1_4["Layer 1 Proxy · Layer 4"]
        direction TB
        ST1@{ shape: hex, label: "Layer 1 Stream" }:::stream
        SE1@{ shape: hex, label: "Select Next Layer Stream" }:::streamSelect
        S1 --> ST1
        ST1 --> SE1
    end
    
    subgraph L2_4["Layer 2 Proxy · Layer 4"]
        direction TB
        ST2@{ shape: hex, label: "Layer 2 Stream" }:::stream
        SE2@{ shape: hex, label: "Select Next Layer Stream" }:::streamSelect
        SE1 --> ST2
        ST2 --> SE2
    end
    
    subgraph L3_7["Layer 3 Proxy · Layer 7"]
        direction TB
        A3@{ shape: hex, label: "Layer 3 Proxy" }:::proxy7
        LB7@{ shape: hex, label: "Layer 7 Load Balancer" }:::lb7
        Backend@{ shape: rounded, label: "Backend System" }:::backend
        SE2 --> A3
        A3 --> LB7
        LB7 --> Backend
    end

Step-by-Step Guide

Purchase a Cloud Server and Deploy Docker

You need a public VPS as a relay, because OpenVPN requires a public entry point for clients to connect to. It’s recommended to purchase a Linux system that supports systemctl for easier management, and deploy Docker:

Format external storage as an XFS partition:

shell
1
2
  # mkfs.xfs /dev/vdb5
  # echo "/dev/vdb5 /mnt/data xfs defaults 1 1" |tee -a /etc/fstab

Adjust the Docker directory using one of two methods:

Mount the /var/lib/docker directory:

shell
1
2
3
4
5
6
  # systemctl stop docker
  # mkdir -p /mnt/data/docker
  # rsync -aXS /var/lib/docker/.  /mnt/data/docker/
  # echo "/mnt/data/docker /var/lib/docker none bind 0 0"|tee -a /etc/fstab
  # mount -a
  # systemctl start docker

Specify a custom directory: Modify /etc/systemd/system/multi-user.target.wants/docker.service as follows:

shell
1
  ExecStart=/usr/bin/dockerd --storage-driver=overlay2 -g /mnt/hhd/docker

Install the docker-openvpn Service

docker-openvpn

Pick a name for the $OVPN_DATA data volume container, it will be created automatically.

shell
1
  # OVPN_DATA="ovpn-data"

Initialize the $OVPN_DATA container that will hold the configuration files and certificates

shell
1
2
3
4
5
6
  # docker volume create --name $OVPN_DATA
  # docker run -v $OVPN_DATA:/etc/openvpn \
    --rm kylemanna/openvpn ovpn_genconfig \
    -u udp://VPN.SERVERNAME.COM
  # docker run -v $OVPN_DATA:/etc/openvpn \
    --rm -it kylemanna/openvpn ovpn_initpki

If using TCP

shell
1
2
3
  # docker run -v $OVPN_DATA:/etc/openvpn \
    --rm kylemanna/openvpn ovpn_genconfig \
    -u tcp://VPN.SERVERNAME.COM:1443

Start OpenVPN server process

shell
1
2
3
  # docker run -v $OVPN_DATA:/etc/openvpn -d \
    -p 1194:1194/udp \
    --cap-add=NET_ADMIN kylemanna/openvpn

shell
1
2
3
  # docker run -v $OVPN_DATA:/etc/openvpn -d \
    -p 1443:1194/tcp \
    --cap-add=NET_ADMIN kylemanna/openvpn

Running a Second Fallback TCP Container

shell
1
2
3
4
  # docker run -v $OVPN_DATA:/etc/openvpn \
    --rm -p 1443:1194/tcp \
    --privileged kylemanna/openvpn ovpn_run \
    --proto tcp

Generate a client certificate without a passphrase. Retrieve the client configuration with embedded certificates. “CLIENTNAME” can be customized:

shell
1
2
3
4
  # docker run -v $OVPN_DATA:/etc/openvpn \
    --rm -it kylemanna/openvpn easyrsa build-client-full CLIENTNAME nopass
  # docker run -v $OVPN_DATA:/etc/openvpn \
    --rm kylemanna/openvpn ovpn_getclient CLIENTNAME > CLIENTNAME.ovpn

Add routing rules so that the Docker internal network can access VPN client nodes Add a routing rule on the Docker host so that other containers can access OpenVPN client nodes through the default network:

shell
1
  # ip route add 192.168.255.0/24 via $DOCKER_OPENVPN_IP

Configure static internal/external IPs for clients to ensure consistent IPs across connections, simplifying proxy configuration

shell
1
2
# cat ccd/CLIENTNAME
ifconfig-push 192.168.255.10 192.168.255.9

Deploy Frontend Proxy
- Use DOCKER-CADDY as the reverse proxy, mainly for automatic HTTPS and simple configuration

shell
1
2
3
4
5
6
7
docker run -d \
  -v $(pwd)/Caddyfile:/etc/Caddyfile \
  -v $HOME/.caddy:/root/.caddy \
  -p 80:80 -p 443:443 \
  --name caddy \
  --link openvpn:openvpn \
  abiosoft/caddy

Caddy configuration reference:

ini
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
  http://git.mickeybee.cn {
    redir https://git.mickeybee.cn{url}
  }
  https://git.mickeybee.cn {
    gzip
    proxy / 192.168.xxx.xx:3000
    tls [email protected] {
      max_certs 10
      key_type  p256
    }
  }

Deploy TCP Proxy
- Use DOCKER-HAPROXY for deployment, suitable for handling database connections, SSH, and other long-lived connections

shell
1
2
3
4
5
6
docker run -d \
  -v $(pwd)/haproxy:/usr/local/etc/haproxy:ro \
  -p xxx:xxx -p yyy:yyy \
  --name haproxy \
  --link openvpn:openvpn \
  haproxy

Map Backend Ports
Add backend service mappings in the HAProxy configuration file, specifying the IP addresses and port numbers, supporting both TCP and HTTP proxy modes. Restart the HAProxy service after configuration changes to apply them.

Overview

Network Topology

Proxy Flow

Step-by-Step Guide

Purchase a Cloud Server and Deploy Docker

Install the docker-openvpn Service

Deploy Frontend Proxy

Deploy TCP Proxy

Map Backend Ports