PVE下 LXC 容器安装 Zerotier 一直 OFFLINE

前言

目前是在PVE下安装Openwrt做主路由，基于LXC容器安装Debian11 做服务器使用，有时候想在外面访问家里的某些服务或文件，没有公网IP但是有公网服务器那就只能做内网穿透了，之前用过nps、frp、zerotier，这三种使用起来感觉各有千秋，但是访问大文件的时候，zerotier还是很有优势的，也不怎么占用公网服务器的流量。

整理教程时的系统环境

CT模板：debian-11-standard_11.3-1_amd64.tar.zst
zerotier-one (1.10.1)
zerotier-one (1.8.10)

先常规创建特权容器，然后编辑配置文件，添加 /dev/net/tun，启动容器后安装软件。

LXC 添加 /dev/net/tun

官方文档 OpenVPN in LXC 中，nano /etc/pve/lxc/123.conf

Add the following lines at the end (if you’re using PVE < 7.0, change cgroup2 with cgroup)
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir

还有官方论坛中，有网友提出的解决方案：PVE 7 OpenVPN (LXC) problem: Cannot open TUN/TAP dev

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.hook.autodev: sh -c "modprobe tun; cd ${LXC_ROOTFS_MOUNT}/dev; mkdir net; mknod net/tun c 10 200; chmod 0666 net/tun"

The LXC also need to be privileged

再有国内大佬：解决在 LXC容器 (PVE) 运行 Openwrt 时无法使用 Zerotier 问题，这个和官方文档差不多

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

我感觉用哪种都可以，都是加 /dev/net/tun 设备，主要看你的容器是否为特权容器，非特权容器参数不同，这里不做说明。接下来安装Zerotier，两种方式看个人需求了。

安装 Zerotier

使用脚本一键安装

使用官方在线一键安装脚本，直接安装 zerotier

curl -s https://install.zerotier.com/ | sudo bash

如果提示-bash: sudo: command not found，那就安装 sudo 后再执行apt install sudo -y

root@docker:~# curl -s https://install.zerotier.com/ | sudo bash

*** ZeroTier Service Quick Install for Unix-like Systems

*** Tested OSes / distributions:

***   MacOS (10.13+) (just installs ZeroTier One.pkg)
***   Debian Linux (7+)
***   RedHat/CentOS Linux (6+)
***   Fedora Linux (16+)
***   SuSE Linux (12+)
***   Mint Linux (18+)

*** Supported architectures vary by OS / distribution. We try to support
*** every system architecture supported by the target.

*** Please report problems to contact@zerotier.com and we will try to fix.

*** Detecting Linux Distribution

*** Found Debian, creating /etc/apt/sources.list.d/zerotier.list
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK

*** Installing zerotier-one package...
Hit:1 https://mirrors.ustc.edu.cn/debian bullseye InRelease
Get:2 https://mirrors.ustc.edu.cn/debian-security bullseye-security InRelease [48.4 kB]
Get:3 https://mirrors.ustc.edu.cn/debian bullseye-updates InRelease [44.1 kB]                                
Get:4 https://mirrors.ustc.edu.cn/debian bullseye-backports InRelease [49.0 kB]                              
Get:5 https://mirrors.ustc.edu.cn/debian-security bullseye-security/main Sources [143 kB]                    
Get:6 https://mirrors.ustc.edu.cn/debian-security bullseye-security/main amd64 Packages [175 kB]                       
Get:7 https://mirrors.ustc.edu.cn/debian-security bullseye-security/main Translation-en [109 kB]           
Get:8 http://download.zerotier.com/debian/bullseye bullseye InRelease [36.9 kB]                            
Get:9 https://mirrors.ustc.edu.cn/debian bullseye-backports/main Sources.diff/Index [63.3 kB]
Get:10 https://mirrors.ustc.edu.cn/debian bullseye-backports/main amd64 Packages.diff/Index [63.3 kB]
Get:11 https://mirrors.ustc.edu.cn/debian bullseye-backports/main Sources T-2022-08-09-2007.14-F-2022-08-09-1403.00.pdiff [1047 B]
Get:11 https://mirrors.ustc.edu.cn/debian bullseye-backports/main Sources T-2022-08-09-2007.14-F-2022-08-09-1403.00.pdiff [1047 B]
Get:12 https://mirrors.ustc.edu.cn/debian bullseye-backports/main amd64 Packages T-2022-08-09-2007.14-F-2022-08-09-2007.14.pdiff [636 B]
Hit:13 https://download.docker.com/linux/debian bullseye InRelease                                    
Get:12 https://mirrors.ustc.edu.cn/debian bullseye-backports/main amd64 Packages T-2022-08-09-2007.14-F-2022-08-09-2007.14.pdiff [636 B]
Get:14 http://download.zerotier.com/debian/bullseye bullseye/main amd64 Packages [3682 B]
Fetched 737 kB in 1s (558 kB/s)      
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  zerotier-one
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 3157 kB of archives.
After this operation, 10.9 MB of additional disk space will be used.
Get:1 http://download.zerotier.com/debian/bullseye bullseye/main amd64 zerotier-one amd64 1.10.1 [3157 kB]
Fetched 3157 kB in 3s (1208 kB/s)       
Selecting previously unselected package zerotier-one.
(Reading database ... 22223 files and directories currently installed.)
Preparing to unpack .../zerotier-one_1.10.1_amd64.deb ...
Unpacking zerotier-one (1.10.1) ...
Setting up zerotier-one (1.10.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/zerotier-one.service -> /lib/systemd/system/zerotier-one.service.
Processing triggers for man-db (2.9.4-2) ...

*** Enabling and starting ZeroTier service...
Synchronizing state of zerotier-one.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable zerotier-one

*** Waiting for identity generation...

*** Success! You are ZeroTier address [ 70f814151f ].

看到Success且没有报错，那就安装成功了

使用 Docker 安装

这里使用官方镜像，最后的 ID 需要替换成自己的

docker run -d           \
  --name zerotier-one   \
  --restart=always      \
  --net=host            \
  --cap-add=NET_ADMIN   \
  --device=/dev/net/tun \
  -v /zerotier-one:/var/lib/zerotier-one \
  zerotier/zerotier mtuj99m3qfjknkpb

加入网络

使用 Docker 安装方式的这一步就可以跳过了，因为最后的参数就是要加入的网络 ID，常规安装方式需要执行此命令，最后的 ID 需要替换成自己的

zerotier-cli join mtuj99m3qfjknkpb

出现问题

OFFLINE

输入zerotier-cli status后得到的不是想要的，状态是OFFLINE，正常应该是ONLINE才对

root@docker:~# zerotier-cli status      
200 info 70f814151f 1.10.1 OFFLINE

REQUESTING_CONFIGURATION

输入zerotier-cli listnetworks后得到的也不是想要的

root@docker:~# zerotier-cli listnetworks
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks mtuj99m3qfjknkpb  12:34:56:78:9a:bc REQUESTING_CONFIGURATION PRIVATE ztswmn9yzn -

正常应该是这种：

root@iStoreOS:~# zerotier-cli listnetworks
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks mtuj99m3qfjknkpb xxx-network 12:34:56:78:9a:bc OK PRIVATE ztswmn9yzn 10.1.1.2/24

connection failed

更换 docker 容器安装方式

root@docker:~# docker logs zerotier-one 
=> Configuring networks to join
=> Joining networks: []
=> Starting ZeroTier
===> ZeroTier hasn't started, waiting a second
=> Writing healthcheck for networks: []
=> zerotier-cli info: [Error connecting to the ZeroTier service: connection failed

Please check that the service is running and that TCP port 9993 can be contacted via 127.0.0.1.]
=> Sleeping infinitely
=> Killing zerotier
=> Configuring networks to join
=> Joining networks: []
=> Starting ZeroTier
=> Writing healthcheck for networks: []
=> zerotier-cli info: [200 info 0b5ada83ff 1.8.10 OFFLINE]
=> Sleeping infinitely

解决方案

一开始还以为是 /dev/net/tun 设备没加上，于是网上搜了一圈，基本上都是上面的两种添加方案，挨个试了都不行，重新新建容器编辑配置也不行，而且我并没有出现网友截图中所出现的PORT ERROR问题，所以可以排除我没配置好。当看到REQUESTING_CONFIGURATION的时候想到可能是因为 OpenWrt 主路由的防火墙问题，因为我目前方案，所有容器都走虚拟网桥，能够保证下面所有设备都可以访问各个容器，但是我又不太会玩OpenWrt，这就导致很久没解决。看到恩山论坛有大佬在防火墙命令导致zerotier故障帖子2、3楼给出解决方案，我这和他的应该不太一样，因为我是在容器中生成zt开头的接口，并不是在主路由中安装Zerotier，试了一下手动加也没成功。如果安装精简版OpenWrt后又在里面安装Zerotier的同学出现了这个问题，可以尝试一下这个方案。

虽然没解决，但是方向应该是找对了的，一计不成又生一计，我想到用端口转发来解决，系统自带的防火墙端口转发功能，外部区域只能选 WAN，内部区域只能选 LAN，而且我这 WAN 口是 DHCP 上网方式，结果使用 WAN 口获取的上级 IP 地址加转发的端口可以访问容器内的服务，却解决不了 zerotier 无法在线的问题！还是不太会玩这个系统啊，看到OpenWrt中有Zerotier，试了一下很好用，还用命令加入了Moon 服务器，不过主路由和容器虽然一个网段但是不能直接使用Zerotier访问，还得选别的路才行，网上搜了一下说有一个工具Socat端口转发很强，拿来试试果真好用，没有外部区域和内部区域，可以转发到 LAN 口，那么容器内就不需要装Zerotier了，属于曲线解决了需求，但是问题依旧没解决。

目录CONTENT