tunnel
Tunnel acts as the bridge between edge and cloud. It consists of tunnel-cloud and tunnel-edge, which is responsible for maintaining persistent cloud-to-edge network connection. It allows edge nodes with no public IP assigned to be managed by Kubernetes on the cloud to obtain unified and centralized operation and maintenance.
Architecture Diagram
Implementation
Node Registration
- The tunnel-edge on the edge node actively connects to tunnel-cloud service, and tunnel-cloud service transfers the request to the tunnel-cloud pod according to the load balancing policy.
- After tunnel-edge establishes a gRPC connection with tunnel-cloud, tunnel-cloud will write the mapping of its podIp and nodeName of the node where tunnel-edge is located into DNS。If the gRPC connection is disconnected, tunnel-cloud will delete the podIp and node name mapping.
Cloud Request Forwarding
- When apiserver or other cloud applications access the kubelet or other applications on the edge node, the tunnel-dns uses DNS hijacking (resolving the node name in the host to the podIp of tunnel-cloud) to forward the request to the pod of the tunnel-cloud.
- The tunnel-cloud forwards the request information to the gRPC connection established with the tunnel-edge according to the node name.
- The tunnel-edge requests the application on the edge node according to the received request information.
Configuration File
The tunnel component includes tunnel-cloud and tunnel-edge. The tunnel-edge running on the edge node establishes a gRPC long-lived with the tunnel-cloud running on the cloud, which is used to forward the tunnel from the cloud to the edge node.
Tunnel-cloud
The tunnel-cloud contains three modules of stream, TCP and HTTPS. The stream module includes gRPC server and dns components. The gRPC server is used to receive gRPC long-lived requests from tunnel-edge, and the dns component is used to update the node name and IP mapping in the tunnel-cloud memory to the coredns hosts plug-in configmap.
Tunnel-cloud Configuration
tunnel-cloud-conf.yaml
apiVersion:v1kind:ConfigMapmetadata:name:tunnel-cloud-confnamespace:edge-systemdata:tunnel_cloud.toml:| [mode]
[mode.cloud]
[mode.cloud.stream] # stream module
[mode.cloud.stream.server] # gRPC server component
grpcport = 9000 # listening port of gRPC server
logport = 8000 # The listening port of the http server for log and health check,use (curl -X PUT http://podip:logport/debug/flags/v -d "8") to set the log level
channelzaddr = "0.0.0.0:6000" # The listening address of the gRPC [channlez](https://grpc.io/blog/a-short-introduction-to-channelz/)server, used to obtain the debugging information of gRPC
key = "../../conf/certs/cloud.key" # The server-side private key of gRPC server
cert = "../../conf/certs/cloud.crt" # Server-side certificate of gRPC server
tokenfile = "../../conf/token" # The token list file (nodename: random string) is used to verify the token sent by the edge node tunneledge. If the verification fails according to the node name, the token corresponding to the default will be used to verify
[mode.cloud.stream.dns] # DNS component
configmap= "proxy-nodes" # configmap of the configuration file of the coredns hosts plugin
hosts = "/etc/superedge/proxy/nodes/hosts" # The path of the configmap of the configuration file of the coredns hosts plugin in the mount file of the tunnel-cloud pod
service = "proxy-cloud-public" # tunnel-cloud service name
debug = true # DNS component switch, debug=true dns component is closed, the node name mapping in the tunnel-cloud memory will not be saved to the configmap of the coredns hosts plug-in configuration file, the default value is false
[mode.cloud.tcp] # TCP module
"0.0.0.0:6443" = "127.0.0.1:6443" # The parameter format is "0.0.0.0:cloudPort": "EdgeServerIp:EdgeServerPort", cloudPort is the server listening port ofthe tunnel-cloud TCP module, EdgeServerIp and EdgeServerPort are the IP and port of the edge node server forwarded by the proxy
[mode.cloud.https] # HTTPS module
cert ="../../conf/certs/kubelet.crt" # HTTPS module server certificate
key = "../../conf/certs/kubelet.key" # HTTPS module server private key
[mode.cloud.https.addr] # The parameter format is "httpsServerPort": "EdgeHttpsServerIp: EdgeHttpsServerPort", httpsServerPort is the listening port of the HTTPS module server, EdgeHttpsServerIp:EdgeHttpsServerPort is the proxy forwarding the IP and port of the edge node HTTPS server, The server of the HTTPS module skips verifying the client certificate, so you can use (curl -k https://podip:httpsServerPort) to access the port monitored by the HTTPS module. The data type of the addr parameter is map, which can support monitoring multiple ports.
"10250" = "101.206.162.213:10250"Tunnel-edge
The tunnel-edge also contains three modules of stream, TCP and HTTPS. The stream module includes the gRPC client component, which is used to send gRPC long-lived requests to the tunnel-cloud.
Tunnel-edge Configuration
tunnel-edge-conf.yaml
apiVersion:v1kind:ConfigMapmetadata:name:tunnel-edge-confnamespace:edge-systemdata:tunnel_edge.toml:| [mode]
[mode.edge]
[mode.edge.stream] # stream module
[mode.edge.stream.client] # gRPC client component
token = "6ff2a1ea0f1611eb9896362096106d9d" # Authentication token for access to tunnel-cloud
cert = "../../conf/certs/ca.crt" # The ca certificate of the server-side certificate of the gRPC server of tunnel-cloud is used to verify the server-side certificate
dns = "localhost" # The IP or domain name signed by the gRPC server certificate of tunnel-cloud
servername = "localhost:9000" # The IP and port of gRPC server of tunnel-cloud
logport = 7000 # The listening port of the http server for log and health check,use (curl -X PUT http://podip:logport/debug/flags/v -d "8") to set the log level
channelzaddr = "0.0.0.0:5000" # The listening address of the gRPC channlez server, used to obtain the debugging information of gRPC
[mode.edge.https]
cert= "../../conf/certs/kubelet-client.crt" # The client certificate of the HTTPS server forwarded by tunnel-cloud proxy
key= "../../conf/certs/kubelet-client.key" # The private key of the client side of the HTTPS server forwarded by the tunnel-cloud proxyTunnel Forwarding Mode
Tunnel proxy supports either TCP or HTTPS request forwarding.
Local Debugging
Tunnel supports HTTPS (HTTPS module) and TCP protocol (TCP module). The data of the protocol module is transmitted through gRPC long-lived (stream module ), so it can be divided into modules for local debugging. Local debugging can use go’s testing framework. The configuration file can be generated by calling config_test test method Test_Config (where the constant variable config_path is the path of the generated configuration file relative to the path of the config_test go file, and main_path is the configuration file relative to the test file Path), such as the local debugging of the stream module: config_path = “../../../conf” (The generated configuration file is in the conf folder under the root directory of the project), then main_path="../../../../conf"(the path of stream_test relative to conf), the configuration file is generated to support the configuration of ca.crt and ca.key (when configpath/certs/ca.crt and configpath/certs/ca.key exist, the specified ca is used to issue the certificate).
stream module debugging
start of the stream server
func Test_StreamServer(t *testing.T) {
err := conf.InitConf(util.CLOUD, "../../../../conf/cloud_mode.toml")
if err != nil {
t.Errorf("failed to initialize stream server configuration file err = %v", err)
return
}
model.InitModules(util.CLOUD)
InitStream(util.CLOUD)
model.LoadModules(util.CLOUD)
context.GetContext().RegisterHandler(util.MODULE_DEBUG, util.STREAM, StreamDebugHandler)
model.ShutDown()
}
Load configuration file(conf.InitConf)->Initialize the module(model.InitMoudule)->Initialize the stream module(InitStream)->Load the initialized module->Register a custom handler (StreamDebugHandler)->Shut down the module (model.ShutDown)
StreamDebugHandler is a custom handler for debugging cloud side messaging
start of the stream client
func Test_StreamClient(t *testing.T) {
os.Setenv(util.NODE_NAME_ENV, "node1")
err := conf.InitConf(util.EDGE, "../../../../conf/edge_mode.toml")
if err != nil {
t.Errorf("failed to initialize stream client configuration file err = %v", err)
return
}
model.InitModules(util.EDGE)
InitStream(util.EDGE)
model.LoadModules(util.EDGE)
context.GetContext().RegisterHandler(util.MODULE_DEBUG, util.STREAM, StreamDebugHandler)
go func() {
running := true
for running {
node := context.GetContext().GetNode(os.Getenv(util.NODE_NAME_ENV))
if node != nil {
node.Send2Node(&proto.StreamMsg{
Node: os.Getenv(util.NODE_NAME_ENV),
Category: util.STREAM,
Type: util.MODULE_DEBUG,
Topic: uuid.NewV4().String(),
Data: []byte{'c'},
})
}
time.Sleep(10 * time.Second)
}
}()
model.ShutDown()
}
Set the node name environment variable->Load configuration file (conf.InitConf)->Initialization module (model.InitMoudule)->Initialize the stream module (InitStream)->Load the initialized module->Register a custom handler (StreamDebugHandler)->Shut down the module (model.ShutDown)
The node name is loaded through the environment variable of NODE_NAME
TCP module debugging
TCP server debugging
func Test_TcpServer(t *testing.T) {
err := conf.InitConf(util.CLOUD, "../../../../conf/cloud_mode.toml")
if err != nil {
t.Errorf("failed to initialize stream server configuration file err = %v", err)
return
}
model.InitModules(util.CLOUD)
InitTcp()
stream.InitStream(util.CLOUD)
model.LoadModules(util.CLOUD)
model.ShutDown()
}
Need to initialize the TCP module (InitTcp) and the stream module (stream.InitStream) at the same time
TCP client debugging
func Test_TcpClient(t *testing.T) {
os.Setenv(util.NODE_NAME_ENV, "node1")
err := conf.InitConf(util.EDGE, "../../../../conf/edge_mode.toml")
if err != nil {
t.Errorf("failed to initialize stream client configuration file err = %v", err)
return
}
model.InitModules(util.EDGE)
InitTcp()
stream.InitStream(util.EDGE)
model.LoadModules(util.EDGE)
model.ShutDown()
}
HTTPS module debugging
Similar to TCP module debugging, HTTPS module and stream module need to be loaded at the same time.
Debugging of tunnel main() function
In the main test file of the tunnel tunnel_test, you need to use init() set the parameters, at the same time you need to use TestMain to parse the parameters and call the test method
Feedback
Was this page helpful?
Glad to hear from you! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.
