The OpenShift Router is the entry point for almost all north-south traffic, making it crucial to understand its operational mechanism, particularly the configuration update and reload mechanism.In case of service request anomalies, we can quickly analyze the problem’s cause, fix it in a timely manner, and ensure application continuity.This chapter mainly introduces the configuration loading mechanism of the OpenShift Router.
The OpenShift Router is primarily based on Haproxy.When there are updates to Pods or certificate updates, the Haproxy configuration will be reloaded to ensure that the cluster’s routing information is up to date.Whether reloading the configuration will impact the currently online services is a concern for system administrators.
1. The Process of Haproxy Configuration Reload
The Haproxy configuration reload process consists of two steps.
1. Generate the latest configuration
2. Restart the Haproxy process
Generating the Latest Haproxy Configuration
The following three resource changes in OpenShift will trigger an update to the Haproxy configuration:
1. Changes to Routes
2. Changes to Pod IP/Endpoints
3. Changes to Certificates
$ oc exec router-3-r2scd cat haproxy-config.template | head -n5{{/* haproxy-config.cfg: contains the main config with helper backends that are used to terminate encryption before finally sending to a host_be which is the backend that is the final backend for a route and contains all the endpoints for the serviceThere are two processes running in the OpenShift Router Pod.
$ oc exec router-3-r2scd -n default -- ps -efUID PID PPID C STIME TTY TIME CMD1000000+ 1 0 0 Nov07 ? 00:00:08 /usr/bin/openshift-router1000000+ 5076 1 0 Nov07 ? 00:00:02 /usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/run/haproxy.pid -x /var/lib/haproxy/run/haproxy.sock -sf 5061...The parent process of the Haproxy process is openshift-router, which manages the haproxy process.The openshift-router will query the status of Routes, EndPoints, and certificates through the Master API, generate the latest Haproxy configuration, and execute the reload operation.After each reload, the haproxy process will be terminated and restarted, resulting in a change in the process ID of haproxy.
$ oc exec router-3-r2scd -n default -- ps -efUID PID PPID C STIME TTY TIME CMD1000000+ 1 0 0 Nov07 ? 00:00:08 /usr/bin/openshift-router1000000+ 5208 1 0 Nov07 ? 00:00:02 /usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/run/haproxy.pid -x /var/lib/haproxy/run/haproxy.sock -sf 5148 5193...2. Environment Variables for openshift-router
The openshift-router accepts many environment variables to control the haproxy configuration and the haproxy reload process.
RELOAD_SCRIPT
This is the script for reloading haproxy, with the default address being /var/lib/haproxy/reload-haproxy.Generally, it is not changed.The openshift-router process will run this script at regular intervals, which is set through the environment variable RELOAD_INTERVAL, with a default of 5 seconds.
RELOAD_INTERVAL
This sets the time interval for the openshift-router process to execute the RELOAD_SCRIPT, defaulting to 5 seconds.Increasing this value can reduce the frequency of haproxy reloads by the openshift-router.
3. Speeding Up Haproxy Reload
For Haproxy, it is crucial to complete the configuration loading quickly.This way, clients will not hit their SYN retry limits, which could lead to connection failures.
OpenShift Router Optimization Methods
1. Upgrade to version 3.9 or higher to support seamless reloads.
2. Use route sharding to reduce the policies for each route, thereby speeding up loading.
3. Increase the RELOAD_INTERVAL environment variable value to reduce periodic reload calls.
Checking Reload Speed
By checking the reload speed of HAProxy, we can confirm whether connection issues were caused by the reload.Prometheus will store the HAProxy loading time information under the monitoring item template_router_reload_seconds.
sh-4.2$ curl -s http://admin:aAIKAyrX1s@localhost:1936/metrics | grep template_router_reload_seconds# HELP template_router_reload_seconds Measures the time spent reloading the router in seconds.# TYPE template_router_reload_seconds summarytemplate_router_reload_seconds{quantile="0.5"} NaNtemplate_router_reload_seconds{quantile="0.9"} NaNtemplate_router_reload_seconds{quantile="0.99"} NaNtemplate_router_reload_seconds_sum 7.299802469000001template_router_reload_seconds_count 140We see that the total reload count is 140, and the total reload time is about 7.3 seconds.The average loading time is 7.3 / 140 = 0.05 seconds, which is quite fast!
Automatically Closing Old Haproxy Processes After Router Reload
In OpenShift versions 3.9 and above, the Router will retain the old Haproxy process for a while after reloading Haproxy.This is to wait for ongoing requests to close, so the old process will hang for a while.These requests will close in one of two ways:
1. Client closure
2. Connection timeout
The default connection timeout in OpenShift is 1 hour, and you can reduce the hang time of the Haproxy process by lowering the value of the ROUTER_DEFAULT_TUNNEL_TIMEOUT environment variable, while also adjusting the subsequent health check time by lowering the ROUTER_BACKEND_CHECK_INTERVAL environment variable.
If you have any questions, you can click on the original text at the end of the article to comment and communicate in the community. <br/> If you find this article useful, please share it or click "View More" to let more peers see it. <br/> Recommended materials/articles: <br/> OpenShift Production Environment Deployment Specifications <br/> How to Achieve OpenShift-F5 Integration (North-South Traffic via F5)? <br/> Open Source Container Cloud OpenShift Builds an Enterprise Application Cloud Platform Based on Kubernetes. part1 <br/> http://www.talkwithtrend.com/Document/detail/tid/418219 <br/> Open Source Container Cloud OpenShift Builds an Enterprise Application Cloud Platform Based on Kubernetes. part2 <br/> http://www.talkwithtrend.com/Document/detail/tid/418221 <br/> Click to read the original text to focus on the community "Container Cloud" technology topics, which will continue to update quality materials and articles. You can also raise difficult questions there to discuss and exchange with peers: http://www.talkwithtrend.com/Topic/98447 <br/> Download the twt community client APP to connect with more peers and get answers to your questions anytime. <br/> Easily subscribe to technology topics in various fields and browse the latest articles and materials. <br/> Long press to recognize the QR code to download or search for "twt" in the app store. <br/> Long press the QR code to follow the public account. <br/> * The content published by this public account only represents the author's views and does not represent the community's position.