Unlocking Client IP Visibility with Nginx Ingress in Kubernetes

Our organization has around 90% of our applications containerized and deployed in a Kubernetes environment, both on-premise and in the cloud. We primarily use the Nginx Ingress Controller to proxy requests to various backend services. We chose Nginx as our reverse proxy due to its high performance, straightforward configuration, detailed documentation, and robust community support. Most importantly, it suits our needs perfectly in all aspects, so we have no reason to consider other reverse proxies at this time.

Initially, we didn't focus on logging or tracking client IPs. However, circumstances have changed recently, and we now need to log the client IP in Nginx access logs and pass it to backend applications for validation, anomaly detection, and audit purposes. We initially thought this would be straightforward, but finding a solution that works effectively in a Kubernetes environment has proven to be more challenging than expected.

I will now share the different methods to capture the client IP in the Nginx Ingress Controller for Kubernetes environments and explain why this functionality does not work by default in your cluster without additional configuration.

Understanding Client IP Preservation in Kubernetes

Typically, Nginx sets the client IP in either the X-Real-IP header or as the first element in the X-Forwarded-For header, depending on the configuration and upstream headers received.

Headers Used for Client IP

• X-Real-IP Header: This header is commonly used by Nginx to store the client's real IP address. It is straightforward and directly reflects the client's IP address without additional proxy information.

• X-Forwarded-For Header: In cases where requests pass through multiple proxies or load balancers, the X-Forwarded-For header lists all IP addresses that the request has traversed. The client's IP address is typically the first element in this header, separated by commas. For example, X-Forwarded-For: client_ip, proxy1_ip, proxy2_ip.

Why Kubernetes NodePort/LoadBalancer Services Don't Preserve Client IP by Default

When deploying an application on Kubernetes using a LoadBalancer service, the client's original IP address is often lost due to how traffic routing works. Kube-proxy, which handles the routing, assigns a random port on each node and redirects client traffic to this port using low-level routing logic. It then selects a backend and proxies the traffic, replacing the original client IP with a cluster internal IP. This happens because kube-proxy randomly selects a backend node to forward the traffic to, causing the client IP to be lost in the process. The thing is the IP gets lost in the *X-Forwarded-For header as well.

Solutions

Note: The solution below applies to all reverse proxies except Proxy Protocol V2, as it depends on specific features of the reverse proxy.

Modifying External Traffic Policy to Preserve Client IP

The External Traffic Policy parameter determines whether a service should route external traffic to node-local or cluster-wide endpoints:

1. Local: Preserves the client source IP and may cause a second hop to another node, but ensures good overall load-spreading. However, if there are no local endpoints, packets will be dropped.
2. Cluster: Does not preserve the client source IP but avoids a second hop for LoadBalancer and NodePort type Services, potentially resulting in imbalanced traffic spreading.

This parameter should be changed under spec.externalTrafficPolicy in the service object.

Based on this explanation, the Local setting is suitable for single-node clusters but may not be efficient or recommended for multi-node clusters.

Using Proxy Protocol V2

Proxy Protocol v2 is designed to safely transport connection information such as the original client IP address, port number, and protocol (TCP or UDP) over a proxied connection. It improves upon Proxy Protocol v1 with a more compact and efficient binary format, making it better suited for high-performance and low-latency environments. Since the proxy protocol is used to communicate in binary format, it's not feasible to send normal HTTP/HTTPS/TCP/UDP/TLS requests directly to the NGINX ingress service using tools like Postman or curl. Requests must be sent to the Load Balancer (LB) endpoint, as the LB accepts HTTP/HTTPS/TCP/UDP/TLS requests and communicates with the NGINX ingress via proxy protocol v2 (binary format).

Requirements to Implement Proxy Protocol V2

1. Reverse Proxy Support:

   • The reverse proxy must support Proxy Protocol v2. Nginx Ingress, for instance, has this support.

2. Load Balancer Support:

   • The load balancer must also support Proxy Protocol v2.

To enable Proxy Protocol in Nginx Ingress via Helm chart set the value controller.config.use-proxy-protocol to true and perform helm install or upgrade.

Since the load balancer should also support Proxy Protocol, most cloud providers have official documentation for enabling this functionality. example for AWS NLB service.beta.kubernetes.io/aws-load-balancer-proxy-protocol would be the the annotation to enable the protocol and value should be *

Using HostNetwork

If your load balancer doesn't support Proxy Protocol v2, you can use the HostNetwork solution. This approach simplifies the configuration and avoids the need for the previously mentioned Helm values. To implement this, set the controller.hostNetwork to true and service.type to ClusterIP and perform helm install or upgrade.

With this configuration, all your pods will bind to each node's interface on port 80 for HTTP and port 443 for HTTPS by default. For OpenShift (OC) users, you need to add the SCC (Security Context Constraints) to the Nginx Ingress service account.

If ports 443 and 80 are already in use by another application, you can change the ports for the Nginx Ingress pods under controller.ports.

Ensure that your load balancer listens on port 443 for HTTPS and port 80 for HTTP traffic, and forwards this traffic to the Kubernetes worker nodes' ingress ports. In this case, the ingress ports would be either 443 or 8443 for HTTPS and 80 or 8080 for HTTP.

Using CDN

Another approach is to leverage Content Delivery Networks (CDNs) such as Cloudfront, Cloudflare, or Fastly, which offer features to pass client IP information to the backend via custom headers. For example, Cloudfront and Cloudflare use True-Client-IP, while Fastly uses Fastly-Client-IP to store the client's IP address. Each CDN provider has its own method of transmitting the client IP in a header that can be utilized within your application or substituted for the X-Real-IP header using the proxy_set_header module in Nginx under the location, server, or http directive:

proxy_set_header X-Real-IP $http_cdn_client_header_name; # Change the header here according to your CDN Client IP header name

This configuration ensures that Nginx forwards requests to the backend server with the client's real IP address provided by the CDN's custom header.

Conclusion

Preserving the client IP in Kubernetes environments can be challenging but is achievable with the right strategies. By understanding the role of reverse proxies and the intricacies of traffic routing within Kubernetes, you can implement effective solutions tailored to your needs.

1. Modifying externalTrafficPolicy to Local helps preserve client IPs but is best suited for single-node clusters and requires careful consideration in multi-node environments.

2. Using Proxy Protocol v2 ensures that connection information like the client IP is preserved across proxies.

3. Leveraging HostNetwork offers a straightforward approach to bind pods directly to node interfaces, though it requires managing network ports and security contexts, especially in environments like OpenShift.

4. Using CDN allows you to utilize features from providers like Cloudfront, Cloudflare, or Fastly etc to pass client IP information via custom headers such as True-Client-IP or Fastly-Client-IP etc. This method simplifies IP preservation by ensuring Nginx can directly use these headers to replace or supplement X-Real-IP, facilitating accurate client IP forwarding to your application.

Each of these strategies has its own benefits and considerations. By carefully evaluating your specific requirements and environment constraints, you can choose the best approach to maintain client IP visibility for your applications.

Thanks for reading! Feel free to share your thoughts and reach out if you have any questions. Until next time! ????