Over the years I’ve been trying to run a custom-made layer-4 transparent-proxy service for most of my entire network to use automatically (zero client configuration). It first started because I did not like the general idea of a VPN server using a sub-1500 MTU setting while all of the clients on a network auto-assume a 1500 MTU themselves. In addition, instead of reading 1500 bytes at a time off of a TUN interface, you can instead read 8192 bytes off of a TCP socket at a time which you can then feed to a fast stream cipher without the need for packet fragmentation. However, it took me quite a while to reach some stability with tracking all of the connection state types and to iron out all the issues that could arise from transparently proxying both UDP and TCP connections. Some of the lessons I learnt that might help others trying a similar approach include the following:

• Make sure to increase the number of file descriptors to handle all of the sockets and pipes per each process/thread running
  • Ex: ulimit -n 65536
• Make sure to check for any remapped duplicated source port entries in the connection state tracking table based on dport after checking sport first
  • Ex: conn=$(echo "${outp}" | grep -i " src=${addr} .* dport=${port} " | grep -i "${prot}")
• Make sure to DNAT load balance UDP client traffic based on source IP+PORT ranges instead of the connection state or statistic mode modules
  • Ex: multiport sports 0:53133 to:192.168.1.1:3135
  • Ex: multiport sports 53134:57267 to:192.168.1.2:3135
• Make sure to pay attention to the finer details of properly managing connection states and process/thread states throughout the entire code base
  • Ex: Create a separate thread that is dedicated to managing the file descriptors and processing states

I will try to post more tips as time goes on and I learn more but these small issues can cause a lot of headaches when you’re trying to translate and redirect thousands of network wide connections down into separated processes for load balancing purposes! 🙂

~