Yesterday I made the decision (read: mistake) to update my Netgear DG834G router (hardware v4, firmware v5.01.09) to firmware v5.01.14 – and, as is the way with these things, it brought trouble.  After the upgrade I couldn’t reach www.nikrivers.com from the LAN side of the router.

The problem is caused by the way the router handles traffic coming from an internal IP address and destined for the WAN (i.e. external) IP address.  In this situation it requires that the router first transfers the traffic from the internal network to the external network, and then immediately passes it back whilst applying any firewall or routing rules that are relevant to incoming external traffic.

This behaviour is called ‘NAT loopback’, and it seems the vast majority of routers built for the home market have this ability turned off, or do not have the ability at all.  It can be a big problem if you host a website and wish to access that same website using its domain name.  The domain name will resolve to the WAN IP address of your router, and any traffic headed there (such as an HTTP GET request on port 80) from the internal network will be ignored by the router.

There are a few ways to solve this, but none of them are ideal.

1. Use the server name instead of the domain name to access your website
2. Modify the list of known network hosts on each client to point your domain name straight to the server in question.
3. Run your own DNS server, using a view to return the server’s local IP address to requests for your domain name originating from your network.

Of course, the situation gets more complex if you’re also using your router to send TCP traffic on port 80 to your webserver and UDP traffic on port 8668 to a game server.

The solution is to get NAT loopback working on your router.  With some routers, such as the Touchspeed 535 as provided by Be Broadband, this feature can be enabled using a simple CLI command.  For other routers, such as the Netgear DG834G, it’s not quite so easy.

For the purposes of this post I’ll assume the internal network is on the 192.168.0.x range, the router is 192.168.0.254, and the web server is 192.168.0.1.  You will need to modify these IP addresses according to your own network setup.

The first thing to do is to enable debug mode on the router.  Simply go to http://192.168.0.254/setup.cgi?todo=debug and you’ll be rewarded with an appropriate message, “Debug Enable!”.  Nice.

Now connect to the router with ‘telnet 192.168.0.254′ to gain access to the router’s cut-down installation of Linux.  All that is required is to add one additional entry to the router’s iptables (which is a standard Linux feature; Google it or more info).  Type the following, amending any IP addresses according to the network setup:

iptables -t nat -A POSTROUTING -d 192.168.0.1 -s 192.168.0.0/24 -p tcp –dport 80 -j SNAT –to 192.168.0.254

This adds a rule to the POSTROUTING chain on the nat table which applies to all TCP traffic on port 80 (HTTP) coming from the private network and headed to the router.  The rule redirects the traffic to the server, and then processing jumps to the SNAT chain.

If the server is more than just a simple web server, such as an NTP server or mail server as well, the above step needs to be performed (changing the -p and –dport parameters accordingly) for each port and protocol combination you require.  Alternatively, those parameters could be omitted altogether, which will allow all traffic types on all ports through:

iptables -t nat -A POSTROUTING -d 192.168.0.1 -s 192.168.0.0/24 -j SNAT –to 192.168.0.254

If you do this, I recommend you run a firewall on your server, with only the appropriate ports opened.

There is more information in section 10 of Rusty Russell’s Linux 2.4 NAT Howto.

Unfortunately, the iptables change isn’t retained when the router restarts, so it is necessary to go through the process every time – which is a pain in the backside.  Fortunately, however, the Netgear support website has a download link for previous firmware versions, so I downgraded my router back to firmware v5.01.09 and everything worked fine again–including NAT loopback–with no iptables hack required.

A robust solution with DNS

Simply put, proper DNS is the best way to get around a router’s lack of/poorly implemented NAT loopback.

If you have the resources to host a website then you most likely also have the resources to host a DNS server for your internal network.  Simply create an ACL list describing all the clients on your internal network (probably as simple as specifying the CIDR block for your network, maybe something like 192.168.1.0/24).  Then create a view whose clients match that ACL, and define that view as a master DNS server for your website domain.  You then need to create a zone file for that domain – but instead of using an external IP for your webserver, use its internal IP.

All requests for other domains will be routed to the DNS forwarders, but requests for your webserver’s domain will be handled locally, and internal IP addresses will be returned.

The benefit is that you avoid traversing your gateway router to simply come back inside your network.  It doesn’t make sense that you rely on your gateway router to access a website within your own network.  In addition, the firewall on your router can be hardened to a much greater degree: for example, you needn’t leave FTP ports open on your router if you’re only connecting locally.  Or, to put it another way, you’re likely going to want to give yourself more access to your server than you want to give to the outside world; configuring a router’s firewall rules for this kind of conditional logic is simply asking for trouble.

When it comes out of the factory the Slug has an inherrent bug which causes the internal clock to lose time; apparently Linksys tried to fix this problem by adjusting the time regularly (using cron) but it seems the time is adjusted in the wrong direction – thus, the error is doubled. This problem is fixed in Unslung 2.6 and higher anyway (Unslung 2.10 is the latest release as at Jan 2009), so we can remove the relevant crontab entry.

Edit the crontab file:

vi /etc/crontab

Comment out the call to hwclock (the Linksys ‘fix’). The hash (#) symbol is used to denote a comment:

#1 * * * * root /usr/sbin/hwclock -s &>/dev/null

Now install ntpd, and edit the config file to contain time servers that are geographically close to you – you can find out what these are by visiting http://www.pool.ntp.org/. Editing the config file is quite self-explanatory.

install ntp
vi /opt/etc/ntp/ntp.conf

Now edit the ntp startup script; there are a few things we need to do to make sure the time is set and maintained correctly.

vi /opt/etc/init.d/S77ntp

Add the following lines just before the existing call to ntp:

/opt/bin/tickadj 10000 > /dev/null
/opt/bin/ntpd -q -c /opt/etc/ntp/ntp.conf > /dev/null

The first line corrects the Slug’s tick value, and the second line makes ntp quickly (-q) set the time using the specified time servers, and exits. It is important to note that when ntp is running, it does not just simply set the correct time regularly; it constantly checks the time against that of the time servers, and gradually brings the system time in line. The call we just added, with the -q flag, ensures that the Slug has pretty accurate time before ntp starts properly.

Speaking of which, you will need to make sure the Slug has accurate time now. This can be done using the web interface, for example. It only needs to be as accurate as you can get it, and ntp will take care of the rest.

Now simply restart ntp:

/opt/etc/init.d/S77ntp restart

It will take a while for ntp to synchronise with the time servers you specified in the config file, sometimes upwards of 15 minutes – so don’t be too concerned if nothing seems to be happening straight away. You can check what’s going on with the ntpq tool:

ntpq -p

The command will list information about the time servers. If ntp has decided on a server’s suitability for synchronisation, the time server’s name will be preceeded by * (the chosen server), + (server is suitable), or - (server is unsuitable).

Note: if you are running Dovecot on your Slug, and ntp adjusts the time backwards after you initially set the Slug’s time (using the web interface), Dovecot will complain about time going backwards, and die. You will need to restart Dovecot; but this only happens while you’re configuring ntp, and won’t happen during normal operation once ntp is running.

If this information was useful, please leave a comment to let me know!