Detecting the Proxies

There’s no doubt that your online experience can be extremely limited from certain countries. More and more countries are seeking to take control over things like access to social media, political and religious websites and worryingly even many of the independent news sources available online. The practice of filtering and censorship grows everyday and there’s little sign of this trend reversing.

Most of the time, censorship of the internet is justified by a fight on crime, child abuse or other illegal practice. Generally those aims are flexible and can be extended to suit whatever the State decides, often the associated legislation is particularly vague to allow whatever interpretation is needed. Most countries who heavily police the internet have ‘catch all’ phrases which can cover pretty much anything they decide at the time. All over the world innocent journalists, bloggers and web users have been imprisoned for little more than expressing the ‘wrong opinion’ online, sometimes all it takes is a ‘like’ on a Facebook post to land people in trouble.

For people unfortunate enough to live in such places, using the internet safely involves protecting both their internet connection and maintaining anonymity. There are simple things like using pseudonyms and never putting any real information in social media profiles for instance. Yet ultimately it is the technical details which are most important, hiding your real ip address is vital.

This is because it doesn’t matter what information you leave online, your IP address can be linked to your physical location. Obviously in a public access point like a cafe or library this isn’t as important but if you’re using a home internet connection you have to be even more careful. This is why proxies and VPN are so important as they sit between you and the website you’re visiting in order to hide your location. Instead of your own IP being left, instead it’s that of the VPN/proxy that you are using.

Here’s an example of such a service being used, not for security but instead to fool the region locking of a big media site – it’s called a proxy for Netflix you can see it here.

In this scenario, Netflix is not able to see the true location of the viewer only that of the proxy server which allows full access. However using proxies to hide your location is fraught with dangers simply because they are relatively insecure. Firstly they do nothing to hide or secure any information you transmit and secondly proxies are by default set up to transmit a X-Forwarded-For HTTP header when they contact any server. This can be used to both detect the presence of the proxy and worse the originators IP address.

Any decent anonymous proxy server would be configured to not send this header but remember it is default behaviour and any upgrade or misconfiguration could easily override these changes. The danger is that as soon as any misconfiguration happens, the proxy will be identified and picked up by services such as IP2Location which maintain extensive databases of proxies, VPNs and TOR nodes.

In the case of media sites this simply means that attempts to bypass the region locks won’t work but for a political activist in somewhere like Thailand, China or Turkey then the repercussions can be much more serious.
Additional Reading

Using a VPN/Proxy on Different Devices

For the internet free spirit who wants full access to the web without restrictions, filters or logs then using a proxy or VPN is pretty much essential. I say that, although the use of proxies is pretty much in terminal decline at least as far as bypassing blocks such as the region restrictions applied by big media sites. A proxy or VPN server operates in a very similar way fundamentally by acting as a buffer to forward and receive requests to hide your location. This means that whatever web sites you visit they will only see the proxy/VPN and never your real location additionally your ISP will only log the visit to the proxy not the end destination.

However a VPN server crucially adds an additional layer – that of encryption which protects the data being transmitted and is almost impossible to detect. The problem is that nowadays even the most cleverly configured proxy server is fairly easy for web sites to detect, if the site enforces region locking it will normally just block access from a proxy. A VPN though is much more difficult to detect and most sites cannot directly detect the use of a VPN although they can use other methods. So if you want to hide your identity from a snooping government, or want to watch Hulu from outside the USA, you’re going to need a VPN service not a proxy. Ignore the hundreds of free proxies available online, most are completely useless now and are often used to steal user credentials and passwords!

A VPN (virtual private network) service is somewhat more sophisticated than a proxy so will need an additional service to make it run. A proxy can be used simply by configuring it’s settings in most standard web browsers but that’s because it merely transparently forwards and receives data without protecting it in any way. A VPN actually sets up an encrypted tunnel between the client and the server which obviously needs some software component to run. Most VPN providers will provide some client connection software to establish the connection on their PC, you can see an example in this video:

If you watch the whole video it shows you the difference between a proxy and a VPN, demonstrating the client software of a program called Identity Cloaker in action.

How Does a VPN Work on other Devices?

This is what often confuses people, after all how does your smart phone or tablet use a VPN after all it’s the platform that many of us use to watch TV or movies sites. You can of course still configure a proxy on these devices but as mentioned these are largely useless now and should be avoided unless you have a specific need (and technical knowledge). For most of these devices there are only two options – use an application specifically designed for the device to enable the VPN connection or use the device’s own operating system to establish the connection.

It’s generally better to use the second option, that is to manually connect to your VPn service using the devices own operating system. This for example is how you connect a VPN using an iPad in this post. All you need as you can see is the login and user authentication details, input them into the VPN configuration screen and save the connection. You can then enable the VPN connection whenever you wish.

It’s worth remembering what actually happens when you establish a VPN connection either through some client software or as illustrated on a computer tablet manually. Firstly an encrypted tunnel is established, making sure all your data is hidden from view, secondly your IP address is effectively hidden all web sites will only see the address of the VPN server. This is what allows people to bypass the myriad of region blocks on large media websites – the ability to use the IP address of the VPN server rather than your own. This is why most VPN services offer a range of servers in different countries so that you can use a UK one for the BBC iPlayer, a USA one for Netflix and Hulu and so on.

Escaping Region Locking from Media Sites

In the early years of the internet there were very few restrictions on what you could see and download. If you started a web browser in the US you’d get pretty much the same experience as someone who started in Cairo. Obviously there might be some variation in speed of course, but what you could see and do was almost identical.

That’s changed a lot now with the growing popularity of region locking and control. It started off fairly helpfully – your search engines would switch you to the appropriate location based on your IP address. This meant that if you were searching from London for electricians you wouldn’t get directed to results in Sydney which would obviously be useful. We’ve got used to this and it generally makes everyone’s life much easier.

However the use of region locking has extended greatly in the last few years, in fact any major web site will usually operate some level of control. Often it’s again beneficial, Amazon will make sure you go to the UK site, Costco will direct you to your local store and so on. However for many of the world’s biggest media sites it’s a much different story – region locking usually means region blocking.

Ever tried to access Pandora from outside the US? Well it doesn’t work, the wonderful music site is only accessible for those located in the US. Want to watch the BBC News, sorry if you’re outside the UK it’s not going to happen. Those are just two but the list is extensive, in fact it’s unlikely you’ll now find a large media site which doesn’t lock down access based on the location of your IP address.

It’s crazy when you think about it, a global communication medium deliberately trying to segregate and restrict our world. Worse too that in a time when many of us travel extensively, we are blocked and filtered at every turn when we’re online.

So What’s the Solution?
Well to take back control and stop being blocked you need to be able to control your IP address. Unfortunately for most of us that’s not possible, the IP address is assigned when you connect to the internet and there is no way of modifying it. You can of course modify your local address but that’s not important, region locking uses your external internet facing IP address.

However although you cannot modify your address, you can hide it by using VPN servers to protect your connection. If you connect to a UK VPN server for example, it will look as though your have a UK IP address and watching the BBC works without problems. You can use a US VPN to gain a US IP address for Netflix irrespective of where you actually are. Many firms have developed services to support this demand and the top VPN providers will allow access to a network of servers in different countries.

This means that although you cannot change your real IP address, you can hide it behind a VPN server. It gives you back control and neatly sidesteps the pervasive region locking and filtering which seems likely to keep expanding.

Further Reading – British VPN

Methods of Designating IP Addresses

There are a variety of methods that can be used to assign an IP address to a windows client. You can obviously assign directly by allocating a static address or by using a variety of other methods such as ARP, BOOTP or DHCP. The main methods will be discussed briefly below:

Static Configuration
The majority of network enabled devices including Windows computers can be assigned an IP address manually. This is normally allocated in the TCP/IP properties on windows machines and in the network configuration options on other devices like games consoles and media streamer for instance. It is important that although any address can be allocated here, it should be the correct address for the network configuration. You should also ensure that the IP address is unique on that network,otherwise network connectivity will be affected for both clients with duplicate addresses.

RARP and BOOTP
Two of the main options for assigning an ip address dynamically are Reverse ARP (RARP) and the Bootstrap protocol (BootP). RARP requires a server which maintains a list of hardware addresses and a pool of IP addresses to be allocated. The server would normally be contacted via a broadcast initiated from the client before an ip address is allocated and then assigned to the hardware id. There are often issues with this method of address allocation though:

  • Clients broadcasts will sometimes not reach the RARP server. This might be for various reasons but is often to do with network topology, perhaps a router is incorrectly configured on the network. This can be resolved by configuring an IP helper address on any routers. It may also be simpler to configure the router to be a RARP servers depending on the model.
  • RARP server does not have an IP address which corresponds to the client’s hardware address.

Mostly due to various inefficiencies RARP is rarely used in modern networks and is usually replaced by the more sophisticated BOOTP to assign IP addresses.  This runs over UDP and sets up a port for client requests and another port is assigned for server responses.   The response from the BOOTP server actually contains additional information such as the address of the local gateway.   BootP suffers similar problems to RARP mainly to do with network connectivity.  UDP often suffers connectivity problems and the routers should be configured to allow UDP traffic and without interruption,  other problems can be with access lists filtering out the UDP ports.

Further Reading

DNS Lookups on Web Proxies

Proxy servers will commonly be required to perform two kinds of DNS lookups those to resolve IP addresses from the hostnames and reverse lookups to find the hostname given the IP address. The DNS lookups will normally require contacting the DNS service and therefore there will be an impact on speed and some latency. It is therefore important to optimize these lookups in order to minimize the impact on the proxy performance.

The main goal in optimizing DNS lookups of all sorts is to actually avoid doing external lookups whenever possible. The more DNS lookups that are performed the bigger the impact on the performance of the proxy server. DNS lookups are of course pretty much essential in running any sort of proxy, without a method to determine IP addresses and hostnames they will be unable to retrieve the information and URLs requested. Unfortunately there’s no way to completely replace these requests however one method can reduce the number that is required – DNS caching.

Reverse DNS lookups will be utilised when the IP address is available but we need the DNS Hostname. This is usually the situation when the connection is inbound and the receiver wants to find out which host the connection is coming from. In this situation the socket can actually be queried to obtain the IP address (that the connection is from) however the DNS Hostname would not be available in that information. This is because the TCP/IP protocol works with IP address and not DNS hostnames.

Reverse DNS requests are commonly needed to apply access rights and controls. This is because these are usually assigned by client hostname or domain name not IP addresses. For example it is typical to assign internet rights based on physical clients or membership of a domain group, the IP address is not typically used to control rights in this way. Also most logs store information on proxies in hostname format as they are much easier to track and follow than simply numerical addresses. This makes it easier to troubleshoot things like people using external Dns servers to watch American version of Netflix from their office!

If there is no requirement for DNS host names to be used for access control, then it is often feasible to turn reverse DNS lookups off – doing so will heavily boost the performance of any internet connected proxy server. Although having hostnames in logs is convenient, it is not alone worth the performance impact. The logs can be updated after with hostnames if required by resolving the IP addresses afterwards if required.

The updating of logs with hostname resolution is actually much more efficient if done in a single batch. This is because it is likely that there are individual IP addresses repeated in the logs and these can be resolved with a single request. Especially on proxy servers this can be a significant reduction because there will likely be a fixed number of IP addresses which are repeatedly requested.

John Halliwell
http://www.iplayerabroad.com/2016/07/20/bbc-vpn-block-real/

Protocol Verification by Proxies

It’s worth noting in any environment which actively uses proxies, that circuit level (generic) tunnelling such as those use by SOCKS and SSL tunnels will normally allow any protocol to be passed through the proxy. The implication of course is that the proxy doesn’t understand the protocol merely passes it on, which means also that the server cannot verify what is happening at the protocol level. This can be a dangerous situation especially if the proxy offers a gateway to internet or external traffic from the internet.

For illustration, any SSL tunnelling protocol can usually tunnel any TCP based protocol – so it could actually be used to telnet directly into the server. There are some huge dangers to allowing any server to transport protocols like this with no consideration to the operational requirements. It’s like leaving a huge back door to your network unless it is properly managed. There are of course options to control these protocols and one of the simpler is to restrict tunneling based on specific ports. So you would allow 443 open to allow HTTPS traffic, 563 for News and maybe 636 for secure LDAP. You’d have to extend this list to consider any other application or protocol requirements needed such as Windows Active Directory or Remote Access.

It works, is simple to implement however the reality is it’s not that secure. The well-known ports are only recommendations and there’s nothing to stop protocols being used on non-standard ports although of course this could cause issues in receiving the data if the servers are not configured to listen on these ports too. This VPN solution discussed on this page called Identity Cloaker which is used to access US versions of Netflix tunnels on a non-standard port for SSL traffic and allows the user to switch to any port.

This means you’ll be left with the unenviable situation where you’ll suspect dangerous traffic is being transported on a non standard port. This method of control means you’ll end up breaking other vital services if you attempt to block the port too. It’s not a long term solution although it is something that can be implemented whilst you try and create something more sophisticated.

The most efficient solution of course is to ensure that the proxy server can verify the protocol it is transporting. Therefore if someone is using non-standard ports or tunneling using a banned protocol or attempting to use a fake IP address like this for example then the proxy wil be able to highlight this issue. Once you have this awareness you can expand the functionality of proxy by building in more intelligence. The server can be used to identify common misuse and external attacks including attempting to use SSL to tunnel their terminal connections via Telnet.

Take Control of your IP Address

On a computer network, much like in real life, there are different levels of access dependent on a variety of reasons. It may be due rights assigned to username or account, perhaps an access token or often simply your physical location. These rights are assigned in different ways but the most popular method across the internet is based on your IP address.

The IP address is that unique number which is assigned to every single device which is connected to the internet, from computers and laptops to phones and tablets and even your internet enabled fridge. Every single device that is accessible online has a unique IP address and can be tracked by this number. Although you IP address can ultimately be traced back to a specific location and owner, this information is not available to any website that it visits. However even without access to an ISP record the IP address can be used to determine two pieces of information very easily – classification and location.


The first classification refers to the type of connection the IP address is registered to specifically residential or commercial. This piece of information is not always used as there can be some overlaps with this classification. The physical location however is used extensively by the vast majority of major web sites. Some may use it to help serve relevant content, perhaps supplying specific language versions depending on your location or serving up adverts which are more applicable to you. This is usually helpful although it can be very annoying if you are genuinely trying to access different content.

The most common use though is to block access based on this location, a practice used by virtually every large media site on the web. If you are in the USA for example, you will not be able to watch any of the UK media sites such as the BBC iPlayer or ITV Hub. Similarly every single one of the big American media sites will block non-US addresses. These blocks and controls are growing exponentially every year for instance there are now thousands of YouTube videos only accessible to specific locations.

Fortunately for the enlightened it isn’t such a big problem, because using VPNs and proxies you can actually control your own IP address. A simple method of using a British VPN server can give you access to the BBC iPlayer in the USA like this. It merely hides your physical location and instead the web site sees only the address of the VPN and it works with the vast majority of web sites.