Escaping Region Locking from Media Sites

In the early years of the internet there were very few restrictions on what you could see and download. If you started a web browser in the US you’d get pretty much the same experience as someone who started in Cairo. Obviously there might be some variation in speed of course, but what you could see and do was almost identical.

That’s changed a lot now with the growing popularity of region locking and control. It started off fairly helpfully – your search engines would switch you to the appropriate location based on your IP address. This meant that if you were searching from London for electricians you wouldn’t get directed to results in Sydney which would obviously be useful. We’ve got used to this and it generally makes everyone’s life much easier.

However the use of region locking has extended greatly in the last few years, in fact any major web site will usually operate some level of control. Often it’s again beneficial, Amazon will make sure you go to the UK site, Costco will direct you to your local store and so on. However for many of the world’s biggest media sites it’s a much different story – region locking usually means region blocking.

Ever tried to access Pandora from outside the US? Well it doesn’t work, the wonderful music site is only accessible for those located in the US. Want to watch the BBC News, sorry if you’re outside the UK it’s not going to happen. Those are just two but the list is extensive, in fact it’s unlikely you’ll now find a large media site which doesn’t lock down access based on the location of your IP address.

It’s crazy when you think about it, a global communication medium deliberately trying to segregate and restrict our world. Worse too that in a time when many of us travel extensively, we are blocked and filtered at every turn when we’re online.

So What’s the Solution?
Well to take back control and stop being blocked you need to be able to control your IP address. Unfortunately for most of us that’s not possible, the IP address is assigned when you connect to the internet and there is no way of modifying it. You can of course modify your local address but that’s not important, region locking uses your external internet facing IP address.

However although you cannot modify your address, you can hide it by using VPN servers to protect your connection. If you connect to a UK VPN server for example, it will look as though your have a UK IP address and watching the BBC works without problems. You can use a US VPN to gain a US IP address for Netflix irrespective of where you actually are. Many firms have developed services to support this demand and the top VPN providers will allow access to a network of servers in different countries.

This means that although you cannot change your real IP address, you can hide it behind a VPN server. It gives you back control and neatly sidesteps the pervasive region locking and filtering which seems likely to keep expanding.

Further Reading – British VPN

Methods of Designating IP Addresses

There are a variety of methods that can be used to assign an IP address to a windows client. You can obviously assign directly by allocating a static address or by using a variety of other methods such as ARP, BOOTP or DHCP. The main methods will be discussed briefly below:

Static Configuration
The majority of network enabled devices including Windows computers can be assigned an IP address manually. This is normally allocated in the TCP/IP properties on windows machines and in the network configuration options on other devices like games consoles and media streamer for instance. It is important that although any address can be allocated here, it should be the correct address for the network configuration. You should also ensure that the IP address is unique on that network,otherwise network connectivity will be affected for both clients with duplicate addresses.

RARP and BOOTP
Two of the main options for assigning an ip address dynamically are Reverse ARP (RARP) and the Bootstrap protocol (BootP). RARP requires a server which maintains a list of hardware addresses and a pool of IP addresses to be allocated. The server would normally be contacted via a broadcast initiated from the client before an ip address is allocated and then assigned to the hardware id. There are often issues with this method of address allocation though:

  • Clients broadcasts will sometimes not reach the RARP server. This might be for various reasons but is often to do with network topology, perhaps a router is incorrectly configured on the network. This can be resolved by configuring an IP helper address on any routers. It may also be simpler to configure the router to be a RARP servers depending on the model.
  • RARP server does not have an IP address which corresponds to the client’s hardware address.

Mostly due to various inefficiencies RARP is rarely used in modern networks and is usually replaced by the more sophisticated BOOTP to assign IP addresses.  This runs over UDP and sets up a port for client requests and another port is assigned for server responses.   The response from the BOOTP server actually contains additional information such as the address of the local gateway.   BootP suffers similar problems to RARP mainly to do with network connectivity.  UDP often suffers connectivity problems and the routers should be configured to allow UDP traffic and without interruption,  other problems can be with access lists filtering out the UDP ports.

Further Reading

Protocol Verification by Proxies

It’s worth noting in any environment which actively uses proxies, that circuit level (generic) tunnelling such as those use by SOCKS and SSL tunnels will normally allow any protocol to be passed through the proxy. The implication of course is that the proxy doesn’t understand the protocol merely passes it on, which means also that the server cannot verify what is happening at the protocol level. This can be a dangerous situation especially if the proxy offers a gateway to internet or external traffic from the internet.

For illustration, any SSL tunnelling protocol can usually tunnel any TCP based protocol – so it could actually be used to telnet directly into the server. There are some huge dangers to allowing any server to transport protocols like this with no consideration to the operational requirements. It’s like leaving a huge back door to your network unless it is properly managed. There are of course options to control these protocols and one of the simpler is to restrict tunneling based on specific ports. So you would allow 443 open to allow HTTPS traffic, 563 for News and maybe 636 for secure LDAP. You’d have to extend this list to consider any other application or protocol requirements needed such as Windows Active Directory or Remote Access.

It works, is simple to implement however the reality is it’s not that secure. The well-known ports are only recommendations and there’s nothing to stop protocols being used on non-standard ports although of course this could cause issues in receiving the data if the servers are not configured to listen on these ports too. This VPN solution discussed on this page called Identity Cloaker which is used to access US versions of Netflix tunnels on a non-standard port for SSL traffic and allows the user to switch to any port.

This means you’ll be left with the unenviable situation where you’ll suspect dangerous traffic is being transported on a non standard port. This method of control means you’ll end up breaking other vital services if you attempt to block the port too. It’s not a long term solution although it is something that can be implemented whilst you try and create something more sophisticated.

The most efficient solution of course is to ensure that the proxy server can verify the protocol it is transporting. Therefore if someone is using non-standard ports or tunneling using a banned protocol or attempting to use a fake IP address like this for example then the proxy wil be able to highlight this issue. Once you have this awareness you can expand the functionality of proxy by building in more intelligence. The server can be used to identify common misuse and external attacks including attempting to use SSL to tunnel their terminal connections via Telnet.

Take Control of your IP Address

On a computer network, much like in real life, there are different levels of access dependent on a variety of reasons. It may be due rights assigned to username or account, perhaps an access token or often simply your physical location. These rights are assigned in different ways but the most popular method across the internet is based on your IP address.

The IP address is that unique number which is assigned to every single device which is connected to the internet, from computers and laptops to phones and tablets and even your internet enabled fridge. Every single device that is accessible online has a unique IP address and can be tracked by this number. Although you IP address can ultimately be traced back to a specific location and owner, this information is not available to any website that it visits. However even without access to an ISP record the IP address can be used to determine two pieces of information very easily – classification and location.


The first classification refers to the type of connection the IP address is registered to specifically residential or commercial. This piece of information is not always used as there can be some overlaps with this classification. The physical location however is used extensively by the vast majority of major web sites. Some may use it to help serve relevant content, perhaps supplying specific language versions depending on your location or serving up adverts which are more applicable to you. This is usually helpful although it can be very annoying if you are genuinely trying to access different content.

The most common use though is to block access based on this location, a practice used by virtually every large media site on the web. If you are in the USA for example, you will not be able to watch any of the UK media sites such as the BBC iPlayer or ITV Hub. Similarly every single one of the big American media sites will block non-US addresses. These blocks and controls are growing exponentially every year for instance there are now thousands of YouTube videos only accessible to specific locations.

Fortunately for the enlightened it isn’t such a big problem, because using VPNs and proxies you can actually control your own IP address. A simple method of using a British VPN server can give you access to the BBC iPlayer in the USA like this. It merely hides your physical location and instead the web site sees only the address of the VPN and it works with the vast majority of web sites.

HTTP Tracing

One of the most useful tools for troubleshooting in the HTTP/1.1 protocol is the TRACE method, which can provide lots of information for tracing routes between proxy chains.   Although the command is similar to the traceroute command, it is not identical as this tracks hops on the network router level whilst TRACE provides tracking based on the intermediate proxies involved in the route.

What can we use the HTTP TRACE command for?

  • identity the route between the proxies that the HTTP request makes.
  • identify each specific proxy in the chain
  • identify the server software, proxy version on each server
  • identify all versions of HTTP involved in the communication
  • detecting any loops in communication
  • tracking invalid responses and server misconfiguration

The command uses a similar format to the GET command, you pass the target and origin server URL as a parameter.  One important parameter to be aware of is the Max-Forwards: setting which specifies the maximum number of hops that are required.   This header is essential for detecting the presence of infinite loops present in a specified chain of proxies. It’s useful if there are complications like people running VPNs or external proxies like this.  If you do not use this parameter then any request will bounce between the proxies indefinitely.

Another useful facet of the TRACE method is the ability to use the command over a Telnet session which makes it extremely useful for troubleshooting remote sessions.  If you telnet to the first proxy in a chain before issuing the command then  you will get more accurate results.  To specify a particular route then the VIA: header can specify the route that trace will take.

Using the Proxy’s Cache for Troubleshooting

Sometimes an error or problem can appear intermittently, there may be a variety of reasons for this but these can be extremely difficult to troubleshoot.   In such situations the easiest way to find the cause is to examine the cache of the proxy servers which are involved.    It is essential that all key proxies are configured correctly to cache server responses

John Williams
http://identityvoucher.co.uk/

Is Anonymity Important Online?

There are many discussions across the world about using the internet and how it should be policed.  Many of the less democratic countries already have rather sweeping digital laws allowing content to be blocked, services closed down and users arrested.  These laws usually are phrased rather vaguely, using excuses like national interest or public safety.  They’re usually designed to be broad enough to cover whichever situation the authorities require without sounding unduly restrictive.  The reality is that in many countries the 140 characters of a Tweet is enough to get you hefty prison sentences.

People seek anonymity for different reasons depending on their location.  Of course in countries like Iran, China and lots of Far Eastern  you have to be very careful what  you say online, if you criticise leaders that can be enough to get you locked away for a very long time.  In 2015 a Thai man ‘liked’ and ‘shared’ a Facebook photograph which was critical of the Thai Royal family, he’s currently awaiting trial and faces 32 years in jail.  Needless to say Thailand is a country where you should be very careful about what you do online particularly if it involves the royal family.

In other more democratic and arguably civilized countries there are somewhat different concerns about privacy online.  You are unlikely to get arrested for being critical of Western leaders online, however don’t assume that your comments are not being monitored.  Most of the advanced countries, particularly in places like the US and UK, online activity is extensively logged.   In the UK legislation is being passed to legitimize this behaviour but it’s fairly certain to assume it’s already being going on for many years prior to this.

Much of the problems about privacy relate to the fact that it’s so easy to monitor people online.  The internet is simply not designed for privacy, it uses insecure clear text protocols like HTTP and email, whilst distributing our connections through a mesh of hardware owned by all sorts of people and corporations.   If you have access to a network hardware in a telecommunications company then there’s little you can’t access with the right resources.  Of course, the morality of this can be quite unclear but there are other areas where legality can be used as a perfectly justifiable excuse.

For example download a Bit Torrent client, join a swarm to download a pirated copy of the latest blockbuster movie and in your screen you’ll instantly see a page full of IP addresses of people illegally downloading copyrighted material.  It’s not hidden, not hard to find and only one step away from turning that into a list of names and addresses.   The people who use these programs are mostly unaware that they are not downloading torrents anonymously, in fact they’re doing it whilst actively broadcasting their identities.

The important factor to remember whatever you’re doing online, wherever you are and irrespective of who you are – you are probably being monitored to some extent.  Whether it’s merely being sucked up by one of the UK security services huge data trawls or more specifically by a media company seeking damages for copyright infringements – it could be happening.

John Herrod

Technology Author and Consultant

On Demand Caching for Proxies

Caching is one of the most important functions performed by proxy servers particularly in a corporate environment.  This is especially relevant when the network has internet connectivity to the desktop, caching is important to help reduce the amount of traffic generated from accessing the web.

If you look at the logs of any corporate network and analyse which external websites are being visited you’ll normally find that a large percentage of traffic is generated to a small number of sites.  News and social media sites if not blocked will often be accessed repeatedly, which means multiple requests for the same information.  Using a proxy server to cache these pages locally can vastly reduce the amount of network traffic generated by these requests.

For example in the UK you may find that a popular website like the BBC is generating hundreds of requests for the news pages.  If you enable on demand caching on a proxy server, when the first page is requested the proxy will store a copy of that page locally.   When the proxy receives the next request for the same page it will provide the cached copy from it’s store and will not need to visit the web page.    This means that no external traffic will be generated in this example and the amount of external bandwidth used will be heavily reduced.

This is called on-demand caching and it means that the web server/proxy only stores documents which are requested by a client.   The server will not attempt to store other pages from that server, only those which are specifically requested by the client browser.  This also helps you filter traffic which is not appropriate for example if someone was using a VPN to stream Netflix to their desktop.

In bigger organisations although proxies configured with caching can dramatically decrease network traffic, one is rarely enough.    However it obviously makes little sense to have duplicate proxies all caching the same external pages.  The question then is how to distribute this data efficiently within the network and to stop any individual proxy from being overloaded.  One of the most common models used in this scenario is that of the replication model, which involves the server mirroring or replicating it’s content to other servers in the network.

John Soames, Working Netflix VPN, Cromer Press, 2015

Introduction to DNS Recursion

The Internet’s DNS structure is often (accurately) described as hierarchical with authoritative servers sitting at the top of the structure.  However because of this setup it is essential that all DNS servers are able to communicate with each other in order to supply response to the name queries which are submitted by clients.

This is because although we would expect our companies internal DNS server to know all the addresses of internal clients and servers, we wouldn’t expect it’s database to contain every external server on the the internet.     Although in the early days of the internet, most DNS servers did contain an entire list of connected server addresses, nowadays that would simply not be feasible or in fact very sensible.

When a DNS server needs to find an address which is not in it’s database, it will query another DNS server on behalf of the requesting client in order to find the answer.    The server in this instance is actually acting in the same way as a client by making a request to another DNS server for the information, this process is known as recursion.

It’s actually quite difficult to detect whether a query is answered by recursion or by directly when troubleshooting DNS queries.    You need to be able to listen to all a DNS servers traffic in order to identify a recursive query.   The additional query (recursive one) is generated after the DNS serverc has checked it’s local database in order to resolve the query.  If this isn’t successful the DNS server will generate the additional request before replying to the client.   This is also dependent on the recursion bit being set in the initial query from the client too, as this allows the server to ask another server if the answer is not in it’s own database.

The recursive query is merely a copy of the initial DNS request and it has the effect of turning the server into a client. You can notice if you analyse the traffic that the transaction ID numbers will change in order to differentiate the initial query from the recursive query sent by the DNS server.   It’s important to keep a note of these transaction IDs when troubleshooting DNS traffic as it’s easy to get confused as many of the packets will look very similar.  If you are trying to analyze something more complicated like the modern, intelligent Smart DNS servers like these – http://www.proxyusa.com/smart-dns-netflix-its-back then it’s even more important to keep track of these transactions.  This is because these DNS servers actually make decisions on how to route the traffic in addition to resolving queries.

 

Residential IP Gateways

For anyone with a significant interest in working online, your IP address is important, it’s a vital part of your online presence.     Most people don’t really care about their address, as long as you have a valid IP address you can get online.   However there are distinctions about these addresses which can make a huge difference to your online experience.

Often the first indication people have that their IP address is of any relevance is when they find themselves getting blocked somewhere.   You might click on a video or website and get redirected to a message ‘sorry not available in  your country’ or you might try and view a website and get redirected somewhere else.   What’s generally to blame is where your IP address is registered and this behaviour is called ‘region locking’.  It’s extremely common and annoying especially if you’re settling down to watch the BBC News live while on holiday outside the UK for example.

This is all factored around the geographical location where you’re IP address is assigned to.  Which is why it usually becomes evident when people travel or go on their holidays, suddenly they find they can’t access the websites that they used to.  Watching domestic TV, streaming videos or accessing their online banking and things like that suddenly become very difficult when you’re outside your usual location.

People have found ways around this, normally you can hide your location by using a proxy or VPN service.  However this only works on a basic level, because there are other restrictions which stop these working mainly centered around the IP classification.   You see many websites now also look one step further than simply location – they look at the classification of the address and whether it originates from a commercial or residential origin.

Anyone who makes their living online is likely to need a little more control.  After all operating in a global market like the internet, getting blocked all the time because of location and what sort of IP address you have is going to be extremely inconvenient.   Sure you can use traditional proxies which are mostly run from datacentres but they too have significant problems.  The issue is that websites increasingly block access to all but residential IP addresses, they just want ordinary home users which means none of these proxy solutions actually work.  The alternative is to use VPNs that have residential IP addresses and gateways built in (read more here)

However it’s much, much harder to set up a residential IP gateway than it is a commercial one.  For instance you can’t just roll up to Comcast or BT and ask it to assign you a few hundred IP addresses, they use those for domestic customers only.   They are appearing but at the moment they are fairly hard to find and extremely expensive.  You have to be careful though as some of these ‘solutions’ actually piggy back domestic customers computers like the not recommended Hola which is a huge security risk to use.

World Wide Web Proxies – Web Proxy List

In the earliest days of the web in 1990, web proxy servers were usually referred to as gateways.  In fact the very first web gateway was created at CERN by the original WWW team , headed by Tim Berners-Lee.

Gateways are effectively devices which are used to forward packets between different networks. These devices can vary in complexity from simple pass through devices to complex system which are able to understand and convert different protocols.   It was in 1993 that the name Web Proxy Server was chosen as a standard term to describe the different types of Web gateways.

Web Proxy Server

These can be further classified into two distinct categories:

Proxy Server – internet/firewall gateways which act in response to client/PC requests.

Information Gateway – gateways which act in response to server requests.

However these are quite broad specification and below you will find some details of the key properties of the proxy servers and associated gateways.  Remember that these classifications can be affected by any application software which is installed on the proxies so they are not necessarily just the simple servers you find on web proxy lists – which are normally just basic Glype installations.   Particularly you may find that destinations and transparency may sometimes  be modified.

Proxy Server Properties

These are the general properties which can be applied to any specific proxy server, there are variations which will affect these.

Transparency: these proxies do not modify the data passing through them. They will perform any filtering specified by rules but this will not affect the end result. The connection will be the same if it was direct or through the proxy server.

Control: the client will determine whether it is uses the proxy or not.  This is typically controlled on the client by specifying the address of the proxy or through client based software.

Destination: the final destination of any request is not affected by any intermediate proxy.  In fact a client or user will often be completely unaware of the existence of the proxy.

Proxies can provide all sorts of features some of which might affect these properties.   They can be used to provide specific access controls, filtering, logging and even simply to speed up access to remote web resources through caching features.

It is in corporate environments that the transparency properties of proxies has usually been modified.  Frequently these firewall proxy servers will sit in the DMZ (Demilitarized Zone) and control both inbound and outbound traffic.  They will accept network requests from clients and forward them out to the internet if approved, then relay the replies back to the clients.   Most of these will also operate caching services to ensure that duplicate requests don’t generate more network traffic and bandwidth charges.

The other advantage of the dual role proxies is that it can act as a single entry point for internet access.   This means that all requests can be logged and monitored allowing an element of control to web access through the company infrastructure.  It also allows replies to be monitored for harmful code such as malware and viruses, this is an important extra layer of security to protect the internal network.