HTTP Tracing

One of the most useful tools for troubleshooting in the HTTP/1.1 protocol is the TRACE method, which can provide lots of information for tracing routes between proxy chains.   Although the command is similar to the traceroute command, it is not identical as this tracks hops on the network router level whilst TRACE provides tracking based on the intermediate proxies involved in the route.

What can we use the HTTP TRACE command for?

  • identity the route between the proxies that the HTTP request makes.
  • identify each specific proxy in the chain
  • identify the server software, proxy version on each server
  • identify all versions of HTTP involved in the communication
  • detecting any loops in communication
  • tracking invalid responses and server misconfiguration

The command uses a similar format to the GET command, you pass the target and origin server URL as a parameter.  One important parameter to be aware of is the Max-Forwards: setting which specifies the maximum number of hops that are required.   This header is essential for detecting the presence of infinite loops present in a specified chain of proxies. It’s useful if there are complications like people running VPNs or external proxies like this.  If you do not use this parameter then any request will bounce between the proxies indefinitely.

Another useful facet of the TRACE method is the ability to use the command over a Telnet session which makes it extremely useful for troubleshooting remote sessions.  If you telnet to the first proxy in a chain before issuing the command then  you will get more accurate results.  To specify a particular route then the VIA: header can specify the route that trace will take.

Using the Proxy’s Cache for Troubleshooting

Sometimes an error or problem can appear intermittently, there may be a variety of reasons for this but these can be extremely difficult to troubleshoot.   In such situations the easiest way to find the cause is to examine the cache of the proxy servers which are involved.    It is essential that all key proxies are configured correctly to cache server responses

John Williams

Travellers Need a Fast US Proxy

Most of us now take some sort of internet enabled devices with us when we travel. Whether it is a smart phone, a tablet or laptop computer there’s usually room in our luggage for at least one of these electronic devices. Indeed many people will often have a variety of electronic gadgets with them at all times. Whether you’re travelling for business or pleasure having access to the internet can make life in an unfamiliar environment much simpler.

Booking hotels, tickets, making travel arrangements is so much simpler if you can research or even pay online. If you spend a lot of time abroad it can become even more vital, enabling you to keep in touch at home, organise your affairs and pay those household bills. Try arranging a direct debit with your bank over an international phone call from a hotel room, believe me it’s not as simple as you can imagine and doesn’t come close to the ease of most online banking systems.

However just as the internet was making life easier for the frequent travellers, many of the services we use are making it harder. You might find logging in to your banking site gets blocked when you’re abroad, accessing your AMC account to watch some TV will be banned to because of copyright issues. The list of US sites that can only be accessed from a US location grows ever week and frankly it’s a huge problem for many of us. To bypass these blocks there is a viable solution use a VPN or proxy server based in the US, if you connect via one of these you won;t get blocked by any region locking restrictions. Try this video – where the user gets an American IP address from outside the US.

As you can see without a VPN being connected, the service is inaccessible. However when you connect using a Fast USA proxy such as the one illustrated your physical location is immaterial. The website determines your location based on the IP address of the VPN server – if it’s in the USA then so you’ll appear to be there too. It’s crazy that we should require these sort of workarounds, the internet is by it’s very definition a global network and yet it’s being increasingly restricted and blocked. After all the most important time for me to use my online banking is when I can get to my physical bank. The time I want to use online entertainment services when I’m away from my home TV set and stuck in a foreign hotel.

If you use these services, remember to try them out first. There are literally thousands of these VPN and proxy service available online but the vast majority of them are pretty hopeless. Without proper support and a decent infrastructure behind them you’ll find many are slow and unreliable. Particularly for accessing online movies and films speed will be essential otherwise it will be a painful experience!

John Simpson

Is Anonymity Important Online?

There are many discussions across the world about using the internet and how it should be policed.  Many of the less democratic countries already have rather sweeping digital laws allowing content to be blocked, services closed down and users arrested.  These laws usually are phrased rather vaguely, using excuses like national interest or public safety.  They’re usually designed to be broad enough to cover whichever situation the authorities require without sounding unduly restrictive.  The reality is that in many countries the 140 characters of a Tweet is enough to get you hefty prison sentences.

People seek anonymity for different reasons depending on their location.  Of course in countries like Iran, China and lots of Far Eastern  you have to be very careful what  you say online, if you criticise leaders that can be enough to get you locked away for a very long time.  In 2015 a Thai man ‘liked’ and ‘shared’ a Facebook photograph which was critical of the Thai Royal family, he’s currently awaiting trial and faces 32 years in jail.  Needless to say Thailand is a country where you should be very careful about what you do online particularly if it involves the royal family.

In other more democratic and arguably civilized countries there are somewhat different concerns about privacy online.  You are unlikely to get arrested for being critical of Western leaders online, however don’t assume that your comments are not being monitored.  Most of the advanced countries, particularly in places like the US and UK, online activity is extensively logged.   In the UK legislation is being passed to legitimize this behaviour but it’s fairly certain to assume it’s already being going on for many years prior to this.

Much of the problems about privacy relate to the fact that it’s so easy to monitor people online.  The internet is simply not designed for privacy, it uses insecure clear text protocols like HTTP and email, whilst distributing our connections through a mesh of hardware owned by all sorts of people and corporations.   If you have access to a network hardware in a telecommunications company then there’s little you can’t access with the right resources.  Of course, the morality of this can be quite unclear but there are other areas where legality can be used as a perfectly justifiable excuse.

For example download a Bit Torrent client, join a swarm to download a pirated copy of the latest blockbuster movie and in your screen you’ll instantly see a page full of IP addresses of people illegally downloading copyrighted material.  It’s not hidden, not hard to find and only one step away from turning that into a list of names and addresses.   The people who use these programs are mostly unaware that they are not downloading torrents anonymously, in fact they’re doing it whilst actively broadcasting their identities.

The important factor to remember whatever you’re doing online, wherever you are and irrespective of who you are – you are probably being monitored to some extent.  Whether it’s merely being sucked up by one of the UK security services huge data trawls or more specifically by a media company seeking damages for copyright infringements – it could be happening.

John Herrod

Technology Author and Consultant

Searching for a VPN with Residential IP ?

So why would anyone be trying to find a VPN with a residential IP? Well, for the sake of clarity, there are certain distinct classifications of IP addresses which are becoming more and more important.  It refers to the actual categorisation of their use rather than any complicated technical property.  The fact is that there are only two of these categories –

  • Commercial IP Addresses – allocated to private companies and datacentres
  • Residential IP Addresses  – allocated to individuals usually through Internet Service Providers (ISPs)

There’s no technical distinction, no difference in structure or allocation – you can’t tell simply by looking at an IP address which is commercial and which is residential address.   In fact it’s entirely possible for addresses to switch between the two categories if they are reallocated.  However the classification is being used increasingly by web sites and services to distinguish between customers.

Is VPN Safe

Take for example a standard residential IP address assigned from a small ISP, any website can determine that this is likely to be a private individual likely to be surfing from their home computer.   The origin of a commercial IP address is much more difficult to determine –  it could be from a commercial organisation, from a wireless access point, directly from a server or bot or perhaps a standard user who is using a VPN or proxy server.   If you’re a website owner looking for customers for example, it’s the residential traffic that is going to interest you most not the commercial stuff.

It’s a classification which is now being used by many websites to block traffic from specific sources.  You can see in this post about VPNs being blocked by Netflix that the media giant is using this classification to stop people bypassing the region blocks by using proxy type servers to hide their locations.  Netflix has simply decided that if you are originating from a commercial based IP address then you can’t access their service irrespective of whether you have a subscription or not.   Which is why people are becoming increasingly desperate to find a VPN with a residential IP address.

It’s not just the media companies who are starting this, other sites are increasingly looking to block all non-residential based addresses too.  There are advertising sites like Craigslist and Gumtree who want to isolate their services to specific local home markets and people using VPNs or commercial servers to access them globally aren’t in that category.   There are casualties of course, VPNs are important ways to maintain the security of your internet connection and privacy yet using one is likely to get you blocked from certain sites.  Additionally there are many countries where it’s not safe to post openly and a VPN is essential to use the internet securely.

There are a few VPN services which now offer residential IP addresses included, like this one at Identity Cloaker which routes Netflix traffic through residential address to avoid being blocked.  However they are quite rare simply because the addresses are very difficult to obtain and cost much more than standard commercial IP addresses available from a datacenter.

On Demand Caching for Proxies

Caching is one of the most important functions performed by proxy servers particularly in a corporate environment.  This is especially relevant when the network has internet connectivity to the desktop, caching is important to help reduce the amount of traffic generated from accessing the web.

If you look at the logs of any corporate network and analyse which external websites are being visited you’ll normally find that a large percentage of traffic is generated to a small number of sites.  News and social media sites if not blocked will often be accessed repeatedly, which means multiple requests for the same information.  Using a proxy server to cache these pages locally can vastly reduce the amount of network traffic generated by these requests.

For example in the UK you may find that a popular website like the BBC is generating hundreds of requests for the news pages.  If you enable on demand caching on a proxy server, when the first page is requested the proxy will store a copy of that page locally.   When the proxy receives the next request for the same page it will provide the cached copy from it’s store and will not need to visit the web page.    This means that no external traffic will be generated in this example and the amount of external bandwidth used will be heavily reduced.

This is called on-demand caching and it means that the web server/proxy only stores documents which are requested by a client.   The server will not attempt to store other pages from that server, only those which are specifically requested by the client browser.  This also helps you filter traffic which is not appropriate for example if someone was using a VPN to stream Netflix to their desktop.

In bigger organisations although proxies configured with caching can dramatically decrease network traffic, one is rarely enough.    However it obviously makes little sense to have duplicate proxies all caching the same external pages.  The question then is how to distribute this data efficiently within the network and to stop any individual proxy from being overloaded.  One of the most common models used in this scenario is that of the replication model, which involves the server mirroring or replicating it’s content to other servers in the network.

John Soames, Working Netflix VPN, Cromer Press, 2015