Buy US Proxy with Transparent Proxying

When we are discussing the technological characteristics of proxies there’s one term which you will see used very often – ‘transparent’.    It can actually be used in two distinct ways when it comes to proxies.  The first is to refer to a definition which implies transparent proxying ensures that any user will see no difference to the original request whether it goes direct to the server or through a proxy.   In an ideal world pretty much all legitimate proxies would be considered ‘transparent’.

Proxies are however significantly more advanced from the early years when this original definition was created.  The term ‘transparent proxying’ now has much more meaning.  The extended definition means that transparent proxying ensures that the client software is not aware of the existence of the proxy server in the communication stream.   This is unusual because the client was usually configured to use a proxy, perhaps by the internet settings in it’s browser configuration.    Software would then make a decision in it’s requests and perhaps distinguish between proxy and direct requests.

When transparent proxying, in it’s modern context, is used the router is programmed to redirect the request through the proxy not the client. This means that the proxy can actually be used to intercept and control all HTTP requests that are targeted by outbound connections.  The request can even be parsed or perhaps even filtered and redirected.  This control allows the network to configure access control rules on all outbound requests,  A company network could use these to ensure unsuitable requests are not being made from a corporate network e.g. illegal web sites.

This level of transparent proxying leaves the client completely unaware of the existence of an intermediate proxy server.   There are some caveats though and the proxy can be detected in certain circumstances.  For  example there is little point in investing in a USA proxy buy if the server only supports HTTP/1.1 because the protocol makes no allowance for transparency in proxying information.

One of the main issues and indeed worries is that allowing completely transparent proxying might cause other issues particularly in the client side applications.  For example one of the fundamentals of using proxies in a corporate network is to reduce traffic by caching locally.  This could cause all sorts of problems if the behaviour of the proxy cache effects communication between the destination server and the client application.

Further Reading – http://www.changeipaddress.net/us-ip-address-for-netflix/

Optimizing Proxies – Protocol Performance

The importance of the data transport protocol is of course crucial to a global information network like the world wide web.  Unfortunately the HTTP/1.0 protocol has some inherent issues which are directly related to performance which have been largely addressed in version 1.1 of the protocol.  It is expected that future developments will further improve the performance of the protocol.

One issue is related to the three way handshake that is required by TCP before it can establish the connection. It is important to remember that during this handshake phase that no application data is transferred at all.  from the user perspective the delay will simply appear as latency in getting the initial connection established.   This three way handshake involves a considerable overhear preceding data transfer and has a noticeable effect on performance particularly in busy networks.

This problem is made worse by using the HTTP 1.0 protocol which makes extensive use of new connections.  In fact every new request requires a new TCP connection to be established, complete with a new three way handshake.  This was originally implemented as a measure to boost performance because it was thought that it would avoid long lived idle connections being left dormant.  The reasoning was that it was more efficient to establish new connections when required as the data burst would be small and frequent.

However the web has not developed like this and it’s is much more than a series of short html files quickly downloaded.  Instead the web is full of large documents and pages embedded with videos and images.  Add to the the multitude of applets, code and other embedded objects and this soon adds up.  What’s more each of these objects usually has it’s own URL and so requires a separate HTTP request for each.    Even if you invest in a high quality US proxy you’ll find some impact on speed using HTTP 1.0 simply due ti the huge number of connection requests it generates.

There were modifications made to increase the perceived performance from the user perspective.  For one, the use of multiple simultaneous connections was allowed and this would allow client software like browsers to download and render multiple components on a page.  This meant that the user wasn’t left waiting as individual components were loaded separately.  However although parallel connections increase performance on an individual level, they generally have a very negative impact on the network as a whole.   The process is still inefficient and allowing parallel connections does little to mitigate this situation.

As any network administrator knows, focussing on a single aspect of network performance is rarely a good idea and will almost never improve overall network performance.    The persistent connection feature was introduced to help solve this, and was added as a non-standard extension to HTPP 1.0 and included by default with HTTP 1.1.

Further Reading: Proxies Blocked by BBC Abroad

Remote Login Methods

The ability to remotely login to a machine that’s miles away from you is perhaps one of the internet’s most popular applications.  It might not seem so, but being able to access a remote host without a hard wire connection has transformed many areas of IT particularly in support and development.   Obviously you need an account on the host that you are trying to login to, but actually using the machine as if you are at the console is extremely useful in many situations.

Two of the most famous applications for remote login access when using a TCP/IP based network (e.g like the internet) are Telnet and Rlogin.   The most famous and probably used by every IT support technician over the age of 25 is Telnet, installed as standard in almost every TCP/IP implementation.   It seems relatively simple but this actually hides some great functionality not least the ability to Telnet from one operating system to another.  It’s incredibly useful to be able to sit at a Microsoft Windows machine with multiple command interfaces open in separate windows to Unix and Linux machines at the same time.

Remember these terminal windows are actually like physically sitting at the remote host’s console.  This is is completely different from just using a web session or using something like an Italian to stream RAI player abroad like this.  Each individual character that you type is entered into the remote host, there’s no streaming, no relaying or filtering.  Obviously there are some restrictions about running a terminal windows on a completely different systems.  However Telnet does an option negotiation phase between the client and server to ensure that only services which are supported at both ends are available.

The other famous remote login application is called Rlogin which was developed from Berkeley Unix.   This application was initially only available on Unix Systems however it has been ported to most other operating systems now and you can Rlogin between Windows and Linux.  Both of these applications use the Client/Server configuration – the client is the system where the initial connection is established to the remote server which is the target.

Nowadays, the most popular of the two application – Telnet has become much more sophisticated.  Over the years lots of functionality has been added to Telnet whereas Rlogin remains quite simple and unmodified.  However it should be noted that although Rlogin lacks features, it is a simple and stable remote access application.

The author – John Herrington has worked in IT for over thirty years in a variety of roles from support to latterly Network manager at a large bank.   He now works for himself and runs one of the largest paid VPN services on the West Coast of America. He obviously works remotely a lot of the time but will rarely use Telnet as it’s too insecure!

Tracking VPN and Proxy Users

There are similar challenges for network administrators in corporate networks and those running firewalls for authoritarian regimes about the use of proxies and VPN services.  The issue is that not only do they allow individuals the freedom to conduct their internet activity without being tracked, a VPN will also prevent most aspects of logging taking place too.

If you imagine a company network it means that an individual could potentially conduct all sorts of behaviour from a company computer whilst sitting in a corporate office whilst at work.   They could be downloading films, streaming Netflix or something perhaps much more sinister even.  Obviously this is potentially a risk to both the network infrastructure and also potentially to the company’s reputation.

So how do you block the use of VPNs and proxies?  For a corporate network there are actually many more options, and the simplest is probably to stop any sort of VPN and proxy being used in the first place.   You can lock down the advanced settings in a web browser quite simply, for example the Internet Explorer Administration Kit (IEAK) allows you to configure and deploy an IE package which cannot be modified onto every client in your organisation.  This stops proxies being used manually and VPN clients can be blocked by ensuring that  standard users have no administrative access to their desktops.

It is certainly easier to block any installation than trying to track the use of VPNs particularly some of the most sophisticated ones.   For example although you could potentially monitor logs in firewalls and routers for specific IP addresses which looked like VPNs some services allow you to switch to a range of IP addresses – Hide My VPN like the one in this video demonstrates:

As you can see if a service is rotated then identifying the VPN by it’s IP address is much more difficult.  However blocking installation of the highlighted service Identity Cloaker can also be difficult as it has a mobile version which can be run directly from a USB disk.

You can see that proxies are fairly irrelevant today as they can be easily blocked, also most content filters can detect their use too.   Significantly their use has now dropped globally for additional reasons mainly that they are mostly detected by websites which operate regional restrictions.   It is the more sophisticated Virtual private networks which are the difficulty, particularly those equipped with various VPN hider technologies and advanced encryption.

VPN Blocking on the Rise

For years people have used VPNs for all sorts of reasons, but it’s origin lay quite simply in the security they provided.  International companies will normally insist that their employees use VPN services when remotely connecting back to their servers using the internet.  It makes sense, otherwise important information and credentials would be trusted to the owners of coffee shop wifi or the administrator of your local Premier Lodge or hotel chain.

The concept is simple, create an encrypted tunnel which ensures that all the data which normally is passed in clear text instead is encrypted and unreadable.  Of course, this security means that as well as being safe from computer criminals and identity thieves – it’s also secure from intelligence services and state controlled snoopers too.  It should come as no surprise that anyone who opposes free speech generally hates VPNs and the protection that they give.

So when we hear stories about different organisations and companies from the Netflix to the Chinese Government trying to block VPNs what are they doing.  Well it depends, obviously the situation that leads to thousands of BBC iPlayer VPN not working is going to be slightly different to the Chinese throwing billions at the great firewall of China.   However the general techniques are basically the same as a small company want to achieve the same thing.

One of the most common options is to block the ports used by these services.  Most VPN tunnelling protocols operate on standard ports, e.g using PPTP or LTP.  They need to establish these connections to transfer and receive data, without them the service won’t function.  Other methods include identifying and blocking specific IP addresses or ranges which are being used by VPN services.   It is these two methods that are mostly used by the big media companies like Hulu and the BBC.

These methods can be time consuming though and it’s possible to switch address and some services allow you to configure alternative ports too. The Chinese Government as you would expect have gone one step forward and use more sophisticated techniques like deep packet inspection.   These involved looking at the data itself to identify if a VPN is being used to transport it.  For example if you are unable to read any data because none of it’s in clear text then there is the likelihood that it is being encrypted.   Of course, there are other methods which encrypt data like SSL so you need to be careful that you don’t block other traffic, it’s a risk that the Chinese would probably be happy to take however.

Even these methods are not foolproof and VPN companies can scramble things like the meta data to make identifying the use of a VPN even harder.  It is worthwhile noting that many people in China still use VPNs routinely and so if the huge resources available to the Chinese State can’t block their use – we should be ok to have a BBC VPN like this for the foreseeable future.

 

 

TCP Extensions – Virtual Circuits

TCP provides lots of additional services which have been added over it’s lifetime one of the more useful ones is that of the virtual circuit transport service. There are three distinct phases in the life of any TCP connection – establishment, transferring data and termination.    There are many applications including things like remote login and those that enable file transfer which are perfectly suited to using a virtual circuit type service.    Many other applications are suited better towards a transaction based service which is basically a client request followed by a server response.  This can be explained by briefly detailing it’s characteristics:

1: Any overhead of connection establishment and the subsequent termination should be minimized.  Ideally one request should be sent followed by the corresponding receive before any other packets are sent.

2: Latency should be reduced to the sum of the round trip time (RTT) plus the server processing time (SPT).

3: Server should be capable of detecting duplicate requests and not processing them again.

A very important application uses this type of service which forms the very backbone of the internet – the Domain Name System (DNS).   Other common applications such as the BBC VPN many people use to bypass the numerous region locking systems which exist online.   The other important decision that an application developer must consider is whether to use UDP or TCP for the transport.  The difficulty is that TCP simply provides too many features for an efficient transaction whilst  UDP doesn’t really provide enough.   Normally UDP is used simply because it avoids the overhead of TCP connections but this involves adding the features that are required like retransmission, dynamics timeouts and congestion avoidance.

The solution that is a better alternative than this is to provide an additional transport layer to provide more efficient handling for the transactions.  The transaction protocol which is commonly used now by many applications is called T/TCP defined in RFC 379 – extending the TCP protocol for transactions.

Remember most TCPs require 7 segments to open and close a connection.  An additional three more segments are added to deal with the requests and replies (initial and the one responding to the ACK).  In addition it may be necessary to add extra control bits to deal with other functionality and connection information required to complete the transactions properly.

Further Reading:

James Hibbert: Polskie Proxy, Haber Press, 2017

 

 

Using a VPN/Proxy on Different Devices

For the internet free spirit who wants full access to the web without restrictions, filters or logs then using a proxy or VPN is pretty much essential. I say that, although the use of proxies is pretty much in terminal decline at least as far as bypassing blocks such as the region restrictions applied by big media sites. A proxy or VPN server operates in a very similar way fundamentally by acting as a buffer to forward and receive requests to hide your location. This means that whatever web sites you visit they will only see the proxy/VPN and never your real location additionally your ISP will only log the visit to the proxy not the end destination.

However a VPN server crucially adds an additional layer – that of encryption which protects the data being transmitted and is almost impossible to detect. The problem is that nowadays even the most cleverly configured proxy server is fairly easy for web sites to detect, if the site enforces region locking it will normally just block access from a proxy. A VPN though is much more difficult to detect and most sites cannot directly detect the use of a VPN although they can use other methods. So if you want to hide your identity from a snooping government, or want to watch Hulu from outside the USA, you’re going to need a VPN service not a proxy. Ignore the hundreds of free proxies available online, most are completely useless now and are often used to steal user credentials and passwords!

A VPN (virtual private network) service is somewhat more sophisticated than a proxy so will need an additional service to make it run. A proxy can be used simply by configuring it’s settings in most standard web browsers but that’s because it merely transparently forwards and receives data without protecting it in any way. A VPN actually sets up an encrypted tunnel between the client and the server which obviously needs some software component to run. Most VPN providers will provide some client connection software to establish the connection on their PC, you can see an example in this video:

If you watch the whole video it shows you the difference between a proxy and a VPN, demonstrating the client software of a program called Identity Cloaker in action.

How Does a VPN Work on other Devices?

This is what often confuses people, after all how does your smart phone or tablet use a VPN after all it’s the platform that many of us use to watch TV or movies sites. You can of course still configure a proxy on these devices but as mentioned these are largely useless now and should be avoided unless you have a specific need (and technical knowledge). For most of these devices there are only two options – use an application specifically designed for the device to enable the VPN connection or use the device’s own operating system to establish the connection.

It’s generally better to use the second option, that is to manually connect to your VPn service using the devices own operating system. This for example is how you connect a VPN using an iPad in this post. All you need as you can see is the login and user authentication details, input them into the VPN configuration screen and save the connection. You can then enable the VPN connection whenever you wish.

It’s worth remembering what actually happens when you establish a VPN connection either through some client software or as illustrated on a computer tablet manually. Firstly an encrypted tunnel is established, making sure all your data is hidden from view, secondly your IP address is effectively hidden all web sites will only see the address of the VPN server. This is what allows people to bypass the myriad of region blocks on large media websites – the ability to use the IP address of the VPN server rather than your own. This is why most VPN services offer a range of servers in different countries so that you can use a UK one for the BBC iPlayer, a USA one for Netflix and Hulu and so on.

Escaping Region Locking from Media Sites

In the early years of the internet there were very few restrictions on what you could see and download. If you started a web browser in the US you’d get pretty much the same experience as someone who started in Cairo. Obviously there might be some variation in speed of course, but what you could see and do was almost identical.

That’s changed a lot now with the growing popularity of region locking and control. It started off fairly helpfully – your search engines would switch you to the appropriate location based on your IP address. This meant that if you were searching from London for electricians you wouldn’t get directed to results in Sydney which would obviously be useful. We’ve got used to this and it generally makes everyone’s life much easier.

However the use of region locking has extended greatly in the last few years, in fact any major web site will usually operate some level of control. Often it’s again beneficial, Amazon will make sure you go to the UK site, Costco will direct you to your local store and so on. However for many of the world’s biggest media sites it’s a much different story – region locking usually means region blocking.

Ever tried to access Pandora from outside the US? Well it doesn’t work, the wonderful music site is only accessible for those located in the US. Want to watch the BBC News, sorry if you’re outside the UK it’s not going to happen. Those are just two but the list is extensive, in fact it’s unlikely you’ll now find a large media site which doesn’t lock down access based on the location of your IP address.

It’s crazy when you think about it, a global communication medium deliberately trying to segregate and restrict our world. Worse too that in a time when many of us travel extensively, we are blocked and filtered at every turn when we’re online.

So What’s the Solution?
Well to take back control and stop being blocked you need to be able to control your IP address. Unfortunately for most of us that’s not possible, the IP address is assigned when you connect to the internet and there is no way of modifying it. You can of course modify your local address but that’s not important, region locking uses your external internet facing IP address.

However although you cannot modify your address, you can hide it by using VPN servers to protect your connection. If you connect to a UK VPN server for example, it will look as though your have a UK IP address and watching the BBC works without problems. You can use a US VPN to gain a US IP address for Netflix irrespective of where you actually are. Many firms have developed services to support this demand and the top VPN providers will allow access to a network of servers in different countries.

This means that although you cannot change your real IP address, you can hide it behind a VPN server. It gives you back control and neatly sidesteps the pervasive region locking and filtering which seems likely to keep expanding.

Further Reading – British VPN

Methods of Designating IP Addresses

There are a variety of methods that can be used to assign an IP address to a windows client. You can obviously assign directly by allocating a static address or by using a variety of other methods such as ARP, BOOTP or DHCP. The main methods will be discussed briefly below:

Static Configuration
The majority of network enabled devices including Windows computers can be assigned an IP address manually. This is normally allocated in the TCP/IP properties on windows machines and in the network configuration options on other devices like games consoles and media streamer for instance. It is important that although any address can be allocated here, it should be the correct address for the network configuration. You should also ensure that the IP address is unique on that network,otherwise network connectivity will be affected for both clients with duplicate addresses.

RARP and BOOTP
Two of the main options for assigning an ip address dynamically are Reverse ARP (RARP) and the Bootstrap protocol (BootP). RARP requires a server which maintains a list of hardware addresses and a pool of IP addresses to be allocated. The server would normally be contacted via a broadcast initiated from the client before an ip address is allocated and then assigned to the hardware id. There are often issues with this method of address allocation though:

  • Clients broadcasts will sometimes not reach the RARP server. This might be for various reasons but is often to do with network topology, perhaps a router is incorrectly configured on the network. This can be resolved by configuring an IP helper address on any routers. It may also be simpler to configure the router to be a RARP servers depending on the model.
  • RARP server does not have an IP address which corresponds to the client’s hardware address.

Mostly due to various inefficiencies RARP is rarely used in modern networks and is usually replaced by the more sophisticated BOOTP to assign IP addresses.  This runs over UDP and sets up a port for client requests and another port is assigned for server responses.   The response from the BOOTP server actually contains additional information such as the address of the local gateway.   BootP suffers similar problems to RARP mainly to do with network connectivity.  UDP often suffers connectivity problems and the routers should be configured to allow UDP traffic and without interruption,  other problems can be with access lists filtering out the UDP ports.

Further Reading

Protocol Verification by Proxies

It’s worth noting in any environment which actively uses proxies, that circuit level (generic) tunnelling such as those use by SOCKS and SSL tunnels will normally allow any protocol to be passed through the proxy. The implication of course is that the proxy doesn’t understand the protocol merely passes it on, which means also that the server cannot verify what is happening at the protocol level. This can be a dangerous situation especially if the proxy offers a gateway to internet or external traffic from the internet.

For illustration, any SSL tunnelling protocol can usually tunnel any TCP based protocol – so it could actually be used to telnet directly into the server. There are some huge dangers to allowing any server to transport protocols like this with no consideration to the operational requirements. It’s like leaving a huge back door to your network unless it is properly managed. There are of course options to control these protocols and one of the simpler is to restrict tunneling based on specific ports. So you would allow 443 open to allow HTTPS traffic, 563 for News and maybe 636 for secure LDAP. You’d have to extend this list to consider any other application or protocol requirements needed such as Windows Active Directory or Remote Access.

It works, is simple to implement however the reality is it’s not that secure. The well-known ports are only recommendations and there’s nothing to stop protocols being used on non-standard ports although of course this could cause issues in receiving the data if the servers are not configured to listen on these ports too. This VPN solution discussed on this page called Identity Cloaker which is used to access US versions of Netflix tunnels on a non-standard port for SSL traffic and allows the user to switch to any port.

This means you’ll be left with the unenviable situation where you’ll suspect dangerous traffic is being transported on a non standard port. This method of control means you’ll end up breaking other vital services if you attempt to block the port too. It’s not a long term solution although it is something that can be implemented whilst you try and create something more sophisticated.

The most efficient solution of course is to ensure that the proxy server can verify the protocol it is transporting. Therefore if someone is using non-standard ports or tunneling using a banned protocol or attempting to use a fake IP address like this for example then the proxy wil be able to highlight this issue. Once you have this awareness you can expand the functionality of proxy by building in more intelligence. The server can be used to identify common misuse and external attacks including attempting to use SSL to tunnel their terminal connections via Telnet.