----------------------------------------------
Choosing a corporate Internet Service Provider
Martin Cooper <mjc@cooper.org.uk>
----------------------------------------------
The UK Internet industry's rate of growth isn't slowing down, although
market forces have been causing and are continuing to cause consolidation,
as small service providers are acquired by larger ones - often telecomms
suppliers looking to buy in Internetworking expertise to expand into that
market.
As the link between voice and data networking becomes increasingly blurred
by technologies such as Frame Relay ("FR") and Asynchronous Transfer Mode
("ATM"), services such as Real-Audio (tm) and Voice over IP ("VoIP"), and
the further liberalisation of the telecomms market leading to ISPs
diversifying into providing telecomms services as well as the reverse,
choice has been vastly increased. Inevitably this increased choice makes
selecting the right supplier to meet your needs that much harder.
In this article I intend to address the integrated ISP and Telecomms
service provision business by considering a small cross-section of the 67
members of the London Internet Exchange (LINX), the London Docklands based
Internet Protocol Exchange Point and Trade Association.
I have chosen to examine 9 ISPs, most of whom have been industry players
for several years in various guises (UUNET UK, PSInet UK, BTnet, GX
Networks, INSnet, VBCnet, Easynet and Netkonect), and a more recent market
entrant (Level3).
On the way to comparing UK suppliers I intend to consider the business in
terms of IP and telecomms network architecture, connectivity and hosting
services, value-added services, customer and network support, and Service
Level Agreements ("SLAs") / pricing policy.
Network architecture
--------------------
IP networks can be considered as part of the Open Systems Interconnection
("OSI") model in which they appear at layer-3 (also known as the "Network"
layer).
Such networks can be carried over a variety of OSI layer-2 (the
"Data-Link" layer) networks including IEEE 802.3 (Ethernet),
High-level Data Link Control protocol ("HDLC") or Point to Point
Protocol ("PPP") over X.21, G.703 or High Speed Serial Interface
("HSSI"), Frame Relay and Asynchronous Transfer Mode, the
Synchronous Optical Network ("SONET") and the Synchronous Digital
Hierarchy ("SDH").
There are a number of factors to consider in the selection of the
underlying layer-2 network upon which the IP network architecture
will be built.
- What speed do you wish to run trunk links at?
- Do you require resiliance?
- Do you want to be able to easily split the bandwidth into chunks?
- Do you need to define Quality of Service ("QoS")?
- Do you need to dynamically assign bandwidth?
- Do you need to specify a complex topology with multiple links and
complex routing?
[ Box 1 - START ]
Comparison of types of layer-2 network
--------------------------------------
IEEE 802.3 (Ethernet)
- Asynchronous but non-deterministic (CSMA/CD protocol)
- Broadcast
- Specifications exist for speeds of up to 10Mbits/sec
(Ethernet), 100Mbits/sec (Fast Ethernet) and 1000Mbits/
sec (Gigabit ethernet)
- Usually switched to allow some degree of Quality of Service
("QoS") to be provided
- Can be run over copper or fibre cables ("bearers")
HDLC/PPP over X.21, G.703 or HSSI
- Layered over X.21 and copper cables ("bearers") for relatively
low-speed (64 - 768Kbits/sec) use, or G.703 (overlayed with
G.704 if channelisation is required) for higher speed (2Mbits/
sec) use. HSSI links require special cables and are generally
only used over very short distances, but are capable of speeds
of up to 52Mbits/sec
- Synchronous
- Point to point
- Multilink PPP can load-balance over two parallel circuits
- Built-in error correction protocols
Frame Relay/ATM
- Optical Carrier ("OC") / Synchronous Transport Module ("STM") speeds
i.e. 155Mbits/sec, 622Mbits/sec, 2.4Gbits/sec
- Asynchronous
- Point to multipoint
- Expected to be run over reliable physical network or with higher layer
protocols overlayed, therefore limited error correction
- Network is conceptualised as a shared bandwidth frame/cell relay "cloud"
with multiple nodes
- Permanent and Switched Virtual Circuits ("PVCs" and "SVCs") with Quality
of Service ("QoS") parameters e.g. committed bandwidth, delay, burstable
bandwidth etc.
- Switching of small fixed size frames (125 bytes with FR) or cells (53
bytes with ATM) increases bandwidth overhead
- Can handle dynamic routing over multiple physical links with ATM Private
Network to Network Interface protocol ("PNNI")
SONET/SDH
- OC/STM speeds
- Synchronous
- Point to point
- Expected to be run over reliable i.e. optical, network, therefore
limited error correction
- Specifies breakout speeds for tributary synchronous data-streams
(channels)
- "Automatic protection and restoration" mechanism for fail-over to
redundant fibre if primary fibre fails
[ Box 1 - END ]
At the IP layer every entity that is connected to the global Internet must
have a globally unique IP address.
Global IP address space is assigned in chunks ("prefixes") with a specific
starting point in the global address space, and of specific length
("prefix length").
IP address space is allocated hierarchically from large chunks
("aggregates") to small ones ("assignments") to best match the Internet
routing hierarchy.
Routing of IP traffic on the Internet requires the distribution of
knowledge about the reachability and location of IP prefixes. On the
contemporary Internet this is done using the Border Gateway Protocol
("BGP") version 4.
IP networks on the Internet interconnect using BGP4 based on two broad
politico-commercial models, "peering" and "transit".
Peering is usually a non-settlement (i.e. non-retail) interconnection
between two networks for whom exchanging their respective locally
originated routes (and therefore traffic) is of approximately equal mutual
benefit.
Transit is usually a settlement (i.e. retail) interconnection between two
networks where one provides the other with non-locally originated routes
(and therefore carries traffic that is not destined for its own networks).
Most UK corporate ISPs use a combination of peering and transit to achieve
full global connectivity and use BGP4 configuration to "tune" this for
optimal routing.
Good networks are those that have the best (most optimal) routes to any
particular global destination, and alternate routes should the best routes
be temporarily unavailable.
Ideally an ISP of choice will be present at multiple geographically
distributed Internet Exchange Points ("IXPs") and have good peering
agreements with other ISPs at them. They will also have at least 2 transit
providers to get the routes they don't see via their peerings.
[ Box 2 - START ]
Possible characteristics of optimal routes
------------------------------------------
* Shortest path to destination network
* Highest bandwidth intermediate networks
* Least congested intermediate networks
* Most reliable intermediate networks
* Most secure intermediate networks
[ Box 2 - END ]
Selection of network hardware platform, duplication of key equipment,
telecomms infrastructure and power supplies, and diverse routing of
physical network links are of high value in developing a high-performance,
high-reliability network architecture.
Cisco hardware is used by most of the Internet for IP routing equipment,
and increasingly its Lightstream and Stratacom ATM hardware is gaining in
popularity.
Fore Systems has a good reputation for its cutting edge ATM technology as
does Cascade for its Frame Relay switches.
Nokia is widely respected for its SDH equipment.
Connectivity services
---------------------
Standard corporate ISP connectivity services fall into 4 main categories:
- Routed ISDN
- Frame Relay/ATM cloud
- Leased line
- Direct
Special corporate ISP connectivity services include:
- Pay per bit
- Committed bandwidth with burst capability
- Split national/international bandwidth
- Regional only transit e.g. UK, EU, or US only
- ISDN backup
- Dual connect with diverse routing
- BGP4 multihoming
Telecomms services include:
- Dark fibre lease
- SDH bandwidth
- Frame Relay/ATM cloud access
- FR/ATM Virtual Private Networks ("VPNs")
Hosting services
----------------
ISP hosting services usually involve one of:
- leasing web space on a shared server
- leasing a dedicated server
- leasing rack space and co-locating your own equipment
Value-add services
------------------
* IP and domain name registration
IP addresses are initially assigned by ISPs based on your predictions of
how many end-systems ("hosts") you will have on your network in 1-2 years.
Domain names are usually registered when you first connect and are most
often used to identify web services (www.domain) or email users (@domain).
* Domain Name Service ("DNS")
If you are unable to run your own primary DNS server to authoritatively
answer queries from the Internet about your domain name(s), your ISP may
be able to run it for you.
Most ISPs will provide secondary DNS for your domains as a backup and to
load-balance with the primary.
* Mail Exchanger ("MX") backup
This acts as a protective buffer for your primary MX when it becomes
available after a hardware failure or extended loss of Internet
connectivity, by load-balancing inbound email with the primary and
forwarding it on to the primary in a controlled manner to avoid
overloading it.
* Smarthosting
This avoids the need to run a local mail server to handle queueing and
retry for temporary failures to send outbound email to particular
destinations.
* USENET news
News is a series of discussion groups organised into hierarchies and
distributed between "peered" news servers by means of a flooding
algorithm.
News can either be provided as a "streaming feed" using the Network News
Transfer Protocol "NNTP") or by giving Network News Reader Protocol
("NNRP") access to your network to allow news reader software to read news
directly off the ISPs news server.
* Web caching
A web cache is a central "proxy" for directing requests to download World
Wide Web documents at, which retains local copies to reduce transit
bandwidth usage and latency for popular documents.
* Multicast/MBONE
The MBONE (Multicast Backbone) is a virtual network overlayed on the
Internet for carrying real-time multicast services such as streams of
audio-visual data.
* Network Time
The Network Time Protocol ("NTP") is a distributed clock update protocol
and can be used to automatically keep the time on your servers correct and
synchronised with each other.
Customer and network support
----------------------------
Most ISPs have a dedicated "new site" team who co-ordinate new customers'
service installations.
After this, support is usually provided through a combination of the
use of email and the 'phone for configuration changes and fault reporting
respectively.
24x7 support based on shift patterns is becoming increasingly common
although all ISPs should (at a minimum) provide a pager number for urgent
out of hours fault reports.
You should also expect all ISPs to be able to do:
- Advance warning of scheduled maintenance
- Notification of unscheduled maintenance owing to network problems
- Customer access-circuit outage notification
- Collection of bandwidth statistics on your access-circuit or
switch/router port
- Replacement of failed or faulty equipment with on-site spares or a 2-4
hour maintenance contract
Service Level Agreements ("SLAs") / pricing policy
--------------------------------------------------
SLAs and pricing are closely linked and it makes sense not to pay for
guarantees you don't need. It is worth considering exactly what level of
service you require and negotiating on price if you don't need the
standard SLA. SLAs usually cover areas such as network availability,
available bandwidth, and average latency.
Pricing of Internet services varies a great deal from one provider to
another. Different packages cost different amounts to provide and clearly
are perceived and valued differently by prospective customers.
It is always worth checking what is and isn't "bundled" into the price and
negotiating if you don't need particular elements of a package.
64K and 2M lines have become somewhat commoditised so don't expect to be
able to negotiate too much here. The UK industry does appear to be quite
polarised on price, from almost unbelieveably inexpensive, to
astonomically expensive, and curiously the quality does not always match
the obvious expectations at either end of the scale.
[ Box 3 - START ]
ISP leased line prices
----------------------
Leased line prices vary widely acccording to provider geography and
access-circuit availability. All price ranges are in UK pounds.
Bandwidth Approximate price range
--------- -----------------------
64K #6,000 - #9,600
128K #12,000 - #16,000
256-512K #17,000 - #31,000
2M #32,000 - #48,000
[ Box 3 - END ]
Players
-------
* UUNET UK
- 8 UK Points Of Presence ("POPs") (including Telehouse London)
- Peering at LINX and various US and European NAPs/IXPs
- Transit from UUNET US
- Duplication of all key functions
- Cisco and Juniper equipment
- 24x7 monitoring and telephone fault reporting
- Optional leased and managed router
- Can do pay per bit service
- Dual connection for resiliance with "Multi-connect" service
- Will do BGP4 multihoming
- Can do server hosting
- Try to be quite standard in their approach to service provision
- Usual value-add services
- Multicast routes carried internally
- Support for real-time services
- Good SLAs on network availability, available bandwidth and latency
- Most expensive UK corporate ISP!
* PSInet UK
- 8 UK POPs (including Telehouse and London hosting centre)
- Peering at LINX. PSInet US will do free private peering in US
- Transit from PSInet US
- Increasing ownership/lease of own dark-fibre network, so low cost of
ownership
- UK backbone network based on Frame Relay
- Cisco routers and Cascade Frame Relay equipment
- 2-4 hour fix for service affecting faults
- 24x7 pager for fault reporting
- SNMP customer equipment monitoring
- Service can be supplied with or without a router included
- Can do hosting at London hosting centre
- Will do BGP4 multihoming
- Usual value-add services
- Good SLAs on network availability, and available bandwidth
- Good value for money. Always prepared to negotiate on price
* BTnet
- 14 UK POPs (including Telehouse)
- Peering at LINX
- Transit from Cable and Wireless US, GTEI, Digex, UUNET US, Concert
(Europe), Level3
- Implement resiliancy wherever possible
- Aim to keep backbone network circuit loading below 60%
- Cisco router and Cascade Frame Relay equipment used
- Spares/support contracts handled internally within BT with strict SLA
- 24x7 monitoring and (full) support
- Option of purchasing router with connection
- Will do BGP4 multihoming
- UK only bandwidth service (will also do US or EU only if required)
- Usual value-add services
- No formal SLAs as yet (i.e. best efforts service), but they are due
"Real Soon Now"
* GX Networks
- POPs in London, Manchester and Cambridge
- Peering at LINX, MAE-East (US), MAE-West (US), D-GIX (Stockholm), and
AMS-GIX (Amsterdam)
- Transit from CERFnet, Verio and Ebone
- Indefeasible Right of Use ("IRU") on STM-1 to New York
- Transit/hosting swap arrangement with Concentric Network (US)
- Access to Fibernet's UK SDH network
- Access to Energis' UK Frame Relay network
- Data/hosting centre at Harbour Exchange, London Docklands
- ATM switches for providing managed ATM services
- Aim to keep backbone network circuit loading below 70%
- Cisco routing and ATM switching equipment, and Cascade Frame Relay kit
- "Cisco Powered Network" accreditation
- Spares holding for most equipment. 4 hour support contracts for major
problems
- Leased and managed router with most services
- Can still opt for collected (rather than delivered) service
- 24x7 monitoring and (full) support
- Will do BGP4 multihoming
- Usual value-add services
- "Flextra" staged bandwidth upgrade service
- "X-Frame 2000" 128K/2M IP over Frame Relay burst service
- "Headroom" split national/international bandwidth service
- Fairly flexible on providing special services
- Fairly good SLAs on network availability, and available bandwidth
- Has DTI telco licence
* INSnet
- UK POPs in London and Manchester (including Telehouse)
- Peering at LINX, MAE-East (US), D-GIX (Stockholm), and AMS-GIX
(Amsterdam)
- Transit from UUNET US and CAIS
- SDH capacity on Gemini transatlantic cable
- STM-4 SDH ring around London with STM-1 to Wimbledon and ATM overlayed
on this for easier bandwidth management
- SDH automatic protection and restoration and ATM PNNI routing for
resiliance
- Cisco IP and ATM equipment. Nokia SDH kit.
- Full set of spares at Wimbledon NOC including Power Supply Units
("PSUs") with various support contracts
- Aim to keep backbone network circuit loading below 65%
- 24x7 pager/mobile for fault reporting. Soon to move to 24x7 manning
- 4 hour fix for service affecting faults
- Customers get access to bandwidth graphs
- Use leased and managed or own router
- Will do BGP4 multihoming
- Usual value-add services
- Happy for customers to run real-time services over their network
- Will do special services if it can be done with a reasonable SLA
- Like with like pricing policy
- Has DTI telco licence
* VBCnet
- 8 UK POPs (including Telehouse London and Telecity Manchester)
- Lines to West Coast USA and Manhattan
- Peering at LINX, Manchester Network Access Point ("MaNAP"), MAE-West
(US), the Commercial Internet Exchange ("CIX"), and D-GIX (Stockholm)
- Transit from NACnet and Abovenet in the US
- Key equipment duplicated and distributed across POPs
- Some BSD UNIX servers used as routers. Moving towards standardising on
Cisco equipment.
- Aim to keep backbone network circuit loading under 65-70%
- Has on-site equipment spares holding
- 24x7 network monitoring, with telephone support for customer fault
reporting
- Will do BGP4 multihoming
- Can do server hosting at most POPs
- Usual value-add services
- Support for real-time services
- Very flexible on special/non-standard services
* Easynet
- London based (Whitfield Street and Telehouse)
- Peering at LINX (no US NAPs yet)
- STM-1 to New York. Connectivity to Easynet France (Paris).
- Transit from PSInet US, GTEI and Teleglobe in New York. Also Ebone for
European transit.
- Key equipment duplicated
- Some BSD UNIX servers used as routers. Moving towards standardising on
Cisco equipment
- Aim to keep backbone network circuit loading under 50%
- Has on-site equipment spares holding with some 2-4 hour equipment
support contracts
- 24x7 monitoring and telephone support for fault reporting
- Will do BGP4 multihoming
- Can do pay per bit service
- Can do server hosting
- Usual value-add services
- Multicast routes carried internally
- Fairly flexible on special services
- Has DTI telco licence
* Netkonect
- London based (Telehouse)
- Peering at LINX and MAE-East (US)
- Transit from UUNET US (via INSnet US)
- International circuits run over Gemini transatlantic cable system
(diverse routing/SDH automatic protection and restoration)
- Combination of Cisco routers, Extreme ethernet switches, with
Livingstone and US Robotics equipment used for dial-up
- Customers can use their own access-circuits to connect
- 24x7 telephone support for fault reporting
- Will do BGP4 multihoming
- Can do server hosting
- Usual value-add services
- Economical pricing
* Level3
- London based (including Telehouse)
- Peering at LINX and various US NAPs
- Transit from CERFnet via Level3 US
- SDH fibre rings used for resiliance in physical networks
- Multiple diverse fibre routes used in POPs
- All key routers and servers duplicated
- All POPs have spares for all equipment
- Customers can use their own access-circuits to connect
- 24x7 monitoring and (full) support
- "WECSS" system (planned for summer 1999) will allow customers to monitor
their own services and trouble tickets
- 2 hour Mean Time to Fix ("MTF") across all products
- Will do BGP4 multihoming
- Usual value-add services (primary DNS extra).
- Has a specials request process. Will try to support services if
possible, but prefer to have sound technical understanding of all
supported products
- Good SLAs
- Has DTI telco licence
Conclusions
-----------
Procuring ISP and telecomms services in the UK is quite a complex process,
particularly as the old distinctions between the two areas become
increasingly blurred.
It is likely to be of particular benefit to either try out the service(s)
you wish to buy (INSnet provide such a facility) or ask to speak to
satisfied reference customers.
The market for ISP services in the UK in particular is highly competitive,
and it is always worth trying to negotiate on price, particularly where
your requirements fall well within the standard Service Level Agreement
and you do not require all its provisions to be supported.
Thanks to all those from the UK ISP industry for their contributions
to my research for this article.
Martin Cooper works for the University of Cambridge Computing Service in
the Network Systems Group.