1. A personal perspective on the evolving Internet
and Research and Education Networks
Bill.St.Arnaud@gmail.com
Latest Version February 15, 2010
--Draft—Draft—Draft-
For discussion purposes only
Abstract
Over the past few years the Internet has evolved from an “end-to-end” telecommunications
service to a distributed computing information and content infrastructure. In the academic
community this evolving infrastructure is often referred to as cyber-infrastructure or
eInfrastructure .This evolution in the Internet was predicted by Van Jacobson several years ago
and now seems readily evident by recent studies such as the Arbor report indicating that the bulk
of Internet traffic is carried over this type of infrastructure as opposed to a general purpose
routed Internet/optical network. This evolving Internet is likely to have profound impacts on
Internet architectures and business models in the both the academic and commercial worlds.
Increasingly traffic will be “local” with connectivity to the nearest cloud, content distribution
network and or social network gateway at a local Internet Exchange (IX) point. Network topology
and architecture will increasingly be driven by the needs of the applications and content rather
than as general purpose infrastructure connecting users and devices. As a consequence the need
to deploy to IPv6 addressing or ID/loc split may be superseded by DNS-type such as laye r 7 XML
routing. This new Internet will more easily allow the deployment of low carbon infrastructure and
create new challenges and opportunities in terms of last mile ownership and network neutrality.
Customer owned networks and tools like User Controlled LightPath (UCLP) and Reverse Passive
Optical Networks (RPON) may become more relevant in order to allow users to connect directly to
the distributed content and application infrastructure at a nearby IX. Research and Education
(R&E) networks may play a critical leadership role in developing new Internet business strategies
for zero carbon mobile Internet solutions interconnecting next generation multi-channel RF mobile
devices to this infrastructure using “white space” and Wifi spectrum. Perhaps ultimately these
developments will lay the foundation for a National Public Internet (NPI) where R&E networks
deploy transit and/or internet exchanges in smaller communities and cities to support distribution
of content and information from not for profit organizations to the general public.
Executive Summary
The Internet appears to be once again is enabling another major wave of innovation. Although
compared to past transformative changes in the Internet such as the invention of the web or
packet switching this new wave to date has received very little public notice or attention. This
lack of awareness is surprising given that investment by private sector companies and
government in this transformative technology now rivals what the carriers are investing in
traditional backbone and last mile Internet technology.

2. This evolution of the Internet was predicted several years ago by an Internet pioneer Dr Van
Jacobson [VAN JACOBSON] where he argued that network engineers and researchers are in
danger of being stuck with a tunnel vision of the need for an ”end-to-end” network much like a
previous generation of engineers and researchers were trapped in thinking that all networks had
to be connection oriented. Dr Van Jacobson’s predictions have recently been vindicated by data
from the Arbor [ARBOR] study showing that over 50% of data on the Internet now comes from
mostly content and application providers using distributed computing and storage infrastructure
independent of the traditional carrier Internet backbones.
Note that end-to-end network should not be confused with the end-to-end principle established by
MIT researchers Reed and Clark where that, whenever possible, communication control operations
should be defined to occur at the end-points of the Internet or as close as possible to the resource
being controlled.
Up to most recently the text book model of the Internet was for businesses and consumers to
access the internet through a last mile provider such as telephone or cable company. Their traffic
would be sent across the backbone to its destination by an Internet service provider. This model
worked reasonably well in the early days of the Internet but as new multimedia content such as
video and network applications evolved the current model failed to provide a satisfactory quality
of experience for users in terms of responsiveness and speed. As a result a host of content,
application and hosting companies invested in something for the purposes of this paper I have
collectively labeled as a Application Content Infrastructure (ACI) that complemented and
expanded the original Internet through the integration of computing, storage and network links.
In the academic world these developments have been underway for some time and the resulting
infrastructure is often referred to cyber-infrastructure or eInfrastructure. As in the commercial
world it is creating profound changes in how we think about networking as a generalized end-to-
end telecommunications service connecting users or institutions to a facility that is seamlessly
integrated with distributed computing and storage delivering specific applications or data
solutions for targeted communities. Examples include Optiuter, Teragrid, LHCnet, etc
These investments by ACIs are quite distinct from that made by the networks that transport traffic
across the Internet, even though both they use many of the same technologies such as optical
networks, routers, etc. Rather, ACIs are integrated networks and other associated infrastructure
used by application and content providers used to host, cache and distribute their services.
Fundamentally ACIs are about the transport of content and applications rather than bits.
Examples of ACIs include large distributed caching networks such as Akamai, cloud service
providers such as Amazon and Azure, Application Service Providers (ASPs) like Google and Apple,
Content Distribution Networks (CDNs) such as Limelight and Hulu, and social networking services
like Facebook and Twitter. Many Fortune 500 companies like banks and airlines have also
deployed their own ACIs as an adjunct to their own private wide area networks in order to
provide secure and timely service to their customers. Most major content and application
organizations have contracted with commercial ACIs or have deployed their own infrastructure.
ACIs also allows the content provider to load balance demand, so that traffic in regions
expressing excessive loads can be re-directed to nodes where there is spare capacity.
The end result is that with very little fanfare the Internet has been transformed so much so over
the past decade that virtually all major content and every advanced application on the
Internet is now delivered over an ACI independent of the traditional carrier Internet
backbones.
This enormous investment in ACIs have resulted in major benefits not only for the content and
hosting companies but also have significantly improved Internet performance as perceived by the

3. end user. In many ways they have also relieved the major carriers from having to make major
investments in their own backbone and last mile infrastructure. These benefits and market drivers
for the development of ACIs can be briefly summarized as follows:
(a) They provide for faster and more responsive Internet experience for users as they
allow a organization’s content and services to be largely delivered locally over the ACI
rather than over a long distance end-to-end network with all the attendant issues of
latency, congestion and packet loss;
(b) They can deliver their data and services much cheaper than across traditional
Internet because reliability and redundancy can be achieved in many different ways
through distributed computing and storage at the application layer rather than the
network layer; and
(c) They enable new Internet business model where revenue is not related to the
number of bits a user downloads but linked to the specific application such as
advertising, product purchase and eventually such things as energy savings and carbon
offsets.
There is growing evidence that ACIs now constitute one of the fastest growing parts of the overall
Internet eco-system and represents a substantial, if not the largest, portion of the investment by
companies in the Internet itself. In some cases their investment in Internet infrastructure equal or
outweigh those of the carriers and ACIs now girdle the globe and rival the carrier’s infrastructure
in terms of scale and complexity. This major growth in ACIs is also supported by recent study
done from Arbor Networks that demonstrated that the majority of Internet traffic is now being
generated by ACIs directly at major peering points. This traffic from ACIs is continued to expect
to grow significantly faster over time versus than traditional Internet traffic and will largely
dominate traffic patterns and network architecture in the coming decades.
Some people mistakenly think ACI provide a "privileged" access channel for major content and
application companies. In fact the just the opposite has happened. Many companies now
specialize in providing third party ACI services which results in greater competition and reduced
costs. Content and application providers now have much greater number of choices in how they
want to deliver their product and service to the end consumer.
As a result ACI also helps lower the barriers for small companies and start-ups as well, enabling
them to grow and compete. It is near-impossible for small providers to quickly deploy and
maintain additional servers during a wave of sudden popularity; at the same time, over-
provisioning and maintaining infrastructure up-front can be very costly. ACI helps solve this
dilemma - small companies can work with third-party ACI providers, and "scale" up their services
efficiently and cost-effectively.
It is not only small businesses but also academic research and education projects that benefit
from third party ACI facilities. A good example is the distribution of HDTV video from the ocean
floor with project Neptune [NEPTUNE] and new Federal government services such as "apps.gov"
[GSA]. In both cases it would be cost prohibitive for a university or government department to
deploy the thousands of servers and purchase Internet access to deliver these services. More
importantly according to Internet 2 [INTERNET2] cost savings of up to 40% on Internet transit
traffic are possible by connecting to ACIs at various Internet Exchange points.

4. The same ACI drivers of enhanced customer experience and low deployment cost have driven the
market for clouds and ASPs. More importantly because of the low deployment cost of ACIs
application and cloud providers can focus on business models where revenue is based on the
service provided rather than charging by bit of data transferred across the network. This is a
fundamental difference in the view of the world compared to end-to-end Internet carriers. Carriers
come largely from a centuries old telephone view of the world where bandwidth was expensive
and had to be carefully rationed by charging users by the minute or by the bit. But the advent of
low cost, high bandwidth optical networks and new data architectures enabled by ACIs allows
them to effectively ignore the cost of bandwidth and distance and instead focus on the value of
the actual service in terms of functionality and ease of use.
New promising business models demonstrate that revenue can be obtained from other value add
services such as cloud computing and eCommerce solutions such as SalesForce. On the horizon
other promising business models for ACIs can be made from energy reduction and the sale of
carbon offsets. For example two recent papers from researchers at MIT [MIT] and Rutgers
[RUTGERS] indicate that a ACI can save up to 35% and 45% respectively in energy costs by
moving data and storage to sites in the network with the lowest instantaneous energy costs.
Some organizations are also investigating how carbon offsets can be used to pay for the
deployment of this type of ACI which in a small way address the challenges of climate change.
ACIs in many ways are dis-intermediating many of the traditional end-to-end telecommunication
services offered by carriers. As the ACI infrastructure reaches ever closer to the end user it may
require new business models for the last and undoubtedly will result in renewed debates on
network neutrality. This will be especially true as the mobile Internet develops with a
corresponding infrastructure to deliver ACI to these devices. As ACI does not require an end-to –
end wireless network, offering wireless data locally through WiFi or new “white space” spectrum
may become the norm.
Background
When the Internet was first conceptualized and implemented in the 1970s and 1980s it was
largely seen as an “end-to-end” network for data similar to the telephone voice network which
was then the dominant form of networking. But instead of making telephone call connections it
was used for the transmission of packets between connected computers.
The telephone network by its very nature is an “end-to-end” network as it is necessary to setup
connections between caller and receiver who may be separated by thousands of miles. The
telephone companies invested billions of dollars building global networks to allow users to make
calls to virtually any place on the planet where there was a telephone receiver.
Although the Internet used a different network technology i.e packets rather than circuit
connections, the network topology was much the same in that it was assumed users (or more
specifically their computers) would want to communicate with other user’s computers and or
servers which could be many thousands of miles away. For this reason it was easy to adapt the
Internet as an overlay network over the existing telephone infrastructure as they both had to
reach the same homes and businesses around the world.

5. The original Internet was made up of several different components and identified as “last mile”,
“regional Internet Service Provider (ISP)”, Internet Exchanges (IXs) and “backbone ISPs” as
illustrated in the following diagram:
The link from the user’s home or residence is identified as the last mile and is usually provided by
the local telephone or cable company. In some cases the last mile may be a wireless link or even
use optical fiber. The last mile usually terminates at the regional ISP’s hub or aggregation point.
In many cases these days the last mile provider is the one and the same as the regional ISP, but
this is not necessarily the case in all jurisdictions. From the regional ISP’s hub or integration point
the Internet traffic is carried by the regional ISP to a local or regional IX. IXs are located in just
about every major city in North America. It is at the IX where the regional ISP exchanges traffic
with one or more national backbone carrier ISPs. The backbone carrier ISPs carry the traffic to
another IX elsewhere in the county or around the world where another regional ISP then carries
the traffic across the destination last mile to the designated recipient.
The big disadvantage of this original architecture is that all traffic from a content company or
application must transit all these links and attendant probabilities of congestion and/or packet
loss. Any “end-to-end” network whether it is the public Internet or telephone system has also an
embedded liability as it is extremely expensive and technically challenging to build a reliable and
robust end-to-end infrastructure of any kind. Each link and node in the network can be a single
point of failure or congestion, of which the loss of any one can disrupt thousands of phone calls or
Internet communication sessions. This poses a major engineering challenge and the solution is to
have many redundant paths across the network with sophisticated routing and switching protocols
to quickly re-route traffic in the advent of congestion or failure at any node or link.
Application Content Infrastructure (ACIs) on the other hand are specialized infrastructure that
provides advanced services to the global Internet community that are delivered locally. This
avoids all the attendant problems of packet loss and congestion on the backbone network (and
also in the last mile). Invariably ACIs connect to the public Internet at the IXs or at private
peering points located in most cities. Their development was driven by several factors, but
primarily it was to enhance the user’s Internet experience, especially when there is significant
congestion and packet loss in the last mile networks. This new evolving model is shown in the
following diagram.The important thing to note about ACIs is that their service is built around
applications and content and not simply the transport of bits and packets across a network.
The use of distributed computing and storage to support their advanced applications allows ACIs
to keep content local. For example when a user connects to Facebook their traffic is carried over
their last mile cable or telephone broadband connection to the closest public Internet Exchange
point or private peering connection usually located in the same city. Their traffic is then handed
off to the Facebook ACI where a copy of the user’s home page is stored. As a user updates or
adds comments to their page their posting is distributed simultaneously to their Facebook’s
friends homepages over the Facebook ACI. Other than the last mile connection virtually none of
their communication on Facebook is carried over the public end-to-end Internet. More importantly
as perceived by the user all Facebook communication effectively stays local, despite the fact that
the user may Facebook friends scattered all over the world.
ACIs are not the same as private IP networks, often referred to as Virtual Private Networks (VPNs)
which are extensively used by Fortune 500 companies, governments and other large organizations

6. for internal traffic purposes. Some ACI networks are global in scale and rival the carrier public
Internet networks in terms of size and complexity. Good examples of such large scale ACIs
include Microsoft MSN’s network, Amazon’s EC2 cloud, Google’s Gmail and YouTube infrastructure
as well as Level3’s Content Distribution Service. Many Fortune 500 companies like banks and
airlines have also deployed their own ACIs as an adjunct to their own internal networks in order
to provide secure and reliable service to their customers.
It is important to note that ACIs constitute a lot more than just network links and routing nodes
as is typical public carrier Internet network. A major, if not the largest component of most ACIs is
the vast array of computing and storage devices that are located at various nodes throughout the
ACI infrastructure. As well the network links and elements tend to be more tightly integrated into
the ACI infrastructure of computing and storage infrastructure rather than being treated as an
abstract and independent network service. This allows for more creative and systemic solutions in
terms of reliability and load balancing.
The other distinguishing feature between ACIs and the public Internet is the number and types of
routes that advertised by either network. Generally because they operate at the application and
service layer ACIs only advertise a very small set of the Internet routing table. Instead they
depend on Domain Name Service (DNS) processes and content routing for the forwarding of
traffic. The public Internet, on the other hand must carry the full routing table (over 400,000
routes) because its primary mandate is to provide end-to end service. As a result the routing
infrastructure for a public Internet provider tends to much more complicated and expensive than
that of ACIs. However ACIs generally have made a much larger investment in storage and
computational resources in order to provide the services they offer.
The growth of ACIs is expected to continue dominate the future Internet. Many organizations still
operate legacy applications on their own servers such as e-mail, web hosting, etc. Increasingly,
these organizations are moving these applications to ACIs as the costs of maintaining their own
servers and applications continue to escalate. As well many organizations have come to discover
that ACIs provide their customers and users with much faster and responsive Internet experience
then if the same service was delivered over the traditional Internet. Many are now contracting
with commercial ACIs to deliver their content and/or applications. Several of the large carrier
Internet providers like Level3 offer such a service. In today’s marketplace offering your
customers a faster performing service can provide companies with a significant competitive
advantage.
In fact some pundits claim that if we were to build the Internet from scratch we would rarely need
“end-to-end” communication services as most application such as e-mail, social networking etc
can easily exist in the “cloud” supported by ACIs rather than being an end-to-end service. Even
supposed “end-to-end” Internet applications like voice or video can easily supported by such ACI
services. One of the great Internet pioneers –Van Jacobsen [VAN JACOBSEN] has been doing
considerable research in this field and there is a famous YouTube video of his vision for the Future
Internet along these very lines. His vision of the future Internet is essentially a global network of
integrated ACIs and that end- to-end networks essentially reflect yesterday’s thinking.
The Major Components of ACI eco-system
Globally there are now hundreds of companies who provide some or all of these ACI services. As
mentioned previously there are also many third-party ACIs like Akamai and Limelight who
integrate these services in various ways and provide them to application and content companies.
There are also companies who specialize in delivery of medical, education and government
content and applications. In addition, ACI are in some cases wholly owned by a content or

7. application company as in the case of large organization such as Google, Microsoft, Amazon, and
others. Rather than leasing services from a third-party, they build their own infrastructure, and
then either own or lease fiber to connect their facilities together.
While these companies do not provide last-mile service to end-users, some last-mile providers,
such as Verizon, are also deploying their own CDN and other ACI.
Hosting and Serving Facilities
Hosting and serving facilities are companies that have built large data centers around the world
where they lease space to content and application companies who want to deploy their own
servers. Sometimes these hosting and serving companies also deploy their own severs and lease
them to content and application providers. Generally the connectivity between separate facilities is
arranged independently by the content and application provider. Many of these facilities are also
collocated with IXs around the world. Examples of commercial data center and hosting providers
include Savvis, QualityTech, Opsource, Switch and Data, Equinix , Rackspace, etc
Governments and universities also take advantage of hosting and serving facilities. For example
the General Service Administration operates 8 data centers [GSA-CENTER] throughout the US.
Content Distribution Networks (CDN)
As more and more organizations started distributing content and applications over the Internet,
the cost of deploying and managing servers in remote data centers became quite burdensome.
This was particularly true for smaller media and content companies. In response several
companies started deploying what are called Content Distribution Networks (CDNs) where they
undertook complete responsibility for the distribution, replication and load balancing of content
and media to various IXs and other interconnection points throughout the Internet. The
quintessential examples of CDNs are Akamai and Limelight. There are over a dozen companies in
this space including players such as CDNetworks, Edgecast, Voxel, Mirror Image, Amazon
Cloudfront, BitGravity, CacheFly, Edgecast Networks, etc
As with the hosting companies there are many content distribution networks that are available for
smaller content organizations and education or medical applications. Examples include Inuk
Networks [INUK] which distributes educational video in the US and the UK
Cloud service providers
While CDN facilities are useful and cost effective for distributing web pages and streaming video
they could not address the need for computing intensive applications or storage. "Clouds" were
developed to address this need. Clouds are essentially a global ACI facility of tightly coupled and
interconnected computing and storage resources. An entire separate paper could be written on
clouds but generally they are broken down into three major applications areas:
• Infrastructure as a Service (IaaS)
• Platform as a Service (PaaS)
• Software as a Service (SaaS)
Infrastructure as a service is cloud ACI where users can configure their own virtual computing
infrastructure and deploy their own applications. The Amazon EC2 cloud is the prototypical
example of this type of cloud ACI. Platform as a Service, on the other hand, removes much of the

8. complexity of creating virtual machines and allows users to focus on deploying an application.
The Microsoft Azure is built along these lines.
The big advantage here for application developers is that they can scale their applications without
having to invest in expensive computer servers and hardware. Users can add and subtract
computation and storage services as they need and they only pay for what they use. It
significantly reduces the cost of implementation as time and resources do not have to be spent
configuring physical servers and systems.
In addition to the big name clouds like Amazon Web Services, Google App Engine, and Azure
there are many dozens of other cloud ACI facilities including IBM, 3Tera, Netsuite, Joyent,
Rackspace
Software-as-a-service is a much more specialized type of cloud service. Some companies provide
particular applications that are are offered as services for users to access online, rather than to
download to their computer as client software. SalesForce being the poster child for this type of
cloud. These types of cloud services allow organizations to offload their labor intensive servers
like e-mail and word processing to the net and make these services easily accessible from
anywhere around the world.
Again, as with content networking there are several clouds being deployed by the research and
education community such as NASA's Nebula [NASA]
Internet Exchanges (IX)
The key component that ties together hosting companies, CDN facilities and clouds into a
collective ACI service is the Internet Exchange (IX) point. They often serve a dual purpose:
(a) This is where media, content and application companies connect to third party ACI
facilities for distribution of their product; and
(b) This is where ACI facilities connect with last mile access providers for delivery of the
content and applications to the consumer.
There is a whole range and variety of IXs from large public facilities such as the Peering and
Internet Exchange (PAIX) [PAIX] in Palo Alto California and the Amsterdam Internet Exchange
(AMS-IX) [AMS-IX] to smaller private facilities called peering points where a last mile provider
may directly interconnect with an ACI under terms and conditions that are of mutual benefit to
both parties. Backbone and regional ISPs also connect to these IXs in order to reduce their costs
of carrying Internet traffic as well.
There can be dozens if not hundreds of media, content and applications companies at any given
single IX. As well there may be a large number of backbone and regional ISPs. For example the
Amsterdam IX has over 345 connected organizations while the PAIX in San Jose has over 75
connected organizations.
The IX can be thought of as a large cross connect switch that allows all these parties to exchange
traffic with each other. In most cases this is done on what is called a settlement free basis i.e.
there is no exchange of money to transmit or receive traffic. The only exception is between
backbone and regional ISPs who historically pay for the exchange of traffic between each other.
As well several R&E networks such as BCnet in British Columbia, KAREN in New Zealand and
UNINETT in Norway provide transit or internet exchange services for smaller communities and
cities which are too small to support a commercial IX.

9. There are several hundred IXs around the world. A reasonably comprehensive list of these
exchange points and the connected ACIs and Internet backbones can be found at
http://en.wikipedia.org/wiki/List_of_Internet_exchange_points.
How ACIs improve delivery of online services
Content and application delivery is a critical issue for many media and information companies. In
today's Internet, traffic volumes can vary widely and unexpectedly in what are called "flash"
events. For instance, President Obama's inauguration generated over 7 million live, simultaneous
video streams delivered via Akamai's infrastructure, with total traffic in excess of 2 terabits per
second [AKAMAI]. Such huge traffic events can also happen unexpectedly, leading a given
services' traffic to surge in a short period of time.
In the past content and application providers would have to deploy their own server infrastructure
and often do a worst case over provisioning in anticipation of the fluctuating traffic volumes on the
Internet. This approach was also suspect to single points of failure from any outage on the
Internet which would make the site completely unreachable.
By locating servers that host the content in a highly distributed manner at various data centers
around the global Internet close to Internet users, content and application providers could insure
that there was no single point of failure. If there was a network outage at some point in the
network, content and applications could still be delivered elsewhere in the global Internet. This
architecture also allows content and application providers to load balance demand for their
content and redirect requests for some applications or content to less busy servers somewhere
else on the network.
How ACIs Improve Consumer Internet Experience
One of the original and still critically important drivers for ACIs is to improve the user's Internet
experience by reducing latency and enhance application responsiveness. As most congestion
occurs in the last mile networks the impact of such congestion is significantly reduced by moving
content and applications as close as possible to the consumer. Keeping data local and thereby
reducing transmission time can significantly enhance a users’ perception of the quality of the
responsiveness of an application – even under congestion and packet loss. As long as there is
sufficient bandwidth to prevent egregious congestion in the last mile virtually all new innovative
content and applications can be delivered over a best effort last mile network if they originate
from an ACI. For example, only a short time ago it seemed unimaginable that one could deliver
HD quality movies over the Internet – but this is done routinely today by many services delivered
using ACI, such as Hulu, Netflix, and YouTube.
On the Internet, each time an application on an originating computer sends a packet of data to a
distant user it must be acknowledged by the recipient’s computer (who sends a return packet
called an (ACK) before another packet from the originating computer is sent. If the originating
computer does not receive this acknowledgement packet within a specified time it assumes it
hasn’t been lost somewhere in the network and it retransmits the original packet. This simple
“handshake” protocol ensures that no packets are lost in transmission across the network.

10. Packet loss also signals to the originating computer there may be congestion on the network.
Presumably the original packet got dropped on the floor somewhere where there was too much
data for the network to handle. If you look closely at Internet traffic you will see it is made up of
billions and billions of these types of data and acknowledgement packets back and forth across
the network.
If the originating computer does not receive an acknowledgement packet within a specified time
frame not only does it retransmit the original packet it also immediately slows down its
transmission rate. Then very slowly the originating computer speeds up the transmission of
packets as long as it continues to receive ACKs for every transmitted packet. But if at any time it
does not receive an ACK it immediately throttles back the transmission of packets.
The time to reach maximum transmission speed is very important and is directly related to the
distance between the originating and receiving computer – the greater the distance the longer it
takes to reach maximum transmission speed. There are a number of other factors that may
control transmission speed, but a basic rule of thumb is that for every doubling of distance, the
time to reach maximum speed of transmission is reduced by half. This speed reduction is
necessary because greater distances means it takes a much longer time to send packets and
receive ACKs.
However if there is a single packet drop then transmission is immediately reduced by up to 50%
and then slowly allowed to ramp back up. It is obvious that the probability of packet loss will
increase the greater the distance between the originating and recipient computers as there are
many network links and nodes to traverse – any one of which could be briefly overloaded and
cause congestion.
Packet loss can occur anywhere along a network from the user’s home, in the last mile and/or
across the carrier backbone network. However a study done by StreamCenter [STREAMCENTER]
indicates that most congestion occurs in the last mile.
Regardless of where the congestion and packet loss occurs in the network the effect is immediate
in terms of the responsiveness of the application as perceived by the user. The data transmission
rate is immediately reduced which is further aggravated if the originating and recipient computer
are a great distance apart. As applications incorporate more multimedia elements such as images
and video the impact can be quite dramatic in terms of the speed of which the data is received.
The same application if hosted a distant server will take orders of magnitude longer to download
then it is delivered locally given typical last mile congestion and packet loss. Internet users
experience this phenomena every day.
It's worth noting here that some have tried to make a false equivalence between last-mile
broadband providers' plans to prioritize traffic on their networks and the use of ACI to enhance
performance. This comparison is inapt on a number of levels.
As described above, ACIs enhance performance by serving content and applications at a location
that is closer to end-users, and by optimizing within an ACI providers' network for that purpose.
There is no upper bound on how many content and application providers could benefit from this
practice; in theory, everyone could deliver their content and applications from locations nearby to
broadband providers. This practice does not in any way impact the delivery of anyone else's non-
local traffic across the last-mile broadband provider's network, however.
In contrast, a broadband provider prioritizing traffic across its network is ultimately zero-sum. It is
impossible for everyone to take advantage of prioritization at the same time.. Prioritization means
that some packets are being moved to head of a network router's queue; in turn, some packets
are being moved further back in the queue. What's more, only the broadband provider itself has

11. the ability to prioritize traffic across its last-mile network, in contrast to the many different
providers that already provide ACI.
How ACIs Enable Innovation and create new economic opportunities
Numerous larger Internet companies have deployed their own ACI, in order to improve delivery of
their services. However, ACIs also have benefits through the ecosystem, including to start-ups
and small companies. Contrary to some perceptions ACIs are not a special channel or privileged
service for large media or application providers to deliver their content and services. Although
many large content and application companies have deployed their own ACI, there are many
companies who specialize in providing ACI as a third party service. It is these companies that are
enabling a great deal of new innovation on the Internet by lowering the barriers to entry and
enabling small innovative companies and not-for-profit organizations to scale their services as
required.
Prior to the development of ACIs any small startup with a creative idea faced a daunting challenge
in scaling up their business as demand accelerated for their product or service. The cost of
deploying hundreds of servers and paying for Internet transit in many cases was a significant
challenge for the business case for an Internet startup.
But with the advent of ACI facilities such as content distribution networks and clouds, the
economics have completely changed. In order to get off the ground, small startups don't even
have to invest in a single hosting server, network link or virtually any type of physical
infrastructure. By lowering the up-front costs for innovation, it lowers the barriers to entry.
For example, Amazon's Web Services is used by thousands of entities, who essentially "rent"
server and computing capacity from Amazon. They can operate their own web services on this
platform, without deploying or having to maintain their own infrastructure. Amazon started this
service, because it wanted "to specifically target small businesses such as software developers,
much as it catered to consumers and merchants who sell on Amazon's marketplace."
For instance, as discussed in the Wall Street Journal:"Jeffrey McManus operates a Web site called
Approver.com that lets people share and edit documents online. He started the site in August
2006 and funds it himself, using Amazon's online storage service to help him when there is a
usage spike on his site.
'The key thing about Amazon's storage service is that it's a pay-as-you-go service,' says the 40-
year-old San Francisco resident. "We don't have to make this giant investment up front.'"[WSJ]
Another good example is a small startup called Renkoo [WSJ] which developed a Facebook
application that lets users send virtual "drinks" to friends. Rather than deploying their own
infrastructure the company turned to Amazon to "rent" server computers and computing. This
allowed the small startup to scale their application as demand warranted. Renkoo's application
became a big hit attracting 30 million users.
As customer demand increase, start-ups can scale up their business incrementally using ACI. One
can think of the ecosystem of ACI as creating a ladder of investment, with virtual services like
AWS at the bottom all the way up to owning one's own ACI and fiber connectivity at the top.
Rather than having to make huge investments up front in order to start their business, start-ups
can invest over time as their service expands, moving up one rung at a time as they grow their
business.

12. Impact of ACIs on Innovation in Research and Education
Although ACIs are often associated with large media, content and application companies. They are
playing an increasingly vital role for delivering of educational material and supporting collaborative
research.
A good example of this approach is the joint Canadian-US research project called Neptune
[NEPTUNE]. Neptune is a multi-million dollar research project to deploy several hundred mile long
fiber network on the Pacific Ocean floor off the west coast of Canada and the US. The fiber
network will connect a host of undersea instruments in particular HDTV cameras and undersea
robots at various locations on the ocean floor. These cameras and equipment will not only be
accessible by researchers but also by students around the world across the global Internet. The
intent is to ultimately create a "virtual aquarium" where students and researchers can view and
interact with undersea aquatic life in real time.
Obviously the network demands of "virtual aquarium" are huge and so Neptune is using an ACI
provided by Akamai to deliver the hundreds if not thousands of real time video streams around
the world.
Many schools and universities have large archives of educational video which they would like to
distribute to students and educators around the world. But as we have seen previously the cost of
distributing content, especially video, can be very expensive as it requires a host of servers to
deliver multiple streams or downloading of content. In addition, it imposes a huge cost on both
the originating and receiving institution in terms of Internet transit costs. As a result many
institutions are now using ACIs such as Content Delivery Networks and clouds for delivery of their
content.
Other research projects are using ACI for crowd sourcing applications. For instance, there are
"Citizen Science" projects [CITIZEN] that engage with students and the general public to help
them identify exo-planets orbiting far off stars and catalog thousands of previously unknown
species in our forests and oceans. IBM has deployed an ACI facility called the World Community
Grid [WORLD] that taps into computing and storage resources around the world that is voluntarily
provided by many organizations. This ACI is contributing to research in finding new cures for
diseases such as Alzheimers and AIDS.
And of course computation clouds provided such as those provided by Amazon EC2 and Microsoft
Azure are having a huge impact on university computational research and burgeoning field of
what is called "eScience" or "cyber-infrastructure" Cyber-infrastructure is a specialized form of
ACI earmarked for the research and education community but accomplishes many of the same
goals in that it links instruments, computation and storage across the Internet to enable new ways
of doing science.
In the research and education sector good examples of global scaling ACIs include the distributed
computing and network infrastructure for the Large Hadron Collier (LHC) [LHCNET] based in
Geneva. Another example is the University of San Diego Optiputer project [OPTIPUTER]. In
many of these examples the ACIs have acquired their own fiber networks, as well as trans-ocean
wavelengths.
Some experts such as Dan Reed [REED], Vice President of Microsoft and formerly a member of
the President's Committee on the Advancement of Science and Technology (PCAST) claim that in
the future most university research will be entirely dependent on ACIs.

13. ACIs and Innovative Delivery of Government Services
The General Services Administration of the US government several months ago announced a cloud
services portal called "apps.gov". The portal allows agencies and department to develop
applications using third party commercial ACI facilities such as clouds and social media services.
As the "apps.gov" web site mentions it allows agencies and government departments to scale
their applications without having to invest in expensive computer servers and hardware. Users can
add and subtract computation and storage services at they need and they only pay for what they
use. It significantly reduces the cost of implementation as time and resources do not have to be
spent configuring physical servers and systems. Most importantly it provides enhanced service
quality to citizens who use these services.
How ACIs benefit the rest of the Internet Ecosystem
To understand the impact of ACIs are having on the evolving Internet one only has to look at a
recent study by Arbor Networks [ARBOR]. Arbor Networks – in collaboration with University of
Michigan and Merit Network presented the largest study of global Internet traffic since the start
of the commercial Internet at recent Internet engineering conference in October 2009.
The study included analysis of Internet traffic across 110 large and geographically diverse cable
operators, international end-to end carrier backbones, regional networks and content providers.
Results from the study show that over the last five years, Internet traffic has migrated away from
the traditional Internet core of ten to twelve backbone providers and now flows directly between
large content providers and last mile networks (and then to their end-users). As the study notes
these networks “have become so popular they are changing the shape of the Internet itself such
that many organizations and ISPs are directly peering with content providers rather than
purchasing transit from [backbone] ISPs”.
As a result the costs of Internet transit have dropped dramatically for all the parties in the
Internet eco-system. Major media and content now often deliver their content directly to third
party ACIs at major IXs where most traffic is exchanged on a settlement free basis. Regional ISPs
and last mile providers have then been able to reduce the amount of Internet transit they
purchase from backbone Internet providers by directly connecting to an ACI at an IX. Backbone
ISPs have also benefited in that their costs are also significantly reduced as they can also access
the same traffic delivered by ACIs rather than carrying it on their backbones.
ACIs have not only reduced Internet transit costs but they have also introduced a greater degree
of competition. A content and application provider now has many different ways of delivering their
product to end users. Organization with a few very small applications and with a limited market
can still choose to host their own servers and purchase Internet transit from a regional or
backbone ISP. Alternatively they can decide to deploy multiple servers at different hosting data
centers or move their content or application to any number of third party ACI companies. As we
have seen there are a plethora of such companies with a wide range of services and options.
ACIs have also significantly reduced the cost of Internet infrastructure by adopting new network
architectures that simply and reduce the complexity of the traditional backbone ISP facilities.
Again this benefits not only the ACI but reduces cost of Internet delivery to all members of the
Internet eco-system. Because ACIs provide load balancing and redundancy through the wide
distribution of servers at various nodes around the world they do not need complex and highly
redundant network facilities between these node. In previous studies the author has estimated
the cost savings of this type of architecture is a fraction of the cost of a traditional backbone ISP
infrastructure [ST. ARNAUD]

14. Future Evolution for ACIs
The distinguishing and unifying feature of ACIs is that they are an Internet infrastructure the
closely couples network links with compute storage and content to enhance user’s Internet
experience by delivering content and applications locally. ACIs are likely to be a major component
of the next generation of wireless and Internet networks as exemplified by research programs
such as GENI and the future “green” or “energy-aware” Internet. These programs generalize the
concept of ACIs as fundamental feature of the Internet through virtualization of the network,
computing and storage links.
The UCLP [UCLP] initiative extended this concept further by allowing users to deploy and manage
their own ACIs. Although UCLP is often confused with end-to-end circuit switched optical network
solutions as it also uses lightpaths, its original intended purpose was to allow researchers and
businesses to construct their own ACI (also referred to as an Articulated Private Network (APN))
and/or establish direct peering to one or more Internet Exchange points.
The Zero Carbon Internet
Since ACIs achieve redundancy and reliability at the application or content layer they are
inherently more robust and reliable than a traditional end-to-end network which can have many
points of failure. As a consequence they are ideally suited to an environment that has clean but
perhaps unreliable sources of power. As mentioned previously research by MIT and Rutgers
indicate the ACIs may reduce computing and storage energy costs by as much as 45% compared
to distributing and storing data over the public end-to-end Internet. These energy savings are
possible because it is a fundamental feature of ACIs to support distributed computation, data sets
and information where data and computation quickly moved from one node to another in the wide
scale infrastructure.
Research projects such as GreenStar [GREENSTAR] are deploying “green” ACIs where the
infrastructure is powered solely by renewable energy may play an important role in addressing the
challenge of climate change. A future Internet may be made up of multiple Green ACIs like
Greenstar whose revenue may be derived not only from traditional content and applications but
through carbon offsets or “gCommerce” applications under a national Cap and Dividend or Cap
and Reward program.[CAP]
ACI, Network Neutrality Challenges and Last Mile Networks
The growing dominant role of ACIs in the Internet infrastructure may pose future challenges for
regulators to insure that ACIs are able to peer and interconnect to last mile Internet service
providers and that their services are accessible and unconstrained to the ultimate consumer.
As more and more Internet Exchanges are deployed closer to consumers, the future Internet
regulatiry battle ground will be the last mile.
The inadequate investment in last mile infrastructure, particularly in North America, was one of
the principle drivers to deploy ACI facilities so that content and application companies could
ensure a certain degree of responsiveness and interactivity with users. As ACI facilities are

15. increasingly bypassing the backbone ISPs, despite the drop in Internet transit costs, the last
redoubt for the cable and telephone companies will be ownership and control of the last mile
infrastructure. But as discussed earlier it is in the last mile where most Internet congestion
occurs. Despite the huge investment made by ACIs in building out their infrastructure as close as
possible to the end consumer, it may not be sufficient to overcome the limitations to today’s last
mile infrastructure particularly with the next generation of high bandwidth applications and as the
carriers increasingly rely on traffic management techniques to address the challenges of
congestion.
Historically building a last mile infrastructure that was part of a national or global end-to-end
network was very challenging and you needed large companies to maintain and build this
infrastructure. But with the development of ACIs and the disintermediation of end-to-end network
the last mile is a stand alone element which does not require the complexity of previous end-to-
end facilities.
Concepts like “Customer Owned Networks” [CUSTOMER] are thereby easier to imagine and may
represent the future direction of how we deploy and manage last mile infrastructure. The concept
of RPON [RPON] was a further articulation of this idea to allow consumers to control and manage
their peering and multi-home with multiple ACIs at a nearby Internet Exchange point.
The recent announcement by Google to fund a number of last mile fiber to the home pilot projects
may be the necessary catalyst to enable these types of innovative last mile solutions.
ACI and Wireless Internet
The existing cell phone networks are architected very much along the lines of a traditional end-to-
end network as their primary application is voice. 3G/4G networks have very complex
infrastructures as they need to provide a reliable and continuous voice service with a mobile end
device. But as Internet data application start to dominate with the new generation of wireless
devices the same architectural changes we have seen with wire line networks may also become
evident with wireless networks as well.
It is likely that “mobile” ACIs will soon appear on the scene perhaps using the new “white space”
spectrum. This part of the spectrum as well as other bands such as those earmarked for WiFi and
WiMax may be ideal for mobile ACI services, as opposed to traditional end-to-end voice cell phone
networks which require a much higher grade of uncongested spectrum. New multi-receiver radios
that can share radio spectrum and mobile data platforms as exemplified by the iPhone and
Android will soon be able to connect simultaneously to several different wireless ACI and end-to-
end 3G/4G networks. This will also present new challenges to regulators to insure that these
mobile data platforms remain open and that users will have unrestricted access to mobile ACIs.
The Kindle or iPhone are early examples of the next generation of mobile Internet devices. But if
they remain locked to one 3G/4G vendor then their practicality and usefulness will be limited
especially as users start to download large content such as textbooks, videos, etc. And this is not
mention there are other challenges in terms of their archaic and concerted practice of using Digital
Rights Management tools to limit access and sharing.
R&E networks may be well positioned to offer wireless Internet services to students and faculty at
university and college campuses throughout the country. As universities and colleges migrate
student e-mail away from their own servers and as social networking tools like Facebook and
texting become the more prevalent way of communicating the need for high speed wireless access
will be critical both off and on campus.

16. Most R&E networks are uncongested and have already established peerings with major ACI
facilities it would be very easy for them to offer wireless ACI services using WiFi and “white space”
for the next generation of mobile devices as exemplified by the Android and/or iPad. Deployment
of “zero carbon” white space base stations at universities and schools in partnership with major
ACI companies could extend reach of their networks beyond the campus so students and faculty
could still access high speed data even off campus. It also may offer new revenue opportunities
in terms of both “gCommerce” and traditional applications for R&E networks and their ACI
partners.
ACI and IPv6 addressing
As more and more traffic originates with ACI facilities the bulk of Internet routing will be done
through DNS hacks. The DNS system was never intended to be a routing protocol, but there has
been little research on developing new routing protocols for this type of architecture other than
Service Oriented Architecture (SOA) XML style routing.
In any event the pressure for IPv6 routing may be somewhat mitigated by the development of
ACI. If ACIs continue to expand and dominate Internet traffic we may need to rethink the need for
IPv6 deployment or addressing techniques such Loc/ID split [LOC/ID]. Instead URI routing and/or
combinations of Delay Tolerant Networking (DTN) with late binding between DNS and IP
forwarding addresses may be a better long term solution.
An important area of research, such as that undertaken by Van Jacobson needs to be undertaken
to explore these issues. ACIs present host of unique challenges in terms of addressing, security
and information redundancy that yet to have been adequately addressed.
Role of R&E networks and ACI
Originally most R&E networks were also designed as end-to-end networks. But the advent of the
academic equivalent to ACI referred to as cyber-infrastructure or eInfrastructure is fundamentally
changing their role in the same way it is affecting traditional Internet backbone providers.
Increasingly R&E networks will have four critical roles with respect to ACI:
(a) Enabling the deployment of ACI or cyber-infrastructure by research discipline or
community of interest;
(b) Enabling researchers and educators to access commercial ACI services by direct
peering;
(c) Hosting transit or Internet exchange points in smaller communities and cities where no
commercial IX exists; and
(d) Perhaps ultimately deploying wireless ACI services as well as content and peering for
not-for-profit organizations to create a National Public Internet (NPI) in the same spirit as
National Public Radio (NPR) or PBS.
The deployment of specialized network facilities supporting cyber-infrastructure is well underway
in many countries and there are several large cyber-infrastructure facilities that integrate
computing, storage and networking. But the challenge for most R&E network operators in this
environment, as we have seen with Internet backbone operators, is to realize that their role as a
general network service provider is being supplanted by the cyber-infrastructure IT staff who
manage and control the cyber-infrastructure including its network components. As mentioned
previously the UCLP technology was developed specifically for this purpose to allow R&E network

17. operators to hand off control and management of portions of the network to the cyber-
infrastructure management team.
However R&E networks can play an important intermediary role with respect to providing
interconnections to commercial ACIs at IXs who provide a variety services to the R&E community.
This role will become critically important as major ACIs like Microsoft’s recent announcement to
provide free access to their computation cloud for the research community. The Internet 2 and
NLR content peering service is a good example of the value of this type of arrangement.
A more important role for R&E networks is perhaps to extend the benefits of ACIs to smaller
communities that do not have a carrier neutral IX. Several R&E networks already provide this
service such as BCnet in British Columbia, KAREN in New Zealand and UNINETT in Norway.
Deploying these IXs will also enable small not-for-profit content and application providers to
access ACI facilities in larger centers in order to distribute their content globally. R&E network can
play an important aggregation and intermediary role for these smaller players. Cultural content
organizations, in particular don’t have the clout and wherewithal to distributed their content
globally or even locally. Providing a service for this community may be ultimately be the most
important role for R&E networks leading to the concept of a National Public Internet.
National Public Internet
One of the recurring drivers for ACI is to overcome the limitations of the last mile infrastructure
by locating content and applications as close as possible to the consumer. While carrier neutral
IXs exist in most major cities they are very rare in smaller communities. Governments and not for
profit organizations who wish to distribute content, especially video, to these smaller communities
over the Internet face a serious challenge. While they can contract with commercial ACIs to
deliver their content to consumers in major centers where there is local IX this is not possible in
smaller cities or rural areas.
One possible important role for R&E networks is to extend commercial ACI services to these
smaller centers for the distribution of not for profit content to consumers. As broadcast
organizations such as PBS, CBC or TVO look to have their entire content accessible over the
Internet they have an obligation to the public to insure that it accessible by all citizens especially
in smaller cities and rural areas. Backhauling this content over commercial networks may impose
a significant burden to ISPs and huge cost to the content provider as we have witnessed in the
complaints in the UK by BBC’s Internet broadcast service. Using a commercial ACI service for
major centers in combination with R&E networks for smaller cities may provide a workable
solution. The commercial last mile ISPs are still responsible for delivering the content and remain
the customer facing organization, but the R&E network, in effect, becomes an extension of a
commercial ACI for the purposes of distribution of not-for-profit content.
The concept of a National Public Internet will probably generate considerable debate, but as we
have seen in the past the Internet revolution never ceases and with every new development there
will always be those who want to maintain the status quo of outdated business models. But we
cannot ignore the fact that the Internet has now become probably the most important vehicle for
the dissemination and sharing of information and content.
References
[WSJ] Mangalidan, M, "Small Firms Tap Amazon's Juice", January15, 2008, Wall St. On Line,
http://www.thestrategiccounsel.net/downloads/articles/amazons_juice.pdf

18. [NEPTUNE] Neptune Canada Outreach and Education Program,
http://neptunecanada.ca/education-outreach/
[GSA] Vivek Kundra, Federal CIO video speaking about new Federal Government "apps.gov" web
site, http://www.greenm3.com/2010/01/cloud-computing-open-source-containers-federal-govt-
new-appsgov-model.html
[INTERNET2] Internet 2 Content and Peering Service,
http://www.internet2.edu/pubs/commercial-peering.pdf
[GARTNER] "Gartner says Cloud market is expected to reach $150.1 billion in 2013.",
http://www.gartner.com/it/page.jsp?id=920712
[CROWE] Interview with President of Level 3 on future of CDN,
http://blog.streamingmedia.com/the_business_of_online_vi/2008/08/qa-with-jim-cro.html
[INUK] Inuk distance education video service, http://www.inuknetworks.com/universities.html
[PAIX] Peering and Internet Exchange Point now operated by Switch and Data, History and
background http://en.wikipedia.org/wiki/PAIX
[AMS-IX] The world's largest Internet exchange, http://www.ams-ix.net/
[AKAMAI] http://www.akamai.com/html/about/press/releases/2009/press_012009.html
[STREAMCENTER] Cataltepe, Z. Moghe, P., Characterizing nature and location of congestion on
the public Internet, 30 June-3 July 2003, p 741- 746
vol.2,http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1214206
[AKAMAI] Leighton, T., "Akamai and Cloud Computing: A perspective from the edge of the cloud"
http://www.akamai.com/cloud
[WSJ] Mangalidan, M, "Small Firms Tap Amazon's Juice", January15, 2008, Wall St. Journal On
Line
[FACEBOOK] http://blog.facebook.com/blog.php?post=262655797130
[CITIZEN] Citizen Science, Blog on many examples of how clouds, CDN networks, crowd sourcing
etc help students and citizens participate in scientific discovery http://citizen-
science.blogspot.com/
[WORLD] World Community Grid, http://www.worldcommunitygrid.org/
[LHCNET] Global network linking computers and databases for high energy physics,
http://lhcnet.caltech.edu/
[OPTIPUTER] Optical network linking computers, visulization servers, etc
http://www.optiputer.net/
[ST. ARNAUD] St. Arnaud, W., et al. "Architectural and Engineering Issues for Building an Optical
Internet", 1998, www.canarie.ca

19. [REED] Reed, D., "Outsourcing: Perhaps its time" January 14, 2009
http://billstarnaud.blogspot.com/2008/01/some-excellent-comments-by-dan-reed-on.html
[ARBOR] C. Labovitz, et al "ATLAS Internet Observatory 2009 Annual Report"
http://www.nanog.org/meetings/nanog47/presentations/Monday/Labovitz_ObserveReport_N47_M
on.pdf
[AccuStream] "CDN Market Growth 2006 - 2009"
http://www.researchandmarkets.com/reportinfo.asp?report_id=615197&t=d&cat_id=
[UCLP] User controlled LightPaths, http://www.uclp.ca/index.php?
option=com_content&task=view&id=80&Itemid=87
[RPON] Reverse Passive Optical Networks, Free fiber to the home, http://free-fiber-to-the-
home.blogspot.com/
[GREENSTAR] Building a zero carbon network, http://www.greenstarnetwork.com/
[CAP] Cap and Dividend versus Cap and Reward
[CUSTOMER] D. Slater, T.Wu, Homes with Tails, What if you could own your Internet connection,
http://commlaw.cua.edu/articles/v18/18.1/Slater-Wu-FINAL-12-3-09.pdf
[BABCOCK] Babcock school of Management textbooks for 160 students alone produces
45 Tons CO2 http://www.stewartmarion.com/carbon-footprint/html/carbon-footprint-stuff.html
[LOC/ID]
Author Bio
Bill St Arnaud was previously Chief Research Officer CANARIE Inc., an industry government
consortium to promote and develop information highway technologies in Canada, where he
worked for 15 years. He is Bill is currently pursuing pursing new opportunities related to his on-
going passion for Internet networking, especially in the area of developing network and ICT tools
to mitigate climate change.
At CANARIE, Bill St. Arnaud was been responsible for the coordination and implementation of
Canada's next generation optical Internet initiative called CA*net 4. Previously he was President
and founder of a network and software engineering firm called TSA ProForma Inc. TSA was a LAN/
WAN software company that developed wide area network client/server systems for use primarily
in the financial and information business fields in the Far East and the United States. Bill is a
frequent guest speaker at numerous conferences on the Internet and optical networking and is a
regular contributor to several networking magazines.