This blog post is made to help me study for my upcoming CS4226 exams. Lets find out how the internet works!

Autonomous Systems (AS)

An autonomous system is a network of interconnected routers that are identified by a unique AS number (ASN). They are controlled by a single administrative domain and use common routing protocols and policies.

The most commonly used protocol on the internet is the Border Gateway Protocol (BGP)

They are the fundamental units that make up the internet.

What is the architecture of the internet?

Architecture of the internet

The structure of the internet is a network of networks. At the center of all of this is a small number of well connected Tier 1 networks. These Tier 1 networks are the backbone of the internet.

Some content providers (EG: Google, Netflix, etc) connect their own private networks that connects themselves to the internet, often bypassing the Tier 1 and regional networks.

AS Topology

AS Topology

The routers within an AS does not have to be connected to each other directly. As you can see in the above diagram, AS X is separated by AS Y but they are still part of the same AS.

What does it look like?

You can see the current AS topology here: Internet Topology

The redder the color, the more connected the AS is to the rest of the internet.

The top 3 ASes (as of 2020) are as follows:

  1. Level 3
  2. Arelion
  3. Cogent Communications

Another interesting thing is that Singtel is actually within the top 20 at 16th place with AS number 7473. It also contains 5.39% of the world’s IP address.

You can see the rankings yourself here.

Internet Peering

Internet peering is a voluntary interconnection of administratively separate Internet Networks for the purpose of exchanging traffic between the customers of each network.

Why will they peer with each other.

There are various reasons why ASes would want to peer with each other.

Here are the list of some of the reasons:

  1. They have a customer-provider relationship
  2. They have discussed a peer-to-peer agreement

Note: Customers do not need to have AS Numbers.

Based on this relations, ASes can be classified further into 2 categories:

  1. Transit AS
  2. Non-Transit AS

Transit AS vs Non-Transit AS

An AS is a transit AS when traffic from outside your AS is routed through your AS to another outside AS. In this case your AS is a transit AS. When setting up such a network, the network administration must all the eBGP peer to communicate with other eBGP peers.

An AS is a non-transit AS when traffic from outside your network is not allowed to connect to any network that is inside your current network. As a network admin, you will only allow traffic from inside your network to connect to the outside world and the traffic from the outside world to connect to your internal devices.

Transit AS Example

Let’s say AS X wants to route traffic to AS Z. AS X will send the traffic to AS Y and AS Y will send the traffic to AS Z. In this case, AS Y is a transit AS. Assuming AS X and AS Z do not route traffic to anyone else, they are Non-Transit ASes.

Some examples of Transit ASes are Level 3, Arelion, Cogent Communications, etc. Some examples of Non-Transit ASes are content providers like Google, Netflix, Schools etc.

Other than just being a transit AS or a non-transit AS, you can also be a selective transit AS.

A selective transit AS is an AS that allows transit from a selective few while blocking transit from others.

Customer-Provider relationships

In this relation, customers pay provider for access to the internet and to be reachable from anyone.

Customer-Provider Relation

The provider routes the traffic of the customers to the internet. However, the customer does not help to route provider traffic through its networks. We can say that the provider provides transit service to the customer.

A customer can also have 2 providers, this is called multi-homing. When one network goes down, they can make use of the other network.

Peer-to-Peer relationships

Peer to peer relationships are when 2 ASes have agreed to exchange traffic between each other. They provide transit between their respective customers but do not provide transit from one peer to a separate peer. Often, these agreements are free (settlement-free).

Peer-to-Peer Relation

In the example above, Peer 1 has a peering agreement with Peer 2 and Peer 3. However, Peer 2 does not have a peering agreement with Peer 3.

If customer 1 wants to send data to customer 2, it will go through the peer network between Peer 1 and Peer 2. This will be similar between customer 1 and 3. However, if customer 1 wants to send data to customer 3, it will not go through any peer connections. In this case he will not be able to send data to customer 3.

Peering provides shortcuts between different customers and result in less traffic to the providers between the ASes

Choosing the appropriate relations

Many ASes have the dilemma of choosing between a customer-provider relationship or a peer-to-peer relationship.

Advantages of peer to peer

  1. Reduce upstream transit cost
  2. Improve end-to-end performance
  3. Might be the only way to connect customers (For Tier 1 ISPs)

Advantages of customer-provider

  1. Rather have them pay for the connection
  2. Peers are usually competition
  3. Peering relationships require periodic renegotiation

Peering agreements are usually confidential and one of the most contentious issues in the ISP world.

ISP / AS Tier list

Tier 1 ISPs

Tier 1 ISPs generally:

  1. Have access to the entire internet through settlement free peering
  2. Have no upstream provider
  3. Peer with other Tier 1 ISPs to form a full mesh

There are only 10-12 ASes that are Tier 1 ISPs.

Lower tier ISP

Lower tier ISPs generally:

  1. Usually provide transit to their downstream customers but have at least 1 provider upstream.
  2. Have national or regional coverage

There is only a few thousand ASes which are considered lower tier ISPs.

Stub ASes

Stub ASes generally:

  1. Do not provide transit services
  2. Only connect to upstream providers

This is similar to a customer with an AS number. A large majoring of all the ASes fall into this category.

Notable Peering Disputes

Here are a list of notable disputes due to peering.

  1. Cogent-Telia Peering Wars
  2. Cogent-Level3 Peering War

To see even more disputes, you can refer to the Anatomy of Internet Peering Disputes paper.

Internet Exchange Points (IXP)

Internet exchange points are an open and neutral location where ASes can freely interconnect to exchange traffic. They are similar to an ethernet switch for ASes and are often found in neutral locations.

Why do we need IXP

Imagine n different providers, each wanting to peer with each other. If they laid out cabling to connect to each other, they will have to lay out (n-1) * n cables. In the long run when there are more and more peers, the amount of cables will increase more and more.

This is where IXPs come in. Instead of laying cables to each other directly, they can now all lay cables to a neutral location where they are connected to each other (similar to an ethernet switch).

Goals of IXP

The goals of an IXP include:

  1. Save upstream transit costs
  2. Keep local traffic local
  3. Better network performance and Quality of Service
  4. Better scalability (See above)

How to join an IXP

Any network which has its own address space, AS number and transit agreements can join an IXP. Each member brings their own router to the IXP and connects it to the switch at the IXP.

Architecture of an IXP

IXP Architecture

The diagram above shows a simplified architecture of an IXP. The IXP have a series of core routers which help to route traffic between each of the IXPs.

Choosing a location for an IXP

Here are some of the criteria used to choose a location of an IXP

  1. Neutral
  2. Secure
  3. Accessible
  4. Safe
  5. Expandable

An example of good IXP locations are:

  1. Carrier Neutral Data Center
  2. Universities
  3. Tech Parks

Bad locations include:

  1. ISP data center
    1. It is not neutral
  2. Government data center
    1. There might be a security risk
  3. Cable Landing Station
    1. Physical access might be difficult

How to Operate an IXP

Operations of an IXP must be carried out by a neutral party. Usually they make use of a consortium model which consists of a management board with representative members of each of the members.

The costs are usually covered equally by all the participants and other services by the IXP.

You can see a list of real world IXPs here.

References