08 April 2012

Peering at JINX

The Internet - or any internet for that matter - is, by definition, a collection of connected networks.

The Internet started its history in 1969 as the ARPANET when four nodes were connected from four campuses: the University of California Los Angeles (UCLA), the Stanford Research Institute (SRI) in Menlo Park, California, the University of California at Santa Barbara (UCSB) and the University of Utah.  The well known map of the ARPANET below shows the original topology.

To the best of my knowledge this map was drawn by Alex McKenzie.  His notes are  archived by the Charles Babbage Institute who also now owns the copyright.  They have given permission for researchers to quote from the notes.

Even this simple topology (by modern standards) raises a number of interesting questions about interconnecting sites.  Since Utah is only connected to SRI, traffic between Utah and UCSB or UCLA has to be relayed via SRI.  Who pays SRI for this service?  How much does Utah have to contribute to the links between SRI and UCSB and between SRI and UCLA?  Is the link between UCSB and UCLA of concern to Utah at all?  Of course, this being a US military project at the time made the questions about funding easy to answer.  The other question that this old map illustrates is the one about whom a fifth node should connect to: one, two, three or all four other nodes?  In a world where countless organisations and individuals connect their networks or individual computers to the network these questions have, in principle, become impossible to answer.

In fact very few of the myriad of organisations and individuals connected to the Internet have the skills to decide whom to connect to and how to determine sharing costs.  Very few of them are willing to relay information for others, and very few of them want to entrust the relaying of their own information to a competing neighbour down the road.  And thus emerged Internet Service Providers (ISPs).  Rather than connecting to the Internet 'directly' organisations and individuals rather connect to an ISP.  This does not remove the interconnection problem - but delegates it to the much smaller (but still large) number of skilled specialists.

The connection between ISPs is known as peering.  (Technically, peering is the connection of two Autonomous Systems, but a discussion of Autonomous Systems will have to wait for another post.)  Two ISPs can (and often do) peer with one another by simply connecting their networks to one another and configuring their gateway routers to route traffic according to their agreed upon policies.  But, as we have seen above, this solution does not scale and is only appropriate where there is some special reason for individual networks to peer.

The more general solution is to establish an 'exchange' where all the ISPs in the area can connect to one another.  In South Africa at the time of writing the Internet Service Providers' Association (ISPA) operates two such Internet exchanges: one in Johannesburg (JINX, Johannesburg InterNet eXchange) and one in Cape Town (CINX).

An Internet exchange is often hosted by one of the bigger ISPs.  Both JINX and CINX are hosted by Internet Solutions (IS).

While I would love to post some pictures taken inside an Internet exchange, I haven't even attempted to ask permission to go and take some.  I would imagine that such information (in pictures) is rather sensitive and affects the critical infrastructure of a country.  So the best alternative is to show the outside of the buildings that house one.

JINX is indeed housed in two buildings.

JINX Rosebank is located at 158 Jan Smuts Avenue, Rosebank, Johannesburg.

The satellite and microwave dishes on the balconies are an indication that this is not an ordinary office complex.  The JINX Rosebank facility is located in vaults 14 and 16 somewhere on the second floor of this building.

The picture above was taken from the east. Walking around to the south presents one with the following view. Note again the various dishes on the balcony. The bank of air conditioners may not be easy to see from this angle, but there are indeed many.

From the west - from Jan Smuts Avenue - it looks as shown in the picture below.

Finally, looking from the north towards Johannesburg city centre, one sees the billboard of Internet Solutions claiming its presence (amidst others) in this building.

As usual a map may be useful to illustrate our journey thus far (as well as the journey that remains).  On the map below the JINX Rosebank facility is on the left.  We started our journey on its western side - more or less where the label (W) has been added in green.  From there we travelled clockwise until we reached its eastern side - more or less where the label (E) has been added in green.

Map derived from satellite imagery by Google Earth and others.
Immediately opposite the JINX Rosebank facility - towards its west in the same street block - is the Parklands Centre which contains the JINX Parklands facility.  On the map above we will follow the passage through the Parklands Centre that starts at its eastern side.  The picture below looks towards the Parklands Centre - approximately towards the point labelled (E) in yellow on the map.  The journey proceeds through the centre of the Centre and exists on the west - just below the spot labelled (W) in yellow.  We now enter the Parklands Centre from the east.

Apart from a Post Office, there does not seem to be happening much in the Parklands Centre.

But we know somewhere in this centre - to be more precise, in Vault 4 in this centre - is the JINX Parklands facility.  Exiting on the east the facade of the building is composed of drab concrete slabs housing what seems like emergency power generators.  Pictures from a few years back (which, because of copyright issues I cannot reproduce here) show a facebrick facade starting slightly deeper in.  Hence the part housing the generators must have been a fairly recent addition.

Note that the Rosebank facility (two vaults) and Parklands facility (one fault) have been interconnected to form one virtual Internet exchange.  In fact, at the time of writing, the Rosebank facility is filled to capacity and space only remains in the Parklands facility.  On a logical level this is unimportant to an ISP that wants to join the exchange.  On a physical level it does make a difference - the new ISP's cabling has to terminate in Parklands.

More information about the Internet exchanges is available on the ISPA site - click on INX.  Interesting information includes the names of the ISPs that peer at JINX (as well as those that peer at CINX).  Here are some other FAQs straight from their site:
Q: What equipment is required for connecting to an INX?A: You can connect to the INX switch fabric using either a suitable router or a MetroEthernet service. INX switches have copper based Ethernet connections and singlemode or multimode fibre Ethernet interfaces available. You require a router that supports BGP4 since all peering is done using BGP. BGP4 is supported on numerous devices from many vendors. Speak to your router vendor in order to obtain the best hardware. It is common to have dozens of BGP sessions with other members at the exchange and your hardware should be powerful enough to handle this.
Q: Who can I get backhaul links to an INX from?A: Backhaul links into the INXs are typically provided as either MetroEthernet or SDH circuits over fibre. You can approach any licensed ECNS holder to provide you with these services. A number of ISPA’s large and medium members will already have a point-of-presence (PoP) at or near the INX environment and may be able to provide these services. For larger capacity circuits it may be feasible to obtain dark fibre pairs or DWDM wavelengths into the INX environment.
Q: Can I get wireless access to an INX?A: Roof or tower space near the INXs is normally limited and it is preferable to connect to an INX via a fibre based circuit. Contact the host of the INX you wish to connect to in order to determine the rules and processes for installation of radio equipment.
Q: What’s the price of a link between Rosebank and Parklands?A: Existing cables between the Rosebank and Parklands cages are only for peering traffic on the switch fabric. ISPA is not able to provide these links to members. Members should speak to the host or a licensed operator to obtain these cross-connects.
Statistics about the traffic flowing through JINX are available at http://stats.jinx.net.za, and the statistics for CINC at http://stats.cinx.net.za