Magic Internet Kit

Magic Internet Kit
Programmer's Guide


TCP/IP Communications in Depth


TCP/IP communications is a very broad topic, so this document will only focus on the Magic Cap aspects of TCP/IP that are different from equivalent implementations on other platforms. This chapter assumes that you are somewhat familiar with the workings of TCP/IP, so if you need a more information, see Internetworking with TCP/IP by Douglas Comer or TCP Illustrated by Richard Stevens.

Like the previous chapter, Serial Communications in Depth, this chapter dives below the level of the Magic Internet Kit framework and talks about the guts that MIK takes care of for you. Regardless, this information is still important for understanding what's going on under the hood of the Magic Internet Kit.

Using ConnectableMeans vs. low-level APIs

As with serial communications, the first issue to discuss is which level of programming interfaces that your application should use. The low-level communications code in the system is what developers formerly had to use, but the Magic Internet Kit provides a higher level of abstraction that lets the developer program with one easy-to-use API and let the MIK framework worry about the messy and means-specific details.

In general, programming to the MIK framework is preferable to writing code directly for the low-level APIs. The Magic Internet Kit was designed to be both lightweight and modular, so your application only has to use the parts that it wants. In this case, selecting only TCP/IP communications support when creating your application will ensure that only the proper TCP support classes are included, so there will be minimal overhead even though you get the fancy MIK interfaces.

The Serial and TCP/IP In Depth chapters are included with the Magic Internet Kit documentation since they may lend insight into the design choices made in the implementation of the MIK framework. Additionally, if you're trying to debug a comms problem, the information presented here can be extremely valuable. For these reasons, you should read the chapters that apply to your application even if you are using the high-level MIK framework.


Note: Pay special attention to the Using a TCP stream section; this information is very useful for all applications that use TCP streams. Additionally, the Managing data link servers section can be useful for applications that want to manage link servers on their own.

Link servers, protocol stacks, and streams, oh my!

TCP/IP is not a simple beast, but it can be easily mastered. There are several protocols in action that are layered on top of each other, hence the name "protocol stack," and this section will briefly describe how they are related. Magic Cap's object-oriented systems maps these protocol layers into objects, so this section will also discuss the objects that are responsible for implementing the functionality of each layer.

The data link server

At the bottom of the stack is an object that knows how to talk to some type of hardware. This hardware can be a modem, Ethernet card, or anything else that can exchange bits. If there was a digital interface for two tin cans and a string, a link server could be written for it.

The job of the data link server is to throw bits back and forth with a remote host. It can be as smart or a stupid as it needs to be, but a good link server would implement some type of error correction. This server then talks above itself to the next higher level in the stack: IP.

Having a data link server between IP and the hardware allows IP to be generic and run over whatever server the programmer desires. This modularity keeps IP flexible. Examples of common data link servers are dial-up Point-to-Point Protocol (PPP) and Ethernet.

In the Magic Internet Kit, PPPLinkServer implements the core PPP protocol and is used by classes like DialupPPPMeans and MagicBusPPPMeans. Ethernet is implemented in the EtherServer class and is used by the XircomEthernet driver class and its matching XircomMeans class. Both of the link servers inherit from the DataLinkServer class, which is the abstract class that defines the APIs that all Magic Cap data link servers should implement.

Internet Protocol (IP)

Internet Protocol, as one might expect, is the core of the Internet. This protocol defines the connectionless delivery service that is the foundation upon which transport services rest, and the interface that talks to data link servers below. IP also acts as a multiplexer/demultiplexer for the transport services, e.g. TCP, that sit on top of it. IP can handle multiple transport services, so it decides which packets go to which transport services.

In MIK, the IPSwitch and IPDaemon classes implement IP. Applications will never talk to these classes directly; they are only used by the link servers and the transport protocols.

Transmission Control Protocol is a protocol that sits on top of IP and provides a mechanism for reliable, stream-based transport. IP is an unreliable service, i.e. packets can be lost or duplicated, but TCP manages that problem and requests packet resends and throws away duplicate packets. IP is also packet-based, whereas TCP provides a stream-based interface similar to using direct serial communications. For these reasons, TCP is an extremely important protocol and is the transport service of choice for many applications.

TCPStream is the Magic Internet Kit class that implements TCP. These objects are used by the application layer, and as such will be described in much greater detail shortly.

User Datagram Protocol (UDP)

User Datagram Protocol is another transport service, like TCP, that talks to IP on the bottom end and an application on the top end. UDP is packet based and unreliable by definition, but it is useful for some types of applications. This chapter will mostly deal with TCP, but UDP will be discussed later.

UDPPDUServer is the packet server class that MIK clients talk to, and clients must mix in the PDUClientInterface class and implement its upcall, HandlePDU, to receive incoming packets. Note that PDU is an acronym for Packet Datagram Unit.

Managing data link servers

Before any TCP streams can be created, a data link server must be in place and running. I'll use dial-up PPP as an example for the rest of this chapter, but remember that these concepts apply to any other link server as well. With PPP, the modem must be connected to the remote service provider before the TCP/IP elements of the stack can be started, and the application should also disconnect the modem when it is done. This process of managing the link server is handled automatically by the UsesTCP mixin class in the Magic Internet Kit, but this section will discuss methodologies for handling this if you want to do it by hand.


Source Code Note: The UsesTCP class is defined and implemented in Means:UsesTCP.(Def, c)

Determining if a link server is already in place

Before starting up a link server, it is wise to check and make sure that one is not already in place. The application could keep track of this in a state variable or something, but there is no need to since IP has that functionality already built in. The FindLinkByClass method of the IPSwitch class will let the application find out if a particular class of link server is already actively hooked into IP. Here's an example of getting the active link server instance:

linkServer = FindLinkByClass(iIPSwitch, linkClass);

In this case, linkClass is the class number of the link that you're looking for, e.g. PPPLinkServer_, and linkServer will be the Object ID of the running link server. If this class of link server is not running, the return value of FindLinkByClass will be nilObject.

Stating a new link server

Using UsesTCP to create the link server

The Magic Internet Kit's UsesTCP class provides a method for easily creating link servers: CreateDataLink tells a TCP-based means object to connect its link server if it is not already connected. This method, if successful, will return the object ID of the resulting link server object. If this method fails, it will return nilObject.


Note: Applications creating the link server by hand might want to increment the link server's UseCount attribute by one if they don't want the link to be taken down the next time that UsesTCP_DisconnectTCPService detects that the use count is zero.

Creating the link server by hand

If you don't want to use UsesTCP to create the link server for some reason, here's what is going on behind UsesTCP_CreateDataLink. This section is provided for informational purposes; applications should just use CreateDataLink to do this.

How to start a link server depends somewhat on what type of link server it is. A proper Magic Cap link server would inherit from the CommunicationStream class and implement the Connect method for establishing the connection. Once the hardware is physically connected to the remote host, and the remote host is set up to start the type of data link server that you want, e.g. the terminal server is in PPP mode, it's time to create a new link server and start it up. Here's the code:

/* Connect the hardare, assuming that it supports Connect(). */
Connect(hardware, ticket);

/* Set up paramters for new link server object. */
SetUpParameters(&linkParameters.header, newDataLinkServerObjects);
linkParameters.serialServer = hardware;

/* Create the new link server! */
linkServer = NewTransient(linkClass, &linkParameters.header);

/* If a source IP number is known, let the link server know. */
SetDataLinkSourceIP(linkServer, mySourceIP);

/* Increment the usage count of the link server. */
SetUseCount(linkServer, UseCount(linkServer)+1);

/* Fire up the link server! */
SetEnabled(linkServer, true);
StartDataLink(linkServer);
AttachLink(iIPSwitch, linkServer, DataLinkSourceIP(linkServer));

Creating the link server object is quite straightforward using the NewTransient call. After the object is created, its DataLinkSourceIP attribute should be set to an IP address if one is to be manually assigned, otherwise set to zero and the link server should have one dynamically assigned. The link server's UseCount attribute should also be set to one to keep track of the number of users. This attribute will be used later to determine when the link server should be shut down; when disconnecting a link user, the link user should disconnect the data link if the resulting use count is zero.

Now that the link server object is all ready to go, the last three method calls will start it up and connect it to IP. Note that these methods might fail, and therefore the code above should be surrounded by exception handling code for whatever exceptions might get thrown. For the PPPLinkServer class, a commHardwareError exception will get thrown if PPP negotiation fails. Also keep in mind that connecting the hardware using Connect may throw its own exceptions, so that code needs to be shielded as well in the same way that the equivalent serial connection would. For a more detailed example, see the UsesTCP class.


Note: Applications do not need to create the IPSwitch object manually as it will create itself at package installation time. Don't destroy the IPSwitch object, either!

Destroying a link server

Using UsesTCP to destroy the link server

The DestroyDataLink method of UsesTCP is in the logical inverse of CreateDataLink. This method will check to see if the UseCount attribute of the link server is zero, and if so, it will take down the link. There is a related method, AbortDataLink, that will do a hard take-down of the link server. Applications should not sure this method, though, because AbortDataLink will take the link down regardless of users and will also ensure that the hardware is disconnected. AbortDataLink therefore should only be used by the connection code in UsesTCP.

Destroying the link server by hand

As with creating the link server, there's no reason that applications should have to implement this code since DestroyDataLink does the same thing, but this information is included for the sake of completeness.

As usual, destroying is easier than creating. The only caveat is that the application should check to make sure that it does not destroy a link server that is being used by other streams within the application. This is easily handled by using the UseCount attribute of the server to keep track of the number of users. Here's an example of closing down a link server:

linkServer = FindLinkByClass(iIPSwitch, LinkClass(self));
if (HasObject(linkServer))
{
    ulong useCount = UseCount(linkServer) - 1;
    SetUseCount(linkServer, useCount);
    if (useCount <= 0)
    {
        DetachLink(iIPSwitch, linkServer);
        Destroy(linkServer);
    }
}

DetachLink will tell IP that the link server is no longer active, and Destroy will both call Disconnect on the link server's hardware and destroy the link server object.

Creating and using TCP streams

Creating a TCP stream and initiating a connection

Once the link server and is running and connected to IP, as described in the previous section, the application can create transport layers like TCP on top of it. This is pretty easy to do, and it is similar to setting up any other connection: fill in the blanks of a Means object and pass it to TCPStream's Connect method. Here's what the code looks like:

/* means is a DialupIPMeans object, ticket is a TransferTicket */
ReplaceTextWithLiteral(HostName(means), "192.216.22.142");
SetPort(means, 80);

SetMeans(ticket, means);

/* Create the TCPStream object with default parameters */
stream = NewTransient(TCPStream_, nil);
SetStream(ticket, stream);

/* Connect the stream to the remote host */
Connect(stream, ticket);

The means object passed into TCPStream_Connect must inherit from DialupIPMeans, and the HostName field must contain the IP address of the destination. For more information on IP addresses and name resolution, see the Resolving host names with DNS section later in this chapter. If the Connect method of TCPStream fails, it will throw a serverAborted exception, so be sure to catch these exceptions.

Parameters can also be passed to NewTransient when creating the TCPStream object. The only interesting parameter is the source port, which may need to be set if required to be within a certain range. This is usually never the case, though, and therefore should be set to zero so that IP will assign a port dynamically.


Note: Multiple streams can be created on the same data link! Just remember to keep track of how many exist by incrementing the link server's UseCount attribute each time a stream is created and decrement the count when they are destroyed. Neat!

Listening for an incoming TCP connection

The above section describes how to initiate an outgoing connection, but with the Magic Internet Kit applications can also listen for incoming connections. The high level MIK framework does not include this functionality built-in just yet since most applications will only want to initiate connections, but the TCPStream interface is in place and fully functional.

To listen for an incoming connection, create the data link server as usual, but replace the connect code with the following:

NewTCPStreamParameters parameters;
SetUpParameters(¶meters.header, 0);
parameters.destinationIPAddress = 0;
parameters.destinationPort = 0;

/* port to listen to */
parameters.sourcePort = 80;

stream = NewTransient(TCPStream_, (Parameters*)¶meters);

Listen(stream);

In this case, the stream will listen to port 80, the httpd port. The Listen method will immediately return, but the next Read or Write call will block until a remote host connects to the specified port on the device. Therefore any code following the Listen that relies on the connection being active should be after a Read or Write call! For example, if the application calls CountReadPending immediately following the Listen, the Listen will return immediately and the CountReadPending will also immediately return zero, which is probably not what the programmer wants.


Note: Due to the inherently blocking nature of listening for an incoming connection, or doing most other communications work for that matter, this code should not be running on the User Actor! See the Multithreading with Actors section of this document for more information on creating and using new threads.

Using a TCP stream

TCPStream objects are used pretty much like any other CommunicationStream subclass; CountReadPending returns the number of bytes available to read, Read reads the bytes, and Write writes bytes.

CountReadPending

operation CountReadPending(): Unsigned, noFail;

This method is used to find out how many bytes are in the stream's local buffer and are therefore available for immediate reading. Reading is a synchronous operation, so code that does not want to block while reading should always call CountReadPending first to check how many bytes are available.

Read

operation Read(buffer: Pointer; count: Unsigned): Unsigned, noFail;

This method is used to read a number of bytes from a stream into a buffer. If the number of bytes requested by Read are not available yet, for example the server has not sent them, Read will block until either all the bytes are received or an error has occurred. The return value is the number of bytes that were read, and comparing this value to the number of bytes requested lets the application know if an error occurred; if fewer bytes were returned than requested, then an error occurred but Read still returned as much data as it could get.

One useful tactic when waiting for data is to block on a one character read and then read any other pending data. For example, the following code is used in the CujoTerm template application:

count = Read(stream, &buffer, 1);

if (count != 1)
{
    Fail(serverAborted);
}
else /* count == 1, everything is peachy keen */
{
    Unsigned count = CountReadPending(stream);
    if (count != 0)
    {
        if (count > 254) count = 254;
        Read(stream, &buffer[1], count);
    }
    
    /* Remember that we already read one character up above! */
    count++;
    
    HandleBytes(client, (Pointer)buffer, count);
}

In this case, the one character read will block until either data is available or an error occurred, for example the remote host closing the stream. When Read returns, a check is made to make sure that Read returned a character, and if not an exception is thrown. If the character was indeed read okay, the rest of the code will get the rest of the pending bytes, up to 255 total, and process them.

Write

operation Write(buffer: Pointer; count: Unsigned), noFail;

Write, as one can imagine, is used to write data to a stream. With TCP streams, write will immediately return after placing the bytes into TCP's outgoing buffer. If the TCP stream was closed for some reason, or some other error occurred, Write will instead throw a serverAborted exception. For this reason, all calls to Write should be prepared to catch serverAborted exceptions and handle the error case.

Resolving host names with DNS

While IP addresses look neat, what with all those numbers and periods, humans have a hard time remembering them. For this reason symbolic names are often assigned to refer to IP addresses, for example the name "www.genmagic.com" currently refers to the IP address 192.216.22.142. This feature also allows www.genmagic.com to be repointed to a new machine without forcing users to know about the change or the new address. All of this neat functionality is bestowed upon us by the Domain Name System (DNS).

While DNS is very useful for resolving host names, also keep in mind that resolving a name takes time; a UDP packet requesting the name resolution must be sent a known name resolver, and then the response packet must be received. Fortunately, DNS caches names once they have been resolved, but the initial lookup still takes a second or two. Additionally, adding the DNS resolver to your Magic Cap application increases the memory footprint by about ten kilobytes.

DNS in the Magic Internet Kit framework

When using the MIK framework, the UsesTCP class will resolve host names when needed. Specifically, if a host name is not a properly formatted IP address, the UsesTCP class will figure that it's a host name that a DNS server can figure out. It will then ask DNS if it can resolve the name and, if not, fail up to the Means classes with a serverAborted exception. The Means classes will then return nilObject for the stream.

To tell DNS which name resolution servers to use, add the 32-bit IP address of each server to the list in the DNSServers attribute of the ConnectableMeans object. This list is a FixedList object, so entries can be added with the AddUniqueElem method. If the means object is in the application's object instance file, entries can also be added there.


Source Code Note: The Magic Internet Kit's usage of DNS is contained in the UsesTCP class implemented in Means:UsesTCP.(Def, c)

Using DNS manually

If your application is not using the MIK framework, which automatically handles name resolution, you can still use DNS manually. This section will tell you how to do it.


Source Code Note: The class definition for the DNSResolver class is found in the TCP:DNSResolver.Def file.

Preparing for DNS lookups

The first step in preparing to use DNS is making sure that a link server is active. If there's no data link, DNS can't send its request to the name resolvers. See the above section on managing data link servers for more information on that topic. Once the data link is up, the application needs to tell DNS which domain name resolvers to contact. This is done using the RegisterDNS method. Here's the code:

RegisterDNS(iDNSResolver, 0x12345678);

The server should be specified by its 32-bit IP address. If applications need to translate dotted text IP addresses into 32-bit unsigned format, the MakeIPAddress method of IPSwitch will do this.


Source Code Note: The IPAttributeText class translates IP addresses between dotted text and 32-bit formats. See Means:UsesTCP.(Def, c) for the implementation of IPAttributeText.

Much like the IPSwitch class, the DNS resolver referred to by the iDNSResolver indexical does not need to be created or destroyed by the application. This class will automatically create itself at package installation time.

Looking up a symbolic host name

Once DNS has at least one resolver registered and the link server is active, the application can freely request name and address information. To convert a symbolic host name to an IP address, use the GetHostByName method of the DNSResolver class. Here's an example:

ReplaceTextWithLiteral(hostName, "www.genmagic.com");

address = GetHostByName(iDNSResolver, hostName);

SPrintF(addressString, "%d.%d.%d.%d", (address>>24),
        (address>>16)&0xff, (address>>8)&0xff, address&0xff);

ReplaceTextWithLiteral(hostName, addressString);

In the above code, the hostName parameter is a text object containing the name to be looked up, and the return value is the 32-bit IP address. If the name lookup fails, a DNSLookupFailed exception will be thrown, so the application should plan ahead and catch this exception.

Keep in mind that TCPStream's Connect method expects the host name to be a dotted text IP address, so the last two lines of code convert the 32-bit value into that format and put the result back into the hostName text object.

Looking up an IP address

The process of resolving an IP address is almost identical to looking up a host name. The GetHostByAddress method takes a 32-bit unsigned IP address and returns a text object containing the host name associated with that address. If no name exists a DNSLookupFailed exception is thrown.

Shutting down DNS

The iDNSResolver object will stick around in transient memory when the application is done using it, but applications might want to unregister DNS servers when it is done using these servers. Additionally, if the user has the ability to modify the list of DNS servers that the application should use, a server should be unregistered when the user specifies that it should no longer be used. To do this, call the UnregisterDNS method. The parameters for this method are identical to that of RegisterDNS.

Persistence of DNS

The object pointed to by the iDNSResolver indexical is located in transient memory. This means that the object will go away whenever transient memory gets blasted, but it will always be recreated. This means that the DNS resolver's cache will be cleared as well as its list of servers that were registered with the RegisterDNS method. As a result, the application should always register its DNS servers before it plans to use them if there is a chance that transient memory could have been reset since the last registration.

More to follow

This chapter of the MIK Programmer's Guide is not yet completed at this time. To find the latest version of this document, which should be more complete by the time you read this, check out the Magic Cap Developer Resources web pages located at:

http://www.genmagic.com/Develop/MagicCap/index.html


Magic Internet Kit home page Programmer's Guide index