Manage your application's communication more precicely.

Organization of the Tutorial.

• Explanation
• Combining Simulator/Wire-Protocol
• AppSockets
• Accept
• Read
• Connect
• Write

Similar to the selection mechanism in Java's non-blocking I/O facility, FreePastry notifies you of events (can read / can write / socket available), and then your code can handle the events immeadiately or later. FreePastry always makes these notifications by calling your implementation of the AppSocketReceiver on the Slector Thread. Since these calls are issued on the FreePastry's network thread, if you must do computationally intense or blocking I/O operations you need to do so on a different thread. If you spend too much time on FreePastry's selector you will interfere with overlay maintenance and responses to liveness checks, which will result in degraded performance. You can do time consuming operations on your own thread or on FreePastry's Processor or Blocking-I/O threads before calling read()/write(). See the Environment tutorial for more information.

Unlike FreePastry's routing/messaging interface, the AppSocket interface does not use any serialization technique. You will simply be able to send/receive ByteBuffers.

Why may applications need their own socket interface?

• Complete control of serialization of data: No need to chunk or aggregate into individual messages. No need to use Java Serialization or FreePastry serialization which may not be optimal for bulk data transfers.
• Control resources more closely, and provide end to end flow control:

  For example, your application is disk bound. It is receiving a bulk data transfer over FreePastry. You accomplished this by chunking the data into multiple messages. If your receiver cannot write the items to disk fast enough, you have 3 options:

  1. Drop messages, and build retry-logic.
  2. Try to queue them, and risk running out of memory (RAM).
  3. Attempt to build some kind of flow control mechanism over FreePastry's message interface.

  With the socket interface, you gain more control of your application's communication channel, and often the obvious answer becomes the correct answer. In this case, don't read from the TCP buffer until the previous disk IO completed. This will cause back-pressure to the source and that node will not waste bandwidth by sending data that you cannot process.

• Fully utilize the connection without interfering with FreePastry's overlay traffic, other applications running on the node, or other applications running on the machine/network.

  FreePastry uses only a single TCP connection for it's "Message Based" traffic. This includes FreePastry's maintenance traffic as well as Application's message traffic. FreePastry does not employ a chunking mechanism. Thus sending very large messages in FreePastry which could take several minutes to transfer will stall FreePastry's overlay maintenance messages, which could result in poor overlay performance.

  With these limitations, it is difficult to utilize all available network resources on high-capacity nodes, while still allowing FreePastry to operate properly on lower capacity nodes.

  With the Application Socket Interface, your application will have its own socket, which will be subject to all the benefits of TCP (such as congestion control) that make it interoperate well with other applications. You have more direct information about network performance in your particular environment.

Why not just open my own sockets?

FreePastry has made some effort to work properly in reduced connectivity environments such as NATs and in the presence of temporary routing anomlies. For example FreePastry only requires 1 (UDP/TCP) port to be forwarded, and does not require any specific port, thus making it possible to run multiple nodes behind the same NAT. Also, FreePastry uses source routing to bridge connections during temporary internet anomalies. If your application opens its own sockets then it will be required to allocate additional ports, open them on NATs, and have its own source-routing infrastructure to get through the communication difficluties that FreePastry already handles. As we attempt to handle additional connectivity issues in the future, we plan to continue to support the AppSocket interface.

The AppSocket interface is compatable with FreePastry's simulator. The sockets are simulated rather than requiring sys-calls. Our solution scales better than opening potentially n-squared connections on a single computer that is simulating tens or hundreds of thousands of nodes.

Note that the Application socket interface is no substitute for FreePastry's Routing mechanism which provides all of the advantages of the Key Based Routing primitive.

Download the tutorial files: Tutorial.java MyApp.java, into a directory called rice/tutorial/appsocket/.

First the command line args are more sophisticated. You can either use the previous syntax of localbindport bootIP bootPort numNodes to use sockets, or specify -direct numNodes to use the simulator.

Usage: 
java [-cp FreePastry-.jar] rice.tutorial.appsocket.Tutorial localbindport bootIP bootPort numNodes
  or
java [-cp FreePastry-.jar] rice.tutorial.appsocket.Tutorial -direct numNodes

example java rice.tutorial.DistTutorial 9001 pokey.cs.almamater.edu 9001 10
example java rice.tutorial.DistTutorial -direct 10

Here is the code that accepts the arguments from the command line, and builds the appropriate environment. I won't spend time explaining it because it is not very specific to FreePastry.

      boolean useDirect;
      if (args[0].equalsIgnoreCase("-direct")) {
        useDirect = true;
      } else {
        useDirect = false; 
      }
            
      // Loads pastry settings
      Environment env;
      if (useDirect) {
        env = Environment.directEnvironment();
      } else {
        env = new Environment(); 
        
        // disable the UPnP setting (in case you are testing this on a NATted LAN)
        env.getParameters().setString("nat_search_policy","never");      
      }
    
      int bindport = 0;
      InetSocketAddress bootaddress = null;
      
      // the number of nodes to use is always the last param
      int numNodes = Integer.parseInt(args[args.length-1]);    
      
      if (!useDirect) {
        // the port to use locally
        bindport = Integer.parseInt(args[0]);
        
        // build the bootaddress from the command line args
        InetAddress bootaddr = InetAddress.getByName(args[1]);
        int bootport = Integer.parseInt(args[2]);
        bootaddress = new InetSocketAddress(bootaddr,bootport);    
      }      

  /**
   * This constructor launches numNodes PastryNodes.  They will bootstrap 
   * to an existing ring if one exists at the specified location, otherwise
   * it will start a new ring.
   * 
   * @param bindport the local port to bind to 
   * @param bootaddress the IP:port of the node to boot from
   * @param numNodes the number of nodes to create in this JVM
   * @param env the environment for these nodes
   * @param useDirect true for the simulator, false for the socket protocol
   */
  public DistTutorial(int bindport, InetSocketAddress bootaddress, int numNodes, Environment env, boolean useDirect) throws Exception {

    // construct the PastryNodeFactory
    PastryNodeFactory factory;
    if (useDirect) {
      NetworkSimulator sim = new EuclideanNetwork(env);
      factory = new DirectPastryNodeFactory(nidFactory, sim, env);
    } else {
      factory = new SocketPastryNodeFactory(nidFactory, bindport, env);
    }

      if (bootHandle == null) {
        if (useDirect) {
          bootHandle = node.getLocalHandle();
        } else {
          // This will return null if we there is no node at that location
          bootHandle = ((SocketPastryNodeFactory)factory).getNodeHandle(bootaddress);
        }
      }

Additional changes to Tutorial.java from Lesson 4

• We removed the "route 10 messages" loop in the tutorial. You cannot "route" in the overlay with the direct sockets. To route or lookup the nearest node to an Id, you must use the messaging layer as in lesson 4. To use the AppSocket interface, you must have a NodeHandle to a specific Node in the network that you wish to open the connection to. See Lessons 1, 2 and 3 for the explanation of these terms.
• We changed the name of the method from routeMyMessageDirect() to sendMyMessageDirect().
• We want to show that AppSockets also work when connecting to yourself (the local node). Thus in the sending messages to the leafset loop, we took out the "if (i != 0) {" conditional, so that we also send to ourself.

When requested, FreePastry will provide your application with an AppSocket which you will use to read and write your data. The read() and write() calls are non-blocking, and calling them when the socket is not ready to read or write will be a waste of time: it will read or write zero bytes. Thus FreePastry needs to notify you when the socket is ready.

The AppSocketReceiver interface is how FreePastry's Application Socket Interface notifies your application that it may read or write or that a new socket is available. As you can see from the JavaDoc, it has 3 callbacks:

• receiveSocket() -- called when the socket has been established.
• receiveSelectResult() -- called whenever your app has permission to read or write.
• receiveException() -- called when there is a problem.

Acquiring a socket

Sockets are inheritently an asynchronus connection. There is always a connector and a connectee or usually we call these the client and server. Typically all of your nodes will both be servers and clients. The server/client designation only relates to the asymmetry of the connection process, not necessarally the intent of the application.

For example, we have node Alice who would like to get() a large object from a DHT. It turns out that the primary replica holder of the requested key is Bob. But Alice doesn't know that yet. Here is how this could work:

1. Alice calls get(key) which routes the request to the primary replica of the "key." The request contains the key of the object to be returned, and her NodeHandle. This is done using FreePastry's routing/message interface which you learned in lesson 3.
2. The request is routed to Bob, the primary replica holder.
3. Bob uses Alice's NodeHandle to connect to Alice using FreePastry's Application Socket Interface.
4. When the socket becomes available for writing, Bob sends the requested data.
5. Once Alice receives the object, she checks the key of the object she received to determine which request the opened socket referred to.

Note that there is no need for Bob to read from the socket he opened, and there is no need for Alice to write on the socket she received. In this case, even thouhgh Alice is the requester, she is the server, and must accept the socket to handle the response. Note that both the server and the client need access to an AppSocket:

• Alice, the server -- so she can read the response
• Bob, the client -- so he can send the response.

Our example will skip the routing/messaging component from the above example. We expect that you should be able to accomplish this if you have completed tutorial less #4.

To accept a socket, you must call endpoint.accept() with an AppSocketReceiver. When another node's application attempts to connect to your application, the new AppSocket will be delivered via the receiveSocket() method on this AppSocketReceiver.

    // register the endpoint
    this.endpoint = node.buildEndpoint(this, "myinstance");
    
    ...
    
    // example receiver interface
    endpoint.accept(new AppSocketReceiver() {
      /**
       * When we accept a new socket.
       */
      public void receiveSocket(AppSocket socket) {
        ... handle socket here ...
        
        // it's critical to call this to be able to accept multiple times
        endpoint.accept(this);
      }    
      
      ... rest of the interface ...
    }
    
    ...
    
    // register after we have set the AppSocketReceiver
    endpoint.register();

You may receive an AppNotRegisteredException if you attempt to open a socket on a remote node who has not yet registered the application.

Congratulations! You now know how to accept sockets in FreePastry's Application Socket Interface! Next, you will learn how to read from the AppSocket.

  /**
   * Must be called every time a Read/Write occurs to continue operation.
   *
   * @param wantToRead if you want to read from this socket
   * @param wantToWrite if you want to write to this socket
   * @param timeout // the socket's timeout value (this is a TCP level param)
   * @param receiver will have receiveSelectResult() called on it
   * note that you must call select() each time receiveSelectResult() is called.  This is so
   * your application can properly handle flow control
   */
  void register(boolean wantToRead, boolean wantToWrite, int timeout, AppSocketReceiver receiver);

        // this code reuses "this" AppSocketReceiver, and registers for reading only, and a timeout of 30000. 
        socket.register(true, false, 30000, this);

      public void receiveSelectResult(AppSocket socket, boolean canRead, boolean canWrite) {

        ...
	      
        // read from the socket into ins
        long ret = socket.read(ins, 0, ins.length);    

        ...
        
        // only need to do this if expecting more messages
        // socket.register(true, false, 3000, this);        
      }

The call to read() is pretty straightforward. It is nearly identical to SocketChannel.read().

The socket.register() call works in a similar way to endpoint.accept(). You only get notified once each time you register, and must re-register for additional data once you are ready to read or write more data. Usually, you will re-register with the same arguments as the initial registration. In our example, we don't expect to send additional data, thus we don't re-retister. (The call is commented out.)

Congratulations! You now know how to register for notificaton of read events and read from the AppSocket.

    endpoint.connect(nh, new AppSocketReceiver() {
      
      /**
       * Called when the socket comes available.
       */
      public void receiveSocket(AppSocket socket) {
        // handle the socket once it connects
        
        ...
        
      }    

      /**
       * Called if there is a problem.
       */
      public void receiveException(AppSocket socket, Exception e) {
        e.printStackTrace();
      }
      
      ...

    }, 30000);

        // register for writing
        socket.register(false, true, 30000, this);

      /**
       * Example of how to write some bytes
       */
      public void receiveSelectResult(AppSocket socket, boolean canRead, boolean canWrite) {   

	      ...
	      
	      long ret = socket.write(outs,0,outs.length);        
	      
	      ...
	      
        // keep writing
        socket.register(false, true, 30000, this); 

        ...  
        
      }
    }, 30000);
  }  

Congratulations! You have just sent and received messages over FreePastry's Application Level Socket Interface. Now you can more precicely control your application's bandwidth and its data-transfer mechanism.