Hercules CTCI-W32

Table of Contents

• Overview   (How it Works)

• Configuring Windows Networking

  • IP Forwarding
    

• Configuring Hercules

• Configuring your Guest Operating System's TCP/IP Settings

  • Defining your guest's default gateway
    

• Tweaking CTCI-W32

Since Hercules runs as a user process under the control of a driving system (Windows in this case), it does not normally have direct access to the driving system's network adapter.  Until recently, this presented a problem in establishing connectivity between the network and the TCP/IP stack of an operating system running under Hercules.

But thanks to some fancy coding by yours truly (with the help of WinPCap's device driver), it is now possible to establish a virtual point-to-point link between the TCP/IP stack running under Hercules and Window's TCP/IP stack, allowing you to use Windows as a router to pass Ethernet frames between the Hercules TCP/IP stack and the rest of the network, as shown in the following diagram:

    +----------------------------------------------------------------------+
    |                                                                      |
    |                    W I N D O W S   H O S T                           |
    |                                                                      |
    |                                                                      |
    | +-------------+    00-00-5E-A8-00-04                                 |
    | |   Hercules  |     +-------------+                                  |
    | | ----------- |     |  Hercules's |                    +-------------|
    | |   OS/390    |     |  "virtual"  |   (FishPack dll)   |  WinPCap    |        (to other LAN work-
    | |   TCP/IP    |     |   adapter   |<------------------>|   device    |          stations, Internet,
    | | ----------- |     |             |                    |   driver    |            etc...)
    | |    CTCA     |     |(192.168.0.4)|                    + - - - - - - +
    | |(192.168.0.4)|     +-----+-------+                    |  Windows's  |          +-----------------+
    | +------+------+           |                            |    REAL     *--------->|   192.168.0.1   |
    |        |                  |        +------------------>|  adapter    |   (LAN)  | Gateway adapter |
    |        +------------------+        |  (Winsock dll)    | 192.168.0.2 |          +-----------------+
    |          (TunTap32 dll)            |                   +-------------+           00-02-48-8B-8D-60
    |                                    |                  00-80-B3-E1-DF-69
    |                          +---------V-------+                         |
    |                        +-|                 |                         |
    |                      +-| |  (Windows       |                         |
    |                      | | |   Applications) |                         |
    |                      | | |                 |                         |
    |                      | | +-----------------+                         |
    |                      | +-----------------+                           |
    |                      +-----------------+                             |
    |                                                                      |
    +----------------------------------------------------------------------+

The TunTap32 dll uses the FishPack dll to send/receive packets on the actual physical adapter (via WinPCap's device driver).  By responding appropriately to Address Resolution Protocol packets, it makes Windows think there is another physical adapter connected somewhere on the local network.  This adapter (that Windows thinks exists) however, is simply a "virtual" one "created" by simply having TunTap32 respond appropriately to Windows's ARP request packets with a MAC address constructed from Hercules's assigned IP address.  (I got the idea after reading a discussion of the "Virtual Router Redundancy Protocol (VRRP)" in RFC 2338.)

Herc's virtual adapter MAC address (network card address) is calculated to be: 00-00-5E-xn-nn-nn, where 'xn-nn-nn' is always the last three octets of the IP address assigned to Hercules with the high-order x'80' bit 'OR'ed on (that's what the 'x' means in 'xn-nn-nn').  Thus, in the above example, the MAC address of Herc's virtual adapter would be 00-00-5E-A8-00-04  (because the last three octets of Herc's IP address (192.168.0.4) are: '168' ('A8' in hex), '0' and '4').

TunTap32 dll simply reads packets from, and writes packets to, the real Windows adapter via the FishPack dll (which simply uses Windows ReadFile and WriteFile and DeviceIoControl APIs to communicate with the WinPCap device driver to pass packets to/from the actual hardware), prepending them with the internally generated Ethernet MAC address of Hercule's "virtual" adapter whenever it does.  From Windows's point of view, these packets came from some workstation's real network adapter somewhere out there on the local network and summarily does whatever it does with them, which is usually route them to the default gateway and thus "to the outside world" (i.e. out on the 'net or wherever).

When the packets come back to Windows (from the Internet or wherever), Windows then sends them back to the adapter they came from (which is Herc's "virtual" adapter) by prepending the Ethernet packet with the MAC address of Herc's virtual adapter, and then writing the packet onto the LAN.

The real adapter (whose IP address is 192.168.0.2 in this example, and which is physically a part of the real network, being a real adapter after all) sees this packet and asks the WinPCap device driver whether or not it's interested in it.

The WinPCap device driver, because it's a packet sniffer, is always interested in all packets and thus saves the packet in its internal device driver buffer.

Then the next time TunTap32 dll does an I/O to the real adapter (via the FishPack dll via the WinPCap device driver), it receives the packet, examines it, and notices that it's a packet whose destination MAC address matches Herc's virtual adapter and consequently returns it back to Herc.

Simple, yes?   :)

Anyway . . . so ends the discussion of how it works.  Now for how to actually use it . . .

There's really not much, if anything, to configure in order to use CTCI-W32.  The only thing you need is the WinPCap device driver (including their accompanying "packet" dll), so that's the first thing you'll need to acquire and install: WinPCap. Download and install WinPCap according to their instructions and you're halfway home.

IMPORTANT! Be sure to only use the latest version of WinPCap fully supported by CTCI-W32 and not the very latest version that may be availble! As of this writing the most current version of WinPCap that is fully supported by CTCI-W32 is version 4.0. Thus even though by the time you read this version 4.1 of WinPCap may become available, that doesn't necessarily mean CTCI-W32 will necessarily support it! ALWAYS, ALWAYS, ALWAYS use only the latest version of WinPCap that is fully supported by CTCI-W32!! (In other words, the very latest, most current version of WinPCap may not necessarily be supported by CTCI-W32!!)

The next thing you need to do is double-check that your network adapter has an IP address assigned to it, and that it also has a default gateway assigned.  For most people, this is already the way your network adapter is setup, so for most individuals, you won't have to do anything at all, but just in case, you should double check to make sure.

Note: if you're using DHCP rather than assigning your own IP address to your network card/adapter, then you'll need to tell Herc the exact hardware address (MAC) of the network card/adapter that you want Herc to use.  See the "Configuring Hercules" section further below for more information.

Right-click on My Network Places icon on your desktop and select Properties. A "Network and Dial-up Connections" window should appear (I'm doing this on my Windows 2000 Pro system. The procedure for Windows 9x would be slightly different):

Right click on the icon for your local area network connection (called "Local Area Connection" on my system) and again, select "Properties" from that context menu and a "Local Area Connection Properties" dialog should appear:

Double-click on the "Internet Protocol (TCP/IP)" component (or select it and then click on the "Properties" button) and a "Internet Protocol (TCP/IP) Properties" dialog should appear:

Make sure you have Use the following IP address selected and have entered a valid IP address, Subnet mask, and Default gateway value.

That's pretty much it (except for the IP Forwarding issue explained just below).  That's all the configuration you need to do to your Windows system.  The rest of the CTCI-W32 functionality is controlled via Hercules device statements.

Note: if you're using DHCP rather than assigning your own IP address to your network card/adapter (i.e. you prefer to have the "Obtain an IP address automatically" selected in the above dialog instead), then you'll need to tell Herc the exact hardware address (MAC) of the network card/adapter that you want Herc to use.   See the "Configuring Hercules" section further below for more information.

You may or may not need to have "IP Forwarding" enabled on your Windows system. Whether you do or not depends on whether you're using a router to route traffic on your network or not. If you're using a router then you shouldn't need IP Forwarding enabled on your Windows system, but if you're not using a real physical hardware router then you probably do need to enable it.

To allow your Hercules guest o/s to communicate with the outside world (beyond the system it's running on), you need to either have a hardware router or else have IP Forwarding enabled on your Windows system. This is so your Hercules virtual guest's packets can be properly routed to their final destination, wherever that may be. If you have a router that's what it does. If you don't, that's what Windows' "IP Forwarding" does: it causes Windows to behave like a router and thus becomes your network router in essence.

Without a router and without IP Forwarding enabled, you will only be able to communicate with your Hercules guest from the same Windows computer that Herc is running on (and vice-versa; Herc will only be able to communicate with the Windows system it is running on).

So, if you don't have a router and you want Herc to be able to communicate with the outside world (i.e. beyond the Windows system it's running on), then you need to make sure IP Forwarding is enabled in Windows.

To enable IP Forwarding on Windows 2000/XP, make the following registry change:

  [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
  "IPEnableRouter"="1"

Note: the procedure for Windows 2000 / 2003 Server is slightly different still. Basically (I'm going from memory here) you use the "Routing and Remote Access" snapin and right-click on your server and select "Properties", and then on the dialog that appears, simply check the "Enable this Server to Act as a Router" checkbox (for at least "Local Area Network (LAN)" traffic). That may not be exactly right, but it's close.   :)

For Windows 98/Me, make the following registry change instead:

  [HKEY LOCAL MACHINE\System\CurrentControlSet\Services\VxD\MSTCP]
  "EnableRouting"="1"

Note: if the "IPEnableRouter" (or "EnableRouting") registry value does not yet exist under the aforementioned registry key, you will have to manually add it. Select 'New' from the 'Edit' menu and add a new DWORD value with the indicated name.

* * *   N O T E   * * *

It is also very important that you disable IP Forwarding on your guest operating system (running under Hercules) if you are using an older version of TunTap32.dll.

If you you mistakenly forget to do this, your guest o/s will get stuck in a sort of "feedback loop". (See item #7 in the CTCI-W32 FAQ for more information).

Let me say that again: if you are running an older version of TunTap32.dll, you need to remember to disable "IP Forwarding" in your guest o/s running under Herc.

Also note that this is completely different from enabling IP Forwarding on your Windows host system.  As explained further above, you may wish to enable IP Fowarding in your Windows host (so Windows properly routes packets beyond the workstation Herc is running on; i.e. so your Herc system can communicate with the rest of the world), but regardless of whether you wish to enable IP Forwarding on your Windows host system, you must still be sure to disable IP Fowarding on your guest o/s running under Hercules  (if you're using an older version of TunTap32.dll).

With the newer versions of TunTap32.dll (version 2.0.0.367 or greater) this shouldn't be a problem. With the newest version of TunTap32 you should be able to use IP Forwarding on your Herc guest o/s just fine, but with older versions you can't.

Two IP addresses must ultimately be assigned: one for the Hercules end of the link and another for the driving system's (host systems's) end of the link (i.e. TunTap32.dll + Windows).

Note that the format of the Hercules configuration file statements for all of the networking emulations has changed from the previous releases of Hercules. The older 'CTCI-W32' format has been deprecated. Please ensure your CTC device statements are updated to use the newer 'CTCI' (or 'LCS') format in order to remain compatible with current and future releases of Hercules.

For example:

  0E20-0E21   CTCI   192.168.0.4   192.168.0.2

The first IP address (192.168.0.4) is the IP address of your Hercules system. The second IP address (192.168.0.2) identifies to the CTCI-W32 logic (i.e. FishPack.dll) the network adapter that CTCI-W32 is to use in order to reach Windows's TCP/IP stack.

It is also very important that the IP address you choose for your virtual guest (the '192.168.0.4' in the the above example) be in the same subnet as your Windows host in order for CTCI-W32 to work properly. Basically the way you need to approach things (think of things during configuration) is to configure your guest just as if it were a real system connected to your network. Thus it should normally be assigned an IP address within the same subnet as all the other workstations on your local network.

Note however that it is possible however, to place it into a subnet separate from your Windows host if you're prepared to define the proper routing entries, but it's usually a lot simpler and less problematic to keep things simple and set it up to be in the same subnet unless you know what you're doing.

If your network adapter does not have a permanent (static) IP address assigned to it (i.e. you instead use DHCP to have an IP address dynamically assigned to it instead), then instead of specifying an IP address as the second parameter in the Hercules device statement, you need to specify the MAC address of the adapter you wish to use instead:

  0E20-0E21   CTCI   -n  00-80-B3-E1-DF-69   192.168.0.4   0.0.0.0

Most people find it more convenient to assign their own IP addresses rather than mess with DHCP however.

(And in case you're wondering, the 0.0.0.0 in the above example is simply a placeholder for where the ip address of the adapter would normally go. Two IP addresses always need to be specified on the configuration statement in order to be syntactically correct, but since you don't wish to specify a specific IP address, you use 0.0.0.0 instead. The adapter you wish to use is identified by the "-n <mac-address>" parameter.)

The procedure for configuring TCP/IP on whatever guest operating system you plan to run under Hercules varies depending on what operating system you plan to use of course. The procedure is completely different for VM than it is for OS/390 for example. For detailed instructions on how to configure MVS's TCPIP PROFILE for using a CTCI device, please refer to question #17 of the CTCI-W32 FAQ: "MVS TCPIP configuration using CTCI-W32".

If you're interested in how to set things up for using an 'LCS' (LAN Channel Station) device instead (which is highly recommended since they can handle any Ethernet packet and not just 'IP' packets), then you might be interested to know that the following setup does indeed work.

            -------------------------------------------------------------------------
            Sample regular / normal LCS device definitions...


             0E20-0E21  LCS  -n  172.16.9.163   -m  00-00-5e-90-09-5d   172.16.9.93



            -------------------------------------------------------------------------
            Sample LCS device definitions for Enterprise Extender...


             0E20-0E23  LCS  -n  172.16.9.163   -o  my_oatfile.txt



            -------------------------------------------------------------------------
            my_oatfile.txt:


              *********************************************************
              * Dev  Mode Port Entry specific information             *
              *********************************************************

                0E20  IP   00   PRI   172.16.9.93
                HWADD      00         00-00-5E-90-09-5D


                0E22  IP   01   SEC   172.16.10.93
                HWADD      01         00-00-5E-90-0A-5D



            -------------------------------------------------------------------------
            Sample TCPIP PROFILE statements for Exterprise Extender...

            (Note: not all statements are shown)


             IPCONFIG  DATAGRAMFWD  VARSUBNETTING  SYSPLEXROUTING

             DEVICE  LCS1     LCS       E20  AUTORESTART
             LINK    ETH1     ETHERNET  0    LCS1

             DEVICE  VDEV1    VIRTUAL   0
             LINK    VLINKA   VIRTUAL   0    VDEV1

             DEVICE  IUTSAMEH MPCPTP
             LINK    EELINK   MPCPTP         IUTSAMEH

             START   LCS1
             START   IUTSAMEH

             HOME

              172.16.9.93   ETH1
              172.16.10.93  VLINKA

             BEGINROUTES

              ROUTE  172.16.0.0  255.255.0.0  =             ETH1  MTU 1492
              ROUTE  DEFAULT                  172.16.13.1   ETH1  MTU 1492

             ENDROUTES

             PORT

              12000 UDP VTAM
              12001 UDP VTAM
              12002 UDP VTAM
              12003 UDP VTAM
              12004 UDP VTAM

            -------------------------------------------------------------------------
            (Missing / not shown: VTAM definitions)

Your guest's "default gateway" depends on whether or not you are using a real router to perform your routing. If you don't have a real router and are instead using the Windows "IP Forwarding" (IP Routing) feature to perform your routing, then your default gateway should be the physical adapter on your Windows system that your virtual interface is using.

Otherwise if you do have (are using) a real router, then your default gateway should obviously be the IP address of your actual router. In the sample LCS configuration immediately above, '172.16.13.1' is a real network router (someplace else on the same physical network that Hercules's Windows host is running on) and thus knows how to route traffic to the other end of the guest's Exterprise Extender link. Note too that, as previously suggested, the guest's IP addresses (172.16.9.93 and 172.16.10.93) are within the same subnet (255.255.0.0) as the Windows host that Hercules is running on (172.16.9.163).

With the above setup this user had no problem whatsoever having Exterprise Extender completely saturating a T1 line running from the west coast all the way cross country to the east coast.

The CTCI-W32 protocol also supports some additional tuning parameters in addition to the above mentioned required parameters.

For example, you can adjust both the size of the WinPCap kernel device driver's internal packet buffer as well as the size of TunTap32.dll's own packet I/O buffer to try and maybe tweak some additional speed out of your network for example:

  0E20-0E21   CTCI   -k 1024   -i 64   192.168.0.4   192.168.0.2

The numbers 1024 and 64 on the above statement are the size of the WinPCap kernel device driver's internal packet buffer (in K) and the size of TunTap32 dll's internal packet I/O buffer (in K). Please note that because the memory for WinPCap's kernel device driver's packet buffer is taken from Windows's "non-paged" pool memory (i.e. from your system's real physical memory), the size of this buffer has a direct impact on Windows performance. If you choose a ridiculously large number here, Windows will then have very little memory to work with and the overal performance of your Windows system will quite likely suffer, so choose your value wisely.

The second number (64 in the above example) is the size in K of the I/O buffer inside of TunTap32 dll.  This defines how much data TunTap32 will ask to retrieve from the device driver each time it needs to do an I/O on the physical adapter (i.e. it defines how much data will be transferred from the device driver's internal buffer to TunTap32's I/O buffer). The memory for this buffer however is virtual memory (allocated from the Hercules process's virtual address space) and thus should not impact Windows's performance to any large degree (unless you choose such a ridiculously large number that Windows begins thrashing).

The TunTap32 dll feeds packets back to Herc one at a time, and when it runs out of packets in its internal I/O buffer, it asks the WinPCap device driver (via the FishPack dll) for another bunch of packets.  This number defines how much data (and utimately how many packets) it will get from WinPCap (via FishPack) each time it asks for packets (if there are any available at all, of course).

The larger this buffer is, then the fewer actual I/Os TunTap32 will have to do to WinPCap via FishPack (which is good because such I/Os can be expensive), but at the same time the longer the actual time interval will be between those I/Os too, meaning that the WinPCap device driver will have to buffer its data for a longer period of time between those infrequent I/Os, thereby requiring you to specify a larger kernel buffer in order to ensure that no packets are ever lost during periods of high network activity.

On the other hand, the larger the actual I/O (i.e the more bytes that are transferred per I/O), the longer each I/O is going to actually take too.  Yes we're only talking a few microseconds here, but we're holding up the entire Windows operating system when you do it too (transferring memory from kernel-land to user-land), so choose you values wisely.

Unless you're using a Gigabit or faster network it's usually best to just leave them off altogether and let them default to their default values.

The currently implemented minimum, maximum, and default values are:

If you do decide to try and adjust these buffer size values in order to try and increase network throughput/performance however, you might find the Hercules tt32 stats command to be useful:

23:59:11.098 0000066C tt32 stats e20
23:59:11.098 0000066C TunTap32.dll Statistics:
23:59:11.118 0000066C Size of Kernel Hold Buffer:       1024K
23:59:11.118 0000066C Size of DLL I/O Buffer:             64K
23:59:11.128 0000066C Maximum DLL I/O Bytes Received:      2K
23:59:11.138 0000066C            7 Write Calls
23:59:11.148 0000066C            8 Write I/Os
23:59:11.148 0000066C          319 Read Calls
23:59:11.158 0000066C          259 Read I/Os
23:59:11.168 0000066C          282 Packets Read
23:59:11.168 0000066C            8 Packets Written
23:59:11.178 0000066C        38172 Bytes Read
23:59:11.178 0000066C          542 Bytes Written
23:59:11.188 0000066C            1 Internal Packets
23:59:11.198 0000066C            2 Ignored Packets

If the reported "Maximum DLL I/O Bytes Received" value is identical to the defined "Size of DLL I/O Buffer" value, then that means each time TunTap32 asked for more packets, there were at least a full buffer's worth (OR MORE!) sitting in the device driver's buffer, waiting to be delivered to TunTap32. This would tend to indicate that the default buffer sizes were too small and that your network happens to be very busy. (That is to say, in my own experience, the previously mentioned situation is rather unusual! Rarely is TunTap32 unable to deliver an entire buffer's worth of packets to Hercules before another buffer's worth arrives, unless either Hercules is sick, or your network is very busy).

Well, that's it I guess.  If you have any problems, check the FAQ.  If your answer isn't in there, then post your question to the list (i.e. ask the group for help) or feel free to send me an email.

Enjoy!   :)