The previous version of CTCI-W32 contains a serious flaw in its initialization logic that causes it to enter into a "soft deadlock" situation when used with WinPCap version 3.1 or greater. Symptoms include tt32info and tt32test (i.e. 'test.exe') hanging for several seconds and then exiting without actually doing anything (i.e. without actually ever starting up at all), as well as CTCI-W32 not functioning properly (if at all) when used with Hercules.
The technical details of the flaw (as well as how it was accidentally introduced) are largely unimportant (except perhaps to Windows programmer geeks) compared with the importance that it has now been identified and fixed.
This new version of CTCI-W32 should now work properly with WinPCap 3.1 and, more than likely, all future versions of WinPCap as well (until such time as they make a major change requiring me to add additional support for it, that is).
Thus, if you wish to take advantage of WinPCap's latest capabilities (whatever they may be) you must use this new 3.1 version of CTCI-W32 and not any of the older ones.
to work around an as-yet unidentified/unresolved bug involving TunTap32.dll's inability to directly set the caller's 'errno' variable. These functions are identical to all of the existing functions except that there is a new pointer variable argument passed as the last argument. To use any of these new functions, the caller should pass in this new last argument the address of an integer variable which TunTap32.dll then sets to the proper 'errno' value that the caller should then set its own errno variable to upon return. (i.e. the easiest way would be to point it directly to your module's (executable's/dll's) errno var so TunTap32 ends up directly setting it [to its proper value])
One of either O_RDONLY, O_WRONLY or O_RDWR must be specified. Specifying any other flags causes an error. (impacted tuntap32 calls: tuntap32_open, tuntap32_open_ip_tuntap) The flag is honored too. If the interface is opened O_RDONLY then any writes will cause an error and vice versa. (impacted tuntap32 calls: tuntap32_read, tuntap32_write, tuntap32_read_ip_tap, tuntap32_write_ip_tun)
Both are now specified in bytes. (Previously, the 'Get' functions returned the value in 'K'. They now both return the value in bytes, the same format that is passed to the 'Set' function). (impacted ioctl codes: TT32GDEVBUFF, and TT32GIOBUFF):
Since the timeout value is specified in milliseconds, it is thus returned the same way: in milliseconds. (impacted ioctl codes: TT32GTIMEOUT)
IFF_RUNNING means the interface is physically operational and is set internally by TunTap32 whenever the interface is succesfully initialized. If IFF_RUNNING is specified, it is silently ignored without error. (impacted ioctl codes: SIOCSIFFLAGS)
IFF_RUNNING means interface is physically operational. If it is not set, then the interface cannot be enabled (IFF_UP) because it isn't operational. If IFF_UP is specified but IFF_RUNNING is not set, then an EACCES error is returned. (impacted ioctl codes: SIOCSIFFLAGS)
to define the interface type to either IFF_TUN or IFF_TAP. (impacted ioctl codes: SIOCSIFFLAGS)
Previously the flags passed to SetIFaceFlags would be combined (ORed) with existing (previously set) flags. Now whatever flags you pass are used to completely replace existing flag settings (i.e. subsequent calls override previous calls). This means that if you call SetIFaceFlags multiple times, only the flags set in the very last call are actually used. (impacted ioctl codes: SIOCSIFFLAGS)
WARNING! The output is very verbose! Every read/written packet is traced. (impacted ioctl codes: SIOCSIFFLAGS)
(to reflect the new IP address since the default virtual interface MAC address is based on its IP address). This automatic updating of the interface's MAC address only occurs if the default MAC address hasn't already been manually changed to some other value via a prior call to SetHercVirtMACAddr (tuntap32_ioctl(SIOCSIFHWADDR)). If a prior call was made to manually set the MAC to some other value then that value is honored and the MAC address is then not automatically changed. (impacted ioctl codes: SIOCSIFDSTADDR)
ioctl(SIOCSIFADDR) cannot be used to "switch" to using a different IFace (host physical adapter) once an interface has been 'open'ed. Attempts to do so will be ignored and result in an error if the ip address specified in the call is not zero (meaning "don't care") and different from the one currently being used.
That is to say, calls which specify either zero as the ip address, or which specify the same ip address as would be returned by a SIOCGIFADDR ioctl call, will continue to succeed as normal (since the iface (host physical adapter) is not actually being changed). Specifying any other non-zero ip address different from the one used in the 'open' call however, will be rejected and result in an error.
It defines the size of your write buffer (i.e. the maximum number of bytes that you can write to the adapter). (impacted ioctl codes: SIOCSIFMTU; impacted tuntap32 calls: tuntap32_write, tuntap32_write_ip_tun)
The buffer size will be automatically reallocated larger (if possible) whenever needed. If automatic reallocation fails however, then a warning is issued and the packets that wouldn't fit into the smaller buffer are discarded. (impacted calls: ioctl(TT32SIOBUFF) and tuntap32_read)
You must manually call SetIFaceFlags with IFF_UP to enable the interface (i.e. "bring it up" or "enable it"). (impacted tuntap32 calls: tuntap32_read, tuntap32_write, tuntap32_read_ip_tap, tuntap32_write_ip_tun, and, indirectly, tuntap32_ioctl code: SIOCSIFFLAGS)
in the Ethernet header that it automatically builds for TUN interfaces. It now scans internal objects for all known real/virtual MAC addresses, queries Windows via GetBestInterface, etc, including actually sending an ARP packet out on the wire via the the Windows 'SendARP' API.
Only when all such attempts fail does it then fall back to using the prior logic of relying on on Windows' "IP Forwarding" feature to route the packet. (impacted tuntap32 calls: tuntap32_write, tuntap32_write_ip_tun)
Previously it wasn't even checking to see whether its TunTap calls were successful or not. Oops! (impacted tuntap32 calls: tuntap32_ioctl)
(** RESCINDED in v3.1.7 **) which allow one to directly manipulate Window's routing tables.
(impacted tuntap32 calls: tuntap32_ioctl)
to allow one to directly manipulate Window's ARP tables. (impacted tuntap32 calls: tuntap32_ioctl)
is deprecated since it never contained any useful information anyway (its value was always zero). Additionally, the structure was extended to include some new fields pertaining to checksum offloading (see next item below), a new flag indicating whether a networking cable is actually plugged into the adapter or not, and, for WinNT variant operating systems only, the IP Helper "adapter index" value.
meaning you may now freely enable whatever Task Offloading capabilities your adapter may happen to support (including Checksum Offloading, since TunTap32 has now been coded to both detect and compensate for it).
Due to the current lack of Checksum Offloading support in WinPCap however, Checksum Offloading will never actually be used (even if your adapter happens to support it) in the following two situations:
- For packets sent by your virtual guest,
- For packets received by your virtual guest when sent directly from your Windows host to your virtual guest.
That is to say, normally packets sent by other systems to your guest can have the verification of their checksums offloaded to the host's adapter that receives them (since they obviously have to pass through it and are thus able to take advantage of your adapter's 'Receive' Task Offloading capabailities), but in the case of your host sending packets to your guest, Checksum Offloading cannot occur since such packets never actually pass through the physical hardware (adapter) and are instead intercepted by WinPCap and passed directly to TunTap32 for delivery to your virtual guest.
Thus, regardless of whether or not your adapter happens to support 'Receive' Checksum Offloading, all packets sent from your Windows host to your virtual guest will never be able to taker advantage of the Checksum Offloading capabilities of your adapter (even though from your virtual guest's point of view the packets are indeed being received and not sent).
Furthermore, due to the very nature of the way Task Offloading is designed and implemented at the Windows device-driver level, all such packets (sent from your Windows host to your virtual guest) will incur a slight (but hopefully not noticible) performance penalty as a result of TunTap32 having to manually re-calculate the checksums for all such packets (since your Windows host never did so since it was naively expecting the hardware to do it).
All packets sent to your virtual guest from other systems however (i.e. external to your Windows host, such as from other workstations on your local LAN or from the Internet) should not incur such a penalty however, since they should already have proper checksum values (by virtue of the fact that they were never sent from your host's but rather only received by it).
(Clear as mud, right? <g>)
and now supports testing on two different TunTap32 interfaces at the same time (the first one is a TUN interface by default (like before) and the second one is (by default) a TAP interface), thereby allowing testing of both TunTap32's TUN logic as well as its TAP logic at the same time (as well as how well it behaves when combinations of both are used at the same time).
Detailed (very verbose) debug tracing messages are also provided to both verify and debug proper TunTap32 functioning, as well as it now displays, for each adapter, whether or not checksum offloading is enabled (although due to the previously mentioned change that shouldn't be an issue anymore).
that displays all trace/testing messages while testing is active (so you can see what's going on) and now no longer asks for an o/p file filename until the application is closed (at which point you can then specify the filename of what file you wish to save the test messages in, if so desired).
Also, because the view is now an edit view, you can now copy the test messages directly from the main window for pasting, if needed, into some other program or document (such as an email asking for help for example).
for each TUN and TAP interface that it creates, just like Hercules does. The entries are of course automatically removed whenever the interfaces are closed.
that is responsible for routing traffic through your LAN. If your LAN does not have a router however, then you will still need it of course (so that your virtual guest's packets can be routed properly through your network).
Additionally, it should it mentioned that if you are using a real router to route traffic on your LAN, then, technically, you should not ever enable 'IP Forwarding' on your Windows host. If you do, double packets will result as a consequence of both your router and Windows both attempting to route traffic to your virtual guest. (Note that due to the way networking is designed to operate however, such duplicate packets should not actually cause any problems other than the obvious slight degradation that would result due to the extra redundant packets flowing through it thereby needlessly consuming available bandwidth).
As a result, CTCI-W32 now ships in two flavors: normal and UNICODE. At the moment both are logically identical to one another however (and thus should behave identically too), so it currently does not matter which one you use. In the future however, the UNICODE versions may (should I decide to do it) be customized for different languages and thus be different. (For now though they're the same)
since they are all always packaged together as one "product" anyway. In the future, whenever a new release of an individual component is needed, all components will be released together with a uniform updated new version number (in order to make it easier to determine whether or not you have the most current version of each component installed or not).
have been changed to 0x400000 (4MB) in order to maximize available virtual address space.
Previously they were using the Windows default (which is currently 0x10000000 (256MB)), thereby causing valuable potential address space to be wasted due to fragmentation. Using a preferred base of 4MB however, potentially frees up an additional 230MB+ of valuable virtual address space, thereby allowing more memory to be used by your applications rather than having it wasted due to memory fragmentation. (Refer to the new "Maximizing Hercules Available Memory" web page for more information on rebasing Windows DLLs to increase available memory.)
unoriginally called 'FishLib.dll'. The FishLib DLL is simply a collection of routines and functions common to most of the programs I write and splitting that common functionality out into a separate maintainable module makes it easier for me to maintain all of my products without having to duplicate my efforts.
CTCI-W32 now supports the new 3.0 version of WinPCap. Version 3.0 of WinPCap includes support for running WinPCap (and thus CTCI-W32) on SMP (multi-processor) systems, so if you have such a system, you can now take advantage of that with this new version of CTCI-W32.
The "virtual adapter" (hardware interface) that TunTap32 "creates" [via responding to ARP request packets; see readme for more information] now actually does what the documentation mistakenly already said it was doing.
This one I have to apologize for. I mistakenly updated the CTCI-W32 readme documentation with information regarding the how the MAC address is supposedly now constructed (last 3 octets of destination IP address used to build last 3 octets of hardware interface MAC address), but the only problem was, that wasn't how the current version actually did it. I mistakenly updated the documentation with information ahead of time. Sorry about that.
Anyway, this new release now actually does what the documentation [prematurely] said it was already doing (i.e. last 3 octets of IP becomes last 3 of MAC, etc). Again, my apologies for getting ahead of myself.
Previously, if, for whatever reason, FishPack was unable to auto-start the WinPCap kernel driver service, it prematurely returned from its initialization function thereby causing all subsequent requests to fail. This has been corrected.
I don't believe this minor bug was actually causing anyone any problems (although I have no way of knowing for sure) since it is quite unlikely that FishPack would be unable to auto-start WinPCap's kernel driver service as long as WinPCap was properly installed, but once again I feel compelled to mention it anyway just in case.
(self explanatory)
The 'tt32stats xxx' Hercules command should now work correctly once Hercules 2.18 is released.
The 'tuntap32_open_ip_tuntap' function, while deprecated (one should use Jim's new TUNTAP compatible 'tuntap32_open' function instead), was (and still is) being used by my silly 'tt32test' program, but it hadn't been updated to construct the MAC address in the new format explained further above. This has now been corrected. (Note: this change shouldn't really affect anyone (at least not Herc functionality anyway), so it's really a pretty minor issue. Just thought I'd mention it just in case it did somehow affect someone/something).
Thanks to Jim Pierson's truly Herculean effort, Hercules and CTCI-W32 now support the LCS (LAN Channel Station) protocol.
This new version of TunTap32.dll is now required if you wish to use the CTCI-W32 feature of Hercules starting with version 2.16a3 of Hercules (the pre-release CVS version of Hercules version 2.17) since it contains the many new (and now standardized) DLL entry points that Hercules now uses, but should be backward compatible with previous releases (versions) of Hercules as well (i.e. you should be able to use this new version of TunTap32.dll on older CTCI-W32-compatible versions of Hercules as well as the new 2.16a3/2.17 version too).
Jim put a lot of hard work into this version of TunTap32, standardizing the various entry points and basically trying, as much as possible, to make it as compatible as possible to the way one uses tuntap on Linux/Unix systems (via ioctl calls, etc). This release of TunTap32.dll is the result of his work and not mine, so we should all thank him for his efforts. Thanks, Jim! :)
Ever since I originally released CTCI-W32 it has worked just fine on local networks (LANs), but has never worked when going through a router. You could access your Herc system just fine from any station on your LAN, but whenever you'd try accessing it from outside your LAN it just wouldn't work. People using routers noticed this aggravation right away and no matter what we tried doing, nothing would resolve it.
But no more! THE bug has been found. (thanks to Don Thimsen)
Don was thoughful enough to do some packet captures and examine the actual traffic going over the network in his attempt to try and figure out what was going on. What he saw was incorrectly constructed ARP response packets wherein the 'target' IP address in the ARP response packets (built and sent by TunTap32.dll) had the wrong IP & MAC addresses in them. Instead of having the IP & MAC address of the original requestor (like they should have), they had the IP & MAC address of Herc in them.
The router, upon receiving such a packet, would then route the packet right back to Herc because, according to the IP address, that's who was supposed to receive it. The 'target' IP address was Herc so the router dutifully "routed" the packet right back to Herc since that's what TunTap32 was [incorrectly] telling it to do.
Anyway, once I fixed this glaring bug, the ARP packets started getting properly routed back to their intended receiver (i.e. the workstation on the other physical network on the other side of the router) and CTCI-W32 began magically working just fine. You could now access your Herc machine from outside your local network (i.e. through a router).
I apologize profusely for the agravation this bug has obviously caused to those people trying to get Herc networking (CTCI-W32) working with their routers. I should have tested my product a little more thoroughly before releasing it (I should have examined the packets my program was generating to make sure they were correct) but I didn't. I simply tried "talking" to Herc from each workstation on my local network (LAN) and when that seemed to work just fine, I figured all was well. I suppose if I had had a router at the time (which I didn't) I might have caught the problem sooner, but since I didn't, I didn't.
Anyway, the bug has now been squashed (fixed) so all you people with routers who have been pulling your hair out trying to get Herc working beyond your local network should now have no trouble doing so with the new TunTap32. :))
Again, ever since day one, many people have reported problems getting CTCI-W32 working for them after installing WinPCap. What would all too frequently happen was the person would install WinPCap and then immediately try to use CTCI-W32 only to discover that it just wouldn't work. The reason it wouldn't work for them is explained in hercules-390 group message 22877: simply installing WinPCap isn't enough.
WinPCap's installation process/program fails to create the packet driver service (the "NPF (Netgroup Packet Filter)" kernel driver service) that every program using WinPcap needs. Without this service, using WinPcap is impossible (because the service that needs to be started (in order to use it) doesn't exist yet).
This issue doesn't affect most any other product that uses WinPcap because the first thing all of them do is call WinPcap's Packet.DLL function "PacketOpenAdapter", and the "PacketOpenAdapter" function in their Packet.DLL is coded to always check to see if the service has been created yet and creates it if it hasn't been. That's why message 22877 explains why it's necessary to always run their 'testapp' utility at least once after first installing WinPCap: so that the "WinPCap installation" can be completed (i.e. so the required service can get created. Hardly does any good to try and start a service if the service to be started doesn't even exist yet)
But CTCI-W32 is different. It doesn't use (or rather, didn't use (until now)) their [crappy] Packet.DLL. Instead, it uses its own DLL to make device driver calls: FishPack.
Until now, FishPack would simply attempt to start the driver service that it needed to use, correctly (but naively) relying on WinPCap to properly install its own product (creating the necessary services a product needs is something the product's installation program should do, since services only need to be created once and never again. Once they're created, they then exist and can thus, from then on, be started any time the product in question needs to use the service).
But as I said, it was awfully naive of me to rely on Politecnico di Torino to do something correctly. :(
Anyway, FishPack now makes a "dummy" call into the WinPcap Packet.DLL's "PacketOpenAdapter" function at startup to force WinPCap to finish its installation process and create the needed kernel driver service, so CTCI-W32 users now no longer need to manually run WinPCap's 'testapp' utility before trying to use CTCI-W32. CTCI-W32 should work just fine immediately after installing WinPcap so new CTCI-W32 users should no longer get confused as to why CTCI-W32 just isn't working after [seemingly] properly installing everything.
These two issues alone -- the above ARP routing issue and this one -- should hopefully greatly reduce the volume of support emails we receive since they were a significant portion of them. (I don't know why I hadn't thought of doing this (calling into Packet.DLL) in the first place, but hindsight is always 20-20, isn't it? (sigh) >;-)
Without getting into a lot of technical details, the Politecnico di Torino folks changed something else in their version 2.3 device driver besides just the name of the driver: they also completely reversed the meaning (and thus the behavior) of their the query/set OID i/o control codes used by the Win9x version of their device driver.
This had the immediate effect of causing all programs that communicated directly to their device driver (such as as FishPack.dll) to no longer work. Worse, it ended up causing such programs to end up doing things they never actually intended to do (such as setting certain device driver values instead of simply querying them).
FishPack.dll was changed to detect the fact that it is running on a system with version 2.3 or newer of WinPCap installed and to react appropriately. Note that this fix only affects WIn9x users of CTCI-W32, and then, only those users who use CTCI-W32 with version 2.3 of WinPCap. Win2K/XP users and users still using the older version (2.2) of WinPCap are unaffected by this fix.
Minor fixes to issue additional debugging information in all modules (FishPack, TunTap32, tt32info and tt32test).
There was a bug in the original release of FishPack.dll that caused it to not recognize more than one adapter on Win9x (98/Me) systems. The bug was such that FishPack would only ever return "the 'last' adapter" whenever TunTap32 or tt32info asked it for a list of available adapters to use. This resulted in preventing users with more than one adapter from being able to use any of the "non-first" ones.
What constitutes a "non-first" adapter (or "last" adapter) depended entirely on the order in which the system returned its list of adapters back to FishPack and thus is hard to explain. The order was always consistent (which adapter was returned first and which adapter was returned last remainded the same from one run to the next), but for any given system, which adapter ended up being the first to be returned and which ended up being the last to be returned was completely unpredictable. (I think it had to do with the order in which they were installed).
In any case, this bug has been corrected in version 1.0.1.286 of FishPack.dll so Win9x users should now be able to use whatever other adapters they might also have installed (as long as WinPCap is able to use them of course. CTCI-W32 can only use adapters that WinPCap can use since CTCI-W32 uses WinPCap to do what it does.)
The original release of CTCI-W32 would [apparently] not work at all on WinXP systems even though I said it would. I sincerely apologize for that.
I thought that since WinPCap supported Windows 2000, that it also supported Windows XP too, but I was wrong. According to Politecnico di Torino's web site, WinPCap only works on Windows XP systems starting with their new 2.3 version. (See question #14 of their FAQ.)
Furthermore, starting with their 2.3 version, they switched to using a different name for device driver (see next item just below) so even if you did happen to install their new 2.3 version on your Windows XP system, CTCI-W32 still wouldn't work.
But that's fixed now, and again I apologize to all WinXP users for whatever agravation/confusion this oversight has caused you.
As briefly mentioned in the previous item just above, the makers of WinPCap (Politecnico di Torino) just recently came out with a new 2.3 version of their product. Starting with this new version, however, they began calling their device driver by a different name.
In versions prior to 2.3 (i.e. version 2.2 and earlier), their device driver was named "packet.sys" (on WinNT/2K systems) and "packet.vxd" (on Win9x/Me systems). Thus, my original version/release of CTCI-W32 was coded to load and start the device driver by that name.
Starting with version 2.3 however, they began calling their device driver "npf.sys" and "npf.vxd". Thus, all CTCI-W32 users who just so happened to install version 2.3 of WinPCap would find that CTCI-W32 just wouldn't work for them.
Some users managed to figure out on their own that falling back to the previous 2.2 version of WinPCap resolved their inability to get CTCI-W32 working (and I applaud those users that figured that out for themselves and thus managed to solve their problem for themselves), but many others I'm sure scratched their head and/or pulled their hair out trying to figure out why the heck CTCI-W32 just wouldn't work for them. (Note that this category of user includes all CTCI-W32 users -- Windows 98, Windows Me and Windows 2000 users -- as well as Windows XP users. This issue -- WinPCap's new driver naming scheme starting with version 2.3 -- affects all CTCI-W32 users and not just Windows XP users.)
What all this means to you, the user(s) of CTCI-W32, is this: if you're still using the original version of CTCI-W32 with version 2.2 of WinPcap, then everything should be working okay for you and should continue to work okay for you as long as you don't upgrade to version 2.3 of WinPCap.
(Note: I honestly have no idea whether or not there is any advantage to upgrading to the new version of WinPCap, but if you feel you must (and, as explained in the previous item, Windows XP users must use version 2.3 of WinPCap), then what follows in the next paragraph applies to you.)
As soon as you upgrade to version 2.3 of WinPCap, however, then you'll also need to upgrade to version 1.0.1.286 of FishPack.dll in order for CTCI-W32 to continue working for you. The new 1.0.1.286 version of FishPack.dll contains the necessary modifications to load/start the new/differently named WinPCap device driver that CTCI-W32 relies on, and the older original release version of CTCI-W32 will not work with the new 2.3 version of WinPCap.
Let me say that again: the older (original release) version of CTCI-W32 will not work with the new 2.3 version of WinPCap.
The new version of CTCI-W32 (FishPack.dll version 1.0.1 build 286) however, should work with the new 2.3 version of WinPCap, and is coded differently (i.e. correctly) so as to be completely unaffected by Politecnico di Torino's renaming their device driver again at some point in the future (should they ever decide to do that).
FishPack.dll is now coded the way it should have been all along, so that Politecnico di Torino can now rename their device driver to whatever the heck they want to from now on, and the current version of FishPack.dll will still be able to load and start it just fine, so hopefully this shouldn't ever be an issue again.
And again, I apologize profusely for whatever confusion and/or agravation this issue has caused anyone. I'm not perfect ( yet >;-> ), and thus mistakes do unfortunately (still) happen from time to time, so I greatly appreciate your not getting too upset whenever they do.
;-)
"Fish" (David B. Trout)
fishsoftdevlabs.com
"Programming today is a race between
software engineers striving to build bigger
and better idiot-proof programs, and the
Universe trying to produce bigger and better
idiots. So far, the Universe is winning."
- Rich Cook