Maximizing Hercules Available Memory

Introduction

This document provides instructions and hints on how to maximize the amount of available memory on Windows. It is the result of one day's effort researching the subject from a Hercules development/developer point-of-view, but pertains in general to any/all Windows application development.

It is not formatted very well because, well, I'm basically lazy. So sue me. :)

Windows Memory Layouts

The following is how Windows carves up each process's 4GB address space on 32-bit Windows:

Win9x memory layout 0x00000000 - 0x00000FFF (4KB) MS-DOS & 16-bit Windows (inaccessible; reserved for NULL ptr assignments) 0x00001000 - 0x0003FFFF (4MB) MS-DOS & 16-bit Windows (read/write, but don't touch) 0x00400000 - 0x7FFFFFFF (2GB) Private to Win32 processes (unreserved, usable) 0x80000000 - 0xBFFFFFFF (1GB) Memory-mapped files, shared Win32 DLLs, 16-bit apps, and memory allocations; shared by all Win32 processes. (usable, read/write) 0xC0000000 - 0xFFFFFFFF (1GB) VxDs, memory manager, and file system code; shared by all Win32 processes. (read/write, but don't touch)

                      WinNt memory layout


  0x00000000 - 0x0000FFFF   (64K)   (inaccessible; reserved
                                    for NULL ptr assignments)

  0x00010000 - 0x7FFeFFFF   (2GB)   Private to Win32 processes
                                    (unreserved, usable)

  0x7FFF0000 - 0x7FFFFFFF   (64K)   (inaccessible; reserved
                                    for NULL ptr assignments)

  0x80000000 - 0xFFFFFFFF   (2GB)   operating system use
                                    (inaccessible)

As you can see, even though each process has 4GB of address space (32 bits), only slightly less than 2GB (31 bits) is actually available to any given application. The rest is reserved for operating system use and can't be used.

To further complicate matters, just as with most operating systems, any application (any binary image actually (i.e. .EXE, .DLL, etc)) can define, at link time, a "preferred base address" (i.e. the preferred load address, i.e. the address at which the image in question prefers to be loaded at).

The reason images (.EXEs and .DLLs, etc) have defined preferred base/loading addresses is simply to relieve the operating system from having to relocate the image to another spot in memory. At link-edit time (when the image gets built), all of the program's address constants, etc, are pre-initialized with whatever preferred base address and offset you define on your LINK statement. That way if the image can be loaded at your preferred base address (load address), the operating system has nothing else to do; all address constants, etc, are already what they should be.

If the image cannot be loaded at the preferred base address (load address) however, then the operating system, after loading it at a different address, must then adjust all of the addresses in the entire image (which I suppose with applications that do a lot of image loading (and running on slow systems) can take a few extra microseconds to do).

Today, however, such things are not really a concern, but unfortunately Widows is still designed to care about it. :(

Thus, a default build always creates EXEs with a "preferred base address" of 0x00400000 (4MB), since that's the lowest address an image can be loaded at on Win9x. If you try specifying a smaller (lower) base address the linker will complain with a warning that "the resulting executable may not run on Win9x."

Additonally, the default "preferred base address" for DLLs is (for some REALLY odd reason I cannot fathom) 0x10000000 (256MB)! Thus, up to 256MB of valuable virtual address space is pretty much completely wasted (since it'll never be used for a malloc heap and instead only as room to load additional dlls into).

Why Microsoft decided on this particular value I'll never know.

The 'vadump' report

The SDK utility vadump -sv -p <pid> can report a process's memory layout. Running it on a default MSVC build of Herc reveals the following memory layout on my system:

Note: the default build for MSVC has probably since been changed since the time this document was written. The below information thus might no longer be accurate. It is included as-is for illustrative purposes only.

Note: the vadump utility, amazingly, does not display its report in any meaningful sequence. The below displays were manually sorted into ascending virtual address sequence.

PRODUCTION SYSTEM (before rebase) 00230000 : 0023F000 hsys.dll 00240000 : 00276000 hdasd.dll 00280000 : 00290000 hutil.dll 00290000 : 002A3000 zlib1.dll 002B0000 : 002CE000 LIBBZ2.dll 00400000 : 00406000 Hercules.exe 00BC0000 : 00BC6000 hdteq.dll 00BD0000 : 00BEC000 dyncrypt.dll 00BF0000 : 00BF7000 dyngui.dll 00C00000 : 00C09000 hdt3505.dll 00C10000 : 00C16000 hdt3525.dll 00C20000 : 00C27000 hdt1403.dll 00C30000 : 00C3B000 hdt3270.dll 02800000 : 028C1000 DbgHelp.dll 10000000 : 101BA000 hengine.dll 43000000 : 43005000 GoogleDesktopNetwork1.dll 68590000 : 685A6000 rsvpsp.dll 68C40000 : 68C49000 RAPILIB.dll 74FD0000 : 74FEE000 msafd.dll 75010000 : 75017000 wshtcpip.dll 75020000 : 75028000 WS2HELP.DLL 75030000 : 75044000 WS2_32.dll 759B0000 : 759B6000 LZ32.DLL 77820000 : 77827000 VERSION.dll 77D30000 : 77DA8000 RPCRT4.dll 77E10000 : 77E6F000 USER32.dll 77F40000 : 77F7C000 GDI32.dll 77F80000 : 77FFC000 ntdll.dll 78000000 : 78045000 msvcrt.dll 7C2D0000 : 7C335000 ADVAPI32.DLL 7C370000 : 7C409000 MSVCR80.dll 7C570000 : 7C623000 KERNEL32.dll

Using 'rebase'

Adding the command: " rebase -b 0x400000 $(X)*.dll" (without the quotes of course) immediately following the: "all: allzlib alllibbz2" build-rule in the dllmod makefile results in the following memory layout:

PRODUCTION SYSTEM (after rebase) 00370000 : 0038C000 dyncrypt.dll 00400000 : 00406000 Hercules.exe 00420000 : 00427000 dyngui.dll 00450000 : 00486000 hdasd.dll 004A0000 : 004A7000 hdt1403.dll 004D0000 : 004DB000 hdt3270.dll 004F0000 : 004F9000 hdt3505.dll 00500000 : 00506000 hdt3525.dll 00510000 : 00516000 hdteq.dll 00530000 : 006EA000 hengine.dll 006F0000 : 006FF000 hsys.dll 00710000 : 00720000 hutil.dll 00720000 : 0073E000 LIBBZ2.dll 00740000 : 00753000 zlib1.dll 02800000 : 028C1000 DbgHelp.dll 43000000 : 43005000 GoogleDesktopNetwork1.dll 68590000 : 685A6000 rsvpsp.dll 68C40000 : 68C49000 RAPILIB.dll 74FD0000 : 74FEE000 msafd.dll 75010000 : 75017000 wshtcpip.dll 75020000 : 75028000 WS2HELP.DLL 75030000 : 75044000 WS2_32.dll 759B0000 : 759B6000 LZ32.DLL 77820000 : 77827000 VERSION.dll 77D30000 : 77DA8000 RPCRT4.dll 77E10000 : 77E6F000 USER32.dll 77F40000 : 77F7C000 GDI32.dll 77F80000 : 77FFC000 ntdll.dll 78000000 : 78045000 msvcrt.dll 7C2D0000 : 7C335000 ADVAPI32.DLL 7C370000 : 7C409000 MSVCR80.dll 7C570000 : 7C623000 KERNEL32.dll

Note how hengine.dll got moved down from its default 0x10000000 slot, thus freeing up valuable contiguous memory. This is good! This is what we want!

Continuing my efforts further with a Win2K Server virtual machine I happen to have:

W2KSRVR VIRTUAL MACHINE Using old Ivan snapshot; MAINSIZE 64 00230000 : 0023F000 hsys.dll 00240000 : 00278000 hdasd.dll 00280000 : 00290000 hutil.dll 00290000 : 002A2000 zlib1.dll 00400000 : 00406000 hercules.exe 00BA0000 : 00BA6000 hdteq.dll 00BB0000 : 00BB6000 hdt1052c.dll 00BC0000 : 00BC8000 hdt3505.dll 00BD0000 : 00BD6000 hdt3525.dll 00BE0000 : 00BE6000 hdt1403.dll 00BF0000 : 00BFB000 hdt3270.dll 10000000 : 101DD000 hengine.dll 68590000 : 685A6000 rsvpsp.dll 68C40000 : 68C49000 RAPILIB.dll 72A00000 : 72A2D000 DbgHelp.dll 74FD0000 : 74FEE000 msafd.dll 75010000 : 75017000 wshtcpip.dll 75020000 : 75028000 WS2HELP.DLL 75030000 : 75044000 WS2_32.dll 77D30000 : 77DA8000 RPCRT4.dll 77E10000 : 77E6F000 USER32.dll 77F40000 : 77F7C000 GDI32.dll 77F80000 : 77FFC000 ntdll.dll 78000000 : 78045000 MSVCRT.DLL 7C2D0000 : 7C335000 ADVAPI32.DLL 7C340000 : 7C396000 MSVCR71.dll 7C570000 : 7C623000 KERNEL32.dll Using new rebased version; MAINSIZE 64 00400000 : 00406000 hercules.exe 00410000 : 0042C000 dyncrypt.dll 00430000 : 00436000 hdt1052c.dll 00440000 : 00447000 hdt1403.dll 00450000 : 00486000 hdasd.dll 004E0000 : 004EB000 hdt3270.dll 004F0000 : 004F9000 hdt3505.dll 00500000 : 00506000 hdt3525.dll 00510000 : 00516000 hdteq.dll 00530000 : 006EA000 hengine.dll 006F0000 : 006FF000 hsys.dll 00710000 : 00720000 hutil.dll 00720000 : 0073E000 LIBBZ2.dll 00740000 : 00753000 zlib1.dll 68590000 : 685A6000 rsvpsp.dll 68C40000 : 68C49000 RAPILIB.dll 72A00000 : 72A2D000 DbgHelp.dll 74FD0000 : 74FEE000 msafd.dll 75010000 : 75017000 wshtcpip.dll 75020000 : 75028000 WS2HELP.DLL 75030000 : 75044000 WS2_32.dll 77D30000 : 77DA8000 RPCRT4.dll 77E10000 : 77E6F000 USER32.dll 77F40000 : 77F7C000 GDI32.dll 77F80000 : 77FFC000 ntdll.dll 78000000 : 78045000 msvcrt.dll 7C2D0000 : 7C335000 ADVAPI32.DLL 7C370000 : 7C409000 MSVCR80.dll 7C570000 : 7C623000 KERNEL32.dll un-rebased snapshot: old max MAINSIZE = 1500 00230000 : 0023F000 hsys.dll 00240000 : 00278000 hdasd.dll 00280000 : 00290000 hutil.dll 00290000 : 002A2000 zlib1.dll 00400000 : 00406000 hercules.exe 00BA0000 : 00BA6000 hdteq.dll 00BB0000 : 00BB6000 hdt1052c.dll 00BC0000 : 00BC8000 hdt3505.dll 00BD0000 : 00BD6000 hdt3525.dll 00BE0000 : 00BE6000 hdt1403.dll 00BF0000 : 00BFB000 hdt3270.dll 10000000 : 101DD000 hengine.dll 6E0B0000 : 6E0C6000 rsvpsp.dll 6E0D0000 : 6E0D9000 RAPILIB.dll 72A00000 : 72A2D000 DbgHelp.dll 74FD0000 : 74FEE000 msafd.dll 75010000 : 75017000 wshtcpip.dll 75020000 : 75028000 WS2HELP.DLL 75030000 : 75044000 WS2_32.dll 77D30000 : 77DA8000 RPCRT4.dll 77E10000 : 77E6F000 USER32.dll 77F40000 : 77F7C000 GDI32.dll 77F80000 : 77FFC000 ntdll.dll 78000000 : 78045000 MSVCRT.DLL 7C2D0000 : 7C335000 ADVAPI32.DLL 7C340000 : 7C396000 MSVCR71.dll 7C570000 : 7C623000 KERNEL32.dll rebased snapshot: New max MAINSIZE = 1800! 00400000 : 00406000 hercules.exe 00410000 : 0042C000 dyncrypt.dll 00430000 : 00436000 hdt1052c.dll 00440000 : 00447000 hdt1403.dll 00450000 : 00486000 hdasd.dll 004E0000 : 004EB000 hdt3270.dll 004F0000 : 004F9000 hdt3505.dll 00500000 : 00506000 hdt3525.dll 00510000 : 00516000 hdteq.dll 00530000 : 006EA000 hengine.dll 006F0000 : 006FF000 hsys.dll 00710000 : 00720000 hutil.dll 00720000 : 0073E000 LIBBZ2.dll 00740000 : 00753000 zlib1.dll 719C0000 : 719D6000 rsvpsp.dll 719E0000 : 719E9000 RAPILIB.dll 72A00000 : 72A2D000 DbgHelp.dll 74FD0000 : 74FEE000 msafd.dll 75010000 : 75017000 wshtcpip.dll 75020000 : 75028000 WS2HELP.DLL 75030000 : 75044000 WS2_32.dll 77D30000 : 77DA8000 RPCRT4.dll 77E10000 : 77E6F000 USER32.dll 77F40000 : 77F7C000 GDI32.dll 77F80000 : 77FFC000 ntdll.dll 78000000 : 78045000 msvcrt.dll 7C2D0000 : 7C335000 ADVAPI32.DLL 7C370000 : 7C409000 MSVCR80.dll 7C570000 : 7C623000 KERNEL32.dll

The above displays illustrates both how a simple rebasing of DLLs helps to free up valuable virtual memory (over 300MB in this particular case!) as well as how, apparently, specifying a large MAINSIZE causes Windows to apparently shuffle around already loaded DLLs in order to try and make room for a heap of the specified size. (Note: that's admittedly just an educated guess but the above evidence seems to support it)

Continuing the same effort but on my production development system provides the following results:

Note: rebasing GoogleDesktopNetwork1.dll was not trivial. Please refer the the discussion further below for additional information on rebasing DLLs that are currently in use by system services.

PRODUCTION SYSTEM With only Herc rebased, using my MVS config: max MAINSIZE = 1000 00370000 : 0038C000 dyncrypt.dll 00400000 : 00406000 Hercules.exe 00420000 : 00427000 dyngui.dll 00450000 : 00486000 hdasd.dll 004A0000 : 004A7000 hdt1403.dll 004D0000 : 004DB000 hdt3270.dll 004F0000 : 004F9000 hdt3505.dll 00500000 : 00506000 hdt3525.dll 00510000 : 00516000 hdteq.dll 00530000 : 006EA000 hengine.dll 006F0000 : 006FF000 hsys.dll 00710000 : 00720000 hutil.dll 00720000 : 0073E000 LIBBZ2.dll 00740000 : 00753000 zlib1.dll 02800000 : 028C1000 DbgHelp.dll 43000000 : 43005000 GoogleDesktopNetwork1.dll 68590000 : 685A6000 rsvpsp.dll 68C40000 : 68C49000 RAPILIB.dll 74FD0000 : 74FEE000 msafd.dll 75010000 : 75017000 wshtcpip.dll 75020000 : 75028000 WS2HELP.DLL 75030000 : 75044000 WS2_32.dll 759B0000 : 759B6000 LZ32.DLL 77820000 : 77827000 VERSION.dll 77D30000 : 77DA8000 RPCRT4.dll 77E10000 : 77E6F000 USER32.dll 77F40000 : 77F7C000 GDI32.dll 77F80000 : 77FFC000 ntdll.dll 78000000 : 78045000 msvcrt.dll 7C2D0000 : 7C335000 ADVAPI32.DLL 7C370000 : 7C409000 MSVCR80.dll 7C570000 : 7C623000 KERNEL32.dll After rebasing both "DbgHelp.dll" and "GoogleDesktopNetwork1.dll" to 0x400000 (still using my MVS config): max MAINSIZE now = 1800! 00370000 : 0038C000 dyncrypt.dll 00400000 : 00406000 Hercules.exe 00420000 : 00427000 dyngui.dll 00450000 : 00486000 hdasd.dll 004A0000 : 004A7000 hdt1403.dll 004D0000 : 004DB000 hdt3270.dll 004F0000 : 004F9000 hdt3505.dll 00500000 : 00506000 hdt3525.dll 00510000 : 00516000 hdteq.dll 00530000 : 006EA000 hengine.dll 006F0000 : 006FF000 hsys.dll 00710000 : 00720000 hutil.dll 00720000 : 0073E000 LIBBZ2.dll 00740000 : 00753000 zlib1.dll 00BB0000 : 00C71000 DbgHelp.dll 00E80000 : 00E85000 GoogleDesktopNetwork1.dll 722E0000 : 722F6000 rsvpsp.dll 72300000 : 72309000 RAPILIB.dll 74FD0000 : 74FEE000 msafd.dll 75010000 : 75017000 wshtcpip.dll 75020000 : 75028000 WS2HELP.DLL 75030000 : 75044000 WS2_32.dll 759B0000 : 759B6000 LZ32.DLL 77820000 : 77827000 VERSION.dll 77D30000 : 77DA8000 RPCRT4.dll 77E10000 : 77E6F000 USER32.dll 77F40000 : 77F7C000 GDI32.dll 77F80000 : 77FFC000 ntdll.dll 78000000 : 78045000 msvcrt.dll 7C2D0000 : 7C335000 ADVAPI32.DLL 7C370000 : 7C409000 MSVCR80.dll 7C570000 : 7C623000 KERNEL32.dll

As you can see from the above displays, the result of rebasing both Hercules's DLLs -- and perhaps more importantly, the Google desktop DLL -- resulted in my being able to recover over 800MB of usable virtual address space!

Rebasing DLLs which are currently in use

If you encounter the error:

      REBASE: *** RelocateImage failed (foobar.dll). Image may be corrupted

when trying to rebase a particular DLL, then it simply means the DLL is either marked read-only or is already loaded and in use by some existing process and thus cannot be rebased.

You need to identify which process or processes is/are using it and shut that process down (see further below). If it's a service then you need to stop that service. (Note that stopping a key system service when other system services depend on it has its own ramifications. You need to be careful and know what you're doing).

In my case rebasing "DbgHelp.dll" was easy (no one was using it) whereas rebasing "GoogleDesktopNetwork1.dll" was a proverbial pain-in-the-butt! (The RpcSs ("Remote Procedure Call (RPC)") service was the one using it, and shutting down this particular critical system service was a pain. You need to go into regedit and set its 'Start' value from 2 (Automatic) to 4 (Disabled) and then reboot into Safe Mode. Note that when you do, Explorer may take much longer (several minutes) to start up. Also note that once disabled, you will NOT be able to re-enable it via the Services snapin either. You will thus need to go back into regedit and change its 'Start' value back to 2 (Automatic) and then reboot.

Once you've disabled RpcSs and have rebooted into Safe Mode, go into the "Program Files\Google\Google Desktop" directory and rebase Google's network dll as needed (it should work now that RpcSs isn't running):

      rebase -b 0x400000 GoogleDesktopNetwork1.dll

Once you've rebased it, then go back into regedit, change RpcSs back to automatic, and reboot normally.

I don't know of any Windows utility that will tell you what process is using a given DLL (does anyone know of one?). What I personally did was use Sysinternals' free "Process Explorer" utility instead. It's similar to Windows Task Manager but much nicer IMO and very much more powerful.

Note: you may also want to make some registry tweaks to HercGUI to get it to accept MAINSIZE values larger than 1024. See the "Registry Tweaks" section of the HercGUI help pages for more information.

FINALLY, while it may be obvious to some people it might not be obvious to others: the size of your Windows swap file limits the amount of memory a program can acquire and thus may also need to be increased in size as well. If your Windows swap file is too small, then no matter what Hercules does it's never going to be able to obtain the memory it needs, so if possible, set your Windows "Maximum paging size" value to the maximum supported.

That's it.

Hope this information has been helpful to you!

"Fish" (David B. Trout)
fishsoftdevlabs.com

"Programming today is a race between
software engineers striving to build bigger
and better idiot-proof programs, and the
Universe trying to produce bigger and better
idiots. So far, the Universe is winning."

- Rich Cook