Representation of data types in memory - Part 3

You have probably written statements like this thousand of times, but do you know what's going on under the hood?

float foo = 1.21;

This is the continuation of my series about representation of data types in memory. In my previous post, I've discussed the numerical integral types. Now it's time to dig into the wonderful world of floating types. There is a lot to discuss about floating types, so in this post I will focus on the Normalized form and its representation in memory.

Every C++ programmer has dealt with the data types float and double and we will soon see how they are represented in memory. But before doing that, we need to understand some basic concepts and formats, so let's discuss some theory before proceeding to the practical part.

My intention is not to give a complete description about the theory of floating points, I will give a very brief summary here (and I mean it!), the details can easily be found on the net.

Below is a very good description (from this site), which describes how a floating point is expressed when stored in memory.

A floating-point number is typically expressed in the scientific notation, with a fraction (F), and an exponent (E) of a certain radix (r), in the form of F×rE. Decimal numbers use radix of 10 (F×10E); while binary numbers use radix of 2 (F×2E).

Example:
Let's say we have 16.2510. By using the scientific notation, it can be written as 16.2510 * 100 or 1.62510 * 101 and so forth. In binary, it can be written as 10000.012 * 2⁰, 1000.0012 * 2¹, 100.00012 * 2², 10.000012 * 2³ or 1.0000012 * 2⁴ and so forth. The representation used in a floating point is 1.0000012 * 2⁴, which is called the Normalized form.

The IEEE standard 754 describes the single precision format and double precision format. It is important to have a brief understanding of these formats, because the floating types in C++ is based on them.

According to MSDN, following is stated (Visual Studio 2010 specific) for the data type float;

The float type is stored as a four-byte, single-precision, floating-point number. It represents a single-precision 32-bit IEEE 754 value.

According to MSDN, following is stated (Visual Studio 2010 specific) for the data type double;

The double type is stored as an eight-byte, double-precision, floating-point number. It represents a double-precision 64-bit IEEE 754 value.

Voila! There we have the basic definition of the data types float and double.

The single precision format consists of 32 bits (4 bytes), where the Most Significant Bit (MSB) represent the sign (S) bit, the following 8 bits represent the exponent (E), and the 23 Least Significant Bit(s) (LSB) represent the fraction (F). Note that a bias is applied to the exponent (E) in order to represent both positive and negative exponents. The bias is 127 in single precision format, meaning exponent (E) = 0 is represented as 127, E=1 is represented as 128 and so on.

The double precision format consists of 64 bits (8 bytes), where the MSB represent the sign (S) bit, the following 11 bits represent the exponent (E), and the 52 LSB(s) represent the fraction (F). Note that a bias is applied to the exponent (E) in order to represent both positive and negative exponents. The bias is 1023 in double precision format, meaning exponent (E) = 0 is represented as 1023, E=1 is represented as 1024 and so on.

Before proceeding to the practical part, just a few words about Normalized form.

As we have seen above, Normalized form means we have an implicit leading 1 to the left of the radix point, which is used in the fraction (F), for instance 1.0000012. This leading 1 is not represented in the 32/64 bit format, but we know the leading 1 is there, if Normalized form is used.

Let's look into a simple application. This is very similar to my previous simple application, except from the fact the data type is float.

int main()
{
   float a = 1.0;
   float b = 2.0;
   float c = 3.0;
   float d = 4.0;

   return 0;
}

When starting WinDbg and execute all the initializations statements, we see the result below.

WinDbg - Memory view after initializations

As discussed in previous posts, in the Memory view, we can see the blue rectangle which indicates the inserted block of 0xCC which is typically done in Debug mode. The blue arrow shows the offset of the Memory view, which is where the insertion of 0xCC starts. Each float is also "guarded" by four bytes 0xCC. In the Disassembly view, we can notice that special floating point instructions are used. I will not discuss them in detail here, if you want more information, they are described in Intel x86 reference manual, more specific in the section "Intel 64 and IA-32 Architectures Software Developer's Manual Volume 2A: Instruction Set Reference, A-M"

Binary representation of 1.0

To understand how a float is stored in the memory, I will work through a couple of examples. Let's start with the first statement in the simple application;

float a = 1.0;

In binary we have 1.010 = 1.02, which is equal to 1.02 * 20 in Normalized form. Now let's see how this number 1.02 * 20 is stored in memory bit by bit.
Positive number -> Sign (S) bit = 02
Exponent: 0, i.e. the exponent (E) is represented as 12710 = 011111112
Fraction (F): 02 = 000000000000000000000002
Binary representation: 00111111 10000000 00000000 000000002
Hexadecimal representation: 3F80000016

This number is stored in the memory by this instruction;

fstp    dword ptr[ebp-8h]

The fstp instruction (Store Floating Point Value) copies the value from the FPU register stack to the memory in either single- or double precision format (single in this case due to float type). Note that the fld1 (Load Floating Point Value) instruction pushed this value (1.0) onto the FPU register stack in the first place. 0x3F800000 will be stored at memory address ebp-0x8. Taking little-endian into account, each byte will be saved according to below;

ebp-0x8: 0x00
ebp-0x7: 0x00
ebp-0x6: 0x80
ebp-0x5: 0x3F

Binary representation of 2.0

float b = 2.0;

Since 1.0 already was converted above, I will give a briefer explanation below.
Binary form: 10.02
Normalized form: 1.02 * 21
Positive number -> Sign (S) bit = 02
Exponent: 1, i.e. the exponent (E) is represented as 12810 = 100000002
Fraction (F): 02 = 000000000000000000000002
Binary representation: 01000000 00000000 00000000 000000002
Hexadecimal representation: 4000000016

This number is stored in the memory by this instruction;

fstp    dword ptr[ebp-14h]

Taking little-endian into account, each byte will be saved according to below;

ebp-0x14: 0x00
ebp-0x13: 0x00
ebp-0x12: 0x00
ebp-0x11: 0x40

Binary representation of 3.0

float a = 3.0;

Binary form: 11.02

Normalized form: 1.12 * 21
Positive number -> Sign (S) bit = 02
Exponent: 1, i.e., the exponent (E) is represented as 12810 = 100000002
Fraction (F): 12 = 100000000000000000000002
Binary representation: 01000000 01000000 00000000 000000002
Hexadecimal representation: 4040000016

This number is stored in the memory by this instruction;

fstp    dword ptr [ebp-20h]

Taking little-endian into account, each byte will be saved according to below;

ebp-0x14: 0x00
ebp-0x13: 0x00
ebp-0x12: 0x40
ebp-0x11: 0x40

Binary representation of 4.0

float a = 4.0;

Binary form: 100.02

Normalized form: 1.02 * 22
Positive number -> Sign (S) bit = 02
Exponent: 2, i.e. the exponent (E) is represented as 12910 = 100000012
Fraction (F): 02 = 000000000000000000000002
Binary representation: 01000000 10000000 00000000 000000002
Hexadecimal representation: 4080000016

This number is stored in the memory by this instruction;

fstp    dword ptr[ebp-2Ch]

Taking little-endian into account, each byte will be saved according to below;

ebp-0x14: 0x00
ebp-0x13: 0x00
ebp-0x12: 0x80
ebp-0x11: 0x40

The four examples above was only dealing with numbers in Normalized form. Later, I will have a look at the Denormalized form.

You are welcome to leave comments, complaints or questions!

Representation of data types in memory - Part 2

This is the continuation of the series "Representation of data types in memory". In this part, I will investigate how signed numerical integers are stored in memory. I'm using Windows Vista 32 bit with Microsoft Visual C++ 2010 Express (in Debug Mode) and WinDbg. Note that I'm not considering C++11.

Well, in my previous post, we saw how the unsigned numerical integers were saved in little-endian format in a block of 0xCC. This is of course true for signed numerical integers as well, but signed integers need to take the sign into account as well.

I will start this post in a similar way like my previous one, by using the same simple program, but with signed numerical integers.

int main()
{
   int a = -1;
   int b = -2;
   int c = -3;
   int d = -4;

   return 0;
}

When starting WinDbg and execute all the initializations statements, we see the result below.


WinDbg - Memory view after initializations

As discussed before, in the Memory view, we can see the blue rectangle which indicates the inserted block of 0xCC, which typically is done in Debug mode. The blue arrow shows the offset of the Memory view, which is where the insertion of 0xCC starts. Each integer is also "guarded" by four bytes 0xCC. Now we will focus on how the signed numerical integers are saved in memory. This is done by using the two's-complement representation. What is two's-complement? You can read all about it on the net, but here is a short explanation cited from wikipedia.

"Two's complement is a mathematical operation on binary numbers, as well as a binary signed number representation based on this operation. Its wide use in computing makes it the most important example of a radix complement."

There are a lot of in-depth information on the net how two's-complement conversion is done, here is my short version;

Invert all bits of the (positive) number and add one (+1).

Let's use an actual value from our simple program as an example, for instance;

int a = -1;

Positive number: 110

It's an int, i.e. four bytes in size -> 110 = 0000000116.

Then invert all bits; 0000000116 -> FFFFFFFE16
and then add one (+1); FFFFFFFE16 -> FFFFFFFF16

In other words, the number FFFFFFFF16, represent -1 in two's complement representation.

We can see this number in the Disassembly view, more specific, we can see this instruction;

mov     dword ptr [ebp-8],0FFFFFFFFh

It means that -1 is represented as 0xFFFFFFFF and will be stored at memory address ebp-0x8. Taking little-endian into account, each byte will be saved according to below;

ebp-0x8: 0xFF
ebp-0x7: 0xFF
ebp-0x6: 0xFF
ebp-0x5: 0xFF

Now let's continue with a similar program as in the first part in this series, this time with signed numerical integers.

int main()
{
   char a = -1;
   short int b = -2;
   int c = -3;
   long int d = -4;

   return 0;
}

When starting WinDbg and execute all the initializations statements, we see the result below.

WinDbg - Memory view after initializations

Like the first simple program, the blue arrow and blue rectangle, shows the inserted 0xCC done by the rep stos instruction.
Let's investigate how each type is stored. We know they are stored in two's-complement little-endian format.

Below, I've used following shorthand to denote the two's complement conversion for each type;

Positive number (using correct size) -> Invert all bits -> Added 1

char a = -1;

0116 -> FE16 -> FF16

ebp-0x5: 0xFF

short int b = -2;

000216 -> FFFD16 -> FFFE16

ebp-0x14: 0xFE
ebp-0x13: 0xFF

int c = -3;

0000000316 -> FFFFFFFC16 -> FFFFFFFD16

ebp-0x20: 0xFD
ebp-0x1F: 0xFF
ebp-0x1E: 0xFF
ebp-0x1D: 0xFF

long int d = -4;

0000000416 -> FFFFFFFB16 -> FFFFFFFC16

ebp-0x2C: 0xFC
ebp-0x2B: 0xFF
ebp-0x2A: 0xFF
ebp-0x29: 0xFF

You are welcome to leave comments, complaints or questions!

Representation of data types in memory - Part 1

Normally we don't need to bother how fundamental data types are stored in memory. We just expect it to work. Recently I became curios, so I learned the details of how a fundamental data type is represented in memory, especially on the stack. As usual, I'm using Windows Vista 32 bit with Microsoft Visual C++ 2010 Express and WinDbg. Note that I'm not considering C++11.

I've divided this topic into several posts. In the first one, I will focus on one group of fundamental data types; Numerical integer types, and more specific, the unsigned ones. In following posts, I will consider the signed numerical integers, floating types and other aspects.

Well, let's look into the numerical integer types. You probably know the integer types already. Below is a recap cited from www.cplusplus.com.

"Numerical integer types:

They can store a whole number value, such as 7 or 1024. They exist in a variety of sizes, and can either be signed or unsigned, depending on whether they support negative values or not."

Let's start with a very simple program, here we just use the unsigned integer type. When compiling this program in Debug mode, and executing it, how is the unsigned integers represented in memory?

int main()
{
   unsigned int a = 1;
   unsigned int b = 2;
   unsigned int c = 3;
   unsigned int d = 4;

   return 0;
}

When starting WinDbg and execute all the initializations statements, we see the result below.

WinDbg - Memory view after initializations

The blue arrow above, indicates that I've set the Memory view to ebp-0x0F0 according to the Disassembly view. Before any initializations has been done, a block of memory is initialized to 0xCC thanks to the rep stos instruction. This is typically done in Debug mode. The rep stos instruction starts inserts 0xCC at ebp-0f0.

Further, we can also see that each initialized integer is "guarded" by four bytes of 0xCC (before and after each integer).

The x86 architecture is using the little-endian format. I will briefly explain the little-endian format here. If you want more in-depth information, just search the net.

The endianness describes how a sequence of bytes are stored in the memory. It means that the endianness only matters for data types with more than one byte in size. Little-endian means that the Least Significant Byte (LSB) is stored at the lowest address and the Most Significant Byte (MSB) is stored at the highest address.

From the example above, we are dealing with integers, so let's use integer as an example. An integer is four bytes in size, this can be seen in the Memory view. An integer can be written like "ByteA ByteB ByteC ByteD", where ByteA is the MSB and ByteD is the LSB. The memory will look like this when we are using the little-endian format.

Base Address: ByteD
Base Address + 1: ByteC
Base Address + 2: ByteB
Base Address + 3: ByteA

Let's make the example above more realistic by using an actual value from our simple program, for instance;

   unsigned int a = 1;

As we know, the unsigned int is four byte in size, so the number will be 0x00000001. According to the Disassembly view, the statement "unsigned int a = 1;", will be saved on the stack at memory address ebp-0x8. The LSB i.e. 0x01 is saved at memory address ebp-0x8, and the other bytes is saved according to below.

ebp-0x8: 0x01
ebp-0x7: 0x00
ebp-0x6: 0x00
ebp-0x5: 0x00

The example above, was only dealing with the unsigned integers. Now we move on to another simple program, which shows the data representation of each unsigned numerical integers; char, short int, int and long int.

int main()
{
   unsigned char a = 1;
   unsigned short int b = 2;
   unsigned int c = 3;
   unsigned long int d = 4;

   return 0;
}

When starting WinDbg and execute all the initializations statements, we see the result below.

WinDbg - Memory view after initializations

Like the first simple program, the blue arrow and blue rectangle, shows the inserted 0xCC done by the rep stos instruction.

Let's investigate how each type is stored. We know they are stored in little-endian format, meaning the LSB is stored in the lowest address.

   unsigned char a = 1;

ebp-0x5: 0x01

   unsigned short int b = 2;

ebp-0x14: 0x02
ebp-0x13: 0x00

   unsigned int c = 3;

ebp-0x20: 0x03
ebp-0x1F: 0x00
ebp-0x1E: 0x00
ebp-0x1D: 0x00

   unsigned long int d = 4;

ebp-0x2C: 0x04
ebp-0x2B: 0x00
ebp-0x2A: 0x00
ebp-0x29: 0x00

From the results above, we can note that the unsigned char is only one byte in size, so the endianess format does not matter. We can also see that both unsigned int and unsigned long int is four byte in size.

You are welcome to leave comments, complaints or questions!

Base relocation table

A Portable Executable (PE) file usually contains some headers and some sections. One of the sections, that may exist, is the .reloc section, and within this section is a base relocation table. The base relocation table is needed to fix up virtual addresses in the PE file if the PE file not was loaded at its preferred load address.

In this post, I will have a look at a base relocation table and see what's going on there. Further, I will discuss how to read the entries in the base relocation table, and what they are doing in the code. I'm using PE Insider from Cerbero, PEBrowserPro from Smidgeonsoft and WinDbg.

Before checking out a .reloc section in a PE file, let's discuss the .reloc section briefly.

The .reloc section contains a serie of blocks. There is one block for each 4 KB page that contains virtual addresses, which is in need for fix ups. Each block contains an IMAGE_BASE_RELOCATION struct and a serie of entries.

The IMAGE_BASE_RELOCATION is defined as below, according to the header file winnt.h.

typedef struct _IMAGE_BASE_RELOCATION {
    DWORD   VirtualAddress;
    DWORD   SizeOfBlock;
//  WORD    TypeOffset[1];
} IMAGE_BASE_RELOCATION;

The VirtualAddress holds a Relative Virtual Address (RVA) of the 4 KB page, which the relocation applies to. The SizeOfBlock holds the size of the block in bytes (including the size of the IMAGE_BASE_RELOCATION struct).

Each entry is a 2-byte value and represent a location (offset within the 4 KB page pointed out by the VirtualAddress member in the IMAGE_BASE_RELOCATION struct), which needs to be fixed up in case of the PE file is not loaded at its preferred load address.

That was some theory. Now let's take a look at a .reloc section from a simple "Hello world!" program. I've compiled and linked it in Visual Studio 2010 Express in Release mode.

#include <cstdio>

int main()
{
   std::printf("Hello world!");
}

Before proceeding, just a quick note. The program above lack the return 0 statement. This does not cause any trouble, the compiler actually add this statement. This can be seen in the disassembly views which follows below (xor eax, eax, ret).

Let's open the "Hello world!" program in PE Insider.

First, let's check out the preferred load address.

PE Insider - Preferred load address

The preferred load address is the ImageBase value, i.e. 0x00400000.

Next, check out the section table.

PE Insider - section table

Apparently there are five sections needed for my simple "Hello world!" program. We are only interested in the .reloc section, so lets check it out.

PE Insider - .reloc section

The .reloc section contains two blocks (within the red border). Since there is 2 blocks, it means that there is 2 4KB pages that contain virtual addresses which is in need for fix ups.

The first member in the IMAGE_BASE_RELOCATION in the first block is the VirtualAddress member, with the value 0x00001000 (green underscored value). The second member is the SizeOfBlock member, with the value 0x00000128 (black underscored value). In order to calculate the number of entries, we must subtract the the size of the IMAGE_BASE_RELOCATION struct from the SizeOfBlock member, i.e. 0x128-0x8 = 0x120 bytes. Further, we know that each entry is a 2-byte value, which gives the number of entries to 0x90 (DEC: 144).

The VirtualAddress value, in the second block, is 0x00002000 (green underscored value) and the SizeOfBlock value is 0x00000020 (black underscored value). Applying same idea as a for the first block, the number of entries is 0xC (DEC: 12).

So what is the meaning of the information above?

Regarding the first block, the loader understand it must do some fix ups in the 4 KB page that starts at RVA 0x1000. As you can see in the section table above, 0x1000 is where the .text section starts. In other words, the loader must fix up 0x90 (DEC: 144) virtual addresses in the .text section, i.e in the code. Note that the .text section is covered in 1 4 KB page.

Regarding the second block, the loader understand it must do some fix ups in the 4 KB page that starts at RVA 0x2000. As you can see in the section table above, this is the .rdata section. In other words, the loader must fix up 0xC (DEC: 12) virtual addresses in the .rdata section. Note that the. rdata section is covered in 1 4 KB page.

So how do the loader know how to fix up the virtual address, and where it should do it?

In order for the loader to know how to fix up the address, it takes the delta value of the preferred load address and the actual load address. The delta value is then added to the virtual address.

In order for the loader to know where to do the fix ups, it check out each entry in the blocks in the .reloc section. As mentioned above, an entry is a 2-byte value. The 4 most significants bits in the value, represents the type of relocation. In the x86 world, the type is always 3 (HIGHLOW), which means that we add the complete 32 bit value of the delta value to the virtual address that is in need for fix up. The remaining bits (12 bits) of the entry, tells us the offset in the 4 KB page (pointed out by the VirtualAddress member in the IMAGE_BASE_RELOCATION struct), i.e. where to do the fix up in the page.

Let's clarify this with an example. As mentioned above, the .text section will have 0x90 fix ups. Let's go in to detail for the first 2 fix ups.

According to the .reloc section above, the first entry is 0x3001. This means that the type is 3 and the offset, in the page, is 0x001. The second entry is 0x3007, which means that the type is 3 and the offset, in the page, is 0x007.

Thanks to these offset values, the loader now knows that it must fix up the virtual address at RVA 0x1000+0x1 = 0x1001, and RVA 0x1000+0x7 = 0x1007.

Below is the .text section on file, and the affected RVA's within the red border. I've also marked them as 1 and 2, cause I will deal with them seperatly in my explanation. I also use them as references between the screenshots.

PE Insider - .text section - HEXVIEW

So above, we see the binary machine code. This does not really tells us so much, so let's disassemble this part using PEBrowserPro.



PEBrowserPro - .text section - disassembled

Thanks to the disassembled view, we can easily see what's occurs on RVA 0x1001 and RVA 0x1007. Note that the view using virtual addresses (with the preferred load address 0x0040000).

Lets start with case 1, i.e the RVA 0x1001.

At this position, we can see the value 0xF4204000. This is a virtual address (0x004020F4, remember little-endian), which needs to be fixed up according to the .reloc section. Actually, it is a virtual address within the .rdata section. The string literal "Hello world!" is put in the .rdata section. In the disassemble view above, we push the address, to the string literal, on the stack. In case of the PE file not will be loaded at its preferred load address, this value 0xF4204000 needs to be fixed up.

Now let's move on to case 2, i.e. the RVA 0x1007.

At this position, we can see the value 0xA0204000. This is a virtual address (0x004020A0, remember little-endian), which needs to be fixed up according to the .reloc section. This is a virtual address within the .rdata section as well. More specific, the virtual address 0x004020A0 is a part of the Import Address Table (IAT). It can be verified by checking the Directories.

PE Insider - Directories

Each entry in the IAT contains virtual addresses to functions in other modules. In this case, the IAT entry at the virtual address 0x004020A0 contains a virtual address to the function printf in the msvcr100.dll module.

So when executing the CALL instruction (see the disassemble view), the processor finds out printf's virtual address by checking the virtual address 0x004020A0. Before load time, we don't know the virtual address of printf yet, because the DLL (msvcr100.dll) has not been loaded into virtual memory.

As you can understand, if the loader does not load the "Hello world!" program at its preferred load address, the .rdata section will not be located at 0x2000, and the entry, which tells us where to find the printf function is invalid. Therefore the loader must add the delta value to 0x004020F4.

Finally, I will just show what's going on in the .text section, when the "Hello world!" program is executed (loaded in memory). To this, I'm using WinDbg. Note that the "Hello world!" program is linked with the DYNAMICBASE switch, i.e. it's Address Space Layout Randomization (ASLR) compatible.

First, let's find out the load address.

WinDbg - Load address

Remember that according to the PE file, the preferred load address is 0x00400000. The actual load address is 0x00250000, which gives us a delta value 0x0040000-0x00250000 = 0x1B0000 (negative difference).

Second, let's check out the .text section



WinDbg - .text section - Disassembled view

As you see above, the virtual addresses differ from the disassembled view in PEBrowserPro, due to the "Hello world!" program is loaded at 0x00250000.

For instance, according to PEBrowserPro, the push instruction pushes the virtual address 0x004020F4 on the stack. Perfectly OK when the program is loaded at its preferred load address 0x00400000.
But in this case, the program is loaded at 0x00250000. The delta value is 0x1B0000 (calculated above). This is a negative difference. In other words, the loader must fix up virtual address 0x4020F4 by reducing it with 0x1B0000. This gives 0x004020F4-0x1B0000 = 0x002520F4. This is exactly what happened in the screenshot above.

You are welcome to leave comments, complaints or questions!

INT in notepad.exe

The Import Name Table (INT) contains information how to fill the Import Address Table (IAT) during load time, with appropriate virtual addresses, i.e. virtual addresses to functions in other modules (but still within the same virtual address space). I will explain where to find the INT in the file on disk (and not in memory) and also what kind of information is within the INT. I'm using notepad.exe as example and Windows Vista 32 bit.

So where do we find the INT?

To find the INT, we must start with the headers in the Portable Executable (PE) file (notepad.exe). Within one of the headers, you will find a Directory Table, containing directories. In the case of notepad.exe, there is a directory called Import Directory, which we will inspect in more detail. Using PE Insider from Cerbero, we can easily see the Directory Table.

PE Insider - Directory Table

Above you see three directories. The Export Directory is the first one appearing in the PE file, then follows the Import Directory and the Resource Directory. There are more directories in the PE file, but explaining the directories is not within the scope of this post.

There are a lot of good information about the structures in the Import Directory on the net. I will discuss it briefly here.

The Import Directory holds a Relative Virtual Address (RVA), which point to an array of IMAGE_IMPORT_DESCRIPTOR's. Above, you can find the RVA on the Import Directory RVA row and the right most column, i.e. 0x00008BF4.

The IMAGE_IMPORT_DESCRIPTOR is defined as below, according to the header file winnt.h.

typedef struct _IMAGE_IMPORT_DESCRIPTOR {
    union {
        DWORD   Characteristics;            
        DWORD   OriginalFirstThunk;         
    } DUMMYUNIONNAME;
    DWORD   TimeDateStamp;                  
    DWORD   ForwarderChain;                 
    DWORD   Name;
    DWORD   FirstThunk;                     
} IMAGE_IMPORT_DESCRIPTOR;

The OriginalFirstThunk holds a RVA, which point to an array of IMAGE_THUNK_DATA's. So does the FirstThunk.

It can also be mentioned that the Name member in the IMAGE_IMPORT_DESCRIPTOR holds a RVA, which points to an Ascii string, which is the name of the DLL, e.g. kernel32.dll.

The IMAGE_THUNK_DATA is defined as below, according to the header file winnt.h.

typedef struct _IMAGE_THUNK_DATA32 {
    union {
        DWORD ForwarderString;       
        DWORD Function;             
        DWORD Ordinal;
        DWORD AddressOfData;        
    } u1;
} IMAGE_THUNK_DATA32;

The IMAGE_THUNK_DATA struct represents an imported function from a PE file. In the PE file, the IMAGE_THUNK_DATA is either an Ordinal value or an AddressOfData value. The latter value is a RVA, which points to an IMAGE_IMPORT_BY_NAME struct. How do the loader knows if the value in the IMAGE_THUNK_DATA is an Ordinal value or an AddressOfData value? The loader checks the most significant bit in the value. If the bit is 0, the value is an AddressOfData value, if the bit is 1, the value is an Ordinal value.

The IMAGE_IMPORT_BY_NAME is defined as below, according to the header file winnt.h.

typedef struct _IMAGE_IMPORT_BY_NAME {
    WORD    Hint;
    BYTE    Name[1];
} IMAGE_IMPORT_BY_NAME, *PIMAGE_IMPORT_BY_NAME;

We are now ready to find the INT in file on disk, but first we must find the array of IMAGE_IMPORT_DESCRIPTOR's. We can't really use the RVA 0x00008BF4 directly, since the RVA only can be used when the PE file is loaded into memory. We must in some way translate the RVA to an Offset in the PE file (notepad.exe). I will present a concept here, that I call RVA-to-Offset translation.

First we check where the sections will be loaded in memory, more specific, let's check the section table in the PE file (notepad.exe). See below.

PE Insider - Section Table

We can see that the .text section will be loaded at RVA 0x00001000 with section size 0x00008F40 bytes, and the next section (.data section) will be loaded at RVA 0x0000A000. The array of IMAGE_IMPORT_DESCRIPTOR's starts at RVA 0x8BF4, i.e. within the .text section. This means that the array is 0x00008BF4 - 0x00001000 = 0x00007BF4 bytes from the start of the .text section. On disk, the .text section starts at Offset 0x400 according to the PointerToRawData column. So the array of IMAGE_IMPORT_DESCRIPTOR's on disk, can be found at Offset 0x400+0x00007BF4 = 0x7FF4.

Now when we have translated the RVA 0x00008BF4 to Offset 0x7FF4 in file on disk, we are going to check out what's going on at this location.

Below is a HEXVIEW of the IMAGE_IMPORT_DESCRIPTOR array, name strings (DLLs) and IMAGE_THUNK_DATA arrays. We are going to interpret the HEXVIEW byte by byte, so remember that x86 is using little-endian architecture!

PE Insider - HEXVIEW

The array of IMAGE_IMPORT_DESCRIPTOR's (within the black border) starts at Offset 0x7FF4 (as calculated above) and the first member in the first struct is the OriginalFirstThunk with the value 0x00008DB0 (first underscored value).

For an easier overview above, I have underscored each OriginalFirstThunk member, in each IMAGE_IMPORT_DESCRIPTOR. Counting elements in the IMAGE_IMPORT_DESCRIPTOR array, gives us 14 descriptors. Note that the last one is a NULL Descriptor that marks the end of the array. Each descriptor represents a DLL and its imported functions. This means that notepad.exe uses 13 DLLs. For instance, let's check the first descriptor and its name member. The name member is the fourth member in the IMAGE_IMPORT_DESCRIPTOR struct. At this location, we find the value 0x00008D0C, which is a RVA, which points to the name string. Using the RVA-to-Offset Translation, the RVA 0x00008D0C is translated to Offset 0x810C. This Offset is located directly after the IMAGE_IMPORT_DESCRIPTOR array above, and contains the Ascii string advapi32.dll (and 2 NULL terminations).

Well, back to the OriginalFirstThunk. As mentioned above the first OriginalFirstThunk has the value 0x00008DB0. This is a RVA, which points to an array of IMAGE_THUNK_DATA's. Using the RVA-to-Offset Translation, the RVA 0x00008DB0 will be translated to the Offset 0x81B0. At this location (see HEXVIEW above), we find an array (within the red border). Actually, this array is the Import Name Table (INT) that we was looking for in the first place. In this case, the INT (for the first DLL), contains 6 elements, where the last element is a NULL Data element. In other words, notepad.exe is importing 5 functions from advapi32.dll.

As you can see in the HEXVIEW above, the first element in the IMAGE_THUNK_DATA array contains the value 0x00009138. Remember that the loader must know how to deal with this value, so it checks the first bit, which in this case is a 0, meaning that it is the AddressOfData member. AddressOfData is a RVA, which points to a IMAGE_IMPORT_BY_NAME struct. Using the RVA-to-Offset translation, the RVA 0x00009138 corresponds to the Offset 0x8538 on disk. Let's check out what's going on here. Remember that the information at this location is an IMAGE_IMPORT_BY_NAME struct.

PE Insider - HEXVIEW

From the HEXVIEW at Offset 0x8538, we find the word 0x0268 (underscored), which is the hint member. Then follows the name (Ascii string) of the imported function, in this case ReqQueryValueExW (and two NULL terminations). Then follows another IMAGE_IMPORT_BY_NAME struct, with a hint and an Ascii string, and so on.

So finally we know where the INT is located, and how to read it. The loader is using the INT to fill the IAT. For instance, the loader realize that notepad.exe is importing 5 functions (by name, and not by ordinal) from advapi32.dll and will look up their virtual addresses, and write them to notepad.exe's IAT.

You are welcome to leave comments, complaints or questions!

IAT in notepad.exe

A couple of weeks ago, I was inspecting and comparing the .text section of a PE file (notepad.exe) and the corresponding .text section which was loaded in memory. I realized that the first part of the .text section was not even close to be equal (file on disk v.s. file in memory). I googled a lot about this and finally learned that some part of the .text section may consist of the Import Address Table (IAT).

In this post I'm going to show you what I learned and what is going on in the IAT of notepad.exe. As usual, I'm using Windows Vista 32 bit.

If you don't know it, the IAT is a lookup table. This table is filled with virtual addresses to functions in other modules (in this case, functions outside notepad.exe) when the executable is loaded into memory. I will describe it in more detail later in this post. Do not confuse the IAT with the Import Name Table (INT). The INT is also a part of the PE file, but is not the same as the IAT. However, they do have some connections.

Let's start where I started. It was a sunny day and I was in a very good mood for some coding. Well, more seriously, I was inspecting the .text section using PE Insider from Cerbero. This is what I saw in the HEX view. Note that the .text section starts at offset 0x400 in the file.

PE Insider - Content of PE file (file on disk)

Back a few weeks, when I was inspecting notepad.exe in PE Insider, I thought that the .text section only contained binary machine code. And the binary machine code is not supposed to change when the file is loaded to memory. So I was expecting that the contents to be more or less the same in memory. Let's check out the memory when notepad.exe is executed. To do this I'm using WinDbg.



WinDbg - Content of memory (file loaded in memory)

The .text section starts at 0x00721000 since Windows decided to place notepad.exe at memory location 0x00720000 this time (remember that Windows Vista use ASLR and that notepad is ASLR compatible), and the Relative Virtual Address (RVA) for the .text section (according to the PE file, i.e. notepad.exe) is 0x1000.

It really puzzled me first time I compared the file content with the memory content. There were a lot of differences. First I thought WinDbg was injecting some code to handle the debug break, or maybe it was some unwanted code in the memory? I started to compare the contents (PE Insider - WinDbg) manually (byte by byte) and realized at some memory location, the contents started to be the same. See below.



PE Insider - Content of PE file (file on disk)

WinDbg - Content of memory (file loaded in memory)



Somewhere around the red marking, the contents starts to be equal. Why was only the first part differing? I was thinking that PE Insider maybe was not showing the correct file content. So I downloaded another PE viewer, PEBrowser Pro from SmidgeonSoft. I started PEBrowser Pro, opened notepad.exe and clicked the .text section, which opened a new window with the contents below.

PEBrowser Pro - .text section disassembled

PEBrowser Pro is a little more sophisticated than PE Insider, it can disassembles the .text section among other things. This is what we see above. But why did it start disassemble at 0x100138D and not 0x1001000? Note that PEBrowser Pro thinks notepad.exe should be loaded at its preferred load address which is 0x1000000 according to the ImageBase field in the Optional Header of notepad.exe (which means the .text section starts at 0x1001000). Apparently PEBrowser Pro realize that the first bytes in the .text section is not actually binary machine code, but something else. By having a look at the treeview of notepad.exe in PEBrowser Pro, it gave me some hints.

PEBrowser Pro - Treeview of file contents

Yes, apparently the .text section may consist of more than just binary machine code. It contains the IAT among other things.

But how do PEBrowser Pro know that the binary machine code starts at 0x100138D? I wasn't sure about this, but I guessed that the header of the PE file (notepad.exe) may give a clue. Back in PE Insider and viewing the Data Directories in the Optional Header I saw this.

PE Insider - Data Directories

The Data Directories, more specific, the Import Address Table Directory, tells us that the table starts at RVA 0x1000 i.e. this is where the .text section starts! Its size is 0x388 bytes. At this point, things began to be clear for me.

Next thing to understand was how to interpret the bytes in the IAT.

After some googling, I realized that when the PE file (notepad.exe) is loaded into memory, the IAT is filled with virtual addresses. This is virtual addresses in other modules loaded within the same virtual memory space as notepad.exe.

So let's take an example. Let's investigate the first address in the IAT (in memory), the address at RVA 0x1000. Below we can see that the first address in the IAT is 0x77ae765e. This virtual address is in another module, more specific in advapi32.dll and it is the address of the function "RegQueryValueExW".



WinDbg - IAT (file loaded in memory)

Below we see some part of the disassembled function.

WinDbg - Disassembling RegQueryValueExW

Well, now we know the meaning of the IAT bytes when the PE file is loaded in memory, but why does they differ comparing to the bytes on disk?

As mentioned above, the IAT is filled with addresses during load time, so the IAT can't really be filled before load time, since we have no idea where in virtual memory the DLLs will be loaded (since Windows Vista 32 bit use ASLR). Normally the IAT table (file on disk) contains information how to find the virtual addresses for each function in other DLLs (i.e. how to fill it self with proper addresses during load time). The Import Name Table (INT) contains the necessary information for this mission. Usually the INT and the IAT contains same information in the file on disk.

However, in the case of notepad.exe, the IAT (file on disk) is already filled with virtual addresses! The reason is because notepad.exe is bound to several DLLs. Using dumpbin (dumpbin /IMPORTS notepad.exe) we can see which DLLs notepad.exe is bound to.



dumpbin of notepad's import

I will describe binding here briefly, but I will probably have a more detailed explanation in another post.

Binding an EXE to a DLL makes the start up time shorter, when launching the application. Why? Because the virtual addresses to functions, outside the module, are already in the EXE (file on disk) and the loader does not need to bother about this. However, in order for the loader to not be bothered, there are some constraints that's need to be fulfilled. The virtual addresses in the IAT in the EXE (file on disk) assumes that the DLL will be loaded at its preferred load address, i.e. according to the ImageBase field in the Optional Header of the DLL. Further, the loaded DLL must be the version that the EXE expect (correct timestamp). Above notepad.exe expect the versions of all DLLs to be from Jan 19 2008.

Since Windows Vista 32 bit use ASLR, and notepad is ASLR compatible, the DLLs will probably not be loaded at its preferred load address. Further, most DLLs has probably been updated (since 2008), i.e. new versions/timestamps. These issues will force the loader to use the INT information even though the IAT is already filled with virtual addresses (since these virtual addresses on disk are incorrect in this case).

So finally, the answer to my introductory question, why is the first part of the .text section differing so much (file on disk v.s. file in memory)? The first part of the .text section is the IAT, which contains virtual addresses. File on disk contains virtual addresses to functions in DLLs of a certain (old) version (due to binding), but in memory, these virtual addresses are updated by the loader (if needed).

You are welcome to leave comments, complaints or questions!