Multiple definitions in COFF

A couple of days ago, I was curious about how the linker deals with multiple definitions of a function. So I read Microsoft's online PE documentation about the COFF format and how COFF was dealing with multiple defintions. In this post, I'm going to investigate some simple object files with dumpbin from Microsoft. I'm using Visual studio 2019, C++ 14, 32 bit. The compiler flags will be stated further down in this post. So grab another cup of coffee and let's get started! 

As mentioned above, the online PE documentation is of great help when understanding the structure of the object file. It states that a section is a basic unit of code or data within a PE or COFF file. Below, we will learn that COMDAT is a special type of section, and we will further dig into this. 

First, I will have a brief discussion with an introductory example of the symbol table. Then I will present a couple of setups, where each setup is slightly different. My focus will be on the .text sections, since the code resides inside those. Below is the code for this introductory example.

int main()
{
   return 0;
}

The code resides inside main.cpp and is compiled into main.obj, using these flags.

/JMC /permissive- /ifcOutput "Debug\" /GS /analyze- /W3 /Zc:wchar_t /ZI /Gm- /Od /sdl /Fd"Debug\vc142.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_DEBUG" /D "_CONSOLE" /D "_UNICODE" /D "UNICODE" /errorReport:prompt /WX- /Zc:forScope /RTC1 /Gd /Oy- /MDd /FC /Fa"Debug\" /EHsc /nologo /Fo"Debug\" /Fp"Debug\project.pch" /diagnostics:column

I will dump the symbol table by using dumpbin, like this:

dumpbin /symbols main.obj

Dumping the symbols gives output below (truncated).

COFF SYMBOL TABLE
000 010575C9 ABS    notype       Static       | @comp.id
001 80010391 ABS    notype       Static       | @feat.00
002 00000002 ABS    notype       Static       | @vol.md
003 00000000 SECT1  notype       Static       | .drectve
    Section length   85, #relocs    0, #linenums    0, checksum        0
    Relocation CRC 00000000
006 00000000 SECT2  notype       Static       | .debug$S
    Section length  1E0, #relocs    2, #linenums    0, checksum        0
    Relocation CRC BE724FAD
009 00000000 SECT3  notype       Static       | .msvcjmc
    Section length    1, #relocs    0, #linenums    0, checksum 77073096
    Relocation CRC 00000000
00C 00000000 SECT3  notype       Static       | __D3C22540_main@cpp
00D 00000000 SECT4  notype       Static       | .text$mn
    Section length    5, #relocs    0, #linenums    0, checksum 672BE856, selection    2 (pick any)
    Relocation CRC 4A6A8444
010 00000000 SECT5  notype       Static       | .debug$S
    Section length   98, #relocs    3, #linenums    0, checksum        0, selection    5 (pick associative Section 0x4)
    Relocation CRC B9D5806A
013 00000000 SECT6  notype       Static       | .text$mn
    Section length   37, #relocs    3, #linenums    0, checksum A2398015, selection    1 (pick no duplicates)
    Relocation CRC 2E7CF6B9
016 00000000 SECT7  notype       Static       | .debug$S
    Section length  104, #relocs    9, #linenums    0, checksum        0, selection    5 (pick associative Section 0x6)
    Relocation CRC 351E60C8
019 00000000 SECT6  notype ()    External     | _main

In the truncated table above, we can see 26 entries from the symbol table, where the 26th entry is telling us that the symbol _main is external and belongs to section #6. 

From the online PE documentation, we can understand that the sections are defined inside the symbol table as well. For such entry, SECT6 (entry 0x13 above), there follows another entry (entry 0x014 above), providing information about the section. This entry is in Auxiliary Format 5 and since section #6 is a COMDAT section (see below for explanation), there is a selection type defined, in this case 1 (pick no duplicates). The selection type tells linker how to deal with multiple definitions of COMDATA sections. In this particular case, the linker only accepts one definition, if there are more, the linker issues a multiple definition error.

Further, I will dump the headers for each section, like this:

dumpbin /headers main.obj

Dumping the headers gives the output below (only section header #6 pasted).

SECTION HEADER #6
.text$mn name
       0 physical address
       0 virtual address
      37 size of raw data
     501 file pointer to raw data (00000501 to 00000537)
     538 file pointer to relocation table
       0 file pointer to line numbers
       3 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= _main
         16 byte align
         Execute Read

In the truncated output above, we can see section flags 60501020. The online PE documentation (Section Flags) can be used to understand the 60501020. It's also briefly described in the output above; the section contains executable code, the section contains COMDAT data, the section can be executed as code, the section can be read.

Now let's look into the different setups. Each setup contains a main.cpp with a main function and a variation of additional source files. The figure summarize the source files and resulting object files with its symbols. The compiler options are according to the introductory example above. In each setup, after the figure, I will present the important parts of the object files (using the commands described above) and have a brief discussion. Let's look at the first one.

Setup 1

Figure 1

void helperfunc1()
{}

00D 00000000 SECT4  notype       Static       | .text$mn
    Section length   35, #relocs    3, #linenums    0, checksum  DB372CC, selection    1 (pick no duplicates)
    Relocation CRC E4C10A68

019 00000000 SECT4  notype ()    External     | ?helperfunc1@@YAXXZ (void __cdecl helperfunc1(void))

SECTION HEADER #4
.text$mn name
       0 physical address
       0 virtual address
      35 size of raw data
     45A file pointer to raw data (0000045A to 0000048E)
     48F file pointer to relocation table
       0 file pointer to line numbers
       3 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl helperfunc1(void)" (?helperfunc1@@YAXXZ)
         16 byte align
         Execute Read

From above, we see entry 0x00D and entry 0x019 in the symbol table, which defines SECT4 and the external symbol ?helperfunc1@@YAXXZ, which is the name mangled name of the function helperfunc1. 

Setup 2

Figure 2

static void helperfunc1()
{}

COFF SYMBOL TABLE
000 010575C9 ABS    notype       Static       | @comp.id
001 80010391 ABS    notype       Static       | @feat.00
002 00000002 ABS    notype       Static       | @vol.md
003 00000000 SECT1  notype       Static       | .drectve
    Section length   41, #relocs    0, #linenums    0, checksum        0
    Relocation CRC 00000000
006 00000000 SECT2  notype       Static       | .debug$S
    Section length  148, #relocs    2, #linenums    0, checksum        0
    Relocation CRC 3A286AB6
009 00000000 SECT3  notype       Static       | .msvcjmc
    Section length    1, #relocs    0, #linenums    0, checksum 77073096
    Relocation CRC 00000000
00C 00000000 SECT3  notype       Static       | __5291C45E_helperfuncs@cpp
00D 00000000 SECT4  notype       Static       | .debug$T
    Section length   5C, #relocs    0, #linenums    0, checksum        0
    Relocation CRC 00000000
010 00000000 SECT5  notype       Static       | .chks64
    Section length   28, #relocs    0, #linenums    0, checksum        0
    Relocation CRC 00000000

Running the dumpbin command with the symbol option gives no helperfunc1 symbol. The symbol is not present in the object file, since it is not used in the translation unit and is declared as static.

Setup 3

Figure 3

static void helperfunc1()
{}

void libfunc1()
{
   helperfunc1();
}

00D 00000000 SECT4  notype       Static       | .text$mn
    Section length   35, #relocs    3, #linenums    0, checksum  DB372CC, selection    1 (pick no duplicates)
    Relocation CRC E4C10A68

013 00000000 SECT6  notype       Static       | .text$mn
    Section length   3A, #relocs    4, #linenums    0, checksum 81ED1D3A, selection    1 (pick no duplicates)
    Relocation CRC B844FE03

01F 00000000 SECT4  notype ()    Static       | ?helperfunc1@@YAXXZ (void __cdecl helperfunc1(void))

020 00000000 SECT6  notype ()    External     | ?libfunc1@@YAXXZ (void __cdecl libfunc1(void))

SECTION HEADER #4
.text$mn name
       0 physical address
       0 virtual address
      35 size of raw data
     4AA file pointer to raw data (000004AA to 000004DE)
     4DF file pointer to relocation table
       0 file pointer to line numbers
       3 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl helperfunc1(void)" (?helperfunc1@@YAXXZ)
         16 byte align
         Execute Read

SECTION HEADER #6
.text$mn name
       0 physical address
       0 virtual address
      3A size of raw data
     65B file pointer to raw data (0000065B to 00000694)
     695 file pointer to relocation table
       0 file pointer to line numbers
       4 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl libfunc1(void)" (?libfunc1@@YAXXZ)
         16 byte align
         Execute Read

Now libfunc is calling helperfunc1. From above, we see entry 0x00D and entry 0x013 in the symbol table, which defines SECT4 and SECT6 respectively. We can also see the the symbol ?helperfunc1@@YAXXZ again, this time it's defined as a static symbol, since we defined helperfunc1 as static in the code. Further, we also see the external symbol ?libfunc1@@YAXXZ. Both symbols states for the linker that there is only one defintion allowed.

Setup 4

Figure 4

inline void helperfunc1()
{}

COFF SYMBOL TABLE
000 010575C9 ABS    notype       Static       | @comp.id
001 80010391 ABS    notype       Static       | @feat.00
002 00000002 ABS    notype       Static       | @vol.md
003 00000000 SECT1  notype       Static       | .drectve
    Section length   41, #relocs    0, #linenums    0, checksum        0
    Relocation CRC 00000000
006 00000000 SECT2  notype       Static       | .debug$S
    Section length  148, #relocs    2, #linenums    0, checksum        0
    Relocation CRC 3A286AB6
009 00000000 SECT3  notype       Static       | .msvcjmc
    Section length    1, #relocs    0, #linenums    0, checksum 77073096
    Relocation CRC 00000000
00C 00000000 SECT3  notype       Static       | __5291C45E_helperfuncs@cpp
00D 00000000 SECT4  notype       Static       | .debug$T
    Section length   5C, #relocs    0, #linenums    0, checksum        0
    Relocation CRC 00000000
010 00000000 SECT5  notype       Static       | .chks64
    Section length   28, #relocs    0, #linenums    0, checksum        0
    Relocation CRC 00000000

Running the dumpbin command with the symbol option gives no helperfunc1 symbol. The symbol is not present in the object file, since it is not used in the translation unit and declared inline. As a matter of fact, this symbols table looks identical with the symbol table from Setup 2.

Setup 5

Figure 5

inline void helperfunc1()
{}

void libfunc1()
{
   helperfunc1();
}

00D 00000000 SECT4  notype       Static       | .text$mn
    Section length   35, #relocs    3, #linenums    0, checksum  DB372CC, selection    2 (pick any)
    Relocation CRC E4C10A68

013 00000000 SECT6  notype       Static       | .text$mn
    Section length   3A, #relocs    4, #linenums    0, checksum 81ED1D3A, selection    1 (pick no duplicates)
    Relocation CRC B844FE03

01F 00000000 SECT4  notype ()    External     | ?helperfunc1@@YAXXZ (void __cdecl helperfunc1(void))

020 00000000 SECT6  notype ()    External     | ?libfunc1@@YAXXZ (void __cdecl libfunc1(void))

SECTION HEADER #4
.text$mn name
       0 physical address
       0 virtual address
      35 size of raw data
     4AA file pointer to raw data (000004AA to 000004DE)
     4DF file pointer to relocation table
       0 file pointer to line numbers
       3 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl helperfunc1(void)" (?helperfunc1@@YAXXZ)
         16 byte align
         Execute Read

SECTION HEADER #6
.text$mn name
       0 physical address
       0 virtual address
      3A size of raw data
     65B file pointer to raw data (0000065B to 00000694)
     695 file pointer to relocation table
       0 file pointer to line numbers
       4 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl libfunc1(void)" (?libfunc1@@YAXXZ)
         16 byte align
         Execute Read 

Now libfunc1 is calling helperfunc1. From above, we see entry 0x00D and entry 0x013 in the symbol table, which defines SECT4 and SECT6 respectively. We can also see the the external symbol ?helperfunc1@@YAXXZ and libfunc1@@YAXXZ. Unlike Setup 3, helperfunc1 is external and its selection type is 1, meaning any symbol can be linked.

Setup 6

Figure 6

inline void helperfunc1()
{}

#include "helperfuncs.hpp"

void libfunc1()
{
   helperfunc1();
}

00E 00000000 SECT4  notype       Static       | .text$mn
    Section length   35, #relocs    3, #linenums    0, checksum  DB372CC, selection    2 (pick any)
    Relocation CRC 16A75AA6

014 00000000 SECT6  notype       Static       | .text$mn
    Section length   3A, #relocs    4, #linenums    0, checksum 81ED1D3A, selection    1 (pick no duplicates)
    Relocation CRC B844FE03

020 00000000 SECT4  notype ()    External     | ?helperfunc1@@YAXXZ (void __cdecl helperfunc1(void))

021 00000000 SECT6  notype ()    External     | ?libfunc1@@YAXXZ (void __cdecl libfunc1(void))

SECTION HEADER #4
.text$mn name
       0 physical address
       0 virtual address
      35 size of raw data
     53B file pointer to raw data (0000053B to 0000056F)
     570 file pointer to relocation table
       0 file pointer to line numbers
       3 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl helperfunc1(void)" (?helperfunc1@@YAXXZ)
         16 byte align
         Execute Read

SECTION HEADER #6
.text$mn name
       0 physical address
       0 virtual address
      3A size of raw data
     6EC file pointer to raw data (000006EC to 00000725)
     726 file pointer to relocation table
       0 file pointer to line numbers
       4 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl libfunc1(void)" (?libfunc1@@YAXXZ)
         16 byte align
         Execute Read

This setup does not differ from Setup 5. The only difference is the created helperfuncs.hpp, which is included in the created libfuncs1.cpp. This is a more typical setup, where the inline function is defined inside the header file. Note that the symbol entries and sections are identical with Setup 5.

Setup 7

Figure 7

inline void helperfunc1()
{}

#include "helperfuncs.hpp"

void libfunc1()
{
   helperfunc1();
}

#include "helperfuncs.hpp"

void libfunc2()
{
   helperfunc1();
}

00E 00000000 SECT4  notype       Static       | .text$mn
    Section length   35, #relocs    3, #linenums    0, checksum  DB372CC, selection    2 (pick any)
    Relocation CRC 16A75AA6

014 00000000 SECT6  notype       Static       | .text$mn
    Section length   3A, #relocs    4, #linenums    0, checksum 81ED1D3A, selection    1 (pick no duplicates)
    Relocation CRC 4796C86E

020 00000000 SECT4  notype ()    External     | ?helperfunc1@@YAXXZ (void __cdecl helperfunc1(void))

021 00000000 SECT6  notype ()    External     | ?libfunc2@@YAXXZ (void __cdecl libfunc2(void))

SECTION HEADER #4
.text$mn name
       0 physical address
       0 virtual address
      35 size of raw data
     533 file pointer to raw data (00000533 to 00000567)
     568 file pointer to relocation table
       0 file pointer to line numbers
       3 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl helperfunc1(void)" (?helperfunc1@@YAXXZ)
         16 byte align
         Execute Read

SECTION HEADER #6
.text$mn name
       0 physical address
       0 virtual address
      3A size of raw data
     6E4 file pointer to raw data (000006E4 to 0000071D)
     71E file pointer to relocation table
       0 file pointer to line numbers
       4 number of relocations
       0 number of line numbers
60501020 flags
         Code
         COMDAT; sym= "void __cdecl libfunc2(void)" (?libfunc2@@YAXXZ)
         16 byte align
         Execute Read

This is even a more typical setup. The inline function is defined inside the header file. The header file is included in two cpp files. 

Above is only the symbol table for the additional file libfuncs2.obj stated. We can compare the symbols from libfuncs1.obj (from Setup 6) with the symbols from libfuncs2.obj in this Setup. Both object files contains the external symbol?helperfunc1@@YAXXZ, with selection type 2. Thanks to this selection type, the linker will not complain about multiple defintions during link time.

You are welcome to leave comments, complaints or questions!

IAT in notepad.exe

A couple of weeks ago, I was inspecting and comparing the .text section of a PE file (notepad.exe) and the corresponding .text section which was loaded in memory. I realized that the first part of the .text section was not even close to be equal (file on disk v.s. file in memory). I googled a lot about this and finally learned that some part of the .text section may consist of the Import Address Table (IAT).

In this post I'm going to show you what I learned and what is going on in the IAT of notepad.exe. As usual, I'm using Windows Vista 32 bit.

If you don't know it, the IAT is a lookup table. This table is filled with virtual addresses to functions in other modules (in this case, functions outside notepad.exe) when the executable is loaded into memory. I will describe it in more detail later in this post. Do not confuse the IAT with the Import Name Table (INT). The INT is also a part of the PE file, but is not the same as the IAT. However, they do have some connections.

Let's start where I started. It was a sunny day and I was in a very good mood for some coding. Well, more seriously, I was inspecting the .text section using PE Insider from Cerbero. This is what I saw in the HEX view. Note that the .text section starts at offset 0x400 in the file.

PE Insider - Content of PE file (file on disk)

Back a few weeks, when I was inspecting notepad.exe in PE Insider, I thought that the .text section only contained binary machine code. And the binary machine code is not supposed to change when the file is loaded to memory. So I was expecting that the contents to be more or less the same in memory. Let's check out the memory when notepad.exe is executed. To do this I'm using WinDbg.



WinDbg - Content of memory (file loaded in memory)

The .text section starts at 0x00721000 since Windows decided to place notepad.exe at memory location 0x00720000 this time (remember that Windows Vista use ASLR and that notepad is ASLR compatible), and the Relative Virtual Address (RVA) for the .text section (according to the PE file, i.e. notepad.exe) is 0x1000.

It really puzzled me first time I compared the file content with the memory content. There were a lot of differences. First I thought WinDbg was injecting some code to handle the debug break, or maybe it was some unwanted code in the memory? I started to compare the contents (PE Insider - WinDbg) manually (byte by byte) and realized at some memory location, the contents started to be the same. See below.



PE Insider - Content of PE file (file on disk)

WinDbg - Content of memory (file loaded in memory)



Somewhere around the red marking, the contents starts to be equal. Why was only the first part differing? I was thinking that PE Insider maybe was not showing the correct file content. So I downloaded another PE viewer, PEBrowser Pro from SmidgeonSoft. I started PEBrowser Pro, opened notepad.exe and clicked the .text section, which opened a new window with the contents below.

PEBrowser Pro - .text section disassembled

PEBrowser Pro is a little more sophisticated than PE Insider, it can disassembles the .text section among other things. This is what we see above. But why did it start disassemble at 0x100138D and not 0x1001000? Note that PEBrowser Pro thinks notepad.exe should be loaded at its preferred load address which is 0x1000000 according to the ImageBase field in the Optional Header of notepad.exe (which means the .text section starts at 0x1001000). Apparently PEBrowser Pro realize that the first bytes in the .text section is not actually binary machine code, but something else. By having a look at the treeview of notepad.exe in PEBrowser Pro, it gave me some hints.

PEBrowser Pro - Treeview of file contents

Yes, apparently the .text section may consist of more than just binary machine code. It contains the IAT among other things.

But how do PEBrowser Pro know that the binary machine code starts at 0x100138D? I wasn't sure about this, but I guessed that the header of the PE file (notepad.exe) may give a clue. Back in PE Insider and viewing the Data Directories in the Optional Header I saw this.

PE Insider - Data Directories

The Data Directories, more specific, the Import Address Table Directory, tells us that the table starts at RVA 0x1000 i.e. this is where the .text section starts! Its size is 0x388 bytes. At this point, things began to be clear for me.

Next thing to understand was how to interpret the bytes in the IAT.

After some googling, I realized that when the PE file (notepad.exe) is loaded into memory, the IAT is filled with virtual addresses. This is virtual addresses in other modules loaded within the same virtual memory space as notepad.exe.

So let's take an example. Let's investigate the first address in the IAT (in memory), the address at RVA 0x1000. Below we can see that the first address in the IAT is 0x77ae765e. This virtual address is in another module, more specific in advapi32.dll and it is the address of the function "RegQueryValueExW".



WinDbg - IAT (file loaded in memory)

Below we see some part of the disassembled function.

WinDbg - Disassembling RegQueryValueExW

Well, now we know the meaning of the IAT bytes when the PE file is loaded in memory, but why does they differ comparing to the bytes on disk?

As mentioned above, the IAT is filled with addresses during load time, so the IAT can't really be filled before load time, since we have no idea where in virtual memory the DLLs will be loaded (since Windows Vista 32 bit use ASLR). Normally the IAT table (file on disk) contains information how to find the virtual addresses for each function in other DLLs (i.e. how to fill it self with proper addresses during load time). The Import Name Table (INT) contains the necessary information for this mission. Usually the INT and the IAT contains same information in the file on disk.

However, in the case of notepad.exe, the IAT (file on disk) is already filled with virtual addresses! The reason is because notepad.exe is bound to several DLLs. Using dumpbin (dumpbin /IMPORTS notepad.exe) we can see which DLLs notepad.exe is bound to.



dumpbin of notepad's import

I will describe binding here briefly, but I will probably have a more detailed explanation in another post.

Binding an EXE to a DLL makes the start up time shorter, when launching the application. Why? Because the virtual addresses to functions, outside the module, are already in the EXE (file on disk) and the loader does not need to bother about this. However, in order for the loader to not be bothered, there are some constraints that's need to be fulfilled. The virtual addresses in the IAT in the EXE (file on disk) assumes that the DLL will be loaded at its preferred load address, i.e. according to the ImageBase field in the Optional Header of the DLL. Further, the loaded DLL must be the version that the EXE expect (correct timestamp). Above notepad.exe expect the versions of all DLLs to be from Jan 19 2008.

Since Windows Vista 32 bit use ASLR, and notepad is ASLR compatible, the DLLs will probably not be loaded at its preferred load address. Further, most DLLs has probably been updated (since 2008), i.e. new versions/timestamps. These issues will force the loader to use the INT information even though the IAT is already filled with virtual addresses (since these virtual addresses on disk are incorrect in this case).

So finally, the answer to my introductory question, why is the first part of the .text section differing so much (file on disk v.s. file in memory)? The first part of the .text section is the IAT, which contains virtual addresses. File on disk contains virtual addresses to functions in DLLs of a certain (old) version (due to binding), but in memory, these virtual addresses are updated by the loader (if needed).

You are welcome to leave comments, complaints or questions!