Seneca How-To Page

For contribution marks I created a How To Page on the Seneca Wiki. This How to page will have walkthroughs and tutorials for all the open source projects Seneca is working on.

Right now I have step by step instructions for how to run different tests on Firefox. Others might add how to write a test.

I tried to make the walkthroughs nicer looking than black on white. It is not necessary for people to follow my lead but adding a little extra flare is always nice.

PGO Patch

I have submitted a patch to for the bug 465592.

The Bug

Currently PGO doesn't work for Thunderbird. For Windows builds the compilation breaks in the LCMS module for the Mozilla core. If PGO is disable for this module then Thunderbird can be built with PGO enabled.

The Patch

After talking with David Humphrey and Ted Mielczarek I modified the Makefile.in in the module/lcms/src directory. To include the following lines.

ifndef MOZ_ENABLE_LIBXUL
NO_PROFILE_GUIDED_OPTIMIZE = 1
endif

The MOZ_ENABLE_LIBXUL flag is used to link libraries into a single xul.dll library. On Firefox this flag is used and a number of libraries are linked in the xul.dll, one of these libraries is the mozlcms.lib. This does not happen with Thunderbird. Instead the libraries are made into DLL's. The mozlcms.dll crashes during the optimization build for PGO.


Checking if the flag has not been set means that the mozlcms.lib will be made into a DLL. Since this will crash if PGO is enabled, the NO_PROFILE_GUIDED_OPTIMIZE is set to disable PGO for this one module.

The Result

Well, it works otherwise I would not have posted the patch. I am waiting on a review.

Understanding Thunderbird Build Process

I've been reading the Makefiles for Thunderbird to understand how it compiles. My interest is in the Mozilla core module lcms.

I'm trying to understand if it is possible to disable PGO for that module through the Thunderbird Makefiles instead of modifying the lcms Makefile.

I notice that the flag LCMS_CFLAGS=-DLCMS_DLL is set in the Makefile in the Mozilla core that gets downloaded when using the client.py file. This flag is not set in the Mozilla core when checking out Firefox.

Talking to David Humphrey he mentioned a flag MOZ_ENABLE_LIBXUL. This flag was set in my build output for the Firefox build but not in the Thunderbird build.

In the build output for Thunderbird I noticed that the lcms objs are combined into a mozlcms.dll and in the Firefox build lcms objs are combined into a lib and later it is combined with the xul.dll.

Options for disabling PGO for the LCMS module:
Option1:

A flag could be added in the makefile for the lcms module that will enable or disable PGO. But I would prefer not to modify the Mozilla Core files.

Option2:

After the mozilla core is built, could use another script to build the lcms module again but this time as a lib and re-compile the xul.dll and add the mozlcms.lib into the xul.dll. But this is kind of a hack.

I'll keep looking at it and asking the guys "who know" the best way to proceed.

Let's start a converation!!

I've posted a bug for Thunderbird to start a conversation on the best way to approach enabling PGO for Thunderbird.

Currently I have to modify a file in the Mozilla core to enable PGO for Thunderbird and I shouldn't have to. I am looking into how to modify the Thunderbird build process so it can successfully compile without modifying the Mozilla core.

0.2 Release

I have released my 0.2 Release.

I am happy to say that I have successfully built Thunderbird with PGO enabled. A closed bug has been opened as I continue to for on this. The problem is that code in the Mozilla core needs to be modified for Thunderbird to compile with PGO but Firefox which uses the same code does not.

Looking Ahead
For 0.3 Release I want to figure out how to modify the Thunderbird make files to correct the pgo related issue without modifying the Mozilla core.

If I have time I want to enable PGO for modules specific to Thunderbird.

Success!!!!

I have successfully compiled Thunderbird with PGO.

I had to disable pgo for the Color Management module.
I included NO_PROFILE_GUIDED_OPTIMIZE = 1 right before the line that indicated what optimization this module is to have.

The Problem
I was receiving an internal linker error when the compiler was trying to optimize the mozlcms.dll. The error that was displayed was f:\mozilla\thunderbird\mozilla\modules\lcms\src\cmscgats.c (...) LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage. I blogged about this error.

Steps to Resolve
At first I was compiling with VS 2008 which broke my original Firefox build. I figured that was the problem. I decided to create a VM of Windows 2003 and setup an environment as close as I could to the tinderboxes.

I installed Visual Studio 2005, VS 2005 SP1, and the Vista SDK. I tried to compile again and I received the same error. At this point I figured it was the Vista SDK since the environment I used with VS 2008 also had Vista SDK. So.....I have an environment that has Visual Studio 2005 without the Vista SDK.

I still got the error.

At this point I decided that I would disable that one module and hoped it all goes well.... and it DID!!!

In between I tried a bunch of other things that had no affect so I won't mention them but I have been compiling and compiling and installing and installing since Monday.

Details
A bug has been re-opened and I have posted to it.

I noticed that Thunderbird creates 27 DLL's in the dist/bin directory and Firefox creates 17 DLL's in the same directory.

The xul.dll file is 10x larger for Firefox than Thunderbird.

I'll post more details about what objects were optimized and what percentages they were optimized. I accidentally blew away my last build.

So Close!

I appears that the enabling of PGO for Thunderbird might be really easy. All I had to due was change the Thunderbird client.mk file to include the profiledbuild option that is available in the Firefox client.mk file.

Thunderbird has a clone of the mozilla-central source which already has PGO enabled for individual modules.

To turn it own all that really happens is the setting of two environment variables.

####################################
# Profile-Guided Optimization
#  To use this, you should set the following variables in your mozconfig
#    mk_add_options PROFILE_GEN_SCRIPT=/path/to/profile-script
#
#  The profile script should exercise the functionality to be included
#  in the profile feedback.
#
#  This is up here, outside of the MOZ_CURRENT_PROJECT logic so that this
#  is usable in multi-pass builds, where you might not have a runnable
#  application until all the build passes and postflight scripts have run.
ifdef MOZ_OBJDIR
  PGO_OBJDIR = $(MOZ_OBJDIR)
else
  PGO_OBJDIR := $(TOPSRCDIR)
endif

profiledbuild::
 $(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_GENERATE=1
 OBJDIR=${PGO_OBJDIR} $(PROFILE_GEN_SCRIPT)
 $(MAKE) -f $(TOPSRCDIR)/client.mk maybe_clobber_profiledbuild
 $(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_USE=1

So the first build has the environment variable MOZ_PROFILE_GENERATE=1 set. This causes a instrumental build to be built. This is used for profiling and gathering performance data. Then a script is run to profile the build. After some clean up it is built again with the performance data included. The environment variable MOZ_PROFILE_USE=1 indicates that.

But alas my build failed. It created the first build and opened Thunderbird but the second build failed. It looks like a compiler error. I have seen the error before but I cannot remember what caused it.

f:\mozilla\thunderbird\mozilla\modules\lcms\src\cmscgats.c(875) : fatal error C1001: An internal error has occurred in the compiler.
(compiler file 'f:\rtm\vctools\compiler\utc\src\P2\main.c[0x10CA10D0:0x00000008]', line 182)
 To work around this problem, try simplifying or changing the program near the locations listed above.
Please choose the Technical Support command on the Visual C++ 
 Help menu, or open the Technical Support help file for more information

LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage

  Version 8.00.50727.42

  ExceptionCode            = C0000005
  ExceptionFlags           = 00000000
  ExceptionAddress         = 10CA10D0 (10B00000) "f:\Program Files\Microsoft Visual Studio 8\VC\BIN\c2.dll"
  NumberParameters         = 00000002
  ExceptionInformation[ 0] = 00000000
  ExceptionInformation[ 1] = 00000008

CONTEXT:
  Eax    = 00000000  Esp    = 0012ED28
  Ebx    = 00000015  Ebp    = 0012ED3C
  Ecx    = 028DDA01  Esi    = 02866060
  Edx    = 0297AE12  Edi    = 00000000
  Eip    = 10CA10D0  EFlags = 00010246
  SegCs  = 0000001B  SegDs  = 00000023
  SegSs  = 00000023  SegEs  = 00000023
  SegFs  = 0000003B  SegGs  = 00000000
  Dr0    = 00000000  Dr3    = 00000000
  Dr1    = 00000000  Dr6    = 00000000
  Dr2    = 00000000  Dr7    = 00000000
make[6]: *** [mozlcms.dll] Error 232
make[6]: *** Deleting file `mozlcms.dll'
make[6]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla/modules/lcms/src'
make[5]: *** [libs] Error 2
make[5]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla/modules/lcms'
make[4]: *** [libs_tier_external] Error 2
make[4]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla'
make[3]: *** [tier_external] Error 2
make[3]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla'
make[2]: *** [default] Error 2
make[2]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb'
make: *** [build] Error 2

So close....

Thunderbird Testing Environment

I downloaded from Seneca a copy of Windows 2003 using the key provided to me through Seneca. On my XP machine using Virtual Box I have created a VM for win2k3.

I've been reading up and it looks like there is no real testing for Thunderbird. Currently the tinderboxes check to see if the code compiled and as long as it compiles everything is ok.

MailDev
I was chatting with some fella's and it turns out they have a few leak tests and Standard8 is working on the Tp tests ( performance startup tests ). Since I'll need some performance tests to verify my findings I'll probably be contributing those too.

Where I am!
I am trying to setup my win2k3 VM with the right patches and updates so I can duplicate the tinderboxes. I also need to install Visual 2005 with the Vista SDK and any patches mentioned on the Reference Platforms page on the Mozilla Wiki.

Next!
Start looking into enabling one module to use PGO for Thunderbird. The good stuff.

Time to put the nose to the grind.