0.3 Release

So this is my last contribution to the OSD 600 Course but not my last contribution to my project. With this release I will be slowing down my activity but I hope I find the time to finish what I have started.

0.3 Release Details
My 0.3 Release project release can be found here.

The summary is I have landed a patch that keeps PGO for Thunderbird breaking on Windows. A module in the mozilla-core called lcms breaks when it is not linked into the xul library. The patch basically disables pgo for that module if it is not being linked into the library otherwise pgo is enabled for it.

As I was trying to enable pgo for Thunderbird related modules I discovered that PGO gets enabled for all modules. This was good news until I noticed that only 5 modules were getting optimized out of 74.

As I looked into this it turned out that the folder that most of the profile data was in was being deleted before used to optimize the modules. I created a new bug for this problem.

I have added my proposed patch and using that patch and doing some simple profiling 69 of 74 modules were optimized.

And that is my 0.3 Release.

I have learned about using Make and Python and compiler options for PGO. I still don't understand PGO entirely and don't understand why it does what it does. Example would be when using the option to optimize for size the compile spits out that the module was optimized for speed. I guess at some point I should make an inquiry on the Microsoft forums.

I plan on taking the OSD700 class but am looking at working with Eclipse.

Thunderbird components getting Optimized

So the problem I ran into with optimizing Thunderbird was all the profile data was getting deleted before it was used. I opened a new bug for this.

The other part of the problem was the python script, pgomerge.py, was not looking in the objdir/dist/bin/components for the profile data. So I have edited the pgomerge.py file and it does find and merge in all the profile data.

AWESOME!!!

New problem...the profile data in the dist/bin is not getting touched. I pretty sure my script is broken.

Time to look into this python stuff some more.

New Bug for PGO

I've been trying to enable pgo on different Thunderbird components but I've been having problems.

Problem
The build output says these files are compiled with optimization flags but no optimization data is being collected.

It turns out the data is being collected and placed in the objdir/dist/bin/components directory. BUT!!!! This folder gets deleted.

I told ted about this. Here is a paraphrase of our conversation.

Basically, i need a flag to tell the make not to delete the folder when second profile build is being done and to look in the right directory for the optimization files.

I have posted a bug about this. This will be interesting to see how many components will get optimized without breaking.

Patch Landed!

My Patch has made it into mozilla-central.
Here is the patch.
Here is the bug.
Here is my happy dance.

Patch Reviewed!

Sweet! My patch was reviewed.

I just have to fix the comments in the patch and it is good to go.

I'm not sure what to do after I attach the updated patch.
This is the bug.

Optimizing the Thunderbird Address Book - cont'd.

I am still attempting to generate some optimization data for the address book.

One Success
The address book objects were compiled with the speed optimization. I must not have been hitting any xpcom calls to the c++ code. The opening and closing of windows must have been all js and xul code.

I'm looking through the c++ code to see what functions are available. I am going to try and profile to hit these functions. I also am trying to enable speed optimizations for the entire mail.dll in the Makefile in the mail.dll directory. This might not work.

Note:
I also notice that when I turn on PGO for Thunderbird that all the modules get the default optimization which is the size optimization.

I will have to build a non-pgo build of Thunderbird to see if it makes a difference.

Optimizing the Thunderbird Address Book.

I did a quick edit for the Makefile.in file for the address book in the Thunderbird source. I was hoping this would enable pgo for this module.

Here is the diff for the changes I made.

I was able to successfully build thunderbird with these options but during the profiling no optimization data was collected. If data is collected a .pgc file is created. This library is added into the mail.dll so a mail.pgc file should have been created. None was.

I haven't looked at my build output to confirm that the library and mail.dll were compiled with the optimizations. I did notice that mail.dll library was compiled with the instrumental option. This means the mail.dll had the extra instructions to collect profile data. It is possible that my profiling did not actual affect the address book.

I hope to get some more information before my 0.3 is released.

Patch Review Process

I've had my patch posted for about a week and still haven't not had a review. So I decided to jump on IRC and try and ask for a review.

My Project is PGO and apparently there is only one guy who owns that sort of code, Ted Mielczarek. I sent a private message to Ted and an open message on the #maildev channel to see if I could get a review. Ted responded that he is working on his review queue and would be getting to my patch. On the #maildev channel Standard8 pointed me to Ted.

So it all rests with Ted.

I was nervous posting on IRC or messaging Ted because I still don't understand the etiquette of that sort of thing. But everyone responded nicely which makes me feel better.

This experience with IRC is helping me feel better about asking questions from people I don't know.

Seneca How-To Page

For contribution marks I created a How To Page on the Seneca Wiki. This How to page will have walkthroughs and tutorials for all the open source projects Seneca is working on.

Right now I have step by step instructions for how to run different tests on Firefox. Others might add how to write a test.

I tried to make the walkthroughs nicer looking than black on white. It is not necessary for people to follow my lead but adding a little extra flare is always nice.

PGO Patch

I have submitted a patch to for the bug 465592.

The Bug

Currently PGO doesn't work for Thunderbird. For Windows builds the compilation breaks in the LCMS module for the Mozilla core. If PGO is disable for this module then Thunderbird can be built with PGO enabled.

The Patch

After talking with David Humphrey and Ted Mielczarek I modified the Makefile.in in the module/lcms/src directory. To include the following lines.

ifndef MOZ_ENABLE_LIBXUL
NO_PROFILE_GUIDED_OPTIMIZE = 1
endif

The MOZ_ENABLE_LIBXUL flag is used to link libraries into a single xul.dll library. On Firefox this flag is used and a number of libraries are linked in the xul.dll, one of these libraries is the mozlcms.lib. This does not happen with Thunderbird. Instead the libraries are made into DLL's. The mozlcms.dll crashes during the optimization build for PGO.


Checking if the flag has not been set means that the mozlcms.lib will be made into a DLL. Since this will crash if PGO is enabled, the NO_PROFILE_GUIDED_OPTIMIZE is set to disable PGO for this one module.

The Result

Well, it works otherwise I would not have posted the patch. I am waiting on a review.

Understanding Thunderbird Build Process

I've been reading the Makefiles for Thunderbird to understand how it compiles. My interest is in the Mozilla core module lcms.

I'm trying to understand if it is possible to disable PGO for that module through the Thunderbird Makefiles instead of modifying the lcms Makefile.

I notice that the flag LCMS_CFLAGS=-DLCMS_DLL is set in the Makefile in the Mozilla core that gets downloaded when using the client.py file. This flag is not set in the Mozilla core when checking out Firefox.

Talking to David Humphrey he mentioned a flag MOZ_ENABLE_LIBXUL. This flag was set in my build output for the Firefox build but not in the Thunderbird build.

In the build output for Thunderbird I noticed that the lcms objs are combined into a mozlcms.dll and in the Firefox build lcms objs are combined into a lib and later it is combined with the xul.dll.

Options for disabling PGO for the LCMS module:
Option1:

A flag could be added in the makefile for the lcms module that will enable or disable PGO. But I would prefer not to modify the Mozilla Core files.

Option2:

After the mozilla core is built, could use another script to build the lcms module again but this time as a lib and re-compile the xul.dll and add the mozlcms.lib into the xul.dll. But this is kind of a hack.

I'll keep looking at it and asking the guys "who know" the best way to proceed.

Let's start a converation!!

I've posted a bug for Thunderbird to start a conversation on the best way to approach enabling PGO for Thunderbird.

Currently I have to modify a file in the Mozilla core to enable PGO for Thunderbird and I shouldn't have to. I am looking into how to modify the Thunderbird build process so it can successfully compile without modifying the Mozilla core.

0.2 Release

I have released my 0.2 Release.

I am happy to say that I have successfully built Thunderbird with PGO enabled. A closed bug has been opened as I continue to for on this. The problem is that code in the Mozilla core needs to be modified for Thunderbird to compile with PGO but Firefox which uses the same code does not.

Looking Ahead
For 0.3 Release I want to figure out how to modify the Thunderbird make files to correct the pgo related issue without modifying the Mozilla core.

If I have time I want to enable PGO for modules specific to Thunderbird.

Success!!!!

I have successfully compiled Thunderbird with PGO.

I had to disable pgo for the Color Management module.
I included NO_PROFILE_GUIDED_OPTIMIZE = 1 right before the line that indicated what optimization this module is to have.

The Problem
I was receiving an internal linker error when the compiler was trying to optimize the mozlcms.dll. The error that was displayed was f:\mozilla\thunderbird\mozilla\modules\lcms\src\cmscgats.c (...) LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage. I blogged about this error.

Steps to Resolve
At first I was compiling with VS 2008 which broke my original Firefox build. I figured that was the problem. I decided to create a VM of Windows 2003 and setup an environment as close as I could to the tinderboxes.

I installed Visual Studio 2005, VS 2005 SP1, and the Vista SDK. I tried to compile again and I received the same error. At this point I figured it was the Vista SDK since the environment I used with VS 2008 also had Vista SDK. So.....I have an environment that has Visual Studio 2005 without the Vista SDK.

I still got the error.

At this point I decided that I would disable that one module and hoped it all goes well.... and it DID!!!

In between I tried a bunch of other things that had no affect so I won't mention them but I have been compiling and compiling and installing and installing since Monday.

Details
A bug has been re-opened and I have posted to it.

I noticed that Thunderbird creates 27 DLL's in the dist/bin directory and Firefox creates 17 DLL's in the same directory.

The xul.dll file is 10x larger for Firefox than Thunderbird.

I'll post more details about what objects were optimized and what percentages they were optimized. I accidentally blew away my last build.

So Close!

I appears that the enabling of PGO for Thunderbird might be really easy. All I had to due was change the Thunderbird client.mk file to include the profiledbuild option that is available in the Firefox client.mk file.

Thunderbird has a clone of the mozilla-central source which already has PGO enabled for individual modules.

To turn it own all that really happens is the setting of two environment variables.

####################################
# Profile-Guided Optimization
#  To use this, you should set the following variables in your mozconfig
#    mk_add_options PROFILE_GEN_SCRIPT=/path/to/profile-script
#
#  The profile script should exercise the functionality to be included
#  in the profile feedback.
#
#  This is up here, outside of the MOZ_CURRENT_PROJECT logic so that this
#  is usable in multi-pass builds, where you might not have a runnable
#  application until all the build passes and postflight scripts have run.
ifdef MOZ_OBJDIR
  PGO_OBJDIR = $(MOZ_OBJDIR)
else
  PGO_OBJDIR := $(TOPSRCDIR)
endif

profiledbuild::
 $(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_GENERATE=1
 OBJDIR=${PGO_OBJDIR} $(PROFILE_GEN_SCRIPT)
 $(MAKE) -f $(TOPSRCDIR)/client.mk maybe_clobber_profiledbuild
 $(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_USE=1

So the first build has the environment variable MOZ_PROFILE_GENERATE=1 set. This causes a instrumental build to be built. This is used for profiling and gathering performance data. Then a script is run to profile the build. After some clean up it is built again with the performance data included. The environment variable MOZ_PROFILE_USE=1 indicates that.

But alas my build failed. It created the first build and opened Thunderbird but the second build failed. It looks like a compiler error. I have seen the error before but I cannot remember what caused it.

f:\mozilla\thunderbird\mozilla\modules\lcms\src\cmscgats.c(875) : fatal error C1001: An internal error has occurred in the compiler.
(compiler file 'f:\rtm\vctools\compiler\utc\src\P2\main.c[0x10CA10D0:0x00000008]', line 182)
 To work around this problem, try simplifying or changing the program near the locations listed above.
Please choose the Technical Support command on the Visual C++ 
 Help menu, or open the Technical Support help file for more information

LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage

  Version 8.00.50727.42

  ExceptionCode            = C0000005
  ExceptionFlags           = 00000000
  ExceptionAddress         = 10CA10D0 (10B00000) "f:\Program Files\Microsoft Visual Studio 8\VC\BIN\c2.dll"
  NumberParameters         = 00000002
  ExceptionInformation[ 0] = 00000000
  ExceptionInformation[ 1] = 00000008

CONTEXT:
  Eax    = 00000000  Esp    = 0012ED28
  Ebx    = 00000015  Ebp    = 0012ED3C
  Ecx    = 028DDA01  Esi    = 02866060
  Edx    = 0297AE12  Edi    = 00000000
  Eip    = 10CA10D0  EFlags = 00010246
  SegCs  = 0000001B  SegDs  = 00000023
  SegSs  = 00000023  SegEs  = 00000023
  SegFs  = 0000003B  SegGs  = 00000000
  Dr0    = 00000000  Dr3    = 00000000
  Dr1    = 00000000  Dr6    = 00000000
  Dr2    = 00000000  Dr7    = 00000000
make[6]: *** [mozlcms.dll] Error 232
make[6]: *** Deleting file `mozlcms.dll'
make[6]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla/modules/lcms/src'
make[5]: *** [libs] Error 2
make[5]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla/modules/lcms'
make[4]: *** [libs_tier_external] Error 2
make[4]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla'
make[3]: *** [tier_external] Error 2
make[3]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla'
make[2]: *** [default] Error 2
make[2]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb/mozilla'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/i/mozilla/thunderbird/tb-pgo/tb'
make: *** [build] Error 2

So close....

Thunderbird Testing Environment

I downloaded from Seneca a copy of Windows 2003 using the key provided to me through Seneca. On my XP machine using Virtual Box I have created a VM for win2k3.

I've been reading up and it looks like there is no real testing for Thunderbird. Currently the tinderboxes check to see if the code compiled and as long as it compiles everything is ok.

MailDev
I was chatting with some fella's and it turns out they have a few leak tests and Standard8 is working on the Tp tests ( performance startup tests ). Since I'll need some performance tests to verify my findings I'll probably be contributing those too.

Where I am!
I am trying to setup my win2k3 VM with the right patches and updates so I can duplicate the tinderboxes. I also need to install Visual 2005 with the Vista SDK and any patches mentioned on the Reference Platforms page on the Mozilla Wiki.

Next!
Start looking into enabling one module to use PGO for Thunderbird. The good stuff.

Time to put the nose to the grind.

New Focus for PGO Project

For the rest of my PGO project I'll be focusing on adding PGO to Thunderbird. So my first task is to build Thunderbird. Since I'll be building and testing a lot I have scripts to automate this task for me. I just tweaked my scripts to now build for Thunderbird.

Bash Script Problems
I don't know why but when I tried to create a bash script from scratch it would behave as if it executed, as in no bash could find command error, but nothing would happen. I could get any output. So I copied an existing working script and modified it. Works for me.

Next Steps
Now that I can build Thunderbird I am going to try and replicate the Mozilla tinderbox setup for Thunderbird on my home machine so I can duplicate the performance testing environment. Once I have a baseline of how Thunderbird performs on my machine I can then start enabling PGO and see what kind of performance boost I get.

After that I would have lots of opportunities for other Seneca students to contribute by compiling and running the tests on their machines to see if they are also seeing a performance gain.

So now to get to it.....

0.1 Release - Firefox PGO

0.1 Release is done.

I have learned:

How to implement PGO on any application.
How to turn PGO on and off for modules in Firefox.
That profiling firefox anymore than opening and closing the browser doesn't yeild better performance.
How to run xpcshell, mochitest, and reftests on a Firefox build.
How to create bash scripts to build firefox and create reports from the build output.
How to reproduce the sqlite bug as report in this bug.
How to attach my results of test to a bug.
That sqlite crashes even in a stand alone application when compiled with PGO.
That I am going to turn my focus to Thunderbird and turn on PGO for Windows/Linux/Mac.

0.1 Release more detailed look

Experience
I have found myself frustrated, upset, surprised and excited as I work my way into the Mozilla Community. I am asking questions more openly, more willing to look dumb in the hopes of getting the information I need. I always search first, of course.

It really takes time to get a feel for how Mozilla does things. I just recently discovered that the trunk has all these readme's scattered everywhere. Probably the first place to look for help.

I still find the Mozilla wiki's and documentation to be lacking for someone who is not as experienced as the folks at Mozilla. I tell my wife that "it's documentation written for people who already know it". I find I have to research and look stuff up to understand what I read on Mozilla.

Now that I have my feet wet I am excited to push forward.

Review of PGO Data

45% increase not a result of PGO profiling of Google Maps.

I used build Firefox with only profiling of a page opening and then closing to check it against the loading times for Google Maps.

I re-ran all the tests and here is the summary of the results:

Firefox 3.0.3

Minefield PGO *

Minefield PGO **

2.15s (avg)

1.18s (avg)

1.17s (avg)

* Profiled using the Google Maps website.
** Profiled with only the browser opening then closing.

I also saved the Google Maps page on to my local server and when the pages were served from a local server the load times were all identical.

Serving Google Maps locally resulted in non of the javascript files Google Map uses being saved since they are loaded dynamically.

I have to conclude from this second performance test that minefield has some performance improvements not seen in Firefox 3.0.3 since the profiling did not improve Google Maps site loading.

Firefox PGO Details

According to the build output:

• 2938 objects are created
• 2246 of the 2938 objects are created with pgo optimizations
• 2101 of the 2246 objects are optimized for size
• 145 of the 2246 objects are optimized for speed
• 73 objects are optimized with profile data
• 69 of 73 objects didn't have any profile data to optimize with
• 49 of 73 objects are tests objects

In the end only 4 objects are optimized.

Merging ..\..\dist\bin\js3250!1.pgc
882 of 2904 ( 30.37%) profiled functions will be compiled for speed

Merging ..\..\dist\bin\xul!1.pgc
3433 of 84021 (  4.09%) profiled functions will be compiled for speed

Merging ..\..\dist\bin\xpcom!1.pgc
55 of 55 (100.00%) profiled functions will be compiled for speed

Merging ..\..\dist\bin\firefox!1.pgc
61 of 61 (100.00%) profiled functions will be compiled for speed

More PGO Testing

45% Increase in the loading of maps.google.com. I profiled Google Maps by running a few searches.

I used a website called WebWait. It calculates the load time for a webpage. Of course internet lag will play apart but the numbers were pretty close to each other.

The data I collected is here.

Google Maps - 45% faster ( was profiled )
CNN.com - 21% faster ( was profiled )
tomshardware.com - the same ( was not profiled )

I need to look into setting up talos to find out if doing profiling actually speeds up some aspects of Firefox or not.

Next
Create a script to build multiple PGO's during the night. Each PGO will do different profiling. Example. go to just cnn.com or just maps.google.com or just sunspider javascript test.

Profiling Attempt

I tried to optimize Firefox by using the Acid3 Test and Sunspider. These test the DOM and JavaScript respectively. Afterwards I loaded www.cnn.com 5 times, it was saved on the hard driver, and ran the Sunspider Test 5 times.

Profiling Results

Firefox Build	CNN.com (AVG)	SunSpider (AVG)
Minefield (non-pgo)	2206.4	n/a
Minefield (pgo)	2126	3020.6
Firefox 3.0.3	2186.4	3002

***Note***
For each of the five runs I used the same Test User profile I created and cleared the private data with everything checked to ensure that nothing was pre-cached.

I didn't quite get the results I was expecting. I realize that Mozilla is probably doing some sort of profiling that captures the typical usage of an Internet User.

Dave Humphrey mentioned that Mozilla optimizes its browser for the top 100 sites ( cannot remember how many ). If these were used as the profiling data that would make sense why there is almost no difference between my PGO build and the Firefox release.

Next

• Find out how Mozilla optimizes Firefox
• Look into what is getting optimized in my pgo builds
• Understand the pgo process a bit better
  

Mozilla is feeling HOT HOT HOT!

I felt helpless when I loaded the tinderbox page seeing the winnt 5.2 boxes on fire. How was I to get the most recently working copy for the trunk so I can build?

The Problem
The last time I looked at the tree it was green so I proceeded to clone a new copy using Mercurial. Before I had finish getting a clone or right before I started, the tree was on fire. The clone I downloaded failed to build.

The Solution
After talking with Ted and Dave ( humph), on IRC, they helped me get the latest working revision. On the tinderbox page I looked at the boxes that I am running, Windows XP SP2, and I scrolled down the page until I found the green sections. In the green sections there is a rev: followed by a hex number. This green column or section represents a build that past all the tests. The rev: number is the revision in Mercurial.

I then did the following steps:

• Get the latest changes - hg pull -u
• Rolled back to a revision - hg up

And I was able to successfuly build firefox.

Finding Optimized Functions

Using pgomgr for Visual Studio I am able to look at the pgd ( profile guided database ) files that are created during profiling.

Functions
The pgd files hold the following information:

• Number of times a function was entered.
• Time spent in a function.
• Time spend in the sections of a function.
• Number of times a loop was iterated.
• Number of times the loop was broken out of.
• Number of times an if statement was hit.
• Number of times each branch was hit.
  

With the help of contributors I hope to create a profiling scenario that will optimize Firefox for everyday use.

The pgd files can help determine if my profiling scenario is broad enough to hit most of the Firefox functions that get used.

I'll need to research what the average users usage habits are.

Tests
To know if my profiling is having a positive or negative effect I'll need to run performance tests against my builds. I'm looking into the performance tests mozilla uses for their tinderbox builds.

Next
Create a list of performance tests to run.
Create a comparison between a non-pgo build and a pgo-build.

Change of Direction

After talking with Ted and Dave Humphrey on irc I decided on not focusing on the bugs in PGO since PGO is turned off for the code that had the bugs. I'm focusing on increasing the amount of profiling of Firefox to improve its performance. I had notice that the pgo output mentioned that only 3% of all the functions are optimized. I would like to increase that. Of course I also need to determine what percentage of all Firefox functions are used on average.

My first step is to run some automated tests to gather performance data and also test that the pgo didn't break anything.

Mozilla and Tests
I read up on the Mozilla test suite at the Mozilla Automated Testing page and still wasn't very clear on what it had to say. I find I need to re-read the Mozilla documentation a couple of times to get a feel for the flow of information. I also had to visit a lot of links to answers some questions I had about how to use the existing automated tests and create future tests.

I mentioned this on irc and Ted said if I re-wrote the Mozilla Automated Testing so that it would be easier to follow he would review and submit it for me. Maybe once I get this testing thing under my belt I'll take that beast.

If only time wasn't money....
I changed my config script to run the Mochitest suite after my initial build of the pgo. I use the Test Suite to gather performance data for the second build of the pgo, this produces the optimized build. The first run took 249 minutes. I had 35 failed tests.

Failed Tests
33 of the failed tests were in relation to bug 391728. It looks like this bug was fixed on 2008-09-29. I'm pretty sure I've updated my trunk since then.
2 of the failed tests were in relation to test_jquery.html. I've blown away my details but once I compile again and re-run the tests I will post more details.

Build Problems
I am having random build problems and as Ted pointed out on irc it is most likely user error and indeed it was/is. I only had 1GB of ram and noticed very long build times so I bought 4GB.

Well it turned out that my motherboard only supports 2GB so I returned the 4GB and bought 2GB with better timings. I have a good cpu and fast memory so I decided to do some overclocking.

Here is where the user error comes in. I forgot to make sure not to overclock my pci bus when I overclocked my cpu. This resulted in weird corruption errors on windows and build errors with Firefox. I believe I have finally cleaned up my harddrives from the invalid file handles that were created.

Goal
My goal this weekend it to build an optimized Firefox that was profiled using the xpcshell tests and Mochitest.

Links of Interest
Mozilla Automated Testing
Running Automated Tests
Complied-Code Automated Tests
Writing xpcshell-based Unit Tests
Mochitest

Failed to Reproduce Bugs

I commented the lines in the makefiles as suggested by Ted in his comment about my Successful PGO build. I rebuilt and ran firefox but it would not crash. I browsed some sites, added bookmarks, and made changes to the bookmarks but it would not crash.

I checked the makefiles created in the objdir and they show that the lines that I commented were indeed commented.

I have read some bugs in bugzilla and have not found anything on a specific bug or how to reproduce a specific bug on Windows.

More research is needed.

Successful PGO Build

On IRC a few folks pointed out that I should be using Visual Studio 2005 instead of Visual Studio 2008 for the PGO build of Firefox and it worked.

The error that indicated the problem is probably with Visual Studio 2008 is posted here under Firefox Compilation Errors ( Sept 26th, 2008 ).

Next steps are to duplicate the bugs or crashes experienced from other users using PGO builds of Firefox.

Compiling with VS 2005

I was talking on irc in the #developers channel and it was suggested I try building my profile build using Visual Studio 2005 instead of with Visual Studio 2008. I'm also using the Vista SDK which might be causing a problem. Might.

Real Error
The folks on irc asked to look at the errors I received so I used pastebin to show the error and they felt that the actual error was not listed and wanted to see the lines before the error I posted. Well I had used redirection for stdout and stderr to 2 files and it turns out that some of the errors were going to stdout. As I looked through my output.txt file I noticed a huge section that was about linker errors. This is when they suggested I try using Visual Studio 2005 after I mentioned I had it.

The Error

e:\mozilla\trunk\modules\lcms\src\cmscgats.c(875) : fatal error C1001: An internal error has occurred in the compiler.
(compiler file 'f:\dd\vctools\compiler\utc\src\p2\main.c[0x511953E5:0x00000008]', line 182)
To work around this problem, try simplifying or changing the program near the locations listed above.
Please choose the Technical Support command on the Visual C++
Help menu, or open the Technical Support help file for more information

LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage

 Version 9.00.21022.08

 ExceptionCode            = C0000005
 ExceptionFlags           = 00000000
 ExceptionAddress         = 511953E5 (51030000) "e:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\c2.dll"
 NumberParameters         = 00000002
 ExceptionInformation[ 0] = 00000000
 ExceptionInformation[ 1] = 00000008

CONTEXT:
 Eax    = 00000000  Esp    = 0012ED18
 Ebx    = 02D5D215  Ebp    = 0012ED2C
 Ecx    = 02D96601  Esi    = 02D5D280
 Edx    = 02DAAE12  Edi    = 00000305
 Eip    = 511953E5  EFlags = 00010246
 SegCs  = 0000001B  SegDs  = 00000023
 SegSs  = 00000023  SegEs  = 00000023
 SegFs  = 0000003B  SegGs  = 00000000
 Dr0    = 00000000  Dr3    = 00000000
 Dr1    = 00000000  Dr6    = 00000000
 Dr2    = 00000000  Dr7    = 00000000

Mozilla Community
I like how helpful the Mozilla community is and as a student that is not an avid irc user I forget about the culture and community. I had logged out and switched OS's so I could compile with Visual Studio 2005 and when I jumped back on irc it was about 30 minutes later. I had messaged one of the guys to let them know I was now compiling and he/she didn't seem to remember me. He/she was probably busy and I need to learn how to better discern what is relevant and what is not.

And I cannot worry about making such mistakes. Better to fall and learn then to never jump.

Close but no Cigar

As suggested by Ted, in a comment, I built Firefox with disable debug and tests. I was able to open and close Firefox without any assertions during the profiling, since they were not compiled, but it still failed to compile the optimized build.

After talking to a few guys in the #seneca channel on irc://moznet.org I decided to ask on #developers channel for any help. Unfortunately when I asked it was getting late and the channel was pretty quiet. I plan on asking again to get some help since I cannot really go forward without help compiling or debugging the cryptic errors I'm getting.

Mozilla Source - Looking for Find in This Page...

The Search
Chinmay and I decided to look for how "Find in This Page...", on a Mac it is just "Find", works.
Searching mxr.mozilla.org for "Find in This Page..." found only a test xml file, strangely enough. So we looked for "Edit".

We found a DTD file that had a label and a accesskey for Find, the entity was called findcmd. A search on findcmd found a XUL file for the overlay of the Edit Menu. The overlay XUL file had an element for findcmd entity and a command attribute with the value of key_find. The key_find is a key element in the same XUL file. Its command attribute is cmd_find.

A search for cmd_find found a findbar.xml file. The cmd_find is associated with a method called onFindCommand. This method calls startFind method. The startFind method calls the open command which generates the findbar at the bottom of the screen. In the startFind method after the bar has been opened it then populates the text box with any selected text and enables the findbar buttons if any text is in the textbox.

That is how we found the "Find in This Page..." code.

PGO Build Problems

Failed to Build
Last night I worked on creating a profile build of Firefox. From the comment on this bug suggest others were able to successfully to build Firefox using pgo, unfortunately I am having issues.

Initial Build Successful
I was able to complete the initial profile build of Firefox. This build adds extra instructions to Firefox to watch for: the functions that are used the most, which branches of if/else or switches get hit the most and anything else that can be used to optimize the build. The data collected gets stored in files. To gather the data, the client.mk file runs a script after the initial build to run scenario's to gather the data.

Errors and Crashes
The script I used only opened the browser. I tried to go to a website but an assertion was triggered. I chose to retry which brought me into debug mode so I chose to continue and the browser closed. The make file proceeded to compile the optimized build using the profile data but failed with errors. I tried the process again but this time I closed the browser after it opened and an assertion was triggered on closure and the optimized build failed again.

Output and Help
I have posted the build output here. I'm going to look at some newsgroups to find out if my experience is expected since others have created successful pgo builds. I might need to disable more features to be able to create a successful build. More research is needed.

Microsoft's PGO really improved my Program

My project for OSD is to investigate bugs and crashes in Firefox when it has been built using profile guided optimizations.

Working with PGO
I am unfamiliar with pgo's so I decided to use it on one of my old class assignments. First a program is built to include extra code to monitor the activity in the program and spit out a file with possible optimizations. To best optimize the program, the program should be run through multiple scenario's to generate more files with optimizations for the program. Then all the optimizations are to be merged into the program to created an optimized build. Of course there are many options to choose what to optimize and what optimizations should be merged.

The Optimize Program
I choose to test my assignment from my course Data Structures and Algorithymes.

The excerpt below is from my Project Page.

---

The program loads 15,000 random records from a text file into 4 tables each using a different method for storage and searching.

1. Simple Table: Uses an array and a linear search.
2. Chain Table: Uses an array of link lists. Search is done with a Hash Key.
3. Hash Table: Uses an array and searches with linear probing. Also uses a Hash Key.
4. Tree Table: Uses a binary search tree.

After the data is loaded into a table, 30 different tests are run against it. The same 30 tests for each table. Here are the Results.

	Non-PGO	PGO	Improved By
Simple Table:	28 Secs	4 Secs	7x
Chain Table:	0.109 Secs	0.047 Secs	2.3x
Hash Table:	0.141 Secs	0.047 Secs	2.3x
Tree Table:	0.287 Secs	0.078 Secs	3.6x

---

I'm very impressed with the results. I tried to look at the files to see if I could determine what it decided on optimizing but of course it is in some machine language I don't understand. Probably references to memory locations and such.

So I'm still not sure how to tackle my project. I think I'll mess around with the PGO settings and other optimizations. I can also do this from the command line with I might try. I'll need to talk to ted on IRC or someone about how to compile with pgo and debugging it.

Firefox Lab

I built firefox yestarday ( Sept 17th ) and created my FireFox build page today. I didn't understand the directions for making my build page so I called it Firefox build and then realized I should call it cbishop Firefox build and created a new page. Now I can't figure out how to delete Firefox build page and tried to move it to cbishop Firefox build page 2. I'm gonna get kicked off the wiki if I keep this up. HAHA!

The build went pretty smooth for me. I wrote down the steps I took to build FireFox on Windows XP with Visual Studio 2008 compiler. I haven't gone through the steps to verify if they are complete. I'll do that soon.

Class

I have been spending a lot of time just writing up on my progress. I spent two hours this evening doing up my Firefox build page and updating all the necessary pages. I think I'll have to cut back on the detail because I cannot keep this up with all my other classes and projects that I have on the go.

I am a little surprised at the lack of direction in the Open Source class. I did expect a little more hand holding. Today we went over how to build Firefox which was good but I feel there needs to be a little more help with getting started.

What I would have like to have seen was jumping into compiling Firefox in the first week or two with information focused on how to get around Mozilla and find information and documentation then how to reach the community through IRC.

But that's just my two cents.

Moz Dev Days - Not what I expected

Mozilla Developer Day - Tuesday

On Tuesday I was at the second day of the Mozilla Developer Day and it was not what I had expected. After looking at the schedule I had assumed that I would be shown, taught and walked-through how to create addons using XUL in the XUL add-on workshop. Now, I have never been to a conference or to a workshop at a conference. I probably should not have had any expectations but I did. What ended up happening was more of a free-for-all. The people in the group expressed what they were interested in and the Mozilla Guys tried to work with each person to help them progress in these areas. This was not what I had expected but I enjoyed the approach.

What Happened in the Workshop

I was interested in learning about XUL and how to use it to create an application. Mark Finkle explained how to get started on the creating a bare bones application in XUL. More importantly he explained where to find information on how to get started cause that's half the battle.

Getting Started with XULRunner

Meeting with Ted Mielczarek

My project for OSD ( Open Source Development ) course is looking at the bugs found during the PGO ( profile guided optimization ) builds. Ted was fun to talk to and with in five minutes gave me some direction to start with. I will start with building Firefox on my Windows computer and then add the profile guided optimizations and then we'll go from there. After meeting and talking with Ted I feel better about tackling this project.

The Mozilla Guys

I really enjoyed have the Mozilla guys at Seneca. It was great having so many knowledgable and experienced developers in one room. The main thing I noticed was how down to earth the guys are. I don't know the really cool technology they are developing and at no time did I feel as if they were talking over my head. I would definitely go to another conference with these guys.

Added Preview to the Directions Command

I was able to add a preview of the directions map from Google in the preview pane for ubiquity. I wanted to add some more customization to the preview pane to pretty it up but I was having difficulties.

I'm still don't understand the flow of control or changing of scoping in JavaScript. Or I do understand but I forget and get lost inside the whirlwind of event handlers, callbacks and anonymous functions.

The ubiquity channel on irc has been very helpful. I suggest poking your head in there if you need any assistance with ubiquity.

Discovery Channel Commercial

My wife and I just love this Commercial.

http://www.youtube.com/watch?v=V5BxymuiAxQ

Now that is good Branding. I know that it is the Discovery Channel when that song enters my head randomly.

Ubiquity Bug Ticket Submitted

So today I submitted my first ticket for a bug I found in Ubiquity.

I was trying to add an iframe to the preview pane in Ubiquity but it would stall or crash ubiquity. The error doesn't happen right away all the time but eventually it does happen. A restart of Firefox fixes the problem.

Problem Code

CmdUtils.CreateCommand({
name: "testIframe",
takes: {"word": noun_arb_text},
execute: function(directObj) {
},
preview: function( pblock, directObj )
{


var iframe = pblock.ownerDocument.createElement("iframe");
iframe.setAttribute("src","http://www.jslint.com");
pblock.innerHTML = "";
pblock.appendChild(iframe);
}

})

When I discovered this bug I jumped on the ubiquity channel on irc and started asking if this should be submitted as a ticket. Two people tried my code saw the problem and agreed that it should be posted as a ticket.

It was nice to have some one agree that it should be a ticket since I'm new to the process and didn't want to submit a ticket that should be submitted.

Exciting new ground for me.

Ubiquity To the Rescue!!!

I am a computer person and I like short-cuts. I am not a command line freak that just loves typing commands instead I prefer pointing and clicking. I'm lazy. I do the lazy thing. I'll never love VI. Not true, one day our eyes will meet and it will be love but not while my mouse is so fuzzy and warm.

Ubiquity helps combine my love of pointing and clicking with command typing. I enjoy the define function. Double click a word then ctrl+space ( on Windows ) and we have that word's definition.

I liked the map command but I really wanted to be able to put in multiple addresses and get the directions to those addresses. So I created one. It can be found on my Projects Page on the Seneca Wiki.

The command does not give a preview map. I might spend some time trying to ratify that. I would like to add some window dressing to the preview section also. Pretty it up some.

I have a few ideas on how to implement a pasteBin command. In the preview page add a textbox or editable field that the user can paste code into and a button to convert to a pasteBin url. I might give it a whirl.

Back to drowning in the sea of code.......

Project Candidates

After reading the potential projects list on the Seneca wiki I chose 4 candidates for my Open Source Development project.

1. gdb Symbol Server Support

   Adding support for gdb to grab symbols from a server. Mozilla points out in a bug thread that it has no plans on fixing this. It also points out that this would be a great student project.

2. Profile the Build System

   The building of Firefox on Windows takes longer than on Linux or Mac. This project would be profiling the performance of the build to determine performance problems.

3. Mercurial History Browsing

   To create nice graphical interface for the changes made to the software.
   

4. Research PGO Related Bugs

   Mozilla is using profile guided optimizations on Windows but some of the builds have bugs. This project would find and fix some or all of these bugs.

I'm excited about these projects and cannot wait to finally pick one and get working. If anyone in ODS600/DPS909 would like to partner up on one of these projects you can post here.

Open Source

The essay by Eric Raymond "Cathedral and Bazaar" and the movie "Revolution OS" are quite interesting. I didn't realize that people are willing to work hard and passionately for software that they will never see a dime from.

I don't believe that we would be under the rule of Microsoft right now if was not for Richard Stallman creating the GNU license or Linus Torvalds for creating Linux. Some one, some where would of realized the same things and created such a license or Operating System.

I do agree that how software is being made has changed. No longer is it a small team working hard to release poorly documented and tested software but people from all around the world working together to develop, patch, and release well documented and tested software.

I'm excited to see where this community will take us or how it will evolve.

Test

This should not appear on planet seneca.

First post!

Hi! I am Chris Bishop and I am a CPA Student with Seneca College in North York. This is my first personal blog. I currently am blogging on a site for a project I am working on. www.c3dl.org, this project is about creating a library in JavaScript to support the native OpenGL calls in the Firefox browser.

I am excited to be in the Open Source Development class and cannot wait to see what project I will be tackling.