For the rest of my PGO project I'll be focusing on adding PGO to Thunderbird. So my first task is to build Thunderbird. Since I'll be building and testing a lot I have scripts to automate this task for me. I just tweaked my scripts to now build for Thunderbird.
Bash Script Problems
I don't know why but when I tried to create a bash script from scratch it would behave as if it executed, as in no bash could find command error, but nothing would happen. I could get any output. So I copied an existing working script and modified it. Works for me.
Next Steps
Now that I can build Thunderbird I am going to try and replicate the Mozilla tinderbox setup for Thunderbird on my home machine so I can duplicate the performance testing environment. Once I have a baseline of how Thunderbird performs on my machine I can then start enabling PGO and see what kind of performance boost I get.
After that I would have lots of opportunities for other Seneca students to contribute by compiling and running the tests on their machines to see if they are also seeing a performance gain.
So now to get to it.....
Tuesday, October 28, 2008
Friday, October 17, 2008
0.1 Release - Firefox PGO
0.1 Release is done.
I have learned:
0.1 Release more detailed look
Experience
I have found myself frustrated, upset, surprised and excited as I work my way into the Mozilla Community. I am asking questions more openly, more willing to look dumb in the hopes of getting the information I need. I always search first, of course.
It really takes time to get a feel for how Mozilla does things. I just recently discovered that the trunk has all these readme's scattered everywhere. Probably the first place to look for help.
I still find the Mozilla wiki's and documentation to be lacking for someone who is not as experienced as the folks at Mozilla. I tell my wife that "it's documentation written for people who already know it". I find I have to research and look stuff up to understand what I read on Mozilla.
Now that I have my feet wet I am excited to push forward.
I have learned:
How to implement PGO on any application.
How to turn PGO on and off for modules in Firefox.
That profiling firefox anymore than opening and closing the browser doesn't yeild better performance.
How to run xpcshell, mochitest, and reftests on a Firefox build.
How to create bash scripts to build firefox and create reports from the build output.
How to reproduce the sqlite bug as report in this bug.
How to attach my results of test to a bug.
That sqlite crashes even in a stand alone application when compiled with PGO.
That I am going to turn my focus to Thunderbird and turn on PGO for Windows/Linux/Mac.
0.1 Release more detailed look
Experience
I have found myself frustrated, upset, surprised and excited as I work my way into the Mozilla Community. I am asking questions more openly, more willing to look dumb in the hopes of getting the information I need. I always search first, of course.
It really takes time to get a feel for how Mozilla does things. I just recently discovered that the trunk has all these readme's scattered everywhere. Probably the first place to look for help.
I still find the Mozilla wiki's and documentation to be lacking for someone who is not as experienced as the folks at Mozilla. I tell my wife that "it's documentation written for people who already know it". I find I have to research and look stuff up to understand what I read on Mozilla.
Now that I have my feet wet I am excited to push forward.
Saturday, October 11, 2008
Review of PGO Data
45% increase not a result of PGO profiling of Google Maps.
I used build Firefox with only profiling of a page opening and then closing to check it against the loading times for Google Maps.
I re-ran all the tests and here is the summary of the results:
Firefox 3.0.3
Minefield PGO *
Minefield PGO **
2.15s (avg)
1.18s (avg)
1.17s (avg)
* Profiled using the Google Maps website.
** Profiled with only the browser opening then closing.
I also saved the Google Maps page on to my local server and when the pages were served from a local server the load times were all identical.
Serving Google Maps locally resulted in non of the javascript files Google Map uses being saved since they are loaded dynamically.
I have to conclude from this second performance test that minefield has some performance improvements not seen in Firefox 3.0.3 since the profiling did not improve Google Maps site loading.
I used build Firefox with only profiling of a page opening and then closing to check it against the loading times for Google Maps.
I re-ran all the tests and here is the summary of the results:
Firefox 3.0.3
Minefield PGO *
Minefield PGO **
2.15s (avg)
1.18s (avg)
1.17s (avg)
* Profiled using the Google Maps website.
** Profiled with only the browser opening then closing.
I also saved the Google Maps page on to my local server and when the pages were served from a local server the load times were all identical.
Serving Google Maps locally resulted in non of the javascript files Google Map uses being saved since they are loaded dynamically.
I have to conclude from this second performance test that minefield has some performance improvements not seen in Firefox 3.0.3 since the profiling did not improve Google Maps site loading.
Friday, October 10, 2008
Firefox PGO Details
According to the build output:
- 2938 objects are created
- 2246 of the 2938 objects are created with pgo optimizations
- 2101 of the 2246 objects are optimized for size
- 145 of the 2246 objects are optimized for speed
- 73 objects are optimized with profile data
- 69 of 73 objects didn't have any profile data to optimize with
- 49 of 73 objects are tests objects
Merging ..\..\dist\bin\js3250!1.pgc
882 of 2904 ( 30.37%) profiled functions will be compiled for speed
Merging ..\..\dist\bin\xul!1.pgc
3433 of 84021 ( 4.09%) profiled functions will be compiled for speed
Merging ..\..\dist\bin\xpcom!1.pgc
55 of 55 (100.00%) profiled functions will be compiled for speed
Merging ..\..\dist\bin\firefox!1.pgc
61 of 61 (100.00%) profiled functions will be compiled for speed
More PGO Testing
45% Increase in the loading of maps.google.com. I profiled Google Maps by running a few searches.
I used a website called WebWait. It calculates the load time for a webpage. Of course internet lag will play apart but the numbers were pretty close to each other.
The data I collected is here.
Google Maps - 45% faster ( was profiled )
CNN.com - 21% faster ( was profiled )
tomshardware.com - the same ( was not profiled )
I need to look into setting up talos to find out if doing profiling actually speeds up some aspects of Firefox or not.
Next
Create a script to build multiple PGO's during the night. Each PGO will do different profiling. Example. go to just cnn.com or just maps.google.com or just sunspider javascript test.
I used a website called WebWait. It calculates the load time for a webpage. Of course internet lag will play apart but the numbers were pretty close to each other.
The data I collected is here.
Google Maps - 45% faster ( was profiled )
CNN.com - 21% faster ( was profiled )
tomshardware.com - the same ( was not profiled )
I need to look into setting up talos to find out if doing profiling actually speeds up some aspects of Firefox or not.
Next
Create a script to build multiple PGO's during the night. Each PGO will do different profiling. Example. go to just cnn.com or just maps.google.com or just sunspider javascript test.
Thursday, October 9, 2008
Profiling Attempt
I tried to optimize Firefox by using the Acid3 Test and Sunspider. These test the DOM and JavaScript respectively. Afterwards I loaded www.cnn.com 5 times, it was saved on the hard driver, and ran the Sunspider Test 5 times.
Profiling Results
***Note***
For each of the five runs I used the same Test User profile I created and cleared the private data with everything checked to ensure that nothing was pre-cached.
I didn't quite get the results I was expecting. I realize that Mozilla is probably doing some sort of profiling that captures the typical usage of an Internet User.
Dave Humphrey mentioned that Mozilla optimizes its browser for the top 100 sites ( cannot remember how many ). If these were used as the profiling data that would make sense why there is almost no difference between my PGO build and the Firefox release.
Next
Profiling Results
| Firefox Build | CNN.com (AVG) | SunSpider (AVG) |
|---|---|---|
| Minefield (non-pgo) | 2206.4 | n/a |
| Minefield (pgo) | 2126 | 3020.6 |
| Firefox 3.0.3 | 2186.4 | 3002 |
For each of the five runs I used the same Test User profile I created and cleared the private data with everything checked to ensure that nothing was pre-cached.
I didn't quite get the results I was expecting. I realize that Mozilla is probably doing some sort of profiling that captures the typical usage of an Internet User.
Dave Humphrey mentioned that Mozilla optimizes its browser for the top 100 sites ( cannot remember how many ). If these were used as the profiling data that would make sense why there is almost no difference between my PGO build and the Firefox release.
Next
- Find out how Mozilla optimizes Firefox
- Look into what is getting optimized in my pgo builds
- Understand the pgo process a bit better
Wednesday, October 8, 2008
Mozilla is feeling HOT HOT HOT!
I felt helpless when I loaded the tinderbox page seeing the winnt 5.2 boxes on fire. How was I to get the most recently working copy for the trunk so I can build?
The Problem
The last time I looked at the tree it was green so I proceeded to clone a new copy using Mercurial. Before I had finish getting a clone or right before I started, the tree was on fire. The clone I downloaded failed to build.
The Solution
After talking with Ted and Dave ( humph), on IRC, they helped me get the latest working revision. On the tinderbox page I looked at the boxes that I am running, Windows XP SP2, and I scrolled down the page until I found the green sections. In the green sections there is a rev: followed by a hex number. This green column or section represents a build that past all the tests. The rev: number is the revision in Mercurial.
I then did the following steps:
The Problem
The last time I looked at the tree it was green so I proceeded to clone a new copy using Mercurial. Before I had finish getting a clone or right before I started, the tree was on fire. The clone I downloaded failed to build.
The Solution
After talking with Ted and Dave ( humph), on IRC, they helped me get the latest working revision. On the tinderbox page I looked at the boxes that I am running, Windows XP SP2, and I scrolled down the page until I found the green sections. In the green sections there is a rev: followed by a hex number. This green column or section represents a build that past all the tests. The rev: number is the revision in Mercurial.
I then did the following steps:
- Get the latest changes - hg pull -u
- Rolled back to a revision - hg up
Tuesday, October 7, 2008
Finding Optimized Functions
Using pgomgr for Visual Studio I am able to look at the pgd ( profile guided database ) files that are created during profiling.
Functions
The pgd files hold the following information:
The pgd files can help determine if my profiling scenario is broad enough to hit most of the Firefox functions that get used.
I'll need to research what the average users usage habits are.
Tests
To know if my profiling is having a positive or negative effect I'll need to run performance tests against my builds. I'm looking into the performance tests mozilla uses for their tinderbox builds.
Next
Create a list of performance tests to run.
Create a comparison between a non-pgo build and a pgo-build.
Functions
The pgd files hold the following information:
- Number of times a function was entered.
- Time spent in a function.
- Time spend in the sections of a function.
- Number of times a loop was iterated.
- Number of times the loop was broken out of.
- Number of times an if statement was hit.
- Number of times each branch was hit.
The pgd files can help determine if my profiling scenario is broad enough to hit most of the Firefox functions that get used.
I'll need to research what the average users usage habits are.
Tests
To know if my profiling is having a positive or negative effect I'll need to run performance tests against my builds. I'm looking into the performance tests mozilla uses for their tinderbox builds.
Next
Create a list of performance tests to run.
Create a comparison between a non-pgo build and a pgo-build.
Friday, October 3, 2008
Change of Direction
After talking with Ted and Dave Humphrey on irc I decided on not focusing on the bugs in PGO since PGO is turned off for the code that had the bugs. I'm focusing on increasing the amount of profiling of Firefox to improve its performance. I had notice that the pgo output mentioned that only 3% of all the functions are optimized. I would like to increase that. Of course I also need to determine what percentage of all Firefox functions are used on average.
My first step is to run some automated tests to gather performance data and also test that the pgo didn't break anything.
Mozilla and Tests
I read up on the Mozilla test suite at the Mozilla Automated Testing page and still wasn't very clear on what it had to say. I find I need to re-read the Mozilla documentation a couple of times to get a feel for the flow of information. I also had to visit a lot of links to answers some questions I had about how to use the existing automated tests and create future tests.
I mentioned this on irc and Ted said if I re-wrote the Mozilla Automated Testing so that it would be easier to follow he would review and submit it for me. Maybe once I get this testing thing under my belt I'll take that beast.
If only time wasn't money....
I changed my config script to run the Mochitest suite after my initial build of the pgo. I use the Test Suite to gather performance data for the second build of the pgo, this produces the optimized build. The first run took 249 minutes. I had 35 failed tests.
Failed Tests
33 of the failed tests were in relation to bug 391728. It looks like this bug was fixed on 2008-09-29. I'm pretty sure I've updated my trunk since then.
2 of the failed tests were in relation to test_jquery.html. I've blown away my details but once I compile again and re-run the tests I will post more details.
Build Problems
I am having random build problems and as Ted pointed out on irc it is most likely user error and indeed it was/is. I only had 1GB of ram and noticed very long build times so I bought 4GB.
Well it turned out that my motherboard only supports 2GB so I returned the 4GB and bought 2GB with better timings. I have a good cpu and fast memory so I decided to do some overclocking.
Here is where the user error comes in. I forgot to make sure not to overclock my pci bus when I overclocked my cpu. This resulted in weird corruption errors on windows and build errors with Firefox. I believe I have finally cleaned up my harddrives from the invalid file handles that were created.
Goal
My goal this weekend it to build an optimized Firefox that was profiled using the xpcshell tests and Mochitest.
Links of Interest
Mozilla Automated Testing
Running Automated Tests
Complied-Code Automated Tests
Writing xpcshell-based Unit Tests
Mochitest
My first step is to run some automated tests to gather performance data and also test that the pgo didn't break anything.
Mozilla and Tests
I read up on the Mozilla test suite at the Mozilla Automated Testing page and still wasn't very clear on what it had to say. I find I need to re-read the Mozilla documentation a couple of times to get a feel for the flow of information. I also had to visit a lot of links to answers some questions I had about how to use the existing automated tests and create future tests.
I mentioned this on irc and Ted said if I re-wrote the Mozilla Automated Testing so that it would be easier to follow he would review and submit it for me. Maybe once I get this testing thing under my belt I'll take that beast.
If only time wasn't money....
I changed my config script to run the Mochitest suite after my initial build of the pgo. I use the Test Suite to gather performance data for the second build of the pgo, this produces the optimized build. The first run took 249 minutes. I had 35 failed tests.
Failed Tests
33 of the failed tests were in relation to bug 391728. It looks like this bug was fixed on 2008-09-29. I'm pretty sure I've updated my trunk since then.
2 of the failed tests were in relation to test_jquery.html. I've blown away my details but once I compile again and re-run the tests I will post more details.
Build Problems
I am having random build problems and as Ted pointed out on irc it is most likely user error and indeed it was/is. I only had 1GB of ram and noticed very long build times so I bought 4GB.
Well it turned out that my motherboard only supports 2GB so I returned the 4GB and bought 2GB with better timings. I have a good cpu and fast memory so I decided to do some overclocking.
Here is where the user error comes in. I forgot to make sure not to overclock my pci bus when I overclocked my cpu. This resulted in weird corruption errors on windows and build errors with Firefox. I believe I have finally cleaned up my harddrives from the invalid file handles that were created.
Goal
My goal this weekend it to build an optimized Firefox that was profiled using the xpcshell tests and Mochitest.
Links of Interest
Mozilla Automated Testing
Running Automated Tests
Complied-Code Automated Tests
Writing xpcshell-based Unit Tests
Mochitest
Subscribe to:
Posts (Atom)