Unfortunately from seeing some pictures from developer events I noticed that many of the developers seem to be on Macs. This seems to hurt Linux and Windows users to a degree because it affects the speed at which platform specific issues are fixed. There exists all sorts of tweaks using about
:config but the major problem is that doing a normal Google search will yield you mostly unusable data with a mess of techniques for various versions.
I gave up on it myself, for now. Somehow I have it to the point where it is usable and doesn't hog all of my memory beyond 25% of total system memory. However I still need to close it out, have it save tabs, and then reopen it about once a day or the interface can become extremely sluggish.
You might try using the development version of 3.7 and 4.0 (You will likely need to disable version checking for many of your extensions to work)
http://nightly.mozilla.org/ <---- Nightlies (latest development code each day)
Btw, to be fair I should point out that 64 bit will by nature use more than 32 bit so when doing comparisons with one another, keep this in mind.