[Evergreen-dev] Profiling Evergreen with More recent PostgreSQL Releases

[Jason Stephenson]

Hello, all.

As some of you are aware, I have been running Evergreen against various 
PostgreSQL releases on an old email server that has 128 GB of RAM and 
8TB of disk space.  I currently have PostgreSQL versions 9.6 through 13 
installed.

During my testing, I have identified a few areas where Evergreen 
database performance is slower on newer Pg versions and other areas 
where it is faster.

I am presently setting up to use the plprofiler extension 
(https://github.com/bigsql/plprofiler) to test these areas, beginning 
with those where I know that performance degrades in PostgreSQL version 
greater than 10.

To make the test fair, I am loading a dump of CW MARS' production 
database from Sunday, July 11, 2021 into the various Pg instances. 
Since we are currently on Evergreen 3.5.3, I intend to also run a 
database upgrade script on each instance.

I have a question regarding the database upgrade: Should I upgrade to 
Evergreen 3.7.1 or to an arbitrary commit on the master branch, i.e. the 
one that is current as of the time that I generate the db upgrade?

I also intend to use optimization settings on the database instance 
being tested. I presently have 2 of the instances configured to use 
roughly 40% of the server's capabilities and the other instances are 
left at the default settings. I may continue with the 40% optimization 
or go with settings that will utilize more of the hardware for just the 
instance being tested and either shut off or run the other instances 
with default settings.

If anyone has any additional ideas of things that should be tested or 
would like to help devise a script to more thoroughly test the Evergreen 
database, let me know.

Cheers,
Jason