<div dir="ltr"><div>Hi Brian,</div><div><br></div><div>Some possibilities I can see include:</div><div><br></div><div>- although you're running Pg 10, possibly some settings related to WAL file generation and archiving are still pointing at subdirectories of 9.6.bak</div><div>- similar to above, maybe directories related to WAL are configured to be under 10/, but are actually symlinks to subdirectories under 9.6.bak</div><div>- the upgrade was done using pg_upgradecluster --link --method=upgrade, which would create hard links between files in the original 9.6 cluster and the new 10 one. In that case, removing the links from the old cluster will likely be safe (after of course doing a full backup)<br></div><div><br></div><div>To check whether the third case would be a possibility, compare the output of "du -hs" as applied to each data directory as compared to "du -lhs". If "du -lhs" shows that each data directory is larger as compared to "du -hs", that would suggest that there are hard links present.</div><div><br></div><div>It would be helpful to know which files under 9.6.bak are open according to lsof. If you do an ls -l on them, look at the number following the permissions:<br><br><font size="4"><span style="font-family:monospace">-rw-r--r-- <b><span style="color:rgb(255,0,0)">2</span></b> gmc gmc 2097152 Jul 10 17:33 /tmp/link-test/a/foo</span></font></div><div><br></div><div>That is the number of hard links to the file, so if it is larger than 1, that would strongly suggest that the "pg_upgardecluster --link --method=upgrade" scenario is what's at play.<br></div><div><br></div><div>Also, what is the output of pg_lsclusters? If it is still showing an inactive cluster named "9.6.bak", pg_dropcluster can be used to remove it.<br></div><div><br></div><div>Regards,</div><div><br></div><div>Galen<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 10, 2024 at 3:02 PM Brian Holda via Evergreen-dev <<a href="mailto:evergreen-dev@list.evergreen-ils.org">evergreen-dev@list.evergreen-ils.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-5918171866412117151">
<div dir="ltr">
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
On my Evergreen Database server, in /var/lib/postgresql/</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
I have the following 4 directories: </div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
1) 9.6.bak/ = 111GB, edits happening every minute </div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
2) 10/ = 58GB, last edited a couple hours ago </div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
3) 9.6/ = 45MB, last edited 1 year ago </div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
4) backups/ = 2.2GB, last edited 2AM today </div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
The stumper to me is the 9.6.bak. If we're using psql 10, and already have a backup directory updated daily, why are things actively writing to 9.6.bak/?</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
We need to free up some storage, and I was tempted to get rid of 9.6.bak, but don't want to without figuring out what is writing to it (and why).</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
If helpful, I find no instance of 9.6.bak in </div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
ps -ef | grep postgres</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
But it did appear in </div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
lsof +D /var/lib/postgresql/9.6.bak</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
Any clues?!</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
Brian</div>
<div style="font-family:"Calibri Light","Helvetica Light",sans-serif;font-size:16pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_-5918171866412117151Signature" style="color:inherit">
<p style="line-height:18pt"><span style="font-family:Georgia,serif;font-size:12pt;color:rgb(51,51,51)">Brian Holda<br>
Library Technology Manager<br>
Hekman Library<br>
Calvin University<br>
(616) 526-8673</span></p>
<p style="line-height:18pt"><span style="font-family:Georgia,serif;font-size:12pt;color:rgb(51,51,51)"><a href="https://library.calvin.edu/" id="m_-5918171866412117151OWAa2ca0983-0030-892a-95d0-b5401811df36" style="margin-top:0px;margin-bottom:0px" target="_blank"><img width="245" height="34" style="width: 245px; height: 34px; margin-top: 0px; margin-bottom: 0px;" src="cid:ii_1909e8fa8cbc3e509791"></a></span></p>
<p style="line-height:18pt"><span style="font-family:Georgia,serif;font-size:12pt;color:rgb(51,51,51)"> </span></p>
</div>
</div>
_______________________________________________<br>
Evergreen-dev mailing list<br>
<a href="mailto:Evergreen-dev@list.evergreen-ils.org" target="_blank">Evergreen-dev@list.evergreen-ils.org</a><br>
<a href="http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev" rel="noreferrer" target="_blank">http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev</a><br>
</div></blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr">Galen Charlton<br>Implementation and IT Manager<br>Equinox Open Library Initiative<br><a href="mailto:gmc@equinoxOLI.org" target="_blank">gmc@equinoxOLI.org</a><br><a href="https://www.equinoxOLI.org" target="_blank">https://www.equinoxOLI.org</a> <br>phone: 877-OPEN-ILS (673-6457)<br>direct: 770-709-5581<br><a href="http://evergreen-ils.org" target="_blank"></a></div></div>