[open-ils-commits] [GIT] Evergreen ILS branch rel_2_6 updated. 7821de27b91453d0e972314f45f0ccf1cc117513
Evergreen Git
git at git.evergreen-ils.org
Fri Apr 10 10:55:17 EDT 2015
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "Evergreen ILS".
The branch, rel_2_6 has been updated
via 7821de27b91453d0e972314f45f0ccf1cc117513 (commit)
via 1b38344c3676ff4d8624f02b2c785173a30866e9 (commit)
via c3149e4bacb18d84585120e09948f99a327770b6 (commit)
via ce8037155c9478685490a257e402f13e1f8e8dec (commit)
via 9d5be6b4490b0cacf83d1c861f98510f8e9bb358 (commit)
via 10a4bf6e5bd2eb9f60018d991dcedb5189443ded (commit)
from 8f792a37bd1249ab99341f82e9fec7691e435019 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 7821de27b91453d0e972314f45f0ccf1cc117513
Author: Ben Shum <bshum at biblio.org>
Date: Fri Apr 10 10:54:39 2015 -0400
Docs: Remove these separate bug fix files now that they are part of core RELEASE NOTES
Signed-off-by: Ben Shum <bshum at biblio.org>
diff --git a/docs/RELEASE_NOTES_NEXT/Administration/set_resource_limits_for_reporter.txt b/docs/RELEASE_NOTES_NEXT/Administration/set_resource_limits_for_reporter.txt
deleted file mode 100644
index 9c04c95..0000000
--- a/docs/RELEASE_NOTES_NEXT/Administration/set_resource_limits_for_reporter.txt
+++ /dev/null
@@ -1,32 +0,0 @@
-Set resource limits for Clark Kent
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Several parameters are now available for the reporter daemon process
-(`clark-kent.pl`) to control resource usage. These can be used to
-reduce the chances that a malformed report can cause indigestion
-on a database or reports server. The new parameters, which can be
-set in `opensrf.xml` or as command-line switches for `clark-kent.pl` are
-
-* `//reporter/setup/statement_timeout` / `--statement-timeout`
-
-Number of minutes to allow a report's underlying SQL query
-to run before it gets cancelled. Default value is
-60 minutes. If a report's query gets cancelled, the
-error_text value will be set to a valid that indicates that
-the allowed time was exceeded.
-
-* `//reporter/setup/max_rows_for_charts` / `--max-rows-for-charts`
-
-Number of rows permitted in the query's output before
-Clark Kent refuses to attempt to draw a graph. Default
-value is 1,000 rows.
-
-* `//reporter/setup/resultset_limit` / `--resultset-limit`
-
-If set, truncates the report's output to the specified
-number of hits. Note that it will not be apparent
-to a staff user if the report's output has been
-truncated. Default value is unlimited.
-
-The report concurrency (i.e., the number of reports that Clark
-Kent will run in parallel) can now also be controlled via
-the `opensrf.xml` setting `//reporter/setup/parallel`.
diff --git a/docs/RELEASE_NOTES_NEXT/security-bug-fix.txt b/docs/RELEASE_NOTES_NEXT/security-bug-fix.txt
deleted file mode 100644
index 03d83cd..0000000
--- a/docs/RELEASE_NOTES_NEXT/security-bug-fix.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-IMPORTANT SECURITY INFORMATION
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-A serious security flaw that allows unauthorized remote access to
-organizational unit settings is fixed in the following releases of
-Evergreen: 2.5.9, 2.6.7, and 2.7.4. All prior releases of Evergreen
-are vulnerable to exploitation of this flaw to reveal sensitive system
-information. If you are running a vulnerable release of Evergreen you
-are *strongly* encouraged to upgrade to a non-vulnerable release as
-soon as possible.
commit 1b38344c3676ff4d8624f02b2c785173a30866e9
Author: Ben Shum <bshum at biblio.org>
Date: Fri Apr 10 10:53:59 2015 -0400
Docs: Update 2.6 RELEASE NOTES to include new section on Bug Fixes
Signed-off-by: Ben Shum <bshum at biblio.org>
diff --git a/docs/RELEASE_NOTES_2_6.txt b/docs/RELEASE_NOTES_2_6.txt
index 23f86d5..af7701d 100644
--- a/docs/RELEASE_NOTES_2_6.txt
+++ b/docs/RELEASE_NOTES_2_6.txt
@@ -490,6 +490,52 @@ revisions target level "AA" of compliance.
For more information on WCAG, see http://www.w3.org/WAI/intro/wcag
+Bug Fixes
+---------
+
+IMPORTANT SECURITY INFORMATION
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A serious security flaw that allows unauthorized remote access to
+organizational unit settings is fixed in the following releases of
+Evergreen: 2.5.9, 2.6.7, and 2.7.4. All prior releases of Evergreen
+are vulnerable to exploitation of this flaw to reveal sensitive system
+information. If you are running a vulnerable release of Evergreen you
+are *strongly* encouraged to upgrade to a non-vulnerable release as
+soon as possible.
+
+Set resource limits for Clark Kent
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Several parameters are now available for the reporter daemon process
+(`clark-kent.pl`) to control resource usage. These can be used to
+reduce the chances that a malformed report can cause indigestion
+on a database or reports server. The new parameters, which can be
+set in `opensrf.xml` or as command-line switches for `clark-kent.pl` are
+
+* `//reporter/setup/statement_timeout` / `--statement-timeout`
+
+Number of minutes to allow a report's underlying SQL query
+to run before it gets cancelled. Default value is
+60 minutes. If a report's query gets cancelled, the
+error_text value will be set to a valid that indicates that
+the allowed time was exceeded.
+
+* `//reporter/setup/max_rows_for_charts` / `--max-rows-for-charts`
+
+Number of rows permitted in the query's output before
+Clark Kent refuses to attempt to draw a graph. Default
+value is 1,000 rows.
+
+* `//reporter/setup/resultset_limit` / `--resultset-limit`
+
+If set, truncates the report's output to the specified
+number of hits. Note that it will not be apparent
+to a staff user if the report's output has been
+truncated. Default value is unlimited.
+
+The report concurrency (i.e., the number of reports that Clark
+Kent will run in parallel) can now also be controlled via
+the `opensrf.xml` setting `//reporter/setup/parallel`.
+
Acknowledgments
---------------
The Evergreen project would like to acknowledge the following
commit c3149e4bacb18d84585120e09948f99a327770b6
Author: Galen Charlton <gmc at esilibrary.com>
Date: Thu Apr 9 20:50:24 2015 +0000
LP#1435494: add release notes entry
Signed-off-by: Galen Charlton <gmc at esilibrary.com>
Signed-off-by: Ben Shum <bshum at biblio.org>
diff --git a/docs/RELEASE_NOTES_NEXT/Administration/set_resource_limits_for_reporter.txt b/docs/RELEASE_NOTES_NEXT/Administration/set_resource_limits_for_reporter.txt
new file mode 100644
index 0000000..9c04c95
--- /dev/null
+++ b/docs/RELEASE_NOTES_NEXT/Administration/set_resource_limits_for_reporter.txt
@@ -0,0 +1,32 @@
+Set resource limits for Clark Kent
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Several parameters are now available for the reporter daemon process
+(`clark-kent.pl`) to control resource usage. These can be used to
+reduce the chances that a malformed report can cause indigestion
+on a database or reports server. The new parameters, which can be
+set in `opensrf.xml` or as command-line switches for `clark-kent.pl` are
+
+* `//reporter/setup/statement_timeout` / `--statement-timeout`
+
+Number of minutes to allow a report's underlying SQL query
+to run before it gets cancelled. Default value is
+60 minutes. If a report's query gets cancelled, the
+error_text value will be set to a valid that indicates that
+the allowed time was exceeded.
+
+* `//reporter/setup/max_rows_for_charts` / `--max-rows-for-charts`
+
+Number of rows permitted in the query's output before
+Clark Kent refuses to attempt to draw a graph. Default
+value is 1,000 rows.
+
+* `//reporter/setup/resultset_limit` / `--resultset-limit`
+
+If set, truncates the report's output to the specified
+number of hits. Note that it will not be apparent
+to a staff user if the report's output has been
+truncated. Default value is unlimited.
+
+The report concurrency (i.e., the number of reports that Clark
+Kent will run in parallel) can now also be controlled via
+the `opensrf.xml` setting `//reporter/setup/parallel`.
commit ce8037155c9478685490a257e402f13e1f8e8dec
Author: Galen Charlton <gmc at esilibrary.com>
Date: Thu Apr 9 20:09:59 2015 +0000
LP#1435494: suggest 1048575 as a default resultset_limit
Per a suggestion by Thomas Berezansky; this magic number
represents the number of rows supported by XSLX, less one
for a header row.
Signed-off-by: Galen Charlton <gmc at esilibrary.com>
Signed-off-by: Ben Shum <bshum at biblio.org>
diff --git a/Open-ILS/examples/opensrf.xml.example b/Open-ILS/examples/opensrf.xml.example
index f28e4ae..d21bba2 100644
--- a/Open-ILS/examples/opensrf.xml.example
+++ b/Open-ILS/examples/opensrf.xml.example
@@ -201,7 +201,7 @@ vim:et:ts=4:sw=4:
A value of 0 means that no limit should be set.
-->
- <resultset_limit>0</resultset_limit>
+ <resultset_limit>1048575</resultset_limit>
</setup>
</reporter>
commit 9d5be6b4490b0cacf83d1c861f98510f8e9bb358
Author: Galen Charlton <gmc at esilibrary.com>
Date: Thu Apr 9 20:06:25 2015 +0000
LP#1435494: do not encourage <resultset_limit></resultset_limit>
An empty node in opensrf.xml gets parsed as an empty hashref,
not an empty scalar, so we'll use <resultset_limit>0</resultlet_limit>
in the suggested opensrf.xml config. This fixes an issue
noticed by Ben Shum during testing where reports could fail with
the following message:
DBD::Pg::st execute failed: ERROR: syntax error at or near "0"
LINE 43: ) limited_to_HASH(0x2a974f8)_hits LIMIT HASH(0x2a974f8)
^ at /openils/bin/clark-kent.pl line 243.
Signed-off-by: Galen Charlton <gmc at esilibrary.com>
Signed-off-by: Ben Shum <bshum at biblio.org>
diff --git a/Open-ILS/examples/opensrf.xml.example b/Open-ILS/examples/opensrf.xml.example
index 17df348..f28e4ae 100644
--- a/Open-ILS/examples/opensrf.xml.example
+++ b/Open-ILS/examples/opensrf.xml.example
@@ -198,8 +198,10 @@ vim:et:ts=4:sw=4:
has been limited in this fashion. This setting can be
overriden by the -resultset-limit command-line switch of
clark-kent.pl.
+
+ A value of 0 means that no limit should be set.
-->
- <resultset_limit></resultset_limit>
+ <resultset_limit>0</resultset_limit>
</setup>
</reporter>
diff --git a/Open-ILS/src/reporter/clark-kent.pl b/Open-ILS/src/reporter/clark-kent.pl
index 90e67b1..6664c6d 100755
--- a/Open-ILS/src/reporter/clark-kent.pl
+++ b/Open-ILS/src/reporter/clark-kent.pl
@@ -109,7 +109,9 @@ my $max_rows_for_charts = $opt_max_rows_for_charts //
1000;
$max_rows_for_charts = 1000 unless $max_rows_for_charts =~ /^\d+$/;
my $resultset_limit = $opt_resultset_limit //
- $sc->config_value( reporter => setup => 'resultset_limit' );
+ $sc->config_value( reporter => setup => 'resultset_limit' ) //
+ 0;
+$resultset_limit = 0 unless $resultset_limit =~ /^\d+$/; # 0 means no limit
my ($dbh,$running,$sth, at reports,$run, $current_time);
commit 10a4bf6e5bd2eb9f60018d991dcedb5189443ded
Author: Galen Charlton <gmc at esilibrary.com>
Date: Fri Mar 20 20:33:39 2015 +0000
LP#1435494: set limits on Clark Kent's resource usage
Clark Kent can sometimes consume more RAM, swap space, or CPU
than is reasonable or productive. For example:
- a badly constructed query with multiple Cartesian joins may
never terminate, potentially tying up a Clark child process,
pegging a CPU on the database server, and/or causing significant
scratch disk usage on the database server keeping a snapshot alive.
- a query that returns a very large number of rows can cause a Clark
child to bloat, and in extreme cases cause a OOM on the server
running Clark.
- a report that asks for a chart of an unreasonably large number of
rows can peg a CPU on the Clark server as GD::Graph attempts to
compute sub-pixel graph elements.
In each of these cases, a requested report may never finish.
This patch adds the ability set set some limits on Clark. These
limits can be set either in opensrf.xml for the settings service
to distribute or via command-line switches to clark-kent.pl:
//reporter/setup/statement_timeout / --statement-timeout
Number of minutes to allow a report's underlying SQL query
to run before it gets cancelled. Default value is
60 minutes. If a report's query gets cancelled, the
error_text value will be set to a valid that indicates that
the allowed time was exceeded.
//reporter/setup/max_rows_for_charts / --max-rows-for-charts
Number of rows permitted in the query's output before
Clark Kent refuses to attempt to draw a graph. Default
value is 1,000 rows.
//reporter/setup/resultset_limit / --resultset-limit
If set, truncates the report's output to the specified
number of hits. Note that it will not be apparent
to a staff user if the report's output has been
truncated. Default value is unlimited.
This patch also adds the ability for the concurrency
to be set via an opensrf.xml setting (//reporter/setup/parallel).
If both a command-line switch and an opensrf.xml setting
are supplied, the value set in the command line takes
precedence.
Signed-off-by: Galen Charlton <gmc at esilibrary.com>
Signed-off-by: Ben Shum <bshum at biblio.org>
diff --git a/Open-ILS/examples/opensrf.xml.example b/Open-ILS/examples/opensrf.xml.example
index cf6d1a5..17df348 100644
--- a/Open-ILS/examples/opensrf.xml.example
+++ b/Open-ILS/examples/opensrf.xml.example
@@ -174,6 +174,32 @@ vim:et:ts=4:sw=4:
<success_template>LOCALSTATEDIR/data/report-success</success_template>
<fail_template>LOCALSTATEDIR/data/report-fail</fail_template>
</files>
+ <!-- Number of reports that can be processed simultaneously. This
+ value can overriden by the -c/-concurrency command-line switch
+ of clark-kent.pl.
+ -->
+ <parallel>1</parallel>
+ <!-- Maximum number of rows in the query results allowed before
+ Clark will refuse to draw a pie, bar, or line chart. This
+ value can be overriden by the -max-rows-for-charts command-line
+ switch of clark-kent.pl.
+ -->
+ <max_rows_for_charts>1000</max_rows_for_charts>
+ <!-- Maximum amount of time (in minutes) that an SQL query initiated
+ by Clark Kent will be allowed to run before it is terminated.
+ This value can be overriden by the -statement-timeout
+ command-line switch of clark-kent.pl.
+ -->
+ <statement_timeout>60</statement_timeout>
+ <!-- Maximum number of results permitted. If set to a numeric value,
+ Clark will limit the number of rows returned by report queries
+ to this value. Note that it will not be apparent to a user
+ running a report from the staff interface that their report
+ has been limited in this fashion. This setting can be
+ overriden by the -resultset-limit command-line switch of
+ clark-kent.pl.
+ -->
+ <resultset_limit></resultset_limit>
</setup>
</reporter>
diff --git a/Open-ILS/src/perlmods/lib/OpenILS/Reporter/SQLBuilder.pm b/Open-ILS/src/perlmods/lib/OpenILS/Reporter/SQLBuilder.pm
index 940972a..41d76ba 100644
--- a/Open-ILS/src/perlmods/lib/OpenILS/Reporter/SQLBuilder.pm
+++ b/Open-ILS/src/perlmods/lib/OpenILS/Reporter/SQLBuilder.pm
@@ -38,6 +38,13 @@ sub relative_time {
return $self->builder->{_relative_time};
}
+sub resultset_limit {
+ my $self = shift;
+ my $limit = shift;
+ $self->builder->{_resultset_limit} = $limit if (defined $limit);
+ return $self->builder->{_resultset_limit};
+}
+
sub resolve_param {
my $self = shift;
my $val = shift;
@@ -237,6 +244,8 @@ sub toSQL {
if ($self->is_subquery) {
$sql = '(';
+ } elsif ($self->resultset_limit) {
+ $sql = 'SELECT * FROM (';
}
$sql .= "SELECT\t" . join(",\n\t", map { $_->toSQL } @{ $self->{_select} }) . "\n" if (@{ $self->{_select} });
@@ -251,6 +260,9 @@ sub toSQL {
if ($self->is_subquery) {
$sql .= ') '. $self->{_alias} . "\n";
+ } elsif ($self->resultset_limit) {
+ $sql .= ') limited_to_' . $self->resultset_limit .
+ '_hits LIMIT ' . $self->resultset_limit . "\n";
}
return $self->{_sql} = $sql;
diff --git a/Open-ILS/src/reporter/clark-kent.pl b/Open-ILS/src/reporter/clark-kent.pl
index 6f71e67..90e67b1 100755
--- a/Open-ILS/src/reporter/clark-kent.pl
+++ b/Open-ILS/src/reporter/clark-kent.pl
@@ -29,12 +29,20 @@ use Email::Send;
use open ':utf8';
-my ($count, $config, $sleep_interval, $lockfile, $daemon) = (1, 'SYSCONFDIR/opensrf_core.xml', 10, '/tmp/reporter-LOCK');
+my ($config, $sleep_interval, $lockfile, $daemon) = ('SYSCONFDIR/opensrf_core.xml', 10, '/tmp/reporter-LOCK');
+
+my $opt_count;
+my $opt_max_rows_for_charts;
+my $opt_statement_timeout;
+my $opt_resultset_limit;
GetOptions(
"daemon" => \$daemon,
"sleep=i" => \$sleep_interval,
- "concurrency=i" => \$count,
+ "concurrency=i" => \$opt_count,
+ "max-rows-for-charts=i" => \$opt_max_rows_for_charts,
+ "resultset-limit=i" => \$opt_resultset_limit,
+ "statement-timeout=i" => \$opt_statement_timeout,
"bootstrap|boostrap=s" => \$config,
"lockfile=s" => \$lockfile,
);
@@ -88,6 +96,21 @@ my $base_uri = $sc->config_value( reporter => setup => 'base_uri' );
my $state_dsn = "dbi:" . $state_db{db_driver} . ":dbname=" . $state_db{db_name} .';host=' . $state_db{db_host} . ';port=' . $state_db{db_port};
my $data_dsn = "dbi:" . $data_db{db_driver} . ":dbname=" . $data_db{db_name} .';host=' . $data_db{db_host} . ';port=' . $data_db{db_port};
+my $count = $opt_count //
+ $sc->config_value( reporter => setup => 'parallel' ) //
+ 1;
+$count = 1 unless $count =~ /^\d+$/ && $count > 0;
+my $statement_timeout = $opt_statement_timeout //
+ $sc->config_value( reporter => setup => 'statement_timeout' ) //
+ 60;
+$statement_timeout = 60 unless $statement_timeout =~ /^\d+$/;
+my $max_rows_for_charts = $opt_max_rows_for_charts //
+ $sc->config_value( reporter => setup => 'max_rows_for_charts' ) //
+ 1000;
+$max_rows_for_charts = 1000 unless $max_rows_for_charts =~ /^\d+$/;
+my $resultset_limit = $opt_resultset_limit //
+ $sc->config_value( reporter => setup => 'resultset_limit' );
+
my ($dbh,$running,$sth, at reports,$run, $current_time);
if ($daemon) {
@@ -167,6 +190,7 @@ while (my $r = $sth->fetchrow_hashref) {
$r->{resultset}->set_pivot_label($report_data->{__pivot_label}) if $report_data->{__pivot_label};
$r->{resultset}->set_pivot_default($report_data->{__pivot_default}) if $report_data->{__pivot_default};
$r->{resultset}->relative_time($r->{run_time});
+ $r->{resultset}->resultset_limit($resultset_limit) if $resultset_limit;
push @reports, $r;
}
@@ -203,6 +227,7 @@ for my $r ( @reports ) {
RaiseError => 1
}
);
+ $data_dbh->do('SET statement_timeout = ?', {}, ($statement_timeout * 60 * 1000));
try {
$state_dbh->do(<<' SQL',{}, $r->{id});
@@ -522,28 +547,40 @@ sub build_html {
# Time for a pie chart
if ($r->{chart_pie}) {
- my $pics = draw_pie($r, $file);
- for my $pic (@$pics) {
- print $index "<img src='report-data.html.$pic->{file}' alt='$pic->{name}'/>$br4";
+ if (scalar(@{$r->{data}}) > $max_rows_for_charts) {
+ print $index "<strong>Report output has too many rows to make a pie chart</strong>$br4";
+ } else {
+ my $pics = draw_pie($r, $file);
+ for my $pic (@$pics) {
+ print $index "<img src='report-data.html.$pic->{file}' alt='$pic->{name}'/>$br4";
+ }
}
}
print $index $br4;
# Time for a bar chart
if ($r->{chart_bar}) {
- my $pics = draw_bars($r, $file);
- for my $pic (@$pics) {
- print $index "<img src='report-data.html.$pic->{file}' alt='$pic->{name}'/>$br4";
+ if (scalar(@{$r->{data}}) > $max_rows_for_charts) {
+ print $index "<strong>Report output has too many rows to make a bar chart</strong>$br4";
+ } else {
+ my $pics = draw_bars($r, $file);
+ for my $pic (@$pics) {
+ print $index "<img src='report-data.html.$pic->{file}' alt='$pic->{name}'/>$br4";
+ }
}
}
print $index $br4;
# Time for a bar chart
if ($r->{chart_line}) {
- my $pics = draw_lines($r, $file);
- for my $pic (@$pics) {
- print $index "<img src='report-data.html.$pic->{file}' alt='$pic->{name}'/>$br4";
- }
+ if (scalar(@{$r->{data}}) > $max_rows_for_charts) {
+ print $index "<strong>Report output has too many rows to make a line chart</strong>$br4";
+ } else {
+ my $pics = draw_lines($r, $file);
+ for my $pic (@$pics) {
+ print $index "<img src='report-data.html.$pic->{file}' alt='$pic->{name}'/>$br4";
+ }
+ }
}
# and that's it!
-----------------------------------------------------------------------
Summary of changes:
Open-ILS/examples/opensrf.xml.example | 28 +++++++++
.../perlmods/lib/OpenILS/Reporter/SQLBuilder.pm | 12 ++++
Open-ILS/src/reporter/clark-kent.pl | 63 ++++++++++++++++----
docs/RELEASE_NOTES_2_6.txt | 46 ++++++++++++++
docs/RELEASE_NOTES_NEXT/security-bug-fix.txt | 9 ---
5 files changed, 137 insertions(+), 21 deletions(-)
delete mode 100644 docs/RELEASE_NOTES_NEXT/security-bug-fix.txt
hooks/post-receive
--
Evergreen ILS
More information about the open-ils-commits
mailing list