[Bps-public-commit] app-wsgetmail branch branch-for-multiple-improvements created. 0.06-9-g3e0bcb9
BPS Git Server
git at git.bestpractical.com
Fri Sep 23 14:00:08 UTC 2022
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "app-wsgetmail".
The branch, branch-for-multiple-improvements has been created
at 3e0bcb901a894161ff8e59d8fdfabb66a9ff1725 (commit)
- Log -----------------------------------------------------------------
commit 3e0bcb901a894161ff8e59d8fdfabb66a9ff1725
Author: Brian Conry <bconry at bestpractical.com>
Date: Wed Sep 21 19:10:53 2022 -0500
Add documentation on error/warnings and diagnostics
THis change adds documentation about error messages, warnings, and
troubleshooting issues with message processing.
diff --git a/Changes b/Changes
index 3084053..60782c6 100644
--- a/Changes
+++ b/Changes
@@ -1,6 +1,8 @@
Revision history for App-wsgetmail
0.07 ??/??/??
+ * Add documentation about error messages, warnings, and
+ diagnosing problems with processing.
* Count and report messages with errors in post-fetch processing
* Upgrade failure to fetch folder details to a fatal error
from a warning and successful exit code.
diff --git a/lib/App/wsgetmail.pm b/lib/App/wsgetmail.pm
index ca37fd1..44b463d 100644
--- a/lib/App/wsgetmail.pm
+++ b/lib/App/wsgetmail.pm
@@ -471,6 +471,20 @@ runs successfully.
Once your configuration is stable, you can configure wsgetmail to run
periodically through cron or a systemd service on a timer.
+=head1 ERRORS AND DIAGNOSTIC MESSAGES
+
+wsgetmail sends warning, error, and debug messages to STDERR, while purely
+informational messages are sent to STDOUT. Operators may want to capture both
+output streams as a merged stream for diagnostic purposes. For example:
+
+ wsgetmail --debug --dry-run --config=wsgetmail.json > wsgetmail.debug 2>&1
+
+When the mail processing command exits with an error (non-zero) status the
+action_on_fetched is not performed on that message so that it will be processed
+on the next run.
+
+Full output of the processing command is produced with C<--debug>.
+
=head1 LIMITATIONS
=head2 Fetching from Multiple Folders
commit 8ef1da91bfb1f5d38766fefafef6b6b64173f475
Author: Brian Conry <bconry at bestpractical.com>
Date: Wed Sep 21 18:39:31 2022 -0500
Count and report on failures in message processing.
Prior to this change wsgetmail reported only on the messages that were
sucessfullly processed. Messages that resulted in errors during
post-fetch processing were not counted or reported on in any way.
With this change a count is kept of messages that resulted in errors
during post-fetch processing and this count is reported if non-zero.
wsgetmail now exits with a failure code if there were any errors
processing messages.
diff --git a/Changes b/Changes
index ad383d5..3084053 100644
--- a/Changes
+++ b/Changes
@@ -1,6 +1,7 @@
Revision history for App-wsgetmail
0.07 ??/??/??
+ * Count and report messages with errors in post-fetch processing
* Upgrade failure to fetch folder details to a fatal error
from a warning and successful exit code.
* Correct error message for issues with deleting messages
diff --git a/bin/wsgetmail b/bin/wsgetmail
index 4d06d07..d0bc4b7 100755
--- a/bin/wsgetmail
+++ b/bin/wsgetmail
@@ -98,15 +98,22 @@ my $getmail = App::wsgetmail->new({config => $config});
print "\nfetching mail using configuration $config_file\n";
-my $count = 0;
+my $done_count = 0;
+my $error_count = 0;
while (my $message = $getmail->get_next_message()) {
my $ok = $getmail->process_message($message);
- $count++ if ($ok);
+ if ($ok) {
+ $done_count++;
+ }
+ else {
+ $error_count++;
+ }
}
-print "\nprocessed $count messages\n";
+print "\nprocessed $done_count messages\n";
+die "there were errors with $error_count messages\n" if $error_count;
__END__
commit 57baf42a22bed75694b3cac3730921d12a58dc2c
Author: Brian Conry <bconry at bestpractical.com>
Date: Wed Sep 21 18:35:17 2022 -0500
Make failure to fetch folders fatal
Prior to this change, a failure to fetch the folder details was treated
as a minor error, even though it completely prevents wsgetmail from
fetching and processing messages.
With this change wsgetmail will die with an error message and exit with
a non-zero return code if there is an issue fetching the folder details.
diff --git a/Changes b/Changes
index acb78cd..ad383d5 100644
--- a/Changes
+++ b/Changes
@@ -1,6 +1,8 @@
Revision history for App-wsgetmail
0.07 ??/??/??
+ * Upgrade failure to fetch folder details to a fatal error
+ from a warning and successful exit code.
* Correct error message for issues with deleting messages
* Add documentation of the dump_messages option
* Add strip_cr option to convert CRLF -> LF
diff --git a/lib/App/wsgetmail/MS365.pm b/lib/App/wsgetmail/MS365.pm
index b188caa..056e1e8 100644
--- a/lib/App/wsgetmail/MS365.pm
+++ b/lib/App/wsgetmail/MS365.pm
@@ -415,8 +415,7 @@ sub _get_message_list {
my $folder = $self->get_folder_details;
unless ($folder) {
- warn "unable to fetch messages, can't find folder " . $self->folder;
- return { '@odata.count' => 0, value => [ ] };
+ die "unable to fetch messages, can't find folder " . $self->folder;
}
# don't request list if folder has no items
commit c2b00125b22300cec6d1980018f6fa29682e1240
Author: Brian Conry <bconry at bestpractical.com>
Date: Wed Sep 21 18:30:35 2022 -0500
Correct error message for issues deleting messages
Before this change the warning emitted for errors when trying to delete
a message incorrectly referred to marking the message as read.
diff --git a/Changes b/Changes
index 9ec068b..acb78cd 100644
--- a/Changes
+++ b/Changes
@@ -1,6 +1,7 @@
Revision history for App-wsgetmail
0.07 ??/??/??
+ * Correct error message for issues with deleting messages
* Add documentation of the dump_messages option
* Add strip_cr option to convert CRLF -> LF
* Add locking based on the username/folder
diff --git a/lib/App/wsgetmail/MS365.pm b/lib/App/wsgetmail/MS365.pm
index d4d58ab..b188caa 100644
--- a/lib/App/wsgetmail/MS365.pm
+++ b/lib/App/wsgetmail/MS365.pm
@@ -325,7 +325,7 @@ sub delete_message {
my @path_parts = ($self->global_access) ? ('users', $self->username, 'messages', $message_id) : ('me', 'messages', $message_id);
my $response = $self->_client->delete_request([@path_parts]);
unless ($response->is_success) {
- warn "failed to mark message as read " . $response->status_line;
+ warn "failed to delete message " . $response->status_line;
warn "response from server : " . $response->content if $self->debug;
}
commit 79139e24975414f00e4951c2e68a1f8def9ce1b7
Author: Brian Conry <bconry at bestpractical.com>
Date: Mon Sep 19 18:08:22 2022 -0500
Document the dump_messages option
This option was previously undocumented.
diff --git a/Changes b/Changes
index 86a39f1..9ec068b 100644
--- a/Changes
+++ b/Changes
@@ -1,6 +1,7 @@
Revision history for App-wsgetmail
0.07 ??/??/??
+ * Add documentation of the dump_messages option
* Add strip_cr option to convert CRLF -> LF
* Add locking based on the username/folder
* Remove '+ 5' adjustment to timeout in MDA.pm
diff --git a/lib/App/wsgetmail.pm b/lib/App/wsgetmail.pm
index 0b51488..ca37fd1 100644
--- a/lib/App/wsgetmail.pm
+++ b/lib/App/wsgetmail.pm
@@ -449,6 +449,12 @@ If you set this to C<"mark_as_read">, wsgetmail will only retrieve and
deliver messages that are marked unread in the configured folder, so it does
not try to deliver the same email multiple times.
+=item dump_messages
+
+Set this to 1 to preserve the temporary files after processing.
+
+When C<"debug"> is also set the filenames will be reported on STDERR.
+
=back
=head1 TESTING AND DEPLOYMENT
commit 32374a4c67b98b71d73f6ed3419657a0ed6a17e1
Author: Brian Conry <bconry at bestpractical.com>
Date: Mon Sep 19 09:05:06 2022 -0500
Add config option strip_cr
This option directs wsgetmail to convert CRLF line endings in the
fetched message to LF line endings. The default value is to preserve
CRLFs in fetched messages.
This behavior is tenchnically non-standard according to the MIME and
SMTP standards, but is the default behavior for at least one other mail
fetching utility and some *nix programs commonly used for mail
processing don't properly handle messages with CRLFs.
diff --git a/Changes b/Changes
index c509610..86a39f1 100644
--- a/Changes
+++ b/Changes
@@ -1,6 +1,7 @@
Revision history for App-wsgetmail
0.07 ??/??/??
+ * Add strip_cr option to convert CRLF -> LF
* Add locking based on the username/folder
* Remove '+ 5' adjustment to timeout in MDA.pm
* Remove interpreter substitution for wsgetmail
diff --git a/lib/App/wsgetmail.pm b/lib/App/wsgetmail.pm
index 8f8ca34..0b51488 100644
--- a/lib/App/wsgetmail.pm
+++ b/lib/App/wsgetmail.pm
@@ -73,6 +73,7 @@ where C<wsgetmail.json> looks like:
"global_access": 1,
"username": "rt-comment at example.com",
"folder": "Inbox",
+ "strip_cr": 0,
"command": "/opt/rt5/bin/rt-mailgate",
"command_args": "--url=http://rt.example.com/ --queue=General --action=comment",
"command_timeout": 30,
@@ -411,6 +412,14 @@ configuration file.
Set this to the name string of a mail folder to read.
+=item strip_cr
+
+Set this to 1 to make wsgetmail convert the messages from the CRLF
+line-ending encoding to the LF line-ending encoding.
+
+This is technically not standards-compliant, but some unix utilities
+don't work with CRLF line-endings.
+
=item command
Set this to an executable command. You can specify an absolute path,
diff --git a/lib/App/wsgetmail/MS365.pm b/lib/App/wsgetmail/MS365.pm
index c2cb7c2..d4d58ab 100644
--- a/lib/App/wsgetmail/MS365.pm
+++ b/lib/App/wsgetmail/MS365.pm
@@ -186,6 +186,18 @@ has post_fetch_action => (
required => 1
);
+=head2 strip_cr
+
+A boolean. If true, the message content will have CRLF line terminators
+converted to LF line terminators.
+
+=cut
+
+has strip_cr => (
+ is => 'ro',
+ required => 0,
+);
+
=head2 debug
A boolean. If true, the object will issue a warning with details about each
@@ -238,7 +250,7 @@ around BUILDARGS => sub {
grep {
defined $config->{$_}
}
- qw(client_id tenant_id username user_password global_access secret folder post_fetch_action debug)
+ qw(client_id tenant_id username user_password global_access secret folder post_fetch_action strip_cr debug)
};
return $class->$orig($attributes);
@@ -296,7 +308,9 @@ sub get_message_mime_content {
# can we just write straight to file from response?
my $tmp = File::Temp->new( UNLINK => 0, SUFFIX => '.mime' );
- print $tmp $response->content;
+ my $content = $response->content;
+ $content =~ s/\r$//mg if $self->strip_cr;
+ print $tmp $content;
return $tmp->filename;
}
commit cfa4ec14c38de28970c2c3f3a26fac40d762adf9
Author: Brian Conry <bconry at bestpractical.com>
Date: Fri Sep 16 16:27:47 2022 -0500
Add file locking to prevent contention
This change adds file locking based on the username and folder from the
merged configuration (config file + comamnd-line overrides).
This prevents multiple isntances running simultaneously on the same
username/folder while still allowing multiple instances to run, even
when they use the same base configuration.
The Microsoft Graph API is based on REST, which is stateless so there's
nothing at the remote endpoint that will prevent multiple instances of
an application from working in parallel.
In order to reduce the number of remote data fetches, wsgetmail executes
a search and pulls down a worklist of messages before it starts
processing the first message. This decision means that multiple copies
of wsgetmail working on the same username/folder can have overlapping
sets of messages, leading to messages being processed multiple times.
diff --git a/Changes b/Changes
index 7195d1a..c509610 100644
--- a/Changes
+++ b/Changes
@@ -1,6 +1,7 @@
Revision history for App-wsgetmail
0.07 ??/??/??
+ * Add locking based on the username/folder
* Remove '+ 5' adjustment to timeout in MDA.pm
* Remove interpreter substitution for wsgetmail
diff --git a/bin/wsgetmail b/bin/wsgetmail
index 4631c32..4d06d07 100755
--- a/bin/wsgetmail
+++ b/bin/wsgetmail
@@ -49,6 +49,7 @@
use v5.10;
use strict;
+use Fcntl ':flock';
use FindBin;
use lib "$FindBin::Bin/../lib";
use JSON;
@@ -80,6 +81,18 @@ foreach my $option ( keys %$extra_options ) {
$config->{dry_run} = $dry_run if (defined $dry_run);
$config->{debug} = $debug if (defined $debug);
+my $foldername = $config->{folder};
+
+$foldername =~ s{/}{_}g;
+
+my $lock_file_name = '/tmp/' . join( '.', 'wsgetmail', $config->{username}, $foldername, 'lock' );
+
+open my $lock_file_fh, '>', $lock_file_name or die "unable to open lock file $lock_file_name ($!)";
+
+if ( !flock $lock_file_fh, LOCK_EX | LOCK_NB ) {
+ print "$0 is already running for $config->{username}/$config->{folder} ($!)\n";
+ exit;
+}
my $getmail = App::wsgetmail->new({config => $config});
commit 3c804d44a6f9996918f962d8cf829abf68ef191b
Author: Brian Conry <bconry at bestpractical.com>
Date: Wed Sep 14 12:39:51 2022 -0500
Remove 5 second fudge from configured timeout
Prior to commit b56eff4 when no timeout was configured for the timeout
the value would be left undef instead of the intended default. The '+
5' both coerced the undef to a defined value and provided a long-enough
timeout for most cases, though not the full 30 seconds intended to be
the default.
Since that commit the intended default has been used and the extra time
no longer needs to be added.
This change also documents that 'inf' should be used for no timeout, as
with IPC::Run a value of '0' is treated as an immediate timeout.
diff --git a/Changes b/Changes
index 904a4f6..7195d1a 100644
--- a/Changes
+++ b/Changes
@@ -1,6 +1,7 @@
Revision history for App-wsgetmail
0.07 ??/??/??
+ * Remove '+ 5' adjustment to timeout in MDA.pm
* Remove interpreter substitution for wsgetmail
0.06 22/04/22
diff --git a/README.md b/README.md
index c7ef00d..507e03a 100644
--- a/README.md
+++ b/README.md
@@ -59,7 +59,7 @@ A hash ref that is passed to construct the `mda` and `client` (see below).
## mda
-An instance of [App::wsgetmail::MDA](https://metacpan.org/pod/App::wsgetmail::MDA) created from our `config` object.
+An instance of [App::wsgetmail::MDA](https://metacpan.org/pod/App%3A%3Awsgetmail%3A%3AMDA) created from our `config` object.
## client\_class
@@ -219,7 +219,7 @@ configuration file.
- command\_timeout
Set this to the number of seconds the `command` has to return before
- timeout is reached. The default value is 30.
+ timeout is reached. The default value is 30. Use 'inf' for no timeout.
- action\_on\_fetched
@@ -278,9 +278,9 @@ of time.
# SEE ALSO
- [wsgetmail](https://metacpan.org/pod/wsgetmail)
-- [App::wsgetmail::MDA](https://metacpan.org/pod/App::wsgetmail::MDA)
-- [App::wsgetmail::MS365](https://metacpan.org/pod/App::wsgetmail::MS365)
-- [App::wsgetmail::MS365::Message](https://metacpan.org/pod/App::wsgetmail::MS365::Message)
+- [App::wsgetmail::MDA](https://metacpan.org/pod/App%3A%3Awsgetmail%3A%3AMDA)
+- [App::wsgetmail::MS365](https://metacpan.org/pod/App%3A%3Awsgetmail%3A%3AMS365)
+- [App::wsgetmail::MS365::Message](https://metacpan.org/pod/App%3A%3Awsgetmail%3A%3AMS365%3A%3AMessage)
# AUTHOR
diff --git a/lib/App/wsgetmail.pm b/lib/App/wsgetmail.pm
index 4bb0004..8f8ca34 100644
--- a/lib/App/wsgetmail.pm
+++ b/lib/App/wsgetmail.pm
@@ -428,7 +428,7 @@ double quotes.
=item command_timeout
Set this to the number of seconds the C<command> has to return before
-timeout is reached. The default value is 30.
+timeout is reached. The default value is 30. Use "inf" for no timeout.
=item action_on_fetched
diff --git a/lib/App/wsgetmail/MDA.pm b/lib/App/wsgetmail/MDA.pm
index eb18e47..0f46960 100644
--- a/lib/App/wsgetmail/MDA.pm
+++ b/lib/App/wsgetmail/MDA.pm
@@ -199,7 +199,7 @@ sub _run_command {
return 1;
}
- my $ok = run ([ $self->command, _split_command_args($self->command_args, 1)], $fh, \$output, \$error, timeout( $self->command_timeout + 5 ) );
+ my $ok = run ([ $self->command, _split_command_args($self->command_args, 1)], $fh, \$output, \$error, timeout( $self->command_timeout ) );
unless ($ok) {
warn sprintf('failed to run command "%s %s" for file %s : %s',
$self->command,
commit ef1166688563c07c0f7e9a15121aa56c0132ca3e
Author: Brian Conry <bconry at bestpractical.com>
Date: Wed Sep 14 10:48:53 2022 -0500
Remove shebang line substitution from bin/wsgetmail
This avoids certain possible issues when doing 'make dist' and
eliminates the possibility of unintentionally commiting a change to the
shebang line to the author's current perl path.
diff --git a/Changes b/Changes
index 6b2e719..904a4f6 100644
--- a/Changes
+++ b/Changes
@@ -1,5 +1,8 @@
Revision history for App-wsgetmail
+0.07 ??/??/??
+ * Remove interpreter substitution for wsgetmail
+
0.06 22/04/22
* Update 0.05 Changes to conform to spec
* Sync module pod with recent changes in script
diff --git a/Makefile.PL b/Makefile.PL
index 9e03110..224bccb 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -31,13 +31,6 @@ if ($^O ne 'VMS' and $secure_perl_path !~ /$Config{_exe}$/i) {
$secure_perl_path .= $Config{_exe};
}
-substitute(
- {
- PERL => $ENV{PERL} || $secure_perl_path,
- },
- 'bin/wsgetmail',
-);
-
install_script('bin/wsgetmail');
auto_install();
sign;
diff --git a/bin/wsgetmail b/bin/wsgetmail
index 2218417..4631c32 100755
--- a/bin/wsgetmail
+++ b/bin/wsgetmail
@@ -1,6 +1,4 @@
#!/usr/bin/env perl
-### before: #!@PERL@
-#
# BEGIN BPS TAGGED BLOCK {{{
#
# COPYRIGHT:
diff --git a/inc/Module/Install/Substitute.pm b/inc/Module/Install/Substitute.pm
deleted file mode 100644
index 56af7fe..0000000
--- a/inc/Module/Install/Substitute.pm
+++ /dev/null
@@ -1,131 +0,0 @@
-#line 1
-package Module::Install::Substitute;
-
-use strict;
-use warnings;
-use 5.008; # I don't care much about earlier versions
-
-use Module::Install::Base;
-our @ISA = qw(Module::Install::Base);
-
-our $VERSION = '0.03';
-
-require File::Temp;
-require File::Spec;
-require Cwd;
-
-#line 89
-
-sub substitute
-{
- my $self = shift;
- $self->{__subst} = shift;
- $self->{__option} = {};
- if( UNIVERSAL::isa( $_[0], 'HASH' ) ) {
- my $opts = shift;
- while( my ($k,$v) = each( %$opts ) ) {
- $self->{__option}->{ lc( $k ) } = $v || '';
- }
- }
- $self->_parse_options;
-
- my @file = @_;
- foreach my $f (@file) {
- $self->_rewrite_file( $f );
- }
-
- return;
-}
-
-sub _parse_options
-{
- my $self = shift;
- my $cwd = Cwd::getcwd();
- foreach my $t ( qw(from to) ) {
- $self->{__option}->{$t} = $cwd unless $self->{__option}->{$t};
- my $d = $self->{__option}->{$t};
- die "Couldn't read directory '$d'" unless -d $d && -r _;
- }
-}
-
-sub _rewrite_file
-{
- my ($self, $file) = @_;
- my $source = File::Spec->catfile( $self->{__option}{from}, $file );
- $source .= $self->{__option}{sufix} if $self->{__option}{sufix};
- unless( -f $source && -r _ ) {
- print STDERR "Couldn't find file '$source'\n";
- return;
- }
- my $dest = File::Spec->catfile( $self->{__option}{to}, $file );
- return $self->__rewrite_file( $source, $dest );
-}
-
-sub __rewrite_file
-{
- my ($self, $source, $dest) = @_;
-
- my $mode = (stat($source))[2];
-
- open my $sfh, "<$source" or die "Couldn't open '$source' for read";
- print "Open input '$source' file for substitution\n";
-
- my ($tmpfh, $tmpfname) = File::Temp::tempfile('mi-subst-XXXX', UNLINK => 1);
- $self->__process_streams( $sfh, $tmpfh, ($source eq $dest)? 1: 0 );
- close $sfh;
-
- seek $tmpfh, 0, 0 or die "Couldn't seek in tmp file";
-
- open my $dfh, ">$dest" or die "Couldn't open '$dest' for write";
- print "Open output '$dest' file for substitution\n";
-
- while( <$tmpfh> ) {
- print $dfh $_;
- }
- close $dfh;
- chmod $mode, $dest or "Couldn't change mode on '$dest'";
-}
-
-sub __process_streams
-{
- my ($self, $in, $out, $replace) = @_;
-
- my @queue = ();
- my $subst = $self->{'__subst'};
- my $re_subst = join('|', map {"\Q$_"} keys %{ $subst } );
-
- while( my $str = <$in> ) {
- if( $str =~ /^###\s*(before|replace|after)\:\s?(.*)$/s ) {
- my ($action, $nstr) = ($1,$2);
- $nstr =~ s/\@($re_subst)\@/$subst->{$1}/ge;
-
- die "Replace action is bad idea for situations when dest is equal to source"
- if $replace && $action eq 'replace';
- if( $action eq 'before' ) {
- die "no line before 'before' action" unless @queue;
- # overwrite prev line;
- pop @queue;
- push @queue, $nstr;
- push @queue, $str;
- } elsif( $action eq 'replace' ) {
- push @queue, $nstr;
- } elsif( $action eq 'after' ) {
- push @queue, $str;
- push @queue, $nstr;
- # skip one line;
- <$in>;
- }
- } else {
- push @queue, $str;
- }
- while( @queue > 3 ) {
- print $out shift(@queue);
- }
- }
- while( scalar @queue ) {
- print $out shift(@queue);
- }
-}
-
-1;
-
-----------------------------------------------------------------------
hooks/post-receive
--
app-wsgetmail
More information about the Bps-public-commit
mailing list