Here’s another bunch of images made last week in Vienna. The bicycle with the poisonous red is from Tuesday morning, I just couldn’t pass it, the others are from Thursday. It is Saturday night now, it’s raining outside, I am uploading 340 images, six sizes each, to my website and have done so for nine hours. ADSL is a bitch 🙂
Uploading again? Why that? Oh, in the morning, after I had processed these four images, I found some images missing from some posts, sometimes thumbnails, sometimes the main image. With the current layout this seems to be a problem in certain cases. I don’t use “width” attributes on my images (which I should, I know) and on certain posts the browser simply hung indefinitely, seemingly refusing to carry on without an actual image and its width. That’s obviously pretty bad 🙂
I made a check: from an export file of my blog I extracted the image “SRC” attributes pointing to “manessinger.com”. The result was a file with 1887 URLs, one URL per line.
#!/bin/sh
XMLFILE=$1
if [ "x$XMLFILE" = x ] ; then
echo "usage: $0 <xmlfile>"
exit 1
fi
cat $XMLFILE | sed -e 's/ /\n/g' | \
egrep 'src="https://d25zfm9zpd7gm5.cloudfront.net/' | cut -d'"' -f2
I’m printing the code here without explanation, those for whom it may be useful, will understand it anyway.
The next step was to check if all of these URLs are present and which are not. The result was a file with the file names (tail of the URL) of the files that were missing. Again here’s the code, a Perl script expecting an input file with one URL per line, producing an output file with one filename per missing image.
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request;
use File::Basename;
use Time::HiRes qw(usleep);
# arguments checking
if (($#ARGV != 0) || (! -f $ARGV[0])) {
print "usage: $0 <url-per-line-file>\n";
exit(1);
}
# construct a user agent
my $ua = LWP::UserAgent->new();
$ua->agent("");
$ua->timeout(10);
open(URLS, "<".$ARGV[0]);
while (<URLS>😉 {
chomp;
my $url = $_;
my $new_url = grab_image($ua, $url);
}
exit(0);
sub grab_image {
my $ua = shift;
my $url = shift;
my $request = HTTP::Request->new(HEAD=>$url);
my $response = $ua->request($request);
my $base = basename($url);
if ($response->is_success) {
print STDERR ("$base\n");
} else {
print "ERROR\@$url\n\t".$response->status_line."\n\n";
print STDERR ("$base MISSING\n");
open(MISSING, ">>missing.txt");
print MISSING "$base\n";
close(MISSING);
}
usleep(100);
}
Yes, I could have given the file reference as a parameter 🙂 Anyway. This is slightly more convenient than checking 1887 URLs manually, isn’t it? The result were a whopping 340 URLs of missing images. Oops!
Then it dawned on me: I had simply uploaded all images from my image database that were tagged “smugmug”. I have mostly given up tagging images. It’s convenient to know exactly where you’ve taken a certain images, find all images with a sundown taken in Italy, find all images showing a certain person, etc, but when I tag images that thoroughly, it takes me 15 minutes per day. Well, maybe I should do it again, mostly ignoring the backlog.
Anyway. While I had believed that the tag “smugmug” was the one that I had always given, I must have been much more lazy than I had thought. Indeed, some checks in IMatch, my image database, revealed that the missing images were not tagged and thus had not been uploaded.
Now what? Manually finding 340 images, tagging them and copying them to my conversion directory? Possible but tedious. Thankfully IMatch can be programmed. I had to learn it, but learning it and writing a program was still faster than doing it manually.
Sub Main
' We need an active database to run this script
Dim db As Database
Set db = Application.ActiveDatabase
If db Is Nothing Then
MsgBox "Please open a database first"
Exit Sub
ElseIf db.ReadOnly Then
MsgBox "The database is read-only!"
Exit Sub
End If
' Check if the result category exists
Dim rescat As Category
Set rescat = db.Categories("UNTAGGED")
If rescat Is Nothing Then
MsgBox "Please create a category 'UNTAGGED'"
Exit Sub
End If
If Not rescat.Images.Count = 0 Then
If MsgBox("Clear the category?",vbYesNo) = vbYes Then
' Avoid flicker when updating the category!
db.Redraw = False
Dim rci As Image
For Each rci In rescat.Images
rescat.RemoveImage rci
Next rci
db.Redraw = True
End If
End If
Dim names(339) As String
names(0) = "20061017_161745.jpg"
names(1) = "20061020_173954.jpg"
' ... and so on until
names(338) = "20090919_173043_ps.jpg"
names(339) = "20090925_154035.jpg"
Dim i As Long
For i = LBound(names) To UBound(names)
' Find the file in the database
Set img = Nothing
Set iset = db.GetImages(names(i))
If Not iset Is Nothing Then
If iset.Count > 0 Then
Set img = iset(1)
End If
End If
If Not img Is Nothing Then
rescat.AddImage(img)
End If
Next
End Sub
Believe it or not, this was my first Basic program. Hmm … well, not really. Now that I think of it, 12 years ago I have written a code generator that allowed Visual Basic programmers on PCs to call C and Cobol subroutines on UNIX and VMS servers via RPC. The tool had generated client- and server-side stubs, thus I already have written Basic, at least indirectly via a program 🙂
Now I had all missing images tagged “UNTAGGED” and could export them to my conversion directory, ready for the resize-and-upload script to process them. And that’s what this script still does. It’s not the resizing, it’s the uploading. On the other hand, I don’t care. It runs in the background and I’m free to do whatever I want.
In the afternoon, after I had started the uploader, I drove to the lake. Weather was fine, and really, the water was still … well … tolerable. It’s quite cool now, but it was in no way uncomfortable.
Here we are. I will post more code examples during the next week. If you don’t care about code, just ignore it. One or the other fragment may be valuable for someone though.
The Song of the Day is “Missing You” from the 1995 Mavericks album “Music for All Occasions”. When I first heard the Mavericks, I was not sure if this is a joke or if they really mean it. It sounded so fifties retro-style, so much like “Kottan’s Kapelle” (a satirical Autrian TV series about Vienna’s police), that I was not sure what to make of it. I liked it though and have bought one more Mavericks CD. See a video on YouTube.