Is that a DLL in your pocket…

Shock! Horror! Bug found where Windows applications will open DLLs that are in the current working directory of a process!

Except it’s not a bug. It’s by design, and it’s existed since NT.

Microsoft is being smacked in the head by a required feature of Windows due to the initial weakness of the LoadLibrary call. If you don’t specify a path to the file to load, it uses the standard library search path.

Dear god, you would think that this was news. It is not news, nor has it been since the goddamned operating system shipped. Granted, the issue is severe, but the fact of the matter is if an application is executed using a working directory that isn’t under your control, then what can you do? if there are libraries in the same directory that launched the program that happen to share the name of system libraries then you’re hosed.

Hey, guess what asshole, if you link a linux binary with a search path containing ‘.’, then you get the same problem. It’s just as well that nobody links their binaries with -R. …. eh?

The documentation is blatant in this regard. I’ve known it was a security issue since I first learned of the LoadLibrary call, as any even half decent developer should have known when they started using the damned function.

The rule is simple. Resolve the full path to a library before you load it. Validate that it ‘looks right’ at that point. Then load it.

BTW .init section in .so files – so totally a security hole. You can’t dlopen a file to determine if it’s good without executing the .init code. Game over man, game f**king over!

My .init code does a setenv(“LD_LIBRARY_PATH”, “.” + getenv(“LD_LIBRARY_PATH”)) … now piss off and write secure code for once…

Password recovery from open applications

Well I had a minor hiccup today when I decided it was ‘password change day’. I duly went around changing the password on all my systems. Then I got back to work. 10 minutes later I turned to my other system and typed in the password.
… It didn’t work …
I smacked my head and said to myself “D’oh”, I need to use the new password. But I couldn’t remember all of it. All I had was a few characters I could remember and the fact that my mail program was checking the mail every few minutes and still working.

First I got the pid of thunderbird…

~% ps -fe | grep thunder
1000     17509     1  0 13:19 ?        00:00:00 /bin/sh /usr/bin/thunderbird
1000     17521 17509  0 13:19 ?        00:00:00 /bin/sh /usr/lib/thunderbird/run-mozilla.sh /usr/lib/thunderbird/thunderbird-bin
1000     17526 17521  0 13:19 ?        00:00:24 /usr/lib/thunderbird/thunderbird-bin
1000     19101 19006  0 14:09 pts/10   00:00:00 grep thunder

Then I got the address of the heap from the process’ maps

~% grep 'heap' /proc/17526/maps
08d02000-0a9ad000 rw-p 08d02000 00:00 0          [heap]

I compiled up memory_dumper, and ran it against the process and heap addresses listed.

% ./memory_dumper 08d02000 0a46a000 17526 heap

Then I ran strings on the resulting file, looking for the pattern that matched my remembered password

% strings heap | grep t%7
cheat%7Ladel
cheat%7Ladel
cheat%7Ladel
cheat%7Ladel
%

4 copies of the password in memory in the program. That is just in-freaking-sane. It should be present in the program only once, and should probably be concealed using some form of obfuscation. Mind you, it has kept the new password in my mind now, so I should be grateful.

And just in case you feel like trying the password listed, don’t. It’s not the real password ๐Ÿ˜‰

Can you people please collate FFS

Nothing more than a rant….
It’s not that tough – when using most english locales, we sort case insensitively. a==A, B==B and so on. Pragmatically, the only reason for picking a locale other than UTF-8.generic is because I would really, really like these rules obeyed.
I am sick to death of having to work around stupidity.
I’m just complaining as I look at the output from ls and it’s pretty much a case sensitive sort. I’m sure that accents are sorted correctly in EN_ie – after all รก is the same as a, but apparently it’s different to A.
Sorting it difficult… the rules are so complicated… stop complaining! you’re able to perform at least 600 million operations per second, and a table lookup for a case insensitive sort is probably going to cost 20.
Bear in mind that the number above was a quick back of the envelope number of an iPhone. I’m sure a real computer will be able to do something a little better…

Update:

Looks like it’s not Linux, it’s only Leopard that doesn’t understand EN_ie collation. Oh well, that’s life I supposeโ€ฆ

URL shortening…

Not a lot of code. We create a rewrite rule for apache to remap any 5 character requests at root to this script.
RewriteRule ^/([A-Za-z0-9][A-Za-z0-9][A-Za-z0-9][A-Za-z0-9][A-Za-z0-9])$ /cgi-bin/shorten.cgi?$1 [PT].
Requests for http://site/….. lookup the entry in the database, requests to shorten.cgi?URL return the shortened uri in a text/plain output when it works.
There isn’t a lot of checking, and you probably need to create the db/ directory with mode 777 so you can update the database under cgi, but it… works on my box ๐Ÿ˜‰

#!/usr/bin/perl -w
# shorten or unshorten a url passed in
use strict;
use DBI;
use Sys::Hostname;

my %keys;
my $value=0;
# 26 + 26 + 10 = 62
my $keys = join("", A..Z) . join("", a..z) . join("", 0..9);
my @keys = split(//, $keys);
for my $i (@keys) {
    $keys{$i} = $value++;
}

my $file = "shorten.db";
my $dir;
my $var;

if (defined($ENV{SCRIPT_FILENAME})) {
    $var = $ENV{SCRIPT_FILENAME};
} else {
    $var = $0;
}

($dir) = $var =~ m/(.*)\/[^\/]+/;

$file = $dir . "/db/" . $file;

if (! -d $dir . "/db") {
    mkdir($dir . "/db", 0777);
    chmod(0777, $dir . "/db");
}

my $dbh = DBI->connect("dbi:SQLite:dbname=$file", "", "") || die "Could not open $file";
chmod(0666, $file);

$dbh->do("create table if not exists mapping (id INTEGER PRIMARY KEY, url TEXT)");
$dbh->do("create index if not exists mappurl on mapping(url)");

exit(0) if (!defined($ENV{QUERY_STRING}));
my $qs = $ENV{QUERY_STRING};

if (length($qs) == 5) { # from short -> long
    my $key = 0;
    map { $key = $key * 62 + $keys{$_} } split(//, $qs);
    my $ary = $dbh->selectall_arrayref("select url from mapping where id = $key");
    if ($ary) {
        my @ary=@$ary;
        print "Location: " . $ary[0][0] . "\n\n";
    }
} else {
    my $sth = $dbh->prepare("select id from mapping where url = ?");
    my $ret = $sth->execute($qs);
    die "Failed to execute " . $sth->errstr unless($ret);
    my @row = $sth->fetchrow_array();
    my $value;
    if (!@row) {
        $sth = $dbh->prepare("insert or replace into mapping (url) values (?)");
        $sth->execute($qs) or die "Failed to insert" . $sth->errstr;
        $value = $dbh->last_insert_id("","","","");
    } else {
        $value = $row[0];
    }
    if (defined($value) && ($value > 0)) {
        my $op = "";
        while(length($op) != 5) {
            $op = $keys[$value % 62] . $op;
            $value /= 62;
        }
        my $base;
        if (!defined($ENV{HTTP_HOST})) {
            $base = hostname();
        } else {
            $base = $ENV{HTTP_HOST};
        }
        print "Content-Type: text/plain\n\nhttp://" . $base . "/" . $op . "\n";
    } else {
        print "Content-Type: text/plain\n\nFailed to shorten $qs.";
        exit(0);
    }
}

# vim: ts=4:sw=4:et

bash pip-isms or right hand side of pipe variables

Unlike my default shell (zsh), bash has a wonderful feature where it doesn’t keep variables that are set at the other end of a pipe, so for example:
i=
cat foo | while read bar; do
    i=$bar
done
echo $i

Yields an empty line. I’ve been stung once or twice on this as I prototype the code initially in an interactive shell, which doesn’t exhibit the issue.
The simplest solution is to use a named pipe.

i=
mkfifo /tmp/foo$$
cat foo >/tmp/foo$$&
pid=$!
while read bar; do
    i=$bar
done </tmp/foo$$


This gives the last line of the file in the i variable.

Lies my computer told me… (threads != processes)

So there I am looking at the sysinfo from a particular machine and I look at the content of the procs field. It looked to be a bit out. Went hunting through the kernel source and noticed that the procs field is filled with the value of the number of threads in the system. This is a little bit odd, as I’m used to separating my threads from my processes.
Turns out that there is an nr_processes() call, which returns the number of processes in the system, rather than the number of threads. A little bit of a change, rebuild and testing now gives me the correct number of processes from the proc field, and I also have a separate result for the number of threads.
There we go, much more sensible ๐Ÿ˜‰
diff -Naur linux-2.6.25.18/include/linux/kernel.h linux-2.6.25.18.new/include/linux/kernel.h
— linux-2.6.25.18/include/linux/kernel.h      2008-10-09 03:58:32.000000000 +0100
+++ linux-2.6.25.18.new/include/linux/kernel.h  2009-03-16 16:23:39.000000000 +0000
@@ -415,7 +415,8 @@
        unsigned long totalhigh;        /* Total high memory size */
        unsigned long freehigh;         /* Available high memory size */
        unsigned int mem_unit;          /* Memory unit size in bytes */
–       char _f[20-2*sizeof(long)-sizeof(int)]; /* Padding: libc5 uses this.. */
+       unsigned int threads;           /* Number of current threads */
+       char _f[20-2*sizeof(long)-2*sizeof(int)];       /* Padding: libc5 uses this.. */
 };
 
 /* Force a compilation error if condition is true */
diff -Naur linux-2.6.25.18/kernel/compat.c linux-2.6.25.18.new/kernel/compat.c
— linux-2.6.25.18/kernel/compat.c     2008-10-09 03:58:32.000000000 +0100
+++ linux-2.6.25.18.new/kernel/compat.c 2009-03-16 16:43:31.000000000 +0000
@@ -1031,7 +1031,8 @@
        u32 totalhigh;
        u32 freehigh;
        u32 mem_unit;
–       char _f[20-2*sizeof(u32)-sizeof(int)];
+       u32 threads;
+       char _f[20-2*sizeof(u32)-2*sizeof(int)];
 };
 
 asmlinkage long
@@ -1076,7 +1077,8 @@
            __put_user (s.procs, &info->procs) ||
            __put_user (s.totalhigh, &info->totalhigh) ||
            __put_user (s.freehigh, &info->freehigh) ||
–           __put_user (s.mem_unit, &info->mem_unit))
+           __put_user (s.mem_unit, &info->mem_unit) ||
+           __put_user (s.threads, &info->threads))
                return -EFAULT;
 
        return 0;
diff -Naur linux-2.6.25.18/kernel/timer.c linux-2.6.25.18.new/kernel/timer.c
— linux-2.6.25.18/kernel/timer.c      2008-10-09 03:58:32.000000000 +0100
+++ linux-2.6.25.18.new/kernel/timer.c  2009-03-16 16:20:02.000000000 +0000
@@ -37,6 +37,7 @@
 #include <linux/delay.h>
 #include <linux/tick.h>
 #include <linux/kallsyms.h>
+#include <linux/sched.h>
 
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
@@ -1166,7 +1167,8 @@
                info->loads[1] = avenrun[1] << (SI_LOAD_SHIFT – FSHIFT);
                info->loads[2] = avenrun[2] << (SI_LOAD_SHIFT – FSHIFT);
 
–               info->procs = nr_threads;
+               info->procs = nr_processes();
+               info->threads = nr_threads;
        } while (read_seqretry(&xtime_lock, seq));
 
        si_meminfo(info);

The Full patch.

Consistency checking a block device

I’ve been testing the resizing of the drives located on a Dell MD3000, and i’ve seen errors when resizing after the 2TB mark. This is on the new firmware which supports > 2TB logical drives. I wrote a script to write to random locations of a block device. It can then read them back and verify that they’re still the same as what was written. Rather than writing to the entire device I use random sampling, with a few fixed points on the block device. I pretty much get consistent failures. If I put in the failed locations into the next write run they come out again in the subsequent run. Kind of makes resizing a dangerous operation, even though it is stated that resizing is non-destructive.

I realize that the array is nothing more than a rebrand of another device, but it would be great if it was tested in a lab before something this bad got out to the customers.

#! /usr/bin/perl -w

use strict;
use Getopt::Long;
use Digest::MD5 qw(md5_hex);
use File::Basename;

my $fs;
my $readfile;
my $writefile;

my $numpatterns = 2048;
my $seed = undef;
my $size;
my $real_size;
my $help;

my %vars;
my @def_offsets = (0);

sub usage($) {
        print <<EOM;
Usage: $0 –fs=<filesystem> –read=<file>|–write=<file>
        [–num=<number of blocks>] [–offset=<offset to test>]
        [–seed=<random number seed>]
EOM
        exit ($_[0]);
}

my $result = GetOptions( fs=s => \$fs,
        num=i => \$numpatterns,
        seed=i => \$seed,
        read=s => \$readfile,
        offset=i => \@def_offsets,
        write=s => \$writefile,
        h|help => \$help);

usage(0) if defined($help);
warn "Need file system to use" if (!defined($fs));
warn "Need either a read or write file" if (!(defined($readfile) || defined($writefile)));

usage (1) if (!defined($fs) || !(defined($readfile) || defined($writefile)));
my $base = basename($fs);

open (IN, "</proc/partitions") || die "Could not load partition tables";
while (<IN>) {
        chomp();
        my ($major, $minor, $blocks, $name) = m/(\w*)\s+(\w*)\s+(\w*)\s+(\w*)$/;
        next if (!defined($major));
        if ($name eq $base) {
                $real_size = $blocks;
                last;
        }
}
close(IN);

die "Could not get size" if (!defined($real_size));

# Write to the offset in blocks
sub write_to_offset($$) {
        my ($offset, $buffer) = @_;
        sysseek(INFS, $offset * 1024, 0);
        my $write = syswrite(INFS, $buffer, 1024);
        if (!defined($write) || $write != 1024) {
                warn "Failed to write: $offset $!\n";
        } else {
                $vars{$offset} = md5_hex($buffer);
        }
}

sub read_from_offset($) {
        my ($offset) = @_;
        my $buffer;
        sysseek(INFS, $offset * 1024, 0);
        my $read = sysread(INFS, $buffer, 1024);
        if (!defined($read) || $read != 1024) {
                warn "Could not read 1024 bytes at $offset $!";
                return (1);
        }
        if (md5_hex($buffer) ne $vars{$offset}) {
                warn "Data at offset $offset was not the same as expected";
                return (1);
        }
        return (0);
}

sub get_buffer {
        my $i = 0;
        my $buffer = "";
        while ($i++ < 256) {
                my $randval = int(rand(255 * 255 * 255 * 255));
                $buffer .= chr($randval >> 24) . chr(($randval >> 16) & 255) .
                        chr(($randval >> 8) & 255) . chr($randval & 255);
        }
        (length($buffer) == 1024) || die "Buffer was " . length($buffer);
        return $buffer;
}

if (defined($readfile)) {
        # reading from previous file
        open (INPUT, "<$readfile") || die "Could not open previous run log";
        while(<INPUT>) {
                chomp();
                my ($key, $value) = m/(.*)=(.*)/;
                if ($key eq "patterncount") {
                        $numpatterns = $value;
                        next;
                }
                if ($key eq "size") {
                        $size = $value;
                        next;
                }
                if ($key eq "seed") {
                        $seed = $value;
                        next;
                }
                $vars{$key} = $value;
        }
        close(INPUT);
} else {
        $seed = time ^ $$ ^ unpack "%L*", `ls -l /proc/ | gzip -f` if (!defined($seed));
        $size = $real_size if (!defined($size));
        open (OUTPUT, ">$writefile") || die "Could not open new run log";
        print OUTPUT "patterncount=$numpatterns\n" .
                "size=$size\n" .
                "seed=$seed\n";
}

print "Size: $real_size [$size] Seed: $seed\n";
srand($seed);

my $mode = "<";
$mode = "+<" if ($writefile);
open(INFS, "$mode$fs") || die "Could not open raw device";

if ($writefile) {
        map { write_to_offset($_, get_buffer()) } @def_offsets;
        write_to_offset($size – 1, get_buffer());
        while($numpatterns > 0) {
                my $offset = int(rand($size));
                print "Writing pattern: $numpatterns           \r";
                next if defined($vars{$offset});
                write_to_offset($offset, get_buffer());
                $numpatterns–;
        }
        map { print OUTPUT "$_=" . $vars{$_} . "\n" } keys(%vars);
        close(OUTPUT);
} else {
        my $failcount = 0;
        my $tocount = scalar(keys(%vars));
        map { $failcount += read_from_offset($_); printf("To Count: %0.7d\r", $tocount–); } sort(keys(%vars));
        print "Count difference: $failcount\n";
}


consistency.pl.txt

signal versus sigaction

the use of the

signal(int signum, void (*handler)(int))

is a smidgin dangerous on various operating systems. Under Solaris, for example once the signal has been delivered to the process the signal handler is reset, so a typical piece of code that wants to reuse the signal handler repeatedly will typically set the signal handler again when receiving the signal. This leads to a minor race condition where upon receipt of the signal and the re-setting of the handler the process receives another copy of the same signal. Some of these signals cause Bad things to happen – such as the stopping of the process (SIGTSTP for example). Under Linux it keeps the signal handler in place, so you have no fear of the event triggering an unwanted event.
The manual page for

signal

under Linux makes it clear that the call is deprecated in favour of the much more functional

sigaction(int sig, const struct sigaction *restrict act, struct sigaction *restrict oact)

call, which keeps signal handlers in place when you don’t pass the SA_RESETHAND parameter as part of the sa_flags parameter of the sigaction structure. So you get to explicitly choose to accept a signal once, and then have the system deal with it in the default manner afterwards.
Signals, are of course a real pain in the ass when dealing with sub-processes. For example the use of ptrace to perform profiling works well until you fork. If another SIGPROF signal arrives before you can create your signal handler then the child process is terminated as that’s the default behaviour in that situation.
Under Solaris (and Leopard) you can make use of dtrace to perform profiling on a set of processes without needing to deal with vagaries of signal handling, making this a non-issue. For those of you stuck in LD_PRELOAD land, probably the only thing that can be done is to set the signal disposition to be ignored before execing the new process. you have a small window where the profiling is missing, but the overall increased stability of the application is improved by preventing it from accidentally being terminated due to a profiling signal being received too soon. I know the accuracy nuts would hate that, but it’s part of the price of dealing with standards.

Cheap and cheerful pwait for linux

#!/bin/bash -p
if [ $# -eq 0 ]; then
echo "Usage: $(basename $0) " 1>&2
exit 1
fi
while [ -d /proc/$1 ]; do sleep 0.5; done

If I implemented it using inotify, I presume I can get rid of the sleep, but that entails compiled code.

Meh, lets leave Xgl out of Ibex

Not really stated anywhere obvious, but apparently Xgl isn’t the preferred mechanism for all the special effects in Xorg post 7.1, it’s AIGLX that you should be using. It’s integrated too ๐Ÿ™‚
So after wasting a lot of time trying to get the nvidia driver working properly with XGL after the latest updates I just searched the docs here, there and everywhere and finally picked up on the ‘just use AIGLX’ vibe.
Maybe I should be reading the release notes… or are they in the release notes at all? I managed to survive gutsy and hardy without changing over to AIGLX and it still worked, so I may just have missed this in the notes somewhere.