Hey there! I see you're not using JavaScript. Just FYI, I use MathJax to display things that deserve to look like math.

Alex

Computer "scientist"

Alex Clemmer is a computer programmer. Other programmers love Alex, excitedly describing him as "employed here" and "the boss's son".

Alex is also a Hacker School alum. Surely they do not at all regret admitting him!

Day 266: The unfamiliar world of OS X syscalls

February 25, 2014

Hacker School: day 266.

(My batch ended on August 22, 2013, but as they say, never graduate.)

After learning a bunch about concurrency primitives Yesterday, I decided it would be fun to have an operational understanding of their implementation. So I decided to boot up dtruss (which is like dtrace, but for OS X) and look at the syscall pattern under different cool concurrent scenarios.

I started by running dtruss on this very simple program that I wrote:

#include <stdio.h>
#include <semaphore.h>

int main()
{
    sem_t mutex;
    sem_init(&mutex, 0, 1);
    sem_post(&mutex);

    return 0;
}

… which promptly caused my terminal to explode with a huge chain of syscalls that I did not recognize:

SYSCALL(args)        = return
open(".\0", 0x0, 0x1)        = 3 0
fstat64(0x3, 0x7FFF6AFE1260, 0x0)        = 0 0
fcntl(0x3, 0x32, 0x7FFF6AFE14E0)         = 0 0
close(0x3)       = 0 0
stat64("/Users/alex/Desktop/fun/scratch/locks\0", 0x7FFF6AFE11D0, 0x0)       = 0 0
issetugid(0x7FFF6B01D530, 0x7FFF6AFE1A30, 0x7FFF6B01D530)        = 0 0
csops(0x0, 0x0, 0x7FFF6AFE14BC)      = 0 0
shared_region_check_np(0x7FFF6AFDF408, 0x2, 0x55)        = 0 0
stat64("/usr/lib/dtrace/libdtrace_dyld.dylib\0", 0x7FFF6AFE05D0, 0x7FFF6AFE14C0)         = 0 0
open("/usr/lib/dtrace/libdtrace_dyld.dylib\0", 0x0, 0x0)         = 3 0
pread(0x3, "\312\376\272\276\0", 0x1000, 0x0)        = 4096 0
pread(0x3, "\317\372\355\376\a\0", 0x1000, 0x1000)       = 4096 0
mmap(0x10B3E6000, 0x2000, 0x5, 0x12, 0x3, 0x100001F)         = 0xB3E6000 0
mmap(0x10B3E8000, 0x1000, 0x3, 0x12, 0x3, 0x100001F)         = 0xB3E8000 0
mmap(0x10B3E9000, 0x1F40, 0x1, 0x12, 0x3, 0x100001F)         = 0xB3E9000 0
close(0x3)       = 0 0

[... AND SO ON ...]

Full printout here. Where’s the initial call to execve? Have I really never run dtrace on OS X before? There were some similarities here (for example fstat is a Linux function too), but overall I had no idea what was going on.

UPDATE: found this neat article that actually has a remarkably similar trace! Wish I found that hours ago.

A lot of this looked to me like dynamic library loading, so I tried to run clang with -static, but I couldn’t quite get it to work. Something about a missing library.

Anyway, a lot of the differences between my prior experiance and this seem to be because OS X is a BSD rather than a Linux. In any event I began looking everything up to see if I could piece together what was happening in kernel land to run this program.

The story the syscalls tell

If you don’t know OS X syscalls very well (I also don’t!), you can have a look at the appendix at the bottom of this entry, where I kept my notes as I looked each of them up.

Here is what I think is going on in the syscalls above. A lot of this I’m not quite sure about/not sure how to find out more about. So some of it is pure speculation.

After this, we check the status of quite a few library files:

stat64("/usr/lib/libSystem.B.dylib\0", 0x7FFF6AFE0410, 0x7FFF6AFE1290)       = 0 0
stat64("/usr/lib/system/libcache.dylib\0", 0x7FFF6AFE0110, 0x7FFF6AFE0F90)       = 0 0
stat64("/usr/lib/system/libcommonCrypto.dylib\0", 0x7FFF6AFE0110, 0x7FFF6AFE0F90)        = 0 0
stat64("/usr/lib/system/libcompiler_rt.dylib\0", 0x7FFF6AFE0110, 0x7FFF6AFE0F90)         = 0 0
stat64("/usr/lib/system/libcopyfile.dylib\0", 0x7FFF6AFE0110, 0x7FFF6AFE0F90)        = 0 0
stat64("/usr/lib/system/libdispatch.dylib\0", 0x7FFF6AFE0110, 0x7FFF6AFE0F90)        = 0 0

[...]

I guess it’s hard to know why we’d inspect so many of these files. I suspect they’re dynamic libraries, and we’re only loading libdtrace into memory with mmap because we need it. Notice that none of them return error codes, so we’re definitely asking for information about files that exist.

Something that’s a bit more mysterious is the huge number of calls to mmap that follow:

mmap(0x0, 0x2000, 0x3, 0x1002, 0x1000000, 0x4)       = 0xB3EB000 0
mprotect(0x10B3EB000, 0x88, 0x1)         = 0 0
mmap(0x0, 0x17000, 0x3, 0x1002, 0x1000000, 0x6)      = 0xB3ED000 0
mprotect(0x10B3ED000, 0x1000, 0x0)       = 0 0

[...]

Notice the file descriptor is 0x1000000 — that seems very high, and I don’t know what it’s for.

But, all this comes to a head when, at the final two lines:

sem_init(0x7FFF6AFE1908, 0x0, 0x1)       = -1 Err#78
sem_post(0x7FFF6AFE1908, 0x0, 0xFFFFFFFFFFFFFFFF)        = -1 Err#9

We learn that actually I programmed these wrong, they error out, and nothing much happens.

Ah, oh well.

Final notes

Oddly enough, when you run this multiple times, things execute in a different order! If you have a look at the other tests in the gist, you’ll see that many of them are quite alike, but occur in different order.

Appendix: my notes on the syscalls

I’ve never really been much of a systems programmer. So I started cracking them open one by one. These are some of my notes. Some of them might be wrong!

Another curious note: a lot of string literals in these calls have a \0 at the end. For example: open(".\0", 0x0, 0x1). I think this is because they’re all null-terminated C strings. So, this syscall is basically opening with no flags and (I think) read-mode.



comments powered by Disqus