David's Computer Stuff Journal: Process creation monitoring

Process creation monitoring

David N. Welton — Mon, 06 Aug 2007 02:53:00 -0700

So, I'm stumped - maybe someone knows the answer to this one. I noticed that on my new Ubuntu system, if I ran ps every 5 seconds or so, the process number jumped significantly, by 5 or 6. Being a bit of a control freak, I want to know exactly what's creating those processes, so I wrote a little Tcl script to try and catch any new [0-9]* directories in proc and read its cmdline file entry, but that seems to not be fast enough to catch the culprit in the act. I had a look at inotify, but that doesn't seem to work with /proc. So at this point, I'm stumped, and am looking around for suggestions. I just want to have a record of newly created processes... seems like it ought to be possible.

"Process creation monitoring" by Benoît Dejean

Tue, 07 Aug 2007 18:02:38 -0700

http://bugzilla.gnome.org/show_bug.cgi?id=439930

"Process creation monitoring" by Stuart Yeates

Tue, 07 Aug 2007 09:02:02 -0700

What are your cron settings?

I log authentication, so my logs are full of cron waking up, shedding previledges, forking a process to check whether any jobs need running, finding that none are, and exiting.

"Process creation monitoring" by Steve

Mon, 06 Aug 2007 15:36:26 -0700

You can log all processes with snoopy:

http://www.debian-administration.org/articles/88

"Process creation monitoring" by Dave Welton

Mon, 06 Aug 2007 07:41:33 -0700

And the gnome cups thing is spawning threads in order to do some HTTP calls to cups without blocking, apparently. Didn't think it was anything important, but it's sort of annoying all the same.

"Process creation monitoring" by Dave Welton

Mon, 06 Aug 2007 07:21:17 -0700

Ok, the solution!

I looked at how ps -eLF was doing things, and modified my Tcl code accordingly:

proc main {} {
    set glob {/proc/*/task/[0-9]*}

    set procs [lsort [glob $glob]]

    while {1} {
        set newprocs [lsort [glob $glob]]
        if { [llength $procs] != [llength $newprocs] } {
            foreach i $newprocs j $procs {
                if { $i != $j} {
                    puts "New process $i"
                    set fl [open "$i/cmdline"]
                    set data [read $fl]
                    close $fl
                    puts "cmdline: $data"
                    break
                }
            }
        }
        set procs $newprocs
    }
}

main

It's quick and dirty, compared with something like systemtap, but requires no kernel recompile or modifications, and was able to find the culprit:

gnome-cups-icon--sm-client-iddefault3

Now, I wonder why that's spawning so many threads?

"Process creation monitoring" by Dave Welton

Mon, 06 Aug 2007 06:58:49 -0700

accttail doesn't do the job, because it's looking at processes, and I think the posters focusing on threads are on the right trail. The only process I can find doing anything is postgresql doing some cleanups, but even turning that off, I get the mystery threads...

Even so, thanks for pointing it out - it's a neat program that I'll keep in mind for the future.

"Process creation monitoring" by David

Mon, 06 Aug 2007 06:19:54 -0700

The connector reports process events to userspace. It uses the netlink mechanism and your kernel must be built with the following configuration options:

Device Drivers --->

Connector - unified userspace<->kernelspace linker --->

<*> Connector unified userspace <-> kernelspace linker [*] Report process events to userspace

This option is available since 2.6.15.

"Process creation monitoring" by folkert@vanheusden.com

Mon, 06 Aug 2007 04:50:24 -0700

easy. 'acctail' will give you this information nicely. URL: http://www.vanheusden.com/acctail/ it uses the bsd process accounting facility

"Process creation monitoring" by Arvin

Mon, 06 Aug 2007 03:53:36 -0700

Hi,

You could also try the BSD process accounting facility of the linux kernel (General Setup->BSD Process Accounting) and install the "acct" debian package.

"Process creation monitoring" by Adam

Mon, 06 Aug 2007 03:49:08 -0700

What if you do a "ps" every 10 seconds or so? Does the process number jump by 10 or 12, or still just by 5 or 6?

If it's still just 5 or 6, then the short-lived processes might be a consequence of running "ps". Not sure why, but it's a possibility. Try running "strace" on your shell as you launch "ps" to see what happens, or "ps" under "strace". "strace" has options to have it only display fork/exec type calls to cut the output down to something manageable.

Alternatively, write a script to keep "ps"ing in a loop and store the output of each in a file. You'll probably catch one of them after a while as a process that shows up in only one ps list.

"Process creation monitoring" by mla@lausch.at

Mon, 06 Aug 2007 03:37:39 -0700

Each new thread, not only processes increments the PID. use "ps -eLf" to look at the threads(LWP) in a process. you don't find them in /proc. A multithreaded program example:

"ps -eLf" returns 6564 as the PID and 6565-6580 as LWPs for this pid

"ls /proc | grep 6565" returns nothing, but "ls /proc/6565" prints the directory listing for this LWP. This means the /proc directory entries are fake and created on demand. Use systemtop as Mikael Olen suggested to instrument exec/clone calls in the kernel.

"Process creation monitoring" by Mark Brown

Mon, 06 Aug 2007 03:30:08 -0700

There are some kernel patches people have been trying to integrate recently to provide this sort of instrumentation. You really do need kernel support to close the races with short lived processes.

If you have some idea which process is creating new processes you could try ptrace, I expect...

"Process creation monitoring" by Mikael Olen falk

Mon, 06 Aug 2007 03:21:18 -0700

You could try getting the info with systemtap http://sourceware.org/systemtap/