[Novalug] Process intercommunication

Peter Larsen peter@peterlarsen.org
Sat Dec 31 02:05:59 EST 2016


Hey Gary,
I got a bit curious how far this was implemented myself.  As I started
this longer thread with, DBUS is a mechanism mainly used by the
graphical subsystem, so I wondered if the file chooser would react to
changes made by scripts and just server side changes. A quick test, and
the file chooser DID see those files as they changed, and running
dbus-monitor showed that no messages were sent.

So while DBUS definitely can carry events - as long as they're generated
by GUI based programs, there's clearly more at play here. We've already
had other people mention inotify - which is a kernel feature. Without
looking at the code it's hard to say if inotify is exclusively used or
if DBUS is used for file notifications too (it makes little sense to
have both systems do it). But DBUS can still be used for other types of
events, particular around gvfs (Gnome virtual file system). Anyway - I
haven't used inotify in any recent shape or form, so I did a quick man
page or two, and found they've added a nice piece of example code to use
- just to see how things work.

I've attached (if there's no attachment due to mailman restrictions just
grab the copy from the man page) inotify.c   - a pure copy of the
"inotify(7)" man page code. If you're unfamiliar with compiling C code,
it's pretty easy in this case:

Save the file to a "test" directory, and in this directory execute the
following:

$ gcc inotify.c -o inotify

This creates a file "inotify" which is executable. To run it, you simply
type ./inotify

So to use the program, the easiest way is to open two terminal windows
side by side. In both windows you enter the "test" directory with the cd
command. In the first window, you type "./inotify $PWD" and hit enter.
The cursor will just sit there and wait. In the other window, you start
having fun creating files, renaming them, deleting them etc. As you do
you'll will see the output in the first window as inotify events are
received.

I used /tmp and got output like this:
IN_CLOSE_NOWRITE: /tmp// [directory]
IN_OPEN: /tmp// [directory]
IN_CLOSE_NOWRITE: /tmp// [directory]
IN_OPEN: /tmp//mynewfile [file]
IN_CLOSE_WRITE: /tmp//mynewfile [file]
IN_OPEN: /tmp//sh-thd-871127182 [file]
IN_OPEN: /tmp//sh-thd-871127182 [file]
IN_CLOSE_WRITE: /tmp//sh-thd-871127182 [file]
IN_CLOSE_NOWRITE: /tmp//sh-thd-871127182 [file]
IN_OPEN: /tmp//pcp.etvzUwg53 [directory]
IN_CLOSE_NOWRITE: /tmp//pcp.etvzUwg53 [dir

If you read the code, it's not "pretty" event based. It's not using
call-back but a poll loop. But of course, if you simply setup a fork
(thread) with a loop waiting for data, that would certainly resolve the
problem without the need for a callback function in C. But it would
require multiple processes (threads).

If you're not up to playing with C, install inotify-tools (not sure if
that's a generic name - that's the package name on Fedora). In there is
several utilities - and inotifywait sorta does what that little program
is doing, just with a lot more options:

inotifywait -m -r --format '%:e %f' /tmp

This outputs file changes in the  /tmp directory.  Of course it won't
show you how it's implemented.  There's also "inotifywatch" which
collects statistics on the events on the files/directories you give as
parameter.

Note - we could use a loop and test for files, but that takes CPU time.
With inotify you aren't using any system processing time (at least not
that can be measured).   And imagine using a loop to test 1000s of files
for changes. Lot of IO, CPU etc.

You can use inotifywait in your shell scripts to wait without using
system resources for a file change.

As for DBUS, there's definitely DBUS stuff going on with the file open,
but probably not with files. But when you attach drives, create new
favorite places etc. that'll all involve communication across multiple
processes. Imagine for instance you're viewing files using sshfs
(implemented via gvfs) and all of a sudden your network connection goes
down. This sends a message out, and the file dialog can remove the
connected drive from the "open file" dialog. Or when you insert a USB
and do other similar events.

In other words, things were a bit more complex (understatement) than I
initially laid out, so I'm sorry for the diversion here. But hopefully
this gives you insight into the Linux kernel's way of handling these
kind of events.

-- 
Regards
  Peter Larsen



On 12/30/2016 04:54 PM, Peter Larsen wrote:
> On 12/30/2016 04:01 PM, Gary Knott via Novalug wrote:
>> Dear Peter,  I looked at the DBUS "tutorial" you cited.  Thanks.
>> (I object to the term object - and also method.  Linux, and
>> Linux Documentation,  should be kept at the C programmer
>> level in my opinion.)
> Heh - sorry about that. After I sent the message, I wondered if I should
> just have linked the WikiPedia article instead
> (https://en.wikipedia.org/wiki/D-Bus). The difference is, that the link
> I gave here was the developer resource - "begin here" guide. WikiPedia
> tends to be more administrative and theory.
>
> That said, everything we do in Linux eventually invokes a C-library
> (stdlib, glibc etc) feature. That's the nature of the beast here. As
> you'll see in a moment, even the File Chooser dialog is a C function.
> However, as you move up the abstraction layers to languages like Python
> or Java, you'll find these platforms have their own libraries which you
> use. These  libraries will in turn call the C libraries for you - but
> you have an easier way to do things. That said, core functionality is
> defined at the C level - so if the goal is to understand "what can I do
> in Linux" that's really where you have to be looking.
>
> I'm no longer fluent in C - I don't program on a daily basis so I think
> I lost my right to wear the "programmer" badge. I can read a program but
> it's not just a second language anymore. If you speak to developers
> you'll find that they see programming languages as a real language -
> they don't really think about "how" the program works - it's very clear
> from just a glance. To me, it takes "a bit" longer to see what it does
> these days. But I just consider myself a slow-reader :)
>
>> Anyway, I see that DBUS is kinda like the Windows "message queue"
>> system with callbacks being used for receiving "messages".
> Right - the difference is how dbus is implemented behind the scene; to
> you as a user of it it's simple call-back functions you register. 
> That's a very basic programming principle when it comes to event based
> programming; I doubt we could think of better ways than a call back - it
> worked 30 years ago, and it works now :)
>
>> What I did not find was any list of pre-defined "events" that I, as a
>> programmer, could request a call-back for.   It seems likely that some process that implements the "file-selection" window  is getting a callback notifying it that a file owned by user "X" has been closed, but that's just a guess.
> Hang on - I think we have a bit of a misunderstand here. There's a basic
> UI (gtk) library function for File Choosers. I cannot recall ever
> programming to a UI that didn't have such a function/feature. It allowed
> user code to use a standard UI for users to pick files. Windows has it,
> OS/2 has it - X11 has it etc. So when you need a user to pick a file
> from the OS, you'll simple call the function. In gtk you can find the
> spec here (yes, it's C):
>
> https://developer.gnome.org/gtk3/stable/GtkFileChooserDialog.html
>
> This is what Firefox calls. It's pretty basic and Firefox/Chrome really
> isn't involved here.  As you can see, all it does is return a file name
> string - so there's no magic here. There's no need to know how the File
> Chooser works - except it comes with a billion options so reading about
> those options is helpful (ie. should the user be able to pick more than
> one file, which file types should be accepted etc).
>
> What I thought your question was about, was how the File Chooser dialog
> knew you'd changed a file elsewhere. That's where DBUS comes in. But in
> THIS case, you don't need to know how it does it - it's just doing it
> for you :)
>
> But if you were to program your own file chooser, or had a similar need,
> you would need to know what events to listen for. This is where reading
> the fine print around dbus is important. There's a ton of options here,
> but in the end you'll need to know what you're looking for. There's an
> easy way to find out: use dbus-monitor and execute the events you want
> to be notified on - and well, you have all the options/names you need to
> be looking for right there.  There's qdbus too of course to look for the
> registered names/systems.
>
> Here's a few more links to guides:
> *
> https://www.gnu.org/software/emacs/manual/html_node/dbus/Receiving-Method-Calls.html
> * http://blog.fpmurphy.com/2009/02/dbus-monitor.html
> * https://techbase.kde.org/Development/Tutorials#D-Bus
>
>> (Surely, it does not just get a "file-closed" notification and then have to check the dates and times of all files it is displaying?   - Or maybe it just redoes its entire display, accessing the entire directory and formatting its contents without worrying about what file got closed?
> So this is why I choose to show what's actually being sent. Note it's
> the complete file name and action. Ie. there's no need to "refresh" or
> read the directory or anything - you can simply change the data
> structure directly based on the data you're getting. In my case it
> "move" so the file was moved from a source to a destination. My dialogue
> would see that the file was part of the directory it was displaying and
> simply remove the file from the view. Likewise, had I created a file it
> would have been told so, and it could simply display it. Now, in that
> case getting owner, size etc. would most likely require a check of the
> file system. A lazy programmer would simply refresh from the file system
> regardless of the event, as long as it involved the directory it was
> displaying. As long as there's only a "few" files that would work great,
> but if it's thousands of files a refresh could remove the current focus,
> move a lot of files around or even worse take a long time if there are
> 10K+ files to sort through.  Anyway - messaging systems like this,
> allows programmers to make short-cuts and improve performance.
>
>> By the way, is the "file-selection" window  that pops-up i a specific
>> utility program provide3d as part of a Linux distro that is run when I do a thing like click on the "paperclip" in a gmail window being displayed by Firefox?
> It should be the exact same dialog regardless of what software you use.
> Firefox, Chrome, Thunderbird, Evolution, Evince etc. should all present
> you with the exact same dialog when-ever you're accessing a function
> where you have to pick a file.
>
>> [One of the issues that makes understanding "modern"
>> computing daunting , is that understanding the list of programs that
>> run and what they do when you take some action - click or hit a key -
>> is extremely hard  (in fact, often practically impossible!)
>> - it is not just "take an action" => a specific
>> program receives the "action" and acts simply and on its own.]
> Other than the word "modern" would agree that event based programming
> can be tough. Hint - just take a look at NodeJS and let's have a talk
> about "having to twist your head around event programming" :) NodeJS is
> nothing but call-back functions.   It's a hard concept to get through
> for sure. But once understood it's actually beautifully simple in concept.
>
> Btw. I'm sure you know this - but as it's a library function, there's no
> separate process.
>




More information about the Novalug mailing list