[Feature Request] OSX / FSEvents

[robfig]

Hi,

One chap submitted a pull request to watch the entire GOPATH worth of source:
https://github.com/robfig/revel/pull/153

That seems to work in inotify (and I believe windows as well) -- however, kqueue chokes on it due to the number of file descriptors required.

This article suggests that FSEvents would be the mechanism to use for directory-level monitoring:
http://stackoverflow.com/questions/1772209/file-level-filesystem-change-notification-in-mac-os-x

Have you thought about integrating this sort of functionality into your package? I think it would be valuable (for revel in particular!).

Thanks again!
Rob

[DisposaBoy]

FWIW, I believe the inotify limit is per user which means it's shared by all other programs run by the user.

[ `sysctl fs.inotify.max_user_watches` | done: 48.58157ms ]
    fs.inotify.max_user_watches = 8192

that's on Ubuntu and I think that's the default

[howeyc]

Unfortunately I can't add FSEvents functionality as I don't have an OSX machine and from what I can tell BSD doesn't have FSEvents.

[pmonks]

I have an OSX machine and would be happy to help test. I'm an utter go n00b however, so probably won't be much use in writing the code (yet).

[paulhammond]

I just spent a few minutes looking into this. I think there are two main challenges here (aside from your lack of a OS X machine):

Firstly, the FSEvents API is designed to support easily monitoring an entire directory tree, including subdirectories, not just a single directory. This is good; it's the reason FSEvents would work for watching an entire GOPATH, but it doesn't map exactly to the existing fsnotify API. Implementing FSEvents without watching subdirectories isn't going to be much more useful than the existing code. #56 is possibly relevant here.

Secondly, FSEvents is not a set of syscalls like kqueue, it's only available as a C/Objective C framework. Behind the scenes, as with many things on the mac, it's managed by a separate fseventsd process. That in turn listens on /dev/fsevents which you could in theory read from, but the mac kernel developers have said that the non-framework api is unsupported and likely to cause problems. But using the framework means that any code that uses fsnotify on a mac will have to be compiled under cgo, which might cause other problems elsewhere.

At this point I gave up - hopefully someone can find a way to make this work...

[robfig]

I definitely agree with #56 that a recursive watch is super common and should be supported.

To your second point, is there a problem that it requires a framework, or just that it may be difficult to make it work? This project has supposedly done it successfully:
https://github.com/samjacobson/go.fsevents

Using cgo doesn't seem like a problem, especially since it would only be used for the platform that requires/uses it.

[paulhammond]

Good point - I usually avoid cgo based code because it makes cross compilation much harder, if not impossible. That's not a good reason to not offer FSEvents when available.

That said, I'm not going to be building this (working through the issues helped me find an alternate solution to the specific problem I was hoping to solve with FSEvents)

[nathany]

I have OS X and would be willing to help out once the recursive watcher is in place. If @samjacobson adds an open source license to his project, it could be a good basis for adding FSEvents.

[samjacobson]

I was contacted by @nathany regarding this issue. I can't add an open source license to go.fsevents because it's not mine; it's a fork of Steven Degutis' code. I made the fork when I was working on a project that required this functionality. I ended up implementing my own version, which I would be happy to provide under a suitable license.

[nathany]

@samjacobson That would be great! Fsnotify is under a 3-clause BSD license, if that works for you.

[samjacobson]

I've forked fsnotify, and put together a rough implementation. See https://github.com/samjacobson/fsnotify. This is based on my "EventStream" code, and is not perfectly matched to fsnotify semantics yet. At present the Watcher requires a list of paths to watch when it is created. In order to support Watch and RemoveWatch the (internal) FSEventStream will need to be recreated. I'll send a pull request when it's closer to ready.

I changed fsnotify_bsd.go to be used on darwin if cgo is not available. fsnotify_osx will be used on darwin if cgo is available. Any comments / assistance would be appreciated. I understand the FSEventStream (OSX API) well, and of course my original cgo interface, but I don't fully understand the fsnotify API yet.

[howeyc]

@samjacobson That looks very promising, thank you!

[nathany]

@samjacobson This is a great start. Thanks for pitching in.

So there will be some overhead each time a Watch is added to the Watcher?

It may be worth having an internal API that can watch multiple paths at once for adding a recursive watch. Here's some "pseudo-code" of what I have in mind so far, though I haven't worked out the details of internal state (watches, fsnFlags) across subdirectories, and there may be a better OS-specific way to do this.