Boost ASIO makes signal handling pretty painless in C++. It handles the signal and calls a callback in which you can use any function, not just signal safe functions.
Can someone explain the use of "volatile int running" here?
I'm not sure it's correct but it might work here because the only place it's where it's read from without locking is the main thread (`while(cond->running)`), which is also the only place where it's written to (with locks this time).
I'd err on the side of caution here and go with the rule of thumb "volatile is almost always wrong". The slightest change of the code (e.g. allowing another thread to terminate the app) will cause it to fail. It's not much effort to change it to __atomic_load/__atomic_store and remove the volatile qualifier or lock/unlock the mutex every time it's being accessed (costs about 50 nanoseconds).
volatile is usually neither necessary nor sufficient for concurrency as it doesn't have correct semantics. In some simple notification cases (just a polled exit flag for example), given a few assumptions about the compiler and machine model, volatile might have worked. But C11 has _Atomic and there is no excuse for volatile.
In the specific example given in the article, not only volatile, but even _Atomic would be completely superfluous. 'running' it is set once in main before any threads is spawned so there is no concurrency issues; after that every read and write access is done under the protection of the 'lock' mutex, so there is no need for any special qualification in POSIX or C11.
Interestingly, a 'volatile sig_atomic_t' used in a signal handler is both required and sufficient to communicate with the thread of execution interrupted by a signal handler and is the only portable use of the keyword.
edit: there is another portable use in conjunction with {sig,}{set,long}jmp of course.
I'm pretty sure the example code is not correct, but may work out of luck.
The first 'running = 1' is not guaranteed to be visible to other threads because there is no synchronization around it. In practice the pthread_create will probably "fix" the race condition.
My best understanding is that you need a mutex or an atomic every time you access shared data to guarantee visibility. Most CPU architectures don't need the implicit memory barriers from mutex or atomic but it's not wise to rely on it.
the creation of the thread (i.e. the call to pthread_create) in main "synchronizes with" the invocation of 'worker' in the child thread; the store is sequenced before pthread_create; in turn the invocation of 'worker' is sequenced before the load in 'worker', which implies that the store 'happens before' the load, guaranteeing visibility and absence of races.
This is formally guaranteed by C++11 and C11 for std::thread and thrd_create, and also by posix for pthread_create although the language is more informal.
That's wildly inaccurate. It's a language level feature that indicates values can change in ways other than what's suggested by the code that manipulate them.
Yes. signalfd is the easy and correct way to handle signals on Linux, as long as you don't care about portability. I use it in everything I write that needs to handle signals.
The article can be boiled down to the last sentence, which retracts the rest of the article:
>So my claim that there’s a better alternative to signalfd is wrong, but it’s still true that signalfd doesn’t solve any of the other troubles with signal handling: it just saves you a separate thread.
Of course this sentence is disingenuous. Avoiding a separate thread is a huge gain. Dealing with concurrency is one of the main problems with signals. So yes, signalfd is absolutely hugely useful, allows for a lot of simplification, and should be the first tool you reach for if you want to handle signals on Linux and don't need portability.
Basically, the main problem with signalfd is a problem with the Linux implementation of signals in general: if your program requires some kind of logic where each individual signal delivery somehow "counts" in some way, you may run into a problem because the kernel may coalesce/collapse multiple identical signals delivered consecutively into a single signal delivery. So if your program has some kind of logic that relies on responding to each individual signal, you may have a problem. (Note that this problem does not apply to POSIX real time signals.)
But as far as I know, the only reason you'd ever care to actually count individual signal delivery events is with SIGCHLD. So when waiting for SIGCHLD, using signalfd is not sufficient. You'd need to also use one of the wait/waitid/waitpid variants to ensure that you can respond appropriately to each individual SIGCHLD delivered.
On the flip side of coalescing you can end in a situation where volume of signals leads to grinding down in the consumer or worse consumer hanging.
What do you do in that situation? You can force back pressure semantics on senders but then those can grind down, a lot of classic Unix tools were never built with that case in mind.
You could even argue that for signals coalescing should be preferred behavior and multiple signals could be merged together SIGIO, SIGCHLD. In SIGIO you can merge signals impacting FD, with SIGCHLD you should already be running waitpid with WNOHANG.
Sadly I don't believe this can uniformly apply to all signals (don't want to coalesce SIGSEGV/SIGBUS). Also siginfo_t complicates things quite a bit (since the info in there makes it hard to dedup). But such is life in POSIX.... we only now have the hindsight.
Can signalfd be fixed? I think it can, if we explicitly tie
signalfd into the signal-disposition mechanism. If signalfd
could claim responsibility for signal delivery, instead of
requiring that signals be masked or ignored in addition to
using signalfd, this would solve both problems.
If you're going to fix signalfd, fix it right. BSD's kqueue EVFILT_SIGNAL event behaves precisely as you'd want in most situations: an event is triggered regardless of signal disposition. Similarly, and again unlike signalfd, an event will be triggered for each listener. This means different components of a process can react to signals without having to cooperate explicitly or implicitly, whether or not they're using a handler or kqueue.
Basically all Linux needs to do is copy BSD. But that will _never_ happen, because that would be too sane.
There's no way to fix the issue with coalescing. The kernel cannot buffer an unlimited number of events--memory is finite. If your process doesn't loop on wait4 it's fundamentally broken.
The pipe trick doesn't help because you can overflow the pipe buffer, too. On a large, heavily loaded system, receiving several thousand signals before any can be processed could easily happen. In the context of a mechanism like signals where there's no way to throttle the sender with back-pressure, coalescing is the best and correct answer.
The same issue pops up with inotify, which specifies the IN_Q_OVERFLOW event for the same reason. The kernel simply can't buffer an unlimited number of events. Nor can it block file operations when the buffer is full. It can't coalesce events, either, because they could all be from different files, and in any event I think inotify tries to preserve ordering.
Unsurprisingly, with EVFILT_VNODE kqueue implements file notification events only for open file descriptors. That is, you can only listen for events on files for which you have an open reference. File events are then coalesced into a buffer associated with this open reference. Events can never be lost, though ordering can be lost. Which means that compared to inotify, it's much easier to write _correct_ software that doesn't accidentally fail because you forgot to handle the rare overflow scenario, or handled it incorrectly.
Of course, inotify is much more convenient than EVFILT_VNODE. It also uses fewer resources for the common case. Which is a classic distinction between Linux and everybody else: Linux makes the common things blazingly fast and easy but makes it difficult to write resilient software, or to implement novel solutions not conceived of originally.
> There are developers out there that try to catch signals like SIGKILL, SIGABRT, SIGSTOP. This is a pointless attempt the signal will NEVER be delivered to the program.
> The worst offender I have seen was somebody trying to take a pthread lock inside the signal handler. Then trying to fix the "deadlock" by making the lock recursive!! This was so that a pthread_cond_signal could be sent to get the application to exit.
we must have been looking at the same codebase...
for posterity, the correct solution is to use a posix semaphore which is async signal safe.
For the signals that you must handle with a handler (SIGSEGV etc.), Rust doesn't help you. Your code must be "async-signal-safe", that is, everything must work correctly despite being called between two arbitrary opcodes in the host program. Importantly, this means anything that allocates is impermissible, since you might be in the middle of an allocation, and your allocator almost certainly has a per-thread lock and doesn't expect to be called reentrantly on the same thread. And there's no compile-time tracking (e.g., no function type) for functions that don't allocate. You also might be called in the middle of an unsafe block modifying global data, etc.
I've considered designing some sort of complicated DSL in Rust to construct async-signal-safe handlers, but it didn't seem worth the complexity. (Although, now that I write this comment out, I think you can handle SIGSEGV safely from another thread as long as your first thread calls for help in an async-signal-safe way, so maybe that would be a library worth writing.)
For the rest, using sigwaitinfo or signalfd works just as well in Rust as in C/C++, but not really better or worse. There aren't any good libraries for it (which I keep meaning to do something about), but the underlying POSIX APIs are usable equally well in all three languages.
As for whether you should worry about it? Try not to use signals, regardless of language. If you're writing some sort of demand-paging system using SIGSEGV handlers, I hope you know what you're doing already. If you're thinking about SIGALRM, there are other ways to get a notification after a fixed time (and Rust's mio/tokio, for instance, gets you timers in your event loop). If you're trying to use SIGUSR1/SIGUSR2, please rethink your design - you'll be much happier with something like a UNIX socket listening for connections or a localhost-only URL for debugging or whatever. If you want to handle SIGINT (intercept Ctrl-C at a terminal), SIGQUIT (intercept Ctrl-\ at a terminal), SIGTSTP (intercept Ctrl-Z at a terminal), SIGTERM (intercept the "kill" command's default signal), SIGHUP (do something when the terminal exits), SIGWINCH (get notified when the terminal resizes), or SIGCHLD (get notified when a child process exits), you do have to use signals, but all of these can be run through sigwaitinfo or signalfd. And you don't have to actively worry about them if you don't want to customize their behavior, or if (for SIGINT, SIGHUP, etc.) you just want to ignore them and not do anything special.
The only odd case is SIGPIPE, and Rust's standard library blocks SIGPIPE for you and returns EPIPE from a system call instead, and unblocks it in child processes, so you don't need to worry about it. (The default SIGPIPE behavior is basically legacy compatibility with existing C code from the very early days of UNIX.)
> For the rest, using sigwaitinfo or signalfd works just as well in Rust as in C/C++, but not really better or worse. There aren't any good libraries for it (which I keep meaning to do something about), but the underlying POSIX APIs are usable equally well in all three languages.
There are a couple crates which should suffice for basic SIGTERM/SIGHUP/SIGINT handling:
Couldn't a non-libc "standard library" for a hypotethical language be written to be re-entrant safe so dealing with signals would be less painful? For example having a separate malloc-like heap for allocs inside signal handlers etc?
Signal handlers can interrupt other signal handlers. But you probably want to avoid this feature, lest you go mad, so let's ignore it.
Yes, you could write a library for use inside signal handlers that shares no state with the actual libc. Then you could also safely printf, for instance, although you'd jump ahead of any printfs in the host program that are still buffered, etc.
But it's not just libc's globals that are a problem, it's your own globals and those of any other libraries, and if you intend to do nontrivial work in the signal handler (such that having a compiler check that you're async-signal-safe is worth doing), you probably are actually trying to share state with the main program. For instance, say you're installing SEGV handler to virtually map a large data file into memory, fetching it from some remote file storage API as needed. You want access to some data structures from the main program to figure out what to map, probably the ability to make HTTPS connections, etc. And you have to do that all from the signal handler, because whatever pointer segfaulted has to be working by the time you return from your signal handler.
You might be able to do this with Rust plus cleverness, by putting the signal handler in a separate statically-linked crate and using the ownership system to track the fact that the signal handler needs read-only access to some data. (Or with dlmopen, in C.) But it would still be very complex.
mmap is sometimes used as an async signal safe allocation function. IIRC mmap async signal safety is not guaranteed by POSIX but it often is in practice on many platforms. It is not cheap though.
edit: I should have read ben0x539 comment which suggests the same thing.
Can I just call mmap instead of anything requiring userspace coordination beyond function scope if I need dynamic memory in a signal handler?
How easy is it to fuck up the signalfd thing and end up not responding to SIGINT in a timely manner, because I'm being silly and doing some blocking work in the signalfd loop for some reason? Does that happen to people at all?
What kinda usecases are there for handling SIGSEGV? I'm not the kind of person to write on-demand-paging systems so I'm not really sure what that involves and when I need that sort of thing.
> What kinda usecases are there for handling SIGSEGV?
If you're building a runtime that includes go like goroutines, you could handle SIGSEGV when you run out of stack space and graceful kill the go routine.
You can use it for some kind of online migration of VMS.
You could build a moving garbage collector system for native code multi-threaded code where you don't need to stop the world... you can deal with race of not stoping / moving by making those pages in accessible and only stoping the thread / fixing up if happens to hit moved memory.
A runtime for a language that has multi-threading system that doesn't share VM mappings (which is possible using the clone sys call on linux) but can share other things.
All somewhat advanced uses. But uses never the less.
Last time I used C I set a SIGSEGV handler that printed a stack-trace and exited. Which made debugging somewhat easier. (Although It was often segfaulting in libwayland, which didn't help as much as I would like)
SIGBUS is simialr to SIGSEGV (It happens if a mmap is larger that the backing file and you try to read the unbacked memory).
You basically have to handle it if you don't control the backing file, Eg: with shared memory when the other process resizes it after you've mmap'd it.
libwayland-server can install one for you that replaces the map with 0'd memory.
> Can I just call mmap instead of anything requiring userspace coordination beyond function scope if I need dynamic memory in a signal handler?
Yes, but if you're doing nontrivial work, you probably don't just want memory for yourself, you want to share some actual program state with the main process. See the other comment I posted.
In practice, making any raw system call is async-signal-sae. POSIX doesn't specify this because it doesn't specify what is a system call at all - it's purely a C-language, compile-time interface, and a unikernel that makes everything a library call, a Windows subsystem, etc. are all valid ways to implement it. But on a system where mmap is in fact a system call (and the libc wrapper does nothing interesting) it follows from first principles: there's no userspace resource that can be left in an inconsistent state.
> How easy is it to fuck up the signalfd thing and end up not responding to SIGINT in a timely manner, because I'm being silly and doing some blocking work in the signalfd loop for some reason? Does that happen to people at all?
I mean, this is basically the same question as "How easy is it for a system that's only accessible via an HTTP API to get stuck processing an event and not process a POST /quit". If you're worried about messing up your event loop and getting stuck, then yes, SIGINT interrupting your process is useful -- but also, you've already admitted you don't know what the state of your process is. So an orderly shutdown is difficult. Any work you could do to make it easier turns out to be work that makes your original non-blocking design get implemented correctly.
You should probably just leave SIGINT at its default disposition (terminate the process immediately, same as SIGKILL) and save state to disk periodically or something. Or, at best, implement a signal handler that does very little interaction with process state, maybe just writing a log message (but note that syslog isn't async-signal-safe!) and exiting.
SIGSERV is basically a page fault in userspace. Its just named differently. The user mode linux kernel port makes use of it in exactly this way.
Another use case would be a massive one to one hash table. You allocate the address space and when you fault on it you load data from a database on demand and continue execution.
There's also something in Mach for this on OS X, but I forget the details. And a handful of non-UNIX OSes have had support for an outside program being able to get notified efficiently on segfaults and map pages in.
Difficult, though! I'm not sure I know of any queue / buffer implementations that can be interrupted between any two arbitrary instructions and still preserve the buffer in a readable state. What happens if the buffer's updated capacity is written to memory before the updated pointer (remember that the compiler can reorder store instructions in the main program if it's not observable from other threads) and reading it causes another segfault?
If you care deeply about reliable logging in the presence of arbitrary segfaults, get it to a buffer outside of the program: either write(2) / send(2) it to a file/pipe/socket, or write it to a shared memory region / mmaped file and pay attention to the atomicity and ordering of memory writes on that region.
You might also conclude that arbitrary segfaults aren't in scope, and there's some specific high-risk code that is separate from your logging implementation. If you can prevent wild pointers in the high-risk code from reaching the pages used by the logging implementation (e.g., you're a language VM implementation or something), you can relax your requirements. In particular, if you can also prevent them from reaching the libc, and you know the high-risk code cannot induce a segfault in the libc, then you can probably start calling async-signal-unsafe functions. But this requires careful thought.
On the logging system I'm familiar with, the logging queue is lock-free, so any invariant is preserved at every instruction boundary. The log is actually written to disk from a background thread, and the signal handler is responsible of signaling the log thread to flush the logging queue(s).
Of course if the log thread itself has generated the sigsegv (because a log entry or the queue itself is corrupted by a bug) you are out of luck, but you are no worse than not attempting the flush.
Agree that making the log thread a separate process instead is significantly more robust (and not significantly harder in fact).
There's a crate for that. `nix` crate has `signalfd` support. I have recently used it to implement terminal resize in TUI text editor, all in one event loop (`mio` crate).
I needed to write something that needed signal processing a few years ago, its very difficult to find simple help on the subject. Anyway, nice take, I will bookmark for future reference ;)
If I do "kill -HUP <pid>", the SIGHUP is delivered to a process. Each thread in the process has a separate mask of signals its willing to handle. One of those threads that have SIGHUP unmasked will have the signal delivered to it.
If a thread dereferences a null pointer, the sigsegv is only relevant to the thread that dereffed the NULL pointer - that's the only thing that can do anything with it. It makes no sense to send that signal to another thread - it's a signal generated synchronously by the thread itself, rather than asynchronously from anywhere else.
http://www.boost.org/doc/libs/1_64_0/doc/html/boost_asio/ref...