gh-102494: fix MemoryError when using selectors on Solaris

[kulikjak]

When you set ulimit -n unlimited on Solaris and its derivatives (where /dev/poll is available) and import selectors, Python crashes with a MemoryError because there is no upper limit to the allocation size.

This fix adds an arbitrary limit of 2^18 (which results in roughly ~4MB of memory).

Fixes #102494

• Issue: MemoryError when using selectors on Solaris #102494

[arhadthedev]

@jcea (as a Solaris expert)

[jcea]

For reference, documentation at https://docs.oracle.com/cd/E19253-01/816-5177/6mbbc4g9n/index.html for example.

Notice that current array plays two roles:

1. Provide registration and removal of file descriptor information to the kernel. This operations don't require a huge array, you just flush descriptor information when the buffer is full or just right before the "poll" operation. This is already implemented in the "devpoll_flush()" function.

2. Get the result information from the "poll" operation. This requires being able to receive, in the worst case, all the registered file descriptors. Solaris will give back information up to "n" file descriptors if not enough space is available, but I don't know what would happen if these fds become active again. Maybe other file descriptors will starve, begging for service, but not receiving it because other active fds are notified and there is not enough space to receive all pending notifications at the same time (*).

Instead of allocating a huge array and clap it to a max size, I would suggest to keep a count of active file descriptors (registrations minus unregistrations) and resize the array as needed (maybe only growing it if needed, never shrinking it).

I would suggest an initial size of 1024, growing 25% when needed.

Shrinking would be nice too, but probably unimportant.

(*) Would be quite interesting to investigate the kernel implementation. I guess that playing with a handful of fds would be enough to determine if the kernel gives back the active fds in a round robin, random or "start from the beginning until you fill the buffer" way. Some comments at https://github.com/illumos/illumos-gate/blob/2c76d75129011c98e79463bb84917b828f922a11/usr/src/uts/common/io/devpoll.c#L237 suggest that Solaris kernel gives back the active descriptor in a round robin way, so actually using a small (1024 entries?) buffer would be fine enough. See also code around line 293 and 629. This assumption needs to be tested, but the intention seems quite clear.

If this assumption holds true and you want a highly concurrent/performant implementation, you could resize the array when the returned list of active file descriptors use the full array, signaling that a bigger array could reduce syscall traffic, although if you are dealing with Python, this kind of optimizations are probably overkill.

PS: I would support a "port" interface, although Solaris is not a tier-1 platform nowadays for Python.

[kulikjak]

Thank you for the detailed notes; I will look into it.

I've also recently hit another issue with the Devpoll selector when working on Cheroot (cherrypy/cheroot#561), so it needs some love.