Shut down a directory poller's thread pool when the poller stops

[JakeHuneau]

A directory poller running in parallel mode leaks its thread pool every time the partnerships are reloaded.

Cause

DirectoryPollingModule.init creates a fixed thread pool when process_files_in_parallel is set:

if (processFilesAsThreads) {
    executorService = Executors.newFixedThreadPool(maxProcessingThreads);
}

Nothing shuts it down. PollingModule.doStop only cancels the timer, DirectoryPollingModule has no destroy, and BaseSession.destroyPartnershipPollers calls stop() rather than destroy() in any case.

Partnership pollers are destroyed and rebuilt on every partnerships reload, because refreshConfig calls destroyPartnershipPollers and then recreates them. The module object is dropped but the pool's threads are still alive, and live threads are garbage collection roots, so the pool and everything it references survives for the life of the process. Every reload leaks another one, up to max_parallel_files threads per poller.

Fix

Stopping a poller shuts its pool down.

The shutdown refuses new work but lets a file that is part way through being sent finish, rather than interrupting it: an interrupted transmission would leave the partner with an incomplete message and this side unsure whether it arrived. If that work has not drained within ten seconds the wait gives up and logs rather than blocking the reload, and the threads end when the work completes.

Starting a poller again now gives it a usable pool. Without that the restart would leave the poller holding the pool that had just been shut down under it, and a shut down pool rejects everything handed to it, so the poller would have looked alive while failing every file it picked up. That is a hazard the fix introduces if only the shutdown half is done, so both halves are here.

Scope

Only deployments with process_files_in_parallel enabled were affected, as no pool is created otherwise.

Testing

184 tests pass, 4 new. The test runs a real server with parallel mode enabled and takes the poller from the session. Mutation checks: removing the shutdown fails 2 tests, removing the restart guard fails 2.

• aRunningPollerHasAUsablePool
• stoppingThePollerShutsItsPoolDown
• aPollerThatIsStartedAgainGetsAWorkingPool
• theOutboxIsStillPolledAfterARestart