Skip to main content

Dynamically sizing threadpools

Posted by sdo on June 7, 2007 at 10:15 AM PDT

Almost every thread pool implementation takes great pains to make sure
that it can dynamically resize the number of threads it utilizes: you
specify the mininum number of threads you want, the maximum number, and
the thread pool in its wisdom will automatically configure itself to
have the optimal number of threads for your workload. At least, that's
the theory...

But what about in practice? I'd argue that its utility is very limited,
and that in many cases, a dynamically-resizing threadpool will actually
harm to the performance of your system.

First, a quick review of why we have threadpools. From a perfomance
perspective, the most important task of a threadpool is to throttle the
number of simulatneous tasks running on your system. I know that you
may think that the purpose of a threadpool is to allow you to
conveniently run multiple things at once. It does that, but more
importantly, it prevents you from running too many things at once. If
you need to run 100 CPU-bound tasks on a machine with 4 CPUs, you will
get optimal throughput if you run only 4 tasks at a time: each task
fully utilizes the CPU while it is running. Since you can't run more
that 4 tasks at once, you won't get get any better throughput by having
more threads -- in fact, if you add more threads to the saturated
system, your throughput will go down: the threads will compete with
each other for CPU and other system resources, and the operating system
will spend more time than necessary managing the competing threads.

In the real world, of course, tasks are never 100% CPU-bound, so you'll
usually want more threads than CPUs to get optimal use of your system.
How many more is a function of your workload:  how much time it
waits for external resources like a database, and so on. But there will
be an optimal number, usually quite less than the number of
simultaneous tasks your can handle (particularly if those tasks
represent jobs coming in from remote users -- e.g. a web or application
server handling thousands of connections). The determining rule is
this: is you have more tasks to perform AND you have idle CPU time,
then it makes sense to add more threads to the pool. If you have more
tasks to perform but no idle CPU time, then it is counter-productive to
add threads to the pool. And that's my problem with dynamically
resizing threadpools: if they choose to add threads because there are
tasks waiting (even though there is no available CPU time), they will
hurt your performance rather than help it.

Conceivably, you could use some native code to figure out the idle CPU
time on your system and have a threadpool that takes that information
into account. That would be better, but even that is insufficient. Say
you have an application server accessing a remote database using JPA.
Now if the database becomes a bottleneck, you'll have idle CPU time on
your application server, and it will have tasks that are waiting. But
adding threads to run those tasks will again make things worse: it will
increase the work needed to be done by the already-saturated database,
and your overall throughput will suffer. In the final analysis, you are
the only one that will have all the necessary information to know if it
is productive to increase the size of your thread pool.

So you are responsible for setting the maximum size of the threadpool
to a reasonable value, so that the system will never attempt to run too
many threads at once. Given you've done that, is there a point in
having a mininum number of threads? The claim is that there is, because
it can save on system resources. But I would argue that the impact of
that is really minimal. Each thread has a stack and so consumes a
certain amount of memory. But if the thread is idle and the machine
doesn't have enough physical memory to handle everything on the system,
that idle memory will simply be paged out to virtual memory. Even if
the thread exits, the memory it used for its stack still belongs to the
JVM process -- the JVM might reuse that memory for something else, but
in general, the memory cannot be returned to the operating system for
use by other processes. So the memory issue doesn't really have much
impact. Depending on the application, it's conceivable that fewer idle
threads may have a small impact because when a thread is reused, it
might happen to have some important data in the CPU cache (whereas an
idle thread selected to run a task won't have any data in the CPU
cache), but the effects of that in the real world are pretty much
non-existent. So it doesn't hurt to have a minimum number of threads,
but you get no real advantage from that either.

One area that can be very subtle in this regard is the
ThreadPoolExecutor, which can be configured to have three values: a
minimum, a core value, and an absolute maximum. In general, threads are
added when tasks are waiting until the system runs the desired core
value of threads. Then everything chugs along nicely, even though a
certain number of tasks may be waiting in the queue. Now say that the
system can't keep up with the tasks queue: the task queue length grows
beyond some defined value. In response to this, the executor will start
adding threads (up to the absolute maximum). But if the system is
CPU-bound, or if the system is causing a bottleneck on an external
resource, adding those threads is exactly the wrong thing to do. And
because this happens only under circumstances such as an increased
load, it might be something that you fail to catch in normal testing:
during normal testing, you'll usually run with the core number of
threads and may not even notice that you've misconfigured the maximum
number of threads to a value the system cannot handle. The converse of
this argument is that the thread pool executor can add new threads when
a burst of traffic comes, and as long as there are resources available
to execute those threads, the executor can handle the additional tasks
(and then, once the burst is over, the extra threads can exit and
reduce system resource usage). But given the minimal-at-best effect
that has on system resources, handling a burst like that doesn't make a
lot of sense to me, particularly given the potential for increasing
load on the system at exactly the wrong time.

All of that is why I always choose to ignore dynamically sizing
threadpools, and just configure all my pools with a static size.

Related Topics >>