c - pthread_create fails with EAGAIN

Question

Consider this code snippet here, where I am trying to create a bunch of threads which end up processing a given task which simulates a race-condition.

const int thread_count = 128;
pthread_t threads[thread_count];

for (int n = 0; n != thread_count; ++n)
{
    ret = pthread_create(&threads[n], 0, test_thread_fun, &test_thread_args);
    if( ret != 0 )
    {
        fprintf( stdout, "Fail %d %d", ret, errno );
        exit(0);
     }
 }

Things generally works fine except occasionally pthread_create fails with errno EAGAIN "resource temporarily unavailable", I tried inducing usleep, and retry creating but with no real effect.

the failure is sporadic and on some boxes no failures and on some occurs very frequently.

any Idea what could be going wrong here ?

Edit - 1

update on max-threads

cat /proc/sys/kernel/threads-max
256467

Edit 2

I think the inputs here kept me thinking, i'll probably do the below and post any results that are worth sharing.

1. set stack size to a minimum value, I don't think thread_function uses any large arrays.
2. increase my memory and swap ( and nullify any side effects )
3. write a script to monitor the system behavior and see any other process/system daemon is interfering when this case is run, which is in turn can cause resource crunch.
4. system hard and soft limits are pretty high so I'll leave them as they are at this point.

Answer 1

If your program makes sure that it never creates more threads than the system limit allows for (by joining threads before creating new ones), then you are likely running into this kernel bug:

• Task exit is signaled before task resource allocation, leading to bogus EAGAIN errors

With certain kinds of container technology, the race window appears to be much larger, and the bug is much easier to trigger. (This might depend on the types of cgroups in use.)