multithreading - Python Exception in thread Thread-1 (most likely raised during interpreter shutdown)?

Question

My friend and I have been working on a large project to learn and for fun in python and PyGame. Basically it is an AI simulation of a small village. we wanted a day/night cycle so I found a neat way to change the color of an entire surface using numpy (specifically the cross-fade tutorial) - http://www.pygame.org/docs/tut/surfarray/SurfarrayIntro.html

I implemented it into the code and it WORKS, but is extremely slow, like < 1 fps slow. so I look into threading (because I wanted to add it eventually) and found this page on Queues - Learning about Queue module in python (how to run it)

I spend about 15 minutes making a basic system but as soon as I run it, the window closes and it says

Exception in thread Thread-1 (most likely raised during interpreter shutdown):

EDIT: This is literally all it says, no Traceback error

I don't know what I am doing wrong, but I assume I am missing something simple. I added the necessary parts of the code below.

q_in = Queue.Queue(maxsize=0)

q_out = Queue.Queue(maxsize=0)

def run():    #Here is where the main stuff happens
    #There is more here I am just showing the essential parts
    while True:
        a = abs(abs(world.degree-180)-180)/400.

        #Process world
        world.process(time_passed_seconds)

        blank_surface = pygame.Surface(SCREEN_SIZE)
        world.render(blank_surface)    #The world class renders everything onto a blank surface
        q_in.put((blank_surface, a))
        screen.blit(q_out.get(), (0,0))

def DayNight():
    while True:
        blank_surface, a = q_in.get()
        imgarray = surfarray.array3d(blank_surface)  # Here is where the new numpy       stuff starts (AKA Day/Night cycle)
        src = N.array(imgarray)
        dest = N.zeros(imgarray.shape)
        dest[:] = 20, 30, 120
        diff = (dest - src) * a
        xfade = src + diff.astype(N.int)

        surfarray.blit_array(blank_surface, xfade)
        q_out.put(blank_surface)
        q_in.task_done()

def main():
    MainT = threading.Thread(target=run)
    MainT.daemon = True
    MainT.start()

    DN = threading.Thread(target=DayNight)
    DN.daemon = True
    DN.start()

    q_in.join()
    q_out.join()

If anyone could help it would be greatly appreciated. Thank you.

Answer 1

This is pretty common when using daemon threads. Why are you setting .daemon = True on your threads? Think about it. While there are legitimate uses for daemon threads, most times a programmer does it because they're confused, as in "I don't know how to shut my threads down cleanly, and the program will freeze on exit if I don't, so I know! I'll say they're daemon threads. Then the interpreter won't wait for them to terminate when it exits. Problem solved."

But it isn't solved - it usually just creates other problems. In particular, the daemon threads keep on running while the interpreter is - on exit - destroying itself. Modules are destroyed, stdin and stdout and stderr are destroyed, etc etc. All sorts of things can go wrong in daemon threads then, as the stuff they try to access is annihilated.

The specific message you're seeing is produced when an exception is raised in some thread, but interpreter destruction has gotten so far that even the sys module no longer contains anything usable. The threading implementation retains a reference to sys.stderr internally so that it can tell you something then (specifically, the exact message you're seeing), but too much of the interpreter has been destroyed to tell you anything else about what went wrong.

So find a way to shut down your threads cleanly instead (and remove .daemon = True). Don't know enough about your problem to suggest a specific way, but you'll think of something ;-)

BTW, I'd suggest removing the maxsize=0 arguments on your Queue() constructors. The default is "unbounded", and "everyone knows that", while few people know that maxsize=0 also means "unbounded". That's gotten worse as other datatypes have taken maxsize=0 to mean "maximum size really is 0" (the best example of that is collections.deque); but "no argument means unbounded" is still universally true.