subprocess - Running interactive program from within python

Question

I want to achieve something which is very similar to this.

My actual goal is to run Rasa from within python. Taken from Rasa's site:

Rasa is a framework for building conversational software: Messenger/Slack bots, Alexa skills, etc. We’ll abbreviate this as a bot in this documentation.

It is basically a chatbot which runs in the command prompt. This is how it works on cmd :

Now I want to run Rasa from python so that I can integrate it with my Django-based website. i.e. I want to keep taking inputs from the user, pass it to rasa, rasa processes the text and gives me an output which I show back to the user.

I have tried this (running it from cmd as of now)

import sys
import subprocess
from threading import Thread
from queue import Queue, Empty  # python 3.x


def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()


def getOutput(outQueue):
    outStr = ''
    try:
        while True: #Adds output from the Queue until it is empty
            outStr+=outQueue.get_nowait()
    except Empty:
        return outStr

p = subprocess.Popen('command_to_run_rasa', 
                    stdin=subprocess.PIPE, 
                    stdout=subprocess.PIPE, 
                    stderr=subprocess.PIPE, 
                    shell=False, 
                    universal_newlines=True,
                    )

outQueue = Queue()

outThread = Thread(target=enqueue_output, args=(p.stdout, outQueue))

outThread.daemon = True

outThread.start()

someInput = ""

while someInput != "stop":
    someInput = input("Input: ") # to take input from user
    p.stdin.write(someInput) # passing input to be processed by the rasa command
    p.stdin.flush()
    output = getOutput(outQueue)
    print("Output: " + output + "
")
    p.stdout.flush()

But it works fine only for the first line of output. Not for successive input/output cycles. See output below.

How do I get it working for multiple cycles? I've referred to this, and I think I understand the problem in my code from it but I dont know how to solve it.

EDIT: I'm using Python 3.6.2 (64-bit) on Windows 10

Answer 1

You need to keep interacting with your subprocess - at the moment once you pick the output from your subprocess you're pretty much done as you close its STDOUT stream.

Here is the most rudimentary way to continue user input -> process output cycle:

import subprocess
import sys
import time

if __name__ == "__main__":  # a guard from unintended usage
    input_buffer = sys.stdin  # a buffer to get the user input from
    output_buffer = sys.stdout  # a buffer to write rasa's output to
    proc = subprocess.Popen(["path/to/rasa", "arg1", "arg2", "etc."],  # start the process
                            stdin=subprocess.PIPE,  # pipe its STDIN so we can write to it
                            stdout=output_buffer, # pipe directly to the output_buffer
                            universal_newlines=True)
    while True:  # run a main loop
        time.sleep(0.5)  # give some time for `rasa` to forward its STDOUT
        print("Input: ", end="", file=output_buffer, flush=True)  # print the input prompt
        print(input_buffer.readline(), file=proc.stdin, flush=True)  # forward the user input

You can replace input_buffer with a buffer coming from your remote user(s) and output_buffer with a buffer that forwards the data to your user(s) and you'll get essentially what you're looking for - the sub-process will be getting the input directly from the user (input_buffer) and print its output to the user (output_buffer).

If you need to perform other tasks while all this is running in the background, just run everything under the if __name__ == "__main__": guard in a separate thread, and I'd suggest adding a try..except block to pick up KeyboardInterrupt and exit gracefully.

But... soon enough you'll notice that it doesn't exactly work properly all the time - if it takes longer than half a second of wait for rasa to print its STDOUT and enter the wait for STDIN stage, the outputs will start to mix. This problem is considerably more complex than you might expect. The main issue is that STDOUT and STDIN (and STDERR) are separate buffers and you cannot know when a subprocess is actually expecting something on its STDIN. This means that without a clear indication from the subprocess (like you have the [path]> in Windows CMD prompt on its STDOUT for example) you can only send data to the subprocesses STDIN and hope it will be picked up.

Based on your screenshot, it doesn't really give a distinguishable STDIN request prompt because the first prompt is ... : and then it waits for STDIN, but then once the command is sent it lists options without an indication of its end of STDOUT stream (technically making the prompt just ... but that would match any line preceding it as well). Maybe you can be clever and read the STDOUT line by line, then on each new line measure how much time has passed since the sub-process wrote to it and once a threshold of inactivity is reached assume that rasa expects input and prompt the user for it. Something like:

import subprocess
import sys
import threading

# we'll be using a separate thread and a timed event to request the user input
def timed_user_input(timer, wait, buffer_in, buffer_out, buffer_target):
    while True:  # user input loop
        timer.wait(wait)  # wait for the specified time...
        if not timer.is_set():  # if the timer was not stopped/restarted...
            print("Input: ", end="", file=buffer_out, flush=True)  # print the input prompt
            print(buffer_in.readline(), file=buffer_target, flush=True)  # forward the input
        timer.clear()  # reset the 'timer' event

if __name__ == "__main__":  # a guard from unintended usage
    input_buffer = sys.stdin  # a buffer to get the user input from
    output_buffer = sys.stdout  # a buffer to write rasa's output to
    proc = subprocess.Popen(["path/to/rasa", "arg1", "arg2", "etc."],  # start the process
                            stdin=subprocess.PIPE,  # pipe its STDIN so we can write to it
                            stdout=subprocess.PIPE,  # pipe its STDIN so we can process it
                            universal_newlines=True)
    # lets build a timer which will fire off if we don't reset it
    timer = threading.Event()  # a simple Event timer
    input_thread = threading.Thread(target=timed_user_input,
                                    args=(timer,  # pass the timer
                                          1.0,  # prompt after one second
                                          input_buffer, output_buffer, proc.stdin))
    input_thread.daemon = True  # no need to keep the input thread blocking...
    input_thread.start()  # start the timer thread
    # now we'll read the `rasa` STDOUT line by line, forward it to output_buffer and reset
    # the timer each time a new line is encountered
    for line in proc.stdout:
        output_buffer.write(line)  # forward the STDOUT line
        output_buffer.flush()  # flush the output buffer
        timer.set()  # reset the timer

You can use a similar technique to check for more complex 'expected user input' patterns. There is a whole module called pexpect designed to deal with this type of tasks and I wholeheartedly recommend it if you're willing to give up some flexibility.

Now... all this being said, you are aware that Rasa is built in Python, installs as a Python module and has a Python API, right? Since you're already using Python why would you call it as a subprocess and deal with all this STDOUT/STDIN shenanigans when you can directly run it from your Python code? Just import it and interact with it directly, they even have a very simple example that does exactly what you're trying to do: Rasa Core with minimal Python.