python - How to disable stdout buffer when running shell

Question

I am using Python to call a Shell script with

def run_command(cmd):
    print "Start to run: " + cmd
    run = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    while True:
        line = run.stdout.readline().decode()[:-1]
        if line == '' and run.poll() is not None:
            break
        print line # print the log from shell
    recode = run.returncode
    if recode != 0:
        raise Exception("Error occurs!")

    print "End to run: " + cmd

Then I run

run_command("sh /home/tome/a.sh")

I notice the console output from a.sh is not in real time, looks like that there is a buffer for the stdout and when the stdout buffer is full, then the output is printed out.

I would ask how to disable the shell stdout buffer in my script a.sh

Thanks!

Answer 1

The buffering in question would largely be a problem on the script's side, not the Python side; while Python would buffer the reads, it wouldn't block unless the buffer was emptied and there was nothing available to read.

So really, you need to disable buffering in the script itself. Adding stdbuf -oL (or -o0 for completely unbuffered, but line buffering should cover you since you read by line as well) to your commands should help in some cases (where the programs don't adjust their own buffering internally).

If you're seeing this behavior only by looking at Python's output, be aware that Python itself can buffer output as well. You can disable this by passing -u when running Python, or setting the environment variable PYTHONUNBUFFERED=1 before running it, or from within a script, you can manually call sys.stdout.flush() after any writes (direct, or implicit via print) to stdout. On modern Python, print takes an argument to force a flush after printing, but since you're on Python 2.x, that's not an option.