multithreading - Python TCP Server Accepting Connections and Broadcasting Commands

Question

I've been working on a game using a number of Raspberry Pis, Python, and some buttons/switches. My game requires a central server that issues commands to multiple clients.

I'm not new to programming but new to Python and lower-er level network communication and have been lost in the weeds for the past 2 days on how exactly to write my server code.

The client program is a simple socket.connect and then waits for data to be sent. No problems there.

I have had a tough time determining exactly how to write and how to make work the server.

Here's what my server code looks like at the moment:

import socket, time, sys
import threading

TCP_IP = ''
TCP_PORT = 8888
BUFFER_SIZE = 1024
CLIENTS = {}
clientCount = 0

def listener():
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((TCP_IP,TCP_PORT))
    s.listen(5)
    while True:
        conn, addr = s.accept()
        print("new connection from:"+ str(addr))
        #print(len(CLIENTS))
        global clientCount
        clientCount = clientCount+1
        print (clientCount)
        # register client
        CLIENTS[conn.fileno()] = conn


def broadcast():
     for client in CLIENTS.values():
            client.send('this is a broadcats msg')

if __name__ == '__main__':
    listener()

    while clientCount > 0:
        broadcast()
        print(len(CLIENTS)) #print out the number of connected clients every 5s
        time.sleep(5) 

Here's the desired flow: 1. Server starts and waits for first connection or more. I believe this "Server" should be running on a background thread? 2. If connectionCount > 0 start the main program loop 3. For now the main program loop should just display the number of connected clients and broadcast a message to all of them every 5 seconds.

I have about 5 versions of this server. I have tried async, select.select, and several threaded approaches but can't quite nail down the behavior I seek. Should I be putting the server in a background thread? If so how to broadcast to all the connections?

The only thing I haven't tried is Twisted and that is because I could not get it installed in Windows... so I'm ruling that optino out for the moment. If anyone has a pointer on where to go on this I would really appreciate it!

Update

Ok, based on @Armans suggestion I have updated my code so that there is a server class but it still performs the same.

class server():

    def __init__(self):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.bind((TCP_IP,TCP_PORT))
        s.listen(10)
        while 1:
            client_socket, addr = s.accept()
            print ('Connected with ' + addr[0] + ':' + str(addr[1]))
            global clientCount
            clientCount = clientCount+1
            print (clientCount)
            # register client
            CLIENTS[client_socket.fileno()] = client_socket
            threading.Thread(target=self.handler, args=(client_socket, addr)).start() 



    def handler(self, client_socket, addr):
        while 1:
            data = client_socket.recv(BUFFER_SIZE)
            print ('Data : ' + repr(data) + "
")
            data = data.decode("UTF-8")


    def broadcast(self, message):
        for c in self.CLIENTS:
            c.send(message.encode("utf-8"))

if __name__ == '__main__':
    s = server() #create new server listening for connections

    while clientCount > 0:
        s.broadcast('msg here')
        print(len(CLIENTS)) #print out the number of connected clients every 5s
        time.sleep(5) 

I can connect multiple clients and the console displays the following:

Connected with 10.0.0.194:38406
1
Connected with 10.0.0.169:36460
2

But the code in the "while clientCount" loop never runs. This is the zone where I have been stuck for some time so if you have a couple more ideas I would love to here any ideas!

Answer 1

Finally got it working! Much thanks to @Arman for pointing me in the right direction with the threading. I finally feel like I understand how everything is working!

Here is my complete Server & Client code. Hopefully this helps someone else with a master > client setup. The _broadcast() function is working as you will see it just broadcasts a static msg at the moment but that should be an easy update.

If anyone has any advice on code cleanup, python best practices using this code as the sample I would love to hear and learn more. Thanks again SE!

##Client

import socket
import sys
import json

#vars
connected = False

#connect to server
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(('10.0.0.158',8888))
connected = True

while connected == True:
    #wait for server commands to do things, now we will just display things
    data = client_socket.recv(1024)     
    cmd = json.loads(data) #we now only expect json    
    if(cmd['type'] == 'bet'):
        bet = cmd['value']
        print('betting is: '+bet)
    elif (cmd['type'] == 'result'):        
        print('winner is: '+str(cmd['winner']))
        print('payout is: '+str(cmd['payout']))


##Server

import socket, time, sys
import threading
import pprint

TCP_IP = ''
TCP_PORT = 8888
BUFFER_SIZE = 1024

clientCount = 0

class server():

    def __init__(self):
        self.CLIENTS = []        


    def startServer(self):
        try:
            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            s.bind((TCP_IP,TCP_PORT))
            s.listen(10)
            while 1:
                client_socket, addr = s.accept()
                print ('Connected with ' + addr[0] + ':' + str(addr[1]))
                global clientCount
                clientCount = clientCount+1
                print (clientCount)
                # register client
                self.CLIENTS.append(client_socket)
                threading.Thread(target=self.playerHandler, args=(client_socket,)).start()
            s.close()
        except socket.error as msg:
            print ('Could Not Start Server Thread. Error Code : ') #+ str(msg[0]) + ' Message ' + msg[1]
            sys.exit()


   #client handler :one of these loops is running for each thread/player   
    def playerHandler(self, client_socket):
        #send welcome msg to new client
        client_socket.send(bytes('{"type": "bet","value": "1"}', 'UTF-8'))
        while 1:
            data = client_socket.recv(BUFFER_SIZE)
            if not data: 
                break
            #print ('Data : ' + repr(data) + "
")
            #data = data.decode("UTF-8")
            # broadcast
            for client in self.CLIENTS.values():
                client.send(data)

         # the connection is closed: unregister
        self.CLIENTS.remove(client_socket)
        #client_socket.close() #do we close the socket when the program ends? or for ea client thead?

    def broadcast(self, message):

        for c in self.CLIENTS:
            c.send(message.encode("utf-8"))

    def _broadcast(self):        
        for sock in self.CLIENTS:           
            try :
                self._send(sock)
            except socket.error:                
                sock.close()  # closing the socket connection
                self.CLIENTS.remove(sock)  # removing the socket from the active connections list

    def _send(self, sock):        
        # Packs the message with 4 leading bytes representing the message length
        #msg = struct.pack('>I', len(msg)) + msg
        # Sends the packed message
        sock.send(bytes('{"type": "bet","value": "1"}', 'UTF-8'))


if __name__ == '__main__':
    s = server() #create new server listening for connections
    threading.Thread(target=s.startServer).start()

    while 1:       
        s._broadcast()
        pprint.pprint(s.CLIENTS)
        print(len(s.CLIENTS)) #print out the number of connected clients every 5s
        time.sleep(5)