As far as I know, there is no control at server side for transport type preference. Server should be made generic it should support RTP over UDP, RTP over TCP, RTP over RTSP and RTP over RTSP over HTTP(S). And its clients choice which transport to choose. Transport field is first sent in SETUP request
1) UDP
C->A: SETUP rtsp://audio.example.com/twister/audio.en RTSP/1.0
CSeq: 1
Transport: RTP/AVP/UDP;unicast;client_port=3056-3057
2) TCP
C->A: SETUP rtsp://audio.example.com/twister/audio.en RTSP/1.0
CSeq: 1
Transport: RTP/AVP/TCP;unicast;client_port=3056-3057
3) RTP over RTSP and RTP over RTSP over HTTP(S)
S->C: RTSP/1.0 200 OK
CSeq: 2
Date: 05 Jun 1997 18:57:18 GMT
Transport: RTP/AVP/TCP;interleaved=0-1
As we can see "Transport type" request is sent by client side.
If you want to support TCP only server you can send "400 Bad Request" or "461 Unsupported transport" in response to SETUP request as suggested by you or another way is to send 200 OK but do not transmit any RTP packets. Client will timeout and get to know that it is behind proxy and it will send SETUP request again with RTP/AVP/TCP parameter (Not an ideal case).
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…