I think it might add some value for others facing similar issues in the future to elaborate a bit more on Rodney's answer from a hardware side.
Although I came here looking for the answer to the thread question (why was I getting an ioctl error trying to activate the RS485 mode on pyserial (which I was actually calling from pyModbus) I found the answer to that question above. Plain and simple, and to provide a shorter answer than Rodney's: you can't. You won't be able to activate the RS485 mode on your FTDI because there is no function to use the RTS signal as a drive enable for your RS485 chip.
Some devices, like those using the atmel_serial driver do implement this functionality, but you won't find that many. Digging into this issue you'll find people saying you can write your own driver based on one of those that support the RS485 mode (see, for instance: automatically changing RTS for RS-485 communication, but in my view they are wrong)
Now that you have the bad news, you can ask yourself why would you need to use the RS485 mode? I imagine that, like me, you were trying to establish a half-duplex link between two devices using a two wire RS485 setup (which is surprisingly frequent in many industrial devices where you have a master-slave configuration, where you have one side sending a request for data and the other side duly responding, as you do with Modbus).
If you are looking for a software solution you can still take the RTS signal high and low manually before starting and after finishing transmitting from each side. This solution will work for testing and just for fun in most environments (at least those I tried, it even works with Python on a Raspberry Pi using a GPIO line to control the Drive Enable/~Read Enable signals) but I would not use it if you want to get good reliability. With any non-real time OS there is no warranty you will get the correct timing and some devices are quite picky and will report a timeout error. If you are free to choose both sides (or all of them if you have more than two stations) of the bus you can always increase your tiemout everywhere and hope for the best.
Good news is, as Rodney said, on the FTDI chip you have the TXDEN signal available on pin 13 so you can use that right away. Unfortunately, most cheap USB to serial
adaptor PCBs don't have that signal easily accessible on the board. That's what happened to my Sparkfun board. If you are not up for the challenge of attaching a cable directly to the SMD IC you can do what I did: download mprog 3.5 from FTDI, connect to the EPROM, under I/O Controls select TXDEN instead of TXLED and click save to write the updated firmware to the chip. Then you can easily attach a cable to the LED pad and use it as your Drive Enable/~Read Enable signal (see pictures, the scope snapshot shows it works nicely).
Modified FTDI board with TXDEN signal on TXLED
Scope capture of the modified circuit: blue is TX, yellow TXDEN
Lastly, if you are unlucky enough to run into other chips that don't have this option (TXDEN), like the PL2303 you can always go old school and build a hardware TXDEN with a 555 timer (see here: http://www.embeddedsys.com/subpages/resources/images/documents/microsys_art_RS485.pdf). I tested this circuit and it works fine, at least at 9600 bps (note that I had to use a 39K resistor instead of a 3.9K, I think that's a typo but I might be wrong, I did not go into fine details, I just needed something quick and dirty to pair with my FTDI.