The #Hayes #Smartmodem had a separate data mode (in which bytes are sent/received over the phone line) and command mode (in which the computer sent commands to the modem itself).
Switching from command to data mode was simple: send the command `ATO`.
The reverse was more complicated. The modem would switch from data mode to command mode whenever the computer tried to send the special character sequence `+++`, with a delay before and after the sequence.
1/
This was patent encumbered and also a DoS vulnerability: tricking anyone into transmitting that sequence would knock them offline!
There must have been a better solution than this.
One possibility comes to mind: switch modes when the computer briefly toggles the DTR signal. https://en.wikipedia.org/wiki/Data_Terminal_Ready
2/
Unless I'm mistaken, DTR is kept active at all times while the computer is using the modem. Its only function is that the modem is to hang up if DTR becomes inactive for 2 seconds.
So, if DTR becomes inactive for only 0.5 seconds and then becomes active again, what happens then? If the answer is nothing, then that could be interpreted as a signal to switch the modem to command mode.
Nobody can trick you into toggling DTR, so that should be fine, no?
3/end
'Course, I'm about 45 years too late to be having this thought, but… 
Another possibility comes to mind: use a single-byte escape code instead (ASCII's Data Link Escape code 16 seems appropriate), and add a modem command that sends that literal byte and goes back to data mode.
So, for example, whenever you need to transmit `␐`, you'd instead transmit `␐ATY␍` (for a hypothetical command `ATY` that sends a literal ␐ and switches back to online mode) and then continue transmitting as normal.
That would fail: binary file transfers toss all 256 combos at the port. Compression assured the spread.
In practice, delay +++ delay worked perfectly. I don't think it happened even once to me or anyone I talked to (very large number).
Also the worst that could happen was disconnect; the host could talk to the modem but from the line side, nada.
Modem control and recovery was a deepish art in the FidoNet days. They weren't that determinate when things Went Wrong. And there were like 20 or 50 different modem types in use. Some were just awful. Some were wildly complex. Some could not be used as a BBS incoming.
Getting serial ports with consistent control lines as per anything like rs232 was awful. The damn apple machine had zero handshake lines. Things were wildly inconsistent then.
Hmm yeah, the Wikipedia article about the Smartmodem mentions that problem. “However, while the 25-pin connector on the modem side had more than enough pins for this purpose (even some meant for this purpose), the computer side often had far fewer pins connected and controllable, if it even used a full 25-pin connector at all.”
Using a single-byte control code to switch modes would only work if one of the following is true:
1. The only thing being transmitted is uncompressed text.
2. There is a software layer between the application and the modem, which inserts the aforementioned ATY sequence whenever the application sends the mode-switching byte.
3. The application speaks a protocol like PPP that can avoid ever transmitting the mode-switching byte.
Sadly, although PPP would have been able to avoid sending a DLE just fine, it *cannot* avoid sending the Hayes +++ sequence.
That is, it can avoid sending any byte you want, substituting any instance of such bytes with a sequence of other bytes…except 5E or 20 through 3F.
+ is 2B. 
But wait, if Hayes had gone with the aforementioned DLE/ATY scheme, then wouldn't file transfer programs and the like have known to insert the ATY sequence after every DLE? Only dumb terminal programs wouldn't, and there's no keyboard key that sends a DLE.
The Hayes break-in sequence is ONE SECOND of silence, +++, ONE SECOND of silence, then the modem will respond to an AT command. It's pretty specific and obscure.
Even if it did occur, assuming it's an accident, the modem link would fail, though I think stay connected (*), any data sent by the host intended for the other end would be interpreted as hopefully garbage, and modem control would eventually notice, disconnect, (lol), one sec +++ one sec, disconnect, clear and reconfig modem, redial, connect.
(*) The disconnect sequence is the +++ thing, an empty AT to ensure echo works, then ATH0. +++ breaks the data transmission but doesn't drop the modem to modem connection. AT H0 disconnects.
There are two things in the world in my life so far in which I knew/know more than nearly anyone alive: Nash and Rambler cars based on the 01 series chassis, between 1958 and 1963, and how to use modems. I made FidoNet work with pretty much every modem made between 1984 and early 1990s, and made them reliable. Modem manufacturers would send me new modem products so that I could make them work one BBSs.
Those two skills, plus five bucks, gets me a coffee at Starbucks.
Then you must know that a lot of modems didn't require the one-second pause (because it was patent encumbered and licenses weren't cheap). Those that did not could be trivially knocked offline with an ICMP echo request or similar.
Worse, even if software (e.g. pppd) knew about this issue, it couldn't actually do anything about it, because it didn't know whether or not +++ without pauses will be reacted to by this particular modem.
And anyway, what if you really do need to send +++ with pauses over the wire? Unless I'm mistaken, that's impossible.
Pretty much true.
I can tell you those modems were not successful on BBSs!
System operators, BBS or *nix, simply did not use those toy home modems.
Ah. Well, that's one way to solve the problem.
I'm guessing this wasn't the only thing wrong with those cheap modems, though. Did anything else make you decide against using them?
It's far too long ago to recall anything.