Was there a backup plan in case the Shuttle toilet malfunctioned?

First let's review the nominal operation of the Shuttle Waste Collection System (WCS).

(from Shuttle Crew Operations Manual (SCOM) page 2.25-2)

It was divided into two major systems, one for urine and one for feces.

• The urinal consisted of a hose with a crewperson's personal funnel attached at the business end and fan/separator units at the other. The fan automatically came on when the crewperson removed the hose end from its cradle. The urine passed down the hose driven by air flow from the fan, then was separated from the air by the separator part of the fan/separator, and stored in the waste tank. This tank was basically identical to the potable water storage tanks. It was pressurized using a bellows by nitrogen from the Pressure Control System. Water condensed from the cabin air by the humidity control system was also stored in this tank. When the waste tank got full enough, it was dumped overboard.
• The commode consisted of a seat with a chamber below it for storing the waste. When not in use, the chamber was connected to the vacuum vent system and held at vacuum in order to dry out the waste. The commode was used by operating a handle which controlled a manual linkage that closed off the vacuum system, repressurized the chamber, and opened the aperture at the top below the crewperson. When the commode control handle was placed in the forward position for use, the fan/separator came on to provide air flow to guide the waste into the chamber. After usage, returning the handle to the back position returned the system to the drying out configuration. Every so often "compactor ops" had to be done to allow adequate space in the chamber for continued usage.

The procedures for WCS use are found on the WCS cue card in the Orbit Ops Checklist, reproduced below.

Off nominal operations:

As shown on the cue card,

For a non-operational WCS, contingency waste collection devices are available. Fecal collection is accomplished by using an Apollo fecal bag. Apollo fecal bags can be stowed inside the WCS commode or in the wet trash (volume F) compartment. For men, the urine collection device (UCD) is used for urine collection. A UCD consists of a collection bag, a restraint assembly, and a roll-on cuff. For women, urine collection is accomplished via Absorbent Pants (Pull-Ups). The Pull-ups are a modified version of adult diapers.

(SCOM p. 2.25-6)

Some non-catastrophic failures of the WCS could be worked around.

• The vacuum vent system was critical to proper operation. If the vaccum vent system clogged up, this could be managed by cross-connecting the commode's vacuum vent system to the waste water overboard dump line, and opening the dump valve when the commode was in use.
• If the waste water overboard dump system failed, the waste water could be dumped through the potable water overboard dump system using a contingency interconnect hose. If that too failed, it could be collected in flexible Contingency Water Container bags and returned to the ground.
• Mechanical failures of the commode control handle or linkage could be worked around using procedures in the In-flight Maintenance Checklist.
• Finally, if both fan/separators were failed, urinal operations could be recovered by connecting the urinal hose directly to the waste water overboard dump system. However, this resulted in a direct connection from the crewmember to vacuum, and the procedure contained this immortal warning.

(From In Flight Maintenance Checklist section W, not currently online)