In math mode, how do I make the fraction bar "invisible"?

The frac macro is defined as begingroup #1endgroup over #2 (for the definition enter texdef -t latex frac in a terminal). You can redefine this to print the over line in white:

newcommand{fracnoline}[2]{{begingroup #1endgroup color{white}overcolor{black} #2}}

As Mico noted in comments, to generalize the macro in case the original color is not black you can use the following definition:

newcommand{fracnoline}[2]{colorlet{foo}{.}{begingroup #1endgroup color{white}overcolor{foo} #2}}

Full code:

documentclass[oneside,11pt]{book}

usepackage[semibold,tt=false]{libertine}
usepackage{libertinust1math}
usepackage[T1]{fontenc}
usepackage[
  expansion = false ,
  tracking = smallcaps ,
  letterspace = 40 ,
  final
]{microtype}
usepackage{booktabs}
usepackage[font={sf,small},labelsep=quad,labelfont=sc]{caption}
usepackage[subrefformat=parens]{subcaption}
usepackage{xcolor}

begin{document}
newcommand{fracnoline}[2]{colorlet{foo}{.}{begingroup #1endgroup color{white}overcolor{foo} #2}}

$frac{1}{2};genfrac{}{}{0pt}{}{1}{2}$

begin{table}[!h]
  begin{tabular}{lll}
    Row 1 & 1.23 & Some text.
    Row 2 & 7.89 & Additional text.
    Row 3 & 4.56 & More text.
    addlinespace
    Row 4 & 1.23 & Some text.
    Row 5 & $frac{textit{A}:;3.14}{textit{B}:;6.28} fracnoline{textit{A}:;3.14}{textit{B}:;6.28}$ & Additional text.
    Row 6 & 4.56 & More text.
  end{tabular}
end{table}

end{document}

Result:

With amsmath you can use the substack command:

documentclass[oneside,11pt]{book}
usepackage{amsmath}
usepackage{booktabs}

begin{document}
 
  begin{table}[!h]
   begin{tabular}{lll}
    Row 1 & 1.23 & Some text.
    Row 2 & 7.89 & Additional text.
    Row 3 & 4.56 & More text.
    addlinespace
    Row 4 & 1.23 & Some text.
    Row 5 & $ substack{textit{A}:;3.14[1pt]textit{B}:;6.28}$ & Additional text.
    Row 6 & 4.56 & More text.
  end{tabular}
 end{table}
 
end{document}

That's a bit unexpected, I'll have to read appendix G again... but genfrac is just exposing the underlying primitive and atop and above0pt both omit the rule but increase the height of the numerator. You can specify the negative width of the default rule (-0.4pt) to get the same height as frac (which uses the over primitive).

The first four forms here use the same height asfrac, the last three are higher.

Note this only applies to textstyle not displaystyle.

documentclass[oneside,10pt]{book}
usepackage{amsmath}

usepackage[T1]{fontenc}

begin{document}

hrule
$
frac{1}{2};
genfrac{}{}{-.4pt}{}{1}{2};
{1 above -.4pt 2};
{1over 2};
{1 above 0pt 2};
genfrac{}{}{0pt}{}{1}{2};
{1atop 2}
$
hrule
smallskip
hrule
$displaystyle
frac{1}{2};
genfrac{}{}{-.4pt}{}{1}{2};
{1 above -.4pt 2};
{1over 2};
{1 above 0pt 2};
genfrac{}{}{0pt}{}{1}{2};
{1atop 2}
$
hrule


end{document}

Starting from David's answer, a fixed version that works in display, text and script style (not at scriptscript style, where other optimizations are used).

documentclass[oneside,10pt]{book}
usepackage{amsmath}
usepackage[semibold,tt=false]{libertine}
usepackage{libertinust1math}
usepackage[T1]{fontenc}

makeatletter
DeclareRobustCommand{xabove}[2]{{%
  mathpalettefix@genfrac{{#1}{#2}}%
}}
newcommand{fix@genfrac}[2]{fix@@genfrac#1#2}
newcommand{fix@@genfrac}[3]{%
  genfrac{}{}{fix@@@genfrac{#1}}{}{#2}{#3}%
}
newcommand{fix@@@genfrac}[1]{%
  ifx#1displaystyle 0ptelse
  ifx#1textstyle -fontdimen8textfont2 else
  ifx#1scriptstyle -fontdimen8scriptfont2 else
  0pt % but doesn't really work :-(
  fififi
}
makeatother

begin{document}

$displaystylefrac{1}{2}xabove{1}{2}$

$textstylefrac{1}{2}xabove{1}{2}$

$scriptstylefrac{1}{2}xabove{1}{2}$

%$scriptscriptstylefrac{1}{2}xabove{1}{2}$

end{document}