Designing appropriate filter by choosing values for $A$ and $B$

This is meant as a stepping stone up to Dan's answer.

The units for frequency at the sample level are radians per sample. You've got:

$$ 400 frac{cycles}{second} cdot 2 pi frac{radians}{cycle} / 8000 frac{samples}{second} = frac{pi}{10} frac{radians}{sample} $$

That is your target $omega_t$.

Sound is a real valued signal, so you can model it like this:

$$ s[n] = A cos( omega n + phi ) = A frac{ e^{i(omega n + phi)}+e^{-i(omega n + phi)} }{2} $$

Those are your two complex tones you want to zero out. Notice the the units work out as well.

$$ omega frac{radians}{sample} cdot n; samples = omega n ; radians $$

Starting with:

$$ y[n] - ABy[n-1] +A^2y[n-2] = x[n] - Bx[n-1] +x[n-2] $$

Suppose that: $$ begin{aligned} y[n] &= A_y e^{i(omega n + phi_y) } \ x[n] &= A_x e^{i(omega n + phi_x) } \ end{aligned} $$

It follows that: $$ begin{aligned} y[n+d] &= A_y e^{i(omega (n + d) + phi_y) } = A_y e^{i(omega n + phi_y }e^{i(omega d) } = y[n]e^{iomega d } \ x[n+d] &= A_x e^{i(omega (n + d) + phi_x) } = A_x e^{i(omega n + phi_x }e^{i(omega d) } = x[n]e^{iomega d } \ end{aligned} $$

Plug those into your equation:

$$ begin{aligned} y[n] - ABy[n-1] +A^2y[n-2] &= x[n] - Bx[n-1] +x[n-2] \ y[n] left[ 1 - AB e^{iomega } + A^2 e^{i2omega } right] &= x[n] left[ 1 - B e^{iomega} + e^{i2omega} right] \ end{aligned} $$

For convenience, make a substitution:

$$ z = e^{iomega} $$

$$ begin{aligned} ( 1 - AB z + A^2 z^2 ) cdot y[n] &= ( 1 - B z + z^2 ) cdot x[n] end{aligned} $$

Your $z_1$ and $z_2$ represent the two tones you want to zero. By putting the polynomial for $x[n]$ in factor form, your target values can be plugged right in.

$$ begin{aligned} 0 = ( 1 - B z + z^2 ) = ( z - z_1 )(z - z_2 ) = z^2 - (z_1 + z_2)z + z_1 z_2 end{aligned} $$

$$ begin{aligned} z_1 &= e^{iomega_t} \ z_2 &= e^{-iomega_t} end{aligned} $$

Observe:

$$ z_1 z_2 = e^{iomega_t} e^{-iomega_t} = e^0 = 1 $$

Which fits in the equation without rescaling, therefore

$$ B = z_1 + z_2 = e^{iomega_t} + e^{-iomega_t} = 2 cos(omega_t) $$

Now, continue with Dan's answer.

Bottom Line:

$$A <1$$

$$B =2cos(omega_n)$$

Where $omega_n$ is the normalized angular frequency of the desired notch location (in this case for the OP with a sampling rate of 8KHz and notch at 400 Hz this would be $omega_n = 2pi400/8000 = pi/10$), resulting in $B approx 1.902$ and $A$ is the frequency notch bandwidth parameter; the closer $A$ is to 1 the tighter the bandwidth of the notch.

Details:

The straight-forward approach to eliminating narrow band noise at 400 Hz would be a classic 2nd order notch filter. This is detailed at this post:

Transfer function of second order notch filter

This approach is done by simply placing a zero at the frequency of interest, and by adding a pole close to the zero, but inside the unit circle of course to be stable, we can adjust the bandwidth of the notch by the proximity of this pole to that zero, meaning $|p|<1$, but close to 1.

Using real coefficients, meaning complex conjugate zeros and poles as detailed in the linked post, this results in the transfer function:

$$ H(z) = frac{1+a}{2}frac{z^2-2zcos(omega_n)+1}{(z^2-2azcos(omega_n)+a^2)} $$

Which as a difference equation is exactly in the form that the OP has used and therefore given this is the same filter, answers the OP's question. This is is derived from the z-transform as follows (we can ignore the gain scaling parameter $(1+a)/2$ which for most cases is close to 1, and express in decreasing powers of z (and assuming I didn't make simple algebra errors!):

$$ frac{Y(z)}{X(z)} = frac{1-2z^{-1}cos(omega_n)+z^{-2}}{(1-2az^{-1}cos(omega_n)+a^2z^{-2})} $$

$$ Y(z) (1-2az^{-1}cos(omega_n)+a^2z^{-2})= X(z)(1-2z^{-1}cos(omega_n)+z^{-2})$$

$$Y(z)- Y(z)2acos(omega_n)z^{-1}+Y(z)a^2z^{-2} = X(z)-X(z)2cos(omega_n) z^{-1} +X(z)z^{-2} $$

And then with the inverse Z transform we get the desired coefficients that the OP wanted:

$$y[n] - 2acos(omega_n)y[n-1] + a^2y[n-2] = x[n]-2cos(omega_n)x[n-1] + x[n-2] $$

With comparison to OP's expression:

$$y[n] - ABy[n-1] +A^2y[n-2] = x[n] - Bx[n-1] +x[n-2]$$

Results in the following:

$$A = a$$

$$B =2cos(omega_n)$$

Where $omega_n$ is the normalized angular frequency of the desired notch (for the OP's sampling rate and notch frequency this would be $omega_n = 2pi400/8000 = pi/10$) and $a$ is the notch bandwidth parameter ($a<1$, the closer a is to 1 the tighter the notch, refer to the linked post for complete details).