Blind Estimation of Signal Parameter and Noise Variance

Squaring Method

The input signal $x[n]$ can take two values: $+1$ or $-1$. After being multiplied by $h$, the signal becomes $|h|e^{jangle h}$ (if $x[n]=+1$) or $|h|e^{j(pi + angle h)}$ (if $x[n]=-1$). By squaring this we get, $|h|^2e^{j2angle h}$. Both points get mapped to this. Now we can pick out the magnitude and phase of $h$. In summary, the steps are:

1. Square $y[n]$ to get $z[n]=y^2[n]$.
2. Get the magnitude, $hat{|h|}=sqrt{bigg|frac{1}{N}sum_n z[n]bigg|}$. It was pointed out in the comments that this is a biased estimate with bias $sigma^2$, so it is not favorable for a large $sigma$. I describe another way to estimate $sigma^2$ (without using $hat{h}$) so that we can use this $hat{sigma}^2$ to remove the bias from $hat{|h|}$.
3. Get the phase, $angle h=frac{1}{2N} sum_n angle z[n]$
4. Use $hat{h}$ to get the noise variance, $hat{sigma}^2=bigg(frac{1}{N}sum_n z[n]bigg)-hat{h}^2$. This comes from the fact that $Ebig[z[n]big]=h^2+sigma^2$ (also see the clustering method is a better way to go about this as it does not depend on any other estimates).

Caveat: This method is valid only for phase offsets within $pm frac{pi}{2}$. Anything beyond that, the received signal becomes indistinguishable and there are multiple phase shifts which could give the same signal.

Clustering Method

If $h$, can be correctly determined, then you can take hard decisions and calculate the noise variance using the distance of the received symbols from the decision you made.

If $h$ can't be determined right away, then this clustering based estimate of $|h|$ and $sigma^2$ can be used: We know there should be two clusters ($pm 1$) so we can use a simple method like k-means (https://en.wikipedia.org/wiki/K-means_clustering) with $k=2$. The algorithm is:

1. Run k-means on the received samples and get a list of labels back labeling each sample to either cluster $1$ or $2$. Let $S_1$ be the set of sample indices in cluster $1$ and $S_2$ be the set of sample indices in cluster $2$.

2. Now we are going to take all the points in each cluster, shift them be zero mean and take the variance (this will give us a noise variance estimate for each cluster). Then we will average over all of the cluster noise variance estimates to form the final estimate. $hat{sigma}^2=frac{bigg( text{var}big(y[n]-E[y[n]] big)big|_{n in S_1} + text{var}big(y[n]-E[y[n]] big)big|_{n in S_2} bigg)}{2}$

3. Now we can take the estimate from step $2$ and subtract off the bias term which we estimated here as $hat{sigma}^2$ to get the refined estimate $hat{|h|}_0=hat{|h|}-hat{sigma}^2$. You could also get $|h|$ using the magnitude of the cluster means, $|hat{h}|=frac{text{E}big(y[n]big|_{n in S_1}big)+text{E}big(y[n]big|_{n in S_2}big)}{2}$.

There is also a way get $angle h$ from the clusters but it too has the criteria that $angle h$ must be less than $pm frac{pi}{2}$ so I will not detail it here.

Long story short: $|h|$ and $sigma^2$ are doable, but $angle h$ is only doable for some cases.

Code: https://github.com/B-William/DSPSE/blob/master/blindEstimationOfSignalParameterAndNoiseVariance.m