TransWikia.com

what is correct way to perform normalization on data in Auto encoder?

Data Science Asked by Milan_Harkhani on February 1, 2021

working on anomaly detection problem. i’m using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of input is essential for my dataset.

problem with normalization is that when noise value is very high compare to entire dataset. then prediction follows noise. for example if I add noise (delta=300) to 80% of the data and perform normalization on the dataset which mean value is 250 and standard deviation is 79. here noisy data points(80% of the total dataset) are greater than 300. after normalization, I feed this dataset to the model, prediction follows the noise and giving wrong output. this happens because of feature scaling. when I add noise to most of data points, model consider this points as normal data points and rest of as anomalous data points.

In inverse scaling process, I can not use min-max values of my input to perform inverse scaling on prediction otherwise its follow noise in dataset.

so what is right way to perform feature scaling in denoise kind of problem?

One Answer

Min-Max-Scaling will not perform well for your problem, as you already said. For noisy data, scaling according to the quantile range should perform better.

After scaling, you could also try to clip your data to $[-1,1]$, as it it is often done in adversarial learning methods.

Answered by Andreas Look on February 1, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP