Text vectorizer that capture feature offset in the text?

Data Science Asked by R Sorek on October 24, 2020

I’m using sklearn Tfifdfvectorizer to extract feature from text towards text classification.
I believe the information I need tends to be in the beginning of the document, so I would like to somehow capture the offset of each feature per document (either of the first appearance, or the mean offset over of all appearances).
Is there some vectorizer that can do that? or some other method of extracting this information efficiently?

Thank you!

feature extraction scikit learn text text classification tfidf

One Answer

One approach is to create another matrix that stores this information. Scikit-learn stores text features in a document-by-token matrix. The cells of this matrix would be the token index in a document. This matrix then could be used as features during modeling.

It would require writing a custom vectorizer which would be similar to scikit-learn's CountVector implementation.

Answered by Brian Spiering on October 24, 2020

Text vectorizer that capture feature offset in the text?

One Answer

Add your own answers!

Related Questions

Clustering time series based on monotonic similarity

How do we decide between XGBoost, RandomForest and Decision tree?

Magnification factor in image classification

What is the reason behind Keras choice of default (recurrent) activation functions in LSTM networks

What does the classification report interpret? Class 1 indicates abnormal data

Impact of a new word on word embedding vectors

How do I generate a laplacian matrix for a graph dataset?

Stratified Sampling for XGboost

xgboost in R have different results compared to boosted decision tree in Azure ML

k-means for customer review analysis

Algorithm suggestion for anomaly detection in multivariate time series data

choosing model based on last or best iteration on validation set

Dose finding slope/intercept using the formula of m,b gives best fit line always In linear regression?

Orange wont allow me to set target for corpus

Keras P/R metrics at different thresholds during training

Storing and collecting data

Effect of skewness in data

Automation of finding a starting point of measurement in a large dataset

How to manage missing data in meteorological time series?

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Ask a Question