TransWikia.com

Pruning input layers and output layers after training model with shared layers

Data Science Asked by Howard Wang on July 31, 2020

I’m trying to train a search and item encoder and this is the model I have

input_search = Input(shape=(40,), dtype='int64', name='input_search')
input_title = Input(shape=(40,), dtype='int64', name='input_title')
input_desc = Input(shape=(40,), dtype='int64', name='input_desc')
input_brand = Input(shape=(40,), dtype='int64', name='input_brand')

embedding = Embedding(input_dim=20000, output_dim=50, input_length=40)
s_emb = embedding(input_search)
t_emb = embedding(input_title)
d_emb = embedding(input_desc)
b_emb = embedding(input_brand)

s = GlobalMaxPool1D()(s_emb)
t = GlobalMaxPool1D()(t_emb)
d = GlobalMaxPool1D()(d_emb)
b = GlobalMaxPool1D()(b_emb)

concat = concatenate([t, d, b])
concat = Dense(128)(concat)
s = Dense(128, name='vec')(s)
similarity = Lambda(cos_sim)([s, concat])

model = Model(inputs=[input_search, input_desc, input_brand, input_title], outputs=similarity)
_______________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_search (InputLayer)       [(None, 40)]         0                                            
__________________________________________________________________________________________________
input_title (InputLayer)        [(None, 40)]         0                                            
__________________________________________________________________________________________________
input_desc (InputLayer)         [(None, 40)]         0                                            
__________________________________________________________________________________________________
input_brand (InputLayer)        [(None, 40)]         0                                            
__________________________________________________________________________________________________
embedding_1 (Embedding)         (None, 40, 50)       1526100     input_search[0][0]               
                                                                 input_title[0][0]                
                                                                 input_desc[0][0]                 
                                                                 input_brand[0][0]                
__________________________________________________________________________________________________
global_max_pooling1d_5 (GlobalM (None, 50)           0           embedding_1[1][0]                
__________________________________________________________________________________________________
global_max_pooling1d_6 (GlobalM (None, 50)           0           embedding_1[2][0]                
__________________________________________________________________________________________________
global_max_pooling1d_7 (GlobalM (None, 50)           0           embedding_1[3][0]                
__________________________________________________________________________________________________
global_max_pooling1d_4 (GlobalM (None, 50)           0           embedding_1[0][0]                
__________________________________________________________________________________________________
concatenate_5 (Concatenate)     (None, 150)          0           global_max_pooling1d_5[0][0]     
                                                                 global_max_pooling1d_6[0][0]     
                                                                 global_max_pooling1d_7[0][0]     
__________________________________________________________________________________________________
search (Dense)                  (None, 128)          6528        global_max_pooling1d_4[0][0]     
__________________________________________________________________________________________________
product (Dense)                 (None, 128)          19328       concatenate_5[0][0]              
__________________________________________________________________________________________________
lambda (Lambda)                 (None,)              0           search[0][0]                     
                                                                 product[0][0]                    
==================================================================================================
Total params: 1,551,956
Trainable params: 1,551,956
Non-trainable params: 0
__________________________________________________________________________________________________

Items have 3 features – title, description, and brand. I want to use the same embedding layer between all 4 inputs to generate embeddings in the same vector space. However, for prediction time, I would like to remove the 3 inputs for the item, and make the output of the model the embedding vector from the dense layer named ‘vec’ to store the feature vectors. This is what I have tried:

search_model = Model(model.inputs[0], model.layers[-2].output)
item_model = Model(inputs=model.inputs[1:], outputs=model.layers[-1].output)

but I get the error

ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_brand_1:0", shape=(None, 40), dtype=int64) at layer "input_brand". The following previous layers were accessed without issue: []

Is there a way for me to share the embedding layer weights between the search and the item features that would allow me to prune certain layers come inference time? Or perhaps create 2 separate embedding layers, where 1 is run through the search and the other is run through the 3 features of the item but somehow keep the weights in both layers the same?

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP