How to create an attention layer for deep learning networks?

Hello,
Can you please let me know how to create an attention layer for deep learning classification networks? I have a simple 1D convolutional neural network and I want to create a layer that focuses on special parts of a signal as an attention mechanism.
I have been working on the wav2vec MATLAB code recently, but the best I found is the multi-head attention manual calculation. Can we make it as a layer to be included for the trainNetwork function?
For example, this is my current network, which is from this example:
numFilters = 128;
filterSize = 5;
dropoutFactor = 0.005;
numBlocks = 4;
layer = sequenceInputLayer(numFeatures,Normalization="zerocenter",Name="input");
lgraph = layerGraph(layer);
outputName = layer.Name;
for i = 1:numBlocks
dilationFactor = 2^(i-1);

layers = [
convolution1dLayer(filterSize,numFilters,DilationFactor=dilationFactor,Padding="causal",Name="conv1_"+i)
layerNormalizationLayer
spatialDropoutLayer(dropoutFactor)
convolution1dLayer(filterSize,numFilters,DilationFactor=dilationFactor,Padding="causal")
layerNormalizationLayer
reluLayer
spatialDropoutLayer(dropoutFactor)
additionLayer(2,Name="add_"+i)];
% Add and connect layers.
lgraph = addLayers(lgraph,layers);
lgraph = connectLayers(lgraph,outputName,"conv1_"+i);
% Skip connection.
if i == 1
% Include convolution in first skip connection.
layer = convolution1dLayer(1,numFilters,Name="convSkip");
lgraph = addLayers(lgraph,layer);
lgraph = connectLayers(lgraph,outputName,"convSkip");
lgraph = connectLayers(lgraph,"convSkip","add_" + i + "/in2");
else
lgraph = connectLayers(lgraph,outputName,"add_" + i + "/in2");
end

% Update layer output name.
outputName = "add_" + i;
end
layers = [
globalMaxPooling1dLayer("Name",’gapl’)
fullyConnectedLayer(numClasses,Name="fc")
softmaxLayer
classificationLayer(‘Classes’,unique(Y_train),’ClassWeights’,weights)];
lgraph = addLayers(lgraph,layers);
lgraph = connectLayers(lgraph,outputName,"gapl");
I appreciate your help!
regards,
MohanadHello,
Can you please let me know how to create an attention layer for deep learning classification networks? I have a simple 1D convolutional neural network and I want to create a layer that focuses on special parts of a signal as an attention mechanism.
I have been working on the wav2vec MATLAB code recently, but the best I found is the multi-head attention manual calculation. Can we make it as a layer to be included for the trainNetwork function?
For example, this is my current network, which is from this example:
numFilters = 128;
filterSize = 5;
dropoutFactor = 0.005;
numBlocks = 4;
layer = sequenceInputLayer(numFeatures,Normalization="zerocenter",Name="input");
lgraph = layerGraph(layer);
outputName = layer.Name;
for i = 1:numBlocks
dilationFactor = 2^(i-1);

% Update layer output name.
outputName = "add_" + i;
end
layers = [
globalMaxPooling1dLayer("Name",’gapl’)
fullyConnectedLayer(numClasses,Name="fc")
softmaxLayer
classificationLayer(‘Classes’,unique(Y_train),’ClassWeights’,weights)];
lgraph = addLayers(lgraph,layers);
lgraph = connectLayers(lgraph,outputName,"gapl");
I appreciate your help!
regards,
Mohanad Hello,
Can you please let me know how to create an attention layer for deep learning classification networks? I have a simple 1D convolutional neural network and I want to create a layer that focuses on special parts of a signal as an attention mechanism.
I have been working on the wav2vec MATLAB code recently, but the best I found is the multi-head attention manual calculation. Can we make it as a layer to be included for the trainNetwork function?
For example, this is my current network, which is from this example:
numFilters = 128;
filterSize = 5;
dropoutFactor = 0.005;
numBlocks = 4;
layer = sequenceInputLayer(numFeatures,Normalization="zerocenter",Name="input");
lgraph = layerGraph(layer);
outputName = layer.Name;
for i = 1:numBlocks
dilationFactor = 2^(i-1);

Cart

Cart