Enhanced Syndrome-based Reliability Decoding for Error Correction Code Transformers

Abstract

In this work, dynamically adaptive refinement masking (DARM) and mirror- sharing variational U-shaped architecture are proposed in order to improve the performance of error correction code transformer (ECCT). Instead of fixed binary masking, DARM module is designed to create unique masking for each attention head that reinforces the self-attention mechanism by further enhanc-ement of the contrast in the attention map with dynamic magnitudes taking the distribution of the attention weight as reference. Furthermore, under the use of sequential neural architecture, DARM modules are designed to be sequentially connected, creating an iterative refinement effect. For moderate coding lengths, the mirror-sharing variational U-shaped architecture is introduced to enhance the overall efficiency of the transformer-based decoder. The U-shaped architecture with variational-autoencoder-like skip-connection provides a segmentation like behavior that operates well with moderate length codes, especially at low coding rates. As the U-shaped model requires a certain level of depth to achieve desirable performance, an architectural-level parameter-sharing scheme called mirror-sharing is introduced to effectively scale the U-shaped model to achieve better efficiency and performance. Experimental results show considerable improvements in bit error rates compared to the baseline ECCT, while also significantly increasing the training convergence speed.