Swin Wavelet Transformer (SWT): Mixing Tokens with Wavelet and Multiwavelet Transforms

Authors

  • Waleed A. Mahmoud Al-Jawher College of Engineering, Uruk University, Baghdad, Iraq

The Swin Transformer possess a hierarchical structure, a robust structure, and an efficient self-attention mechanism. This makes it superior in performance for a wide variety of artificial intelligence and machine training tasks. It has revolutionized the field of digital signal processing and its applications by creating vision in multiple fields. It uses non-overlapping windows which leads to creating a distinct perspective and understanding of the context of the digital signals. However, it possess a very complicated hierarchical structure an complexities together. By reducing calculations and complexities together. This will lead to robust performance that make it a powerful tool for a wide range of computer vision tasks.

In this research, simplified symbol mixing methods were developed for coding structures similar to transformers, by forming various semantics in the text through linear mixing transformation and in combination with nonlinearity in the feed-forward layers. Hence, we were able to successfully propose a Swin Wavelet Transformer (SWT) model in which the self-attention sublayer was replaced by a Wavelet transform. In a second attempt, the wavelet transform was replaced by the multi-wavelet transform. These two proposed models are achieved a better performance from their FNets and BERT counterparts and are highly competitive with traditional and efficient transformers.

Keywords:

Swin Transformer, Deep Learning, Wavelet transform, Multiwavelet Transform, Multi Head Attention, BERT, CNN

[1] Z. Liu et al., “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10012–10022.

[2] Rasha Ali Dihin, E AlShemmary and Waleed Al-Jawher “Diabetic Retinopathy Classification Using Swin Transformer with Multi Wavelet” Journal of Kufa for Mathematics and Computer, Vol. 10, Issue 2, PP. 167-172, 2023.

[3] Rasha Ali Dihin, Ebtesam N AlShemmary, Waleed AM Al-Jawher, “Wavelet-Attention Swin for Automatic Diabetic Retinopathy Classification” Baghdad Science Journal, 2024.

[4] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang , Li Dong, Furu Wei and Baining Guo, “Swin Transformer V2: Scaling Up Capacity and Resolution” We2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022.

[5] James Lee-Thorp and Joshua Ainslie and Ilya Eckstein and Santiago Ontañón “FNet: Mixing Tokens with Fourier Transforms” Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4296 – 4313, 2022.

[6] Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, 2018.

[7] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural lan- guage understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, 2020.

[8] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. 2020. Fast transformers with clustered at- tention. Advances in Neural Information Processing Systems, 33:21665–21674, 2020.

[9] Zhenyou Zhang, Yi Wang, and Kesheng Wang. “ Fault diagnosis and prognosis using wavelet packet decomposition, fourier transform and artificial neu- ral network”. Journal of Intelligent Manufacturing, 24(6):1213–1227, 2013.

[10] Ali Akram Abdul-Kareem, Waleed Ameen Mahmoud Al-Jawher” Hybrid image encryption algorithm based on compressive sensing, gray wolf optimization, and chaos” , Journal of Electronic Imaging, Volume 32, Issue 4, Pages 043038-043038, 2023.

[11] Maryam I Mousa Al-Khuzaay, Waleed A Mahmoud Al-Jawher, “New Proposed Mixed Transforms: CAW and FAW and Their Application in Medical Image Classification” International Journal of Innovative Computing, Volume 13, Issue 1-2, Pages 15-21, 2022.

[12] AHM Al-Helali, W. A. Mahmoud, HA Hali, AF Fadhel “Multispectral Image Fusion using Walidlet Transform” Advances in Modelling and Analysis B, Volume 52, Issu. 1-2, pp. 1-20, 2009.

[13] Walid Amin Mahmoud-Jawher “Computation of Wavelet and Multiwavelet Transforms using Fast Fourier Transform” Journal Port Science Research, Vol. 4, Issue 2, PP. 111-117, 2021.

[14] Maryam I Al-Khuzaie, Waleed A Mahmoud Al-Jawher “Enhancing Medical Image Classification: A Deep Learning Perspective with Multi Wavelet Transform” Journal Port Science Research, Vol. 6, Issue 4, PP. 365-373, 2023.

[15] Waleed. A. Mahmoud & I.K. Ibraheem "Image Denoising Using Stationary Wavelet Transform” Signals, Inf. Patt. Proc. & Class. Vol. 46, Issue 4, Pages 1-18, 2003.

[16] W. A. Mahmoud & Z. Ragib “Face Recognition Using PCA and Optical Flow” Engineering Journal, Vol. 13, Issue 1, PP. 35-47, 2007.

[17] SM Saadi, WAM Al-Jawher, “Proposed DeepFake Detection Method Using Multiwavelet Transform,” International Journal of Innovative Computing 13 (1-2), 61-66, 2022

[18] Qutaiba Kadhim, Waleed Ameen Mahmoud Al-Jawher “A new multiple-chaos image encryption algorithm based on block compressive sensing, swin transformer, and wild horse optimization” Multidisciplinary Science Journal, Vol. 7, Issue 1, PP. 2025012-2025012, 2024.

[19] Waleed A Mahmoud Al-Jawher, Shaimaa A Shaaban “K-Mean Based Hyper-Metaheuristic Grey Wolf and Cuckoo Search Optimizers for Automatic MRI Medical Image Clustering” Journal Port Science Research, Volume 7, Issue 3, Pages 109-120, 2024.

[20] Walid A Mahmoud, Majed E Alneby, Wael H Zayer “Multiwavelet Transform and Multi-Dimension-Two Activation Function Wavelet Network Using For Person Identification” Iraqi Journal Of Computers, Communications, Control And Systems Engineering, Vol 11, Issue 1, 2011.

[21] Waleed A Mahmoud, Ahmed S Hadi “Systolic Array for Realization of Discrete Wavelet Transform “Journal of Engineering, Vol. 13, Issue 2, PP. 1-9, 2007.

[22] W. A. Mahmoud & Omama Razaq “Speech recognition using new structure for 3D neural network” University of Technology, 1st Computer Conference, PP. 161-171, 2010.

[23] W. A. Mahmoud & Z. J. M. Saleh “ An Algorithm for Computing Multiwavelets &Inverse Transform Using an Over-Sampled Scheme of Pre& Post processing respectively” Engineering Journal, Vol. 10, Issue 2, PP. 270-288, 2004.

[24] W. A. Mahmoud Z Jalal & N. K. Wafi “A New Method of Computing Multi-wavelets Transform using Repeated Row Preprocessing.” Al-Rafidain Engineering Journal, Vol. 12, Issue 2, PP. 21-31., 2004.

[25] E. Dihin, R. Al-Jawher, Waleed and Al-Shemmary “Implementation of The Swin Transformer and Its Application In Image Classification” Journal Port Science Research, vol. 6, Issue 4, PP. 318-331. 2023.

[26] Waleed Ameen Mahmoud Al-Jawher, A. Barsoum and Entather Mahos “Fuzzy Wavenet (FWN) classifier for medical images” Al-Khwarizmi Engineering Journal, Vol. 1, Issue 2, PP. 1-13, 2005.

[27] Waleed A. Mahmoud, MS Abdulwahab, HN Al-Taai: “The Determination of 3D Multiwavelet Transform” IJCCCE, vol. 2, issue 4, 2005.

[28] Waleed A. M. Al-Jawher, T Abbas – “Feature combination and mapping using multiwavelet transform” IASJ, AL-Rafidain, Issue 19, Pages 13-34, 2006.

[29] Waleed A Mahmoud, MR Shaker “3D Ear Print Authentication using 3D Radon Transform” proceeding of 2nd International Conference on Information & Communication Technologies, Pages 1052-1056, 2006.

[30] H. Al-Taai, Waleed A. Mahmoud & M. Abdulwahab “New fast method for computing multiwavelet coefficients from 1D up to 3D” , Proc. 1st Int. Conference on Digital Comm. & Comp. App., Jordan, PP. 412-422, 2007.

[31] Waleed Ameen Mahmoud “A Smart Single Matrix Realization of Fast Walidlet Transform” Journal of Research and Reviews in Computer Science, Volume 2, Issue, 1, PP 144-151, 2011.

[32] WA Mahmoud, AI Abbas, NAS Alwan “Face Identification Using Back-Propagation Adaptive Multiwavelet” Journal of Engineering 18 (3), 2012

[33] WA Mahmoud, AS Hadi, TM Jawad “Development of a 2-D Wavelet Transform based on Kronecker Product” - Al-Nahrain Journal of Science, Vol. 15, Issue 4, PP. 208-213, 2012.

[34] H. M Hasan, Waleed A. Mahmoud Al- Jawher, M. A Alwan “3-d face recognition using improved 3d mixed transform” Journal International Journal of Biometrics and Bioinformatics, Vo. 6, Issue 1, PP. 278-290, 2012.

[35] Waleed A. Mahmud Al-Jawher, Talib M. J. Abbas Al-Talib, R. Hamudi A. Salman “Fingerprint Image Recognition Using Walidlet Transform” Australian Journal of Basic and Applied Sciences, Australia, 2012.

[36] Saleem MR Taha, Walid A Mahmood "New techniques for Daubechies wavelets and multiwavelets implementation using quantum computing “Journal Facta universitatis-series: Electronics and Energetics, Volume 26, Issue 2, Pages 145-156, 2013.

[37] Waleed A Mahmoud, Dheyaa J Kadhim “A Proposal Algorithm to Solve Delay Constraint Least Cost Optimization Problem” Journal of Engineering, Vol. 19, Iss 1, PP 155-160, 2013.

[38] SMR Taha, WA Mahmood “New techniques for Daubechies wavelets and multiwavelets implementation using quantum computing” Facta universitatis-series: Electronics and Energetics 26 (2), 145-156, 2013.

[39] Waleed A. Mahmoud, J J. Stephan and A. A. Razzak “Facial Expression Recognition Using Fast Walidlet Hybrid Transform” Journal port Science Researchو Volume3, No:1, Pages 59-69 2020.

[40] R. A. Dihin, W. A Mahmoud Al-Jawher, Ebtesam N AlShemmary “Diabetic Retinopathy Image Classification Using Shift Window Transformer”, Journal of Innovative Computing, Vol. 13, Issue 1-2, PP. 23-29, 2022.

[41] Rasha Ali Dihin, Ebtesam N. AlShemmary and Waleed A. Mahmoud Al-Jawher “Automated Binary Classification of Diabetic Retinopathy by SWIN Transformer” Journal of Al-Qadisiyah for computer science and mathematics (JQCM), Vol 15, Issue 1, PP. 169-178, 2023.

[42] L R. Hussssein andJ. M. A. Al-Sammarie W. A. Mahmoud “Image Identification using Minimum Distance Classifier with Multi-Wavelet Transform” Journal of Advances in Modelling and Analysis B, Volume 46, Issue (5-6), pages 1-22, 2003.

[43] Waleed A Mahmoud, MR Shaker “3D Ear Print Authentication using 3D Radon Transform” proceeding of 2nd International Conference on Information & Communication Technologies, Pages 1052-1056, 2006.

[44] W Al-Jowher, N Al-Ramahi, M. Alfaouri "Image Identification And Labeling Using Hybrid Transformation And Neural Network" Neural Network World: International Journal on Neural and Mass - Parallel Computing and Information Systems; Prague, Volume 17, Issue 4, Pages 377-395, 2007.

[45] Ibraheem Al-Jadir, Waleed A Mahmoud “A Grey Wolf Optimizer Feature Selection Method and its Effect on the Performance of Document Classification Problem” Journal Port Science Research, Vol. 4, Issue 2, Pages 125-131, 2021.

[46] W. A. Mahmoud, Jane Jaleel Stephan and A. A. W. Razzak “Facial Expression Recognition from Video Sequence Using Self Organizing Feature Map” Journal port Science Researchو Transaction on Engineering, Technology and Their Applications, Vol. 4, Issue 2, Pages 53-68, 2021.

[47] AAR Sakran, SM Hadi, WAM Al-Jawher “A New Approach for DNA Sequence Analysis Using Multiwavelet Transform (MWT)” Journal of Physics: Conference Series 2432 (1), 012022.

[48] Q. K Abed, W. A Mahmoud Al-Jawher “A Robust Image Encryption Scheme Based on Block Compressive Sensing and Wavelet Transform” International J. of Innovative Computing, Vol. 13, I. 1-2, PP. 7-13, 2022.

[49] Ali Akram Abdul-Kareem, Waleed Ameen Mahmoud Al-Jawher, “Image Encryption Algorithm Based on Arnold Transform and Chaos Theory in the Multi-wavelet Domain”, International Journal of Computers and Applications, Vol. 45, Issue 4, pp. 306-322, 2023

[50] L. F. Katran, Ebtesam N AlShemmary, Waleed Ameen Al Jawher “Deep Learning's Impact on MRI Image Analysis: A Comprehensive Survey” Texas Journal of Engineering and Technology, Vol. 25, PP. 63-80, 2023.

[51] Ali Akram Abdul-Kareem, Waleed Ameen Mahmoud Al-Jawher “A Hybrid Domain Medical Image Encryption Scheme Using URUK and WAM Chaotic Maps with Wavelet–Fourier Transforms” Journal of Cyber Security and Mobility, Pages 435–464-435–464, 2023.

[52] Zahraa A Hasan, Suha M Hadi, Waleed A Mahmoud, “Speech scrambler with multiwavelet, Arnold Transform and particle swarm optimization” Journal Pollack Periodica, Volume 18, Issue 3, Pages 125-131, 2023.

[53] Zahraa A Hasan 1, Suha M. Hadi, Waleed A. Mahmoud al-Jawher “Speech scrambling based on multiwavelet and Arnold transformations”, Indonesian Journal of Electrical Engineering and Computer Science, 30 , 2023.

[54] M. I Al-Khuzaie, Waleed A Mahmoud Al-Jawher “Enhancing Medical Image Classification: A Deep Learning Perspective with Multi Wavelet Transform” Journal Port Science Research, Vol. 6, Issue 4, PP. 365-373, 2023.

[55] Lamyaa Fahem Katran, Ebtesam N AlShemmary, Waleed Ameen Al Jawher “Deep Learning's Impact on MRI Image Analysis: A Comprehensive Survey” Texas Journal of Engineering and Technology, Vol. 25, PP. 63-80, 2023.

[56] L. Fahem Katran, Ebtesam N AlShemmary, Waleed AM Al-Jawher “A Review of Transformer Networks in MRI Image Classification” Al-Furat Journal of Innovations in Electronics & Computer Engineering, PP. 148-162, 2024.

[57] S. Saadi, W. Al-Jawher” Enhancing image authenticity: A new approach for binary fake image classification using DWT and swin transformer” Global Journal of Engineering & Technology Advances, 19(3), 2024.

[58] Maryam I Mousa Al-Khuzaie, Waleed A Mahmoud Al-Jawher “Enhancing Brain Tumor Classification with a Novel Three-Dimensional Convolutional Neural Network (3D-CNN) Fusion Model” Journal Port Science Research, Volume 7, Issue 3, Pages 254-267, 2024.

[59] L. Katran, EN AlShemmary, WAM Al-Jawher “Integrating Swin Transformer with Fuzzy Gray Wolve Optimization for MRI Brain Tumor Classification” Journal of Intelligent Engineering & Systems, Vol. 17, Issue 6. 2024.

[60] Waleed A Mahmoud, MR Shaker “3D Ear Print Authentication using 3D Radon Transform” proceeding of 2nd International Conference on Information & Communication Technologies, Pages 1052-1056, 2006.

[61] WA Mahmoud, ALM Rasheed “3D Image Denoising by Using 3D Multiwavelet” AL-Mustansiriya J. Sci 21 (7), 108-136, 2010.

[62] H. M Hasan, Waleed A. Mahmoud Al- Jawher, M. A Alwan “3-d face recognition using improved 3d mixed transform” Journal International Journal of Biometrics and Bioinformatics, Vo. 6, Issue 1, PP. 278-290, 2012.

[63] Waleed A. Mahmud Al-Jawher, Talib M. J. Abbas Al-Talib, R. Hamudi A. Salman “Fingerprint Image Recognition Using Walidlet Transform” Australian Journal of Basic and Applied Sciences, Australia, 2012.

[64] SMR Taha, WA Mahmood “New techniques for Daubechies wavelets and multiwavelets implementation using quantum computing” Facta universitatis-series: Electronics and Energetics 26 (2), 145-156, 2013.

[65] Waleed A. M. Al-Jawher, T Abbas – “Feature combination and mapping using multiwavelet transform” IASJ, AL-Rafidain, Issue 19, Pages 13-34, 2006.

[66] Walid A Mahmoud, Majed E Alneby, Wael H Zayer “2D-multiwavelet transform 2D-two activation function wavelet network-based face recognition” J. Appl. Sci. Res, vol. 6, issue 8, 1019-1028, 2010.

[67] Waleed Ameen Mahmoud “A Smart Single Matrix Realization of Fast Walidlet Transform” Journal of Research and Reviews in Computer Science, Volume 2, Issue, 1, PP 144-151, 2011.

[68] W. Mahmoud, D Kadhim “A Proposal Algorithm to Solve Delay Constraint Least Cost Optimization Problem” Journal of Engineering, Vol. 19, Iss 1, PP 155-160, 2013.

[69] M. Al-Khuzaie, W. Al-Jawher “Enhancing Brain Tumor Classification with a Novel Three-Dimensional Convolutional Neural Network Fusion Model” Journal Port Science Research, Volume 7, Issue 3, Pages 254-267, 2024.

[70] W. A. Mahmoud & I. A Al-Akialy “A Tabulated Method of Computation Multiwavelet Transform” Al-Rafidain University College, Vol. 15, PP. 161-170, Iraq, 2004.

Swin Wavelet Transformer (SWT): Mixing Tokens with Wavelet and Multiwavelet Transforms. (2024). Journal Port Science Research, 7(3), .271-286. https://doi.org/10.36371/port.2024.3.14

License

Copyright (c) 2024 Journal Port Science Research

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

How to Cite

Swin Wavelet Transformer (SWT): Mixing Tokens with Wavelet and Multiwavelet Transforms. (2024). Journal Port Science Research, 7(3), .271-286. https://doi.org/10.36371/port.2024.3.14

Most read articles by the same author(s)

1 2 > >>