← All projects

Korean voice phishing detection

My Ph.D. research: detecting Korean voice phishing (vishing) calls from what is said and how it sounds. I built the call corpus, a series of text, audio and multimodal detectors, and a small acoustic model that runs on a phone.

Where
Soongsil University, Seoul · 2021–2026 · with Prof. Dong-Joo Park
My role
First author, main researcher: data collection, modeling, experiments and writing
Models
Classical ML, KoBERT, KR-SBERT, FastText with attention CNN-BiLSTM, Wav2Vec2, MFCC and eGeMAPS features, audio–text fusion
Output
Papers in Mathematics (2023) and IEEE Access (2025), Korean journal and conference papers, a granted Korean patent with Prof. Park (KR 10-2640315), an open dataset and open code

Demo and evidence

Try it

Pick a call and press play. The transcript appears sentence by sentence, and three models score the call so far after each sentence: one reads only the text, one hears only the audio, and one uses both. Once the text + audio model flags a call, a separate classifier names the scam type. All calls come from the test fold, so none of these models trained on them. They were chosen as clear examples; the full results are below.

Replay plays back the models' saved outputs in time with the audio; it works offline. The live tab sends the call or your clip to the lab server, which runs the models on the spot.

Evidence

The problem

The data

The system

Each study added a different model. Click a figure to see it in full.

Voice-phishing detectors: call audio goes to a text branch (speech to text, text encoder) and an audio branch (acoustic features), fused by concatenation, FiLM gating or cross-attention into a call score; compact acoustic MLPs use the audio alone on a phone.
Overview of the detectors Text and audio of each segment are encoded separately and fused; segment scores are pooled into one score per call. I compared concatenation, FiLM gating and cross-attention, where the text attends over the audio sequence.
Architecture of the attention-based 1D CNN-BiLSTM with FastText embeddings.
Attention hybrid model · Mathematics 2023 FastText embeddings, a 1D convolution, BiLSTM layers and word- and sentence-level attention. Figure from the paper, CC BY 4.0.
Back-translation flow through English, Chinese and Japanese.
Back-translation · IEEE Access 2025 Vishing transcripts are translated into English, Chinese and Japanese and back into Korean to add training examples. Figure from the paper, CC BY 4.0.
Segment-based acoustic pipeline from call audio to a call score.
Acoustic detectors · IEEE Access, under review Call audio is segmented, and MFCC, eGeMAPS or Wav2Vec2.0 features feed a compact MLP whose segment scores make the call decision. Figure from the submitted manuscript.
The eight conditions of the multimodal comparison.
Multimodal comparison · Ph.D. dissertation Eight separately trained text-only, audio-only and audio–text setups, with one shared call-level decision rule.

What I did

The work moved from text-only models to audio and multimodal models, one question at a time.

2021
TextMachine learning on transcriptsKIPS 2021 · Encouragement Paper Award
TextFirst real-time detection designKIISE KCC 2021
2022
TextKoBERT-based detectionKIPS Transactions
TextAttention CNN-BiLSTMKIISE KCC 2022 · Outstanding Paper Award
PatentDeep-learning detection patent filedOctober 2022
2023
TextAttention hybrid model with FastTextMathematics (SCIE)
2024
PatentPatent grantedKR 10-2640315 · February 2024
TextBack-translation augmentationKIISE KCC 2024
DataTranscript dataset publishedIEEE DataPort
2025
TextBack-translation vs SMOTEIEEE Access (SCIE)
MultimodalAcoustic baselines and audio–text fusionKIISE KCC 2025 · Outstanding Paper Award
2026
AudioAcoustic comparison, codec test, on-phone measurementsIEEE Access · under review
MultimodalControlled comparison of fusion methodsPh.D. dissertation · paper in preparation
DataCall corpus releasedtranscripts, labels, FSS audio
Text text modelsAudio acoustic modelsMultimodal audio and textData data and patents

Results

ApproachDataResultSource
Attention 1D CNN-BiLSTM with FastText embeddings2,927 transcriptsweighted F1 99.31%Mathematics 2023
Back-translation augmentation2,927 transcripts + augmentedF1 98.91%
original data 98.14%, SMOTE 97.23%
mean of 16 classifiers
IEEE Access 2025
Compact acoustic MLP on eGeMAPS, 23K parameters142 test calls (3,925 segments)segment vishing F1 99.71%
call level 99.30%
KIISE KCC 2025; IEEE Access (under review)
Audio–text fusion (concatenation, FiLM, cross-attention) vs. audio or text alone1,188 calls, 5 folds × 5 seedscall macro-F1 0.993–0.998
all eight setups;
fusion no better than audio alone
Ph.D. dissertation (paper in preparation)
Bar chart of F1 scores of five text classifiers.
Attention paper: F1 of five text classifiers on the test set, as reported in the paper (Mathematics 2023, CC BY 4.0).
Bar chart of average F1, precision, recall and accuracy for the original, SMOTE and back-translation training sets.
Back-translation paper: average F1, precision, recall and accuracy for the original, SMOTE and combined back-translation training sets (IEEE Access 2025, CC BY 4.0).
Segment-level accuracy, F1 and recall of four acoustic configurations with confidence intervals.
Acoustic paper: segment-level accuracy, F1 and recall of the four feature configurations, with 95% bootstrap intervals (IEEE Access, under review).
Call-level macro-F1 of the eight multimodal conditions and paired differences.
Multimodal comparison: call-level macro-F1 of the eight setups over five seeds, and cross-attention minus each comparator with 95% intervals (Ph.D. dissertation).

On a phone (Samsung Galaxy S24 Ultra): the compact acoustic models run on the device.

← Korean OCR model Next: Thai Fruit Garden agent →