Korean voice phishing detection
My Ph.D. research: detecting Korean voice phishing (vishing) calls from what is said and how it sounds. I built the call corpus, a series of text, audio and multimodal detectors, and a small acoustic model that runs on a phone.
Demo and evidence
Try it
Pick a call and press play. The transcript appears sentence by sentence, and three models score the call so far after each sentence: one reads only the text, one hears only the audio, and one uses both. Once the text + audio model flags a call, a separate classifier names the scam type. All calls come from the test fold, so none of these models trained on them. They were chosen as clear examples; the full results are below.
The three models run on our lab server (one RTX 2060 GPU). Pick a call to score it live, or upload up to 60 seconds of Korean phone audio. Uploads are processed in memory and not stored. This is a research model, not a security product.
Replay plays back the models' saved outputs in time with the audio; it works offline. The live tab sends the call or your clip to the lab server, which runs the models on the spot.
Evidence
- Code for every paper: github.com/selfcontrol7/Korean_Voice_Phishing_Detection
- Dataset: IEEE DataPort, doi:10.21227/163c-0542
- Papers: Mathematics 2023 · KIPS Transactions 2022 · KIPS 2021 · KIISE KCC 2025 · IEEE Access 2025
The problem
- The scam: callers pretend to be prosecutors, bank staff or loan officers and talk victims into sending money.
- Korean speech: a useful detector has to understand Korean calls, not just keywords.
- Timing: it has to warn while the call is still going on, not afterwards.
- Privacy: ideally it runs on the phone itself, so the call never leaves the device.
- No data: when I started, no public Korean dataset put real scam calls next to ordinary calls, so the first step was to build one.
The data
- KorCCVi v2 calls: 1,417 calls (706 vishing, 711 legitimate), split into 39,429 utterance segments, about 67 hours of speech. The vishing calls are real scam recordings published by Korea's Financial Supervisory Service (FSS). The legitimate calls come from AI Hub conversation data. The transcripts, labels and a one-row-per-call table are in the repository, and its data statement links the FSS audio.
- Transcript dataset: 2,927 transcripts (695 vishing, 2,232 legitimate) and the back-translated training sets of the augmentation study, published on IEEE DataPort (doi:10.21227/163c-0542).
The system
Each study added a different model. Click a figure to see it in full.





What I did
The work moved from text-only models to audio and multimodal models, one question at a time.
Results
| Approach | Data | Result | Source |
|---|---|---|---|
| Attention 1D CNN-BiLSTM with FastText embeddings | 2,927 transcripts | weighted F1 99.31% | Mathematics 2023 |
| Back-translation augmentation | 2,927 transcripts + augmented | F1 98.91% original data 98.14%, SMOTE 97.23% mean of 16 classifiers | IEEE Access 2025 |
| Compact acoustic MLP on eGeMAPS, 23K parameters | 142 test calls (3,925 segments) | segment vishing F1 99.71% call level 99.30% | KIISE KCC 2025; IEEE Access (under review) |
| Audio–text fusion (concatenation, FiLM, cross-attention) vs. audio or text alone | 1,188 calls, 5 folds × 5 seeds | call macro-F1 0.993–0.998 all eight setups; fusion no better than audio alone | Ph.D. dissertation (paper in preparation) |




On a phone (Samsung Galaxy S24 Ultra): the compact acoustic models run on the device.
- MFCC model: median 19 ms to score a 3-second window, end to end (audio, features, classifier, smoothing); 26 KB model file.
- eGeMAPS model: median 63 ms per window; 103 KB model file.
- Wav2Vec2.0 version, for comparison: 1.19 s per window.