Detecting Unseen Android Ransomware Families with GNNs and LLMs

Introduction

My master's thesis at the Chair of Machine Learning and Security at TU Berlin asks how pre-trained large language models (LLMs) can give the nodes of a call graph better features for detecting Android ransomware. Mohammad Ebrahimi Fard supervised the work, and Prof. Dr. Konrad Rieck and Prof. Dr. Klaus-Robert Müller examined it. The answer I arrived at is that the leverage lies in what the model is asked to represent. A code embedding with 768 dimensions collapses on ransomware families it has never seen, while two bits from an LLM that judges what a class does carry over to them.

This post walks through the setup, the three findings, and the limits of the evidence. The figures are interactive rebuilds of slides from my defense, and every number comes from the thesis. An earlier baseline post describes the first experiment that led here.

The problem

In 2025 Android accounted for roughly 73% of mobile operating-system use and ran on more than 3.9 billion active devices. A phone concentrates private messages, banking access, and the second factor for a user's other accounts. Ransomware denies access to the device or encrypts its files and then demands payment, so detection has to happen before the application runs, since afterwards the damage is done.

I work with static analysis because screening without execution scales to hundreds of applications and millions of methods. From each Android package (APK) I build a call graph with one node per method (function call graph, FCG) or one node per class (class call graph, CCG), and a graph neural network (GNN) classifies the graph as benign or ransomware. A GNN sees how nodes connect, but to it an encryption routine and a location query are interchangeable anonymous points. An LLM understands what code does, but it reads one snippet at a time and reasons poorly over graph structure. The hybrid gives every node a feature derived from a pre-trained language model and leaves all structural reasoning to the GNN.

Two ways of deriving that feature are compared, with the graph, the classifier, and the evaluation held fixed. One encodes the source code of a node directly into a code embedding. The other prompts an LLM about the code and uses its answer, a verdict, behavior flags, or a summary, as the feature.

The corpus

Everything builds on the 715-APK corpus of Chew et al., which holds 502 benign applications and 213 ransomware samples from six families.

The three largest families, wipelocker, simplelocker, and wannalocker, account for 185 of the 213 samples and form the "big-3" that carries the evaluation. The green bars already hint at a later finding, because wipelocker's 70 APKs carry only seven different sets of class bodies. The family is close to one application repackaged many times.

I decompile every APK, normalize each class body, and key it by its SHA-256 hash in a single database. The 2,665,731 class occurrences of the corpus collapse to 1,611,589 unique class bodies, so each body needs to be embedded or judged by an LLM only once. The same structure turns questions such as how much code a family shares with other families or with benign apps into single database queries. Folding methods into classes shrinks the median graph from 25,310 to 2,256 nodes, about a tenfold reduction, and every LLM experiment runs on the class level.

How I evaluate

A stratified random split puts samples of every family into training and test. Eleven of the sixteen detector evaluations I surveyed for the thesis rest their headline numbers on random or k-fold protocols of this kind. Such a split rewards family-specific code, and it cannot rank feature sources, since five handcrafted scalars per node already reach 0.982 macro F1 on it.

The primary protocol is family leave-one-out. Each fold holds out one entire ransomware family together with an equal number of random benign apps and trains on the rest. Six folds with three seeds each make 18 training runs per configuration, and I report macro F1 and ransomware recall. A temporal split is not informative on this corpus, because the six families cluster in time and a chronological cut would place essentially one family in the test set.

First attempt: raw code embeddings

The first approach feeds each node's source code through CodeRankEmbed, a 137M-parameter code encoder, and uses the 768-dimensional vector as the node feature of a graph attention network (GAT). On a random split this looks solved, with a macro F1 of 0.938 on function call graphs and up to 0.993 on class call graphs.

Holding out one family at a time changes the picture. The same model drops to 0.477 big-3 macro F1 on function graphs and 0.501 on class graphs, recovering only 0.183 and 0.206 of the held-out ransomware. A control that gives every node a random vector scores 0.627 and 0.630, so the embedding sits below noise.

The wipelocker fold shows the gap most clearly. With random node vectors the GNN detects 62 of 70 held-out wipelocker apps, and with the code embedding it detects none. The training trajectory adds a twist, since on wipelocker the held-out score is highest at the first checkpoint and sits at the all-benign floor of 0.333 by epoch 9 for all three seeds.

I then varied everything I could think of while keeping the raw embedding.

Switching from method graphs to class graphs, swapping the architecture, and doubling or tripling the training set all leave the score below the random control. The one all-six gain from the augmentation, 0.551 to 0.639, comes from blackroselucy alone, most likely because its small graphs of about 80 class nodes read as malicious once hundreds of similarly small malware samples enter training.

Why the embeddings fail

The diagnosis starts with the corpus. A family is largely one codebase repeated, with wipelocker's 70 APKs reducing to 1,060 unique class bodies, 66 of the 70 APKs carrying the same boilerplate under one author's package name. Under a random split, identical class bodies sit in training and test.

Because the database stores every class body once, I can measure how much code a held-out family shares with the ransomware the model trains on. I call the share of that training pool that the held-out family also contains its training relevance.

Simplelocker and wannalocker share about 66,000 class bodies, which is 78.7% of wannalocker's code. Holding out simplelocker leaves a small pool of roughly 87,000 class bodies in which 75.5% is relevant, and the GNN recovers 0.50 of the family. Holding out wannalocker leaves the same 66,000 shared bodies, but simplelocker now floods the pool to almost 276,000, so only 24.0% is relevant and recall falls to 0.09. For wipelocker, 222 of the 223 bodies it shares with other families also occur in benign apps, nothing relevant remains, and recall is 0.04.

Pletor is the exception that sharpens the picture. It shares no class body with any training family, yet the embedding detects all six of its APKs. Its AES encryption classes sit at a cosine similarity of 0.998 beside simplelocker's, identical decompiled code except for one package segment, so the embedding recognizes a near-duplicate and reads no behavior.

A linear probe on single class bodies confirms the pattern. It recovers code that a training family also contains at 0.884 to 0.916 and a held-out family's own code at only 0.072 to 0.351, and a plain bag-of-tokens control matches or beats the encoder on five of the six folds.

The nearest-neighbor matrices show where a held-out family lands. Once its own code is excluded, benign code is the closest remaining group for every family, and for pletor and filecoder it is closer than their own family even before the exclusion. The model learns to read that region of the embedding space as benign, and predicting everything benign becomes locally optimal.

The random control scores well for a related reason. Ransomware graphs are small, with a median of 228 class nodes against 3,542 for benign apps, and a logistic regression on just the log node and edge counts reaches 0.809 big-3 macro F1. That classifier flags 15.0% of benign apps on average, and padding an application with benign libraries defeats it, so graph size is a shortcut that fails as a detector. It still explains why the random control beats the embedding, since with random vectors the GNN cannot memorize families and falls back on size.

Second attempt: ask the LLM what a class does

The embedding fails because it encodes how code is written. So I changed the model's role and prompted an LLM about the code, using its answer as the node feature. Gemma 4 E4B-it is the main generator and Qwen3.5-4B, a model of similar size from a different family, replicates the result. Both run zero-shot over all 1.6 million unique class bodies, served with vLLM and constrained by guided JSON decoding, without access to the family label or the data split.

The model returns three kinds of output. The first is a verdict of two bits, potentially_malicious and potentially_ransomware. The second is seven behavior flags covering crypto use, file enumeration, screen locking, command-and-control networking, SMS abuse, device administration, and anti-analysis. The third is a short behavioral summary of the class that I embed afterwards.

Replacing the 768-dimensional embedding with the two-bit Gemma verdict lifts the class-level big-3 macro F1 from 0.501 to 0.909 and the all-six score from 0.589 to 0.906, with a ransomware recall of 0.847. Qwen reproduces the gain at 0.760 big-3 and 0.878 all-six. Both clear the random control, and the verdict also beats the embedding under a GraphSAGE backbone and under a learned pooling readout, so the result depends on neither generator nor architecture. In distribution nothing separates the features, since every one of them scores between 0.921 and 0.978 on the random split.

Adding information back hurts. Embedding the LLM's free-text summaries recovers little at 0.471 to 0.537, and concatenating the code embedding onto the verdict drops the score from 0.909 to 0.531. A summary re-encodes a family's surface in natural language, and the embedding gives the GNN 768 dimensions in which to memorize training families. The gain lies in what the verdict leaves out.

What the verdict reads

Three checks probe why the verdict transfers.

First, I clustered all 294,004 class bodies that occur in ransomware apps with HDBSCAN. The embedding space splits into 62 dense clusters that are on average 90.6% one provenance, so a cluster marks one library or application and no behavior, and 87.5% of the bodies lie in the noise between them. The classes the LLMs flag scatter through that noise, with 68.4% of Gemma's and 50.0% of Qwen's flags falling there, so the flags have no region of their own in the embedding space.

The LLM flags have no region of their own in the embedding space

62

dense clusters, each mostly one codebase

87.5 %

of 294,004 class bodies are noise

68.4 % / 50.0 %

of Gemma's and Qwen's flags lie in the noise

Second, all six families predate the generators' training cutoffs, so a verdict could stem from recognizing a family. I rewrote 110 flagged payload classes by replacing the package and every declared type name with meaningless placeholders while leaving control flow and Android API calls untouched. Only 8 of 110 verdicts flip toward benign, and asked to name the family, Gemma answers "unknown" for all 110 classes. Simplelocker is the caveat, with 7 of 30 of its classes flipping, and the thesis names it as such.

package org.simplelocker;
public final class d {
    private final List c = Arrays.asList("enc");
    ...
}

package a.a.a;
public final class T0 {
    private final List c = Arrays.asList("");
    ...
}

Third, the verdict still needs the graph. Holding the Gemma verdicts fixed and replacing message passing with a parameter-free readout lowers the big-3 macro F1 from 0.909 to 0.806 for max-pooling, 0.793 for sum-pooling, and 0.585 for mean-pooling. On wipelocker the three readouts reach 0.47 to 0.49 against 0.98 for the GAT.

A hand audit of the ransomware flag, drawn from each model's own positives and read by a single annotator, shows that the two models behave differently. Inside ransomware apps Gemma is correct on 54 of 79 flags, with 8 further flags malicious but not ransomware and 17 benign false positives, while Qwen is correct on all 32 of its flags. Merging both flag sets scores below either alone at 0.721 big-3, so the disagreement between the models does not add coverage.

A realistic benign majority

The base corpus holds about 2.4 benign apps per ransomware sample, far denser in ransomware than any market. I enlarged only the benign side, adding 1,664 apps from AndroZoo for 2,166 benign apps and 2,379 APKs in total, which gives a ratio of about ten to one. The held-out test folds stay balanced, and only the training pool grows.

At this scale the two bits alone are too coarse, with the big-3 macro F1 falling from 0.909 to 0.618, and the loss sits in wipelocker and wannalocker. Joining the verdict with five of the behavior flags recovers 0.849 big-3 macro F1 at a false-positive rate of 0.011, and joining all seven flags reaches the best all-six score of 0.809. The price is blackroselucy, which the verdict alone detects perfectly and which inverts from 1.000 to 0.333 under every behavior variant, because its flag profile matches the benign one. No combination of the two features detects all six families at scale.

The trained models also stay ransomware-specific. I held 783 general-malware apps out of every split and scored them with all 18 models. The behavior-joined models flag 0.083 to 0.086 of them and the verdict alone 0.135, and a post-hoc analysis of the flagged apps shows that all 112 are packed stubs with a median of seven classes against 2,966 in the unflagged rest. Their only ransomware evidence is one shared unpacking-loader class, and what such a stub hides lies outside the reach of static analysis.

Limits

The conclusions hold within bounds that the thesis spells out.

• All 213 ransomware samples come from six public families, and only the three large families carry test folds big enough for a graded estimate. The 213 samples reduce to at most 122 distinct compositions, and further ransomware could not be sourced.
• All six families predate the generators' training cutoffs, so recognizing a family and reasoning about behavior are observationally equivalent under this protocol. A ransomware family from after the cutoff is the definitive test.
• Graph size alone reaches 0.809 big-3 macro F1, so every score contains a share that no feature earned by reading behavior. Padding a payload with benign code would show how large that share is, and I left this test to future work.
• Holding out a family also largely holds out its period, and the balanced test folds never measure detection at a realistic imbalance.
• The audit has a single annotator and bounds precision only, the at-scale results rest on Gemma alone, and static extraction cannot see through packers or reflection.
• Two small LLMs of similar size yield largely disjoint flag sets, so swapping the model silently changes what a pipeline flags.

What I take from it

Random splits are an illusion for malware detection. A model at 0.993 on a random split can recover almost none of an unseen family, and a feature that a near-trivial control saturates ranks nothing.

Distilling beats embedding. Asking an LLM what a class does and keeping two bits produces a feature that transfers to unseen families, while every continuous representation I tried, including embeddings of the LLM's own summaries, re-opens the family shortcut.

Semantics and topology work together. The LLM judges each class in isolation, and the call graph separates connected payloads from isolated false positives, which no pooling of the verdicts reproduces. At realistic scale the behavior flags add the resolution that two bits lack.

For future work I would build a much larger labeled database of malicious and benign apps, label benign behavior as well, and fine-tune a code encoder on the labeled class bodies so that classes with the same behavior sit close together regardless of family. Such an encoder needs one forward pass where an LLM decodes its verdict token by token, which would bring the behavioral space within reach of a market-scale corpus. A held-out family has never shown a detector its code, and every family shows what that code does.

Resources

The trained detectors, one GAT checkpoint per held-out family for the two-bit verdict, the verdict joined with five behavior features, and the CodeRankEmbed embedding, are published on Hugging Face↗ under an MIT license. The pipeline that builds the class-level database from a set of APKs is available on request. The ransomware samples and their decompiled code cannot be shared, so the populated database is not published. If you would like the full thesis or want to discuss the results, get in touch.