HoneyPotter

Synthetic distractor factory for stress-testing vision models. Generate non-semantic patterns, textures, and ViT-focused moirés in both in-distribution and neon out-of-distribution (OOD) color modes.

GitHub PyPI Docs
pip install honeypotter

Why HoneyPotter?

Honeypots are adversarial non-semantic categories built to hijack classifier confidence. Use them to stress-test calibration, measure misclassification rate on distractors, and surface architectural biases—no new data collection required.

Pattern Families

Checkerboards, stripes, dots, blobs, solids, primitives, Perlin textures, and a dedicated moire family.

Color Control

Toggle grayscale, high-contrast BW, neon OOD palettes, or gamut-aware RGB sampling from your data.

Dataset Ready

Category names encode parameters and blur settings. Export labelmap.json, labels.txt, and metadata.csv for immediate integration.

What problems does this solve?

  • Quantify misclassification rate: how often do models map honeypots onto real classes?
  • Audit confidence calibration on non-semantic distractors.
  • Expose architectural biases (e.g., ViT patch aliasing vs. CNN texture priors).
  • Bootstrap robustness benchmarks on top of existing datasets.

Sample class list

01: checker_sz16_ood
02: stripes_45deg_w8_grid16
03: moire_grid16_h15
04: dots_r6_s24_color

What does HoneyPotter export?

Image directories

Each category (e.g., moire_f7_df8_a163_grid16_h15) is a folder filled with numbered PNGs ready to drop into validation sets.

Label metadata

labelmap.json, labels.txt, and metadata.csv bind names to IDs and preserve parameters for every sample.

Patterns-only mode

Need backgrounds instead of labels? Use --patterns_only to dump a flat bank of tiles like pattern_000042.png.

Typical workflow

  1. Generate distractors alongside your validation set.
  2. Append the class IDs to your task (ImageNet → 1000 + K).
  3. Evaluate misclassification rate on distractors vs. clean accuracy.
  4. Drill down by family angles, hues, and grid settings to diagnose biases.

Install

Stable release

pip install honeypotter

From source

git clone https://github.com/benlten/HoneyPotter.git
cd HoneyPotter
pip install -e .[dev]

The CLI is exposed as honeypotter once installed (see below for quick commands).

CLI Examples

OOD Sweep

honeypotter generate \
  --out ./honeypots \
  --n_categories 50 --per_class 100 \
  --families checker,stripes,dots,blob,solid \
  --color_mode ood --seed 0

Gamut-Constrained

honeypotter generate \
  --out ./in_dist \
  --gamut ./imagenet_gamut.json \
  --families checker,stripes,perlin \
  --pattern_blur_sigma 2.0

Programmatic

from honeypotter import GenerateDistractor, CategorySpec

spec = CategorySpec.default(families=["moire"], n_categories=10)
gen = GenerateDistractor(spec=spec, seed=7)
img, name = gen()

Pattern Families at a Glance

Every run can emit both in-distribution (natural) and out-of-distribution (neon) variants. Each figure below comes straight from the generator.

Checker in-distribution sample
Checker · in-distribution
Checker OOD sample
Checker · OOD
Stripes in-distribution sample
Stripes · in-distribution
Stripes OOD sample
Stripes · OOD
Dots in-distribution sample
Dots · in-distribution
Dots OOD sample
Dots · OOD

ViT-Focused Moirés

HoneyPotter’s moiré family generates interference bands from dual-frequency sine waves. Intensities can be snapped to an N×N grid (default 16×16) so each vision transformer patch sees a coherent signal.

  • Randomise or enumerate hues with --moire_hues
  • Control interference with --moire_frequency* and --moire_angle*
  • Adjust contrast via --moire_saturation/value ranges

Outputs are named after their parameters so you can trace frequency, angle, grid size, and hue in downstream analysis.

Grid 16 hue 20 angle 8 moiré
Grid 16 · hue 20° · θ 8°
Grid 16 hue 150 angle 45 moiré
Grid 16 · hue 150° · θ 45°
Grid 8 hue 280 angle 90 moiré
Grid 8 · hue 280° · θ 90°
Freeform hue 45 angle 135 moiré
Freeform · hue 45° · θ 135°

Texture Banks Without Labels

Need synthetic backgrounds or augmentation tiles? Add --patterns_only to skip class directories, label maps, and metadata. Files are emitted as pattern_000123.png so you can drop them straight into your pipeline.

honeypotter generate --out ./texture_bank \
  --families moire,stripes --patterns_only \
  --n_categories 24 --per_class 4

Resources

GitHub Repository PyPI Package Evaluation Guidance Examples & Scripts