fallow dupes

Detect copy-pasted code blocks across your entire codebase using suffix-array analysis.

fallow dupes

Options

Detection

Output

Incremental

Token cache

For projects with at least duplicates.minCorpusSizeForTokenCache source files (default 5000), fallow dupes keeps a per-project tokenized-source cache under <root>/.fallow/cache/dupes-tokens-vN/ so warm runs skip re-tokenizing unchanged files. The cache writes a sibling .gitignore containing * on first save, so its contents stay out of git status. Pass --no-cache to disable the cache for one run, or set a higher minCorpusSizeForTokenCache to keep it disabled even on larger projects. Below the threshold the cache stays off because the load/save overhead exceeds the tokenize savings on small corpora.

When --changed-since is set on a project at or above duplicates.minCorpusSizeForShingleFilter files (default 1024), the detector additionally drops unchanged files whose k-token shingles do not overlap any changed file before building the suffix array. Both knobs are scoped to [duplicates] in .fallowrc.json / .fallowrc.jsonc / fallow.toml / .fallow.toml.

Default generated-output ignores

fallow dupes skips common framework output by default: .next, .nuxt, .svelte-kit, .turbo, .parcel-cache, .vite, .cache, out, and storybook-static. These defaults merge with duplicates.ignore; they do not replace your configured ignores.

Test and mock files are also ignored by default: *.test.*, *.spec.*, __tests__/, and __mocks__/ paths are skipped, since intentionally repetitive test setup is a common source of first-run noise on hand-coded projects. This is part of the same ignoreDefaults set as the generated-output ignores above.

{
  "duplicates": {
    "ignoreDefaults": true,
    "ignore": ["**/__generated__/**"]
  }
}

Set ignoreDefaults to false to opt out and use only your duplicates.ignore list.

Reviewed clone groups

Add an intentional clone group to duplicates.ignoredClones without hiding other duplication findings:

{
  "duplicates": {
    "ignoredClones": ["dup:6f12ab34:2"]
  }
}

The key is <fingerprint>:<instance_count>. Copy the fingerprint from the human listing or clone_groups[].fingerprint in JSON. Token changes and new occurrences make the group reportable again, while formatting-only edits keep the same fingerprint.

Collision handles use dup:<16hex>-rN; ordinary short and widened handles are unchanged. Older dup:<16hex>-N handles no longer select or suppress a group. Re-run fallow dupes and review the reported group before replacing a saved key, rather than inserting r into the old handle. Upgrade all runners before adding -rN keys to a shared ignoredClones configuration. See migrating collision handles for baseline behavior.

Monorepo scoping

Both flags retain only clone groups where at least one instance is under a selected workspace root. The full cross-workspace index is still built; reported groups and the duplication_percentage stat are recomputed from the scoped slice.

Debugging

Detection modes

Ranking and JSON fields

The default report and --top multiply duplicated token count by occurrence count, then apply a spread boost. spread is the maximum directory-tree distance between instances. Same-file groups count one spread step per 250 lines of separation. The boost is capped at 15%.

Each JSON clone_groups[] entry includes:

• spread: maximum directory or same-file line distance
• similarity: lowest all-pairs similarity for near-miss groups, omitted for exact groups

The stats object can also include clone_groups_ignored when duplicates.ignoredClones hides groups, and near_candidates_skipped when bounded-work limits skip near-miss comparisons. Zero values are omitted.

Examples

# Mild mode (default)
fallow dupes

Example output

● Duplicates (3 clone groups)

  5,433 lines  2 instances
    deno/lib/types.ts:45-5477
    src/types.ts:45-5477

    914 lines  2 instances
    deno/lib/__tests__/string.test.ts:8-921
    src/__tests__/string.test.ts:7-920

     42 lines  3 instances
    src/features/forecasting/server/procedures/analytics.ts:141-181
    src/features/forecasting/server/procedures/cashflow.ts:153-194
    src/features/forecasting/server/procedures/income.ts:590-631

  Identical code blocks detected via suffix-array analysis, https://docs.fallow.tools/explanations/duplication#clone-groups

✓ 27,255 lines (19.4%) duplicated across 398 files (0.23s)

● Duplicates (3 clone groups)
  ...

✗ 27,255 lines (19.4%) duplicated across 398 files (0.23s)
Duplication (19.4%) exceeds threshold (15%)

JSON output

mirrored_directories

When fallow detects directory pairs that are largely duplicated (e.g., a deno/ mirror of src/), the JSON output includes a mirrored_directories array:

{
  "mirrored_directories": [
    {
      "dir_a": "deno/lib",
      "dir_b": "src",
      "shared_files": ["client.ts", "handlers.ts", "types.ts", "utils.ts"],
      "total_lines": 6347
    }
  ]
}

Each entry identifies two directories with significant overlap, the shared filenames, and the total duplicated lines across those files.

See also