Contributing code

Semgrep welcomes contributions from anyone. If you have an idea for a feature or notice a bug, open an issue. Open an issue before a pull request so maintainers can confirm the approach. You can also contribute by opening an issue without submitting code. This page gives an overview of the repository. For build instructions, see semgrep-core contributing or semgrep-cli contributing in Making a change.

File structure

Semgrep consists of a Python wrapper (semgrep-cli) around an OCaml engine (semgrep-core) which performs the core parsing/matching work. Within semgrep-core, there are two sources of parsers, pfff and tree-sitter-lang using Tree-sitter. Additionally, semgrep-core contains a subengine, spacegrep, for generic matching. You may also be interested in perf, which contains code for running repositories against specific rulesets. The following diagram shows the file structure.

.
├── cli/  (Python wrapper)
│   └── src/
│       └── semgrep/
│
├── src/  (semgrep-core)
│   │── analyzing/  (Dataflow analysis)
│   │── core_cli/  (Entrypoint for semgrep-core)
│   └── matching/  (Matching engine)
│
├── languages/  (Language parsers)
│
├── libs/  (Library components)
│   │── ast_generic/  (Generic AST)
│   └── spacegrep/  (Generic matching)
│
└── perf/  (Performance benchmarking)

Most of Semgrep’s logic is in cli/src and src.

Code relationship

The semgrep-core binary stands alone. Once built, you can run semgrep-core on a Semgrep rule for a given language with a file or directory and receive matches. For example, say you create the config file unsafe-exec.yaml and the program unsafe-exec.py:

rules:
  - id: unsafe-exec
    pattern: exec(...);
    message: Avoid use of exec; it can lead to a remote code execution.
    severity: MEDIUM
    languages: [python]

exec("ls");

If you run semgrep-core -config unsafe-exec.yaml unsafe-exec.py -lang python, it outputs

unsafe-exec.py:1 with rule unsafe-exec
 exec("ls");

If you run semgrep --config unsafe-exec.yaml unsafe-exec.py, it outputs

running 1 rules...
unsafe-exec.py
severity:warning rule:unsafe-exec: Avoid use of exec; it can lead to a remote code execution.
1:exec("ls");
ran 1 rules on 1 files: 1 findings

The matched code is the same, but with semgrep-cli the output is more polished and includes the message. semgrep-cli invokes the semgrep-core binary as a subprocess, with a flag to request JSON output. It reads the semgrep-core output and transforms it appropriately. Depending on the flags used, spacegrep is invoked both independently by semgrep-cli as a subprocess and by semgrep-core as a subfolder. semgrep-cli requires the spacegrep binary, but building semgrep-core also builds spacegrep.

Making a change

Semgrep runs on Python versions >= 3.8. Install one of these versions before proceeding. Because the Python and OCaml development paths are relatively independent, the instructions are divided into Python (semgrep-cli contributing) and OCaml (semgrep-core contributing). To fully build Semgrep from source, start at semgrep-core contributing. It directs you to semgrep-cli contributing when appropriate. Depending on what change you want to make, it might be simpler to build only semgrep-cli or only semgrep-core. For example, if you only want to modify Python code, you can skip installing OCaml by downloading binaries for the OCaml parts. Similarly, if you only want to modify OCaml code, you can work on semgrep-core/spacegrep directly. If you only want to build semgrep-cli, go straight to semgrep-cli contributing. Otherwise, follow the instructions in semgrep-core contributing. The following guide describes what each of semgrep-cli and semgrep-core controls.

Only `semgrep-cli`

The Python code for Semgrep performs pre-processing and post-processing work. You likely need to touch only semgrep-cli if you want to affect

How output is formatted
What files are scanned for each language
The message that is displayed

Go to semgrep-cli contributing

Only `semgrep-core`

The OCaml code for Semgrep performs all the parsing and matching work. You likely need to touch only semgrep-core if you want to

Fix a parse error
Fix a matching error
Improve Semgrep’s performance
Improve the parsing of a programming language

Go to semgrep-core contributing

Both `semgrep` and `semgrep-core`

There are some features that cross through both OCaml and Python code. You likely need to touch both semgrep-cli and semgrep-core if you want to

Address a Rule-defined fix error
Add support for a new language
Change error reporting

Go to semgrep-core contributing. It directs you to semgrep-cli contributing when appropriate.

Development workflow

Before each commit, Semgrep runs pre-commit to format files and check for basic linting issues. If you don’t have pre-commit installed, run the following command:

python -m pip install pre-commit

The pre-commit configuration uses Docker images. Install Docker before running pre-commit. Install the pre-commit hooks with the following command:

pre-commit install

To ensure pre-commit is working as expected, run the following command:

pre-commit run --all

Once pre-commit is working you may commit code and create pull requests as you would expect. Pull requests require approval of at least one maintainer and CI to be passing.

Explaining code

It’s important for code to be easy to maintain. This allows contributors to spend more time on new features rather than studying legacy code. As a general rule of thumb, assume that unwritten context is lost and forgotten. Useful context includes:

Why does this code exist?
What or who uses this code?
What does this code achieve?
Could this code be replaced by an off-the-shelf component? Why not?
Does it implement a formal specification or a well-known pattern? Where can readers learn more about it?

Start each source file with one comment that concisely answers these questions. Here’s a short example:

(*
   Generate unique names with a given prefix.
*)

It can be improved by explaining the code’s uses:

(*
   Generate unique names with a given prefix. This is used to
   name new grammar rules and new OCaml variables.
*)

Adding a changelog entry

Quick reference

Add a new file named like changelog.d/gh-1234.fixed that contains a single paragraph of Markdown text such as:

Fix emojis absorbed by the fleeb generator

File name format:

gh-1234.fixed
   ^^^^ ^^^^^
   |    |
   |    one of: "added", "changed", "fixed", "infra"
   GitHub issue or pull request ID

Valid changelog file suffixes are:

added - New features or other previously non-existing functionality
changed - Items that have changed the way Semgrep functions
fixed - Bug fixes or other improvements
infra - Workflow improvements or other non-code updates

When to add a changelog entry

If you contribute code that affects users, you must add an entry to the changelog, in the changelog.d folder. At each Semgrep release, these files are automatically gathered and formatted to produce release notes. A changelog entry is required if you are:

Adding new features or other previously non-existing functionality.
Including important changes in the way Semgrep functions.
Submitting bug fixes or other improvements.
Creating workflow improvements or other non-code updates.

A tool called towncrier is used for changelog management.

Troubleshooting pre-commit

On M1 Macs, some pre-commit tests may fail. If those checks are running in Docker containers (such as hadolint) and exit with code 137, they are hitting a memory limit. Running x86_64 images on an M1 Mac uses QEMU emulation, which can increase memory consumption. To fix this, raise the memory limit in Docker Desktop under Preferences > Resources. 8 GB should be sufficient.

Working with Git submodules

A submodule is a reference to a specific commit in another Git repository. This results in a subfolder containing a checkout of that repository at that particular commit. Submodules have a reputation of being tricky to use. To minimize problems, follow these guidelines:

When checking out a new branch or commit, update the submodules using the command git submodule update --init --recursive. Adding a shortcut to your shell can be useful. The following is a Bash function that lets you call gitup. It goes into your ~/.bashrc:

gitup() {
  echo "git submodule update --init --recursive"
  git submodule update --init --recursive
}

When modifying both a parent repo A and one of its submodules B, make one pull request for each (PR A, PR B).

i. Before merging PR B, make sure the branch on repo B is not lagging behind the main branch. This ensures that the submodule includes all the latest changes made by others.

ii. Make sure PR B is merged before PR A. This ensures that other developers pick up the changes on B when making their own changes.

iii. After merging PR A, check that submodule B is still up-to-date with respect to its main branch, especially if PR B was merged more than an hour ago. Good to know:
- Merging in B can be done with a merge commit or by squashing the commits.
- If squashing commits in B, you must know that the original commit referenced by A becomes orphaned when the branch is deleted but remains cached by Git for a while. This is usually sufficient to not require A to point to the newly squashed commit. If this turns out to be problematic in practice, Semgrep may disallow commit squashing in the future.

​File structure

​Code relationship

​Making a change

​Only semgrep-cli

​Only semgrep-core

​Both semgrep and semgrep-core

​Development workflow

​Explaining code

​Adding a changelog entry

​Quick reference

​When to add a changelog entry

​Troubleshooting pre-commit

​Working with Git submodules