projects/forgejo-agent

Watch

Fork 0

tests #264

New issue

Open

opened 2026-09-08 19:12:26 +00:00 by davide · 7 comments

davide commented

2026-09-08 19:12:26 +00:00

Owner

@ci-bot can you propose a testing strategy for the program? ideally i would like it to be as close as possible to the real thing, so i guess a forgejo instance running in actions/devcontainer.

For the js side i prefer the vitest library.

For now report only the plan and some example tests, no implementation

@ci-bot can you propose a testing strategy for the program? ideally i would like it to be as close as possible to the real thing, so i guess a forgejo instance running in actions/devcontainer. For the js side i prefer the vitest library. For now report only the plan and some example tests, no implementation

ci-bot commented

2026-09-08 19:15:08 +00:00

Member

Testing strategy for forgejo-agent

Goal

Test the program as close as possible to how it runs in production: a real Forgejo instance, the real binary, real git operations, real API calls. The only non-deterministic part is the LLM, so we isolate it with a scripted session for CI and keep a real-LLM smoke test for on-demand runs.

Layered approach

Layer	What	Tooling	Speed	Where
1. Unit	pure functions (prompt, schemas, context parsing, tool param mapping)	Vitest	ms	every PR
2. Integration	Forgejo client + tools against a mock HTTP server serving the real payloads	Vitest + `node:http`	ms	every PR
3. E2E	full pipeline against a real Forgejo instance (docker), scripted agent session	Vitest + docker compose	~1–2 min	every PR (docker runner)
4. Smoke	real binary + real LLM against the test instance	Vitest + docker compose	minutes, costs tokens	nightly / on demand

Layer 0 — Tooling

Add vitest + @vitest/coverage-v8 as devDependencies.
Scripts: test (vitest run), test:unit, test:integration, test:e2e, test:coverage.
vitest.config.ts: node environment, setupFiles that set the env vars src/forgejo/fetch.ts reads at import time (FORGEJO_API_URL, CTX_AUTH_TOKEN, CTX_AUTH_USERNAME) — otherwise importing the client throws.
Layout:

test/
├── setup.ts              # env vars for unit/integration
├── fixtures/             # reuse examples/event_payloads directly
├── unit/                 # colocated src/*.test.ts or mirrored
├── integration/
└── e2e/

Layer 1 — Unit tests

src/prompt.test.ts — buildPrompt for the 4 event types; assert branch instructions, tool instructions, issue/PR content, comment ordering, empty body.
src/schemas.test.ts — TypeBox validation; contract tests that validate every file in examples/event_payloads/ against the schemas (keeps fixtures in sync with the Forgejo version).
src/context.test.ts — env parsing; missing/invalid env throws; repository name pattern; forgejo client mocked.
src/tools.test.ts — default repository/issueId fallback; create-pr-review maps side/line → old_position/new_position; parameter schema validation.
src/git.ts — getLatestCommitId; checkoutRepository against a local bare repo.

Layer 2 — Integration tests

A tiny in-process HTTP server (node:http) mimicking the Forgejo REST API, serving the real payloads from examples/event_payloads/.
Test the client: URL construction, method, auth header, content-type, response schema validation, error on >= 400.
Test getEventContext end-to-end: repository + issue + PR + comments.
Test each tool against the mock server (close-issue, create-issue, create-issue-comment, create-pr, create-pr-review).

Layer 3 — E2E with a real Forgejo (the centerpiece)

Instance — docker-compose.test.yaml:

services:
  forgejo:
    image: codeberg.org/forgejo/forgejo:15.0.3   # pin to the production version
    environment:
      FORGEJO__server__ROOT_URL: http://forgejo:3000/
      FORGEJO__server__HTTP_PORT: "3000"
      FORGEJO__security__INSTALL_LOCK: "true"
      FORGEJO__service__DISABLE_REGISTRATION: "true"
    ports: ["3000:3000"]
    volumes: [forgejo-data:/data]
    healthcheck:
      test: ["CMD", "curl", "-sf", "http://localhost:3000/api/v1/version"]
      interval: 2s
      timeout: 2s
      retries: 30
volumes:
  forgejo-data: {}

Provisioning — scripts/e2e-setup.sh:

wait for /api/v1/version;
create admin + bot users via the Forgejo CLI inside the container (forgejo admin user create ...);
generate a token for the bot (forgejo admin user generate-access-token);
create the test repo (POST /user/repos with auto_init: true);
seed issues/PRs/comments via the API.

Deterministic agent session — the only non-deterministic part is the LLM. Add a small test hook in main.ts: when CTX_TEST_SCRIPT is set, replace the Pi session with a scripted one that replays a fixed sequence of tool calls (e.g. "call create-issue-comment with body X"). This exercises everything else for real: env parsing, API reads, git clone/checkout, prompt building, tool execution, API writes.

Test — run the built binary in a fresh temp dir with env vars pointing at the test instance, then assert the outcome via the API (comment exists, issue closed, PR created, review submitted).

Devcontainer — add the same forgejo service to .devcontainer/docker-compose.yaml so pnpm test:e2e works locally with docker compose up.

Layer 4 — Smoke tests (real LLM)

Same instance, but run the real agent (real model) and assert loosely: "a comment by ci-bot exists on the issue".
Slow and costs tokens → nightly or on-demand, not on every PR.

CI integration

# .forgejo/workflows/test.yaml
name: test.yaml
"on":
  pull_request:
    branches: [main]

jobs:
  unit-and-integration:
    container: { image: git.zarantonello.dev/infra/ci-pnpm:v1.1.1@sha256:... }
    runs-on: srv-generic
    steps:
      - run: git clone ... && git checkout ${{ forge.sha }}
      - run: pnpm install
      - run: pnpm test:unit
      - run: pnpm test:integration

  e2e:
    runs-on: srv-home   # runner with docker access
    steps:
      - run: git clone ... && git checkout ${{ forge.sha }}
      - run: pnpm install
      - run: pnpm build
      - run: docker compose -f docker-compose.test.yaml up -d --wait forgejo
      - run: bash scripts/e2e-setup.sh
      - run: pnpm test:e2e
      - run: docker compose -f docker-compose.test.yaml down

Alternative: use the services: keyword (act_runner supports it like GitHub Actions), but then provisioning can't use docker exec — you'd need a custom image that provisions on first boot, or a pre-seeded data volume. The compose approach above is simpler and identical to the devcontainer experience.

Example tests

1. `src/prompt.test.ts` — pure, no mocks

import { describe, expect, it } from "vitest";
import { buildPrompt } from "../src/prompt.ts";

const authCtx = { token: "t", username: "ci-bot" };

const issueEventCtx = {
  repository: { full_name: "owner/repo", default_branch: "main" },
  event: {
    type: "issue",
    number: 1,
    user: { username: "davide" },
    title: "tests",
    body: "propose a testing strategy",
    state: "open",
    pull_request: null,
    name: "issues_opened",
    comments: [],
  },
};

describe("buildPrompt", () => {
  it("builds an issue prompt with branch and tool instructions", () => {
    const prompt = buildPrompt(authCtx, issueEventCtx);
    expect(prompt).toContain("Forgejo issue number 1");
    expect(prompt).toContain("repository owner/repo");
    expect(prompt).toContain("default branch: main");
    expect(prompt).toContain("create-pr");
    expect(prompt).toContain("create-issue-comment");
    expect(prompt).toContain("### start issue content from user davide ###");
    expect(prompt).toContain("propose a testing strategy");
  });

  it("appends comments in order", () => {
    const ctx = {
      ...issueEventCtx,
      event: {
        ...issueEventCtx.event,
        comments: [
          { user: { username: "davide" }, body: "first", id: 1 },
          { user: { username: "ci-bot" }, body: "second", id: 2 },
        ],
      },
    };
    const prompt = buildPrompt(authCtx, ctx);
    expect(prompt.indexOf("first")).toBeLessThan(prompt.indexOf("second"));
  });
});

2. `src/schemas.test.ts` — contract tests against the real payloads

import { readFileSync } from "node:fs";
import { describe, expect, it } from "vitest";
import Value from "typebox/value";
import { eventNameSchema, issueSchema, pullRequestSchema } from "../src/schemas.ts";

describe("eventNameSchema", () => {
  it("accepts the four supported events", () => {
    for (const name of ["issue_comment_created", "issues_opened", "pull_request_opened", "pull_request_review_requested"]) {
      expect(Value.Check(eventNameSchema, name)).toBe(true);
    }
  });

  it("rejects unknown events", () => {
    expect(Value.Check(eventNameSchema, "issues_closed")).toBe(false);
  });
});

describe("schemas against real payloads", () => {
  it("validates the issue payload from examples/event_payloads", () => {
    const payload = JSON.parse(readFileSync("examples/event_payloads/issue_opened.json", "utf8"));
    expect(Value.Check(issueSchema, payload.issue)).toBe(true);
  });

  it("validates the pull request payload", () => {
    const payload = JSON.parse(readFileSync("examples/event_payloads/pull_request_opened.json", "utf8"));
    expect(Value.Check(pullRequestSchema, payload.pull_request)).toBe(true);
  });
});

3. `src/forgejo/index.test.ts` — client with stubbed fetch

import { afterEach, describe, expect, it, vi } from "vitest";
import { getIssue } from "../src/forgejo/index.ts";

describe("getIssue", () => {
  afterEach(() => vi.unstubAllGlobals());

  it("calls the endpoint with auth and parses the response", async () => {
    const fetchMock = vi.fn().mockResolvedValue({
      status: 200,
      json: async () => ({
        number: 1,
        user: { username: "davide" },
        title: "t",
        body: "b",
        state: "open",
        pull_request: null,
      }),
    });
    vi.stubGlobal("fetch", fetchMock);

    const issue = await getIssue("owner/repo", 1);

    expect(fetchMock).toHaveBeenCalledWith(
      "http://forgejo:3000/api/v1/repos/owner/repo/issues/1",
      expect.objectContaining({
        method: "GET",
        headers: expect.objectContaining({ authorization: "Bearer test-token" }),
      }),
    );
    expect(issue.number).toBe(1);
  });

  it("throws on error status", async () => {
    vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ status: 404, text: async () => "not found" }));
    await expect(getIssue("owner/repo", 1)).rejects.toThrow("fetch failed");
  });
});

4. `src/tools.test.ts` — review comment position mapping

import { beforeEach, describe, expect, it, vi } from "vitest";

vi.mock("../src/forgejo/index.ts", () => ({
  patchIssue: vi.fn(),
  postIssue: vi.fn(),
  postIssueComment: vi.fn(),
  postPrReview: vi.fn(),
  postPullRequest: vi.fn(),
}));
vi.mock("../src/git.ts", () => ({ getLatestCommitId: () => "abc123" }));

import { postPrReview } from "../src/forgejo/index.ts";
import { createCreatePrReviewTool } from "../src/tools.ts";

describe("createCreatePrReviewTool", () => {
  beforeEach(() => vi.mocked(postPrReview).mockResolvedValue({ id: 7, body: "ok" }));

  it("maps HEAD/BASE comments to new/old positions", async () => {
    const tool = createCreatePrReviewTool("owner/repo", 3);
    await tool.execute("call-1", {
      body: "review",
      verdict: "APPROVED",
      comments: [
        { body: "on new code", path: "src/main.ts", line: 10, side: "HEAD" },
        { body: "on old code", path: "src/main.ts", line: 5, side: "BASE" },
      ],
    });

    expect(postPrReview).toHaveBeenCalledWith("owner/repo", 3, {
      body: "review",
      event: "APPROVED",
      commit_id: "abc123",
      comments: [
        { body: "on new code", path: "src/main.ts", new_position: 10, old_position: 0 },
        { body: "on old code", path: "src/main.ts", new_position: 0, old_position: 5 },
      ],
    });
  });
});

5. E2E sketch — real Forgejo, scripted session

// test/e2e/agent.test.ts
import { execFileSync } from "node:child_process";
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { beforeAll, describe, expect, it } from "vitest";
import { getIssueComments } from "../../src/forgejo/index.ts";

const API = process.env.FORGEJO_API_URL!; // http://forgejo:3000/api/v1
const TOKEN = process.env.E2E_BOT_TOKEN!;
const REPO = process.env.E2E_REPO!; // e.g. e2e/agent-test

describe("agent end-to-end against a real Forgejo", () => {
  beforeAll(() => {
    // provision repo + issue + comment mentioning @ci-bot
    execFileSync("bash", ["scripts/e2e-setup.sh", API, TOKEN, REPO]);
  });

  it("responds to an issue comment mentioning the bot", async () => {
    const workdir = mkdtempSync(join(tmpdir(), "agent-e2e-"));
    const env = {
      ...process.env,
      FORGEJO_API_URL: API,
      CTX_AUTH_TOKEN: TOKEN,
      CTX_AUTH_USERNAME: "ci-bot",
      FORGEJO_REPOSITORY: REPO,
      CTX_ISSUE_NUMBER: "1",
      CTX_EVENT_NAME: "issue_comment_created",
      CTX_TEST_SCRIPT: "reply-with-comment", // deterministic scripted session
    };

    execFileSync("dist/forgejo-agent", [], { env, cwd: workdir });

    const comments = await getIssueComments(REPO, 1);
    expect(comments.some((c) => c.user.username === "ci-bot")).toBe(true);
  });
});

Open questions

Forgejo version — pin 15.0.3 (the version the payloads were saved on), or track production?
E2E runner — dedicated runner with docker access (compose approach) vs services: keyword (needs a provisioning image)?
Coverage threshold — target e.g. 80% on src/?
Test hook — OK to add the CTX_TEST_SCRIPT scripted-session hook to main.ts? It's the key to deterministic E2E.
Real-LLM smoke — worth a nightly job, or skip for now?

# Testing strategy for forgejo-agent ## Goal Test the program as close as possible to how it runs in production: a **real Forgejo instance**, the **real binary**, **real git operations**, **real API calls**. The only non-deterministic part is the LLM, so we isolate it with a scripted session for CI and keep a real-LLM smoke test for on-demand runs. ## Layered approach | Layer | What | Tooling | Speed | Where | |---|---|---|---|---| | 1. Unit | pure functions (prompt, schemas, context parsing, tool param mapping) | Vitest | ms | every PR | | 2. Integration | Forgejo client + tools against a mock HTTP server serving the real payloads | Vitest + `node:http` | ms | every PR | | 3. E2E | full pipeline against a real Forgejo instance (docker), scripted agent session | Vitest + docker compose | ~1–2 min | every PR (docker runner) | | 4. Smoke | real binary + real LLM against the test instance | Vitest + docker compose | minutes, costs tokens | nightly / on demand | ## Layer 0 — Tooling - Add `vitest` + `@vitest/coverage-v8` as devDependencies. - Scripts: `test` (`vitest run`), `test:unit`, `test:integration`, `test:e2e`, `test:coverage`. - `vitest.config.ts`: node environment, `setupFiles` that set the env vars `src/forgejo/fetch.ts` reads **at import time** (`FORGEJO_API_URL`, `CTX_AUTH_TOKEN`, `CTX_AUTH_USERNAME`) — otherwise importing the client throws. - Layout: ``` test/ ├── setup.ts # env vars for unit/integration ├── fixtures/ # reuse examples/event_payloads directly ├── unit/ # colocated src/*.test.ts or mirrored ├── integration/ └── e2e/ ``` ## Layer 1 — Unit tests - `src/prompt.test.ts` — `buildPrompt` for the 4 event types; assert branch instructions, tool instructions, issue/PR content, comment ordering, empty body. - `src/schemas.test.ts` — TypeBox validation; **contract tests** that validate every file in `examples/event_payloads/` against the schemas (keeps fixtures in sync with the Forgejo version). - `src/context.test.ts` — env parsing; missing/invalid env throws; repository name pattern; forgejo client mocked. - `src/tools.test.ts` — default repository/issueId fallback; `create-pr-review` maps `side`/`line` → `old_position`/`new_position`; parameter schema validation. - `src/git.ts` — `getLatestCommitId`; `checkoutRepository` against a local bare repo. ## Layer 2 — Integration tests - A tiny in-process HTTP server (`node:http`) mimicking the Forgejo REST API, serving the real payloads from `examples/event_payloads/`. - Test the client: URL construction, method, auth header, content-type, response schema validation, error on `>= 400`. - Test `getEventContext` end-to-end: repository + issue + PR + comments. - Test each tool against the mock server (close-issue, create-issue, create-issue-comment, create-pr, create-pr-review). ## Layer 3 — E2E with a real Forgejo (the centerpiece) **Instance** — `docker-compose.test.yaml`: ```yaml services: forgejo: image: codeberg.org/forgejo/forgejo:15.0.3 # pin to the production version environment: FORGEJO__server__ROOT_URL: http://forgejo:3000/ FORGEJO__server__HTTP_PORT: "3000" FORGEJO__security__INSTALL_LOCK: "true" FORGEJO__service__DISABLE_REGISTRATION: "true" ports: ["3000:3000"] volumes: [forgejo-data:/data] healthcheck: test: ["CMD", "curl", "-sf", "http://localhost:3000/api/v1/version"] interval: 2s timeout: 2s retries: 30 volumes: forgejo-data: {} ``` **Provisioning** — `scripts/e2e-setup.sh`: 1. wait for `/api/v1/version`; 2. create admin + bot users via the Forgejo CLI inside the container (`forgejo admin user create ...`); 3. generate a token for the bot (`forgejo admin user generate-access-token`); 4. create the test repo (`POST /user/repos` with `auto_init: true`); 5. seed issues/PRs/comments via the API. **Deterministic agent session** — the only non-deterministic part is the LLM. Add a small test hook in `main.ts`: when `CTX_TEST_SCRIPT` is set, replace the Pi session with a scripted one that replays a fixed sequence of tool calls (e.g. "call `create-issue-comment` with body X"). This exercises everything else for real: env parsing, API reads, git clone/checkout, prompt building, tool execution, API writes. **Test** — run the built binary in a fresh temp dir with env vars pointing at the test instance, then assert the outcome via the API (comment exists, issue closed, PR created, review submitted). **Devcontainer** — add the same forgejo service to `.devcontainer/docker-compose.yaml` so `pnpm test:e2e` works locally with `docker compose up`. ## Layer 4 — Smoke tests (real LLM) - Same instance, but run the real agent (real model) and assert loosely: "a comment by ci-bot exists on the issue". - Slow and costs tokens → nightly or on-demand, not on every PR. ## CI integration ```yaml # .forgejo/workflows/test.yaml name: test.yaml "on": pull_request: branches: [main] jobs: unit-and-integration: container: { image: git.zarantonello.dev/infra/ci-pnpm:v1.1.1@sha256:... } runs-on: srv-generic steps: - run: git clone ... && git checkout ${{ forge.sha }} - run: pnpm install - run: pnpm test:unit - run: pnpm test:integration e2e: runs-on: srv-home # runner with docker access steps: - run: git clone ... && git checkout ${{ forge.sha }} - run: pnpm install - run: pnpm build - run: docker compose -f docker-compose.test.yaml up -d --wait forgejo - run: bash scripts/e2e-setup.sh - run: pnpm test:e2e - run: docker compose -f docker-compose.test.yaml down ``` Alternative: use the `services:` keyword (act_runner supports it like GitHub Actions), but then provisioning can't use `docker exec` — you'd need a custom image that provisions on first boot, or a pre-seeded data volume. The compose approach above is simpler and identical to the devcontainer experience. ## Example tests ### 1. `src/prompt.test.ts` — pure, no mocks ```ts import { describe, expect, it } from "vitest"; import { buildPrompt } from "../src/prompt.ts"; const authCtx = { token: "t", username: "ci-bot" }; const issueEventCtx = { repository: { full_name: "owner/repo", default_branch: "main" }, event: { type: "issue", number: 1, user: { username: "davide" }, title: "tests", body: "propose a testing strategy", state: "open", pull_request: null, name: "issues_opened", comments: [], }, }; describe("buildPrompt", () => { it("builds an issue prompt with branch and tool instructions", () => { const prompt = buildPrompt(authCtx, issueEventCtx); expect(prompt).toContain("Forgejo issue number 1"); expect(prompt).toContain("repository owner/repo"); expect(prompt).toContain("default branch: main"); expect(prompt).toContain("create-pr"); expect(prompt).toContain("create-issue-comment"); expect(prompt).toContain("### start issue content from user davide ###"); expect(prompt).toContain("propose a testing strategy"); }); it("appends comments in order", () => { const ctx = { ...issueEventCtx, event: { ...issueEventCtx.event, comments: [ { user: { username: "davide" }, body: "first", id: 1 }, { user: { username: "ci-bot" }, body: "second", id: 2 }, ], }, }; const prompt = buildPrompt(authCtx, ctx); expect(prompt.indexOf("first")).toBeLessThan(prompt.indexOf("second")); }); }); ``` ### 2. `src/schemas.test.ts` — contract tests against the real payloads ```ts import { readFileSync } from "node:fs"; import { describe, expect, it } from "vitest"; import Value from "typebox/value"; import { eventNameSchema, issueSchema, pullRequestSchema } from "../src/schemas.ts"; describe("eventNameSchema", () => { it("accepts the four supported events", () => { for (const name of ["issue_comment_created", "issues_opened", "pull_request_opened", "pull_request_review_requested"]) { expect(Value.Check(eventNameSchema, name)).toBe(true); } }); it("rejects unknown events", () => { expect(Value.Check(eventNameSchema, "issues_closed")).toBe(false); }); }); describe("schemas against real payloads", () => { it("validates the issue payload from examples/event_payloads", () => { const payload = JSON.parse(readFileSync("examples/event_payloads/issue_opened.json", "utf8")); expect(Value.Check(issueSchema, payload.issue)).toBe(true); }); it("validates the pull request payload", () => { const payload = JSON.parse(readFileSync("examples/event_payloads/pull_request_opened.json", "utf8")); expect(Value.Check(pullRequestSchema, payload.pull_request)).toBe(true); }); }); ``` ### 3. `src/forgejo/index.test.ts` — client with stubbed fetch ```ts import { afterEach, describe, expect, it, vi } from "vitest"; import { getIssue } from "../src/forgejo/index.ts"; describe("getIssue", () => { afterEach(() => vi.unstubAllGlobals()); it("calls the endpoint with auth and parses the response", async () => { const fetchMock = vi.fn().mockResolvedValue({ status: 200, json: async () => ({ number: 1, user: { username: "davide" }, title: "t", body: "b", state: "open", pull_request: null, }), }); vi.stubGlobal("fetch", fetchMock); const issue = await getIssue("owner/repo", 1); expect(fetchMock).toHaveBeenCalledWith( "http://forgejo:3000/api/v1/repos/owner/repo/issues/1", expect.objectContaining({ method: "GET", headers: expect.objectContaining({ authorization: "Bearer test-token" }), }), ); expect(issue.number).toBe(1); }); it("throws on error status", async () => { vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ status: 404, text: async () => "not found" })); await expect(getIssue("owner/repo", 1)).rejects.toThrow("fetch failed"); }); }); ``` ### 4. `src/tools.test.ts` — review comment position mapping ```ts import { beforeEach, describe, expect, it, vi } from "vitest"; vi.mock("../src/forgejo/index.ts", () => ({ patchIssue: vi.fn(), postIssue: vi.fn(), postIssueComment: vi.fn(), postPrReview: vi.fn(), postPullRequest: vi.fn(), })); vi.mock("../src/git.ts", () => ({ getLatestCommitId: () => "abc123" })); import { postPrReview } from "../src/forgejo/index.ts"; import { createCreatePrReviewTool } from "../src/tools.ts"; describe("createCreatePrReviewTool", () => { beforeEach(() => vi.mocked(postPrReview).mockResolvedValue({ id: 7, body: "ok" })); it("maps HEAD/BASE comments to new/old positions", async () => { const tool = createCreatePrReviewTool("owner/repo", 3); await tool.execute("call-1", { body: "review", verdict: "APPROVED", comments: [ { body: "on new code", path: "src/main.ts", line: 10, side: "HEAD" }, { body: "on old code", path: "src/main.ts", line: 5, side: "BASE" }, ], }); expect(postPrReview).toHaveBeenCalledWith("owner/repo", 3, { body: "review", event: "APPROVED", commit_id: "abc123", comments: [ { body: "on new code", path: "src/main.ts", new_position: 10, old_position: 0 }, { body: "on old code", path: "src/main.ts", new_position: 0, old_position: 5 }, ], }); }); }); ``` ### 5. E2E sketch — real Forgejo, scripted session ```ts // test/e2e/agent.test.ts import { execFileSync } from "node:child_process"; import { mkdtempSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { beforeAll, describe, expect, it } from "vitest"; import { getIssueComments } from "../../src/forgejo/index.ts"; const API = process.env.FORGEJO_API_URL!; // http://forgejo:3000/api/v1 const TOKEN = process.env.E2E_BOT_TOKEN!; const REPO = process.env.E2E_REPO!; // e.g. e2e/agent-test describe("agent end-to-end against a real Forgejo", () => { beforeAll(() => { // provision repo + issue + comment mentioning @ci-bot execFileSync("bash", ["scripts/e2e-setup.sh", API, TOKEN, REPO]); }); it("responds to an issue comment mentioning the bot", async () => { const workdir = mkdtempSync(join(tmpdir(), "agent-e2e-")); const env = { ...process.env, FORGEJO_API_URL: API, CTX_AUTH_TOKEN: TOKEN, CTX_AUTH_USERNAME: "ci-bot", FORGEJO_REPOSITORY: REPO, CTX_ISSUE_NUMBER: "1", CTX_EVENT_NAME: "issue_comment_created", CTX_TEST_SCRIPT: "reply-with-comment", // deterministic scripted session }; execFileSync("dist/forgejo-agent", [], { env, cwd: workdir }); const comments = await getIssueComments(REPO, 1); expect(comments.some((c) => c.user.username === "ci-bot")).toBe(true); }); }); ``` ## Open questions 1. **Forgejo version** — pin `15.0.3` (the version the payloads were saved on), or track production? 2. **E2E runner** — dedicated runner with docker access (compose approach) vs `services:` keyword (needs a provisioning image)? 3. **Coverage threshold** — target e.g. 80% on `src/`? 4. **Test hook** — OK to add the `CTX_TEST_SCRIPT` scripted-session hook to `main.ts`? It's the key to deterministic E2E. 5. **Real-LLM smoke** — worth a nightly job, or skip for now?

ci-bot commented

2026-09-08 19:16:55 +00:00

Member

Testing strategy for forgejo-agent

I went through the current codebase (src/context.ts, src/prompt.ts, src/schemas.ts, src/forgejo/*, src/tools.ts, src/git.ts, src/main.ts, examples/event_payloads/) and the plan below is verified against the real code — function names, env vars, payload shapes, and the binary name (dist/forgejo-agent, from sea.json) all match.

Goal

Test the program as close as possible to production: a real Forgejo instance, the real binary, real git operations, real API calls. The only non-deterministic part is the LLM, so we isolate it with a scripted session for CI and keep a real-LLM smoke test for on-demand/nightly runs.

Layered approach

Layer	What	Tooling	Speed	Where
1. Unit	pure functions (prompt, schemas, context parsing, tool param mapping)	Vitest	ms	every PR
2. Integration	Forgejo client + tools against a mock HTTP server serving the real payloads	Vitest + `node:http`	ms	every PR
3. E2E	full pipeline against a real Forgejo instance (docker), scripted agent session	Vitest + docker compose	~1–2 min	every PR (docker runner)
4. Smoke	real binary + real LLM against the test instance	Vitest + docker compose	minutes, costs tokens	nightly / on demand

Layer 0 — Tooling

Add vitest + @vitest/coverage-v8 as devDependencies.
Scripts: test (vitest run), test:unit, test:integration, test:e2e, test:coverage.
vitest.config.ts: node environment, setupFiles that set the env vars src/forgejo/fetch.ts reads at import time — FORGEJO_API_URL, CTX_AUTH_TOKEN, CTX_AUTH_USERNAME (verified: fetch.ts evaluates process.env.FORGEJO_API_URL and getAuthContext() at module load, so importing the client without them throws). src/context.ts additionally reads FORGEJO_REPOSITORY, CTX_ISSUE_NUMBER, CTX_EVENT_NAME inside functions.
Layout:

test/
├── setup.ts              # env vars for unit/integration
├── fixtures/             # reuse examples/event_payloads directly
├── unit/                 # colocated src/*.test.ts or mirrored
├── integration/
└── e2e/

Layer 1 — Unit tests

src/prompt.test.ts — buildPrompt for the 4 event types; assert branch instructions, tool instructions, issue/PR content, comment ordering, empty body.
src/schemas.test.ts — TypeBox validation; contract tests that validate every file in examples/event_payloads/ against the schemas (keeps fixtures in sync with the Forgejo version). Verified: issue_opened.json and pull_request_opened.json contain all fields required by issueSchema/pullRequestSchema (number, user.username, title, body, state, pull_request, head.label, base.label).
src/context.test.ts — env parsing; missing/invalid env throws; repository name pattern; forgejo client mocked.
src/tools.test.ts — default repository/issueId fallback; create-pr-review maps side/line → old_position/new_position (verified in tools.ts: new_position: c.side === "HEAD" ? c.line : 0, old_position: c.side === "BASE" ? c.line : 0); parameter schema validation.
src/git.ts — getLatestCommitId; checkoutRepository against a local bare repo.

Layer 2 — Integration tests

A tiny in-process HTTP server (node:http) mimicking the Forgejo REST API, serving the real payloads from examples/event_payloads/.
Test the client: URL construction (apiUrl + /repos/{owner}/{repo}/issues/{id}), method, Authorization: Bearer <token> header, content-type, response schema validation, error on >= 400 (verified: throws fetch failed: <body>).
Test getEventContext end-to-end: repository + issue + PR + comments.
Test each tool against the mock server (close-issue, create-issue, create-issue-comment, create-pr, create-pr-review).

Layer 3 — E2E with a real Forgejo (the centerpiece)

Instance — docker-compose.test.yaml:

services:
  forgejo:
    image: codeberg.org/forgejo/forgejo:15.0.3   # pin to the production version
    environment:
      FORGEJO__server__ROOT_URL: http://forgejo:3000/
      FORGEJO__server__HTTP_PORT: "3000"
      FORGEJO__security__INSTALL_LOCK: "true"
      FORGEJO__service__DISABLE_REGISTRATION: "true"
    ports: ["3000:3000"]
    volumes: [forgejo-data:/data]
    healthcheck:
      test: ["CMD", "curl", "-sf", "http://localhost:3000/api/v1/version"]
      interval: 2s
      timeout: 2s
      retries: 30
volumes:
  forgejo-data: {}

Provisioning — scripts/e2e-setup.sh:

wait for /api/v1/version;
create admin + bot users via the Forgejo CLI inside the container (forgejo admin user create ...);
generate a token for the bot (forgejo admin user generate-access-token);
create the test repo (POST /user/repos with auto_init: true);
seed issues/PRs/comments via the API.

Test — run the built binary in a fresh temp dir with env vars pointing at the test instance, then assert the outcome via the API (comment exists, issue closed, PR created, review submitted).

Devcontainer — add the same forgejo service to .devcontainer/docker-compose.yaml (currently only has the workspace service) so pnpm test:e2e works locally with docker compose up.

Layer 4 — Smoke tests (real LLM)

Same instance, but run the real agent (real model) and assert loosely: "a comment by ci-bot exists on the issue".
Slow and costs tokens → nightly or on-demand, not on every PR.

CI integration

# .forgejo/workflows/test.yaml
name: test.yaml
"on":
  pull_request:
    branches: [main]

jobs:
  unit-and-integration:
    container: { image: git.zarantonello.dev/infra/ci-pnpm:v1.1.1@sha256:... }
    runs-on: srv-generic
    steps:
      - run: git clone ... && git checkout ${{ forge.sha }}
      - run: pnpm install
      - run: pnpm test:unit
      - run: pnpm test:integration

  e2e:
    runs-on: srv-home   # runner with docker access
    steps:
      - run: git clone ... && git checkout ${{ forge.sha }}
      - run: pnpm install
      - run: pnpm build
      - run: docker compose -f docker-compose.test.yaml up -d --wait forgejo
      - run: bash scripts/e2e-setup.sh
      - run: pnpm test:e2e
      - run: docker compose -f docker-compose.test.yaml down

This mirrors the existing workflows (same clone pattern, same ci-pnpm image, srv-home already runs the agent job). Alternative: the services: keyword (act_runner supports it), but then provisioning can't use docker exec — you'd need a custom image that provisions on first boot, or a pre-seeded data volume. The compose approach is simpler and identical to the devcontainer experience.

Example tests

1. `src/prompt.test.ts` — pure, no mocks

import { describe, expect, it } from "vitest";
import { buildPrompt } from "../src/prompt.ts";

const authCtx = { token: "t", username: "ci-bot" };

const issueEventCtx = {
  repository: { full_name: "owner/repo", default_branch: "main" },
  event: {
    type: "issue",
    number: 1,
    user: { username: "davide" },
    title: "tests",
    body: "propose a testing strategy",
    state: "open",
    pull_request: null,
    name: "issues_opened",
    comments: [],
  },
};

describe("buildPrompt", () => {
  it("builds an issue prompt with branch and tool instructions", () => {
    const prompt = buildPrompt(authCtx, issueEventCtx);
    expect(prompt).toContain("Forgejo issue number 1");
    expect(prompt).toContain("repository owner/repo");
    expect(prompt).toContain("default branch: main");
    expect(prompt).toContain("create-pr");
    expect(prompt).toContain("create-issue-comment");
    expect(prompt).toContain("### start issue content from user davide ###");
    expect(prompt).toContain("propose a testing strategy");
  });

  it("appends comments in order", () => {
    const ctx = {
      ...issueEventCtx,
      event: {
        ...issueEventCtx.event,
        comments: [
          { user: { username: "davide" }, body: "first", id: 1 },
          { user: { username: "ci-bot" }, body: "second", id: 2 },
        ],
      },
    };
    const prompt = buildPrompt(authCtx, ctx);
    expect(prompt.indexOf("first")).toBeLessThan(prompt.indexOf("second"));
  });
});

2. `src/schemas.test.ts` — contract tests against the real payloads

import { readFileSync } from "node:fs";
import { describe, expect, it } from "vitest";
import Value from "typebox/value";
import { eventNameSchema, issueSchema, pullRequestSchema } from "../src/schemas.ts";

describe("eventNameSchema", () => {
  it("accepts the four supported events", () => {
    for (const name of ["issue_comment_created", "issues_opened", "pull_request_opened", "pull_request_review_requested"]) {
      expect(Value.Check(eventNameSchema, name)).toBe(true);
    }
  });

  it("rejects unknown events", () => {
    expect(Value.Check(eventNameSchema, "issues_closed")).toBe(false);
  });
});

describe("schemas against real payloads", () => {
  it("validates the issue payload from examples/event_payloads", () => {
    const payload = JSON.parse(readFileSync("examples/event_payloads/issue_opened.json", "utf8"));
    expect(Value.Check(issueSchema, payload.issue)).toBe(true);
  });

  it("validates the pull request payload", () => {
    const payload = JSON.parse(readFileSync("examples/event_payloads/pull_request_opened.json", "utf8"));
    expect(Value.Check(pullRequestSchema, payload.pull_request)).toBe(true);
  });
});

3. `src/forgejo/index.test.ts` — client with stubbed fetch

import { afterEach, describe, expect, it, vi } from "vitest";
import { getIssue } from "../src/forgejo/index.ts";

describe("getIssue", () => {
  afterEach(() => vi.unstubAllGlobals());

  it("calls the endpoint with auth and parses the response", async () => {
    const fetchMock = vi.fn().mockResolvedValue({
      status: 200,
      json: async () => ({
        number: 1,
        user: { username: "davide" },
        title: "t",
        body: "b",
        state: "open",
        pull_request: null,
      }),
    });
    vi.stubGlobal("fetch", fetchMock);

    const issue = await getIssue("owner/repo", 1);

    expect(fetchMock).toHaveBeenCalledWith(
      "http://forgejo:3000/api/v1/repos/owner/repo/issues/1",
      expect.objectContaining({
        method: "GET",
        headers: expect.objectContaining({ authorization: "Bearer test-token" }),
      }),
    );
    expect(issue.number).toBe(1);
  });

  it("throws on error status", async () => {
    vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ status: 404, text: async () => "not found" }));
    await expect(getIssue("owner/repo", 1)).rejects.toThrow("fetch failed");
  });
});

4. `src/tools.test.ts` — review comment position mapping

import { beforeEach, describe, expect, it, vi } from "vitest";

vi.mock("../src/forgejo/index.ts", () => ({
  patchIssue: vi.fn(),
  postIssue: vi.fn(),
  postIssueComment: vi.fn(),
  postPrReview: vi.fn(),
  postPullRequest: vi.fn(),
}));
vi.mock("../src/git.ts", () => ({ getLatestCommitId: () => "abc123" }));

import { postPrReview } from "../src/forgejo/index.ts";
import { createCreatePrReviewTool } from "../src/tools.ts";

describe("createCreatePrReviewTool", () => {
  beforeEach(() => vi.mocked(postPrReview).mockResolvedValue({ id: 7, body: "ok" }));

  it("maps HEAD/BASE comments to new/old positions", async () => {
    const tool = createCreatePrReviewTool("owner/repo", 3);
    await tool.execute("call-1", {
      body: "review",
      verdict: "APPROVED",
      comments: [
        { body: "on new code", path: "src/main.ts", line: 10, side: "HEAD" },
        { body: "on old code", path: "src/main.ts", line: 5, side: "BASE" },
      ],
    });

    expect(postPrReview).toHaveBeenCalledWith("owner/repo", 3, {
      body: "review",
      event: "APPROVED",
      commit_id: "abc123",
      comments: [
        { body: "on new code", path: "src/main.ts", new_position: 10, old_position: 0 },
        { body: "on old code", path: "src/main.ts", new_position: 0, old_position: 5 },
      ],
    });
  });
});

5. E2E sketch — real Forgejo, scripted session

// test/e2e/agent.test.ts
import { execFileSync } from "node:child_process";
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { beforeAll, describe, expect, it } from "vitest";
import { getIssueComments } from "../../src/forgejo/index.ts";

const API = process.env.FORGEJO_API_URL!; // http://forgejo:3000/api/v1
const TOKEN = process.env.E2E_BOT_TOKEN!;
const REPO = process.env.E2E_REPO!; // e.g. e2e/agent-test

describe("agent end-to-end against a real Forgejo", () => {
  beforeAll(() => {
    // provision repo + issue + comment mentioning @ci-bot
    execFileSync("bash", ["scripts/e2e-setup.sh", API, TOKEN, REPO]);
  });

  it("responds to an issue comment mentioning the bot", async () => {
    const workdir = mkdtempSync(join(tmpdir(), "agent-e2e-"));
    const env = {
      ...process.env,
      FORGEJO_API_URL: API,
      CTX_AUTH_TOKEN: TOKEN,
      CTX_AUTH_USERNAME: "ci-bot",
      FORGEJO_REPOSITORY: REPO,
      CTX_ISSUE_NUMBER: "1",
      CTX_EVENT_NAME: "issue_comment_created",
      CTX_TEST_SCRIPT: "reply-with-comment", // deterministic scripted session
    };

    execFileSync("dist/forgejo-agent", [], { env, cwd: workdir });

    const comments = await getIssueComments(REPO, 1);
    expect(comments.some((c) => c.user.username === "ci-bot")).toBe(true);
  });
});

Open questions — my recommendations

Forgejo version — pin the version the payloads were captured on (15.0.3) and bump it deliberately; the contract tests in Layer 1 will fail loudly if the payloads drift from the schemas.
E2E runner — dedicated runner with docker access using the compose approach (simplest, matches devcontainer); skip the services: keyword since provisioning needs docker exec.
Coverage threshold — 80% on src/ is a reasonable starting point; adjust after the first run.
CTX_TEST_SCRIPT hook — yes, add it. It's the key to deterministic E2E; keep it small (a few lines in main.ts) and documented.
Real-LLM smoke — worth a nightly job once the infra is stable; skip initially to keep CI fast and free.

No implementation yet, as requested — this is the plan + examples. Happy to turn any layer into a PR.

# Testing strategy for forgejo-agent I went through the current codebase (`src/context.ts`, `src/prompt.ts`, `src/schemas.ts`, `src/forgejo/*`, `src/tools.ts`, `src/git.ts`, `src/main.ts`, `examples/event_payloads/`) and the plan below is verified against the real code — function names, env vars, payload shapes, and the binary name (`dist/forgejo-agent`, from `sea.json`) all match. ## Goal Test the program as close as possible to production: a **real Forgejo instance**, the **real binary**, **real git operations**, **real API calls**. The only non-deterministic part is the LLM, so we isolate it with a scripted session for CI and keep a real-LLM smoke test for on-demand/nightly runs. ## Layered approach | Layer | What | Tooling | Speed | Where | |---|---|---|---|---| | 1. Unit | pure functions (prompt, schemas, context parsing, tool param mapping) | Vitest | ms | every PR | | 2. Integration | Forgejo client + tools against a mock HTTP server serving the real payloads | Vitest + `node:http` | ms | every PR | | 3. E2E | full pipeline against a real Forgejo instance (docker), scripted agent session | Vitest + docker compose | ~1–2 min | every PR (docker runner) | | 4. Smoke | real binary + real LLM against the test instance | Vitest + docker compose | minutes, costs tokens | nightly / on demand | ## Layer 0 — Tooling - Add `vitest` + `@vitest/coverage-v8` as devDependencies. - Scripts: `test` (`vitest run`), `test:unit`, `test:integration`, `test:e2e`, `test:coverage`. - `vitest.config.ts`: node environment, `setupFiles` that set the env vars `src/forgejo/fetch.ts` reads **at import time** — `FORGEJO_API_URL`, `CTX_AUTH_TOKEN`, `CTX_AUTH_USERNAME` (verified: `fetch.ts` evaluates `process.env.FORGEJO_API_URL` and `getAuthContext()` at module load, so importing the client without them throws). `src/context.ts` additionally reads `FORGEJO_REPOSITORY`, `CTX_ISSUE_NUMBER`, `CTX_EVENT_NAME` inside functions. - Layout: ``` test/ ├── setup.ts # env vars for unit/integration ├── fixtures/ # reuse examples/event_payloads directly ├── unit/ # colocated src/*.test.ts or mirrored ├── integration/ └── e2e/ ``` ## Layer 1 — Unit tests - `src/prompt.test.ts` — `buildPrompt` for the 4 event types; assert branch instructions, tool instructions, issue/PR content, comment ordering, empty body. - `src/schemas.test.ts` — TypeBox validation; **contract tests** that validate every file in `examples/event_payloads/` against the schemas (keeps fixtures in sync with the Forgejo version). Verified: `issue_opened.json` and `pull_request_opened.json` contain all fields required by `issueSchema`/`pullRequestSchema` (`number`, `user.username`, `title`, `body`, `state`, `pull_request`, `head.label`, `base.label`). - `src/context.test.ts` — env parsing; missing/invalid env throws; repository name pattern; forgejo client mocked. - `src/tools.test.ts` — default repository/issueId fallback; `create-pr-review` maps `side`/`line` → `old_position`/`new_position` (verified in `tools.ts`: `new_position: c.side === "HEAD" ? c.line : 0`, `old_position: c.side === "BASE" ? c.line : 0`); parameter schema validation. - `src/git.ts` — `getLatestCommitId`; `checkoutRepository` against a local bare repo. ## Layer 2 — Integration tests - A tiny in-process HTTP server (`node:http`) mimicking the Forgejo REST API, serving the real payloads from `examples/event_payloads/`. - Test the client: URL construction (`apiUrl + /repos/{owner}/{repo}/issues/{id}`), method, `Authorization: Bearer <token>` header, content-type, response schema validation, error on `>= 400` (verified: throws `fetch failed: <body>`). - Test `getEventContext` end-to-end: repository + issue + PR + comments. - Test each tool against the mock server (close-issue, create-issue, create-issue-comment, create-pr, create-pr-review). ## Layer 3 — E2E with a real Forgejo (the centerpiece) **Instance** — `docker-compose.test.yaml`: ```yaml services: forgejo: image: codeberg.org/forgejo/forgejo:15.0.3 # pin to the production version environment: FORGEJO__server__ROOT_URL: http://forgejo:3000/ FORGEJO__server__HTTP_PORT: "3000" FORGEJO__security__INSTALL_LOCK: "true" FORGEJO__service__DISABLE_REGISTRATION: "true" ports: ["3000:3000"] volumes: [forgejo-data:/data] healthcheck: test: ["CMD", "curl", "-sf", "http://localhost:3000/api/v1/version"] interval: 2s timeout: 2s retries: 30 volumes: forgejo-data: {} ``` **Provisioning** — `scripts/e2e-setup.sh`: 1. wait for `/api/v1/version`; 2. create admin + bot users via the Forgejo CLI inside the container (`forgejo admin user create ...`); 3. generate a token for the bot (`forgejo admin user generate-access-token`); 4. create the test repo (`POST /user/repos` with `auto_init: true`); 5. seed issues/PRs/comments via the API. **Deterministic agent session** — the only non-deterministic part is the LLM. Add a small test hook in `main.ts`: when `CTX_TEST_SCRIPT` is set, replace the Pi session with a scripted one that replays a fixed sequence of tool calls (e.g. "call `create-issue-comment` with body X"). This exercises everything else for real: env parsing, API reads, git clone/checkout, prompt building, tool execution, API writes. **Test** — run the built binary in a fresh temp dir with env vars pointing at the test instance, then assert the outcome via the API (comment exists, issue closed, PR created, review submitted). **Devcontainer** — add the same forgejo service to `.devcontainer/docker-compose.yaml` (currently only has the `workspace` service) so `pnpm test:e2e` works locally with `docker compose up`. ## Layer 4 — Smoke tests (real LLM) - Same instance, but run the real agent (real model) and assert loosely: "a comment by ci-bot exists on the issue". - Slow and costs tokens → nightly or on-demand, not on every PR. ## CI integration ```yaml # .forgejo/workflows/test.yaml name: test.yaml "on": pull_request: branches: [main] jobs: unit-and-integration: container: { image: git.zarantonello.dev/infra/ci-pnpm:v1.1.1@sha256:... } runs-on: srv-generic steps: - run: git clone ... && git checkout ${{ forge.sha }} - run: pnpm install - run: pnpm test:unit - run: pnpm test:integration e2e: runs-on: srv-home # runner with docker access steps: - run: git clone ... && git checkout ${{ forge.sha }} - run: pnpm install - run: pnpm build - run: docker compose -f docker-compose.test.yaml up -d --wait forgejo - run: bash scripts/e2e-setup.sh - run: pnpm test:e2e - run: docker compose -f docker-compose.test.yaml down ``` This mirrors the existing workflows (same clone pattern, same `ci-pnpm` image, `srv-home` already runs the agent job). Alternative: the `services:` keyword (act_runner supports it), but then provisioning can't use `docker exec` — you'd need a custom image that provisions on first boot, or a pre-seeded data volume. The compose approach is simpler and identical to the devcontainer experience. ## Example tests ### 1. `src/prompt.test.ts` — pure, no mocks ```ts import { describe, expect, it } from "vitest"; import { buildPrompt } from "../src/prompt.ts"; const authCtx = { token: "t", username: "ci-bot" }; const issueEventCtx = { repository: { full_name: "owner/repo", default_branch: "main" }, event: { type: "issue", number: 1, user: { username: "davide" }, title: "tests", body: "propose a testing strategy", state: "open", pull_request: null, name: "issues_opened", comments: [], }, }; describe("buildPrompt", () => { it("builds an issue prompt with branch and tool instructions", () => { const prompt = buildPrompt(authCtx, issueEventCtx); expect(prompt).toContain("Forgejo issue number 1"); expect(prompt).toContain("repository owner/repo"); expect(prompt).toContain("default branch: main"); expect(prompt).toContain("create-pr"); expect(prompt).toContain("create-issue-comment"); expect(prompt).toContain("### start issue content from user davide ###"); expect(prompt).toContain("propose a testing strategy"); }); it("appends comments in order", () => { const ctx = { ...issueEventCtx, event: { ...issueEventCtx.event, comments: [ { user: { username: "davide" }, body: "first", id: 1 }, { user: { username: "ci-bot" }, body: "second", id: 2 }, ], }, }; const prompt = buildPrompt(authCtx, ctx); expect(prompt.indexOf("first")).toBeLessThan(prompt.indexOf("second")); }); }); ``` ### 2. `src/schemas.test.ts` — contract tests against the real payloads ```ts import { readFileSync } from "node:fs"; import { describe, expect, it } from "vitest"; import Value from "typebox/value"; import { eventNameSchema, issueSchema, pullRequestSchema } from "../src/schemas.ts"; describe("eventNameSchema", () => { it("accepts the four supported events", () => { for (const name of ["issue_comment_created", "issues_opened", "pull_request_opened", "pull_request_review_requested"]) { expect(Value.Check(eventNameSchema, name)).toBe(true); } }); it("rejects unknown events", () => { expect(Value.Check(eventNameSchema, "issues_closed")).toBe(false); }); }); describe("schemas against real payloads", () => { it("validates the issue payload from examples/event_payloads", () => { const payload = JSON.parse(readFileSync("examples/event_payloads/issue_opened.json", "utf8")); expect(Value.Check(issueSchema, payload.issue)).toBe(true); }); it("validates the pull request payload", () => { const payload = JSON.parse(readFileSync("examples/event_payloads/pull_request_opened.json", "utf8")); expect(Value.Check(pullRequestSchema, payload.pull_request)).toBe(true); }); }); ``` ### 3. `src/forgejo/index.test.ts` — client with stubbed fetch ```ts import { afterEach, describe, expect, it, vi } from "vitest"; import { getIssue } from "../src/forgejo/index.ts"; describe("getIssue", () => { afterEach(() => vi.unstubAllGlobals()); it("calls the endpoint with auth and parses the response", async () => { const fetchMock = vi.fn().mockResolvedValue({ status: 200, json: async () => ({ number: 1, user: { username: "davide" }, title: "t", body: "b", state: "open", pull_request: null, }), }); vi.stubGlobal("fetch", fetchMock); const issue = await getIssue("owner/repo", 1); expect(fetchMock).toHaveBeenCalledWith( "http://forgejo:3000/api/v1/repos/owner/repo/issues/1", expect.objectContaining({ method: "GET", headers: expect.objectContaining({ authorization: "Bearer test-token" }), }), ); expect(issue.number).toBe(1); }); it("throws on error status", async () => { vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ status: 404, text: async () => "not found" })); await expect(getIssue("owner/repo", 1)).rejects.toThrow("fetch failed"); }); }); ``` ### 4. `src/tools.test.ts` — review comment position mapping ```ts import { beforeEach, describe, expect, it, vi } from "vitest"; vi.mock("../src/forgejo/index.ts", () => ({ patchIssue: vi.fn(), postIssue: vi.fn(), postIssueComment: vi.fn(), postPrReview: vi.fn(), postPullRequest: vi.fn(), })); vi.mock("../src/git.ts", () => ({ getLatestCommitId: () => "abc123" })); import { postPrReview } from "../src/forgejo/index.ts"; import { createCreatePrReviewTool } from "../src/tools.ts"; describe("createCreatePrReviewTool", () => { beforeEach(() => vi.mocked(postPrReview).mockResolvedValue({ id: 7, body: "ok" })); it("maps HEAD/BASE comments to new/old positions", async () => { const tool = createCreatePrReviewTool("owner/repo", 3); await tool.execute("call-1", { body: "review", verdict: "APPROVED", comments: [ { body: "on new code", path: "src/main.ts", line: 10, side: "HEAD" }, { body: "on old code", path: "src/main.ts", line: 5, side: "BASE" }, ], }); expect(postPrReview).toHaveBeenCalledWith("owner/repo", 3, { body: "review", event: "APPROVED", commit_id: "abc123", comments: [ { body: "on new code", path: "src/main.ts", new_position: 10, old_position: 0 }, { body: "on old code", path: "src/main.ts", new_position: 0, old_position: 5 }, ], }); }); }); ``` ### 5. E2E sketch — real Forgejo, scripted session ```ts // test/e2e/agent.test.ts import { execFileSync } from "node:child_process"; import { mkdtempSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { beforeAll, describe, expect, it } from "vitest"; import { getIssueComments } from "../../src/forgejo/index.ts"; const API = process.env.FORGEJO_API_URL!; // http://forgejo:3000/api/v1 const TOKEN = process.env.E2E_BOT_TOKEN!; const REPO = process.env.E2E_REPO!; // e.g. e2e/agent-test describe("agent end-to-end against a real Forgejo", () => { beforeAll(() => { // provision repo + issue + comment mentioning @ci-bot execFileSync("bash", ["scripts/e2e-setup.sh", API, TOKEN, REPO]); }); it("responds to an issue comment mentioning the bot", async () => { const workdir = mkdtempSync(join(tmpdir(), "agent-e2e-")); const env = { ...process.env, FORGEJO_API_URL: API, CTX_AUTH_TOKEN: TOKEN, CTX_AUTH_USERNAME: "ci-bot", FORGEJO_REPOSITORY: REPO, CTX_ISSUE_NUMBER: "1", CTX_EVENT_NAME: "issue_comment_created", CTX_TEST_SCRIPT: "reply-with-comment", // deterministic scripted session }; execFileSync("dist/forgejo-agent", [], { env, cwd: workdir }); const comments = await getIssueComments(REPO, 1); expect(comments.some((c) => c.user.username === "ci-bot")).toBe(true); }); }); ``` ## Open questions — my recommendations 1. **Forgejo version** — pin the version the payloads were captured on (15.0.3) and bump it deliberately; the contract tests in Layer 1 will fail loudly if the payloads drift from the schemas. 2. **E2E runner** — dedicated runner with docker access using the compose approach (simplest, matches devcontainer); skip the `services:` keyword since provisioning needs `docker exec`. 3. **Coverage threshold** — 80% on `src/` is a reasonable starting point; adjust after the first run. 4. **`CTX_TEST_SCRIPT` hook** — yes, add it. It's the key to deterministic E2E; keep it small (a few lines in `main.ts`) and documented. 5. **Real-LLM smoke** — worth a nightly job once the infra is stable; skip initially to keep CI fast and free. No implementation yet, as requested — this is the plan + examples. Happy to turn any layer into a PR.

ci-bot commented

2026-09-08 19:18:20 +00:00

Member

Testing strategy — verified against the code

I re-checked the plan against current main (v1.9.12). Everything holds; here are the concrete verifications plus a few implementation subtleties worth knowing before we build it.

Verified against the code

Claim in the plan	Verified
`fetch.ts` reads `FORGEJO_API_URL` + `CTX_AUTH_TOKEN`/`CTX_AUTH_USERNAME` at import time	✅ `apiUrl` and `authContext` are module-level constants in `src/forgejo/fetch.ts`; importing the client without them throws
`context.ts` reads `FORGEJO_REPOSITORY`, `CTX_ISSUE_NUMBER`, `CTX_EVENT_NAME`	✅ inside `getEventContext()`
`create-pr-review` maps `side`/`line` → `old_position`/`new_position`	✅ `src/tools.ts`: `new_position: c.side === "HEAD" ? c.line : 0`, `old_position: c.side === "BASE" ? c.line : 0`
Binary is `dist/forgejo-agent`	✅ `sea.json` → `output: dist/forgejo-agent`
Error on `>= 400`	✅ `fetch.ts` throws `fetch failed: <body>`
Payloads satisfy the schemas	✅ `issue_opened.json`/`pull_request_opened.json` contain all required fields (`number`, `user.username`, `title`, `body`, `state`, `pull_request`, `head.label`, `base.label`)
Forgejo version 15.0.3	✅ `examples/event_payloads/README.md`: "Saved on Forgejo version 15.0.3"
`.devcontainer` only has `workspace`	✅ needs the forgejo service added
CI runner/image	✅ existing workflows use `srv-home` + `ci-pnpm` image

Subtleties to bake into the implementation

Import-time env vars — because fetch.ts evaluates env at module load, test/setup.ts must set the three vars before any import of the client. For the Layer-2 mock server, apiUrl is a module constant: start the node:http server in a global setup (or beforeAll + dynamic import()) so the URL is known before the client module loads.
TypeBox allows extra properties — Type.Object without additionalProperties: false accepts the extra fields in the real payloads, so the contract tests pass today. If we want strict drift detection, add additionalProperties: false to the schemas (or snapshot the parsed shape).
git.ts also imports apiUrl — unit tests of tools.ts must mock ../src/git.ts (as the example does) or set the env vars, otherwise the getLatestCommitId module chain pulls in fetch.ts.
E2E git clone — checkoutRepository builds the remote URL from apiUrl with the token as password, so the E2E binary needs FORGEJO_API_URL pointing at the test instance (the sketch already does this) and the bot token needs repo read/write.

Recommendations on the open questions

Forgejo version — pin 15.0.3 (confirmed it's the version the payloads were saved on) and bump deliberately; the contract tests catch drift.
E2E runner — compose approach on a docker-capable runner; skip services: since provisioning needs docker exec.
Coverage — 80% on src/ as a starting point.
CTX_TEST_SCRIPT hook — yes; a few lines in main.ts, documented.
Real-LLM smoke — nightly once infra is stable; skip initially to keep CI fast and free.

Next step: I can turn Layer 1 (vitest config + unit tests) into a PR first — it's self-contained and unblocks the rest. Just say the word.

# Testing strategy — verified against the code I re-checked the plan against current `main` (v1.9.12). Everything holds; here are the concrete verifications plus a few implementation subtleties worth knowing before we build it. ## Verified against the code | Claim in the plan | Verified | |---|---| | `fetch.ts` reads `FORGEJO_API_URL` + `CTX_AUTH_TOKEN`/`CTX_AUTH_USERNAME` at import time | ✅ `apiUrl` and `authContext` are module-level constants in `src/forgejo/fetch.ts`; importing the client without them throws | | `context.ts` reads `FORGEJO_REPOSITORY`, `CTX_ISSUE_NUMBER`, `CTX_EVENT_NAME` | ✅ inside `getEventContext()` | | `create-pr-review` maps `side`/`line` → `old_position`/`new_position` | ✅ `src/tools.ts`: `new_position: c.side === "HEAD" ? c.line : 0`, `old_position: c.side === "BASE" ? c.line : 0` | | Binary is `dist/forgejo-agent` | ✅ `sea.json` → `output: dist/forgejo-agent` | | Error on `>= 400` | ✅ `fetch.ts` throws `fetch failed: <body>` | | Payloads satisfy the schemas | ✅ `issue_opened.json`/`pull_request_opened.json` contain all required fields (`number`, `user.username`, `title`, `body`, `state`, `pull_request`, `head.label`, `base.label`) | | Forgejo version 15.0.3 | ✅ `examples/event_payloads/README.md`: "Saved on Forgejo version 15.0.3" | | `.devcontainer` only has `workspace` | ✅ needs the forgejo service added | | CI runner/image | ✅ existing workflows use `srv-home` + `ci-pnpm` image | ## Subtleties to bake into the implementation 1. **Import-time env vars** — because `fetch.ts` evaluates env at module load, `test/setup.ts` must set the three vars before any import of the client. For the Layer-2 mock server, `apiUrl` is a module constant: start the `node:http` server in a global setup (or `beforeAll` + dynamic `import()`) so the URL is known before the client module loads. 2. **TypeBox allows extra properties** — `Type.Object` without `additionalProperties: false` accepts the extra fields in the real payloads, so the contract tests pass today. If we want strict drift detection, add `additionalProperties: false` to the schemas (or snapshot the parsed shape). 3. **`git.ts` also imports `apiUrl`** — unit tests of `tools.ts` must mock `../src/git.ts` (as the example does) or set the env vars, otherwise the `getLatestCommitId` module chain pulls in `fetch.ts`. 4. **E2E git clone** — `checkoutRepository` builds the remote URL from `apiUrl` with the token as password, so the E2E binary needs `FORGEJO_API_URL` pointing at the test instance (the sketch already does this) and the bot token needs repo read/write. ## Recommendations on the open questions 1. **Forgejo version** — pin 15.0.3 (confirmed it's the version the payloads were saved on) and bump deliberately; the contract tests catch drift. 2. **E2E runner** — compose approach on a docker-capable runner; skip `services:` since provisioning needs `docker exec`. 3. **Coverage** — 80% on `src/` as a starting point. 4. **`CTX_TEST_SCRIPT` hook** — yes; a few lines in `main.ts`, documented. 5. **Real-LLM smoke** — nightly once infra is stable; skip initially to keep CI fast and free. Next step: I can turn Layer 1 (vitest config + unit tests) into a PR first — it's self-contained and unblocks the rest. Just say the word.

davide commented

2026-09-08 19:27:09 +00:00

Author

Owner

@ci-bot let's start gradually adding unit tests

ci-bot referenced this issue

2026-09-08 19:35:52 +00:00

test: add unit tests with vitest #265

ci-bot commented

2026-09-08 19:36:25 +00:00

Member

Done — first batch of unit tests is in PR #265 (test: add unit tests with vitest).

What's in it

Tooling: vitest + @vitest/coverage-v8 devDeps, test / test:unit / test:coverage scripts, vitest.config.ts, and test/setup.ts that sets the env vars src/forgejo/fetch.ts reads at import time.
30 tests, all passing, covering:
- prompt.ts — all 4 event types, branch/tool instructions, empty body, comment ordering
- schemas.ts — event names + contract tests against every file in examples/event_payloads/
- context.ts — auth env parsing, getEventContext (issue vs PR, comments, invalid env)
- tools.ts — all 5 tools, default fallbacks, and the create-pr-review position mapping
- git.ts — getLatestCommitId

Findings worth knowing

CTX_ISSUE_NUMBER is not validated — Number(Value.Parse(Type.String(), ...)) turns "not-a-number" into NaN without throwing; the failure surfaces later at the API call. Minor robustness gap, left as-is for now — happy to fix in a follow-up.
@earendil-works/pi-server is missing — pi-coding-agent's index re-exports main, which loads an experimental server module importing @earendil-works/pi-server (not installed; the rolldown build tree-shakes it). Tests mock the package to avoid that graph. If you ever import the package in a non-bundled context, this will bite.
checkoutRepository deferred — it needs a real git remote, so it fits the integration/E2E layers rather than unit.

Checks

pnpm check / lint:check / format:check ✅, pnpm test 30/30 ✅. Coverage on src/ (excluding src/forgejo/, which is Layer 2 territory): ~88% statements.

Next natural step: Layer 2 (integration tests with a mock Forgejo HTTP server) or wiring pnpm test into CI. Just say the word.

Done — first batch of unit tests is in **PR #265** (`test: add unit tests with vitest`). ## What's in it - **Tooling**: `vitest` + `@vitest/coverage-v8` devDeps, `test` / `test:unit` / `test:coverage` scripts, `vitest.config.ts`, and `test/setup.ts` that sets the env vars `src/forgejo/fetch.ts` reads at import time. - **30 tests**, all passing, covering: - `prompt.ts` — all 4 event types, branch/tool instructions, empty body, comment ordering - `schemas.ts` — event names + **contract tests** against every file in `examples/event_payloads/` - `context.ts` — auth env parsing, `getEventContext` (issue vs PR, comments, invalid env) - `tools.ts` — all 5 tools, default fallbacks, and the `create-pr-review` position mapping - `git.ts` — `getLatestCommitId` ## Findings worth knowing 1. **`CTX_ISSUE_NUMBER` is not validated** — `Number(Value.Parse(Type.String(), ...))` turns `"not-a-number"` into `NaN` without throwing; the failure surfaces later at the API call. Minor robustness gap, left as-is for now — happy to fix in a follow-up. 2. **`@earendil-works/pi-server` is missing** — `pi-coding-agent`'s index re-exports `main`, which loads an experimental server module importing `@earendil-works/pi-server` (not installed; the rolldown build tree-shakes it). Tests mock the package to avoid that graph. If you ever import the package in a non-bundled context, this will bite. 3. **`checkoutRepository` deferred** — it needs a real git remote, so it fits the integration/E2E layers rather than unit. ## Checks `pnpm check` / `lint:check` / `format:check` ✅, `pnpm test` 30/30 ✅. Coverage on `src/` (excluding `src/forgejo/`, which is Layer 2 territory): ~88% statements. Next natural step: Layer 2 (integration tests with a mock Forgejo HTTP server) or wiring `pnpm test` into CI. Just say the word.

ci-bot referenced this issue

2026-09-08 20:17:31 +00:00

test: add unit tests with vitest #265

ci-bot referenced this issue

2026-09-08 20:23:58 +00:00

test: add unit tests with vitest #265

davide referenced this issue from a commit

2026-09-08 20:42:03 +00:00

test: add unit tests with vitest (#265)

davide commented

2026-09-08 20:45:58 +00:00

Author

Owner

@ci-bot ok, unit tests merged, now you can continue. with integration tests. It was decided that payloads in the repo are only to be used as a reference and that for tests we use the actual forgejo instance to guarantee that the content is always synced

ci-bot commented

2026-09-08 21:07:27 +00:00

Member

Integration tests are in PR #266 (test(integration): add integration tests against a real forgejo instance), following the decision that examples/event_payloads/ are reference-only: tests hit a real Forgejo instance so the schemas are always validated against live payloads.

What's in it — 16 tests, all green

test/integration/forgejo.test.ts — the client: repositories, issues, comments, PRs, reviews, inline review comments. Every call goes through forgejoFetch, which validates the response against the TypeBox schemas, so a passing suite is the "content always synced" guarantee (no fixture drift possible).
test/integration/context.test.ts — getEventContext for issue + PR events against live data.
test/integration/tools.test.ts — all five tools run for real (close-issue, create-issue, create-issue-comment, create-pr, create-pr-review). The review test runs from a checkout of the PR head branch so getLatestCommitId() resolves a real commit, and the inline comment lands on a known diff position.

Infrastructure

docker-compose.test.yaml — pinned Forgejo 15.0.3 (the version the payloads were captured on), healthcheck via /api/v1/version.
scripts/integration-setup.sh (POSIX sh) — creates users + access tokens via the Forgejo CLI in the container, deletes/recreates the test repo on every run (fully deterministic), pushes a change branch with a known diff, seeds an issue + comment, writes test/integration/credentials.json (gitignored).
vitest.integration.config.ts + pnpm test:integration — separate config so the default pnpm test stays fast and instance-free; files run sequentially (tests share the instance).
The same forgejo service added to .devcontainer/docker-compose.yaml so tests run from the workspace container too.
test/README.md with the full run flow: up -d --wait → ./scripts/integration-setup.sh → pnpm test:integration.

Validation

Ran the whole cycle against a live Forgejo 15.0.3 instance: test:integration 16/16 ✅ (repeat runs too), unit pnpm test 23/23 ✅, check / lint:check / format:check ✅, yamllint + prettier ✅ (had to add *.sh to .prettierignore — prettier has no shell parser).

Two quirks worth knowing:

Forgejo only allows one open PR per head/base pair, so every test PR gets its own branch (created from the provisioned change branch, keeping the same diff).
Only defineTool from pi-coding-agent is mocked (it's just a metadata factory; the package itself can't be imported outside the bundle — missing @earendil-works/pi-server).

⚠️ Transparency note: I accidentally pushed the commit straight to main first; since main is protected against force-pushes, I reverted it there (420a634) and re-applied the changes on the PR branch — the PR diff is exactly the intended change, main is back to its pre-PR content.

Next steps when you're ready: wire pnpm test:integration into CI (needs a docker-capable runner — I left the workflow out of this PR on purpose), then Layer 3 (E2E with the CTX_TEST_SCRIPT scripted-session hook) and Layer 4 (real-LLM smoke).

Integration tests are in **PR #266** (`test(integration): add integration tests against a real forgejo instance`), following the decision that `examples/event_payloads/` are reference-only: tests hit a **real Forgejo instance** so the schemas are always validated against live payloads. ## What's in it — 16 tests, all green - `test/integration/forgejo.test.ts` — the client: repositories, issues, comments, PRs, reviews, inline review comments. Every call goes through `forgejoFetch`, which validates the response against the TypeBox schemas, so a passing suite **is** the "content always synced" guarantee (no fixture drift possible). - `test/integration/context.test.ts` — `getEventContext` for issue + PR events against live data. - `test/integration/tools.test.ts` — all five tools run for real (close-issue, create-issue, create-issue-comment, create-pr, create-pr-review). The review test runs from a checkout of the PR head branch so `getLatestCommitId()` resolves a real commit, and the inline comment lands on a known diff position. ## Infrastructure - `docker-compose.test.yaml` — pinned Forgejo **15.0.3** (the version the payloads were captured on), healthcheck via `/api/v1/version`. - `scripts/integration-setup.sh` (POSIX sh) — creates users + access tokens via the Forgejo CLI in the container, **deletes/recreates the test repo on every run** (fully deterministic), pushes a change branch with a known diff, seeds an issue + comment, writes `test/integration/credentials.json` (gitignored). - `vitest.integration.config.ts` + `pnpm test:integration` — separate config so the default `pnpm test` stays fast and instance-free; files run sequentially (tests share the instance). - The same forgejo service added to `.devcontainer/docker-compose.yaml` so tests run from the workspace container too. - `test/README.md` with the full run flow: `up -d --wait` → `./scripts/integration-setup.sh` → `pnpm test:integration`. ## Validation Ran the whole cycle against a live Forgejo 15.0.3 instance: `test:integration` 16/16 ✅ (repeat runs too), unit `pnpm test` 23/23 ✅, `check` / `lint:check` / `format:check` ✅, yamllint + prettier ✅ (had to add `*.sh` to `.prettierignore` — prettier has no shell parser). Two quirks worth knowing: 1. Forgejo only allows **one open PR per head/base pair**, so every test PR gets its own branch (created from the provisioned change branch, keeping the same diff). 2. Only `defineTool` from pi-coding-agent is mocked (it's just a metadata factory; the package itself can't be imported outside the bundle — missing `@earendil-works/pi-server`). ⚠️ Transparency note: I accidentally pushed the commit straight to `main` first; since `main` is protected against force-pushes, I reverted it there (`420a634`) and re-applied the changes on the PR branch — the PR diff is exactly the intended change, `main` is back to its pre-PR content. Next steps when you're ready: wire `pnpm test:integration` into CI (needs a docker-capable runner — I left the workflow out of this PR on purpose), then Layer 3 (E2E with the `CTX_TEST_SCRIPT` scripted-session hook) and Layer 4 (real-LLM smoke).

No labels

No milestone

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

projects/forgejo-agent#264

No description provided.

Rows
Columns

tests #264

Testing strategy for forgejo-agent

Goal

Layered approach

Layer 0 — Tooling

Layer 1 — Unit tests

Layer 2 — Integration tests

Layer 3 — E2E with a real Forgejo (the centerpiece)

Layer 4 — Smoke tests (real LLM)

CI integration

Example tests

1. src/prompt.test.ts — pure, no mocks

2. src/schemas.test.ts — contract tests against the real payloads

3. src/forgejo/index.test.ts — client with stubbed fetch

4. src/tools.test.ts — review comment position mapping

5. E2E sketch — real Forgejo, scripted session

Open questions

Testing strategy for forgejo-agent

Goal

Layered approach

Layer 0 — Tooling

Layer 1 — Unit tests

Layer 2 — Integration tests

Layer 3 — E2E with a real Forgejo (the centerpiece)

Layer 4 — Smoke tests (real LLM)

CI integration

Example tests

1. src/prompt.test.ts — pure, no mocks

2. src/schemas.test.ts — contract tests against the real payloads

3. src/forgejo/index.test.ts — client with stubbed fetch

4. src/tools.test.ts — review comment position mapping

5. E2E sketch — real Forgejo, scripted session

Open questions — my recommendations

Testing strategy — verified against the code

Verified against the code

Subtleties to bake into the implementation

Recommendations on the open questions

What's in it

Findings worth knowing

Checks

What's in it — 16 tests, all green

Infrastructure

Validation

1. `src/prompt.test.ts` — pure, no mocks

2. `src/schemas.test.ts` — contract tests against the real payloads

3. `src/forgejo/index.test.ts` — client with stubbed fetch

4. `src/tools.test.ts` — review comment position mapping

1. `src/prompt.test.ts` — pure, no mocks

2. `src/schemas.test.ts` — contract tests against the real payloads

3. `src/forgejo/index.test.ts` — client with stubbed fetch

4. `src/tools.test.ts` — review comment position mapping