code-testing-extensions
Provides file paths to language-specific extension files for the code-testing pipeline. Call this skill to discover available extension guidance files (e.g., dotnet.md for .NET, cpp.md for C++). Do not use directly — invoked by code-testing agents and skills that need language-sp
Install
npx skills add https://github.com/dotnet/skills/tree/main/plugins/dotnet-test/skills/code-testing-extensions
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install dotnet-skills@llmmart
git clone https://github.com/dotnet/skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole dotnet/skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Code Testing Extensions
This skill provides access to language-specific guidance files used by the code-testing pipeline. Call this skill to get the file paths, then read the relevant file for your target language.
Available Extension Files
| File | Language | Contents |
|---|---|---|
| extensions/dotnet.md | .NET (C#/F#/VB) | Build commands, test commands, project reference validation, common CS error codes, MSTest template |
| extensions/python.md | Python | Framework-adaptive test commands (pytest, custom runners), project layout detection, mocking guidelines, common errors |
| extensions/typescript.md | TypeScript/JavaScript | Build/test commands (Jest/Vitest/Mocha), framework detection, mocking, TS-specific considerations |
| extensions/powershell.md | PowerShell | Test commands (Pester v5), module import patterns, discovery/run pitfalls, mocking, common errors |
| extensions/cpp.md | C++ | Testing internals with friend declarations |
| extensions/go.md | Go | go test commands, table-driven tests, integration vs unit layout, mocking via interfaces, common errors |
| extensions/java.md | Java | Maven/Gradle commands, JUnit 4/5 and TestNG detection, Mockito, Spring Boot slices, common errors |
| extensions/rust.md | Rust | cargo test commands, unit vs integration vs doc tests, features, async test harnesses, common errors |
| extensions/ruby.md | Ruby | RSpec and Minitest commands, Bundler usage, Rails specifics, mocking patterns, common errors |
| extensions/swift.md | Swift | SPM and Xcode test commands, XCTest vs Swift Testing, @testable import, async/throws tests, common errors |
| extensions/kotlin.md | Kotlin | Gradle commands, JUnit/Kotest detection, MockK, coroutines test, KMP and Android specifics, common errors |
| extensions/dotnet-examples.md | .NET (C#/F#/VB) | Concrete pipeline examples: sample research output, plan, generated tests, fix cycles, final report |
| extensions/python-examples.md | Python | Concrete pipeline examples (pytest): research, plan, generated test file, fix cycles, final report |
| extensions/typescript-examples.md | TypeScript/JavaScript | Concrete pipeline examples (Vitest, applicable to Jest): research, plan, generated test file, fix cycles, final report |
| extensions/go-examples.md | Go | Concrete pipeline examples (standard testing): research, plan, table-driven test file, fix cycles, final report |
| extensions/java-examples.md | Java | Concrete pipeline examples (JUnit 5 + Mockito on Maven): research, plan, generated test file, fix cycles, final report |
Usage
Read the appropriate extension file for the target language before writing test code. When an <language>-examples.md file exists for the target language, read it alongside the base extension to see a concrete end-to-end pipeline walkthrough (research output, plan, generated tests, fix cycles, final report).
Files (skills)
-
extensions
-
cpp-examples.md 8.8 KB
# C++ Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a C++ codebase using CMake + Catch2. These show what each pipeline phase produces for a small library project. > GoogleTest follows the same shape. Replace `TEST_CASE` / `SECTION` / `REQUIRE` with `TEST` / `EXPECT_*` and link the test executable to `GTest::gtest_main` instead of `Catch2::Catch2WithMain`. ## Source Under Test A simple `InvoiceService` in a CMake project: ```text CMakeLists.txt include/contoso/billing/ invoice.hpp invoice_repository.hpp invoice_service.hpp src/invoice_service.cpp tests/CMakeLists.txt (links Catch2::Catch2WithMain) ``` ```cpp // src/invoice_service.cpp #include "contoso/billing/invoice_service.hpp" #include <cmath> #include <stdexcept> #include <utility> namespace contoso::billing { InvoiceService::InvoiceService(InvoiceRepository& repository, Clock clock) : repository_(repository), clock_(std::move(clock)) {} double InvoiceService::calculate_total(const Invoice& invoice) const { if (invoice.line_items.empty()) { throw std::invalid_argument("invoice has no line items"); } double subtotal = 0.0; for (const LineItem& item : invoice.line_items) { subtotal += static_cast<double>(item.quantity) * item.unit_price; } return std::round((subtotal + subtotal * invoice.tax_rate) * 100.0) / 100.0; } Invoice InvoiceService::get_by_id(int id) const { auto invoice = repository_.find(id); if (!invoice.has_value()) { throw std::out_of_range("invoice not found"); } return *invoice; } void InvoiceService::mark_as_paid(int id) { auto invoice = repository_.find(id); if (!invoice.has_value()) { throw std::out_of_range("invoice not found"); } if (invoice->status == InvoiceStatus::paid) { throw std::logic_error("invoice is already paid"); } invoice->status = InvoiceStatus::paid; invoice->paid_at = clock_(); repository_.update(*invoice); } } // namespace contoso::billing ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: /work/contoso-billing - **Language**: C++20 - **Build System**: CMake (preset `ninja-debug` present) - **Test Framework**: Catch2 v3 (detected via `find_package(Catch2 3 REQUIRED)`) ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Configure**: `cmake --preset ninja-debug` - **Build tests**: `cmake --build --preset ninja-debug --target invoice_service_tests` - **Test**: `ctest --preset ninja-debug --output-on-failure` - **Coverage (if configured)**: rebuild with `--coverage`, then use `gcov` or `llvm-cov` ## Files to Test ### High Priority | File | Classes/Functions | Testability | Notes | |------|-------------------|-------------|-------| | src/invoice_service.cpp | InvoiceService: calculate_total, get_by_id, mark_as_paid | High | Repository is an interface; clock dependency is injectable | ## Testing Patterns - No existing patterns; recommend Catch2 `TEST_CASE` blocks, `SECTION` cases, `Approx` for floating-point assertions, and a hand-written fake repository. ``` ## Sample Plan Output ```markdown # Test Implementation Plan ## Overview Generate Catch2 tests for InvoiceService covering pure calculation logic, repository lookup behavior, and the paid-state transition. ## Commands - **Build**: `cmake --build --preset ninja-debug --target invoice_service_tests` - **Test**: `ctest --preset ninja-debug --output-on-failure` ## Phase 1: InvoiceService ### Files to Test - **Source**: `src/invoice_service.cpp` - **Test File**: `tests/invoice_service_tests.cpp` **Methods to Test**: 1. `calculate_total` — tax, zero tax, rounding, empty line items 2. `get_by_id` — existing invoice and missing invoice 3. `mark_as_paid` — success with fixed clock, already-paid, missing ``` ## Sample Generated Test File ```cpp // tests/invoice_service_tests.cpp #include "contoso/billing/invoice_service.hpp" #include <catch2/catch_test_macros.hpp> #include <catch2/matchers/catch_matchers_string.hpp> #include <chrono> #include <optional> #include <stdexcept> #include <unordered_map> using Catch::Matchers::ContainsSubstring; namespace contoso::billing { namespace { class FakeRepository final : public InvoiceRepository { public: std::optional<Invoice> find(int id) override { auto it = invoices.find(id); return it == invoices.end() ? std::nullopt : std::optional<Invoice>{it->second}; } void update(const Invoice& invoice) override { updated = invoice; invoices[invoice.id] = invoice; } std::unordered_map<int, Invoice> invoices; std::optional<Invoice> updated; }; Invoice make_invoice(int id = 1) { return Invoice{ .id = id, .status = InvoiceStatus::pending, .tax_rate = 0.10, .line_items = {LineItem{.quantity = 1, .unit_price = 100.0}}, }; } } // namespace TEST_CASE("InvoiceService calculates totals", "[invoice-service]") { FakeRepository repository; InvoiceService sut(repository, [] { return std::chrono::system_clock::time_point{}; }); SECTION("single item with tax") { REQUIRE(sut.calculate_total(make_invoice()) == Catch::Approx(110.0)); } SECTION("multi quantity with zero tax") { Invoice invoice = make_invoice(); invoice.tax_rate = 0.0; invoice.line_items = {LineItem{.quantity = 3, .unit_price = 25.0}}; REQUIRE(sut.calculate_total(invoice) == Catch::Approx(75.0)); } SECTION("rounds to two decimals") { Invoice invoice = make_invoice(); invoice.tax_rate = 0.07; invoice.line_items = {LineItem{.quantity = 2, .unit_price = 9.99}}; REQUIRE(sut.calculate_total(invoice) == Catch::Approx(21.38).epsilon(0.001)); } SECTION("empty line items throw") { Invoice invoice = make_invoice(); invoice.line_items.clear(); REQUIRE_THROWS_WITH(sut.calculate_total(invoice), ContainsSubstring("no line items")); } } TEST_CASE("InvoiceService uses the repository", "[invoice-service]") { const auto fixed_time = std::chrono::system_clock::time_point{std::chrono::seconds{123}}; FakeRepository repository; repository.invoices.emplace(42, make_invoice(42)); InvoiceService sut(repository, [fixed_time] { return fixed_time; }); SECTION("get_by_id returns an existing invoice") { REQUIRE(sut.get_by_id(42).id == 42); } SECTION("get_by_id throws when missing") { REQUIRE_THROWS_WITH(sut.get_by_id(999), ContainsSubstring("not found")); } SECTION("mark_as_paid updates status, date, and repository") { repository.invoices.emplace(1, make_invoice(1)); sut.mark_as_paid(1); REQUIRE(repository.updated.has_value()); REQUIRE(repository.updated->status == InvoiceStatus::paid); REQUIRE(repository.updated->paid_at == fixed_time); } SECTION("mark_as_paid rejects an already-paid invoice") { Invoice invoice = make_invoice(2); invoice.status = InvoiceStatus::paid; repository.invoices.emplace(2, invoice); REQUIRE_THROWS_WITH(sut.mark_as_paid(2), ContainsSubstring("already paid")); } } } // namespace contoso::billing ``` ## Sample Fix Cycle When the implementer hits a compile or runner issue, the fixer agent diagnoses and resolves it. **Build output:** ```text error: cannot declare variable 'repository' to be of abstract type 'FakeRepository' note: missing pure virtual method 'InvoiceRepository::update' ``` **Fixer diagnosis:** The fake repository implemented `find` but not the full `InvoiceRepository` interface. **Fix applied:** Add `void update(const Invoice& invoice) override` to `FakeRepository` and record the updated invoice for assertions. **Rebuild + rerun:** `cmake --build --preset ninja-debug --target invoice_service_tests && ctest --preset ninja-debug --output-on-failure` → SUCCESS ## Sample Final Report ```markdown ## Test Generation Report **Project**: contoso-billing (C++ / CMake) **Strategy**: Direct (single source file in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 9 | | Tests passing | 9 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `tests/invoice_service_tests.cpp` (2 Catch2 test cases, 9 sections) ### Coverage - InvoiceService.calculate_total — tax, zero tax, rounding, empty input - InvoiceService.get_by_id — found and missing branches - InvoiceService.mark_as_paid — success and already-paid branches ### Build / Test Validation - Configure: ✅ `cmake --preset ninja-debug` - Build: ✅ `cmake --build --preset ninja-debug --target invoice_service_tests` - Test: ✅ `ctest --preset ninja-debug --output-on-failure` ``` -
cpp.md 20.1 KB
# C++ Extension Language-specific guidance for C++ test generation. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — find `*_test.cpp`, `*_tests.cpp`, `test_*.cc`, `tests/`, `test/`, `unittests/`, and any CTest/Bazel/Make test targets. Copy the framework, naming, fixtures, assertions, and helper style already in use. 2. **Build configuration** — inspect `CMakeLists.txt`, `CMakePresets.json`, `Makefile`, `WORKSPACE`, `MODULE.bazel`, `BUILD`, `BUILD.bazel`, `meson.build`, or CI scripts before inventing commands. 3. **Dependencies** — detect GoogleTest, GoogleMock, Catch2, doctest, Boost.Test, or a custom harness from package manifests and target links. 4. **Compiler and standard** — identify `CMAKE_CXX_STANDARD`, `-std=c++NN`, toolchain files, compiler wrappers, sanitizers, and warning-as-error flags. 5. **Coverage setup** — determine whether the repo already has `gcov`, `llvm-cov`, `gcovr`, `lcov`, or custom coverage targets. If not, add minimal, test-target-scoped coverage instrumentation. Generated C++ tests must earn coverage, not merely compile. Target uncovered functions, branches, error paths, and boundary conditions; shallow smoke tests that only construct objects rarely move line coverage. ## Environment and Toolchain Detection | Indicator | Meaning | |-----------|---------| | `CMakeLists.txt` | CMake project; prefer configured build directories and existing presets | | `CMakePresets.json` | Use `cmake --preset <name>` and `cmake --build --preset <name>` when present | | `Makefile` without CMake | Use `make`, `make test`, or the repo's documented targets | | `WORKSPACE`, `MODULE.bazel`, `BUILD(.bazel)` | Bazel project; tests are `cc_test` targets | | `compile_commands.json` | Exact compiler flags and include directories used by the build | | `CMAKE_CXX_STANDARD` / `-std=c++17` | Maximum language features allowed in tests | | `clang++` | Prefer `llvm-cov gcov` for coverage data collection | | `g++` | Use matching `gcov` from the same GCC toolchain version | Useful discovery commands: ```bash cmake --version g++ --version clang++ --version find . \( -name CMakeLists.txt -o -name CMakePresets.json -o -name Makefile -o -name BUILD -o -name BUILD.bazel \) find . \( -name '*test*.cpp' -o -name '*test*.cc' -o -name '*tests*.cpp' -o -name '*tests*.cc' \) ``` Do not change the project's C++ standard to make a test compile. Match the production target's standard and include directories. ## Test Framework Detection | Indicator | Framework | |-----------|-----------| | `find_package(GTest)` / `GTest::gtest_main` / `gtest_discover_tests` | GoogleTest | | `GTest::gmock` / `#include <gmock/gmock.h>` | GoogleMock for mocks | | `find_package(Catch2 3)` / `Catch2::Catch2WithMain` / `catch_discover_tests` | Catch2 v3 | | `#include <catch2/catch_test_macros.hpp>` | Catch2 v3 test source | | `#include <gtest/gtest.h>` | GoogleTest test source | | `add_test(NAME ... COMMAND ...)` | Manual CTest registration | | `cc_test(` | Bazel C++ test target | Use the framework already present. Do not add Catch2 to a GoogleTest repo or GoogleTest to a Catch2 repo just because it is familiar. ## Build Commands Prefer repo scripts and presets first. Otherwise use the smallest command that compiles the changed test target. | Scope | CMake command | |-------|---------------| | Configure debug build | `cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug` | | Configure with tests | `cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON` | | Configure with coverage option | `cmake -S . -B build-coverage -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON -DCODE_COVERAGE=ON` | | Build all | `cmake --build build` | | Build one test target | `cmake --build build --target my_component_tests` | | Parallel build | `cmake --build build --parallel` | | Clean rebuild | `cmake --build build --target clean && cmake --build build` | Make equivalents when no CMake build exists: | Scope | Make command | |-------|--------------| | Build all | `make` | | Build tests | `make test` or `make tests` | | Build one target | `make my_component_tests` | | Coverage build | `make clean && CXXFLAGS="--coverage -O0 -g" LDFLAGS="--coverage" make tests` | Bazel equivalents: | Scope | Bazel command | |-------|---------------| | Build tests | `bazel build //...` | | Run all tests | `bazel test //...` | | Run one test | `bazel test //path/to:target_test` | ## Test Commands | Scope | Command | |-------|---------| | All CTest tests | `ctest --test-dir build --output-on-failure` | | Verbose CTest failure | `ctest --test-dir build --output-on-failure -V` | | One CTest test by regex | `ctest --test-dir build -R my_component --output-on-failure` | | Direct GoogleTest binary | `./build/tests/my_component_tests` | | GoogleTest filter | `./build/tests/my_component_tests --gtest_filter=SuiteName.TestName` | | GoogleTest list tests | `./build/tests/my_component_tests --gtest_list_tests` | | Direct Catch2 binary | `./build/tests/my_component_tests` | | Catch2 filter by name | `./build/tests/my_component_tests "test case name"` | | Catch2 filter by tag | `./build/tests/my_component_tests "[parser]"` | | Catch2 list tests | `./build/tests/my_component_tests --list-tests` | For CTest, run from the configured build tree or pass `--test-dir build`; running `ctest` from the source root often reports zero tests. ## Lint Command Use the repo's lint script first (`make lint`, `cmake --build build --target lint`, `ninja clang-tidy`). Otherwise detect tools from configuration: - `.clang-format` present → `clang-format -i path/to/test.cpp` - `.clang-tidy` present → run the repo's clang-tidy target or `clang-tidy path/to/test.cpp -- -Iinclude` - CMake format target present → `cmake --build build --target format` Never silence warnings in generated tests with blanket pragmas. Fix the warning or match the existing project pattern. ## Project Layout and Test File Placement Common layouts: ```text project/ ├── include/ # public headers ├── src/ # implementation files ├── tests/ # test sources and CMakeLists.txt │ ├── CMakeLists.txt │ └── widget_test.cpp ├── CMakeLists.txt └── CMakePresets.json ``` | Source file | Preferred test file | |-------------|---------------------| | `src/widget.cpp` | `tests/widget_test.cpp` or existing repo pattern | | `src/parser/tokenizer.cpp` | `tests/parser/tokenizer_test.cpp` | | `include/lib/widget.hpp` | `tests/widget_test.cpp` using the public API | - Match existing suffixes: `_test.cpp`, `_tests.cpp`, `test_*.cpp`, or `.cc`. - Keep tests near existing test CMake targets instead of creating an isolated harness. - Prefer testing through public headers. Use white-box access only when existing tests already do so or coverage-critical internals cannot be reached otherwise. - Shared fixtures/helpers belong in `tests/support/`, `tests/helpers/`, or the existing helper location, not production `src/` unless the repo already has test-only utilities. ## GoogleTest Setup Minimal test source: ```cpp #include <gtest/gtest.h> #include "calculator.hpp" TEST(CalculatorTest, Add_WithPositiveInputs_ReturnsSum) { Calculator calculator; EXPECT_EQ(calculator.Add(2, 3), 5); } ``` CMake registration with discovery: ```cmake enable_testing() find_package(GTest REQUIRED) include(GoogleTest) add_executable(calculator_tests tests/calculator_test.cpp ) target_link_libraries(calculator_tests PRIVATE calculator_lib GTest::gtest_main ) gtest_discover_tests(calculator_tests) ``` If the repo already has a test helper function such as `add_project_test(...)`, use it instead of writing raw `add_executable` blocks. ## Catch2 v3 Setup Minimal test source: ```cpp #include <catch2/catch_test_macros.hpp> #include "calculator.hpp" TEST_CASE("Calculator adds positive inputs", "[calculator]") { Calculator calculator; CHECK(calculator.Add(2, 3) == 5); } ``` CMake registration with discovery: ```cmake enable_testing() find_package(Catch2 3 REQUIRED) include(Catch) add_executable(calculator_tests tests/calculator_test.cpp ) target_link_libraries(calculator_tests PRIVATE calculator_lib Catch2::Catch2WithMain ) catch_discover_tests(calculator_tests) ``` Use `Catch2::Catch2WithMain` unless the repo already provides a custom `main`. Linking a framework main and defining your own `main` causes duplicate-symbol linker failures. ## Coverage Instrumentation Coverage instrumentation is the highest-risk setup step. Add coverage flags to **both compilation and linking** for the test target. Adding flags only to `CXXFLAGS` often compiles but produces no `.gcda` files or fails with missing gcov runtime symbols at link time. ### CMake target-scoped coverage option Prefer target-scoped flags over global `CMAKE_CXX_FLAGS` so production targets stay clean: ```cmake option(CODE_COVERAGE "Build tests with gcov-compatible coverage instrumentation" OFF) add_executable(calculator_tests tests/calculator_test.cpp ) target_link_libraries(calculator_tests PRIVATE calculator_lib GTest::gtest_main ) if(CODE_COVERAGE AND CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang|AppleClang") target_compile_options(calculator_tests PRIVATE --coverage -O0 -g) target_link_options(calculator_tests PRIVATE --coverage) endif() ``` For libraries built into the test binary, instrument the library target too, otherwise coverage only reports test files: ```cmake if(CODE_COVERAGE AND CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang|AppleClang") target_compile_options(calculator_lib PRIVATE --coverage -O0 -g) target_link_options(calculator_lib PRIVATE --coverage) target_compile_options(calculator_tests PRIVATE --coverage -O0 -g) target_link_options(calculator_tests PRIVATE --coverage) endif() ``` Equivalent long-form flags: ```cmake target_compile_options(calculator_tests PRIVATE -fprofile-arcs -ftest-coverage -O0 -g) target_link_options(calculator_tests PRIVATE -fprofile-arcs -ftest-coverage) ``` ### Coverage command sequence GCC/gcov path: ```bash cmake -S . -B build-coverage -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON -DCODE_COVERAGE=ON cmake --build build-coverage --target calculator_tests ctest --test-dir build-coverage --output-on-failure gcovr --root . --filter 'src/' --filter 'include/' --exclude 'tests/' --print-summary ``` Clang path using gcov-compatible data: ```bash cmake -S . -B build-coverage -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON -DCODE_COVERAGE=ON cmake --build build-coverage --target calculator_tests ctest --test-dir build-coverage --output-on-failure gcovr --root . --gcov-executable 'llvm-cov gcov' --filter 'src/' --filter 'include/' --exclude 'tests/' --print-summary ``` `lcov` / `genhtml` path: ```bash cmake -S . -B build-coverage -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON -DCODE_COVERAGE=ON cmake --build build-coverage ctest --test-dir build-coverage --output-on-failure lcov --capture --directory build-coverage --output-file coverage.info lcov --remove coverage.info '*/tests/*' '*/_deps/*' --output-file coverage.filtered.info genhtml coverage.filtered.info --output-directory coverage-html ``` Make-only coverage path: ```bash make clean CXXFLAGS="--coverage -O0 -g" LDFLAGS="--coverage" make tests ./tests/calculator_tests gcovr --root . --print-summary ``` Important coverage rules: - Compile and link with the same compiler family. Clang-generated coverage data should be read with `llvm-cov gcov`, not system `gcov`. - Run the instrumented test binary before collecting; `.gcda` files are written when the process exits normally. - Use `-O0 -g` for coverage builds to keep line mapping stable. - Exclude test files and vendored dependencies from coverage reports; include production `src/` and `include/`. - If the code under test is a static library, object library, or source list linked into tests, instrument that target as well as the test executable. ## Coverage-Targeting Guidance To avoid low coverage deltas: 1. Read the coverage report and identify uncovered production files, functions, and branch lines. 2. Write tests that drive real behavior through public APIs or stable seams. 3. Prioritize branches: error handling, empty input, boundary values, invalid parse cases, feature flags, and state transitions. 4. Prefer one parameterized test that covers many meaningful paths over many tests that repeat the same happy path. 5. Assert observable outcomes, side effects, return codes, exceptions, and mock interactions. A test that only constructs an object usually adds little or no useful coverage. 6. Re-run the targeted coverage command and confirm the intended files moved. Do not chase coverage by testing implementation details that make the suite brittle when a public API can cover the same lines. ## Test Patterns ### GoogleTest assertions | Need | Pattern | |------|---------| | Non-fatal equality | `EXPECT_EQ(actual, expected)` | | Fatal precondition | `ASSERT_NE(pointer, nullptr)` | | Boolean | `EXPECT_TRUE(value)` / `EXPECT_FALSE(value)` | | String equality | `EXPECT_STREQ(actual.c_str(), "expected")` | | Floating point | `EXPECT_NEAR(actual, expected, 1e-6)` | | Exception | `EXPECT_THROW(call(), std::invalid_argument)` | | No exception | `EXPECT_NO_THROW(call())` | Use `ASSERT_*` only when the rest of the test cannot safely continue. ### GoogleTest fixtures and parameterized tests ```cpp class ParserTest : public ::testing::Test { protected: Parser parser_; }; TEST_F(ParserTest, Parse_WithEmptyInput_ReturnsEmptyResult) { EXPECT_TRUE(parser_.Parse("").empty()); } class ClampTest : public ::testing::TestWithParam<std::tuple<int, int, int, int>> {}; TEST_P(ClampTest, Clamp_WithBoundaryInputs_ReturnsExpectedValue) { const auto [value, min, max, expected] = GetParam(); EXPECT_EQ(Clamp(value, min, max), expected); } INSTANTIATE_TEST_SUITE_P( BoundaryCases, ClampTest, ::testing::Values( std::make_tuple(-1, 0, 10, 0), std::make_tuple(5, 0, 10, 5), std::make_tuple(11, 0, 10, 10))); ``` ### Catch2 assertions and generators ```cpp #include <catch2/catch_approx.hpp> #include <catch2/catch_test_macros.hpp> #include <catch2/generators/catch_generators.hpp> TEST_CASE("Clamp handles boundary inputs", "[math]") { const auto [value, min, max, expected] = GENERATE( std::tuple{-1, 0, 10, 0}, std::tuple{5, 0, 10, 5}, std::tuple{11, 0, 10, 10}); CHECK(Clamp(value, min, max) == expected); } TEST_CASE("Divide rejects zero denominator", "[math]") { REQUIRE_THROWS_AS(Divide(1.0, 0.0), std::invalid_argument); CHECK(Divide(1.0, 3.0) == Catch::Approx(0.333333).epsilon(0.001)); } ``` Use `REQUIRE` when execution must stop after failure; use `CHECK` for independent assertions. ## Mocking Rules Use GoogleMock when the repo already uses gMock or GoogleTest with mocks: ```cpp #include <gmock/gmock.h> class MockClock : public Clock { public: MOCK_METHOD(std::chrono::seconds, Now, (), (const, override)); }; TEST(SchedulerTest, ShouldRun_WhenIntervalElapsed_ReturnsTrue) { MockClock clock; EXPECT_CALL(clock, Now()).WillOnce(::testing::Return(std::chrono::seconds{42})); Scheduler scheduler(clock); EXPECT_TRUE(scheduler.ShouldRun()); } ``` Guidelines: - Mock interfaces with virtual methods and virtual destructors. - Prefer small interfaces or constructor injection over global state. - For code without virtuals, create seams with templates, function objects, adapters, or thin interfaces around external dependencies. - Do not mock standard library containers or value objects; build real values. - If a test needs more than three mocks, treat it as a design smell and look for a higher-level behavioral test. ## Testing Internals If types are not well suited for testing only through their public surface, consider exposing internals to tests using a preprocessor-guarded `friend` declaration: ```cpp class MyClass { #ifdef UNIT_TESTING friend class MyClassTest; #endif // ... }; ``` Define `UNIT_TESTING` only in the test build configuration so production builds remain unaffected: ```cmake target_compile_definitions(my_component_tests PRIVATE UNIT_TESTING) ``` Use this sparingly. Prefer public behavior tests and dependency seams before adding test-only friendship. ## Common Errors | Error | Fix | |-------|-----| | `undefined reference to __gcov_init` / `__gcov_exit` | Add `--coverage` or `-fprofile-arcs -ftest-coverage` to link flags, not only compile flags | | No `.gcda` files produced | Ensure the instrumented binary ran to normal exit, the production target was instrumented, and the build directory is writable | | `profiling: ... cannot merge previous GCDA file` | Delete old coverage files or rebuild clean after changing compiler/options | | `gcov: stamp mismatch` | Clean the build directory; `.gcno` and `.gcda` came from different builds | | Clang coverage unreadable by `gcov` | Run `gcovr --gcov-executable 'llvm-cov gcov'` | | `multiple definition of main` | Link `GTest::gtest_main` or `Catch2::Catch2WithMain`, or provide your own main, not both | | CTest reports `No tests were found!!!` | Add `enable_testing()` and `gtest_discover_tests`, `catch_discover_tests`, or `add_test`; run `ctest --test-dir build` | | `fatal error: gtest/gtest.h: No such file or directory` | Use the repo's dependency mechanism or add `find_package(GTest REQUIRED)` / FetchContent only as last resort | | `undefined reference` to production symbols | Link the test target to the library under test with `target_link_libraries(test PRIVATE my_lib)` | | ABI or standard mismatch | Match `CMAKE_CXX_STANDARD`, compiler, runtime, and flags between production and test targets | | `EXPECT_EQ` prints unreadable custom types | Add `operator==` and, if useful, `operator<<` in the test namespace or production type namespace | | Test passes directly but not under CTest | Check working directory assumptions; set `WORKING_DIRECTORY` in `add_test` or use paths relative to test data | | Segfault in test cleanup | Avoid owning raw pointers in tests; use RAII objects and make mock lifetimes outlive the object under test | ## Dependency Installation (Last Resort) Only add dependencies after investigation confirms the repo has no test framework or the expected framework is missing. Prefer existing package managers and lockfiles. CMake with installed packages: ```cmake find_package(GTest REQUIRED) find_package(Catch2 3 REQUIRED) ``` CMake FetchContent fallback for GoogleTest: ```cmake include(FetchContent) FetchContent_Declare( googletest URL https://github.com/google/googletest/archive/refs/tags/v1.14.0.zip) FetchContent_MakeAvailable(googletest) ``` CMake FetchContent fallback for Catch2 v3: ```cmake include(FetchContent) FetchContent_Declare( Catch2 URL https://github.com/catchorg/Catch2/archive/refs/tags/v3.5.4.zip) FetchContent_MakeAvailable(Catch2) list(APPEND CMAKE_MODULE_PATH ${catch2_SOURCE_DIR}/extras) ``` Package manager examples: | Manager | Command | |---------|---------| | vcpkg | `vcpkg install gtest catch2` | | Conan | `conan install . --build=missing` | | apt | `sudo apt-get install libgtest-dev catch2 gcovr lcov` | | Homebrew | `brew install googletest catch2 gcovr lcov` | On Debian/Ubuntu, `libgtest-dev` historically installs only GoogleTest *sources* (no prebuilt libraries or CMake package config), so `find_package(GTest REQUIRED)` can still fail. On those systems either build/install GoogleTest from the packaged sources, add it via `FetchContent`, or use vcpkg/Conan instead of relying on apt alone. Do not vendor dependencies by copying source into the repo unless that is already the project's dependency policy. ## Skip Skip or avoid these actions unless the repo explicitly requires them: - Do not replace the build system or create a parallel test harness outside CMake/Make/Bazel just for generated tests. - Do not add coverage flags globally to release builds; keep coverage in a Debug/test-only configuration. - Do not use system `gcov` with Clang-generated coverage data. - Do not define a second test `main` when linking framework-provided main targets. - Do not write tests that only instantiate objects without assertions or behavior coverage. - Do not add a new test framework when an existing one is already configured. -
dotnet-examples.md 10.7 KB
# .NET Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a .NET/C# codebase. These show what each pipeline phase produces for a small project. ## Source Under Test A simple `InvoiceService` in a .NET 9 project using MSTest: ```text src/ Contoso.Billing/ Contoso.Billing.csproj InvoiceService.cs Invoice.cs IInvoiceRepository.cs tests/ Contoso.Billing.Tests/ Contoso.Billing.Tests.csproj (exists, references Contoso.Billing) Contoso.Billing.sln ``` ```csharp // InvoiceService.cs namespace Contoso.Billing; public class InvoiceService(IInvoiceRepository repository) { public decimal CalculateTotal(Invoice invoice) { if (invoice is null) throw new ArgumentNullException(nameof(invoice)); if (invoice.LineItems.Count == 0) throw new InvalidOperationException("Invoice has no line items."); var subtotal = invoice.LineItems.Sum(li => li.Quantity * li.UnitPrice); var tax = subtotal * invoice.TaxRate; return Math.Round(subtotal + tax, 2); } public async Task<Invoice> GetByIdAsync(int id) { var invoice = await repository.FindAsync(id); return invoice ?? throw new KeyNotFoundException($"Invoice {id} not found."); } public async Task MarkAsPaidAsync(int id) { var invoice = await repository.FindAsync(id) ?? throw new KeyNotFoundException($"Invoice {id} not found."); if (invoice.Status == InvoiceStatus.Paid) throw new InvalidOperationException("Invoice is already paid."); invoice.Status = InvoiceStatus.Paid; invoice.PaidDate = DateTime.UtcNow; await repository.UpdateAsync(invoice); } } ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: C:\src\Contoso.Billing - **Language**: C# (.NET 9) - **Framework**: .NET 9 (net9.0) - **Test Framework**: MSTest 3.8 ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Build**: `dotnet build Contoso.Billing.sln` - **Test**: `dotnet test Contoso.Billing.sln` - **Lint**: `dotnet format Contoso.Billing.sln` ## Project Structure - Source: `src/Contoso.Billing/` - Tests: `tests/Contoso.Billing.Tests/` (exists, empty) ## Files to Test ### High Priority | File | Classes/Functions | Testability | Notes | |------|-------------------|-------------|-------| | src/Contoso.Billing/InvoiceService.cs | InvoiceService: CalculateTotal, GetByIdAsync, MarkAsPaidAsync | High | Core business logic, repository dependency needs mocking | ### Low Priority / Skip | File | Reason | |------|--------| | src/Contoso.Billing/Invoice.cs | Data model, no logic | | src/Contoso.Billing/IInvoiceRepository.cs | Interface, no implementation | ## Existing Tests - No existing tests found ## Existing Test Projects - **Project file**: `tests/Contoso.Billing.Tests/Contoso.Billing.Tests.csproj` - **Target source project**: `src/Contoso.Billing/Contoso.Billing.csproj` - **Test files**: none ## Testing Patterns - No existing patterns; recommend sealed test classes, AAA structure, `Moq` for mocking IInvoiceRepository ## Recommendations - Start with InvoiceService.CalculateTotal (pure logic, easy to test) - Then async methods (require mocking IInvoiceRepository) ``` ## Sample Plan Output What `code-testing-planner` produces in `<TESTAGENT_DIR>/plan.md`: ```markdown # Test Implementation Plan ## Overview Generate MSTest tests for the Contoso.Billing InvoiceService, covering all three public methods across happy path, edge case, and error scenarios. Single phase since there is only one source file. ## Commands - **Build**: `dotnet build tests/Contoso.Billing.Tests/Contoso.Billing.Tests.csproj` - **Test**: `dotnet test tests/Contoso.Billing.Tests/Contoso.Billing.Tests.csproj` - **Lint**: `dotnet format --include tests/Contoso.Billing.Tests/` ## Phase Summary | Phase | Focus | Files | Est. Tests | |-------|-------|-------|------------| | 1 | InvoiceService | 1 | 9-12 | --- ## Phase 1: InvoiceService ### Overview Cover all public methods of InvoiceService. CalculateTotal is pure logic tested with DataRow. Async methods require a mocked IInvoiceRepository. ### Files to Test #### 1. InvoiceService.cs - **Source**: `src/Contoso.Billing/InvoiceService.cs` - **Test File**: `tests/Contoso.Billing.Tests/InvoiceServiceTests.cs` - **Test Class**: `InvoiceServiceTests` **Methods to Test**: 1. `CalculateTotal` — Pure calculation logic - Happy path: single line item returns quantity × price + tax - Happy path: multiple line items summed correctly - Edge case: zero tax rate returns subtotal only - Error case: null invoice throws ArgumentNullException - Error case: empty line items throws InvalidOperationException 2. `GetByIdAsync` — Repository lookup - Happy path: existing ID returns invoice - Error case: non-existent ID throws KeyNotFoundException 3. `MarkAsPaidAsync` — State transition - Happy path: unpaid invoice transitions to Paid with PaidDate set - Error case: already paid throws InvalidOperationException - Error case: non-existent ID throws KeyNotFoundException ### Success Criteria - [ ] All test files created - [ ] Tests compile with `dotnet build` - [ ] All tests pass with `dotnet test` ``` ## Sample Generated Test File What `code-testing-implementer` produces: ```csharp using Microsoft.VisualStudio.TestTools.UnitTesting; using Moq; using Contoso.Billing; namespace Contoso.Billing.Tests; [TestClass] public sealed class InvoiceServiceTests { private readonly Mock<IInvoiceRepository> _repositoryMock = new(); private readonly InvoiceService _sut; public InvoiceServiceTests() { _sut = new InvoiceService(_repositoryMock.Object); } // --- CalculateTotal --- [TestMethod] [DataRow(1, 100.00, 0.10, 110.00, DisplayName = "Single item with 10% tax")] [DataRow(3, 25.00, 0.0, 75.00, DisplayName = "Multiple quantity, zero tax")] public void CalculateTotal_ValidLineItems_ReturnsExpectedTotal( int quantity, double unitPrice, double taxRate, double expected) { // Arrange var invoice = new Invoice { TaxRate = (decimal)taxRate, LineItems = [new LineItem { Quantity = quantity, UnitPrice = (decimal)unitPrice }] }; // Act var total = _sut.CalculateTotal(invoice); // Assert Assert.AreEqual((decimal)expected, total); } [TestMethod] public void CalculateTotal_NullInvoice_ThrowsArgumentNullException() { Assert.ThrowsExactly<ArgumentNullException>(() => _sut.CalculateTotal(null!)); } [TestMethod] public void CalculateTotal_EmptyLineItems_ThrowsInvalidOperationException() { // Arrange var invoice = new Invoice { LineItems = [] }; // Act & Assert Assert.ThrowsExactly<InvalidOperationException>(() => _sut.CalculateTotal(invoice)); } // --- GetByIdAsync --- [TestMethod] public async Task GetByIdAsync_ExistingId_ReturnsInvoice() { // Arrange var expected = new Invoice { Id = 42 }; _repositoryMock.Setup(r => r.FindAsync(42)).ReturnsAsync(expected); // Act var result = await _sut.GetByIdAsync(42); // Assert Assert.AreSame(expected, result); } [TestMethod] public async Task GetByIdAsync_NonExistentId_ThrowsKeyNotFoundException() { // Arrange _repositoryMock.Setup(r => r.FindAsync(999)).ReturnsAsync((Invoice?)null); // Act & Assert await Assert.ThrowsExactlyAsync<KeyNotFoundException>( () => _sut.GetByIdAsync(999)); } // --- MarkAsPaidAsync --- [TestMethod] public async Task MarkAsPaidAsync_UnpaidInvoice_SetsStatusAndDate() { // Arrange var invoice = new Invoice { Id = 1, Status = InvoiceStatus.Pending }; _repositoryMock.Setup(r => r.FindAsync(1)).ReturnsAsync(invoice); // Act await _sut.MarkAsPaidAsync(1); // Assert Assert.AreEqual(InvoiceStatus.Paid, invoice.Status); Assert.IsNotNull(invoice.PaidDate); _repositoryMock.Verify(r => r.UpdateAsync(invoice), Times.Once); } [TestMethod] public async Task MarkAsPaidAsync_AlreadyPaid_ThrowsInvalidOperationException() { // Arrange var invoice = new Invoice { Id = 1, Status = InvoiceStatus.Paid }; _repositoryMock.Setup(r => r.FindAsync(1)).ReturnsAsync(invoice); // Act & Assert await Assert.ThrowsExactlyAsync<InvalidOperationException>( () => _sut.MarkAsPaidAsync(1)); } } ``` ## Sample Fix Cycle When the implementer encounters a build error, the fixer agent diagnoses and resolves it: **Build output:** ```text error CS0246: The type or namespace name 'Moq' could not be found (are you missing a using directive or an assembly reference?) ``` **Fixer diagnosis:** The test project is missing the Moq NuGet package. **Fix applied:** ```bash dotnet add tests/Contoso.Billing.Tests/Contoso.Billing.Tests.csproj package Moq ``` **Rebuild:** `dotnet build tests/Contoso.Billing.Tests/Contoso.Billing.Tests.csproj` → SUCCESS --- **Another common cycle:** **Build output:** ```text error CS7036: There is no argument given that corresponds to the required parameter 'repository' of 'InvoiceService.InvoiceService(IInvoiceRepository)' ``` **Fixer diagnosis:** Test code instantiated `new InvoiceService()` without passing the required constructor parameter. The source uses a primary constructor with an `IInvoiceRepository` dependency. **Fix applied:** ```csharp // Before (wrong) var sut = new InvoiceService(); // After (fixed) var repositoryMock = new Mock<IInvoiceRepository>(); var sut = new InvoiceService(repositoryMock.Object); ``` **Rebuild:** SUCCESS ## Sample Final Report What `code-testing-generator` produces at Step 9: ```markdown ## Test Generation Report **Project**: Contoso.Billing **Strategy**: Single pass ### Results | Metric | Value | |----------------|-------| | Tests created | 9 | | Tests passing | 9 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `tests/Contoso.Billing.Tests/InvoiceServiceTests.cs` (9 tests) ### Coverage - InvoiceService.CalculateTotal — 3 happy path, 2 error cases - InvoiceService.GetByIdAsync — 1 happy path, 1 error case - InvoiceService.MarkAsPaidAsync — 1 happy path, 1 error case ### Build Validation - Scoped build: ✅ passed - Full solution build (`dotnet build --no-incremental`): ✅ passed ### Next Steps - Add integration tests for repository layer if needed - Consider testing with multiple line items for CalculateTotal ``` -
dotnet.md 9.6 KB
# .NET Extension Language-specific guidance for .NET (C#/F#/VB) test generation. ## Project System Detection Determine the project system before choosing any command or editing a manifest. | Signal | Project system | Consequence | |---|---|---| | Root `<Project Sdk="...">` or an `Sdk` attribute | SDK-style | `dotnet build` / `dotnet test` are normally valid; new `*.cs` files are usually included by glob | | `ToolsVersion`, `Microsoft.Common.props` / `Microsoft.CSharp.targets` imports, explicit `<Reference>` and `<Compile Include>` items | Classic non-SDK | Preserve the repository's MSBuild / test-runner commands; every new source or test file must be added to the project | | `packages.config` beside the project | Classic NuGet dependency management | Preserve `packages.config` and assembly references; do not run `dotnet add package` or introduce `PackageReference` unless the user explicitly requested a migration | For classic projects, inspect repository scripts, CI configuration, `README*`, and `AGENTS.md` for the authoritative build and test commands. Common commands are `MSBuild.exe` followed by `vstest.console.exe` or `MSTest.exe`, but the checked-in command wins. If no compatible runner is installed, report that blocker instead of migrating the project or claiming `dotnet test` succeeded. ## Build Commands | Scope | Command | |-------|---------| | SDK-style test project | `dotnet build MyProject.Tests.csproj` | | SDK-style solution (final validation) | `dotnet build MySolution.sln --no-incremental` | | Classic non-SDK project | Use the repository's existing MSBuild command (often `MSBuild.exe MySolution.sln /t:Build`) | - Use `--no-restore` if dependencies are already restored - Use `-v:q` (quiet) to reduce output noise - Always use `--no-incremental` for the final validation build — incremental builds hide errors like CS7036 ## Test Commands | Scope | Command | |-------|---------| | SDK-style all tests | `dotnet test` | | SDK-style filtered | `dotnet test --filter "FullyQualifiedName~ClassName"` | | SDK-style after build | `dotnet test --no-build` | | Classic non-SDK | Use the checked-in runner command; commonly `vstest.console.exe <test.dll>` after MSBuild | - Use `--no-build` if already built - Use `-v:q` for quieter output ## Lint Command ```bash dotnet format --include path/to/file.cs dotnet format MySolution.sln # full solution ``` ## Project Reference Validation Before writing test code, read the test project's `.csproj` to verify it has `<ProjectReference>` entries for the assemblies your tests will use. If a reference is missing, add it: ```xml <ItemGroup> <ProjectReference Include="../SourceProject/SourceProject.csproj" /> </ItemGroup> ``` This prevents CS0234 ("namespace not found") and CS0246 ("type not found") errors. In a classic project, preserve its existing `<ProjectReference>` metadata and configuration mappings instead of replacing them with the SDK-style shorthand. ## Common CS Error Codes | Error | Meaning | Fix | |-------|---------|-----| | CS0234 | Namespace not found | Add `<ProjectReference>` to the source project in the test `.csproj` | | CS0246 | Type not found | Add `using Namespace;` or add missing `<ProjectReference>` | | CS0103 | Name not found | Check spelling, add `using` statement | | CS1061 | Missing member | Verify method/property name matches the source code exactly | | CS0029 | Type mismatch | Cast or change the type to match the expected signature | | CS7036 | Missing required parameter | Read the constructor/method signature and pass all required arguments | ## `.csproj` / `.sln` Handling - During phase implementation, build only the specific test `.csproj` for speed - For the final validation, build the full `.sln` with `--no-incremental` - Full-solution builds catch cross-project reference errors invisible in scoped builds ### Registering test code with the build (MANDATORY) Before writing a new C# test file, inspect the test project's compile items. - SDK-style projects normally include `*.cs` by glob. Do not add a redundant `<Compile Include>` unless default compile items are disabled. - Classic non-SDK projects require an explicit item for every new file. Add a path relative to the project, preserving its path separator and ordering: ```xml <Compile Include="Services\OrderServiceTests.cs" /> ``` After editing, re-open the project and verify the exact new test path appears once. A file on disk that is missing from a classic project's compile items is not part of the test assembly and must never be reported as generated coverage. ### Registering a new test project (MANDATORY when `dotnet new` was used) A new `.csproj` is **invisible** to `dotnet test <solution>`, to `dotnet test` run from the repo root, and to any CI/benchmark harness until it is added to the solution. Run `dotnet sln add` *immediately* after creating the project as part of Step 3 ("Register Test Project with Build System") — do not defer it to a later step. 1. Use the exact solution or solution-filter target identified in the research or plan document under `<TESTAGENT_DIR>` — do not search for or substitute a different `.sln`, `.slnx`, or `.slnf` target. 2. If that target is a `.sln` or `.slnx`, run `dotnet sln <solution> add <test-project.csproj>`. 3. If the target is a `.slnf` (solution filter), also ensure the new project is included in the filter; adding only to the underlying `.sln` may not be enough for test discovery. 4. Skip this if the project is already included in the solution or solution filter used for testing. 5. Prefer the researched test command. If you need to run the solution directly, use `dotnet test --solution <solution>` only for repos on .NET SDK 10+ with MTP-style syntax; otherwise use the standard positional form `dotnet test <solution>`. ### Harness Discovery Check Before reporting success, run the **harness-equivalent** discovery command from the repo root and confirm the test count went up by at least the number of tests you generated. The harness (CI, msbench, coverage tools) does not know which `.csproj` you targeted — it runs the solution-level command, so a test that passes via `dotnet test MyProject.Tests.csproj` is still worthless if `dotnet test <solution> --list-tests` doesn't enumerate it. ```bash # From repo root, against the solution identified in <TESTAGENT_DIR>/research.md dotnet test <solution> --list-tests --no-build 2>&1 | grep -c '^ [A-Za-z]' ``` If the delta is `0`, the new project isn't in the solution. Run `dotnet sln <solution> add <test-project.csproj>` and re-run the check. Do **not** report success until the harness command sees your new tests. For a classic non-SDK project, use the repository's normal build and discovery command instead of the example above. The minimum acceptable check is: 1. the new file is present exactly once as `<Compile Include="...">`; 2. the classic project builds with its documented MSBuild command; and 3. the repository's test runner discovers the new test(s). If the environment lacks the required Visual Studio/MSBuild/test-runner toolchain, verify item registration, report that execution is blocked, and do not substitute `dotnet test` or modernize the project. ## Test Framework Detection Detect the framework and installed version from the test project's `.csproj`, `packages.config`, and referenced assembly `HintPath` values. Match the existing framework, mocking library, base fixtures, and API level: | Package Reference | Framework | Attributes | Assertion Style | |-------------------|-----------|------------|-----------------| | `MSTest.Sdk` or `MSTest.TestFramework` | MSTest | `[TestClass]`, `[TestMethod]`, `[DataRow]` | `Assert.AreEqual(expected, actual)` | | `xunit` | xUnit | `[Fact]`, `[Theory]`, `[InlineData]` | `Assert.Equal(expected, actual)` | | `NUnit` | NUnit | `[TestFixture]`, `[Test]`, `[TestCase]` | `Assert.That(actual, Is.EqualTo(expected))` | Use the repo's existing framework — do not introduce a different one. For MSTest, load `writing-mstest-tests` only for APIs supported by the installed version. In particular, `Assert.ThrowsExactly` and the unified collection assertions require MSTest 3.8+, while older suites should keep compatible `Assert.ThrowsException`, `StringAssert`, and `CollectionAssert` patterns. Never upgrade MSTest, Moq, NBuilder, or another test dependency merely to use a newer example. ## MSTest Template ```csharp using Microsoft.VisualStudio.TestTools.UnitTesting; namespace ProjectName.Tests; [TestClass] public sealed class ClassNameTests { [TestMethod] public void MethodName_Scenario_ExpectedResult() { // Arrange var sut = new ClassName(); // Act var result = sut.MethodName(input); // Assert Assert.AreEqual(expected, result); } [TestMethod] [DataRow(2, 3, 5, DisplayName = "Positive numbers")] [DataRow(-1, 1, 0, DisplayName = "Negative and positive")] public void Add_ValidInputs_ReturnsSum(int a, int b, int expected) { // Act var result = _sut.Add(a, b); // Assert Assert.AreEqual(expected, result); } } ``` ## Skip Coverage Tools Do not configure or run code coverage measurement tools (coverlet, dotnet-coverage, XPlat Code Coverage) by default. These tools have inconsistent cross-configuration behavior and waste significant time. Coverage is measured separately by the evaluation harness. **SDK-style exception**: if the user or evaluation harness explicitly requires a Cobertura/XML artifact, add `coverlet.collector` as a `PackageReference` so the harness can produce it. For classic non-SDK projects, preserve `packages.config` and use only the repository's existing coverage workflow; never inject a `PackageReference`. Do not run the coverage command yourself. -
go-examples.md 12.4 KB
# Go Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a Go codebase. These show what each pipeline phase produces for a small package. ## Source Under Test A simple `InvoiceService` in a Go module: ```text go.mod (module github.com/contoso/billing) internal/billing/ invoice.go invoice_repository.go (defines the InvoiceRepository interface) invoice_service.go ``` ```go // internal/billing/invoice_service.go package billing import ( "context" "errors" "fmt" "math" "time" ) type InvoiceService struct { repository InvoiceRepository now func() time.Time } func NewInvoiceService(repo InvoiceRepository) *InvoiceService { return &InvoiceService{repository: repo, now: time.Now} } func (s *InvoiceService) CalculateTotal(invoice *Invoice) (float64, error) { if invoice == nil { return 0, errors.New("invoice must not be nil") } if len(invoice.LineItems) == 0 { return 0, errors.New("invoice has no line items") } var subtotal float64 for _, li := range invoice.LineItems { subtotal += float64(li.Quantity) * li.UnitPrice } tax := subtotal * invoice.TaxRate return math.Round((subtotal+tax)*100) / 100, nil } func (s *InvoiceService) GetByID(ctx context.Context, id int) (*Invoice, error) { invoice, err := s.repository.Find(ctx, id) if err != nil { return nil, err } if invoice == nil { return nil, fmt.Errorf("invoice %d not found", id) } return invoice, nil } func (s *InvoiceService) MarkAsPaid(ctx context.Context, id int) error { invoice, err := s.repository.Find(ctx, id) if err != nil { return err } if invoice == nil { return fmt.Errorf("invoice %d not found", id) } if invoice.Status == StatusPaid { return errors.New("invoice is already paid") } invoice.Status = StatusPaid invoice.PaidDate = s.now() return s.repository.Update(ctx, invoice) } ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: /work/billing - **Language**: Go 1.22 (from go.mod) - **Module**: github.com/contoso/billing - **Test Framework**: standard `testing` package (no testify/gomock detected in go.sum) ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Vet**: `go vet ./...` - **Build**: `go build ./...` - **Compile tests**: `go test -count=1 -run=^$ ./internal/billing` - **Test**: `go test -count=1 ./internal/billing` ## Project Structure - Source: `internal/billing/` - Tests: none ## Files to Test ### High Priority | File | Functions | Testability | Notes | |------|-----------|-------------|-------| | internal/billing/invoice_service.go | InvoiceService.CalculateTotal, GetByID, MarkAsPaid | High | Uses InvoiceRepository interface — easy to fake with a hand-written struct | ## Existing Tests - No existing tests found ## Testing Patterns - No existing patterns; recommend white-box `package billing` tests with hand-written fake repository (no testify since the repo doesn't use it), table-driven `t.Run` subtests for CalculateTotal, and an injected `now func() time.Time` for MarkAsPaid. ## Recommendations - Inject `now` instead of stubbing `time.Now` globally — the struct already supports it - Place tests in `internal/billing/invoice_service_test.go` (same package, white-box) ``` ## Sample Plan Output ```markdown # Test Implementation Plan ## Overview Generate standard-library Go tests for InvoiceService using table-driven subtests and a hand-written fake repository. Single phase since there is only one source file. ## Commands - **Compile tests**: `go test -count=1 -run=^$ ./internal/billing` - **Test**: `go test -count=1 -v ./internal/billing` ## Phase 1: InvoiceService ### Files to Test #### 1. invoice_service.go - **Source**: `internal/billing/invoice_service.go` - **Test File**: `internal/billing/invoice_service_test.go` **Functions to Test**: 1. `CalculateTotal` — Table-driven - Happy paths: single item, multi-item, rounding - Error cases: nil invoice, empty line items 2. `GetByID` — happy + missing + repo error 3. `MarkAsPaid` — happy (verifies timestamp via injected clock) + already-paid + missing + repo error ``` ## Sample Generated Test File ```go // internal/billing/invoice_service_test.go package billing import ( "context" "errors" "strings" "testing" "time" ) type fakeRepository struct { findFunc func(ctx context.Context, id int) (*Invoice, error) updateFunc func(ctx context.Context, invoice *Invoice) error updated *Invoice } func (f *fakeRepository) Find(ctx context.Context, id int) (*Invoice, error) { if f.findFunc != nil { return f.findFunc(ctx, id) } return nil, nil } func (f *fakeRepository) Update(ctx context.Context, invoice *Invoice) error { f.updated = invoice if f.updateFunc != nil { return f.updateFunc(ctx, invoice) } return nil } func TestInvoiceService_CalculateTotal(t *testing.T) { tests := []struct { name string invoice *Invoice want float64 wantErr string }{ { name: "single item with 10% tax", invoice: &Invoice{TaxRate: 0.10, LineItems: []LineItem{{Quantity: 1, UnitPrice: 100}}}, want: 110, }, { name: "multi quantity zero tax", invoice: &Invoice{TaxRate: 0, LineItems: []LineItem{{Quantity: 3, UnitPrice: 25}}}, want: 75, }, { name: "rounds half up", invoice: &Invoice{TaxRate: 0.07, LineItems: []LineItem{{Quantity: 2, UnitPrice: 9.99}}}, want: 21.38, }, { name: "nil invoice errors", invoice: nil, wantErr: "invoice must not be nil", }, { name: "empty line items errors", invoice: &Invoice{TaxRate: 0, LineItems: []LineItem{}}, wantErr: "no line items", }, } sut := NewInvoiceService(&fakeRepository{}) for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { got, err := sut.CalculateTotal(tt.invoice) if tt.wantErr != "" { if err == nil || !strings.Contains(err.Error(), tt.wantErr) { t.Fatalf("expected error containing %q, got %v", tt.wantErr, err) } return } if err != nil { t.Fatalf("unexpected error: %v", err) } if got != tt.want { t.Errorf("CalculateTotal = %v, want %v", got, tt.want) } }) } } func TestInvoiceService_GetByID(t *testing.T) { ctx := context.Background() want := &Invoice{ID: 42} t.Run("returns invoice when found", func(t *testing.T) { repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return want, nil }} sut := NewInvoiceService(repo) got, err := sut.GetByID(ctx, 42) if err != nil || got != want { t.Fatalf("got (%v, %v), want (%v, nil)", got, err, want) } }) t.Run("returns not-found error when missing", func(t *testing.T) { repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, nil }} sut := NewInvoiceService(repo) _, err := sut.GetByID(ctx, 999) if err == nil || !strings.Contains(err.Error(), "999") { t.Fatalf("expected error mentioning 999, got %v", err) } }) t.Run("propagates repository error", func(t *testing.T) { boom := errors.New("boom") repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, boom }} sut := NewInvoiceService(repo) _, err := sut.GetByID(ctx, 1) if !errors.Is(err, boom) { t.Fatalf("expected boom error, got %v", err) } }) } func TestInvoiceService_MarkAsPaid(t *testing.T) { ctx := context.Background() fixedTime := time.Date(2025, 1, 1, 12, 0, 0, 0, time.UTC) t.Run("transitions pending invoice to paid", func(t *testing.T) { invoice := &Invoice{ID: 1, Status: StatusPending} repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return invoice, nil }} sut := NewInvoiceService(repo) sut.now = func() time.Time { return fixedTime } if err := sut.MarkAsPaid(ctx, 1); err != nil { t.Fatalf("unexpected error: %v", err) } if invoice.Status != StatusPaid { t.Errorf("status = %v, want %v", invoice.Status, StatusPaid) } if !invoice.PaidDate.Equal(fixedTime) { t.Errorf("paid date = %v, want %v", invoice.PaidDate, fixedTime) } if repo.updated != invoice { t.Errorf("repository was not updated with the invoice") } }) t.Run("rejects already-paid invoice", func(t *testing.T) { invoice := &Invoice{ID: 1, Status: StatusPaid} repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return invoice, nil }} sut := NewInvoiceService(repo) if err := sut.MarkAsPaid(ctx, 1); err == nil || !strings.Contains(err.Error(), "already paid") { t.Fatalf("expected already-paid error, got %v", err) } if repo.updated != nil { t.Errorf("update should not be called for already-paid invoice") } }) t.Run("returns not-found when missing", func(t *testing.T) { repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, nil }} sut := NewInvoiceService(repo) if err := sut.MarkAsPaid(ctx, 999); err == nil || !strings.Contains(err.Error(), "999") { t.Fatalf("expected not-found error, got %v", err) } }) } ``` ## Sample Fix Cycle When the implementer hits a compile or test-runner issue, the fixer agent diagnoses and resolves it. **Test output:** ```text internal/billing/invoice_service_test.go:14:6: cannot use &fakeRepository{} (value of type *fakeRepository) as type InvoiceRepository in argument to NewInvoiceService: *fakeRepository does not implement InvoiceRepository (missing method Update) ``` **Fixer diagnosis:** The fake repository only implemented `Find`. Go enforces full interface implementation at compile time. Add the missing method. **Fix applied:** Add the `Update` method to `fakeRepository` (shown in the test file above). **Rebuild + rerun:** `go test -count=1 ./internal/billing` → SUCCESS --- **Another common cycle — wrong test selection regex:** **Test output:** ```text testing: warning: no tests to run ``` **Fixer diagnosis:** The agent used `go test -run TestInvoiceService_CalculateTotal/single_item` without `^...$` anchors. The Go test runner treats `-run` as a regex; the underscore makes the match too narrow. **Fix applied:** ```bash # Before — bare name without anchors, and an unquoted space would be parsed # by the shell as two separate arguments go test -run 'TestInvoiceService_CalculateTotal/single_item' # After — anchor the subtest name, replace spaces with underscores go test -run '^TestInvoiceService_CalculateTotal$/^single_item_with_10%_tax$' ./internal/billing ``` **Rerun:** SUCCESS ## Sample Final Report ```markdown ## Test Generation Report **Project**: billing (Go) **Strategy**: Direct (single source file in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 11 | | Tests passing | 11 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `internal/billing/invoice_service_test.go` (3 top-level tests, 11 subtests including 5 table cases) ### Coverage - InvoiceService.CalculateTotal — 3 happy + 2 error cases (table-driven) - InvoiceService.GetByID — happy + missing + repo-error - InvoiceService.MarkAsPaid — happy (with fixed clock) + already-paid + missing ### Build / Test Validation - `go vet ./...`: ✅ - `go test -count=1 ./internal/billing`: ✅ PASS ### Next Steps - Add fuzz test (`FuzzCalculateTotal`) if rounding correctness is critical - Consider extracting a `Clock` interface if more time-dependent logic appears ``` -
go.md 7.8 KB
# Go Extension Language-specific guidance for Go test generation. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — find `*_test.go` files and copy their style (table-driven layout, helper usage, assertion library, build tags) 2. **`go.mod` / `go.sum`** — module path, Go version, dependencies (e.g. `testify`, `gomock`, `mockery`) 3. **Build/CI scripts** — `Makefile`, `magefile.go`, `Taskfile.yml`, `.github/workflows/*.yml` 4. **`go.work`** — if present, you are in a workspace; tests for a module must run from that module's directory or use `-C` (Go 1.20+) Use whatever assertion style and test layout the repo already uses. Do not introduce `testify` if the repo uses the standard library only. ## Toolchain Detection | Indicator | Meaning | |-----------|---------| | `go.mod` `go 1.x` directive | Minimum Go version — match it locally with `go version` | | `go.work` at the root | Multi-module workspace; commands resolve dependent modules from sibling directories | | `vendor/` directory | Vendored deps; many commands implicitly add `-mod=vendor` | | `tools.go` with `//go:build tools` | Tool versions pinned in `go.mod` (e.g. `mockgen`); install with `go install` from the listed paths | ## Build Commands | Scope | Command | |-------|---------| | Compile a package | `go build ./path/to/pkg` | | Vet (static analysis) | `go vet ./...` | | Compile tests without running | `go test -count=1 -run=^$ ./path/to/pkg` | | Whole module | `go build ./...` | `go build ./...` is the closest thing to a "does it compile" gate. It does not exercise test files — use `go test -run=^$` to type-check tests as well. ## Test Commands | Scope | Command | |-------|---------| | All tests in a package | `go test ./path/to/pkg` | | All tests in module | `go test ./...` | | Single test | `go test -run '^TestName$' ./path/to/pkg` | | Subtest | `go test -run '^TestName$/^subname$' ./path/to/pkg` | | Verbose | `go test -v ./path/to/pkg` | | Race detector | `go test -race ./...` | | Disable cache | `go test -count=1 ./...` | | Short mode | `go test -short ./...` | - `-run` arguments are **regular expressions anchored** with `^...$`; without anchors the pattern matches as a substring - `go test -count=1` is the canonical way to bypass the test result cache; never use a fake `-count=2` or environment hacks - `-race` significantly slows tests and requires CGO — only enable if the repo's CI does ## Lint Command Use the repo's lint script first (`make lint`, `task lint`). Otherwise detect from `.golangci.yml`/`.golangci.yaml`: - `.golangci.yml` present → `golangci-lint run ./...` - No config → `gofmt -w .` and `go vet ./...` - `goimports` config / pre-commit hook → `goimports -w path/to/file.go` Never disable existing linters in the test files you generate. ## Project Layout and Imports Go uses package paths derived from the module path in `go.mod`. | Scenario | Test placement | Package declaration | |----------|----------------|----------------------| | Internal-only test (white-box) | `foo_test.go` next to `foo.go` | `package foo` (same as production) | | External-only test (black-box) | `foo_test.go` next to `foo.go` | `package foo_test` (forces use of public API) | | Integration / build-tag gated | `foo_integration_test.go` | Add `//go:build integration` at top | - Test files **must** end with `_test.go` — the toolchain ignores other names - A package directory may contain both `package foo` and `package foo_test` test files simultaneously - Helpers shared across tests in one package go in `helpers_test.go` — do not export them; put them in the `_test` package only if integration tests in another package need them - Imports use the full module path: `import "github.com/org/module/pkg"` — copy the exact module path from `go.mod` ## Test Function Signatures | Kind | Signature | |------|-----------| | Standard test | `func TestThing(t *testing.T)` | | Subtests | `t.Run("name", func(t *testing.T) { ... })` | | Benchmark | `func BenchmarkThing(b *testing.B)` | | Example (godoc) | `func ExampleThing()` with `// Output:` comment | | Fuzz (Go 1.18+) | `func FuzzThing(f *testing.F)` | | Per-package setup | `func TestMain(m *testing.M)` — call `m.Run()` and `os.Exit` with its code | Use **table-driven tests** when generating multiple cases for the same behavior — this is idiomatic Go and matches what most repos already use: ```go func TestAdd(t *testing.T) { tests := []struct { name string a, b int want int }{ {"positives", 2, 3, 5}, {"negatives", -1, -1, -2}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := Add(tt.a, tt.b); got != tt.want { t.Errorf("Add(%d,%d) = %d, want %d", tt.a, tt.b, got, tt.want) } }) } } ``` When iterating with `t.Run` over a loop variable on Go < 1.22, capture it with `tt := tt` to avoid closure-over-loop-variable bugs. ## Common Errors | Error | Fix | |-------|-----| | `package X is not in std` / `cannot find module providing package X` | Add the import to `go.mod`: `go get path/to/module@version`, then `go mod tidy` | | `import cycle not allowed in test` | Move shared helpers to a separate package, or switch to a `_test` package for black-box tests | | `undefined: X` in `_test` package | The symbol is unexported; either use `package foo` (white-box) or export it intentionally | | `t.Parallel called multiple times` | Each subtest can call `t.Parallel()` once; do not call it twice in the same test | | `panic: test executed panic(nil) or runtime.Goexit` | A goroutine called `t.Fatal` outside the test goroutine; only the main test goroutine may call `Fatal`/`FailNow` | | `flag provided but not defined: -X` | Flags registered in `init()` of test files must use `flag.NewFlagSet` carefully; place test-only flags in `TestMain` | | `go: cannot find main module` | Run inside the module directory (where `go.mod` lives), or use `-C path` (Go 1.20+) | | `build constraints exclude all Go files in...` | Build tags filtered out every file — match the repo's tag with `-tags=integration` etc. | | `missing go.sum entry for module` | Run `go mod download` or `go mod tidy` | | Race detector reports data race | Fix the race; do not silence it. CGO must be enabled | ## Mocking Rules Go has no reflection-based mocking framework that's universally adopted. Pick what the repo already uses: - **Interfaces + hand-written fakes** (most idiomatic) — define a small interface in the consumer package and pass a struct that implements it - **`gomock` / `mockgen`** — if the repo has `//go:generate mockgen ...` directives or `mocks/` directories, regenerate via `go generate ./...` rather than editing generated files - **`testify/mock`** — used in many repos; instantiate with `new(MockX)` and chain `.On("Method", ...).Return(...)` - **`httptest`** — for HTTP clients/servers; spin up `httptest.NewServer` instead of mocking `http.Client` Always prefer dependency injection over global function patching. If a test needs more than 3 mocks, flag it as a design smell. ## Concurrency and Cleanup - Use `t.Cleanup(func() { ... })` instead of deferring in test bodies — runs even if `t.FailNow` fires - Use `t.TempDir()` for temp files — auto-cleaned at test end - Use `t.Context()` (Go 1.24+) or pass an explicit `context.Background()` — never call real network or filesystem APIs without one in long-running tests ## Dependency Installation (Last Resort) Only install packages after investigation confirms they are missing: ``` go get github.com/stretchr/testify@latest go mod tidy ``` Run `go mod tidy` after any `go get` to keep `go.sum` consistent. Never edit `go.sum` by hand. ## Skip Coverage Tools Do not configure or run coverage tools (`-cover`, `-coverprofile`, `go tool cover`). Coverage is measured separately by the evaluation harness. -
java-examples.md 11.9 KB
# Java Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a Java codebase using JUnit 5 + Mockito. These show what each pipeline phase produces for a small project. ## Source Under Test A simple `InvoiceService` in a Maven project using JUnit 5: ```text pom.xml src/main/java/com/contoso/billing/ InvoiceService.java Invoice.java (mutable POJO with status, taxRate, lineItems and setStatus / setPaidDate mutators) InvoiceStatus.java (enum: PENDING, PAID) InvoiceRepository.java (interface) src/test/java/com/contoso/billing/ (exists, empty) ``` ```java // src/main/java/com/contoso/billing/InvoiceService.java package com.contoso.billing; import java.math.BigDecimal; import java.math.RoundingMode; import java.time.Clock; import java.time.LocalDateTime; import java.util.Optional; public class InvoiceService { private final InvoiceRepository repository; private final Clock clock; public InvoiceService(InvoiceRepository repository) { this(repository, Clock.systemUTC()); } public InvoiceService(InvoiceRepository repository, Clock clock) { this.repository = repository; this.clock = clock; } public BigDecimal calculateTotal(Invoice invoice) { if (invoice == null) { throw new IllegalArgumentException("invoice must not be null"); } if (invoice.lineItems().isEmpty()) { throw new IllegalStateException("Invoice has no line items."); } BigDecimal subtotal = invoice.lineItems().stream() .map(li -> li.unitPrice().multiply(BigDecimal.valueOf(li.quantity()))) .reduce(BigDecimal.ZERO, BigDecimal::add); BigDecimal tax = subtotal.multiply(invoice.taxRate()); return subtotal.add(tax).setScale(2, RoundingMode.HALF_UP); } public Invoice getById(int id) { Optional<Invoice> invoice = repository.find(id); return invoice.orElseThrow( () -> new IllegalArgumentException("Invoice " + id + " not found.")); } public void markAsPaid(int id) { Invoice invoice = repository.find(id) .orElseThrow(() -> new IllegalArgumentException("Invoice " + id + " not found.")); if (invoice.status() == InvoiceStatus.PAID) { throw new IllegalStateException("Invoice is already paid."); } invoice.setStatus(InvoiceStatus.PAID); invoice.setPaidDate(LocalDateTime.now(clock)); repository.update(invoice); } } ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: /work/billing - **Language**: Java 21 (`<maven.compiler.release>21</maven.compiler.release>`) - **Build Tool**: Maven (wrapper `./mvnw` present) - **Test Framework**: JUnit 5 (Jupiter 5.10) + Mockito 5.x (detected in pom.xml) - **Assertion library**: built-in `Assertions` (no AssertJ/Hamcrest in deps) ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Compile**: `./mvnw -q test-compile` - **Test**: `./mvnw -q test` - **Single class**: `./mvnw -q test -Dtest=InvoiceServiceTest` - **Single method**: `./mvnw -q test -Dtest=InvoiceServiceTest#calculateTotal_validLineItems_returnsExpectedTotal` ## Project Structure - Source: `src/main/java/com/contoso/billing/` - Tests: `src/test/java/com/contoso/billing/` (exists, empty) ## Files to Test ### High Priority | File | Classes/Methods | Testability | Notes | |------|-----------------|-------------|-------| | InvoiceService.java | calculateTotal, getById, markAsPaid | High | Repository dependency mockable via Mockito; clock injection available for time-dependent test | ## Testing Patterns - No existing patterns; recommend JUnit 5 + Mockito with `@ExtendWith(MockitoExtension.class)`, `@Mock` / `@InjectMocks` fields, `@ParameterizedTest` + `@CsvSource` for table-driven `calculateTotal`, and `Clock.fixed(...)` for `markAsPaid` timestamp. ## Recommendations - Test class lives in the same package (`com.contoso.billing`) for package-private access if needed - Inject `Clock.fixed(...)` rather than mocking `LocalDateTime.now(...)` — the service already accepts a Clock ``` ## Sample Plan Output ```markdown # Test Implementation Plan ## Overview Generate JUnit 5 + Mockito tests for InvoiceService, covering all three public methods across happy path, edge case, and error scenarios. Single phase since there is only one source file. ## Commands - **Compile**: `./mvnw -q test-compile` - **Test**: `./mvnw -q test -Dtest=InvoiceServiceTest` ## Phase 1: InvoiceService ### Files to Test #### 1. InvoiceService.java - **Source**: `src/main/java/com/contoso/billing/InvoiceService.java` - **Test File**: `src/test/java/com/contoso/billing/InvoiceServiceTest.java` **Methods to Test**: 1. `calculateTotal` — pure logic (parameterized) - Happy paths: single item w/ tax, multi-quantity zero tax, rounding-half-up - Error cases: null invoice → IllegalArgumentException; empty line items → IllegalStateException 2. `getById` — happy + missing 3. `markAsPaid` — happy (verify status + paid date via fixed clock + verify update) + already-paid + missing ``` ## Sample Generated Test File ```java // src/test/java/com/contoso/billing/InvoiceServiceTest.java package com.contoso.billing; import java.math.BigDecimal; import java.time.Clock; import java.time.Instant; import java.time.LocalDateTime; import java.time.ZoneOffset; import java.util.List; import java.util.Optional; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.CsvSource; import org.mockito.InjectMocks; import org.mockito.Mock; import org.mockito.junit.jupiter.MockitoExtension; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertSame; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.never; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; @ExtendWith(MockitoExtension.class) class InvoiceServiceTest { @Mock InvoiceRepository repository; @InjectMocks InvoiceService sut; // --- calculateTotal --- @ParameterizedTest(name = "qty={0} unitPrice={1} taxRate={2} -> {3}") @CsvSource({ "1, 100.00, 0.10, 110.00", "3, 25.00, 0.00, 75.00", "2, 9.99, 0.07, 21.38" }) void calculateTotal_validLineItems_returnsExpectedTotal( int quantity, BigDecimal unitPrice, BigDecimal taxRate, BigDecimal expected ) { Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, taxRate, List.of(new LineItem(quantity, unitPrice))); BigDecimal total = sut.calculateTotal(invoice); assertEquals(0, total.compareTo(expected), () -> "expected " + expected + " but got " + total); } @Test @DisplayName("null invoice throws IllegalArgumentException") void calculateTotal_nullInvoice_throws() { assertThrows(IllegalArgumentException.class, () -> sut.calculateTotal(null)); } @Test void calculateTotal_emptyLineItems_throws() { Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); IllegalStateException ex = assertThrows(IllegalStateException.class, () -> sut.calculateTotal(invoice)); assertEquals("Invoice has no line items.", ex.getMessage()); } // --- getById --- @Test void getById_existingId_returnsInvoice() { Invoice expected = new Invoice(42, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); when(repository.find(42)).thenReturn(Optional.of(expected)); assertSame(expected, sut.getById(42)); } @Test void getById_missingId_throws() { when(repository.find(999)).thenReturn(Optional.empty()); IllegalArgumentException ex = assertThrows(IllegalArgumentException.class, () -> sut.getById(999)); assertEquals("Invoice 999 not found.", ex.getMessage()); } // --- markAsPaid (uses an injected fixed Clock instead of @InjectMocks) --- @Test void markAsPaid_pendingInvoice_transitionsToPaidAndPersists() { Clock fixed = Clock.fixed(Instant.parse("2025-01-01T12:00:00Z"), ZoneOffset.UTC); InvoiceService service = new InvoiceService(repository, fixed); Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); when(repository.find(1)).thenReturn(Optional.of(invoice)); service.markAsPaid(1); assertEquals(InvoiceStatus.PAID, invoice.status()); assertEquals(LocalDateTime.ofInstant(fixed.instant(), ZoneOffset.UTC), invoice.paidDate()); verify(repository).update(invoice); } @Test void markAsPaid_alreadyPaid_throwsAndDoesNotUpdate() { Invoice invoice = new Invoice(1, InvoiceStatus.PAID, BigDecimal.ZERO, List.of()); when(repository.find(1)).thenReturn(Optional.of(invoice)); assertThrows(IllegalStateException.class, () -> sut.markAsPaid(1)); verify(repository, never()).update(any()); } @Test void markAsPaid_missingId_throws() { when(repository.find(999)).thenReturn(Optional.empty()); assertThrows(IllegalArgumentException.class, () -> sut.markAsPaid(999)); } } ``` ## Sample Fix Cycle When the implementer hits a compile or runtime error, the fixer agent diagnoses and resolves it. **Test output:** ```text [ERROR] No tests found for given includes: [com.contoso.billing.InvoiceServiceTest] ``` **Fixer diagnosis:** Surefire only includes `**/*Test.class` (default). The class is `InvoiceServiceTest` (correct) but it was created under `src/test/java/com/contoso/billing/` with **no** package declaration. Maven compiles it into the default package, so `-Dtest=com.contoso.billing.InvoiceServiceTest` doesn't match. **Fix applied:** Add `package com.contoso.billing;` at the top of the test file so it lands in the expected package. **Rebuild + rerun:** `./mvnw -q test -Dtest=InvoiceServiceTest` → SUCCESS --- **Another common cycle — wrong Mockito setup:** **Test output:** ```text org.mockito.exceptions.misusing.UnnecessaryStubbingException: Unnecessary stubbings detected. 1. -> at InvoiceServiceTest.calculateTotal_nullInvoice_throws(InvoiceServiceTest.java:55) ``` **Fixer diagnosis:** `@MockitoExtension` runs in strict mode by default — stubbed calls (`when(repository.find(...)).thenReturn(...)`) must be used. The test stubbed `repository` in a `@BeforeEach` for every test, but `calculateTotal_nullInvoice_throws` never touches the repository. **Fix applied:** Move stubs into the tests that actually need them (as shown in the generated file above), rather than a single shared `@BeforeEach`. **Rebuild + rerun:** SUCCESS ## Sample Final Report ```markdown ## Test Generation Report **Project**: billing (Java / Maven) **Strategy**: Direct (single source file in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 8 | | Tests passing | 8 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `src/test/java/com/contoso/billing/InvoiceServiceTest.java` (8 tests, 3 parameterized cases via @CsvSource) ### Coverage - InvoiceService.calculateTotal — 3 happy path, 2 error cases - InvoiceService.getById — happy + missing - InvoiceService.markAsPaid — happy (fixed Clock) + already-paid + missing ### Build / Test Validation - `./mvnw -q test-compile`: ✅ - `./mvnw -q test`: ✅ Tests run: 8, Failures: 0, Errors: 0 ### Next Steps - Add AssertJ if the team standardises on it (more expressive assertions) - Consider Testcontainers for true repository integration tests ``` -
java.md 9.7 KB
# Java Extension Language-specific guidance for Java test generation. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — find `*Test.java` / `*Tests.java` / `*IT.java` (integration) files and copy their style (JUnit version, assertion library, mock library, lifecycle methods) 2. **Build file** — `pom.xml` (Maven), `build.gradle` / `build.gradle.kts` (Gradle), `BUILD` / `BUILD.bazel` (Bazel) 3. **Java version** — `<maven.compiler.release>`, `sourceCompatibility`, or `toolchains` block 4. **Wrapper scripts** — always prefer `./mvnw` or `./gradlew` over a system-installed Maven/Gradle so you match the project's pinned version Use whatever framework the repo already uses (JUnit 4, JUnit 5/Jupiter, TestNG). Do not migrate to a different framework as a side effect of writing tests. ## Build Tool Detection | Indicator | Build tool | Default test command | |-----------|------------|----------------------| | `pom.xml` | Maven | `./mvnw test` | | `build.gradle` / `build.gradle.kts` | Gradle | `./gradlew test` | | `settings.gradle*` with `include 'subproject'` | Gradle multi-project | `./gradlew :subproject:test` | | `BUILD` / `BUILD.bazel` | Bazel | `bazel test //path/to:test` | If both `pom.xml` and `build.gradle` exist, pick the one used by CI. ## Build Commands | Scope | Maven | Gradle | |-------|-------|--------| | Compile main + test | `./mvnw test-compile` | `./gradlew testClasses` | | Compile only | `./mvnw compile` | `./gradlew classes` | | Full build | `./mvnw verify` | `./gradlew build` | | Skip tests during build | `./mvnw -DskipTests package` | `./gradlew assemble` | - Use `-q` (Maven) / `--console=plain` (Gradle) to reduce output noise - For Gradle, prefer `--no-daemon` only in CI; locally the daemon makes incremental builds far faster ## Test Commands | Scope | Maven | Gradle | |-------|-------|--------| | All unit tests | `./mvnw test` | `./gradlew test` | | Single class | `./mvnw test -Dtest=MyClassTest` | `./gradlew test --tests MyClassTest` | | Single method | `./mvnw test -Dtest=MyClassTest#myMethod` | `./gradlew test --tests MyClassTest.myMethod` | | Tag filter (JUnit 5) | `./mvnw test -Dgroups=fast` | `./gradlew test -PincludeTags=fast` (if configured) or `--tests` | | Integration tests | `./mvnw verify -DskipUnitTests` (with failsafe-plugin) | `./gradlew integrationTest` (if registered) | - `Surefire` runs unit tests (`*Test.java`); `Failsafe` runs integration tests (`*IT.java`) — do not put long integration tests under Surefire - Gradle's `--tests` accepts wildcards: `--tests "*MyMethod*"` - Use `--rerun-tasks` (Gradle) or `-DforkCount=...` (Surefire) only when troubleshooting cache issues ## Lint Command Use the repo's existing lint task first. Otherwise check for: - Checkstyle (`checkstyle.xml`, `<plugin>checkstyle</plugin>`) → `./mvnw checkstyle:check` or `./gradlew checkstyleMain` - Spotless (`spotless` block / plugin) → `./mvnw spotless:apply` or `./gradlew spotlessApply` - ErrorProne / NullAway → integrated into compilation; run a normal build - google-java-format / palantir-java-format → use the repo's configured formatter Never disable existing checks in the test files you generate. ## Project Layout and Imports Maven/Gradle conventional layout: ``` src/ ├── main/java/com/example/foo/Bar.java ├── main/resources/ ├── test/java/com/example/foo/BarTest.java └── test/resources/ ``` | Layout | Test placement | |--------|----------------| | Standard | `src/test/java/<same package as production class>/<ClassName>Test.java` | | Integration tests separated | `src/integrationTest/java/...` (Gradle) or `src/it/java/...` (Maven w/ failsafe) | | Multi-module Maven | Tests live in the same module as the code under test | - Test classes must mirror the production class's **package** to access package-private members - Avoid wildcard imports unless the repo already uses them — match the explicit imports shown in the templates below - For JUnit 5: import `org.junit.jupiter.api.Test` (and other annotations as needed) and `org.junit.jupiter.api.Assertions.assertEquals` etc. as static imports - For JUnit 4: import `org.junit.Test`, `org.junit.Before`, etc., and `org.junit.Assert.assertEquals` etc. as static imports ## Test Framework Detection | Indicator | Framework | Annotations | Assertion style | |-----------|-----------|-------------|------------------| | `junit-jupiter-*` deps | JUnit 5 | `@Test`, `@ParameterizedTest`, `@BeforeEach`, `@DisplayName` | `Assertions.assertEquals(expected, actual)` | | `junit:junit:4.x` | JUnit 4 | `@Test`, `@Before`, `@RunWith` | `Assert.assertEquals(expected, actual)` | | `org.testng:testng` | TestNG | `@Test(groups=...)`, `@BeforeMethod` | `Assert.assertEquals(actual, expected)` (note **reversed** order) | | `org.assertj:assertj-core` | AssertJ (assertions only) | n/a | `assertThat(actual).isEqualTo(expected)` | | `org.hamcrest:hamcrest` | Hamcrest matchers | n/a | `assertThat(actual, is(equalTo(expected)))` | **Argument order matters**: JUnit/AssertJ use `(expected, actual)`; TestNG uses `(actual, expected)`. Reversing them produces confusing failure messages. ## JUnit 5 Template ```java package com.example.foo; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.CsvSource; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; class CalculatorTest { @Test @DisplayName("add returns sum of two positive numbers") void add_positiveNumbers_returnsSum() { Calculator sut = new Calculator(); assertEquals(5, sut.add(2, 3)); } @ParameterizedTest @CsvSource({ "2, 3, 5", "-1, 1, 0" }) void add_validInputs_returnsSum(int a, int b, int expected) { assertEquals(expected, new Calculator().add(a, b)); } @Test void divide_byZero_throws() { Calculator sut = new Calculator(); assertThrows(ArithmeticException.class, () -> sut.divide(1, 0)); } } ``` ## Common Errors | Error | Fix | |-------|-----| | `package X does not exist` | Add the dependency to `pom.xml` / `build.gradle`; run `./mvnw dependency:resolve` or `./gradlew --refresh-dependencies` | | `cannot find symbol` | Verify class name and import path; check that the test source set sees the production source set | | `No tests found for given includes` (Gradle) | `--tests` pattern doesn't match; verify the class/method names, that test methods are annotated with `@Test`, and that the class name matches the test task's `include` pattern (default `**/*Test*.class`). For JUnit 4 only, the class must also be `public` with a public no-arg constructor — JUnit 5 allows package-private classes and methods | | `Test class should have exactly one public zero-argument constructor` (JUnit 4) | Remove constructors with parameters; use `@Before` for setup | | `org.junit.runners.model.InvalidTestClassError` (JUnit 4) | Class is missing `public`, has wrong constructor, or method signature is wrong | | Mixing `org.junit.Test` (4) and `org.junit.jupiter.api.Test` (5) | Pick one framework per test class — imports must match the framework annotation | | `java.lang.NoClassDefFoundError` at runtime | Test runtime classpath is missing a transitive dep; add it to `testRuntimeOnly` (Gradle) or `<scope>test</scope>` (Maven) | | `UnsupportedClassVersionError` | JDK used to run tests is older than the JDK used to compile; align toolchains | | `Mockito cannot mock final class` | Use Mockito's inline mock maker — Mockito 5+ uses it by default; for Mockito 3.x/4.x add the `mockito-inline` artifact (replaces `mockito-core`). Or switch to MockK for Kotlin. `mockito-subclass` does **not** mock final classes | | `WrongTypeOfReturnValue` (Mockito) | The stubbed method returns a different type than the mock was set up for — check return type signatures | ## Mocking Rules - Use whatever the repo already uses: **Mockito** (most common), **EasyMock**, **JMockit**, or hand-written fakes - For JUnit 5 + Mockito, use `@ExtendWith(MockitoExtension.class)` with `@Mock` / `@InjectMocks` fields - For JUnit 4 + Mockito, use `@RunWith(MockitoJUnitRunner.class)` or `MockitoAnnotations.openMocks(this)` in `@Before` - Use `when(mock.method(...)).thenReturn(...)` for stubs and `verify(mock).method(...)` for interactions - Use `ArgumentCaptor` to assert on complex argument values rather than over-specifying matchers - Prefer constructor injection so production code stays testable without `@InjectMocks` - If a test needs more than 3 mocks, flag it as a design smell ## Spring Boot If the repo uses Spring Boot: - `@SpringBootTest` loads the full context — slow; use only when needed - Slice tests are faster: `@WebMvcTest`, `@DataJpaTest`, `@JsonTest` - Use `@MockBean` (Spring) only inside Spring tests; in plain unit tests use `@Mock` - Use `@Testcontainers` for real-DB integration tests if the repo already has it on the classpath ## Dependency Installation (Last Resort) Only add dependencies after investigation confirms they are missing. Maven (`pom.xml`): ```xml <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter</artifactId> <version>5.10.2</version> <scope>test</scope> </dependency> ``` Gradle (`build.gradle.kts`): ```kotlin testImplementation("org.junit.jupiter:junit-jupiter:5.10.2") testRuntimeOnly("org.junit.platform:junit-platform-launcher") ``` If the repo uses BOMs (`<dependencyManagement>` or Gradle platforms), reuse them — don't pin a different version than the BOM publishes. ## Skip Coverage Tools Do not configure or run coverage tools (JaCoCo, Cobertura, OpenClover). Coverage is measured separately by the evaluation harness. -
kotlin-examples.md 9.8 KB
# Kotlin Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a Kotlin JVM codebase using Gradle + JUnit 5. These show what each pipeline phase produces for a small project. > `kotlin.test` follows the same shape for multiplatform projects. Replace JUnit Jupiter parameterization with `@Test` methods or the repo's established KMP data pattern, and place tests under `src/commonTest/kotlin` or the matching target source set. ## Source Under Test A simple `InvoiceService` in a Gradle Kotlin JVM project: ```text settings.gradle.kts build.gradle.kts src/main/kotlin/com/contoso/billing/ Invoice.kt InvoiceRepository.kt InvoiceService.kt src/test/kotlin/com/contoso/billing/ (exists, empty) ``` ```kotlin // src/main/kotlin/com/contoso/billing/InvoiceService.kt package com.contoso.billing import java.math.BigDecimal import java.math.RoundingMode import java.time.Clock import java.time.LocalDateTime class InvoiceService( private val repository: InvoiceRepository, private val clock: Clock = Clock.systemUTC(), ) { fun calculateTotal(invoice: Invoice): BigDecimal { require(invoice.lineItems.isNotEmpty()) { "Invoice has no line items." } val subtotal = invoice.lineItems .map { it.unitPrice.multiply(BigDecimal.valueOf(it.quantity.toLong())) } .fold(BigDecimal.ZERO, BigDecimal::add) val tax = subtotal.multiply(invoice.taxRate) return subtotal.add(tax).setScale(2, RoundingMode.HALF_UP) } fun getById(id: Int): Invoice = repository.find(id) ?: throw NoSuchElementException("Invoice $id not found.") fun markAsPaid(id: Int) { val invoice = getById(id) check(invoice.status != InvoiceStatus.PAID) { "Invoice is already paid." } invoice.status = InvoiceStatus.PAID invoice.paidAt = LocalDateTime.now(clock) repository.update(invoice) } } ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: /work/contoso-billing - **Language**: Kotlin 2.0 JVM - **Build Tool**: Gradle wrapper present (`./gradlew`) - **Test Framework**: JUnit 5 + kotlin.test assertions (`useJUnitPlatform()` and `junit-jupiter-params` detected) - **Mocking**: MockK is not present; repository is an interface and can be faked directly ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Compile tests**: `./gradlew compileTestKotlin --console=plain` - **Single class**: `./gradlew test --tests "com.contoso.billing.InvoiceServiceTest" --console=plain` - **All tests**: `./gradlew test --console=plain` ## Files to Test ### High Priority | File | Classes/Methods | Testability | Notes | |------|-----------------|-------------|-------| | src/main/kotlin/com/contoso/billing/InvoiceService.kt | calculateTotal, getById, markAsPaid | High | Repository interface is fakeable; Clock is injectable | ## Testing Patterns - No existing patterns; recommend JUnit 5 `@Test`, `@ParameterizedTest` + `@CsvSource`, backticked test names, `kotlin.test` assertions, and a hand-written fake repository. ``` ## Sample Plan Output ```markdown # Test Implementation Plan ## Overview Generate JUnit 5 tests for InvoiceService covering calculation, lookup, and paid-state transition behavior. ## Commands - **Compile tests**: `./gradlew compileTestKotlin --console=plain` - **Test**: `./gradlew test --tests "com.contoso.billing.InvoiceServiceTest" --console=plain` ## Phase 1: InvoiceService ### Files to Test - **Source**: `src/main/kotlin/com/contoso/billing/InvoiceService.kt` - **Test File**: `src/test/kotlin/com/contoso/billing/InvoiceServiceTest.kt` **Methods to Test**: 1. `calculateTotal` — parameterized tax, zero tax, rounding, empty-line-items error 2. `getById` — existing invoice and missing invoice 3. `markAsPaid` — success with fixed clock, already-paid, missing ``` ## Sample Generated Test File ```kotlin // src/test/kotlin/com/contoso/billing/InvoiceServiceTest.kt package com.contoso.billing import org.junit.jupiter.api.Test import org.junit.jupiter.api.assertThrows import org.junit.jupiter.params.ParameterizedTest import org.junit.jupiter.params.provider.CsvSource import java.math.BigDecimal import java.time.Clock import java.time.Instant import java.time.LocalDateTime import java.time.ZoneOffset import kotlin.test.assertEquals import kotlin.test.assertSame import kotlin.test.assertTrue class InvoiceServiceTest { private class FakeRepository : InvoiceRepository { val invoices = mutableMapOf<Int, Invoice>() var updated: Invoice? = null override fun find(id: Int): Invoice? = invoices[id] override fun update(invoice: Invoice) { updated = invoice invoices[invoice.id] = invoice } } @ParameterizedTest(name = "qty={0}, unitPrice={1}, taxRate={2} -> {3}") @CsvSource( "1, 100.00, 0.10, 110.00", "3, 25.00, 0.00, 75.00", "2, 9.99, 0.07, 21.38", ) fun `calculateTotal returns expected total for valid line items`( quantity: Int, unitPrice: BigDecimal, taxRate: BigDecimal, expected: BigDecimal, ) { val service = InvoiceService(FakeRepository()) val invoice = invoice( taxRate = taxRate, lineItems = mutableListOf(LineItem(quantity = quantity, unitPrice = unitPrice)), ) val total = service.calculateTotal(invoice) assertEquals(0, total.compareTo(expected), "expected $expected but got $total") } @Test fun `calculateTotal throws for empty line items`() { val service = InvoiceService(FakeRepository()) val invoice = invoice(lineItems = mutableListOf()) val exception = assertThrows<IllegalArgumentException> { service.calculateTotal(invoice) } assertTrue(exception.message!!.contains("no line items", ignoreCase = true)) } @Test fun `getById returns existing invoice`() { val repository = FakeRepository() val expected = invoice(id = 42) repository.invoices[42] = expected val service = InvoiceService(repository) val result = service.getById(42) assertSame(expected, result) } @Test fun `getById throws for missing invoice`() { val service = InvoiceService(FakeRepository()) val exception = assertThrows<NoSuchElementException> { service.getById(999) } assertTrue(exception.message!!.contains("999")) } @Test fun `markAsPaid updates status date and repository`() { val repository = FakeRepository() val invoice = invoice(id = 1) repository.invoices[1] = invoice val fixedClock = Clock.fixed(Instant.parse("2025-01-01T12:00:00Z"), ZoneOffset.UTC) val service = InvoiceService(repository, fixedClock) service.markAsPaid(1) assertEquals(InvoiceStatus.PAID, invoice.status) assertEquals(LocalDateTime.ofInstant(fixedClock.instant(), ZoneOffset.UTC), invoice.paidAt) assertSame(invoice, repository.updated) } @Test fun `markAsPaid throws and does not update already paid invoice`() { val repository = FakeRepository() repository.invoices[1] = invoice(id = 1, status = InvoiceStatus.PAID) val service = InvoiceService(repository) val exception = assertThrows<IllegalStateException> { service.markAsPaid(1) } assertTrue(exception.message!!.contains("already paid", ignoreCase = true)) assertEquals(null, repository.updated) } private fun invoice( id: Int = 1, status: InvoiceStatus = InvoiceStatus.PENDING, taxRate: BigDecimal = BigDecimal("0.10"), lineItems: MutableList<LineItem> = mutableListOf(LineItem(quantity = 1, unitPrice = BigDecimal("100.00"))), ): Invoice = Invoice(id = id, status = status, taxRate = taxRate, lineItems = lineItems, paidAt = null) } ``` ## Sample Fix Cycle When the implementer hits a Gradle or Kotlin compile issue, the fixer agent diagnoses and resolves it. **Build output:** ```text No tests found for given includes: [com.contoso.billing.InvoiceServiceTest] ``` **Fixer diagnosis:** The test file was created under `src/test/java` with Kotlin source, so the Kotlin JVM source set did not compile it into the expected package. **Fix applied:** Move the file to `src/test/kotlin/com/contoso/billing/InvoiceServiceTest.kt` and keep `package com.contoso.billing` at the top. **Rebuild + rerun:** `./gradlew test --tests "com.contoso.billing.InvoiceServiceTest" --console=plain` → SUCCESS --- **Another common cycle — JUnit Platform not enabled:** **Test output:** ```text 0 tests completed ``` **Fixer diagnosis:** The project has JUnit Jupiter dependencies but the Gradle `test` task is not configured with `useJUnitPlatform()`. **Fix applied:** Match the repo's build convention and add `tasks.test { useJUnitPlatform() }` if it is missing. **Rerun:** SUCCESS ## Sample Final Report ```markdown ## Test Generation Report **Project**: contoso-billing (Kotlin / Gradle) **Strategy**: Direct (single source file in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 8 | | Tests passing | 8 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `src/test/kotlin/com/contoso/billing/InvoiceServiceTest.kt` (8 JUnit 5 tests, 3 parameterized cases) ### Coverage - InvoiceService.calculateTotal — 3 happy path, 1 error case - InvoiceService.getById — found and missing branches - InvoiceService.markAsPaid — success and already-paid branches ### Build / Test Validation - Compile tests: ✅ `./gradlew compileTestKotlin --console=plain` - Test run: ✅ `./gradlew test --tests "com.contoso.billing.InvoiceServiceTest" --console=plain` ``` -
kotlin.md 11.2 KB
# Kotlin Extension Language-specific guidance for Kotlin test generation. For pure-Java codebases, use `java.md` instead. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — find files in `src/test/kotlin/`, `src/commonTest/kotlin/`, `src/jvmTest/kotlin/`, etc., and copy their style (framework, assertion library, mock library, coroutine helpers) 2. **Build file** — `build.gradle.kts` / `build.gradle` — note Kotlin version, plugins (`kotlin("jvm")`, `kotlin("multiplatform")`, `kotlin("android")`), and `dependencies { testImplementation(...) }` 3. **`gradle/libs.versions.toml`** — the version catalog if the repo uses one; reference aliases instead of hard-coded versions 4. **Wrapper script** — always invoke `./gradlew` (Unix) or `.\gradlew.bat` (Windows), never a system-installed Gradle 5. **Multiplatform layout** — `src/<sourceSet>/kotlin/` indicates KMP; tests live in matching `*Test` source sets (`commonTest`, `jvmTest`, `nativeTest`) Use whatever framework the repo already uses (JUnit Jupiter, JUnit 4, Kotest, kotlin.test). Do not switch. ## Project Type Detection | Indicator | Project type | |-----------|--------------| | `kotlin("jvm")` plugin | Plain JVM Kotlin | | `kotlin("multiplatform")` plugin with `kotlin { jvm(); js(); ... }` | Kotlin Multiplatform | | `com.android.application` / `com.android.library` plugin | Android | | `org.springframework.boot` plugin | Spring Boot Kotlin | | `kotlin("jvm")` + `application` plugin | Kotlin CLI / server | For **Android**, see also platform-specific test types: `src/test/` for unit tests on the JVM, `src/androidTest/` for instrumented tests on a device/emulator. They use different runners and gradle tasks. ## Build Commands | Scope | Command | |-------|---------| | Compile main + test (JVM) | `./gradlew compileTestKotlin` | | Full build | `./gradlew build` | | Skip tests | `./gradlew assemble` | | Single module | `./gradlew :module-name:build` | | KMP target only | `./gradlew :module:jvmTest` (or `linuxX64Test`, etc.) | - Use `--console=plain` to suppress Gradle's animated output - Use `--build-cache` (often default in CI) to reuse outputs - For Android: `./gradlew assembleDebug` (build APK) and `./gradlew testDebugUnitTest` (run unit tests) ## Test Commands | Scope | Command | |-------|---------| | All tests (JVM) | `./gradlew test` | | Single class | `./gradlew test --tests "com.example.WidgetTest"` | | Single method | `./gradlew test --tests "com.example.WidgetTest.add returns sum"` | | KMP all targets | `./gradlew allTests` | | KMP one target | `./gradlew jvmTest`, `./gradlew jsTest`, `./gradlew linuxX64Test` | | Android unit tests | `./gradlew testDebugUnitTest` | | Android instrumented | `./gradlew connectedDebugAndroidTest` (requires device/emulator) | - `--tests` accepts wildcards: `--tests "*Widget*"`. Method names with spaces or backticks must be quoted: `--tests "com.example.WidgetTest.creates a widget"` - Use `--rerun-tasks` only when troubleshooting cache issues - For Kotest, the runner is registered with JUnit Platform — the standard `./gradlew test` and `--tests` flags work the same way ## Lint Command Use the repo's lint tooling first: - `./gradlew ktlintCheck` (autoformat: `./gradlew ktlintFormat`) when ktlint is configured - `./gradlew detekt` when detekt is configured - `./gradlew spotlessCheck` / `spotlessApply` for the Spotless plugin - Android Studio's IDE inspections; `./gradlew lint` (Android-only) for the Android Lint task ## Project Layout ``` src/ ├── main/kotlin/com/example/foo/Bar.kt ├── main/resources/ ├── test/kotlin/com/example/foo/BarTest.kt # mirrors production package └── test/resources/ ``` KMP layout: ``` src/ ├── commonMain/kotlin/... # shared ├── commonTest/kotlin/... # shared tests using kotlin.test ├── jvmMain/kotlin/... ├── jvmTest/kotlin/... ├── jsMain/kotlin/... └── jsTest/kotlin/... ``` - Test classes mirror the production class's package so they can access `internal` members (Kotlin's `internal` is module-scoped — within the same Gradle module, including the test source set) - For KMP common tests, you can only import from `kotlin.test` and other multiplatform-aware libraries (e.g. `kotlinx.coroutines.test`, Kotest multiplatform, MockK on JVM only) ## Test Framework Detection | Dependency | Framework | Annotations / DSL | |------------|-----------|--------------------| | `org.jetbrains.kotlin:kotlin-test` | kotlin.test (multiplatform) | `@Test`, `@BeforeTest`, `assertEquals`, `assertFailsWith` | | `junit-jupiter-*` | JUnit 5 | `@Test`, `@ParameterizedTest`, `@BeforeEach`, `@DisplayName` | | `junit:junit:4.x` | JUnit 4 | `@Test`, `@Before`, `@RunWith(JUnitPlatform::class)` rare | | `io.kotest:kotest-runner-junit5` | Kotest | `class FooSpec : FunSpec({ test("...") { ... } })` (DSL — many styles: `StringSpec`, `BehaviorSpec`, etc.) | | `org.spekframework.spek2:spek-dsl-jvm` | Spek 2 | `object FooSpec : Spek({ describe(...) { it(...) {} } })` (legacy) | For Kotest, **stick to the spec style the repo already uses** — mixing styles is confusing. ## Test Templates ### JUnit 5 ```kotlin package com.example.foo import org.junit.jupiter.api.DisplayName import org.junit.jupiter.api.Test import org.junit.jupiter.api.assertThrows import kotlin.test.assertEquals class CalculatorTest { @Test @DisplayName("add returns sum of two positive numbers") fun `add returns sum of two positives`() { val sut = Calculator() assertEquals(5, sut.add(2, 3)) } @Test fun `divide by zero throws`() { val sut = Calculator() assertThrows<ArithmeticException> { sut.divide(1, 0) } } } ``` Backticked method names (`` `like this` ``) are idiomatic for Kotlin tests because they read better in failure messages. ### Kotest (StringSpec) ```kotlin package com.example.foo import io.kotest.core.spec.style.StringSpec import io.kotest.matchers.shouldBe import io.kotest.assertions.throwables.shouldThrow class CalculatorSpec : StringSpec({ "add returns sum of two positive numbers" { Calculator().add(2, 3) shouldBe 5 } "divide by zero throws" { shouldThrow<ArithmeticException> { Calculator().divide(1, 0) } } }) ``` ## Coroutines - Use `kotlinx-coroutines-test` when it's already on the classpath; otherwise add it as a `testImplementation` only after confirming it is missing (see Dependency Installation) - Use `runTest { ... }` (replaces the older `runBlockingTest`) for `suspend` test bodies - For virtual time advance, use a `TestDispatcher` built from `testScheduler` — e.g. `StandardTestDispatcher(testScheduler)` or `UnconfinedTestDispatcher(testScheduler)` — rather than calling `delay` and waiting in real time - Inject a `CoroutineDispatcher` into production code instead of using `Dispatchers.Main/IO` directly — then swap it in tests via `Dispatchers.setMain(testDispatcher)` ```kotlin @Test fun `loads data eventually`() = runTest { val repo = FakeRepo() val dispatcher = StandardTestDispatcher(testScheduler) val sut = Loader(repo, dispatcher) sut.start() advanceUntilIdle() assertEquals(LoadState.Done, sut.state.value) } ``` ## Common Errors | Error | Fix | |-------|-----| | `Unresolved reference: X` | Add the import; verify the test source set sees the production source set; for KMP, the dep may be declared only in `jvmTest` | | `Cannot access 'X': it is internal in module Y` | `internal` is module-scoped, so a test in another Gradle module cannot see it. Move the test into the same module, expose a public seam (e.g. a `*-testing` artifact, or change visibility deliberately), or add the consuming module to the source module's `friend modules` via the Kotlin compiler `-Xfriend-paths` option. `@VisibleForTesting` does **not** widen Kotlin visibility | | `Class 'XTest' is not abstract and does not implement abstract member` (Kotest spec) | The spec class needs a no-arg constructor and a primary-constructor block — match the existing spec style | | `No tests found for given includes` (Gradle) | `--tests` pattern doesn't match; verify class name and that the framework's runner is registered on the test task (`useJUnitPlatform()`) | | `kotlin.UninitializedPropertyAccessException: lateinit property X has not been initialized` | The `@BeforeEach` (or `BeforeTest`) didn't run, or the field was reset; use `lateinit` only after confirming the lifecycle hook fires | | `IllegalStateException: Module with the Main dispatcher had failed to initialize` | Coroutines test needs `Dispatchers.setMain(...)` before launching anything that touches `Dispatchers.Main`; reset with `Dispatchers.resetMain()` in teardown | | `Mockito cannot mock final class` | Kotlin classes are `final` by default — either use **MockK** (works with final classes) or apply the `kotlin-allopen` plugin scoped to a marker annotation | | `MissingMockKException` | The mock wasn't initialized; call `MockKAnnotations.init(this)` or use `@MockK` with `@MockKExtension` (JUnit 5) | | KMP common test references a JVM-only API | Move the test to `jvmTest`, or use `expect/actual` declarations | | Android: `Method ... not mocked` | The unit test runs on the JVM and the SDK class is just a stub — either use Robolectric, move the test to instrumented (`androidTest`), or refactor to inject the dependency | ## Mocking Rules - **MockK** is the de-facto standard for Kotlin (final classes, coroutine support): `every { mock.foo() } returns 1`, `coEvery { mock.suspendFn() } returns 1`, `verify { mock.foo() }`, `coVerify { ... }` - Mockito works on Kotlin too with `mockito-kotlin` extensions, but Kotlin classes are `final` by default — use Mockito's inline mock maker (default in Mockito 5+; the `mockito-inline` artifact for Mockito 3.x/4.x). `mockito-subclass` cannot mock final classes - Avoid `mockkStatic`/`mockkObject` for production code you control — refactor to a wrapper instead - Prefer constructor injection so you don't need framework annotations (`@InjectMocks`) at all - If a test needs more than 3 mocks, flag it as a design smell ## Android Specifics - Robolectric tests live under `src/test/` and emulate the Android framework on the JVM — fast but imperfect - Instrumented tests live under `src/androidTest/`, require a connected device/emulator, and are slow — use sparingly - Compose UI tests use `createComposeRule()` and `composeTestRule.onNodeWithText(...).performClick()` — match the existing test setup if Compose is in the project - Hilt: use `@HiltAndroidTest` and `HiltAndroidRule` for instrumented tests; for unit tests pass fakes directly to ViewModels ## Dependency Installation (Last Resort) Only add dependencies after investigation confirms they are missing. `build.gradle.kts`: ```kotlin dependencies { testImplementation("org.junit.jupiter:junit-jupiter:5.10.2") testImplementation("io.mockk:mockk:1.13.10") testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.8.0") } tasks.test { useJUnitPlatform() } ``` If the repo uses a version catalog, add to `gradle/libs.versions.toml` and reference via `libs.junit.jupiter` etc. Match the major versions already in use. ## Skip Coverage Tools Do not configure or run coverage tools (JaCoCo, Kover). Coverage is measured separately by the evaluation harness. -
powershell-examples.md 8.4 KB
# PowerShell Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a PowerShell module using Pester v5. These show what each pipeline phase produces for a small module. ## Source Under Test A simple `InvoiceService` module: ```text src/ Contoso.Billing.psd1 Contoso.Billing.psm1 Tests/ (empty) ``` ```powershell # src/Contoso.Billing.psm1 enum InvoiceStatus { Pending Paid } function Get-InvoiceTotal { [CmdletBinding()] param([Parameter(Mandatory)][pscustomobject]$Invoice) if (-not $Invoice.LineItems -or $Invoice.LineItems.Count -eq 0) { throw 'Invoice has no line items.' } $subtotal = 0 foreach ($lineItem in $Invoice.LineItems) { $subtotal += $lineItem.Quantity * $lineItem.UnitPrice } [math]::Round($subtotal + ($subtotal * $Invoice.TaxRate), 2) } function Get-InvoiceById { [CmdletBinding()] param([Parameter(Mandatory)][int]$Id, [Parameter(Mandatory)][scriptblock]$FindInvoice) $invoice = & $FindInvoice $Id if ($null -eq $invoice) { throw "Invoice $Id not found." } $invoice } function Set-InvoicePaid { [CmdletBinding()] param( [Parameter(Mandatory)][int]$Id, [Parameter(Mandatory)][scriptblock]$FindInvoice, [Parameter(Mandatory)][scriptblock]$UpdateInvoice, [scriptblock]$GetNow = { Get-Date } ) $invoice = Get-InvoiceById -Id $Id -FindInvoice $FindInvoice if ($invoice.Status -eq [InvoiceStatus]::Paid) { throw 'Invoice is already paid.' } $invoice.Status = [InvoiceStatus]::Paid $invoice.PaidDate = & $GetNow & $UpdateInvoice $invoice } Export-ModuleMember -Function Get-InvoiceTotal, Get-InvoiceById, Set-InvoicePaid ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: C:\work\contoso-billing - **Language**: PowerShell 7.4 - **Module**: `src/Contoso.Billing.psd1` imports `Contoso.Billing.psm1` - **Test Framework**: Pester v5 ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Module load**: `Import-Module ./src/Contoso.Billing.psd1 -Force -ErrorAction Stop` - **Discovery**: `Invoke-Pester -Configuration @{ Run = @{ Path = './Tests'; PassThru = $true; SkipRun = $true } }` - **Test**: `Invoke-Pester -Path ./Tests -Output Detailed` ## Files to Test ### High Priority | File | Functions | Testability | Notes | |------|-----------|-------------|-------| | src/Contoso.Billing.psm1 | Get-InvoiceTotal, Get-InvoiceById, Set-InvoicePaid | High | Dependencies are scriptblocks, easy to fake; clock is injectable | ## Testing Patterns - No existing patterns; recommend Pester v5 `Describe` / `Context` / `It`, `BeforeAll` module import, `-TestCases` for total calculations, and scriptblock fakes for repository operations. ``` ## Sample Plan Output ```markdown # Test Implementation Plan ## Overview Generate Pester v5 tests for total calculation, repository lookup, and the paid-state transition. Single phase since there is one module file. ## Commands - **Import**: `Import-Module ./src/Contoso.Billing.psd1 -Force -ErrorAction Stop` - **Test**: `Invoke-Pester -Path ./Tests/Contoso.Billing.Tests.ps1 -Output Detailed` ## Phase 1: Contoso.Billing ### Files to Test - **Source**: `src/Contoso.Billing.psm1` - **Test File**: `Tests/Contoso.Billing.Tests.ps1` **Functions to Test**: 1. `Get-InvoiceTotal` — table-driven happy paths and empty-line-items error 2. `Get-InvoiceById` — existing invoice and missing invoice 3. `Set-InvoicePaid` — status/date update and persistence; already-paid error; missing invoice error ``` ## Sample Generated Test File ```powershell # Tests/Contoso.Billing.Tests.ps1 BeforeAll { Import-Module (Join-Path $PSScriptRoot '..' 'src' 'Contoso.Billing.psd1') -Force -ErrorAction Stop function New-TestInvoice { param( [int]$Id = 1, [InvoiceStatus]$Status = [InvoiceStatus]::Pending, [double]$TaxRate = 0.10, [object[]]$LineItems = @(@{ Quantity = 1; UnitPrice = 100.00 }) ) [pscustomobject]@{ Id = $Id Status = $Status TaxRate = $TaxRate LineItems = $LineItems PaidDate = $null } } } Describe 'Contoso.Billing invoice functions' { Context 'Get-InvoiceTotal' { It 'returns <Expected> for <Name>' -TestCases @( @{ Name = 'single item with tax'; LineItems = @(@{ Quantity = 1; UnitPrice = 100.00 }); TaxRate = 0.10; Expected = 110.00 } @{ Name = 'multi quantity zero tax'; LineItems = @(@{ Quantity = 3; UnitPrice = 25.00 }); TaxRate = 0.00; Expected = 75.00 } @{ Name = 'rounds to two decimals'; LineItems = @(@{ Quantity = 2; UnitPrice = 9.99 }); TaxRate = 0.07; Expected = 21.38 } ) { param($LineItems, $TaxRate, $Expected) $invoice = New-TestInvoice -LineItems $LineItems -TaxRate $TaxRate Get-InvoiceTotal -Invoice $invoice | Should -BeExactly $Expected } It 'throws when the invoice has no line items' { $invoice = New-TestInvoice -LineItems @() { Get-InvoiceTotal -Invoice $invoice } | Should -Throw '*no line items*' } } Context 'Get-InvoiceById' { It 'returns an existing invoice' { $expected = New-TestInvoice -Id 42 $findInvoice = { param($Id) if ($Id -eq 42) { $expected } } $result = Get-InvoiceById -Id 42 -FindInvoice $findInvoice $result | Should -BeSame $expected } It 'throws when the invoice is missing' { $findInvoice = { $null } { Get-InvoiceById -Id 999 -FindInvoice $findInvoice } | Should -Throw '*999*' } } Context 'Set-InvoicePaid' { It 'marks a pending invoice as paid and persists it' { $invoice = New-TestInvoice -Id 1 $script:updatedInvoice = $null $fixedNow = [datetime]'2025-01-01T12:00:00Z' $findInvoice = { param($Id) if ($Id -eq 1) { $invoice } } $updateInvoice = { param($Invoice) $script:updatedInvoice = $Invoice } Set-InvoicePaid -Id 1 -FindInvoice $findInvoice -UpdateInvoice $updateInvoice -GetNow { $fixedNow } $invoice.Status | Should -Be ([InvoiceStatus]::Paid) $invoice.PaidDate | Should -Be $fixedNow $script:updatedInvoice | Should -BeSame $invoice } It 'throws and does not update an already-paid invoice' { $invoice = New-TestInvoice -Status ([InvoiceStatus]::Paid) $script:updatedInvoice = $null $findInvoice = { $invoice } $updateInvoice = { param($Invoice) $script:updatedInvoice = $Invoice } { Set-InvoicePaid -Id 1 -FindInvoice $findInvoice -UpdateInvoice $updateInvoice } | Should -Throw '*already paid*' $script:updatedInvoice | Should -BeNullOrEmpty } } } ``` ## Sample Fix Cycle When the implementer hits a Pester discovery or run issue, the fixer agent diagnoses and resolves it. **Test output:** ```text CommandNotFoundException: The term 'Get-InvoiceTotal' is not recognized ``` **Fixer diagnosis:** The module import was placed at script top level. Import the module in `BeforeAll` so the Pester run phase sees the exported functions. **Fix applied:** Move `Import-Module ... -Force` into `BeforeAll` (as shown above). **Rerun:** `Invoke-Pester -Path ./Tests/Contoso.Billing.Tests.ps1 -Output Detailed` → SUCCESS ## Sample Final Report ```markdown ## Test Generation Report **Project**: contoso-billing (PowerShell) **Strategy**: Direct (single module in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 8 | | Tests passing | 8 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `Tests/Contoso.Billing.Tests.ps1` (8 Pester examples, 3 data-driven total cases) ### Coverage - Get-InvoiceTotal — 3 happy path, 1 error case - Get-InvoiceById — found and missing branches - Set-InvoicePaid — success and already-paid branches ### Build / Test Validation - Module load: ✅ `Import-Module ./src/Contoso.Billing.psd1 -Force -ErrorAction Stop` - Discovery: ✅ Pester found 8 tests - Test run: ✅ `Invoke-Pester -Path ./Tests -Output Detailed` ``` -
powershell.md 10.4 KB
# PowerShell Extension Language-specific guidance for PowerShell test generation using Pester v5. ## Rule #0: Confirm the Test Target If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of scripts already covered by existing `*.Tests.ps1` files. Run these **read-only** discovery commands first — they are the deliberate exception to Rule #1's "before writing any test or running any command" rule, and their output is the ground truth Rule #1's reading is meant to interpret. Do **not** write or execute any tests until Rule #0 and Rule #1 are both complete. | Goal | Command | |------|---------| | List uncommitted edits + untracked files | `git status -s` | | Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | | Recently added scripts/modules | `git log --diff-filter=A --name-only -5 -- '*.ps1' '*.psm1' '*.psd1'` | | Modules with no matching `*.Tests.ps1` | compare `Get-ChildItem -Recurse -Include *.psm1,*.ps1` against `*.Tests.ps1` files | Prefer targets that match **all** of: 1. Untracked or recently added (`git status` / `git log --diff-filter=A`). 2. Small and pure (a few hundred lines, no external state, no `Invoke-WebRequest`/registry/filesystem side effects). 3. Located under a conventional source root (`tools/`, `src/`, `Public/`, `Private/`, or the module root next to a `.psd1`). 4. Have **no** existing matching `*.Tests.ps1` file. If a `.psd1` manifest's `RootModule` (or `ModuleToProcess`) points at a specific `.psm1`, that module is almost certainly the target — start there. ### Test Placement Contract Pester only discovers tests under the path passed to `Invoke-Pester -Path` (or the current directory when no path is given). Verification harnesses (CI, msbench, coverage tools) typically scope discovery to a single directory such as `tools/` or `tests/`. Place every test file there, matching the existing convention in the repo: | Layout used by the repo | Test placement | |-------------------------|----------------| | Co-located convention (`Module.psm1` + `Module.Tests.ps1` side-by-side) | Drop `<Module>.Tests.ps1` next to the source file (`tools/StringUtils.psm1` → `tools/StringUtils.Tests.ps1`). | | Sibling `Tests/` directory | Mirror the source path (`src/Foo/Bar.psm1` → `Tests/Foo/Bar.Tests.ps1`). | | Mixed / unknown | Co-locate next to the source — this is what Pester discovers by default and what most harnesses scope to. | A `*.Tests.ps1` file placed outside the discovery root will be invisible to both `Invoke-Pester` and the harness. ### First-Test Sanity Loop After writing the **first** `*.Tests.ps1` file — before writing any others: 1. Run `Invoke-Pester -Path <dir> -PassThru` and confirm the `TotalCount` is `> 0`. If it is `0`, Pester is not discovering your file; fix the location, filename, or `Describe`/`It` structure before continuing. 2. Run the test (`Invoke-Pester -Path <your.Tests.ps1> -Output Detailed`); fix `Import-Module` / dot-source / `BeforeAll` errors before adding more tests. 3. Only then expand to cover the remaining functions. This catches placement and discovery mistakes on turn 1 instead of after dozens of failed-test iterations. ### Harness Discovery Check Before reporting success, run the **harness-equivalent** discovery command from the repo root and confirm the test count went up by at least the number of tests you generated. CI/msbench/coverage harnesses do not know which directory you targeted with `-Path`; they invoke Pester from the repo root with default discovery, so a test that passes via `Invoke-Pester -Path ./tools/Foo.Tests.ps1` is still worthless if `Invoke-Pester` from the repo root does not enumerate it. ```powershell # From repo root — mirrors what a generic harness sees $result = Invoke-Pester -Configuration @{ Run = @{ PassThru = $true; SkipRun = $true } } "$($result.TotalCount) tests discovered" ``` If the count did not increase, your `*.Tests.ps1` file is outside the harness discovery root. Move it to the convention the repo's existing tests use (or, if there are no existing tests, prefer the repo root's `tests/`, `Tests/`, `tst/`, `test/`, or co-locate next to the source). Do **not** report success until the harness-equivalent command sees your new tests. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — find `*.Tests.ps1` files and copy their style (structure, assertions, mock approach, import method) 2. **Module structure** — look for `.psd1` (manifest), `.psm1` (root module), `Public/`/`Private/` organization 3. **Build/test scripts** — check for `build.ps1`, `Invoke-Build` (`*.build.ps1`), `psake`, or CI scripts 4. **Shell target** — check `.psd1` for `PowerShellVersion`/`CompatiblePSEditions`, CI matrix for `pwsh` vs `powershell.exe` Use the repo's existing test conventions. Only add Pester if the repo has no tests at all. ## Build Commands PowerShell is interpreted — no build step. If the repo has a build script, use it. Otherwise validate with: - **Module loads**: `Import-Module ./MyModule.psd1 -Force -ErrorAction Stop` - **Script analyzer**: `Invoke-ScriptAnalyzer -Path ./src -Recurse` (if PSScriptAnalyzer is available) - **Lint**: `Invoke-ScriptAnalyzer -Path path/to/file.ps1 -Fix` ## Test Commands | Scope | Command | |-------|---------| | All tests | `Invoke-Pester` | | Specific file | `Invoke-Pester -Path ./Tests/Get-Widget.Tests.ps1` | | Filter by name | `Invoke-Pester -FullNameFilter '*Get-Widget*'` | | Filter by tag | `Invoke-Pester -TagFilter 'Unit'` | | Non-interactive (CI) | `Invoke-Pester -CI` | | Detailed output | `Invoke-Pester -Output Detailed` | - Prefer the repo's build/test script over raw `Invoke-Pester` - Use `-Output Detailed` during fix cycles, `-Output Minimal` for final validation ## Project Layout and Imports | Layout | Import in `BeforeAll` | |--------|-----------------------| | Module (`.psd1`) | `Import-Module "$PSScriptRoot/../MyModule.psd1" -Force` | | Library script (defines functions) | `. $PSScriptRoot/Get-Widget.ps1` | | Co-located test | `. $PSCommandPath.Replace('.Tests.ps1', '.ps1')` | | Executable script (has `param()`) | Do **not** dot-source — invoke with `& $PSScriptRoot/script.ps1 -Param value` and assert on output/errors | - **All imports go in `BeforeAll`** — never at script top level - **Use `$PSScriptRoot` or `$PSCommandPath`** — never `$MyInvocation.MyCommand.Path` (returns empty in `BeforeAll`) - Use `-Force` on `Import-Module` to pick up changes between runs ## Test File Naming - Files: `*.Tests.ps1` — match existing convention (co-located vs `Tests/` directory) ## Pester v5 Discovery vs Run (Critical) Pester v5 runs in **two phases**: Discovery (collects test metadata) then Run (executes tests). This is the #1 source of agent errors. **Rules:** - All setup code goes in `BeforeAll` or `BeforeEach` — never at script top level or loose inside `Describe`/`Context` - Code directly inside `Describe`/`Context` (but outside `It`/`Before*`/`After*`) runs during **Discovery** — do not put setup, imports, or variable assignments there - Data for `-ForEach` / `-TestCases` must be set in `BeforeDiscovery`, not `BeforeAll` (BeforeAll runs after discovery) - `-Skip:$condition` evaluates at Discovery time — conditions from `BeforeAll` will be `$null` - Use `foreach` loops for dynamic test generation only with `BeforeDiscovery` data - Use `TestDrive:` for file-based tests instead of touching repo files — Pester cleans it up automatically ## Common Errors | Error | Fix | |-------|-----| | Variable is `$null` in `It` block | Move assignment into `BeforeAll` — variables set there are visible to child `It` blocks without `$script:` | | `-ForEach` data is empty | Move data setup from `BeforeAll` to `BeforeDiscovery` | | `CommandNotFoundException` for Mock target | The function must exist before mocking — import the module in `BeforeAll` first | | `$MyInvocation.MyCommand.Path` returns empty | Use `$PSCommandPath` or `$PSScriptRoot` instead | | `Should Be` (no dash) fails | Use v5 syntax: `Should -Be` (with dash prefix) | | `Assert-MockCalled` not recognized | Use v5 syntax: `Should -Invoke` | | Mock has no effect | Check scope — mocks in `It` only apply to that `It`; use `BeforeAll`/`BeforeEach` for broader scope | | `Should -Throw` doesn't catch cmdlet errors | Most cmdlet errors are non-terminating — wrap with `{ cmd -ErrorAction Stop }` or set `$ErrorActionPreference = 'Stop'` in `BeforeEach` | | Tests pass on Windows but fail on Linux | Use `Join-Path` not string concatenation; match exact file casing; avoid Windows-only cmdlets (Registry, EventLog) | ## Mocking Rules - Place mocks in `BeforeAll` (shared) or `BeforeEach` (reset per test) - Mock where the command is **called from** — use `-ModuleName` to mock inside a module's scope - Use `-ParameterFilter` for selective mocking (no `param()` block needed in v5) - Verify calls with `Should -Invoke` — default scope inside `It` counts only that test's calls - Use `InModuleScope` sparingly and as narrowly as possible — prefer `Mock -ModuleName` for testing via public API - Inside mock bodies, use `$PesterBoundParameters` not `$PSBoundParameters` - If a test needs more than 3 mocks, flag it as a design smell ## Non-Obvious Assertions Most `Should` operators are self-explanatory. These are the ones agents get wrong: - `Should -Throw` requires a **scriptblock**: `{ risky-op } | Should -Throw` — not a direct call - `Should -Contain` is for **collections** — use `Should -Be` for scalar equality - `Should -HaveParameter` validates cmdlet signatures: `Get-Command X | Should -HaveParameter 'Name' -Mandatory` - `Should -Invoke` verifies mock calls: `Should -Invoke Get-Item -Times 1 -Exactly` ## Cross-Platform - Prefer `pwsh` (PowerShell 7+) unless the repo explicitly targets Windows PowerShell 5.1 - Use `Join-Path` for paths — never string concatenation with `\` - Linux/macOS file systems are **case-sensitive** — match exact casing in imports and paths - Windows ships Pester 3.4.0 — if v5 is needed: `Install-Module Pester -Force -SkipPublisherCheck` - Check `$PSVersionTable.PSEdition` to detect Core vs Desktop ## Skip Coverage Tools Do not configure or run coverage tools (Pester CodeCoverage, JaCoCo export). Coverage is measured separately by the evaluation harness. -
python-examples.md 12.1 KB
# Python Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a Python codebase using pytest. These show what each pipeline phase produces for a small project. ## Source Under Test A simple `InvoiceService` in a Python package using pytest: ```text src/ contoso_billing/ __init__.py invoice_service.py invoice.py invoice_repository.py tests/ __init__.py conftest.py (empty, just marks tests/ as a package root) pyproject.toml ``` ```python # src/contoso_billing/invoice_service.py from decimal import Decimal, ROUND_HALF_UP from .invoice import Invoice, InvoiceStatus from .invoice_repository import InvoiceRepository class InvoiceService: def __init__(self, repository: InvoiceRepository) -> None: self._repository = repository def calculate_total(self, invoice: Invoice) -> Decimal: if invoice is None: raise ValueError("invoice must not be None") if not invoice.line_items: raise ValueError("Invoice has no line items.") subtotal = sum( (li.quantity * li.unit_price for li in invoice.line_items), start=Decimal("0"), ) tax = subtotal * invoice.tax_rate return (subtotal + tax).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) def get_by_id(self, invoice_id: int) -> Invoice: invoice = self._repository.find(invoice_id) if invoice is None: raise KeyError(f"Invoice {invoice_id} not found.") return invoice def mark_as_paid(self, invoice_id: int) -> None: invoice = self._repository.find(invoice_id) if invoice is None: raise KeyError(f"Invoice {invoice_id} not found.") if invoice.status == InvoiceStatus.PAID: raise ValueError("Invoice is already paid.") invoice.status = InvoiceStatus.PAID invoice.paid_date = _utcnow() self._repository.update(invoice) def _utcnow(): from datetime import datetime, timezone return datetime.now(timezone.utc) ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: /work/contoso-billing - **Language**: Python 3.11 - **Framework**: pure library (no Flask/Django) - **Test Framework**: pytest 8.x (declared in pyproject.toml [project.optional-dependencies].test) - **Package Layout**: `src/` layout — production package imports as `contoso_billing` ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Install (editable)**: `python -m pip install -e ".[test]"` - **Build/Type-check**: none configured - **Test**: `python -m pytest` - **Lint**: none configured ## Project Structure - Source: `src/contoso_billing/` - Tests: `tests/` (exists, empty besides `conftest.py`) ## Files to Test ### High Priority | File | Classes/Functions | Testability | Notes | |------|-------------------|-------------|-------| | src/contoso_billing/invoice_service.py | InvoiceService: calculate_total, get_by_id, mark_as_paid | High | Core business logic, repository dependency needs mocking | ### Low Priority / Skip | File | Reason | |------|--------| | src/contoso_billing/invoice.py | Dataclass, no logic | | src/contoso_billing/invoice_repository.py | Interface/protocol, no implementation | ## Existing Tests - No existing tests found ## Testing Patterns - No existing patterns; recommend pytest function-style tests in `tests/test_invoice_service.py`, `unittest.mock.Mock(spec=InvoiceRepository)` for repository fakes, and `@pytest.mark.parametrize` for table-driven cases. ## Recommendations - Start with `calculate_total` (pure logic, easy to parametrize) - Then `get_by_id` and `mark_as_paid` (require mocking the repository) - Use `unittest.mock.patch("contoso_billing.invoice_service._utcnow")` to control the timestamp in `mark_as_paid` ``` ## Sample Plan Output What `code-testing-planner` produces in `<TESTAGENT_DIR>/plan.md`: ```markdown # Test Implementation Plan ## Overview Generate pytest tests for the Contoso Billing InvoiceService, covering all three public methods across happy path, edge case, and error scenarios. Single phase since there is only one source file. ## Commands - **Install**: `python -m pip install -e ".[test]"` - **Test**: `python -m pytest tests/test_invoice_service.py -q` - **Test (file-scoped during dev)**: `python -m pytest tests/test_invoice_service.py::test_calculate_total_valid_line_items_returns_expected_total -q` ## Phase Summary | Phase | Focus | Files | Est. Tests | |-------|-------|-------|------------| | 1 | InvoiceService | 1 | 9-12 | --- ## Phase 1: InvoiceService ### Overview Cover all public methods of InvoiceService. `calculate_total` is pure logic tested with `@pytest.mark.parametrize`. The async-looking methods are synchronous but require a mocked InvoiceRepository. ### Files to Test #### 1. invoice_service.py - **Source**: `src/contoso_billing/invoice_service.py` - **Test File**: `tests/test_invoice_service.py` **Methods to Test**: 1. `calculate_total` — Pure calculation logic - Happy path: single line item returns quantity × price + tax - Happy path: multiple line items summed correctly - Edge case: zero tax rate returns subtotal only - Error case: None invoice raises ValueError - Error case: empty line items raises ValueError 2. `get_by_id` — Repository lookup - Happy path: existing ID returns invoice - Error case: missing ID raises KeyError 3. `mark_as_paid` — State transition - Happy path: pending invoice transitions to PAID with `paid_date` set - Error case: already-paid raises ValueError - Error case: missing ID raises KeyError ### Success Criteria - [ ] Test file created at `tests/test_invoice_service.py` - [ ] `python -m pytest` reports all tests passed - [ ] No real network/IO; repository is mocked with `Mock(spec=InvoiceRepository)` ``` ## Sample Generated Test File What `code-testing-implementer` produces: ```python # tests/test_invoice_service.py from datetime import datetime, timezone from decimal import Decimal from unittest.mock import Mock, patch import pytest from contoso_billing.invoice import Invoice, InvoiceStatus, LineItem from contoso_billing.invoice_repository import InvoiceRepository from contoso_billing.invoice_service import InvoiceService @pytest.fixture def repository() -> Mock: return Mock(spec=InvoiceRepository) @pytest.fixture def sut(repository: Mock) -> InvoiceService: return InvoiceService(repository) # --- calculate_total --- @pytest.mark.parametrize( "quantity, unit_price, tax_rate, expected", [ (1, "100.00", "0.10", "110.00"), (3, "25.00", "0.00", "75.00"), (2, "9.99", "0.07", "21.38"), ], ids=["single-item-10pct-tax", "multi-quantity-zero-tax", "rounds-half-up"], ) def test_calculate_total_valid_line_items_returns_expected_total( sut: InvoiceService, quantity: int, unit_price: str, tax_rate: str, expected: str ) -> None: invoice = Invoice( tax_rate=Decimal(tax_rate), line_items=[LineItem(quantity=quantity, unit_price=Decimal(unit_price))], ) total = sut.calculate_total(invoice) assert total == Decimal(expected) def test_calculate_total_none_invoice_raises_value_error(sut: InvoiceService) -> None: with pytest.raises(ValueError, match="invoice must not be None"): sut.calculate_total(None) def test_calculate_total_empty_line_items_raises_value_error(sut: InvoiceService) -> None: invoice = Invoice(tax_rate=Decimal("0"), line_items=[]) with pytest.raises(ValueError, match="no line items"): sut.calculate_total(invoice) # --- get_by_id --- def test_get_by_id_existing_id_returns_invoice( sut: InvoiceService, repository: Mock ) -> None: expected = Invoice(id=42, tax_rate=Decimal("0"), line_items=[]) repository.find.return_value = expected result = sut.get_by_id(42) assert result is expected repository.find.assert_called_once_with(42) def test_get_by_id_missing_id_raises_key_error( sut: InvoiceService, repository: Mock ) -> None: repository.find.return_value = None with pytest.raises(KeyError, match="999"): sut.get_by_id(999) # --- mark_as_paid --- def test_mark_as_paid_pending_invoice_sets_status_and_date( sut: InvoiceService, repository: Mock ) -> None: invoice = Invoice(id=1, status=InvoiceStatus.PENDING, tax_rate=Decimal("0"), line_items=[]) repository.find.return_value = invoice fixed_now = datetime(2025, 1, 1, 12, 0, tzinfo=timezone.utc) with patch("contoso_billing.invoice_service._utcnow", return_value=fixed_now): sut.mark_as_paid(1) assert invoice.status == InvoiceStatus.PAID assert invoice.paid_date == fixed_now repository.update.assert_called_once_with(invoice) def test_mark_as_paid_already_paid_raises_value_error( sut: InvoiceService, repository: Mock ) -> None: invoice = Invoice(id=1, status=InvoiceStatus.PAID, tax_rate=Decimal("0"), line_items=[]) repository.find.return_value = invoice with pytest.raises(ValueError, match="already paid"): sut.mark_as_paid(1) repository.update.assert_not_called() def test_mark_as_paid_missing_id_raises_key_error( sut: InvoiceService, repository: Mock ) -> None: repository.find.return_value = None with pytest.raises(KeyError, match="999"): sut.mark_as_paid(999) ``` ## Sample Fix Cycle When the implementer encounters an import or attribute error, the fixer agent diagnoses and resolves it. **Test output:** ```text ModuleNotFoundError: No module named 'contoso_billing' ``` **Fixer diagnosis:** The package is not installed in editable mode, so the `src/` layout's package is not on `sys.path`. **Fix applied:** ```bash python -m pip install -e ".[test]" ``` **Rerun:** `python -m pytest tests/test_invoice_service.py -q` → SUCCESS --- **Another common cycle — patch target wrong:** **Test output:** ```text AttributeError: <module 'datetime'> does not have the attribute '_utcnow' ``` **Fixer diagnosis:** The test patched `datetime._utcnow` but the production code defines its own `_utcnow` helper inside `contoso_billing.invoice_service`. Patches must target the lookup site, not the definition site. **Fix applied:** ```python # Before (wrong) with patch("datetime._utcnow", return_value=fixed_now): # After (fixed) — patch where the name is looked up with patch("contoso_billing.invoice_service._utcnow", return_value=fixed_now): ``` **Rerun:** SUCCESS --- **Another common cycle — Mock without spec:** **Test output:** ```text AttributeError: Mock object has no attribute 'find_by_id' ``` (but the actual repository method is `find`, not `find_by_id`) **Fixer diagnosis:** `Mock()` happily creates any attribute on access, so a typo in the test went undetected until the production code called `repository.find(...)`. Using `Mock(spec=InvoiceRepository)` would have failed at setup time. **Fix applied:** ```python # Before repository = Mock() repository.find_by_id.return_value = expected # typo, silently accepted # After repository = Mock(spec=InvoiceRepository) repository.find.return_value = expected # typos now raise AttributeError ``` **Rerun:** SUCCESS ## Sample Final Report What `code-testing-generator` produces at Step 9: ```markdown ## Test Generation Report **Project**: contoso-billing **Strategy**: Direct (single source file in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 9 | | Tests passing | 9 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `tests/test_invoice_service.py` (9 tests, 3 parametrized) ### Coverage - InvoiceService.calculate_total — 3 happy path, 2 error cases - InvoiceService.get_by_id — 1 happy path, 1 error case - InvoiceService.mark_as_paid — 1 happy path, 2 error cases ### Build / Install Validation - Editable install: ✅ `python -m pip install -e ".[test]"` - Test run: ✅ `python -m pytest` — 9 passed in 0.12s ### Next Steps - Add tests for repository implementations if any exist - Consider snapshot/property-based testing (`hypothesis`) for `calculate_total` rounding behaviour ``` -
python.md 11.2 KB
# Python Extension Language-specific guidance for Python test generation. ## Rule #1: Investigate the Repo First Before writing any test or running any command, discover what the repo already does: 1. **Find ALL existing test files** — search broadly: `test_*.py`, `*_test.py`, `*.uts`, `test/*.sh`, or any other test format. Do not assume pytest. 2. **Identify the test framework and runner** — check, in order: - `pyproject.toml` `[tool.pytest.ini_options]` `testpaths` - `pytest.ini` - `setup.cfg` `[tool:pytest]` - `tox.ini` `[testenv]` `commands` - `Makefile`, `noxfile.py`, `conftest.py` locations - Project-specific runners such as Django `runtests.py`, `manage.py test`, or `DJANGO_SETTINGS_MODULE` 3. **Find the active test layout** — note the directory, working directory, and fixture scope used by existing tests. Some repos use non-standard layouts such as Ansible-style `test/units/`. 4. **Read existing tests thoroughly** — copy their exact style: file format, imports, fixtures, assertion patterns, helper utilities, setup/teardown conventions 5. **Package layout** — determine import paths from existing code, not guesswork **Use whatever framework and conventions the repo already uses.** If the repo uses a custom test framework (custom file formats, custom runners, domain-specific test utilities), adopt it fully — do not layer pytest on top. Only introduce pytest if the repo has no tests at all. **Never finish with a failing or erroring test.** Run the full new-test suite before finishing. If a test cannot be made to pass within a reasonable number of attempts, delete it. A smaller suite where every test passes is strictly better than a larger suite with any failure — a suite with one failing test can score zero. **Start simple to bank coverage.** Write high-certainty tests for pure functions, validation branches, serializers, small helpers, and deterministic error paths before attempting async views, templates, sessions, network paths, or integration-heavy code. ## Environment Detection Detect the runner from lockfiles/config and prefix all commands accordingly: | Indicator | Prefix | |-----------|--------| | `poetry.lock` / `[tool.poetry]` in `pyproject.toml` | `poetry run` | | `pdm.lock` / `[tool.pdm]` in `pyproject.toml` | `pdm run` | | `uv.lock` / `[tool.uv]` in `pyproject.toml` | `uv run` | | `Pipfile.lock` | `pipenv run` | | `hatch.toml` / `[tool.hatch]` in `pyproject.toml` | `hatch run` | | None of the above | `python -m` | `<prefix>` applies to **module execution** only. With the default `python -m` prefix, `<prefix> pytest` expands to `python -m pytest`, but a script entry point or inline probe must not be double-prefixed — `python -m python manage.py …` / `python -m python -c …` is invalid. Run script entry points (`manage.py`, `runtests.py`) and `python -c` probes with `python` directly, wrapping with the env tool when one is detected (e.g. `poetry run python manage.py test …`, `uv run python -c "…"`) instead of `python -m`. If `Makefile`, `tox.ini`, or `nox` config exists, prefer those scripts over raw commands. ## Build Commands Python has no separate build step. Validate with the type checker if one is configured: | Scope | Command | |-------|---------| | Syntax check | `<prefix> py_compile path/to/file.py` | | Type check | `<prefix> mypy path/to/file.py` or `<prefix> pyright path/to/file.py` | ## Test Commands Run new tests the same way the repo runs existing tests: same working directory, same command wrapper, same `conftest.py` scope, and same settings environment variables. Before choosing a command, inspect runner configuration with copy-pasteable probes: ```powershell Get-ChildItem -Recurse -File -Include pyproject.toml,pytest.ini,setup.cfg,tox.ini,Makefile,noxfile.py,conftest.py,runtests.py,manage.py Select-String -Path pyproject.toml,pytest.ini,setup.cfg,tox.ini -Pattern 'testpaths|\[tool.pytest|\[tool:pytest|commands|DJANGO_SETTINGS_MODULE' -ErrorAction SilentlyContinue ``` If the repo uses a **custom test framework** (custom file formats, custom runner), use its native commands — do not wrap them in pytest. Examples: | Framework | Command | |-----------|---------| | UTscapy (`.uts` files) | `<prefix> scapy.tools.UTscapy -f test/test_file.uts` | | Django runner script | `python runtests.py app_label.tests.test_module` | | Django project | `python manage.py test app_label.tests.test_module` | | Custom runner script | `make test`, `./run_tests.sh`, `tox` | | Repo-defined script | Whatever `scripts.test` in Makefile/tox/nox specifies | For **pytest** projects (the most common case), use the detected `<prefix>`: | Scope | Command | |-------|---------| | All tests | `<prefix> pytest` | | Specific file | `<prefix> pytest tests/test_module.py` | | Specific test | `<prefix> pytest tests/test_module.py::TestClass::test_method` | | Keyword filter | `<prefix> pytest -k "keyword"` | | Stop on first failure | `<prefix> pytest -x --tb=short` | - Prefer `python -m pytest` over bare `pytest` to ensure the correct interpreter - If the project uses `unittest` only (no pytest in deps), use `python -m unittest discover` - If tests must run from a subdirectory, `Set-Location` there first and keep that working directory for verification ## Frameworks Beyond Plain Pytest Mirror the existing tests' import style and invocation exactly. - **Django**: Prefer the repo's runner (`runtests.py`, `manage.py test`, or tox/make target). If the repo uses `pytest-django`, ensure `DJANGO_SETTINGS_MODULE` is set exactly as existing tests/config require. - `python manage.py test app_label.tests.test_module` - `$env:DJANGO_SETTINGS_MODULE='project.settings'; python -m pytest tests/app/test_module.py` - **unittest-style suites**: Use `python -m unittest path.to.test_module` or the repo's discover command; do not force pytest unless existing tests already do. - **Subdir runners**: Some repos expect commands from `tests/`, `test/units/`, or another subdir so relative imports and fixtures work. ## Lint Command Use the repo's existing lint script first (`make lint`, `tox -e lint`). Otherwise detect tools from config: - `ruff.toml` or `[tool.ruff]` → `<prefix> ruff check --fix && <prefix> ruff format` - `[tool.black]` → `<prefix> black` - `.flake8` → `<prefix> flake8` ## Project Layout and Imports | Layout | Import Style | |--------|-------------| | `src/package/module.py` | `from package.module import X` | | `package/module.py` at root | `from package.module import X` | | `module.py` at root | `from module import X` | - **Match existing test imports exactly** — do not invent `src.` prefixes unless existing tests use them - Place new tests where the existing suite lives so the same `conftest.py`, fixtures, helpers, and settings apply - Check `pyproject.toml` `[tool.setuptools.package-dir]` for layout hints - Default test placement: `tests/` mirroring source structure (`src/billing/service.py` → `tests/billing/test_service.py`) ## Heavy or Native Dependencies Before writing tests for a target module, verify it imports cleanly in the same environment and working directory as tests. Run the probe under the **same env wrapper as the test command** (`poetry run`, `pdm run`, `uv run`, `pipenv run`, `hatch run`) so the check reflects the real test interpreter/venv — a bare `python` may resolve to a different environment and report a misleading `ok`: ```powershell # Wrap with the detected env tool, e.g. `poetry run python -c "..."` python -c "import package.module; print('ok')" python -c "from package import module; print('ok')" ``` If a heavy/native dependency such as NumPy, pandas, PyTorch, TensorFlow, cryptography, or a compiled extension cannot be imported or built in the environment: - Do not write tests that import the failing module - Do not spend the budget fighting native build/import failures or installing unrelated packages - Scope down to a pure-Python submodule that imports cleanly, or omit tests for that module rather than shipping ones that cannot run (see *Finalization: Green Suite or Remove*) ## Test File Naming Match the repo's existing conventions. Common patterns: - **pytest**: Files `test_*.py` or `*_test.py`, functions `test_` prefix, classes `Test` prefix - **Custom frameworks**: Use whatever format existing tests use (e.g. `.uts` for UTscapy, custom extensions) If writing new tests in a repo with no tests, default to pytest conventions. ## Common Errors | Error | Fix | |-------|-----| | `ModuleNotFoundError: No module named 'src'` | Import from the package name used by the repo, not from `src` | | `ModuleNotFoundError: No module named 'X'` | Check existing imports for the correct package name; if editable install needed: `<prefix> pip install -e .` | | `ImportError: attempted relative import` | Convert to absolute imports matching existing test patterns | | `fixture 'X' not found` | Check `conftest.py` for existing fixtures; reuse them instead of creating new ones | | `TypeError: missing required argument` | Read the full `__init__`/function signature; pass all required parameters | | `async def functions are not natively supported` | Use `@pytest.mark.asyncio` only if `pytest-asyncio` is already in deps; check for `asyncio_mode = "auto"` in config | | `DJANGO_SETTINGS_MODULE is undefined` | Use the repo's Django runner or set the same settings module used by existing tests | | `ImportError` from `torch`, `numpy`, or compiled extension | Avoid that module; choose a pure-Python target that imports cleanly | | `SyntaxError` | Fix syntax at the indicated line | ## Mocking Rules - Use `unittest.mock` (stdlib) — no extra dependency needed - **Patch where the name is looked up**, not where it is defined: `@patch("mypackage.module.datetime")` not `@patch("datetime.datetime")` - Use `Mock(spec=RealClass)` to catch attribute errors - Use `AsyncMock` for async functions - Prefer dependency injection over `@patch` - If a test needs more than 3 mocks, flag it as a design smell ## Dependency Installation (Last Resort) Only install packages after investigation confirms they are missing. Use the detected prefix: | Manager | Install command | |---------|----------------| | Poetry | `poetry add --group dev pytest` | | PDM | `pdm add -dG test pytest` | | uv | `uv add --dev pytest` | | pip | `python -m pip install -e ".[dev]"` | Never run bare `pip install` in a Poetry/PDM/uv project — it bypasses the lockfile. ## Finalization: Green Suite or Remove Before finishing, run the complete set of tests you added with the repo's native invocation, under the **same env wrapper as the repo's tests** (`poetry run`, `pdm run`, `uv run`, `pipenv run`, `hatch run`). Running the green-suite check in a different interpreter/venv can pass locally yet still fail under the repo's actual runner. ```powershell # Examples; choose the repo-native command/wrapper discovered above poetry run python -m pytest tests/path/to/new_tests.py uv run python -m unittest path.to.new_test_module poetry run python manage.py test app_label.tests.test_module ``` If any new test fails or errors after a reasonable fix attempt, delete that test before finishing. Never leave skipped, xfailed, failing, or collection-error tests just to keep more lines. The final submitted suite must be green. ## Skip Coverage Tools Do not configure or run coverage tools (coverage.py, pytest-cov). Coverage is measured separately by the evaluation harness. -
ruby-examples.md 8.5 KB
# Ruby Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a Ruby codebase using RSpec. These show what each pipeline phase produces for a small gem-style project. > Minitest follows the same coverage shape. Replace `RSpec.describe` / `it` / `expect` with `Minitest::Test` methods and assertions, and run through `bundle exec rake test` or the repo's established Minitest command. ## Source Under Test A simple `InvoiceService` in a Ruby gem: ```text Gemfile lib/ contoso_billing.rb contoso_billing/invoice.rb contoso_billing/invoice_repository.rb contoso_billing/invoice_service.rb spec/spec_helper.rb ``` ```ruby # lib/contoso_billing/invoice_service.rb require 'bigdecimal' require 'time' module ContosoBilling class InvoiceService def initialize(repository:, clock: -> { Time.now.utc }) @repository = repository @clock = clock end def calculate_total(invoice) raise ArgumentError, 'invoice must not be nil' if invoice.nil? raise ArgumentError, 'Invoice has no line items.' if invoice.line_items.empty? subtotal = invoice.line_items.sum { |item| item.quantity * item.unit_price } tax = subtotal * invoice.tax_rate (subtotal + tax).round(2) end def get_by_id(id) invoice = @repository.find(id) raise KeyError, "Invoice #{id} not found." if invoice.nil? invoice end def mark_as_paid(id) invoice = get_by_id(id) raise StandardError, 'Invoice is already paid.' if invoice.status == :paid invoice.status = :paid invoice.paid_at = @clock.call @repository.update(invoice) end end end ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: /work/contoso-billing - **Language**: Ruby 3.3 (from `.ruby-version`) - **Project Type**: Plain gem - **Test Framework**: RSpec 3.x (detected in Gemfile.lock and `spec/spec_helper.rb`) - **Run Prefix**: `bundle exec` required because Gemfile.lock is present ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 examples across 0 files ## Build & Test Commands - **Syntax**: `ruby -c lib/contoso_billing/invoice_service.rb` - **Discovery**: `bundle exec rspec --dry-run` - **Single file**: `bundle exec rspec spec/contoso_billing/invoice_service_spec.rb` - **All specs**: `bundle exec rspec` ## Files to Test ### High Priority | File | Classes/Methods | Testability | Notes | |------|-----------------|-------------|-------| | lib/contoso_billing/invoice_service.rb | InvoiceService: calculate_total, get_by_id, mark_as_paid | High | Repository dependency and clock are injected | ## Testing Patterns - Existing specs use `RSpec.describe`, `subject`, `let`, and `instance_double`. - Recommend `instance_double('InvoiceRepository')` for the repository and a fixed clock lambda for time-dependent behavior. ``` ## Sample Plan Output ```markdown # Test Implementation Plan ## Overview Generate RSpec tests for ContosoBilling::InvoiceService, covering calculation, lookup, and paid-state transition behavior. ## Commands - **Syntax**: `ruby -c spec/contoso_billing/invoice_service_spec.rb` - **Discovery**: `bundle exec rspec --dry-run` - **Test**: `bundle exec rspec spec/contoso_billing/invoice_service_spec.rb` ## Phase 1: InvoiceService ### Files to Test - **Source**: `lib/contoso_billing/invoice_service.rb` - **Test File**: `spec/contoso_billing/invoice_service_spec.rb` **Methods to Test**: 1. `calculate_total` — tax, zero tax, rounding, nil, and empty line items 2. `get_by_id` — existing invoice and missing invoice 3. `mark_as_paid` — success with fixed clock, already-paid, missing ``` ## Sample Generated Test File ```ruby # spec/contoso_billing/invoice_service_spec.rb require 'spec_helper' require 'bigdecimal' require 'ostruct' require 'time' require 'contoso_billing/invoice_service' RSpec.describe ContosoBilling::InvoiceService do subject(:service) { described_class.new(repository: repository, clock: clock) } let(:repository) { instance_double('InvoiceRepository') } let(:fixed_time) { Time.utc(2025, 1, 1, 12, 0, 0) } let(:clock) { -> { fixed_time } } def build_invoice(id: 1, status: :pending, tax_rate: BigDecimal('0.10'), line_items: [OpenStruct.new(quantity: 1, unit_price: BigDecimal('100.00'))]) OpenStruct.new(id: id, status: status, tax_rate: tax_rate, line_items: line_items, paid_at: nil) end describe '#calculate_total' do it 'returns the total for a single item with tax' do expect(service.calculate_total(build_invoice)).to eq(BigDecimal('110.00')) end it 'returns the subtotal when tax is zero' do invoice = build_invoice( tax_rate: BigDecimal('0'), line_items: [OpenStruct.new(quantity: 3, unit_price: BigDecimal('25.00'))] ) expect(service.calculate_total(invoice)).to eq(BigDecimal('75.00')) end it 'rounds to two decimals' do invoice = build_invoice( tax_rate: BigDecimal('0.07'), line_items: [OpenStruct.new(quantity: 2, unit_price: BigDecimal('9.99'))] ) expect(service.calculate_total(invoice)).to eq(BigDecimal('21.38')) end it 'raises for a nil invoice' do expect { service.calculate_total(nil) }.to raise_error(ArgumentError, /must not be nil/) end it 'raises when there are no line items' do expect { service.calculate_total(build_invoice(line_items: [])) }.to raise_error(ArgumentError, /no line items/) end end describe '#get_by_id' do it 'returns an existing invoice' do invoice = build_invoice(id: 42) allow(repository).to receive(:find).with(42).and_return(invoice) expect(service.get_by_id(42)).to be(invoice) end it 'raises KeyError for a missing invoice' do allow(repository).to receive(:find).with(999).and_return(nil) expect { service.get_by_id(999) }.to raise_error(KeyError, /999/) end end describe '#mark_as_paid' do it 'marks a pending invoice as paid and persists it' do invoice = build_invoice(id: 1) allow(repository).to receive(:find).with(1).and_return(invoice) allow(repository).to receive(:update) service.mark_as_paid(1) expect(invoice.status).to eq(:paid) expect(invoice.paid_at).to eq(fixed_time) expect(repository).to have_received(:update).with(invoice) end it 'raises and does not update an already-paid invoice' do invoice = build_invoice(id: 1, status: :paid) allow(repository).to receive(:find).with(1).and_return(invoice) allow(repository).to receive(:update) expect { service.mark_as_paid(1) }.to raise_error(StandardError, /already paid/) expect(repository).not_to have_received(:update) end end end ``` ## Sample Fix Cycle When the implementer encounters a load or mock issue, the fixer agent diagnoses and resolves it. **Test output:** ```text LoadError: cannot load such file -- contoso_billing/invoice_service ``` **Fixer diagnosis:** The generated spec omitted `require 'spec_helper'`, so the gem's load-path setup did not run. **Fix applied:** Add `require 'spec_helper'` as the first require and keep source requires consistent with existing specs. **Rerun:** `bundle exec rspec spec/contoso_billing/invoice_service_spec.rb` → SUCCESS --- **Another common cycle — verifying double mismatch:** **Test output:** ```text The InvoiceRepository class does not implement the instance method: find_by_id ``` **Fixer diagnosis:** `instance_double` caught a typo in the test setup. The production code calls `repository.find`, not `find_by_id`. **Fix applied:** Stub `find` with the expected id instead of `find_by_id`. **Rerun:** SUCCESS ## Sample Final Report ```markdown ## Test Generation Report **Project**: contoso-billing (Ruby) **Strategy**: Direct (single source file in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 9 | | Tests passing | 9 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `spec/contoso_billing/invoice_service_spec.rb` (9 RSpec examples) ### Coverage - InvoiceService#calculate_total — 3 happy path, 2 error cases - InvoiceService#get_by_id — found and missing branches - InvoiceService#mark_as_paid — success and already-paid branches ### Build / Test Validation - Syntax: ✅ `ruby -c spec/contoso_billing/invoice_service_spec.rb` - Discovery: ✅ `bundle exec rspec --dry-run` found the new examples - Test run: ✅ `bundle exec rspec spec/contoso_billing/invoice_service_spec.rb` ``` -
ruby.md 13.9 KB
# Ruby Extension Language-specific guidance for Ruby test generation. ## Rule #0: Confirm the Test Target If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of modules already covered by existing specs. Run these **read-only** discovery commands first — they are the deliberate exception to Rule #1's "before writing any test or running any command" rule, and their output is the ground truth Rule #1's reading is meant to interpret. Do **not** write or execute any tests until Rule #0 and Rule #1 are both complete. | Goal | Command | |------|---------| | List uncommitted edits + untracked files | `git status -s` | | Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | | Recently added files under `lib/` or `app/` | `git log --diff-filter=A --name-only -5 -- 'lib/**' 'app/**'` | | Files referenced by `spec_helper.rb` / `rails_helper.rb` | `grep -nE "^\s*require(_relative)?\s" spec/spec_helper.rb spec/rails_helper.rb 2>/dev/null` | | Modules with no matching spec | compare `lib/**/*.rb` against `spec/**/*_spec.rb` paths | Prefer targets that match **all** of: 1. Untracked or recently added (`git status` / `git log --diff-filter=A`). 2. Small and pure (a few hundred lines, no I/O, no global state). 3. Located under a conventional source root (`lib/`, `app/models/`, `app/services/`). 4. Have **no** existing matching `*_spec.rb` / `*_test.rb`. If `spec/spec_helper.rb` already `require`s one specific file (e.g. `require "string_utils"`), that file is almost certainly the target — start there. ### Test Placement Contract RSpec only discovers specs under `spec/` by default, and verification harnesses (CI, msbench, coverage tools) typically scope discovery to `spec/` alone. Place every spec there, mirroring the source layout: | Source | Spec | |--------|------| | `lib/string_utils.rb` | `spec/string_utils_spec.rb` | | `lib/foo/bar.rb` | `spec/foo/bar_spec.rb` | | `app/models/user.rb` (Rails) | `spec/models/user_spec.rb` | A spec placed anywhere outside `spec/` (e.g. next to the source under `lib/`) will be invisible to `bundle exec rspec` and to the harness. The same applies to Minitest: place tests under `test/` and use `*_test.rb` naming. **Gem-monorepo trap (fastlane, ruby/ruby, large gems with sub-gems):** if the repo contains multiple `*/spec/` directories (each sub-gem with its own specs), `bundle exec rspec` from the repo root only loads `./spec/` by default — sub-gem specs are invisible to the harness. Either: - place the new spec inside the **root** `./spec/` (with a `require_relative` to the sub-gem's `lib/`), or - run the sub-gem's `bundle exec rspec` from the sub-gem dir AND verify in the Harness Discovery Check below that the root command also enumerates it (often it won't — you'll need to extend `.rspec` with `--default-path` or the root `Rakefile`'s test task). For interpreter-build repos (ruby/ruby itself) the test runner requires `make test-all` after `make miniruby` — `ruby test/foo_test.rb` alone is not what the harness runs. ### First-Test Sanity Loop After writing the **first** spec — before writing any others: 1. Run `bundle exec rspec --dry-run` and confirm the example count is `> 0`. If it is `0`, RSpec is not seeing your file; fix the location, filename, or `$LOAD_PATH` before continuing. 2. Run the spec (`bundle exec rspec spec/<your_spec>.rb`); fix `LoadError`, missing `require`, or constant errors before adding more tests. 3. Only then expand to cover the remaining methods. This catches placement and load-path mistakes on turn 1 instead of after dozens of failed-test iterations. ### Harness Discovery Check Before reporting success, run the **harness-equivalent** discovery command from the repo root and confirm the example count went up by at least the number of tests you generated. CI/msbench/coverage harnesses do not know which file or sub-gem dir you targeted; they run the framework's default discovery from the repo root, so a spec that passes via `bundle exec rspec fastlane_core/spec/foo_spec.rb` is still worthless if `bundle exec rspec --dry-run` from the repo root doesn't enumerate it. ```bash # RSpec — from repo root bundle exec rspec --dry-run 2>&1 | grep -E '^[0-9]+ example' # Minitest (Rails) { bundle exec rake test --dry-run 2>/dev/null || bin/rails test --list-tests; } | wc -l # Custom runner (Homebrew, ruby/ruby, etc.) # Use the repo's own runner — `./bin/brew tests --list`, `make test-all`, etc. # If no `--list`/`--dry-run` mode exists, run a single matching test by name and confirm exit 0. ``` If the count did not increase, your spec is invisible to the harness. Move it into `./spec/`, extend `.rspec`/`Rakefile` so the harness picks up the sub-gem dir, or switch to a `require_relative` strategy from a root-level spec. Do **not** report success until the harness-equivalent command sees your new tests. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — find `spec/**/*_spec.rb` (RSpec) or `test/**/*_test.rb` (Minitest) and copy their style (matchers, helpers, factories, contexts) 2. **`Gemfile` / `Gemfile.lock`** — Ruby version, test framework, supporting gems (`rspec`, `minitest`, `factory_bot`, `webmock`, `vcr`, `rails`) 3. **`.ruby-version`** / `.tool-versions` — pinned Ruby version 4. **Test helpers** — `spec/spec_helper.rb`, `spec/rails_helper.rb`, `test/test_helper.rb` — these dictate the load path, requires, and global config 5. **Rake tasks** — `Rakefile` may define a `default` task that runs the full test suite Use the framework the repo already uses. Do not introduce RSpec into a Minitest project (or vice versa). ## Toolchain Detection | Indicator | Manager | Run prefix | |-----------|---------|------------| | `Gemfile.lock` | Bundler | `bundle exec <cmd>` | | `.ruby-version` + `rbenv` | rbenv | combine with `bundle exec` | | `mise.toml` / `asdf` `.tool-versions` | mise/asdf | the wrapper handles version selection; still use `bundle exec` | | Plain Ruby, no Bundler | system Ruby | `ruby <file>` (rare in real projects) | Always run inside `bundle exec` if a `Gemfile.lock` is present — otherwise you may pick up a system gem version that disagrees with the lockfile. ## Build Commands Ruby is interpreted — there is no compile step. The closest validations: | Scope | Command | |-------|---------| | Syntax check | `ruby -c path/to/file.rb` | | Lint (RuboCop) | `bundle exec rubocop path/to/file.rb` | | Type check (Sorbet) | `bundle exec srb tc` (only if `sorbet/` dir exists) | | Type check (RBS/Steep) | `bundle exec steep check` | For Rails: load all classes once with `bundle exec rails zeitwerk:check` to catch missing constants before running tests. ## Test Commands ### RSpec | Scope | Command | |-------|---------| | All specs | `bundle exec rspec` | | Single file | `bundle exec rspec spec/models/widget_spec.rb` | | Single line | `bundle exec rspec spec/models/widget_spec.rb:42` | | By name | `bundle exec rspec -e "creates a widget"` | | Tagged | `bundle exec rspec --tag focus` | | Fail fast | `bundle exec rspec --fail-fast` | | Documentation format | `bundle exec rspec --format documentation` | ### Minitest | Scope | Command | |-------|---------| | All tests | `bundle exec rake test` (Rails) or `bundle exec ruby -Ilib -Itest -e 'Dir.glob("./test/**/*_test.rb").each { |f| require f }'` | | Single file | `bundle exec ruby -Itest test/models/widget_test.rb` | | Single test | `bundle exec ruby -Itest test/models/widget_test.rb -n test_creates_widget` | | By name pattern | `... -n /pattern/` | ### Rails (any framework) | Scope | Command | |-------|---------| | Default suite | `bin/rails test` (Minitest) or `bundle exec rspec` | | Single Rails test file | `bin/rails test test/models/widget_test.rb:42` | | System tests | `bin/rails test:system` | Always prefer the wrapper script (`bin/rails`, `bin/rspec`) when present — they enforce the project's loader/setup. ## Lint Command - `bundle exec rubocop` — autocorrect with `bundle exec rubocop -A` (only if existing tests already conform; do not autocorrect unrelated files) - `bundle exec standardrb --fix` if `standard` is in the Gemfile - Some Rails projects add `rubocop-rails`, `rubocop-rspec`, `rubocop-performance` — they enforce extra rules ## Project Layout and Loading | Layout | Test placement | |--------|----------------| | Plain gem (RSpec) | `spec/` mirrors `lib/` (e.g. `lib/foo/bar.rb` → `spec/foo/bar_spec.rb`) | | Plain gem (Minitest) | `test/` mirrors `lib/` (e.g. `test/foo/bar_test.rb`) | | Rails (RSpec) | `spec/models`, `spec/controllers`, `spec/requests`, `spec/system`, etc. | | Rails (Minitest) | `test/models`, `test/controllers`, `test/integration`, `test/system` | **Loading source code:** - RSpec: `spec/spec_helper.rb` typically does `require 'my_gem'` or sets `$LOAD_PATH`. Match its pattern in new specs by `require 'spec_helper'` (or `require 'rails_helper'` in Rails) - Minitest: each `_test.rb` typically `require 'test_helper'` - Rails uses Zeitwerk autoloading — do **not** add `require_relative '../../app/models/widget'`; just `require 'rails_helper'` and reference the constant ## Test File Naming | Framework | File suffix | Class/example | |-----------|-------------|---------------| | RSpec | `_spec.rb` | `RSpec.describe Widget do ... end`, `it "..." do ... end` | | Minitest (classic) | `_test.rb` | `class WidgetTest < Minitest::Test`, methods `def test_...` | | Minitest (spec) | `_test.rb` | `describe Widget do ... it "..." do ... end end` | | Rails Minitest | `_test.rb` | `class WidgetTest < ActiveSupport::TestCase` | ## RSpec Template ```ruby require 'spec_helper' require 'calculator' RSpec.describe Calculator do subject(:calculator) { described_class.new } describe '#add' do it 'returns the sum of two positive numbers' do expect(calculator.add(2, 3)).to eq(5) end context 'with negative numbers' do it 'returns the correct sum' do expect(calculator.add(-1, 1)).to eq(0) end end it 'raises when given non-numeric input' do expect { calculator.add('a', 1) }.to raise_error(TypeError) end end end ``` ## Common Errors | Error | Fix | |-------|-----| | `LoadError: cannot load such file -- foo` | Missing `require` or load path; check `spec_helper.rb` for the established pattern instead of patching `$LOAD_PATH` ad hoc | | `NameError: uninitialized constant X` | Constant isn't loaded — in Rails, ensure you require `rails_helper`; in plain Ruby, add the appropriate `require` | | `ArgumentError: wrong number of arguments (given X, expected Y)` | Read the method signature; pass keyword vs positional args correctly | | `NoMethodError: undefined method 'foo' for nil:NilClass` | Test setup left a value `nil`; check `let`/`before` ordering and factory data | | `Failure/Error: ... received :foo with unexpected arguments` (RSpec) | Tighten the matcher: `with(hash_including(...))` or relax to `with(any_args)` deliberately | | `expected #<...> to receive :foo (1 time) but received it 0 times` | Either the code path didn't call the stub, or you stubbed the wrong receiver | | `DEPRECATION WARNING` (Rails) | Address the deprecation rather than silencing it; tests that warn today break tomorrow | | `ActiveRecord::PendingMigrationError` | Run `bin/rails db:migrate RAILS_ENV=test` before tests | | `Mysql2::Error / PG::ConnectionBad` in CI | Tests need a database — check `config/database.yml` and CI service containers | | `Capybara::ElementNotFound` (system tests) | Use `find` with explicit waits; do not add `sleep` | ## Mocking Rules (RSpec) - Use `instance_double(Klass)` and `class_double(Klass)` — they verify that the method actually exists, unlike `double` - `allow(obj).to receive(:method).and_return(value)` for stubs; `expect(obj).to receive(:method)` for interaction expectations - Prefer `instance_double` over plain `double`; prefer dependency injection over `allow_any_instance_of` - Use `let` for memoized helpers; use `let!` only when the side effect must run before each example - Avoid global state mutation in tests — wrap in `around` blocks or use `ClimateControl` for env vars - For HTTP, use `webmock` (`stub_request(:get, ...)`) or `vcr` cassettes if the project already uses them - If a test needs more than 3 mocks, flag it as a design smell ## Mocking Rules (Minitest) - Use `Minitest::Mock` for simple cases: `mock = Minitest::Mock.new; mock.expect(:method, return_value, [arg])` - For richer mocking, projects commonly add `mocha`: `obj.expects(:method).returns(value)` (in `test_helper.rb`: `require 'mocha/minitest'`) - Always verify mocks at end of test (`mock.verify` for `Minitest::Mock`); Mocha verifies automatically ## Rails Specifics - Use the **smallest** spec type that covers the behavior: model spec for pure logic, request spec for HTTP, system spec only when JS/UI matters - `rails-controller-testing` gem must be present for `assigns(:foo)` and `assert_template` - `ActiveJob::TestHelper` and `ActiveSupport::Testing::TimeHelpers` (`travel_to`) come with Rails — use them instead of `Timecop` if Rails ≥ 5 - Use fixtures only if the project already uses them; `factory_bot` is more common in modern Rails apps - Database transactions wrap each test by default — for system tests with browser drivers, use `DatabaseCleaner` strategies the project already configures ## Dependency Installation (Last Resort) Only add gems after investigation confirms they are missing. Edit `Gemfile`: ```ruby group :test do gem 'rspec' gem 'webmock' end ``` Then run: ``` bundle install ``` Never `gem install` outside Bundler — it bypasses the lockfile and changes the global Ruby environment. ## Skip Coverage Tools Do not configure or run coverage tools (SimpleCov). Coverage is measured separately by the evaluation harness. -
rust-examples.md 10.2 KB
# Rust Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a Rust crate using the built-in test harness. These show what each pipeline phase produces for a small library crate. ## Source Under Test A simple `InvoiceService` in a Rust crate: ```text Cargo.toml src/ lib.rs invoice.rs invoice_repository.rs invoice_service.rs ``` ```rust // src/invoice_service.rs use crate::invoice::{Invoice, InvoiceStatus}; use crate::invoice_repository::InvoiceRepository; use std::time::SystemTime; #[derive(Debug, PartialEq, Eq)] pub enum InvoiceError { EmptyLineItems, NotFound(i32), AlreadyPaid, Repository(String), } pub struct InvoiceService<R, C> where R: InvoiceRepository, C: Fn() -> SystemTime, { repository: R, clock: C, } impl<R, C> InvoiceService<R, C> where R: InvoiceRepository, C: Fn() -> SystemTime, { pub fn new(repository: R, clock: C) -> Self { Self { repository, clock } } pub fn calculate_total(&self, invoice: &Invoice) -> Result<f64, InvoiceError> { if invoice.line_items.is_empty() { return Err(InvoiceError::EmptyLineItems); } let subtotal: f64 = invoice.line_items.iter().map(|item| item.quantity as f64 * item.unit_price).sum(); Ok(((subtotal + subtotal * invoice.tax_rate) * 100.0).round() / 100.0) } pub fn get_by_id(&self, id: i32) -> Result<Invoice, InvoiceError> { self.repository.find(id).map_err(InvoiceError::Repository)?.ok_or(InvoiceError::NotFound(id)) } pub fn mark_as_paid(&mut self, id: i32) -> Result<(), InvoiceError> { let mut invoice = self.get_by_id(id)?; if invoice.status == InvoiceStatus::Paid { return Err(InvoiceError::AlreadyPaid); } invoice.status = InvoiceStatus::Paid; invoice.paid_at = Some((self.clock)()); self.repository.update(invoice).map_err(InvoiceError::Repository) } } ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: /work/contoso-billing - **Language**: Rust 1.78 (edition 2021, from Cargo.toml) - **Crate Type**: library - **Test Framework**: built-in Rust test harness (`#[test]`), no `mockall`/`rstest` dev-dependencies detected ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Check**: `cargo check --all-targets` - **Compile tests**: `cargo test --no-run` - **Test**: `cargo test` - **Single module**: `cargo test invoice_service::tests` ## Files to Test ### High Priority | File | Types/Methods | Testability | Notes | |------|---------------|-------------|-------| | src/invoice_service.rs | InvoiceService: calculate_total, get_by_id, mark_as_paid | High | Generic repository trait is easy to fake; clock closure is injectable | ## Testing Patterns - No existing patterns; recommend unit tests in `#[cfg(test)] mod tests` at the bottom of `invoice_service.rs` and a hand-written fake repository. ``` ## Sample Plan Output ```markdown # Test Implementation Plan ## Overview Generate built-in Rust unit tests for InvoiceService covering calculation, lookup, and paid-state transition behavior. ## Commands - **Check**: `cargo check --all-targets` - **Compile tests**: `cargo test --no-run` - **Test**: `cargo test invoice_service::tests` ## Phase 1: InvoiceService ### Files to Test - **Source**: `src/invoice_service.rs` - **Test Location**: `#[cfg(test)] mod tests` appended to `src/invoice_service.rs` **Methods to Test**: 1. `calculate_total` — tax, zero tax, rounding, empty-line-items error 2. `get_by_id` — existing invoice, missing invoice, repository error 3. `mark_as_paid` — success, already-paid, missing invoice ``` ## Sample Generated Test File ```rust // Appended to src/invoice_service.rs #[cfg(test)] mod tests { use super::*; use crate::invoice::{Invoice, InvoiceStatus, LineItem}; use std::collections::HashMap; use std::time::{Duration, UNIX_EPOCH}; #[derive(Default)] struct FakeRepository { invoices: HashMap<i32, Invoice>, updated: Option<Invoice>, find_error: Option<String>, } impl InvoiceRepository for FakeRepository { fn find(&self, id: i32) -> Result<Option<Invoice>, String> { if let Some(error) = &self.find_error { return Err(error.clone()); } Ok(self.invoices.get(&id).cloned()) } fn update(&mut self, invoice: Invoice) -> Result<(), String> { self.updated = Some(invoice.clone()); self.invoices.insert(invoice.id, invoice); Ok(()) } } fn make_invoice(id: i32) -> Invoice { Invoice { id, status: InvoiceStatus::Pending, tax_rate: 0.10, line_items: vec![LineItem { quantity: 1, unit_price: 100.0 }], paid_at: None, } } fn service_with(repository: FakeRepository) -> InvoiceService<FakeRepository, fn() -> std::time::SystemTime> { InvoiceService::new(repository, || UNIX_EPOCH + Duration::from_secs(123)) } #[test] fn calculate_total_valid_line_items_returns_expected_total() { let cases = [ ("single item with tax", vec![LineItem { quantity: 1, unit_price: 100.0 }], 0.10, 110.0), ("multi quantity zero tax", vec![LineItem { quantity: 3, unit_price: 25.0 }], 0.0, 75.0), ("rounds to two decimals", vec![LineItem { quantity: 2, unit_price: 9.99 }], 0.07, 21.38), ]; let service = service_with(FakeRepository::default()); for (name, line_items, tax_rate, expected) in cases { let mut invoice = make_invoice(1); invoice.line_items = line_items; invoice.tax_rate = tax_rate; let total = service.calculate_total(&invoice).unwrap_or_else(|err| panic!("{name}: unexpected error: {err:?}")); assert!((total - expected).abs() < 0.001, "{name}: got {total}, expected {expected}"); } } #[test] fn calculate_total_empty_line_items_returns_error() { let service = service_with(FakeRepository::default()); let mut invoice = make_invoice(1); invoice.line_items.clear(); assert_eq!(Err(InvoiceError::EmptyLineItems), service.calculate_total(&invoice)); } #[test] fn get_by_id_existing_invoice_returns_invoice() { let mut repository = FakeRepository::default(); repository.invoices.insert(42, make_invoice(42)); let service = service_with(repository); let invoice = service.get_by_id(42).expect("invoice should exist"); assert_eq!(42, invoice.id); } #[test] fn get_by_id_missing_invoice_returns_not_found() { let service = service_with(FakeRepository::default()); assert_eq!(Err(InvoiceError::NotFound(999)), service.get_by_id(999)); } #[test] fn get_by_id_repository_error_is_preserved() { let repository = FakeRepository { find_error: Some("boom".to_owned()), ..FakeRepository::default() }; let service = service_with(repository); assert_eq!(Err(InvoiceError::Repository("boom".to_owned())), service.get_by_id(1)); } #[test] fn mark_as_paid_pending_invoice_updates_status_date_and_repository() { let mut repository = FakeRepository::default(); repository.invoices.insert(1, make_invoice(1)); let mut service = service_with(repository); service.mark_as_paid(1).expect("mark_as_paid should succeed"); let updated = service.repository.updated.as_ref().expect("repository should be updated"); assert_eq!(InvoiceStatus::Paid, updated.status); assert_eq!(Some(UNIX_EPOCH + Duration::from_secs(123)), updated.paid_at); } #[test] fn mark_as_paid_already_paid_returns_error_without_update() { let mut invoice = make_invoice(1); invoice.status = InvoiceStatus::Paid; let mut repository = FakeRepository::default(); repository.invoices.insert(1, invoice); let mut service = service_with(repository); assert_eq!(Err(InvoiceError::AlreadyPaid), service.mark_as_paid(1)); assert!(service.repository.updated.is_none()); } } ``` ## Sample Fix Cycle When the implementer hits a compiler or test-runner issue, the fixer agent diagnoses and resolves it. **Build output:** ```text error[E0596]: cannot borrow `self.repository` as mutable, as it is behind a `&` reference ``` **Fixer diagnosis:** `mark_as_paid` calls `repository.update(...)`, which requires mutable repository access. The production method must take `&mut self`, and tests must bind the service as `let mut service`. **Fix applied:** Change the method receiver to `&mut self` and update tests to use mutable bindings for `mark_as_paid` cases. **Rebuild + rerun:** `cargo test --no-run && cargo test invoice_service::tests` → SUCCESS --- **Another common cycle — integration test imports:** **Build output:** ```text error[E0432]: unresolved import `crate::invoice_service` ``` **Fixer diagnosis:** The test was created under `tests/invoice_service.rs`, which is an integration test crate. Integration tests import the library by crate name, not `crate::`. **Fix applied:** Move the tests into `#[cfg(test)] mod tests` in `src/invoice_service.rs` and use `use super::*;`. **Rerun:** SUCCESS ## Sample Final Report ```markdown ## Test Generation Report **Project**: contoso-billing (Rust) **Strategy**: Direct (single module in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 7 | | Tests passing | 7 | | Tests failing | 0 | | Files created | 0 (tests appended to source module) | ### Files Modified - `src/invoice_service.rs` (7 unit tests in `#[cfg(test)] mod tests`) ### Coverage - InvoiceService::calculate_total — 3 happy path, 1 error case - InvoiceService::get_by_id — found, missing, repository error - InvoiceService::mark_as_paid — success and already-paid branches ### Build / Test Validation - Check: ✅ `cargo check --all-targets` - Compile tests: ✅ `cargo test --no-run` - Test run: ✅ `cargo test invoice_service::tests` ``` -
rust.md 8.8 KB
# Rust Extension Language-specific guidance for Rust test generation. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — look at `#[cfg(test)] mod tests` blocks inside `src/`, integration tests in `tests/`, doc tests in source comments, and any `examples/` that double as smoke tests 2. **`Cargo.toml`** — workspace layout (`[workspace]`), edition, `dev-dependencies`, feature flags, `[[bench]]` / `[[test]]` declarations 3. **`Cargo.lock`** — if checked in, you must not break it without intent 4. **Toolchain** — `rust-toolchain.toml` pins the channel (stable / nightly / specific version) 5. **`build.rs`** — custom build scripts may set `cfg` flags or generate code that tests rely on Match the repo's existing conventions — assertion macros, mock approach, feature-gating — exactly. Do not introduce `tokio::test` if the repo uses `async-std`, etc. ## Toolchain Detection | Indicator | Meaning | |-----------|---------| | `rust-toolchain.toml` with `channel = "..."` | Use rustup to install/select that channel — `rustup show active-toolchain` | | `rust-version = "1.x"` in `Cargo.toml` | Minimum supported Rust version (MSRV); do not use newer language features | | `[workspace]` in root `Cargo.toml` | Multi-crate workspace; commands accept `-p <crate>` to target one member | | `nightly` channel | Tests may use `#![feature(...)]` flags; do not remove them | ## Build Commands | Scope | Command | |-------|---------| | Type-check fast | `cargo check` | | Type-check whole workspace | `cargo check --workspace --all-targets` | | Build (debug) | `cargo build` | | Build with all features | `cargo build --all-features` | | Build a single crate | `cargo build -p crate-name` | | Build tests without running | `cargo test --no-run` | `cargo check` is far faster than `cargo build` and catches almost the same errors. Prefer it during the fix loop; use `cargo build --tests` (or `cargo test --no-run`) before declaring tests compilable. ## Test Commands | Scope | Command | |-------|---------| | All tests | `cargo test` | | Workspace | `cargo test --workspace` | | Single crate | `cargo test -p crate-name` | | Filter by name | `cargo test substring_of_test_name` | | Exact name | `cargo test -- --exact path::to::test_fn` | | Single integration file | `cargo test --test file_stem` (no `.rs`) | | Doc tests only | `cargo test --doc` | | Show stdout | `cargo test -- --nocapture` | | Single-threaded | `cargo test -- --test-threads=1` | | Ignored tests | `cargo test -- --ignored` | | With features | `cargo test --features "feat1 feat2"` | | All features | `cargo test --all-features` | - Arguments before `--` are for cargo; arguments after `--` go to the test binary - `cargo test foo` runs every test with `foo` in its full path (`module::tests::foo_does_a_thing`) — to avoid surprise matches use `--exact` - `cargo nextest run` is significantly faster if the repo already uses it (`Cargo.toml` `[profile.nextest...]` or `.config/nextest.toml`) — match the repo's choice ## Lint Command Use the repo's lint script first. Otherwise: - `cargo fmt --all -- --check` (CI), `cargo fmt` (apply) - `cargo clippy --all-targets --all-features -- -D warnings` - If `clippy.toml` / `rustfmt.toml` exists, the project has opinions — never override them in your tests ## Project Layout ``` my_crate/ ├── Cargo.toml ├── src/ │ ├── lib.rs # library crate root │ ├── main.rs # binary crate root (mutually OK with lib.rs) │ └── module.rs # private/public module ├── tests/ # integration tests — each .rs is a separate crate │ └── widget.rs ├── benches/ # cargo bench targets └── examples/ # cargo run --example name ``` | Test type | Where | Sees | |-----------|-------|------| | Unit test | `#[cfg(test)] mod tests` inside the source file | Private items in the surrounding module | | Integration test | `tests/<name>.rs` | Only the public API of the crate | | Doc test | `///` doctests in source comments | Only the public API; runs via `cargo test --doc` | - **Unit tests** at the bottom of `module.rs`: ```rust #[cfg(test)] mod tests { use super::*; #[test] fn name_scenario_expected() { // ... } } ``` - **Integration tests** import the crate by name: `use my_crate::PublicType;` - Helpers shared between integration tests must live in `tests/common/mod.rs` (the `mod.rs` form prevents cargo from treating them as a top-level test crate) ## Test Function Patterns | Kind | Attribute | |------|-----------| | Sync test | `#[test]` | | Should panic | `#[test] #[should_panic(expected = "message substring")]` | | Ignored (long/manual) | `#[test] #[ignore = "reason"]` | | Async test (Tokio) | `#[tokio::test]` (or `#[tokio::test(flavor = "multi_thread")]`) | | Async test (async-std) | `#[async_std::test]` | | Returning `Result` | `fn name() -> Result<(), Box<dyn Error>>` — use `?` instead of `.unwrap()` | Pick the async harness the repo already uses. Do not mix `tokio` and `async-std` in tests. ## Common Errors | Error | Fix | |-------|-----| | `cannot find type X in this scope` | Add `use crate::module::X;` or `use super::*;` inside the test module | | `function or associated item not found in 'X'` | Verify the method exists on the exact type; check trait imports (e.g. `use std::io::Read`) | | `the trait bound 'X: Y' is not satisfied` | Either implement the trait, add a `where` bound, or change the test to use a type that already implements it | | `borrow of moved value` | Add `.clone()`, borrow with `&`, or restructure ownership — do not use `mem::transmute` to dodge it | | `cannot borrow as mutable` | Make the binding `let mut x` or restructure to avoid simultaneous mutable + immutable borrows | | `lifetime may not live long enough` | Add explicit lifetime annotations or use owned types (`String` instead of `&str`) in the test | | `mismatched types` between `i32` and `usize` | Use `as` casts deliberately or change the literal type with a suffix (`5usize`, `5u32`) | | `unresolved import 'crate::...'` in `tests/foo.rs` | Integration tests must import via the **crate name** (as listed in `Cargo.toml`), not `crate::` | | `error: no test target found` for `cargo test --test foo` | The file must live directly in `tests/`, not `tests/subdir/foo.rs` (subdirs are treated as helpers) | | `attempt to subtract with overflow` (debug) | Underflow on unsigned types; use `checked_sub`/`saturating_sub` or compare before subtracting | | Doctest fails to compile | Use a leading "# " on hidden setup lines; mark code blocks `ignore`/`no_run`/`should_panic` if needed | | `the following imports are unused` (warning treated as error) | Remove unused `use` statements; do not silence with `#[allow(unused_imports)]` | ## Mocking Rules Rust has no single dominant mocking framework. Match the repo: - **Trait + struct fakes** (most idiomatic): define a trait, pass `Arc<dyn Trait>` or generic `T: Trait`, implement a fake struct in tests - **`mockall`** crate: `#[automock]` on a trait generates `MockTrait` for use in tests - **`mockito`** / **`wiremock`**: HTTP server mocks for client tests - **`tempfile`**: scoped temp directories that auto-clean (`tempfile::tempdir()`) Avoid `unsafe` patches to "mock" free functions. Refactor to inject a trait instead. If a test needs more than 3 mocks, flag it as a design smell. ## Features and `cfg` - Tests behind a feature flag run only when that feature is enabled — use `#[cfg(feature = "foo")]` on the `mod tests` or individual `#[test]` functions - `--all-features` exercises everything but may pull conflicting features in some workspaces; check `cargo test --all-features` is part of CI before relying on it - Use `#[cfg(test)]` to gate test-only helpers in production source files — not `#[cfg(feature = "test")]` ## Concurrency, IO, and `unsafe` - Tests run in parallel by default. If your tests share global state (env vars, current dir, statics), serialize them with the `serial_test` crate (if present) or move state into the test - Never write to `/tmp` or the repo dir directly — use `tempfile::tempdir()` so cleanup is automatic - Tests in `unsafe` code should also run under Miri (`cargo +nightly miri test`) if the repo's CI does ## Dependency Installation (Last Resort) Only add dependencies after investigation confirms they are missing: ```toml [dev-dependencies] mockall = "0.12" tokio = { version = "1", features = ["macros", "rt-multi-thread"] } ``` Or via cargo: ``` cargo add --dev mockall cargo add --dev tokio --features macros,rt-multi-thread ``` Match the major version of any tokio/serde/etc. already pinned by the workspace. ## Skip Coverage Tools Do not configure or run coverage tools (`cargo tarpaulin`, `cargo llvm-cov`, `grcov`). Coverage is measured separately by the evaluation harness. -
swift.md 9.4 KB
# Swift Extension Language-specific guidance for Swift test generation. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — find files in `Tests/` (SPM) or `*Tests/` groups (Xcode) and copy their style. Distinguish **XCTest** (`import XCTest`, classes inheriting `XCTestCase`) from **Swift Testing** (`import Testing`, free functions tagged `@Test`) 2. **Project file** — `Package.swift` (SPM), `*.xcodeproj`, `*.xcworkspace`, or `Project.swift` (Tuist) 3. **Swift toolchain** — `.swift-version`, `swift-tools-version` line in `Package.swift`, `IPHONEOS_DEPLOYMENT_TARGET` and `SWIFT_VERSION` build settings in Xcode 4. **CI scripts** — `.github/workflows/*.yml`, `Fastfile`, `Makefile` — these reveal the canonical build/test invocation Use the testing framework the repo already uses. Both XCTest and Swift Testing can coexist in one target — match what the file you're adding tests next to uses. ## Project Type Detection | Indicator | Project type | Build tool | |-----------|--------------|------------| | `Package.swift` only | Swift Package Manager | `swift build` / `swift test` | | `*.xcodeproj` or `*.xcworkspace` | Xcode project (often app/iOS) | `xcodebuild` | | Both | SPM library + Xcode app shell | Use SPM for library targets, Xcode for app targets | | `Project.swift` (Tuist) | Tuist-generated Xcode project | Run `tuist generate` first, then xcodebuild | | `project.yml` (XcodeGen) | XcodeGen-generated project | Run `xcodegen generate` first | If both an `.xcodeproj` and `.xcworkspace` exist (e.g. CocoaPods), **always pass `-workspace` not `-project`** to xcodebuild. ## Build Commands ### Swift Package Manager | Scope | Command | |-------|---------| | Build all | `swift build` | | Build a target | `swift build --target MyLibrary` | | Build for release | `swift build -c release` | ### Xcode (`xcodebuild`) ``` xcodebuild build \ -workspace MyApp.xcworkspace \ -scheme MyAppScheme \ -destination 'platform=iOS Simulator,name=iPhone 15' \ -configuration Debug ``` - Always specify `-destination` for iOS/tvOS/watchOS — the default may not exist on the build machine - Use `-quiet` to suppress xcodebuild's chatty output, and pipe to `xcbeautify`/`xcpretty` if installed - For deterministic CI builds add `-derivedDataPath ./DerivedData` ## Test Commands ### Swift Package Manager | Scope | Command | |-------|---------| | All tests | `swift test` | | Filter by test name (XCTest) | `swift test --filter MyClassTests/testFooBar` | | Filter by test name (Swift Testing) | `swift test --filter MyTestSuite.fooBar` | | Parallel | `swift test --parallel` | | Single platform | `swift test --triple x86_64-apple-macosx` (rare; usually skip) | ### Xcode ``` xcodebuild test \ -workspace MyApp.xcworkspace \ -scheme MyAppScheme \ -destination 'platform=iOS Simulator,name=iPhone 15' \ -only-testing:MyAppTests/MyClassTests/testFooBar ``` - `-only-testing:` and `-skip-testing:` accept `Bundle/Class/Method` paths and may be repeated - `xcodebuild test-without-building` skips compilation if you've already built - For Swift Testing in Xcode 16+, use the same `-only-testing:` syntax — the runner handles both frameworks ## Lint Command Use the repo's lint tooling first: - `swiftlint lint --quiet` (autocorrect: `swiftlint --fix`) when `.swiftlint.yml` is present - `swiftformat .` when `.swiftformat` is present - Some projects gate format on a build phase — running `xcodebuild` may already invoke it ## Project Layout ### SPM ``` Package.swift Sources/ └── MyLibrary/ ├── Foo.swift └── Bar.swift Tests/ └── MyLibraryTests/ └── FooTests.swift ``` - Test target name conventionally is `<TargetName>Tests` and lives in `Tests/<TargetName>Tests/` - Test target must list its production target as a dependency in `Package.swift`: ```swift .testTarget( name: "MyLibraryTests", dependencies: ["MyLibrary"]), ``` ### Xcode - Tests live in a separate target (e.g. `MyAppTests`) added to the scheme's "Test" action - The test target's "Host Application" determines whether tests run on the simulator with the app loaded (unit tests) or as a UI test runner ## Imports - XCTest: `import XCTest` plus `@testable import MyLibrary` to access `internal` symbols - Swift Testing: `import Testing` plus `@testable import MyLibrary` - `@testable` works only when the production target is built with `-enable-testing` (the SPM test target and Xcode "Debug" config do this by default) - Never mark production code `public` solely to make it visible to tests — use `@testable import` instead ## Test File Templates ### Swift Testing (Xcode 16 / Swift 6) ```swift import Testing @testable import MyLibrary @Suite("Calculator") struct CalculatorTests { @Test("add returns the sum of two integers") func addReturnsSum() { let calc = Calculator() #expect(calc.add(2, 3) == 5) } @Test("add throws on overflow", arguments: [ (Int.max, 1), (Int.min, -1), ]) func addThrowsOnOverflow(a: Int, b: Int) { #expect(throws: ArithmeticError.self) { try Calculator().add(a, b) } } } ``` ### XCTest ```swift import XCTest @testable import MyLibrary final class CalculatorTests: XCTestCase { func testAddReturnsSum() { let calc = Calculator() XCTAssertEqual(calc.add(2, 3), 5) } func testAddThrowsOnOverflow() { XCTAssertThrowsError(try Calculator().add(.max, 1)) { error in XCTAssertEqual(error as? ArithmeticError, .overflow) } } } ``` - XCTest requires test methods to start with `test` and take no arguments - Mark XCTest classes `final` to silence warnings and prevent unintended subclassing - Use `XCTUnwrap` instead of force-unwrapping (`!`) inside tests so the failure is reported rather than crashing the runner ## Async, Throws, and Concurrency - Test methods may be `async` and/or `throws` in both frameworks - For asynchronous expectations under XCTest, use `XCTestExpectation` + `wait(for:timeout:)` only when you cannot refactor to `async` - For Swift Testing, use `await confirmation { ... }` to assert that a callback fires - Cancel tasks deliberately with `Task.cancel()` instead of relying on test timeout ## Common Errors | Error | Fix | |-------|-----| | `cannot find 'X' in scope` from a test | Add `@testable import MyLibrary` (and ensure the test target depends on it) | | `module 'MyLibrary' was not compiled for testing` | Build the production target with `-enable-testing`; SPM test targets do this automatically — Xcode Debug configs need "Enable Testability" = YES | | `failed to launch test runner` (Xcode) | Simulator destination may be invalid; list with `xcrun simctl list devices` and pick an existing one | | `No such module 'XCTest'` outside a test target | XCTest is only available in test targets — do not import it from production code | | `Static method 'expect(_:_:sourceLocation:)' is unavailable` / `No such module 'Testing'` | Swift Testing requires Swift 6 / Xcode 16+. On older toolchains, fall back to XCTest | | `Symbol not found: _OBJC_CLASS_$_...` | Linker missing a framework; add it to the test target's "Link Binary With Libraries" | | `signal SIGABRT` in tests | Often a force-unwrap on `nil`; replace `!` with `XCTUnwrap` to localize the failure | | `MainActor-isolated property cannot be referenced from a non-isolated context` | Mark the test method `@MainActor` or move setup into a `MainActor` task | | `Sandbox: ... deny file-write-create` | Use `FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)` instead of writing to fixed paths | | Test discovery shows zero tests on Linux | XCTest on Linux needs `XCTMain([testCase(MyTests.allTests), ...])` in `Tests/LinuxMain.swift` (legacy SwiftPM only); for Swift 5.4+ this is auto-generated | ## Mocking Rules Swift has no Mockito-equivalent — favor protocol-oriented design: - Define a **protocol** for the dependency, pass it via initializer, and implement a fake/stub struct in the test target - For URL/HTTP, use `URLProtocol` subclasses to intercept `URLSession` requests, or use `MockingbirdSwift` / `Cuckoo` if the repo already adopts them - For dates/clocks, inject a `Clock` (`ContinuousClock`, `SuspendingClock`, or a custom `Clock`-conforming type) — do not call `Date()` directly in business logic - Avoid `swizzling` and runtime hacks — they break under Swift's optimizer If a test needs more than 3 mocks, flag it as a design smell. ## Cross-Platform Considerations - Swift on Linux supports XCTest but **not** all of Foundation — guard with `#if canImport(Darwin)` or `#if os(macOS)` only when necessary - Use `String(decoding:as:)` rather than `String(contentsOf:encoding:)` for cross-platform reads - Be careful with `Bundle.main` in tests — on macOS unit tests it points to `xctest`, not your bundle; use `Bundle(for: type(of: self))` (XCTest) or a resource-bundle helper ## Dependency Installation (Last Resort) Only add dependencies after investigation confirms they are missing. `Package.swift`: ```swift .package(url: "https://github.com/apple/swift-collections.git", from: "1.1.0"), ``` Then add to the test target's `dependencies:`. For CocoaPods/Carthage, edit `Podfile`/`Cartfile` and run `pod install` / `carthage update --use-xcframeworks`. ## Skip Coverage Tools Do not configure or run coverage tools (`-enableCodeCoverage YES`, `xccov`, `slather`). Coverage is measured separately by the evaluation harness. -
typescript-examples.md 12.7 KB
# TypeScript Pipeline Examples Concrete input→output examples for the test generation pipeline targeting a TypeScript codebase using Vitest. These show what each pipeline phase produces for a small project. > Jest, Mocha, and node:test follow the same shape. Replace `vi.fn()` / `vi.mock()` with `jest.fn()` / `jest.mock()` (Jest) or hand-written stubs (node:test/Mocha) and adjust the runner command accordingly. ## Source Under Test A simple `InvoiceService` in a TypeScript library using Vitest: ```text src/ invoiceService.ts invoice.ts invoiceRepository.ts index.ts (re-exports public API) package.json tsconfig.json vitest.config.ts package-lock.json (committed for reproducible installs) ``` ```typescript // src/invoiceService.ts import { Invoice, InvoiceStatus } from "./invoice"; import { InvoiceRepository } from "./invoiceRepository"; export class InvoiceService { constructor(private readonly repository: InvoiceRepository) {} calculateTotal(invoice: Invoice): number { if (invoice == null) throw new TypeError("invoice must not be null"); if (invoice.lineItems.length === 0) { throw new Error("Invoice has no line items."); } const subtotal = invoice.lineItems.reduce( (acc, li) => acc + li.quantity * li.unitPrice, 0, ); const tax = subtotal * invoice.taxRate; return roundTo2(subtotal + tax); } async getById(id: number): Promise<Invoice> { const invoice = await this.repository.find(id); if (invoice == null) { throw new Error(`Invoice ${id} not found.`); } return invoice; } async markAsPaid(id: number): Promise<void> { const invoice = await this.repository.find(id); if (invoice == null) { throw new Error(`Invoice ${id} not found.`); } if (invoice.status === InvoiceStatus.Paid) { throw new Error("Invoice is already paid."); } invoice.status = InvoiceStatus.Paid; invoice.paidDate = new Date(); await this.repository.update(invoice); } } function roundTo2(n: number): number { return Math.round((n + Number.EPSILON) * 100) / 100; } ``` ## Sample Research Output What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`: ```markdown # Test Generation Research ## Project Overview - **Path**: /work/contoso-billing - **Language**: TypeScript 5.4 - **Module system**: ESM (`"type": "module"` in package.json) - **Test Framework**: Vitest 1.x (detected via `vitest.config.ts` and `devDependencies.vitest`) - **Package Manager**: npm (lockfile = `package-lock.json`) ## Coverage Baseline - **Initial Line Coverage**: unknown - **Strategy**: broad - **Existing Test Count**: 0 tests across 0 files ## Build & Test Commands - **Install**: `npm ci` - **Type-check**: `npx tsc --noEmit` - **Test**: `npx vitest run` (NEVER bare `vitest` — that starts watch mode) - **Lint**: none configured ## Project Structure - Source: `src/` - Tests: none (will colocate as `src/invoiceService.test.ts` to match Vitest defaults) ## Files to Test ### High Priority | File | Classes/Functions | Testability | Notes | |------|-------------------|-------------|-------| | src/invoiceService.ts | InvoiceService: calculateTotal, getById, markAsPaid | High | Core business logic, repository dependency needs mocking | ### Low Priority / Skip | File | Reason | |------|--------| | src/invoice.ts | Type definitions and enum | | src/invoiceRepository.ts | Interface only | | src/index.ts | Re-export barrel | ## Existing Tests - No existing tests found ## Testing Patterns - No existing patterns; recommend `describe`/`it` blocks, `vi.fn()` stubs for the repository interface, and `it.each` for table-driven cases. ## Recommendations - Co-locate test next to source (`src/invoiceService.test.ts`) — matches Vitest defaults and avoids reaching into `../src/` - Use a fake-timers helper (`vi.useFakeTimers()`) to control `new Date()` in `markAsPaid` - Use a type-narrowed mock object (`{ find: vi.fn(), update: vi.fn() } satisfies InvoiceRepository`) rather than full module mocking ``` ## Sample Plan Output What `code-testing-planner` produces in `<TESTAGENT_DIR>/plan.md`: ```markdown # Test Implementation Plan ## Overview Generate Vitest tests for InvoiceService, covering all three public methods across happy path, edge case, and error scenarios. Single phase since there is only one source file. ## Commands - **Install**: `npm ci` - **Type-check**: `npx tsc --noEmit` - **Test (file-scoped during dev)**: `npx vitest run src/invoiceService.test.ts` - **Test (full)**: `npx vitest run` ## Phase Summary | Phase | Focus | Files | Est. Tests | |-------|-------|-------|------------| | 1 | InvoiceService | 1 | 9-12 | --- ## Phase 1: InvoiceService ### Overview Cover all public methods of InvoiceService. `calculateTotal` is pure logic tested with `it.each`. Async methods require a fake repository. ### Files to Test #### 1. invoiceService.ts - **Source**: `src/invoiceService.ts` - **Test File**: `src/invoiceService.test.ts` **Methods to Test**: 1. `calculateTotal` — Pure calculation logic - Happy path: single line item returns quantity × price + tax - Happy path: multiple line items summed correctly - Edge case: zero tax rate returns subtotal only - Error case: null invoice throws TypeError - Error case: empty line items throws Error 2. `getById` — Repository lookup - Happy path: existing ID returns invoice - Error case: missing ID rejects with Error 3. `markAsPaid` — State transition - Happy path: pending invoice transitions to Paid with `paidDate` set - Error case: already-paid rejects with Error - Error case: missing ID rejects with Error ### Success Criteria - [ ] Test file created at `src/invoiceService.test.ts` - [ ] `npx tsc --noEmit` succeeds - [ ] `npx vitest run` reports all tests passed - [ ] No real network/timers — repository is a `vi.fn()` fake, `new Date()` is controlled via fake timers ``` ## Sample Generated Test File What `code-testing-implementer` produces: ```typescript // src/invoiceService.test.ts import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { Invoice, InvoiceStatus } from "./invoice"; import type { InvoiceRepository } from "./invoiceRepository"; import { InvoiceService } from "./invoiceService"; function makeRepository(): InvoiceRepository & { find: ReturnType<typeof vi.fn>; update: ReturnType<typeof vi.fn> } { return { find: vi.fn(), update: vi.fn(), }; } describe("InvoiceService", () => { let repository: ReturnType<typeof makeRepository>; let sut: InvoiceService; beforeEach(() => { repository = makeRepository(); sut = new InvoiceService(repository); }); // --- calculateTotal --- describe("calculateTotal", () => { it.each([ { quantity: 1, unitPrice: 100, taxRate: 0.1, expected: 110 }, { quantity: 3, unitPrice: 25, taxRate: 0, expected: 75 }, { quantity: 2, unitPrice: 9.99, taxRate: 0.07, expected: 21.38 }, ])( "returns $expected for $quantity × $unitPrice with tax $taxRate", ({ quantity, unitPrice, taxRate, expected }) => { const invoice: Invoice = { id: 1, status: InvoiceStatus.Pending, taxRate, lineItems: [{ quantity, unitPrice }], }; expect(sut.calculateTotal(invoice)).toBe(expected); }, ); it("throws TypeError when invoice is null", () => { expect(() => sut.calculateTotal(null as unknown as Invoice)).toThrow(TypeError); }); it("throws when line items are empty", () => { const invoice: Invoice = { id: 1, status: InvoiceStatus.Pending, taxRate: 0, lineItems: [], }; expect(() => sut.calculateTotal(invoice)).toThrow("no line items"); }); }); // --- getById --- describe("getById", () => { it("returns the invoice for an existing id", async () => { const expected: Invoice = { id: 42, status: InvoiceStatus.Pending, taxRate: 0, lineItems: [] }; repository.find.mockResolvedValue(expected); await expect(sut.getById(42)).resolves.toBe(expected); expect(repository.find).toHaveBeenCalledWith(42); }); it("rejects with Error when the id is missing", async () => { repository.find.mockResolvedValue(null); await expect(sut.getById(999)).rejects.toThrow(/999/); }); }); // --- markAsPaid --- describe("markAsPaid", () => { beforeEach(() => { vi.useFakeTimers(); vi.setSystemTime(new Date("2025-01-01T12:00:00.000Z")); }); afterEach(() => { vi.useRealTimers(); }); it("transitions a pending invoice to Paid with paidDate set", async () => { const invoice: Invoice = { id: 1, status: InvoiceStatus.Pending, taxRate: 0, lineItems: [], }; repository.find.mockResolvedValue(invoice); repository.update.mockResolvedValue(undefined); await sut.markAsPaid(1); expect(invoice.status).toBe(InvoiceStatus.Paid); expect(invoice.paidDate).toEqual(new Date("2025-01-01T12:00:00.000Z")); expect(repository.update).toHaveBeenCalledWith(invoice); }); it("rejects when the invoice is already paid", async () => { const invoice: Invoice = { id: 1, status: InvoiceStatus.Paid, taxRate: 0, lineItems: [], }; repository.find.mockResolvedValue(invoice); await expect(sut.markAsPaid(1)).rejects.toThrow("already paid"); expect(repository.update).not.toHaveBeenCalled(); }); it("rejects when the id is missing", async () => { repository.find.mockResolvedValue(null); await expect(sut.markAsPaid(999)).rejects.toThrow(/999/); }); }); }); ``` ## Sample Fix Cycle When the implementer encounters a runner or type error, the fixer agent diagnoses and resolves it. **Test output:** ```text Error: Vitest failed to access its internal state. One of the following is possible: - "vitest" is imported directly without running "vitest" command ``` **Fixer diagnosis:** The agent ran `node src/invoiceService.test.ts` (or bare `vitest`, which is watch-mode). The runner must be invoked via `npx vitest run`. **Fix applied:** ```bash # Wrong — bare vitest starts an interactive watcher in CI npx vitest # Right — `run` is the one-shot command npx vitest run ``` **Rerun:** SUCCESS --- **Another common cycle — ESM/CJS mismatch:** **Test output:** ```text SyntaxError: Cannot use import statement outside a module ``` **Fixer diagnosis:** The project's `tsconfig.json` emits ESM (`"module": "NodeNext"`) but `package.json` has no `"type": "module"`. Vitest happens to handle this natively; switching to Jest would require additional configuration. The fix here is to ensure Vitest is the runner being used (as already configured in `vitest.config.ts`) and avoid recompiling test files through a separate non-ESM-aware tool. **Fix applied:** Use `npx vitest run` (which uses esbuild internally and handles both ESM and CJS) instead of compiling with `tsc` and running the emitted `.js` directly. **Rerun:** SUCCESS --- **Another common cycle — wrong mock typing:** **Build output:** ```text src/invoiceService.test.ts:14:5 - error TS2322: Type '{ find: Mock<any, any>; }' is not assignable to type 'InvoiceRepository'. Property 'update' is missing in type '{ find: Mock<any, any>; }' but required in type 'InvoiceRepository'. ``` **Fixer diagnosis:** The fake repository only stubbed `find`, not `update`. The `InvoiceRepository` interface requires both. TypeScript caught this at compile time. **Fix applied:** ```typescript // Before const repository = { find: vi.fn() } as InvoiceRepository; // After — provide both methods, narrow the return type so the test code keeps autocomplete function makeRepository(): InvoiceRepository & { find: ReturnType<typeof vi.fn>; update: ReturnType<typeof vi.fn> } { return { find: vi.fn(), update: vi.fn() }; } ``` **Rebuild + rerun:** SUCCESS ## Sample Final Report What `code-testing-generator` produces at Step 9: ```markdown ## Test Generation Report **Project**: contoso-billing (TypeScript) **Strategy**: Direct (single source file in scope) ### Results | Metric | Value | |----------------|-------| | Tests created | 9 | | Tests passing | 9 | | Tests failing | 0 | | Files created | 1 | ### Files Created - `src/invoiceService.test.ts` (9 tests, 3 parameterized via `it.each`) ### Coverage - InvoiceService.calculateTotal — 3 happy path, 2 error cases - InvoiceService.getById — 1 happy path, 1 error case - InvoiceService.markAsPaid — 1 happy path, 2 error cases ### Build / Test Validation - Install: ✅ `npm ci` - Type-check: ✅ `npx tsc --noEmit` - Test run: ✅ `npx vitest run` ### Next Steps - Add tests for any HTTP/Express adapters once they exist - Consider property-based testing (`fast-check`) for `calculateTotal` rounding ``` -
typescript.md 6.8 KB
# TypeScript Extension Language-specific guidance for TypeScript (and JavaScript) test generation. ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: 1. **Existing tests** — find `*.test.ts` / `*.spec.ts` files and copy their style (imports, describe/it vs test, assertion patterns, mock approach) 2. **`package.json`** — `scripts.test`, `devDependencies`, `type` field 3. **Config files** — `tsconfig.json`, `jest.config.*`, `vitest.config.*`, `eslint.config.*` Use the repo's existing test runner and conventions — do not switch frameworks. If multiple runners are configured, follow whichever `scripts.test` invokes. Only introduce a framework if the repo has no tests at all. ## Package Manager Detection Detect the package manager from lockfiles and use it consistently for **all** commands: | Indicator | Manager | Run script | Execute binary | |-----------|---------|------------|----------------| | `pnpm-lock.yaml` | pnpm | `pnpm test` | `pnpm exec <tool>` | | `yarn.lock` | Yarn | `yarn test` | `yarn <tool>` | | `bun.lockb` / `bun.lock` | Bun | `bun test` | `bunx <tool>` | | `package-lock.json` or none | npm | `npm test` | `npx <tool>` | Use `<exec>` below as shorthand for the detected exec command. ## Build Commands | Scope | Command | |-------|---------| | Type check | `<exec> tsc --noEmit` or the repo's `typecheck` script | | Build (if configured) | The repo's `build` script | Many projects don't need an explicit build step — the test runner handles transpilation. ## Test Commands Detect the runner from `devDependencies` and `scripts.test`. Always prefer the repo's test script first. | Runner | Run once | Filter by file | Filter by name | |--------|----------|----------------|----------------| | **Jest** | `<exec> jest` | `<exec> jest path/to/file` | `<exec> jest -t "name"` | | **Vitest** | `<exec> vitest run` | `<exec> vitest run path/to/file` | `<exec> vitest run -t "name"` | | **Mocha** | `<exec> mocha` | (use config or positional args) | `<exec> mocha --grep "name"` | - **Always use `vitest run`** (not bare `vitest`) — bare `vitest` starts watch mode - **Never use `--watch`** — the agent must not start interactive/watch mode - For Jest: `--bail` to stop on first failure, `--verbose` for detail - Mocha `--grep` filters by **test name**, not file path ## Lint Command Use the repo's lint script first. Otherwise detect from `devDependencies` and config: - `eslint.config.*` or `.eslintrc.*` → `<exec> eslint --fix path/to/file.ts` - `prettier` → `<exec> prettier --write path/to/file.ts` - `biome.json` → `<exec> biome check --write path/to/file.ts` ## Project Layout and Imports | Layout | Import Style | |--------|-------------| | Colocated (`src/module.test.ts`) | `import { X } from './module'` | | `__tests__/` dir | `import { X } from '../module'` | | Top-level `tests/` | `import { X } from '../src/module'` | - **Match existing test imports** — copy path style from neighboring tests - If `tsconfig.json` has `paths` aliases (e.g., `@/`), use them in tests too - For monorepos: import from the package name, not relative cross-package paths - For monorepo workspaces (Nx, Turborepo, Lerna): run tests via the workspace tool (`nx test <project>`, `turbo test`), not from a random package directory ## Test File Naming - Match existing convention — check for `.test.ts` vs `.spec.ts` - Jest/Vitest default: `*.test.ts`, `*.spec.ts`, or files inside `__tests__/` - Place test files to mirror the existing project pattern ## Common Errors | Error | Fix | |-------|-----| | `Cannot find module 'X'` | Check existing imports for correct paths; verify `tsconfig.json` `paths`; check `moduleNameMapper` (Jest) or `resolve.alias` (Vitest) | | `TS2305: has no exported member` | Verify the exact export name from the source file | | `TS2345: type not assignable` | Match the expected type; use type assertion only for mock objects | | `SyntaxError: Unexpected token` / `Jest encountered an unexpected token` | Verify TS transform config (`ts-jest`, `@swc/jest`, or Vitest handles natively) | | `ReferenceError: describe is not defined` | Vitest: import from `vitest` or set `globals: true` in config; Jest: ensure tests run under Jest not bare `node` | | `Cannot use import statement outside a module` / `ERR_REQUIRE_ESM` | ESM/CJS mismatch — align runner config with the project's module system (see ESM section); do **not** blindly set `"type": "module"` | | `ReferenceError: document is not defined` | Set test environment: `testEnvironment: 'jsdom'` (Jest) or `environment: 'jsdom'` (Vitest) | | `jest.mock() ... out-of-scope variables` | Keep `jest.mock()` at top level; don't reference variables declared after the mock call (Jest hoists mocks) | | `Cannot find module '@/...'` | Mirror the project's alias config in the test runner's module resolution | | `Warning: not wrapped in act(...)` | Await async UI updates using the repo's existing pattern (`waitFor`, `act`) | ## ESM vs CommonJS Check these signals to determine the project's module system: - `"type": "module"` in `package.json` → ESM - `"module": "ESNext"` or `"NodeNext"` in `tsconfig.json` → ESM output (but not sufficient alone) - `.mjs`/`.mts` extensions → ESM files If the test runner fails with ESM errors, align the runner's config with the project's module system. **Do not change `package.json` `type` field** — align the test runner to match whatever the project uses: - **Jest**: `--experimental-vm-modules` + `ts-jest` with `useESM: true`, or `@swc/jest` - **Vitest**: handles ESM natively - **Mocha**: `--loader ts-node/esm` ## Mocking Rules - Prefer dependency injection over module mocking - Use typed mocks: `jest.Mocked<T>`, `vi.mocked(obj)`, or `Partial<T>` with `as T` - Jest: `jest.mock()` is hoisted — keep at top level, don't close over local variables - Vitest: `vi.mock()` follows the same hoisting rules - If a test needs more than 3–4 mocks, flag it as a design smell - Mock reset: rely on `clearMocks`/`restoreMocks` config if present; otherwise reset in `beforeEach` ## Framework-Specific Notes - **React/Preact**: use `@testing-library/react`, wrap with necessary providers (router, query client, theme) matching existing test setup - **Express/Koa**: use `supertest` for HTTP testing if the repo already uses it - **NestJS**: build testing module with `Test.createTestingModule` — don't instantiate controllers directly ## Dependency Installation (Last Resort) Only install packages after investigation confirms they are missing. Use the detected package manager: ``` <manager> add --save-dev jest ts-jest @types/jest <manager> add --save-dev vitest ``` Never install test infrastructure that conflicts with what the repo already uses. ## Skip Coverage Tools Do not configure or run coverage tools (istanbul, c8, `vitest --coverage`). Coverage is measured separately by the evaluation harness.
-
-
SKILL.md 3.7 KB
--- name: code-testing-extensions description: >- Provides file paths to language-specific extension files for the code-testing pipeline. Call this skill to discover available extension guidance files (e.g., dotnet.md for .NET, cpp.md for C++). Do not use directly — invoked by code-testing agents and skills that need language-specific references. user-invocable: false disable-model-invocation: true license: MIT --- # Code Testing Extensions This skill provides access to language-specific guidance files used by the code-testing pipeline. Call this skill to get the file paths, then read the relevant file for your target language. ## Available Extension Files | File | Language | Contents | |------|----------|----------| | [extensions/dotnet.md](extensions/dotnet.md) | .NET (C#/F#/VB) | Build commands, test commands, project reference validation, common CS error codes, MSTest template | | [extensions/python.md](extensions/python.md) | Python | Framework-adaptive test commands (pytest, custom runners), project layout detection, mocking guidelines, common errors | | [extensions/typescript.md](extensions/typescript.md) | TypeScript/JavaScript | Build/test commands (Jest/Vitest/Mocha), framework detection, mocking, TS-specific considerations | | [extensions/powershell.md](extensions/powershell.md) | PowerShell | Test commands (Pester v5), module import patterns, discovery/run pitfalls, mocking, common errors | | [extensions/cpp.md](extensions/cpp.md) | C++ | Testing internals with friend declarations | | [extensions/go.md](extensions/go.md) | Go | `go test` commands, table-driven tests, integration vs unit layout, mocking via interfaces, common errors | | [extensions/java.md](extensions/java.md) | Java | Maven/Gradle commands, JUnit 4/5 and TestNG detection, Mockito, Spring Boot slices, common errors | | [extensions/rust.md](extensions/rust.md) | Rust | `cargo test` commands, unit vs integration vs doc tests, features, async test harnesses, common errors | | [extensions/ruby.md](extensions/ruby.md) | Ruby | RSpec and Minitest commands, Bundler usage, Rails specifics, mocking patterns, common errors | | [extensions/swift.md](extensions/swift.md) | Swift | SPM and Xcode test commands, XCTest vs Swift Testing, `@testable import`, async/throws tests, common errors | | [extensions/kotlin.md](extensions/kotlin.md) | Kotlin | Gradle commands, JUnit/Kotest detection, MockK, coroutines test, KMP and Android specifics, common errors | | [extensions/dotnet-examples.md](extensions/dotnet-examples.md) | .NET (C#/F#/VB) | Concrete pipeline examples: sample research output, plan, generated tests, fix cycles, final report | | [extensions/python-examples.md](extensions/python-examples.md) | Python | Concrete pipeline examples (pytest): research, plan, generated test file, fix cycles, final report | | [extensions/typescript-examples.md](extensions/typescript-examples.md) | TypeScript/JavaScript | Concrete pipeline examples (Vitest, applicable to Jest): research, plan, generated test file, fix cycles, final report | | [extensions/go-examples.md](extensions/go-examples.md) | Go | Concrete pipeline examples (standard `testing`): research, plan, table-driven test file, fix cycles, final report | | [extensions/java-examples.md](extensions/java-examples.md) | Java | Concrete pipeline examples (JUnit 5 + Mockito on Maven): research, plan, generated test file, fix cycles, final report | ## Usage Read the appropriate extension file for the target language before writing test code. When an `<language>-examples.md` file exists for the target language, read it alongside the base extension to see a concrete end-to-end pipeline walkthrough (research output, plan, generated tests, fix cycles, final report).
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.