go-development
Use when developing Go applications, implementing job schedulers or cron (netresearch/go-cron, ofelia), Docker API integrations, LDAP/AD clients, building resilient services with retry logic, setting up Go test suites (unit/integration/fuzz/mutation), or running golangci-lint.
Install
npx skills add https://github.com/netresearch/go-development-skill/tree/main/skills/go-development
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install netresearch-go-development-skill@llmmart
git clone https://github.com/netresearch/go-development-skill.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole netresearch/go-development-skill collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Go Development Patterns
Core Principles
Type Safety
- Avoid:
interface{}(useany),sync.Map, scattered type assertions, reflection - Prefer: Generics
[T any],errors.AsType[T](Go 1.26), concrete types - Run
go fix ./...after upgrades
Consistency
- One pattern per problem domain
- Match existing codebase patterns
- Refactor holistically or not at all
- Config precedence: defaults < config file < env vars < flags
Testing
- Build tags isolate test tiers: unit (default),
integration,e2e - Always use
t.Parallel(),t.Helper(), table-driven subtests - Use
log/slogdirectly -- never wrap it in custom Logger interfaces
Conventions
- Naming: ID, URL, HTTP (not Id, Url, Http) — not tool-enforced (ST1003 is off by default)
- Error wrapping:
fmt.Errorf("failed to process: %w", err)
References
Git hooks: ls lefthook.yml 2>/dev/null && lefthook install || echo "Add lefthook — see references/lefthook-template.md"
Load as needed:
| Reference | Purpose |
|---|---|
references/architecture.md |
Package structure, state mutation completeness |
references/logging.md |
Structured logging with log/slog, migration from logrus |
references/cron-scheduling.md |
go-cron patterns: named jobs, runtime updates, resilience |
references/resilience.md |
Pointer to go-cron's built-in retry/circuit-breaker/timeout wrappers |
references/docker.md |
Docker client patterns, buffer pooling |
references/ldap.md |
LDAP/Active Directory integration |
references/testing.md |
Build tags, resource isolation, race gotchas |
references/linting.md |
golangci-lint v2, staticcheck |
references/api-design.md |
Enum/status defensive handling |
references/fuzz-testing.md |
Go fuzzing patterns, security seeds |
references/contracts-and-invariants.md |
Contracts, invariants, property tests |
references/mutation-testing.md |
Gremlins configuration, test quality measurement |
references/makefile.md |
Standard Makefile interface for CI/CD |
references/modernization.md |
go fix modernizers and their build-tag trap, errors.AsType[T], b.Loop |
references/dependencies.md |
Upgrades: go get -u all, majors, build-set scoping |
references/lefthook-template.md |
Ready-to-use lefthook.yml for Go project git hooks |
references/branch-protection.md |
Ruleset watermark: three-ruleset gate, bypass modes |
references/reusable-workflows.md |
Reusable Actions workflow callers, permission propagation, release-gate outputs |
references/single-build-release.md |
Single-build release: cross-compile once, reuse for release+container |
references/awesome-go-submission.md |
awesome-go submission: CI-parsed PR body, name collisions |
Quality Gates
Run before completing any review:
golangci-lint run --timeout 5m # Linting
go vet ./... # Static analysis
staticcheck ./... # Additional checks
govulncheck ./... # Vulnerability scan
go test -race ./... # Race detection
Stdlib Vulnerability Fixes
When govulncheck reports stdlib vulnerabilities: check fix version via vuln.go.dev, update go X.Y.Z in go.mod, run go mod tidy.
Contributing: Submit improvements to https://github.com/netresearch/go-development-skill
Files (go-development-skill)
-
evals
-
evals.json 10.7 KB
[ { "name": "setup_new_go_project", "prompt": "Set up a new Go project with testing, linting, and a Makefile", "assertions": [ { "type": "content", "pattern": "(go mod init|go\\.mod|Makefile)" }, { "type": "content", "pattern": "(golangci-lint|go test|go vet)" } ] }, { "name": "add_ldap_integration", "prompt": "Add LDAP integration to this Go project for user authentication", "assertions": [ { "type": "content", "pattern": "(ldap|go-ldap|LDAP|Active Directory)" }, { "type": "content", "pattern": "(Bind|Search|TLS|connection pool)" }, { "type": "content", "pattern": "go-ldap/ldap" } ] }, { "name": "setup_cron_scheduler", "prompt": "Implement a cron-based job scheduler in Go with named jobs and resilience", "assertions": [ { "type": "content", "pattern": "(go-cron|netresearch/go-cron|cron\\.New)" }, { "type": "content", "pattern": "(AddFunc|AddJob|WithName|RetryWithBackoff|RetryOnError)" } ] }, { "name": "docker_client_integration", "prompt": "Write a Go Docker client that executes containers with buffer pooling", "assertions": [ { "type": "content", "pattern": "(go-dockerclient|fsouza|docker\\.NewClient)" }, { "type": "content", "pattern": "(sync\\.Pool|bufferPool|Buffer)" }, { "type": "content", "pattern": "(fsouza/go-dockerclient|sync\\.Pool)" } ] }, { "name": "retry_with_backoff", "prompt": "Implement exponential backoff retry logic in Go with jitter and context cancellation", "assertions": [ { "type": "content", "pattern": "(RetryConfig|MaxAttempts|BackoffFactor|Jitter)" }, { "type": "content", "pattern": "(context\\.Done|ctx\\.Err|time\\.After)" } ] }, { "name": "graceful_shutdown", "prompt": "Implement graceful shutdown for a Go HTTP server with signal handling", "assertions": [ { "type": "content", "pattern": "(signal\\.Notify|os\\.Signal|SIGTERM|SIGINT)" }, { "type": "content", "pattern": "(Shutdown|context\\.WithTimeout|srv\\.Shutdown)" }, { "type": "content", "pattern": "(signal\\.Notify|Shutdown\\()" } ] }, { "name": "table_driven_tests", "prompt": "Write table-driven tests for a Go function that parses user input strings", "assertions": [ { "type": "content", "pattern": "(tests?\\s*:?=?\\s*\\[?\\]?struct|tc\\.|tt\\.|test\\.name)" }, { "type": "content", "pattern": "(t\\.Run|t\\.Parallel|t\\.Helper)" }, { "type": "content", "pattern": "(t\\.Run|t\\.Parallel)" } ] }, { "name": "fuzz_testing", "prompt": "Add fuzz tests to a Go URL parser to find edge cases and security issues", "assertions": [ { "type": "content", "pattern": "(func Fuzz|f\\.Fuzz|f\\.Add|testing\\.F)" }, { "type": "content", "pattern": "(go test.*-fuzz|fuzz\\s+build\\s+tag|corpus)" } ] }, { "name": "mutation_testing_setup", "prompt": "Set up mutation testing for a Go project to measure test quality", "assertions": [ { "type": "content", "pattern": "(gremlins|go-gremlins|\\.gremlins\\.yaml)" }, { "type": "content", "pattern": "(mutation.*score|unleash|mutant)" }, { "type": "content", "pattern": "(gremlins|\\.gremlins\\.yaml)" } ] }, { "name": "golangci_lint_v2_config", "prompt": "Create a golangci-lint v2 configuration for a production Go project", "assertions": [ { "type": "content", "pattern": "(version:\\s*\"?2|golangci-lint.*v2)" }, { "type": "content", "pattern": "(errcheck|staticcheck|govet|bodyclose|noctx)" }, { "type": "content", "pattern": "(bodyclose|noctx)" } ] }, { "name": "error_handling_conventions", "prompt": "Review this Go code for error handling: `var InvalidInput = errors.New(\"Invalid input.\")` and `return fmt.Errorf(\"Failed to process\")`", "assertions": [ { "type": "content", "pattern": "(lowercase|no punctuation|ErrInvalidInput|Err\\s*prefix)" }, { "type": "content", "pattern": "(%w|errors\\.New|fmt\\.Errorf|wrap)" } ] }, { "name": "slog_structured_logging", "prompt": "Migrate a Go project from logrus to structured logging with log/slog", "assertions": [ { "type": "content", "pattern": "(log/slog|slog\\.New|slog\\.Logger|TextHandler|JSONHandler)" }, { "type": "content", "pattern": "(LevelVar|AddSource|slog\\.Info|slog\\.Error)" } ] }, { "name": "api_design_functional_options", "prompt": "Design a Go API using the functional options pattern for a configurable HTTP client", "assertions": [ { "type": "content", "pattern": "(Option|func\\(.*\\)|With\\w+|functional option)" }, { "type": "content", "pattern": "(WithTimeout|WithRetry|apply|opts)" } ] }, { "name": "makefile_standard_targets", "prompt": "Create a Makefile for a Go project with standard CI targets", "assertions": [ { "type": "content", "pattern": "(test:|build:|lint:|all:)" }, { "type": "content", "pattern": "(-race|-coverprofile|govulncheck|-trimpath)" } ] }, { "name": "go_modernization", "prompt": "Modernize a Go 1.20 codebase to use Go 1.26 features like generics and go fix", "assertions": [ { "type": "content", "pattern": "(go fix|modernize|errors\\.AsType)" }, { "type": "content", "pattern": "(any|interface\\{\\}.*any|generics|\\[T)" }, { "type": "content", "pattern": "(errors\\.AsType|wg\\.Go)" } ] }, { "name": "package_structure", "prompt": "Design the package structure for a Go microservice with HTTP API, job scheduler, and external integrations", "assertions": [ { "type": "content", "pattern": "(cmd/|core/|internal/|web/|config/)" }, { "type": "content", "pattern": "(main\\.go|handler|middleware|domain)" }, { "type": "content", "pattern": "(cmd/|internal/|core/)" } ] }, { "name": "context_propagation", "prompt": "Review this Go code for proper context usage: HTTP handlers that spawn goroutines without passing context", "assertions": [ { "type": "content", "pattern": "(context\\.Context|ctx|context\\.Background|context\\.WithCancel)" }, { "type": "content", "pattern": "(noctx|propagat|goroutine|request.*context)" } ] }, { "name": "setup_lefthook", "prompt": "Set up git hooks for a Go project using lefthook with pre-commit and pre-push stages", "assertions": [ { "type": "content", "pattern": "(lefthook|lefthook\\.yml)" }, { "type": "content", "pattern": "(pre-commit|pre-push|golangci-lint|gofmt|go vet)" } ] }, { "name": "vulnerability_scanning", "prompt": "A Go project's govulncheck reports stdlib vulnerabilities. How do I fix them?", "assertions": [ { "type": "content", "pattern": "(govulncheck|vuln\\.go\\.dev|go\\.mod)" }, { "type": "content", "pattern": "(go mod tidy|go X\\.Y\\.Z|fix version|patch)" }, { "type": "content", "pattern": "(vuln\\.go\\.dev|go mod tidy)" } ] }, { "name": "config_management", "prompt": "Implement configuration management for a Go service with defaults, file, env vars, and flags", "assertions": [ { "type": "content", "pattern": "(defaults|config.*file|env|flag)" }, { "type": "content", "pattern": "(precedence|override|os\\.Getenv|viper|kong)" } ] }, { "name": "integration_test_docker", "prompt": "Write integration tests for a Go service that depend on a PostgreSQL database using Docker", "assertions": [ { "type": "content", "pattern": "(integration|build tag|testcontainers|docker)" }, { "type": "content", "pattern": "(TestMain|setup|teardown|t\\.Cleanup)" } ] }, { "name": "awesome_go_submission", "prompt": "Prepare a pull request to add a Go library to the awesome-go list", "assertions": [ { "type": "content", "pattern": "(Forge link|pkg\\.go\\.dev|goreportcard)" }, { "type": "content", "pattern": "(alphabetical|non-promotional|ends with a period|single item|one (package|item)|exact project name)" } ] }, { "name": "integration_tier_visibility", "prompt": "My Go unit tests all pass locally but the integration job in CI fails on a test I just edited. The test file has no build tag. What is going on and how do I verify a change before pushing?", "assertions": [ { "type": "content", "pattern": "(untagged|no build tag|without a? ?build tag).{0,200}(both|every|all)" }, { "type": "content", "pattern": "(t\\.Skip|stub)" }, { "type": "content", "pattern": "-tags=?[\"']?integration" } ] }, { "name": "tag_and_run_filter_selection", "prompt": "Our Makefile runs `go test -tags=integration -run=\"Test.*Integration\" ./...` for the integration target. Is that correct, and how would I check what it actually runs without starting containers?", "assertions": [ { "type": "content", "pattern": "-test\\.list" }, { "type": "content", "pattern": "go test -c" }, { "type": "content", "pattern": "(subtract|drops?|never run|silently|only the tag)" } ] }, { "name": "coverage_threshold_population", "prompt": "Our Makefile enforces a coverage threshold from `go tool cover -func` over ./... and it has been green for months. Is that a meaningful gate for the library?", "assertions": [ { "type": "content", "pattern": "(population|examples/|testutil|mixes|which packages)" }, { "type": "content", "pattern": "(COVERAGE_PACKAGES|profile (only )?the|narrow)" }, { "type": "content", "pattern": "(codecov|ignore)" } ] } ]
-
-
references
-
api-design.md 2 KB
# Go API Design Patterns Functional options, builders, bitmask flags, and interface-segregation are standard Go idioms with no Netresearch-specific variant — see `references/cron-scheduling.md` § Custom Parser Construction (Bitmask Options) for the org's one concrete instance (go-cron's `ParseOption`). ## Enum & Status Type Safety Defensive handling for enum / status types so invalid values cannot be silently mishandled: - Add a `Valid()` method that returns `false` for unknown values. - Always include a `default` case in `switch` statements over the type. - Write tests for unknown/zero values, not just the known ones. ```go type Policy int const ( PolicyUnknown Policy = iota // zero value — explicitly invalid, so an PolicyRetry // uninitialized Policy is rejected by Valid() PolicySkip PolicyFail ) // Valid reports whether p is a known policy. Values from deserialized data, // API input, or a future enum addition that isn't handled here return false. func (p Policy) Valid() bool { switch p { case PolicyRetry, PolicySkip, PolicyFail: return true default: return false } } func (p Policy) String() string { switch p { case PolicyUnknown: return "unknown" case PolicyRetry: return "retry" case PolicySkip: return "skip" case PolicyFail: return "fail" default: return fmt.Sprintf("Policy(%d)", int(p)) // never panic on unknown } } ``` Why: the zero value of an int-backed enum is always a valid `int` but may be a meaningless policy. A `Valid()` guard at trust boundaries (config load, API input, DB read) turns a silent wrong-branch bug into an explicitly rejected value. ```go func TestPolicy_Valid_RejectsUnknown(t *testing.T) { if Policy(0).Valid() { // the zero value must be rejected t.Error("zero-value Policy(0) should be invalid") } if Policy(99).Valid() { t.Error("Policy(99) should be invalid") } } ``` -
architecture.md 1.8 KB
# Go Architecture Patterns ## Package Structure Standard Go convention: `cmd/` for entry points, `internal/` for private packages, one directory per bounded concern (business logic, CLI, HTTP layer, config). See [golang-standards/project-layout](https://github.com/golang-standards/project-layout) for a common (if unofficial) community reference — there is no Netresearch-specific override. ## State Mutation Completeness When an operation changes an object's state, update **all** tracking fields in the same place — not just the one you came to change. Partial updates leave the object internally inconsistent and produce bugs that are hard to trace back to their cause. ```go // After a run, update every field that describes "what happened", // on both the success and failure paths: func (j *Job) recordRun(start time.Time, err error) { j.LastRunTime = start j.LastDuration = time.Since(start) j.RunCount++ j.LastError = err if err != nil { j.FailureCount++ j.Status = StatusFailed } else { j.Status = StatusCompleted } } ``` Anti-pattern: bumping `RunCount` but forgetting `LastError`/`Status`, so a failed run still reports as "completed". Keep the mutation in one method so the full set is always updated together. ## Job-Scheduler Reference Implementation The concrete job interface hierarchy (`BareJob`, `ExecJob`/`RunJob`/`LocalJob`), resilient-job wrapper, middleware chain (logging/metrics/notification), scheduler core loop, and jobs REST API previously documented here describe ofelia's actual implementation, not a general Go convention. Follow-up (not done in this PR): move them to the `netresearch/ofelia` repo's `AGENTS.md`. In the meantime, see `references/cron-scheduling.md` for the general-purpose `netresearch/go-cron` library API. -
awesome-go-submission.md 7.2 KB
# Submitting a Go Project to awesome-go How to get a Go library accepted into [avelino/awesome-go](https://github.com/avelino/awesome-go) on the first try. The list is curated and gated by an automated CI suite plus maintainer review; most rejections are mechanical (PR-body format, alphabetical order) rather than quality. This doc captures the exact format the CI parses and the gotchas that aren't in the contributing guide. ## Quick index | Piece | Where | |---|---| | Is the project eligible? | [Eligibility](#eligibility) | | What the CI validates automatically | [Automated checks](#automated-checks) | | The single biggest rejection cause | [PR body](#pr-body-1-rejection-cause) | | README entry format + name collisions | [README entry](#readme-entry) | | End-to-end submission steps | [Process](#process) | | Reading the bot's report | [After opening](#after-opening) | ## Eligibility Verify all of these before starting — most are blocking CI checks: - **≥ 5 months of repository history** (since first commit). Hard gate; nothing else matters until this passes. - **Open-source license** — any [OSI-approved](https://opensource.org/licenses/alphabetical) license. *No license = all-rights-reserved = ineligible*, even if the repo is public. - **`go.mod` at repo root** and **≥ 1 SemVer tag** (`vX.Y.Z`). - **`pkg.go.dev` page is live** for the module (visit it once / `GOPROXY=https://proxy.golang.org go get <module>@<tag>` to trigger indexing). - ~~**Go Report Card grade A-, A, or A+**~~ — goreportcard.com was sunset in 2026 and no longer issues grades, so this gate cannot be satisfied as written. Check the current awesome-go contribution guidelines before submitting; expect the requirement to have been dropped or replaced. - **A reachable coverage-service link** (Codecov/Coveralls) — a README badge is *not* enough; the bot fetches the URL. - Category must have **≥ 3 items** (only relevant if creating a new category). ## Automated checks On PR open, a `github-actions` bot posts a sticky **"Automated Quality Checks"** + **"PR Diff Validation"** report. Know which are blocking: **Blocking (PR cannot merge):** repo accessible · `go.mod` present · SemVer tag · pkg.go.dev reachable · Go Report Card ≥ A- · **required links present in PR body** · single item per PR · README link matches forge link · description ends with a period · alphabetical order · no duplicate link · entry-format regex · category ≥ 3. **Warnings only:** OSS license detected · 5-month maturity · CI/CD present · README present · coverage link reachable · link text matches repo name · **non-promotional description** · only `README.md` changed. > The repo-wide `Running test` job (`TestAlpha`, `TestDuplicatedLinks`) fails on **almost every PR** because `main` itself carries pre-existing alphabetical drift and duplicate links in *unrelated* categories. If your category and project name do **not** appear in that log, the failure is not yours — maintainers merge despite it. Don't try to "fix" it in your PR. ## PR body (#1 rejection cause) The most common rejection is a PR body the CI can't parse. It does **not** read prose; it extracts the four required links from the template's labeled lines. Fill the current template and put the **visible URL** on each line (not inside an HTML comment): ```markdown ## Required links - [x] Forge link (github.com, gitlab.com, etc): https://github.com/<org>/<project> - [x] pkg.go.dev: https://pkg.go.dev/github.com/<org>/<project> - [ ] goreportcard.com — service sunset; see the requirement note above - [x] Coverage service link (codecov, coveralls, etc.): https://app.codecov.io/gh/<org>/<project> ## Pre-submission checklist - [x] I have read the Contribution Guidelines - [x] I have read the Quality Standards ## Repository requirements - [x] `go.mod` file and SemVer release (vX.Y.Z) - [x] Open source license (<LICENSE>) - [x] pkg.go.dev link in docs - [ ] goreportcard link — service sunset; see the requirement note above - [x] Coverage service link - [x] Continuous integration (GitHub Actions) ## Pull Request content - [x] Adds only one package. - [x] Added in alphabetical order. - [x] Link text is the exact project name. - [x] Description is clear, concise, non-promotional, and ends with a period. - [x] The link in README.md matches the forge link above. ``` Keep it concise — do not add a marketing "About" section; the non-promotional check scans the whole body. Fetch the live template first in case it changed (the raw media type avoids base64, which is non-portable across GNU/BSD): `gh api repos/avelino/awesome-go/contents/.github/PULL_REQUEST_TEMPLATE.md -H 'Accept: application/vnd.github.raw'`. ## README entry One bullet, in the target category, **alphabetical by visible link text** (case-insensitive), link text = **exact project name**, description **non-promotional and ending with a period**: ```markdown - [<project>](https://github.com/<org>/<project>) - Concise factual description ending with a period. ``` The non-promotional linter rejects superlatives ("blazing fast", "powerful", "production-grade", "world-class"). State capabilities, not adjectives. **Same-name collisions:** a different repo with the same project name may already be listed (e.g. two `go-cron`s). This is allowed — the duplicate check is URL-based, and the list already carries cases like two `scheduler` entries. Handle it by: - Placing your entry **alphabetically adjacent** to the existing one. - Optionally using **`org/project`** as link text to disambiguate — precedented in-list (e.g. `tickstem/cron`) and a likely reviewer request. This still passes the blocking entry-format regex; it only trips the *non-blocking* "link text matches repo name" warning, so it won't fail CI. - **Not** bundling a removal of a stale/abandoned same-name entry into your add PR — the one-item-per-PR rule forbids it; file removals separately (and consider whether it's worth the friction). ## Process ```bash # 1. Fork (no clone) + shallow-clone your fork — the repo history is large gh repo fork avelino/awesome-go --clone=false git clone --depth 1 --single-branch https://github.com/<you>/awesome-go.git cd awesome-go && git checkout -b add-<project> # 2. Edit README.md: add the single bullet in the right category, alphabetically. # Touch ONLY that one line — any unrelated diff hunk gets the PR rejected. # 3. Verify the diff is exactly one insertion git diff --stat # expect: README.md | 1 + # 4. Clean commit (no attribution/co-author trailers), push git commit -am "Add <project> to <Category>" git push -u origin add-<project> # 5. Save the PR body (the template under "PR body" above) to pr-body.md, # then open the PR gh pr create --repo avelino/awesome-go --base main \ --head <you>:add-<project> \ --title "Add <project> to <Category>" --body-file pr-body.md ``` ## After opening - Wait for the bot's sticky report; fix any **blocking** red check and push to the same branch. - "Detect PR type" should pass as a *package* PR; "Skip quality checks (non-package PR)" showing `skipping` is normal. - Ignore the legacy `Running test` failure unless your project/category appears in its log (see [above](#automated-checks)). - awesome-go has a large backlog; merges can take weeks. Don't open duplicate PRs or ping aggressively. -
branch-protection.md 6.4 KB
# Branch Protection Standard for Go Repositories The default-branch gate for Netresearch Go repos, measured as the estate watermark (highest standard per axis across the Go, TYPO3, and skill repos) and applied to all seven Go repos on 2026-08-18. Use it as the minimum for every new Go repo. ## Use rulesets, never classic branch protection Classic protection's review settings are **invisible to the `repos/{r}/rules/branches/{branch}` API** — tooling that reasons from the rules endpoint (pr-status, preflight scripts) cannot see them, so a gate like `require_last_push_approval` surfaces only after everything else is green. Rulesets are the single queryable source of truth. ## Three rulesets, not one Bypass actors attach to a **whole ruleset**. Splitting isolates the deliberate bypass (deps automation on the approval rule) from the rules that must never be bypassed (signatures, checks): | Ruleset | Rules | Bypass | |---|---|---| | `go-baseline` | `deletion`, `non_fast_forward`, `required_status_checks` (strict; the ten `go-check / *` contexts + `drift / Template drift`), `code_scanning` (CodeQL, high_or_higher/errors) | none | | `require-signed-commits` | `required_signatures` | none | | `go-pull-request` | `pull_request`: 1 approval, dismiss stale on push, thread resolution required, merge-commit only, **no** last-push approval, **no** code-owner review | Repo admins, Renovate (app 2740), Dependabot (app 29110) — all `bypass_mode: pull_request` | A separate `Copilot review for default branch` ruleset (`copilot_code_review`, `review_on_push: false`) usually already exists; keep it, do not duplicate the rule. Repos without the shared `go-check` template scale `go-baseline`'s check contexts to the jobs that actually run — a required context that no workflow emits blocks every merge forever. ## Rules with a rationale - **`require_last_push_approval` stays off.** With a bot-maintainer flow (agent pushes review follow-ups, then reviews), the bot is the last pusher and its approval is discounted — structurally unresolvable without a second human (deadlocked go-cron#399). Approver ≠ author, stale-review dismissal, and thread resolution give the protection without the deadlock. - **CI must be in `required_status_checks`.** go-cron required only the template-drift check for months: a red test suite did not block merge. - **`require_code_owner_reviews` only with a CODEOWNERS that resolves.** GitHub ignores unresolvable owners; a CODEOWNERS pointing at a nonexistent team makes the flag vacuous while reading as enforcement (go-cron#400). - **Bypass mode `pull_request`, never `always`.** `always` lets the actor push past non-fast-forward and checks; `pull_request` only relaxes the approval rule for the actor's own PRs (Renovate/Dependabot auto-merge). - **Merge-commit only** (`allowed_merge_methods: ["merge"]`) — atomic commits, preserved signatures. ## Applying to a repo ```bash # Payload shape: {name, target: "branch", enforcement: "active", # conditions: {ref_name: {include: ["~DEFAULT_BRANCH"], exclude: []}}, # bypass_actors: [...], rules: [...]} gh api repos/OWNER/REPO/rulesets -X POST --input go-baseline.json gh api repos/OWNER/REPO/rulesets -X POST --input require-signed-commits.json gh api repos/OWNER/REPO/rulesets -X POST --input go-pull-request.json # Read the EFFECTIVE rules back — a 2xx is not proof: gh api repos/OWNER/REPO/rules/branches/main --jq '[.[].type] | sort' # Then retire classic protection: gh api repos/OWNER/REPO/branches/main/protection -X DELETE ``` Copy a live ruleset as the template instead of hand-writing JSON: `gh api repos/netresearch/go-cron/rulesets --jq '.[]|{id,name}'`, then `GET` the id and adjust. Org-level rulesets would replace the per-repo copies, but require a paid GitHub plan (the free org tier returns 403). ## CODEOWNERS and reviewer routing The standard file (copied verbatim across the Go repos): ```text * @CybotTM @netresearch/netresearch /.github/workflows/ @CybotTM @netresearch/sec /SECURITY.md @CybotTM @netresearch/sec ``` No per-repo maintainer teams exist — use the org-wide teams. Two vacuity traps, both observed live: a CODEOWNERS entry naming a **nonexistent team** is silently ignored, and so is one naming a team **without read access to the repo** (the `netresearch` team had access to none of the seven Go repos, so the default-owner line never routed anywhere). After editing CODEOWNERS, verify access: `gh api orgs/ORG/teams/TEAM/repos/ORG/REPO` — grant with `-X PUT -f permission=pull`. The 1-approval rule is satisfied on solo-maintained repos by the org-shared `pr-quality.yml` caller (`auto-approve-maintainers: true`). A repo without it has **no approval source**: nothing can merge, and ruleset bypass does not apply to a plain `gh pr merge` (only to the explicit admin-bypass path, which is banned). Ship the workflow before or with the ruleset. The auto-approve job runs only for **non-draft, same-repository** PRs whose author passes the authorization test (`author_association` OWNER, MEMBER or COLLABORATOR by default; write permission with `auto-approve-require-write: true`). On a draft the job is `skipped` and the review decision stays `REVIEW_REQUIRED`, and the PR author cannot approve their own PR (the API answers HTTP 422). For an author who passes the test, neither is a reason to ask for a human approver: mark the PR ready (`gh pr ready`) and read the gate again. That only starts a run when the caller's `pull_request` trigger lists `ready_for_review` — the default types do not: ```yaml on: pull_request: branches: [main] types: [opened, synchronize, reopened, ready_for_review] ``` A **fork** PR, or one whose author fails the authorization test, has no automatic approval source; there a maintainer's review is the approval. ## Two rollout traps - **A required check must always report.** Deriving required contexts from a push run on main is not enough: a workflow whose `pull_request` trigger has `paths:` filters never starts on non-matching PRs, and the required check hangs "expected" forever. Remove path filters from the PR trigger of any workflow whose job is a required context. - **Repo merge methods must include the ruleset's allowed method.** A repo with `allow_merge_commit: false` plus a ruleset allowing only `merge` can merge nothing at all. Check `gh api repos/OWNER/REPO --jq '{allow_merge_commit, allow_rebase_merge, allow_squash_merge}'` when applying the ruleset. -
contracts-and-invariants.md 8.2 KB
# Contracts & Invariants Encode preconditions, postconditions, and invariants as runtime checks in the code path. Treat them as the bridge between a spec sentence and the tests that verify it. ## When to Use Use contracts where invariants are crystalline and a violation is a bug, not user error: - State machines (workflows, consensus, session lifecycle, leases) - Protocols (Paxos/Raft, two-phase commit, request/response correlation) - Concurrency primitives (locks, queues, pools, supervised goroutines) - Money, quantities, identifiers (never-negative, monotonic, bounded ranges) - Data migrations (row count preserved, no orphaned references) - Authorization boundaries (see security-audit-skill cross-link) ## When NOT to Use - Plain CRUD glue, HTTP handler plumbing, config parsing — use input validation, not contracts - Anything driven by external input — that is validation (return error), not an invariant (panic) - Frontend/template code, doc generation, scripts A useful test: would a violation indicate the program is in an impossible state? Yes → contract. No → validation. ## Contract Types | Kind | Where | If violated | |------|-------|-------------| | Precondition | First lines of a function | Caller bug — panic | | Postcondition | Just before `return` | Implementation bug — panic | | Invariant | At every public entry/exit of a stateful type | Either — panic | External-input checks are separate: return a typed error, do not panic. ## Go Idioms ### Doc convention Document contracts inline so reviewers (and AI) see intent next to code: ```go // Withdraw debits amount from the account. // // Contract: // pre: amount > 0 // caller bug if violated → panic // validation: amount <= balance // caller's mistake → typed error // post: balance == old(balance) - amount // inv: balance >= 0 func (a *Account) Withdraw(amount Money) error { invariant.Assertf(amount > 0, "precondition: amount > 0, got %v", amount) if amount > a.balance { return ErrInsufficientFunds // validation, not a contract } before := a.balance a.balance -= amount invariant.Assertf(a.balance == before-amount, "postcondition violated") invariant.Assertf(a.balance >= 0, "invariant: balance >= 0") return nil } ``` ### Assertion helper Keep one helper. Do not scatter ad-hoc `panic`s: ```go // Package internal/invariant package invariant import "fmt" // Assert panics with msg when cond is false. // Use only for impossible states. Use returned errors for user input. func Assert(cond bool, msg string) { if !cond { panic("invariant: " + msg) } } func Assertf(cond bool, format string, args ...any) { if !cond { panic("invariant: " + fmt.Sprintf(format, args...)) } } ``` ### Strip in release (optional) For hot paths where the check itself is expensive, gate with a build tag: ```go //go:build assertions package invariant func Assert(cond bool, msg string) { if !cond { panic("invariant: " + msg) } } func Assertf(cond bool, format string, args ...any) { if !cond { panic("invariant: " + fmt.Sprintf(format, args...)) } } ``` ```go //go:build !assertions package invariant func Assert(cond bool, msg string) {} func Assertf(cond bool, format string, args ...any) {} ``` Run tests and staging with `-tags=assertions`; ship release builds without. Most code should keep checks always-on — only strip when profiling proves cost. ### Constructors fail fast Establish invariants at construction so methods can rely on them: ```go func NewAccount(initial Money) (*Account, error) { if initial < 0 { return nil, fmt.Errorf("initial balance: %w", ErrNegative) } return &Account{balance: initial}, nil } ``` Validation at the boundary → typed error. Invariant inside → panic. ## Property Tests from Contracts A postcondition is a property. A property test asserts the postcondition holds for many inputs. ### stdlib `testing/quick` (lightweight) ```go func TestWithdraw_PreservesNonNegativeBalance(t *testing.T) { f := func(initial, amount uint32) bool { a, _ := NewAccount(Money(initial)) _ = a.Withdraw(Money(amount)) return a.balance >= 0 } if err := quick.Check(f, &quick.Config{MaxCount: 1000}); err != nil { t.Fatal(err) } } ``` ### `pgregory.net/rapid` (state machines, recommended for protocols) ```go import "pgregory.net/rapid" func TestAccount_Properties(t *testing.T) { rapid.Check(t, func(t *rapid.T) { initial := rapid.Uint32Range(0, 1_000_000).Draw(t, "initial") a, _ := NewAccount(Money(initial)) ops := rapid.SliceOf(rapid.Uint32Range(0, 10_000)).Draw(t, "ops") for _, op := range ops { _ = a.Withdraw(Money(op)) // No explicit assert here: the contract inside Withdraw panics on // any invariant violation, which rapid.Check surfaces with the // failing input sequence. } }) } ``` For protocols, model the state machine and let `rapid` drive transitions. The contract panics inside the implementation will surface any reachable invariant violation. ## An Unreachable Branch Still Has a Direction When a guard splits on a sentinel and the remaining branch cannot be reached today, its behaviour is still a decision — it is the behaviour that ships the day the assumption stops holding. Choose it by which direction is safe then, not by which looks symmetric now. The trigger is a `switch` or `if`/`else` over a library's error values where the library currently returns only one of them: ```go ckie, err := r.Cookie(name) if errors.Is(err, http.ErrNoCookie) { challenge(w) // no cookie: nothing to clear return } if err != nil { clearCookie(w) // unreachable today -- but which way should it fail? challenge(w) return } ``` `(*http.Request).Cookie` returns `ErrNoCookie` and nothing else, so the second branch is dead. It is also pointed the wrong way: the whole purpose of the first branch is *"we could not read a cookie, so do not emit a deletion"*, and the fallback does the opposite. If a future stdlib release ever returns another error there, the dead branch wakes up and re-creates the bug the guard was written to fix — and in this case attacker-influenced, because `readCookies` already degrades to `ErrNoCookie` when the cookie-count cap trips. Prefer collapsing to a single branch when every error means the same thing: ```go ckie, err := r.Cookie(name) if err != nil { challenge(w) // any lookup failure: nothing to clear return } ``` This also removes the `errors.Is` split, which was only ever justified when the two branches genuinely differ. Keep the split — with a deliberate, documented fallback — when a future unknown error really should behave differently from the sentinel. Applies to any guard whose fallback is currently dead: a `default:` in a switch over a closed enum, an `else` after an exhaustive type assertion, a `case ErrNotFound` chain. Write the branch that fails safe, or delete it and say in the code why one branch is enough. ## Common Mistakes | Mistake | Fix | |---------|-----| | Dead fallback branch pointed the unsafe way | Fail safe, or collapse to one branch — see above | | Panicking on user input | Return a typed error; reserve panic for impossible states | | Sprinkling `assert` decoratively in glue code | Gate by domain — state machines, protocols, money, authz | | Postcondition that re-implements the function | Postcondition states the *property*, not the steps | | Asserting against external systems mid-RPC | Network failure ≠ invariant violation; handle as error | | Catching panics from contracts to "keep serving" | Don't. An invariant violation means state is corrupt — let it crash, restart cleanly | ## Cross-References - `references/testing.md` — build tags, resource isolation, race-condition gotchas - `references/fuzz-testing.md` — input-driven discovery (complementary to property tests) - `references/resilience.md` — panic recovery boundaries (only at goroutine roots, never around contracts) - security-audit-skill `references/authentication-patterns.md` — authorization invariants -
cron-scheduling.md 10.2 KB
# Cron Scheduling with go-cron [`github.com/netresearch/go-cron`](https://github.com/netresearch/go-cron) is a maintained fork of `robfig/cron` — the most popular cron library for Go — with bug fixes, runtime schedule updates, per-entry context, resilience middleware, and modern toolchain support. ## Installation ```bash go get github.com/netresearch/go-cron ``` ```go import cron "github.com/netresearch/go-cron" ``` Drop-in replacement for `robfig/cron/v3` — just change the import path. ## Basic Usage ```go c := cron.New() c.AddFunc("0 9 * * *", func() { fmt.Println("Every day at 9am") }) c.AddFunc("@every 5m", func() { fmt.Println("Every 5 minutes") }) c.Start() defer c.Stop() ``` ## Named Jobs and Lookup Assign names and tags for O(1) lookup, update, and removal: ```go id, _ := c.AddFunc("0 9 * * *", dailyReport, cron.WithName("daily-report"), cron.WithTags("reports", "daily"), ) // Lookup by name (O(1)) entry := c.EntryByName("daily-report") // Filter by tag entries := c.EntriesByTag("reports") // Remove by name c.RemoveByName("daily-report") ``` ## Runtime Updates Update schedules and jobs atomically without remove+re-add: ```go // Update schedule only (preserves job, options, and context) c.UpdateScheduleByName("daily-report", cron.Every(5*time.Minute)) // Update both schedule and job atomically (cancels old entry context) c.UpdateEntryJobByName("daily-report", "30 10 * * *", newJob) // Create-or-update in one call id, err := c.UpsertJob("0 9 * * *", myJob, cron.WithName("my-job")) ``` ### Graceful Job Replacement For long-running jobs, wait for the current execution to finish before replacing: ```go c.WaitForJobByName("my-job") // Block until current execution finishes c.UpsertJob(newSpec, newJob, cron.WithName("my-job")) ``` Check if a job is currently running: ```go if c.IsJobRunningByName("my-job") { log.Println("Job is still running, will wait") c.WaitForJobByName("my-job") } ``` ## Per-Entry Context Each entry gets its own `context.Context` derived from the Cron's base context. The context is automatically canceled when the entry is removed or its job is replaced. ```go c.AddJob("@every 1m", cron.FuncJobWithContext(func(ctx context.Context) { select { case <-ctx.Done(): return // Entry removed or job replaced case <-time.After(10 * time.Second): // Work completed } })) ``` ### Context Hierarchy ``` caller's context └─ cron context (canceled by Stop()) └─ entry context (canceled by Remove/UpdateEntry/UpsertJob) ``` `cron.New(cron.WithContext(parentCtx))` derives a child context. `Stop()` cancels the child, not the caller's context. ## Job Wrappers (Middleware) ### Concurrency Wrappers These implement `JobWithContext` and propagate context to inner jobs: ```go // Apply to all jobs via Cron options c := cron.New(cron.WithChain( cron.Recover(logger), // Catch panics cron.SkipIfStillRunning(logger), // Skip if previous still running cron.DelayIfStillRunning(logger), // Queue until previous finishes cron.Timeout(30*time.Second, nil), // Abandon after duration cron.TimeoutWithContext(30*time.Second, nil), // Cancel context after duration cron.Jitter(5*time.Second), // Random delay )) // Apply to specific job job := cron.NewChain( cron.Recover(logger), cron.DelayIfStillRunning(logger), ).Then(myJob) ``` ### Resilience Wrappers These return `FuncJob` and do NOT forward context: ```go // Retry on panic with exponential backoff retryJob := cron.RetryWithBackoff(myJob, cron.RetryConfig{ MaxRetries: 3, InitialDelay: 100 * time.Millisecond, MaxDelay: 30 * time.Second, Multiplier: 2.0, }) // Retry on error return (job must implement ErrorJob) retryJob := cron.RetryOnError(myErrorJob, cron.RetryOnErrorConfig{ MaxRetries: 3, Delay: time.Second, }) // Circuit breaker — stop after consecutive failures cbJob := cron.CircuitBreaker(myJob, cron.CircuitBreakerConfig{ Threshold: 5, ResetTimeout: time.Minute, }) ``` ### ErrorJob Interface For retry-on-error, implement `ErrorJob`: ```go type myJob struct{} func (j *myJob) Run() {} func (j *myJob) RunWithError() error { // Return error to trigger retry return doWork() } ``` Or use the convenience wrapper: ```go cron.FuncErrorJob(func() error { return doWork() }) ``` ## Observability Monitor cron operations with hooks: ```go c := cron.New(cron.WithObservability(cron.ObservabilityHooks{ OnJobStart: func(id cron.EntryID, name string, scheduled time.Time) { jobsStarted.WithLabelValues(name).Inc() }, OnJobComplete: func(id cron.EntryID, name string, dur time.Duration, recovered any) { jobDuration.WithLabelValues(name).Observe(dur.Seconds()) if recovered != nil { jobPanics.WithLabelValues(name).Inc() } }, })) ``` ## Validation Validate cron expressions before scheduling: ```go // Quick validation if err := cron.ValidateSpec("0 9 * * MON-FRI"); err != nil { log.Fatal(err) } // Instance-level (uses configured parser) c := cron.New(cron.WithSeconds()) if err := c.ValidateSpec("0 30 * * * *"); err != nil { log.Fatal(err) } // Detailed analysis result := cron.AnalyzeSpec("0 9 * * MON-FRI") fmt.Println("Next run:", result.NextRun) fmt.Println("Fields:", result.Fields) ``` ## Custom Parser Construction (Bitmask Options) `cron.New()`'s functional options (`cron.WithSeconds()`, etc.) cover the common cases. For a standalone parser with a specific field set, go-cron exposes a `ParseOption` bitmask (inherited from `robfig/cron/v3`, since go-cron is a drop-in replacement): ```go parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor) schedule, err := parser.Parse("0 9 * * MON-FRI") ``` Available flags: `Second`, `SecondOptional`, `Minute`, `Hour`, `Dom`, `Month`, `Dow`, `DowOptional`, `Descriptor` (allows `@hourly`, `@daily`), `Year`, `Hash` (Jenkins-style `H` expressions). Combine with bitwise OR; define presets as named constants for reuse: ```go const ( StandardParser = cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor ExtendedParser = cron.Second | StandardParser ) ``` ## Missed Job Catch-Up Handle jobs missed during downtime: ```go lastRun := loadFromDatabase("daily-report") c.AddFunc("0 9 * * *", dailyReport, cron.WithPrev(lastRun), cron.WithMissedPolicy(cron.MissedRunOnce), cron.WithMissedGracePeriod(2*time.Hour), ) ``` Policies: `MissedSkip` (default), `MissedRunOnce`, `MissedRunAll`. ## Graceful Shutdown ```go // Block until all running jobs finish c.StopAndWait() // With timeout if !c.StopWithTimeout(30 * time.Second) { log.Println("Warning: some jobs did not complete within 30s") } ``` ## Testing with FakeClock go-cron includes a built-in `FakeClock` for deterministic testing without real time waits: ```go fakeClock := cron.NewFakeClock(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)) c := cron.New(cron.WithClock(fakeClock)) executed := make(chan struct{}, 1) c.AddFunc("0 * * * *", func() { executed <- struct{}{} }) c.Start() defer c.Stop() fakeClock.BlockUntil(1) // Wait for scheduler to register timer fakeClock.Advance(time.Hour) // Trigger the job instantly select { case <-executed: // Job ran successfully case <-time.After(time.Second): t.Fatal("job did not execute") } ``` No wrapper needed — `cron.NewFakeClock` returns a type that satisfies the `cron.Clock` interface directly. ## Common Options ```go c := cron.New( cron.WithSeconds(), // Enable seconds field cron.WithLocation(time.UTC), // Default timezone cron.WithContext(parentCtx), // Parent context cron.WithCapacity(100), // Pre-allocate internals cron.WithMaxEntries(1000), // Limit max entries cron.WithRunImmediately(), // Run @every jobs on Start cron.WithLogger(cron.NewSlogLogger(slog.Default())), cron.WithChain(cron.Recover(logger)), // Default wrappers cron.WithObservability(hooks), // Metrics hooks ) ``` ## Patterns from Production Usage ### Dynamic Job Management (weaviate pattern) ```go func (m *Manager) RescheduleJob(name, newSpec string, newJob cron.Job) error { // Atomic create-or-update — no manual "check then add/update" needed _, err := m.cron.UpsertJob(newSpec, newJob, cron.WithName(name)) return err } ``` ### Graceful Replacement of Long-Running Jobs ```go func (m *Manager) ReplaceJob(name, spec string, job cron.Job) error { // Wait for current execution to finish before replacing m.cron.WaitForJobByName(name) _, err := m.cron.UpsertJob(spec, job, cron.WithName(name)) return err } ``` ### Service Integration with Shutdown ```go func main() { ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer cancel() c := cron.New(cron.WithContext(ctx)) c.AddFunc("@every 5m", healthCheck, cron.WithName("health-check")) c.AddFunc("0 * * * *", syncData, cron.WithName("hourly-sync")) c.Start() <-ctx.Done() if !c.StopWithTimeout(30 * time.Second) { log.Println("Warning: jobs did not complete within 30s") } } ``` ### Context-Aware Long-Running Job ```go c.AddJob("@every 1m", cron.FuncJobWithContext(func(ctx context.Context) { ticker := time.NewTicker(time.Second) defer ticker.Stop() for { select { case <-ctx.Done(): log.Println("Job canceled, cleaning up") return case <-ticker.C: if err := processNextItem(ctx); err != nil { log.Printf("Error: %v", err) } } } }), cron.WithName("item-processor")) ``` ## Migration from robfig/cron Drop-in replacement — just change the import: ```go // Before import "github.com/robfig/cron/v3" // After import cron "github.com/netresearch/go-cron" ``` Key behavior differences: - **DOM/DOW matching**: Uses AND logic (both must match) instead of OR - **DST spring-forward**: Jobs in skipped hour run immediately instead of being silently skipped - **Chain execution**: `Entry.Run()` properly invokes chain wrappers See the [migration guide](https://github.com/netresearch/go-cron/blob/main/docs/MIGRATION.md) for full details. -
dependencies.md 13.3 KB
# Dependency Upgrades Upgrading Go dependencies has three traps: the obvious command does less than it looks, majors are invisible to it, and "everything is updated" is almost never a claim you can make honestly. ## `go get -u ./...` is not a full update `go get -u ./...` upgrades only what is needed to **build the packages matched by the pattern**. Modules elsewhere in the graph are untouched. `go get -u all` covers the whole module graph. ```bash go get -u ./... # build path only go get -u all # whole module graph go mod tidy ``` **Real case:** after `go get -u ./...` reported ~26 upgrades and the tree built green, `go list -m -u all` still listed **44 modules with newer versions**, including `terraform-json` 0.27.2 → 0.28.0 and `terraform-exec` 0.25.1 → 0.25.2 — both in the build. Only `go get -u all` moved them. The claim "all dependencies upgraded" was wrong until challenged. Always confirm with the tool, not the transcript: ```bash go list -m -u all | grep '\[' # any module with an available update prints [newer] ``` ## `-u` never crosses a major version Major versions are distinct module paths (`/v2`, `/v3`), so `-u` cannot reach them — a v2 release is invisible to a v1 module. Check explicitly before claiming a dependency is current: ```bash for m in $(go list -m -f '{{if not .Indirect}}{{.Path}}{{end}}' all); do if [[ "$m" =~ /v([0-9]+)$ ]]; then cur="${BASH_REMATCH[1]}"; base="${m%/v*}"; else cur=1; base="$m"; fi go list -m "$base/v$((cur + 1))@latest" >/dev/null 2>&1 && echo "MAJOR AVAILABLE: $m -> $base/v$((cur + 1))" done ``` A major bump is an import rewrite, not a version bump — scope it as its own change. ## Scope the claim to what is actually in the build After `go get -u all`, `go list -m -u all` will often *still* list modules with newer versions. That is usually correct and not a gap: their versions are selected by MVS from your dependencies' own requirements, and many are test-deps-of-deps that never link into your binary. Forcing them means spurious `require` entries for code you do not ship. Separate "outdated" from "outdated **and in the build**" before reporting: ```bash # outdated AND linked into the binary — the real list comm -12 \ <(go list -deps -f '{{with .Module}}{{.Path}}{{end}}' ./... | grep -v '^$' | sort -u) \ <(go list -m -u all | grep '\[' | awk '{print $1}' | sort) ``` Empty output means every module you actually ship is current; the remainder is graph noise. That is a defensible claim. "All dependencies are latest" usually is not. ## Verify the final tree, not an intermediate one `go get -u` raises the `go` directive when an upgraded dependency demands it (observed: 1.24.0 → 1.25.8). If a workflow pins Go from `.go-version`, that file and `go.mod` can silently disagree. ```bash go build ./... && go vet ./... go mod verify go mod tidy && git diff --exit-code -- go.mod go.sum # tidy drift == CI failure ``` A `depscheck`-style target runs `go mod tidy` then `git diff --exit-code` — it fails on **uncommitted** go.mod/go.sum, so run it after committing, or its red is your own working tree rather than real drift. ## Dependency changes need the test job to actually run A CI `paths:` filter listing only source globs (`'**.go'`) does not match `go.mod`, so **every dependency PR skips the test job** — including Dependabot's. Include the manifest: ```yaml paths: - '**.go' - 'go.mod' - 'go.sum' - '.go-version' ``` ## Replacing an archived dependency: prove the swap, do not assume it An archived module is a reason to move, but the replacement is a behaviour change until measured. Two things to establish before the commit message claims a drop-in. **Enumerate every symbol and every struct tag you use**, not just the one call you remember. A fork can keep a signature and change how it reads a tag. The cheap evidence is a differential probe: a throwaway module importing both libraries, feeding the same inputs through each, comparing results *and* whether an error was returned. Copy the real structs with their real tags — reconstructed fixtures test the reconstruction. Control-test the probe by injecting one difference and checking it reports it; a probe that cannot fail is not evidence. **Read the advisory rather than a summary.** The advisory in this area may be against the module you are migrating *to*: `go-viper/mapstructure/v2` carries CVE-2025-11065 for `<= 2.3.0`, while the archived `mitchellh/mapstructure` has no advisory at all. Take the package, the range and the first patched version from `gh api /advisories?cve_id=<CVE>` or the OSV record. Error *text* is part of the contract when it reaches a user. The same mapstructure change stops quoting the offending value back in a decode error, so any message a provider or CLI surfaces changes wording without changing behaviour. Grep for tests asserting on it, and put it in the release notes. "No advisory" is not "unaffected". The archived module leaks the same value into the same message; it has no advisory because nobody is filing them for it. Read an absent advisory as an absent maintainer. ## Moving to a standard-library replacement: check the acceptance set Go 1.27 ships `uuid`, which makes `github.com/hashicorp/go-uuid` and friends removable. The generation side is a clean swap; the parsing side is not. `hashicorp/go-uuid`'s `ParseUUID` accepts exactly the canonical 36-character hyphenated form. `uuid.Parse` also accepts the URN form (`urn:uuid:...`), the unhyphenated 32-character form, and the brace-wrapped form. Swapping it into a validator therefore *widens* what that validator accepts, silently — and for a value that is echoed back canonicalised by the system it addresses, a widened validator trades an error at validation time for a diff that never settles. Keep the old acceptance set explicitly: ```go func parseGUID(s string) error { if len(s) != 36 { return fmt.Errorf("uuid string is wrong length") } _, err := uuid.Parse(s) return err } ``` The length check is what makes it strict; the three wider spellings are 45, 32 and 38 bytes. Pin it with a table test that lists those three as rejected, and confirm the test fails when the length check is removed — otherwise the guard is asserting nothing. Two more things the swap changes: - **Rejection wording.** `go-uuid` returned one of four messages depending on which check failed — `uuid string is wrong length`, `uuid is improperly formatted` for a misplaced separator, a raw `encoding/hex` error, or `decoded hex is the wrong length`. The standard library returns `invalid uuid` for all of them. Anywhere that interpolates the parse error into a user-facing message now reads differently, and a test matching on one of the four stops matching. - **The generated value's shape.** `go-uuid`'s `GenerateUUID` formatted sixteen random bytes *without* setting the version and variant bits, so it did not produce a v4 by construction — about one output in 64 was a valid v4 by chance, which is why "never" is the wrong word and a sampling check is the wrong test. `uuid.New()` sets both. If the value is stored, say so. One thing that is *not* a hazard here, because it is easy to assume it is: `uuid.UUID` is a `[16]byte`, but `String` has a **value** receiver, so a `%s` verb reaches it for both a value and a pointer. There is no plain `%s` spelling that prints raw bytes. Getting them requires deliberately leaving the type — `u[:]`, `[16]byte(u)`, `%x` — which is not something a refactor does by accident. Assert the rendering only where production code adds formatting of its own, such as composing the value into a larger identifier. Assert it by calling that code, never by rebuilding its expression in the test. ## The `go` directive is not only a floor — it selects runtime behaviour `go.mod` carries two version lines and they do different jobs: ``` go 1.26 # language version AND compatibility baseline toolchain go1.27.1 # which toolchain to fetch and build with ``` The `toolchain` line decides what compiles. The `go` line decides what the result *behaves* like: Go compiles with that version's compatibility defaults and bakes them into the binary. So a module built by go1.27.1 while declaring `go 1.26` ships 1.26 semantics for everything 1.27 changed. **Only the main module's directive counts** (or the workspace's `go.work`). A dependency's `go` line does not reach the importing binary's defaults — verified by building an app at `go 1.26` against a dependency at `go 1.24`, then raising only the app: the dependency's version never showed up either way. Read it back rather than reasoning about it: ```bash go build -o /tmp/probe . && go version -m /tmp/probe | grep -E 'DefaultGODEBUG|^/' # /tmp/probe: go1.27.1 # build DefaultGODEBUG=tracebacklabels=0,x509sslcertoverrideplatform=0 # Same answer without building, straight from the main package: go list -f '{{.DefaultGODEBUG}}' . # tracebacklabels=0,x509sslcertoverrideplatform=0 ``` Both were measured on go1.27.1. `go version -m` reads the *binary*, `go list` reads the *package* — neither reports the `go` directive itself, which is `go mod edit -json | jq -r '.Go'` if that is what you want. Those two settings are 1.27 behaviour changes, pinned back. Raise the main module's `go` directive to 1.27 and the `DefaultGODEBUG` line disappears entirely — same toolchain, same tree, only the directive differs. That two-build diff is the way to show what a floor actually costs, and it takes one minute. One documented exception, from [go.dev/doc/godebug](https://go.dev/doc/godebug): *"GODEBUGs introduced for security releases will have the new behavior apply to all versions."* So a low floor pins back ordinary behaviour changes, not those. ### Deciding the floor The floor is a promise to whoever compiles the module, so ask who that is before keeping it low: - **Library** — importers inherit it through MVS. Keep the floor low deliberately; raising it forces every consumer up. Check who they are: `https://pkg.go.dev/<module>?tab=importedby` says *"No known importers for this package!"* when there are none. - **Application** — users consume binaries and images, not the module. If the Dockerfile packages a prebuilt binary rather than compiling, and CI resolves its toolchain from `go.mod` (`setup-go` with `go-version-file`), then the floor buys nothing and costs the pinned-back behaviour above. A repository that inherited its floor from a sibling library without inheriting the reason is the common case worth checking. ### Building it — a floor nothing compiles is a promise, not a guarantee Which of the two lines `actions/setup-go` resolves from `go-version-file` depends on the major you pin, so do not carry an answer between repositories. Measured on **v7.0.0**: a module declaring `go 1.26.0` alongside `toolchain go1.27.1` was built by 1.27.1 in every job, and the 1.26 floor it advertises to importers was compiled by nothing. Older majors document reading the `go` directive — the `@v5` examples elsewhere in this skill (`makefile.md`, `mutation-testing.md`) predate the change and are not covered by what follows. Because the answer is version-dependent, read it off a run rather than reasoning about it — the jobs say which they got: ```bash gh run view <id> --log | grep -m1 "Setup go version spec" # Setup go version spec 1.27.1 ← while go.mod says go 1.26.0 ``` The fix is a matrix that pins the toolchain per leg. **`GOTOOLCHAIN: local` on every step is the load-bearing part, not boilerplate:** without it the older runner reads the same `toolchain` line and upgrades itself, so the matrix reports a pass for a version it never measured. ```yaml go-compat: strategy: fail-fast: false matrix: go: ['1.26.8', '1.27.1'] # oldest entry = latest patch of the floor's line steps: - uses: actions/setup-go@<sha> with: go-version: ${{ matrix.go }} check-latest: false - env: { GOTOOLCHAIN: local } run: go version # prints what you actually got — keep this step - env: { GOTOOLCHAIN: local } run: go build ./... && go vet ./... && go test -short -race ./... ``` Keep the `go version` step: it is the control that distinguishes a real 1.26 leg from a 1.27 leg wearing a 1.26 label, and it costs one line. Note what that matrix does and does not prove. `go 1.26.0` names an exact minimum, and 1.26.8 satisfies it without exercising it — the leg tests the **supported line**, not the floor. Pin the floor itself (`'1.26.0'`) when the question is whether the declared minimum still compiles; use the latest patch when the question is whether the line still works. They are different claims and only one of them is usually what you want. Locally the same pin applies — `GOTOOLCHAIN=local ~/sdk/go1.26.8/bin/go test ./...`. Without it the SDK binary silently hands off to the newer toolchain, which is the same false green one directory down. ### Raising it `go.mod` is not the only surface. Sweep **without an extension filter** — the files that *enforce* the version are often dotfiles a `--include='*.md'` pattern cannot match: ```bash grep -rn '1\.26' . --exclude-dir=.git --exclude=CHANGELOG.md # go.mod, docs/DEVELOPMENT.md, CONTRIBUTING.md … and .envrc's REQUIRED_VERSION ``` The README's `img.shields.io/github/go-mod/go-version` badge reads the `go` directive, so it follows on its own — and reports the floor, not the toolchain, which is why a repo building with 1.27 can advertise 1.26 and look wrong. -
docker.md 11.1 KB
# Docker Integration Patterns in Go ## Optimized Docker Client ### Client with Connection Pooling ```go package core import ( "context" "sync" docker "github.com/fsouza/go-dockerclient" ) type OptimizedDockerClient struct { client *docker.Client bufferPool *sync.Pool mu sync.RWMutex endpoint string } func NewOptimizedDockerClient(endpoint string) (*OptimizedDockerClient, error) { if endpoint == "" { endpoint = "unix:///var/run/docker.sock" } client, err := docker.NewClient(endpoint) if err != nil { return nil, fmt.Errorf("failed to create Docker client: %w", err) } return &OptimizedDockerClient{ client: client, endpoint: endpoint, bufferPool: &sync.Pool{ New: func() any { return NewCircularBuffer(64 * 1024) // 64KB buffers }, }, }, nil } func NewOptimizedDockerClientFromEnv() (*OptimizedDockerClient, error) { client, err := docker.NewClientFromEnv() if err != nil { return nil, err } return &OptimizedDockerClient{ client: client, bufferPool: &sync.Pool{ New: func() any { return NewCircularBuffer(64 * 1024) }, }, }, nil } func (c *OptimizedDockerClient) Close() error { // fsouza/go-dockerclient doesn't require explicit close // but we can clean up the buffer pool return nil } ``` ## Buffer Pooling ### Circular Buffer Implementation ```go type CircularBuffer struct { data []byte size int head int tail int count int mu sync.Mutex } func NewCircularBuffer(size int) *CircularBuffer { return &CircularBuffer{ data: make([]byte, size), size: size, } } func (b *CircularBuffer) Write(p []byte) (int, error) { b.mu.Lock() defer b.mu.Unlock() n := len(p) if n > b.size { // Only keep the last 'size' bytes p = p[n-b.size:] n = b.size } for _, byte := range p { b.data[b.tail] = byte b.tail = (b.tail + 1) % b.size if b.count < b.size { b.count++ } else { b.head = (b.head + 1) % b.size } } return n, nil } func (b *CircularBuffer) String() string { b.mu.Lock() defer b.mu.Unlock() if b.count == 0 { return "" } result := make([]byte, b.count) if b.head < b.tail { copy(result, b.data[b.head:b.tail]) } else { n := copy(result, b.data[b.head:]) copy(result[n:], b.data[:b.tail]) } return string(result) } func (b *CircularBuffer) Reset() { b.mu.Lock() defer b.mu.Unlock() b.head = 0 b.tail = 0 b.count = 0 } func (b *CircularBuffer) Len() int { b.mu.Lock() defer b.mu.Unlock() return b.count } ``` ### Using Buffer Pool ```go func (c *OptimizedDockerClient) ExecInContainer(ctx context.Context, containerID string, cmd []string) (string, string, error) { // Get buffers from pool stdoutBuf := c.bufferPool.Get().(*CircularBuffer) stderrBuf := c.bufferPool.Get().(*CircularBuffer) defer func() { stdoutBuf.Reset() stderrBuf.Reset() c.bufferPool.Put(stdoutBuf) c.bufferPool.Put(stderrBuf) }() // Create exec instance exec, err := c.client.CreateExec(docker.CreateExecOptions{ Container: containerID, Cmd: cmd, AttachStdout: true, AttachStderr: true, Context: ctx, }) if err != nil { return "", "", fmt.Errorf("failed to create exec: %w", err) } // Start exec and capture output err = c.client.StartExec(exec.ID, docker.StartExecOptions{ OutputStream: stdoutBuf, ErrorStream: stderrBuf, Context: ctx, }) if err != nil { return "", "", fmt.Errorf("failed to start exec: %w", err) } // Check exec exit code inspect, err := c.client.InspectExec(exec.ID) if err != nil { return stdoutBuf.String(), stderrBuf.String(), fmt.Errorf("failed to inspect exec: %w", err) } if inspect.ExitCode != 0 { return stdoutBuf.String(), stderrBuf.String(), fmt.Errorf("command exited with code %d", inspect.ExitCode) } return stdoutBuf.String(), stderrBuf.String(), nil } ``` ## Container Operations ### Create and Run Container ```go func (c *OptimizedDockerClient) RunContainer(ctx context.Context, image string, cmd []string, env map[string]string) (string, error) { // Convert env map to slice envSlice := make([]string, 0, len(env)) for k, v := range env { envSlice = append(envSlice, fmt.Sprintf("%s=%s", k, v)) } // Create container container, err := c.client.CreateContainer(docker.CreateContainerOptions{ Config: &docker.Config{ Image: image, Cmd: cmd, Env: envSlice, }, HostConfig: &docker.HostConfig{ AutoRemove: true, }, Context: ctx, }) if err != nil { return "", fmt.Errorf("failed to create container: %w", err) } // Start container if err := c.client.StartContainer(container.ID, nil); err != nil { // Cleanup on error c.client.RemoveContainer(docker.RemoveContainerOptions{ ID: container.ID, Force: true, }) return "", fmt.Errorf("failed to start container: %w", err) } return container.ID, nil } func (c *OptimizedDockerClient) WaitContainer(ctx context.Context, containerID string) (int, error) { exitCode, err := c.client.WaitContainer(containerID) if err != nil { return -1, fmt.Errorf("failed to wait for container: %w", err) } return exitCode, nil } func (c *OptimizedDockerClient) RemoveContainer(ctx context.Context, containerID string, force bool) error { return c.client.RemoveContainer(docker.RemoveContainerOptions{ ID: containerID, Force: force, RemoveVolumes: true, Context: ctx, }) } ``` ### Container Monitoring ```go type ContainerStats struct { CPUPercent float64 MemoryUsage uint64 MemoryLimit uint64 MemoryPercent float64 NetworkRx uint64 NetworkTx uint64 } func (c *OptimizedDockerClient) GetContainerStats(ctx context.Context, containerID string) (*ContainerStats, error) { statsCh := make(chan *docker.Stats) errCh := make(chan error) go func() { err := c.client.Stats(docker.StatsOptions{ ID: containerID, Stats: statsCh, Stream: false, Context: ctx, }) errCh <- err }() select { case stats := <-statsCh: cpuDelta := float64(stats.CPUStats.CPUUsage.TotalUsage - stats.PreCPUStats.CPUUsage.TotalUsage) systemDelta := float64(stats.CPUStats.SystemCPUUsage - stats.PreCPUStats.SystemCPUUsage) cpuPercent := 0.0 if systemDelta > 0 { cpuPercent = (cpuDelta / systemDelta) * float64(len(stats.CPUStats.CPUUsage.PercpuUsage)) * 100 } return &ContainerStats{ CPUPercent: cpuPercent, MemoryUsage: stats.MemoryStats.Usage, MemoryLimit: stats.MemoryStats.Limit, MemoryPercent: float64(stats.MemoryStats.Usage) / float64(stats.MemoryStats.Limit) * 100, NetworkRx: stats.Network.RxBytes, NetworkTx: stats.Network.TxBytes, }, nil case err := <-errCh: return nil, err case <-ctx.Done(): return nil, ctx.Err() } } ``` ## Docker Events ### Event Listener ```go type EventHandler func(event *docker.APIEvents) func (c *OptimizedDockerClient) ListenEvents(ctx context.Context, handler EventHandler) error { listener := make(chan *docker.APIEvents) err := c.client.AddEventListener(listener) if err != nil { return fmt.Errorf("failed to add event listener: %w", err) } defer c.client.RemoveEventListener(listener) for { select { case event := <-listener: if event == nil { return nil } handler(event) case <-ctx.Done(): return ctx.Err() } } } // Usage: React to container events func handleDockerEvent(event *docker.APIEvents) { switch event.Status { case "start": log.WithField("container", event.ID).Info("Container started") case "die": log.WithField("container", event.ID).Info("Container died") case "destroy": log.WithField("container", event.ID).Info("Container destroyed") } } ``` ## Docker Labels for Configuration ### Reading Labels ```go type JobFromLabels struct { Type string Name string Schedule string Command []string Container string } func (c *OptimizedDockerClient) GetJobsFromLabels(ctx context.Context) ([]JobFromLabels, error) { containers, err := c.client.ListContainers(docker.ListContainersOptions{ Context: ctx, }) if err != nil { return nil, err } var jobs []JobFromLabels for _, container := range containers { // Look for labels like: ofelia.job-exec.job-name.schedule for key, value := range container.Labels { if !strings.HasPrefix(key, "ofelia.") { continue } parts := strings.Split(key, ".") if len(parts) < 4 { continue } jobType := parts[1] // job-exec, job-run, etc. jobName := parts[2] // user-defined name param := parts[3] // schedule, command, etc. // Build job config from labels job := findOrCreateJob(jobs, jobName, jobType) switch param { case "schedule": job.Schedule = value case "command": job.Command = strings.Split(value, " ") case "container": job.Container = value } } } return jobs, nil } ``` ## Health Check ```go func (c *OptimizedDockerClient) Ping(ctx context.Context) error { return c.client.PingWithContext(ctx) } func (c *OptimizedDockerClient) IsHealthy(ctx context.Context) bool { return c.Ping(ctx) == nil } // Health check for API endpoint func DockerHealthCheck(client *OptimizedDockerClient) func(context.Context) error { return func(ctx context.Context) error { if err := client.Ping(ctx); err != nil { return fmt.Errorf("docker daemon unavailable: %w", err) } return nil } } ``` ## Image Operations ```go func (c *OptimizedDockerClient) PullImage(ctx context.Context, image string) error { return c.client.PullImage(docker.PullImageOptions{ Repository: image, Context: ctx, }, docker.AuthConfiguration{}) } func (c *OptimizedDockerClient) ImageExists(ctx context.Context, image string) bool { _, err := c.client.InspectImage(image) return err == nil } func (c *OptimizedDockerClient) EnsureImage(ctx context.Context, image string) error { if c.ImageExists(ctx, image) { return nil } return c.PullImage(ctx, image) } ``` -
fuzz-testing.md 4 KB
# Go Fuzz Testing Go 1.18+ includes built-in fuzzing support. This guide covers patterns for security-focused fuzz testing. ## When to Use Fuzz Testing - Input parsing (URLs, queries, content types) - Data validation and sanitization - Security-sensitive operations (XSS prevention, path traversal detection) - Protocol handling and serialization - Cache key generation ## Basic Pattern ```go //go:build fuzz package mypackage import ( "testing" "unicode/utf8" ) func FuzzMyFunction(f *testing.F) { // 1. Seed with known edge cases f.Add("normal input") f.Add("") // Empty f.Add("\x00null") // Null bytes f.Add("../../../etc/passwd") // Path traversal f.Add("<script>alert(1)") // XSS attempt f.Add("' OR 1=1--") // SQL injection // 2. Define the fuzz target f.Fuzz(func(t *testing.T, input string) { // Skip invalid UTF-8 if needed if !utf8.ValidString(input) { return } // Exercise the function - should not panic result, err := MyFunction(input) // Validate invariants if err == nil { // Check properties that should always hold if result == nil { t.Error("nil result without error") } } }) } ``` ## Security-Focused Seeds ### URL/Path Handling ```go f.Add("/users") f.Add("/users/john%20doe") f.Add("/%2e%2e/etc/passwd") // Path traversal f.Add("/%00null") // Null byte injection f.Add("/users/../../../etc/passwd") // Directory traversal f.Add("/%252e%252e/") // Double encoding f.Add("/路径/用户") // Unicode paths f.Add("//double//slashes//") f.Add("/users;id") // Command injection f.Add("/users|ls") // Pipe injection ``` ### Query Parameters ```go f.Add("key=value") f.Add("key=") f.Add("=value") f.Add("key") f.Add("") f.Add("key=value&key=value2") // Duplicate keys f.Add("key=%00") // Null byte f.Add("key=<script>") // XSS f.Add("key=' OR 1=1--") // SQL injection f.Add("key[]=value1&key[]=value2") // Array syntax ``` ### XSS Payloads ```go f.Add("<script>alert(1)</script>") f.Add("<img src=x onerror=alert(1)>") f.Add("javascript:alert(1)") f.Add("<svg onload=alert(1)>") f.Add("{{.}}") // Template injection f.Add("${7*7}") // Expression injection ``` ## Running Fuzz Tests ```bash # Run specific fuzz test (30 seconds) go test -fuzz=FuzzMyFunction -fuzztime=30s ./... # Run all fuzz tests in package go test -fuzz=. -fuzztime=1m ./path/to/package # Run with race detector (slower but thorough) go test -fuzz=FuzzMyFunction -fuzztime=30s -race ./... # Reproduce a failing case from testdata go test -run=FuzzMyFunction/failing_case ./... ``` ## CI Integration Add to Makefile: ```makefile .PHONY: fuzz fuzz: @echo "Running fuzz tests..." @for pkg in $$(go list ./... | grep -v /vendor/); do \ for fuzz in $$(go test -list='^Fuzz' $$pkg 2>/dev/null | grep '^Fuzz'); do \ echo "Fuzzing $$fuzz in $$pkg..."; \ go test -fuzz=$$fuzz -fuzztime=30s $$pkg || exit 1; \ done; \ done ``` ## Best Practices 1. **Use Build Tags**: Isolate fuzz tests with `//go:build fuzz` 2. **Seed Edge Cases**: Include security payloads, boundary values, unicode 3. **Validate UTF-8**: Skip invalid strings early if your code expects valid UTF-8 4. **Check Invariants**: Assert properties that should always hold 5. **No Panics**: Primary goal is proving code doesn't panic on any input 6. **Limit Resource Usage**: Skip extremely long inputs to prevent timeouts ## File Organization ``` package/ ├── handler.go ├── handler_test.go # Unit tests └── handler_fuzz_test.go # Fuzz tests (//go:build fuzz) ``` ## Related - [Go Fuzzing Documentation](https://go.dev/security/fuzz/) - `references/testing.md` - General testing patterns - `references/mutation-testing.md` - Complementary test quality measurement -
ldap.md 24.3 KB
# LDAP/Active Directory Integration in Go ## Client Setup ### Basic LDAP Client ```go package ldap import ( "crypto/tls" "fmt" "github.com/go-ldap/ldap/v3" ) type Client struct { conn *ldap.Conn baseDN string bindDN string bindPW string userFilter string } type Config struct { Host string Port int BaseDN string BindDN string BindPW string UseTLS bool SkipVerify bool } func NewClient(cfg Config) (*Client, error) { address := fmt.Sprintf("%s:%d", cfg.Host, cfg.Port) var conn *ldap.Conn var err error if cfg.UseTLS { tlsConfig := &tls.Config{ InsecureSkipVerify: cfg.SkipVerify, ServerName: cfg.Host, } conn, err = ldap.DialTLS("tcp", address, tlsConfig) } else { conn, err = ldap.Dial("tcp", address) } if err != nil { return nil, fmt.Errorf("failed to connect to LDAP: %w", err) } // Bind with credentials if err := conn.Bind(cfg.BindDN, cfg.BindPW); err != nil { conn.Close() return nil, fmt.Errorf("failed to bind: %w", err) } return &Client{ conn: conn, baseDN: cfg.BaseDN, bindDN: cfg.BindDN, bindPW: cfg.BindPW, }, nil } func (c *Client) Close() error { if c.conn != nil { c.conn.Close() } return nil } ``` ### Connection Pool ```go type ClientPool struct { cfg Config pool chan *Client maxSize int } func NewClientPool(cfg Config, maxSize int) *ClientPool { return &ClientPool{ cfg: cfg, pool: make(chan *Client, maxSize), maxSize: maxSize, } } func (p *ClientPool) Get() (*Client, error) { select { case client := <-p.pool: // Test connection if err := client.conn.Bind(p.cfg.BindDN, p.cfg.BindPW); err == nil { return client, nil } // Connection dead, create new client.Close() default: // Pool empty } return NewClient(p.cfg) } func (p *ClientPool) Put(client *Client) { select { case p.pool <- client: // Returned to pool default: // Pool full, close connection client.Close() } } ``` ### Identity leak: rebind pooled connections after a user bind A password check binds a pooled connection **as the end user** (`conn.Bind(userDN, userPassword)`). If the pool returns that connection without restoring the service-account bind, the next borrower runs under the user's identity — an authorization leak. The pool above avoids it by rebinding as the service account on `Get`; if yours does not, rebind on the release path (and drop the connection if the rebind fails, rather than pooling a mis-bound one): ```go func (l *LDAP) rebindPooledConnToService(c *ldap.Conn) { if err := c.Bind(l.serviceDN, l.servicePW); err != nil { _ = c.Close() // don't return a mis-bound connection to the pool } } ``` **Testing this leak — do not drain the pool.** The obvious test (perform a password check, then `Get` connections back and inspect them) is a **false-green guard**: the pool's health check runs a probe `Search` on release, which fails on a connection bound as an unprivileged user and **discards it before you can observe it**, so the drain never sees the leak and the test passes even with the fix removed. Instead, inspect a *specific* connection deterministically: ```go // One-connection pool (cache-warm any internal lookup so the check needs only // the bind connection), bind it as the user, then assert the identity directly. conn, _ := client.pool.Get(ctx) _ = conn.Bind(userDN, userPassword) // simulate the verification bind client.rebindPooledConnToService(conn) // the code under test who, _ := conn.WhoAmI(nil) // RFC 4532 require.Contains(t, who.AuthzID, serviceAccount) // not the user ``` Verify it red-green: with the rebind removed, `WhoAmI` still reports the user. ## User Operations ### User Model ```go type User struct { DN string CN string SAMAccountName string UserPrincipalName string Email string DisplayName string FirstName string LastName string Department string Title string Manager string MemberOf []string Enabled bool LastLogon time.Time } func userFromEntry(entry *ldap.Entry) *User { user := &User{ DN: entry.DN, CN: entry.GetAttributeValue("cn"), SAMAccountName: entry.GetAttributeValue("sAMAccountName"), UserPrincipalName: entry.GetAttributeValue("userPrincipalName"), Email: entry.GetAttributeValue("mail"), DisplayName: entry.GetAttributeValue("displayName"), FirstName: entry.GetAttributeValue("givenName"), LastName: entry.GetAttributeValue("sn"), Department: entry.GetAttributeValue("department"), Title: entry.GetAttributeValue("title"), Manager: entry.GetAttributeValue("manager"), MemberOf: entry.GetAttributeValues("memberOf"), } // Parse userAccountControl for enabled status uac := entry.GetAttributeValue("userAccountControl") if uac != "" { uacInt, _ := strconv.Atoi(uac) user.Enabled = (uacInt & 0x2) == 0 // ACCOUNTDISABLE flag } return user } ``` ### Find Users ```go var userAttributes = []string{ "dn", "cn", "sAMAccountName", "userPrincipalName", "mail", "displayName", "givenName", "sn", "department", "title", "manager", "memberOf", "userAccountControl", } func (c *Client) FindUserBySAM(samAccountName string) (*User, error) { filter := fmt.Sprintf("(&(objectClass=user)(sAMAccountName=%s))", ldap.EscapeFilter(samAccountName)) return c.findUser(filter) } func (c *Client) FindUserByEmail(email string) (*User, error) { filter := fmt.Sprintf("(&(objectClass=user)(mail=%s))", ldap.EscapeFilter(email)) return c.findUser(filter) } func (c *Client) FindUserByDN(dn string) (*User, error) { result, err := c.conn.Search(&ldap.SearchRequest{ BaseDN: dn, Scope: ldap.ScopeBaseObject, Filter: "(objectClass=user)", Attributes: userAttributes, }) if err != nil { return nil, err } if len(result.Entries) == 0 { return nil, ErrUserNotFound } return userFromEntry(result.Entries[0]), nil } func (c *Client) findUser(filter string) (*User, error) { result, err := c.conn.Search(&ldap.SearchRequest{ BaseDN: c.baseDN, Scope: ldap.ScopeWholeSubtree, Filter: filter, Attributes: userAttributes, }) if err != nil { return nil, fmt.Errorf("search failed: %w", err) } if len(result.Entries) == 0 { return nil, ErrUserNotFound } return userFromEntry(result.Entries[0]), nil } func (c *Client) ListUsers(filter string, limit int) ([]*User, error) { if filter == "" { filter = "(objectClass=user)" } result, err := c.conn.Search(&ldap.SearchRequest{ BaseDN: c.baseDN, Scope: ldap.ScopeWholeSubtree, Filter: filter, Attributes: userAttributes, SizeLimit: limit, }) if err != nil { return nil, err } users := make([]*User, 0, len(result.Entries)) for _, entry := range result.Entries { users = append(users, userFromEntry(entry)) } return users, nil } ``` ## Authentication ### Validate Credentials The early return on "user not found" below is a **user-enumeration timing side channel**: the not-found path skips the bind entirely, so it answers in a fraction of the time the wrong-password path takes, and an attacker reads account existence off the clock. Both branches must do the same work — see [Constant-time authentication branches](#constant-time-authentication-branches) directly below for the shape that closes it. ```go func (c *Client) Authenticate(username, password string) (*User, error) { // First, find the user user, err := c.FindUserBySAM(username) if err != nil { // DO NOT return here — see the next section. The dummy bind belongs here. return nil, fmt.Errorf("user not found: %w", err) } // Try to bind with user's credentials err = c.conn.Bind(user.DN, password) if err != nil { // Re-bind as service account c.conn.Bind(c.bindDN, c.bindPW) return nil, ErrInvalidCredentials } // Re-bind as service account for subsequent operations c.conn.Bind(c.bindDN, c.bindPW) return user, nil } // Alternative: Create new connection for auth func (c *Client) AuthenticateWithNewConn(username, password string) (*User, error) { user, err := c.FindUserBySAM(username) if err != nil { return nil, err } // Create separate connection for auth cfg := Config{ Host: c.cfg.Host, Port: c.cfg.Port, BaseDN: c.baseDN, BindDN: user.DN, BindPW: password, UseTLS: c.cfg.UseTLS, } authClient, err := NewClient(cfg) if err != nil { return nil, ErrInvalidCredentials } authClient.Close() return user, nil } ``` ### Constant-time authentication branches When the lookup fails, do what the found path does: bind against a non-existent identifier with the supplied password, restore the service bind, and record the attempt. The error the caller sees stays the lookup error; only the *work* is equalised. ```go // buildDummyBindDN is the part every hand-written version of this skips. // The identifier reaches a bind DN, so it is ESCAPED — interpolating a raw // identifier into a DN is its own problem, separate from the timing one. func buildDummyBindDN(identifier, baseDN string) string { return fmt.Sprintf("CN=nonexistent-%s,CN=Users,%s", ldap.EscapeDN(identifier), baseDN) } var bindErr error if lookupErr == nil { bindErr = conn.Bind(user.DN(), password) } else { // same bind, same password, against a DN that cannot exist _ = conn.Bind(buildDummyBindDN(identifier, baseDN), password) bindErr = lookupErr // the caller still learns "not found", not "wrong password" } // the verification bind re-authenticated this connection as the end user (or as // the dummy identity); a pooled connection must be rebound as the service // account before it goes back, in BOTH branches l.rebindPooledConnToService(conn, "CheckPasswordForSAMAccountName") if bindErr != nil { rateLimiter.RecordFailure(key) // both branches, or the counter leaks existence too return nil, bindErr } ``` Three parts, and the escaping is the one that gets dropped: measured across ten independent agent fixes of exactly this defect in `netresearch/simple-ldap-go`, ten of ten closed the timing gap and **none** escaped the dummy DN ([go-development-skill#70](https://github.com/netresearch/go-development-skill/issues/70)). The library's own `CheckPasswordForSAMAccountName` has carried the correct version the whole time, one function away in the same file. The rate limiter counts too. Recording a failure only on the found path turns the counter into the oracle the timing fix just closed. ### Testing a timing fix Assert an **observable the fix changes**, never elapsed time. A wall-clock assertion is noise on a loaded machine and on CI: it fails when a neighbouring container gets busy and passes on a tree where the bind was removed again. What to assert instead, in rough order of preference: - the dummy bind happened: a recorded failed attempt in the rate limiter for the unknown identifier, with the same key shape the found path uses - the bind reached the server: a captured bind DN that is the escaped dummy DN, which pins the escaping at the same time - a metric or log counter the authentication path increments in both branches ```go // the observable, not the clock _, err := client.CheckPasswordForSAMAccountName("no-such-user", "whatever") require.Error(t, err) require.Equal(t, 1, limiter.Failures(normalizeKey("no-such-user")), "the not-found path must record an attempt, or the rate limiter is the oracle") ``` A timing fix with no test is a fix until the next refactor — the same ten trials left **0 of 10** regression tests behind. ## Password Operations ### Change Password ```go func (c *Client) ChangePassword(userDN, oldPassword, newPassword string) error { // AD requires the password in a specific format oldPwdEncoded := encodePassword(oldPassword) newPwdEncoded := encodePassword(newPassword) modifyRequest := ldap.NewModifyRequest(userDN, nil) modifyRequest.Delete("unicodePwd", []string{oldPwdEncoded}) modifyRequest.Add("unicodePwd", []string{newPwdEncoded}) return c.conn.Modify(modifyRequest) } func (c *Client) ResetPassword(userDN, newPassword string) error { // Admin reset - doesn't require old password newPwdEncoded := encodePassword(newPassword) modifyRequest := ldap.NewModifyRequest(userDN, nil) modifyRequest.Replace("unicodePwd", []string{newPwdEncoded}) return c.conn.Modify(modifyRequest) } func encodePassword(password string) string { // AD requires UTF-16LE encoded password surrounded by quotes utf16 := utf16.Encode([]rune("\"" + password + "\"")) pwBytes := make([]byte, len(utf16)*2) for i, v := range utf16 { pwBytes[i*2] = byte(v) pwBytes[i*2+1] = byte(v >> 8) } return string(pwBytes) } ``` ## Group Operations ### Group Model ```go type Group struct { DN string CN string Description string Members []string MemberOf []string } func (c *Client) FindGroup(cn string) (*Group, error) { filter := fmt.Sprintf("(&(objectClass=group)(cn=%s))", ldap.EscapeFilter(cn)) result, err := c.conn.Search(&ldap.SearchRequest{ BaseDN: c.baseDN, Scope: ldap.ScopeWholeSubtree, Filter: filter, Attributes: []string{"dn", "cn", "description", "member", "memberOf"}, }) if err != nil { return nil, err } if len(result.Entries) == 0 { return nil, ErrGroupNotFound } entry := result.Entries[0] return &Group{ DN: entry.DN, CN: entry.GetAttributeValue("cn"), Description: entry.GetAttributeValue("description"), Members: entry.GetAttributeValues("member"), MemberOf: entry.GetAttributeValues("memberOf"), }, nil } func (c *Client) AddUserToGroup(userDN, groupDN string) error { modifyRequest := ldap.NewModifyRequest(groupDN, nil) modifyRequest.Add("member", []string{userDN}) return c.conn.Modify(modifyRequest) } func (c *Client) RemoveUserFromGroup(userDN, groupDN string) error { modifyRequest := ldap.NewModifyRequest(groupDN, nil) modifyRequest.Delete("member", []string{userDN}) return c.conn.Modify(modifyRequest) } func (c *Client) IsUserInGroup(userDN, groupCN string) (bool, error) { user, err := c.FindUserByDN(userDN) if err != nil { return false, err } for _, groupDN := range user.MemberOf { if strings.Contains(strings.ToLower(groupDN), strings.ToLower("CN="+groupCN)) { return true, nil } } return false, nil } ``` ## Error Handling ```go var ( ErrUserNotFound = errors.New("user not found") ErrGroupNotFound = errors.New("group not found") ErrInvalidCredentials = errors.New("invalid credentials") ErrConnectionFailed = errors.New("LDAP connection failed") ErrPermissionDenied = errors.New("permission denied") ) func translateLDAPError(err error) error { if ldapErr, ok := err.(*ldap.Error); ok { switch ldapErr.ResultCode { case ldap.LDAPResultNoSuchObject: return ErrUserNotFound case ldap.LDAPResultInvalidCredentials: return ErrInvalidCredentials case ldap.LDAPResultInsufficientAccessRights: return ErrPermissionDenied } } return err } ``` ## Computer Objects ```go type Computer struct { DN string CN string DNSHostName string OperatingSystem string OSVersion string LastLogon time.Time Enabled bool } func (c *Client) ListComputers(filter string) ([]*Computer, error) { if filter == "" { filter = "(objectClass=computer)" } result, err := c.conn.Search(&ldap.SearchRequest{ BaseDN: c.baseDN, Scope: ldap.ScopeWholeSubtree, Filter: filter, Attributes: []string{ "dn", "cn", "dNSHostName", "operatingSystem", "operatingSystemVersion", "lastLogonTimestamp", "userAccountControl", }, }) if err != nil { return nil, err } computers := make([]*Computer, 0, len(result.Entries)) for _, entry := range result.Entries { computers = append(computers, &Computer{ DN: entry.DN, CN: entry.GetAttributeValue("cn"), DNSHostName: entry.GetAttributeValue("dNSHostName"), OperatingSystem: entry.GetAttributeValue("operatingSystem"), OSVersion: entry.GetAttributeValue("operatingSystemVersion"), }) } return computers, nil } ## Common Gotchas ### simple-ldap-go "localhost" Mock Detection The `simple-ldap-go` library's `isExampleServerName()` function treats `"localhost"` as a mock/example server name. When this is detected, the library returns fake connections that fail with `"connection to example server not available"`. ```go // BAD - simple-ldap-go treats "localhost" as a mock server cfg := simpleldap.Config{ Server: "localhost", Port: 1389, } // Returns: "connection to example server not available" // GOOD - Use 127.0.0.1 to avoid mock detection cfg := simpleldap.Config{ Server: "127.0.0.1", Port: 1389, } ``` This applies to any context using `simple-ldap-go`, including integration tests against a real LDAP server running on localhost. ### IPv6-Safe Host:Port Formatting Use `net.JoinHostPort` instead of `fmt.Sprintf` for constructing address strings. `go vet` flags `fmt.Sprintf("%s:%d", host, port)` because it produces invalid addresses for IPv6 hosts (e.g., `::1:389` instead of `[::1]:389`). ```go // BAD - Fails with IPv6 addresses, flagged by go vet address := fmt.Sprintf("%s:%d", host, port) // GOOD - Handles IPv4, IPv6, and hostnames correctly address := net.JoinHostPort(host, strconv.Itoa(port)) ``` ## Testing LDAP with Testcontainers ### LDAP Lazy Binding Behavior **Critical**: LDAP connections use **lazy binding**. The `Dial()` or `DialTLS()` call only establishes a TCP connection - authentication is not validated until the first LDAP operation. ```go // Connection succeeds even with invalid credentials! conn, err := ldap.Dial("tcp", "ldap.example.com:389") if err != nil { // Only fails on network/DNS errors, NOT auth errors } // Auth is validated HERE, on first operation err = conn.Bind("cn=admin,dc=example,dc=com", "wrong-password") // NOW you get auth errors ``` ### OpenLDAP Anonymous Reads OpenLDAP allows anonymous read access by default. This affects health checks: ```go // Health check using Search works WITHOUT authentication func (c *Client) HealthCheck() error { _, err := c.conn.Search(&ldap.SearchRequest{ BaseDN: c.baseDN, Scope: ldap.ScopeBaseObject, Filter: "(objectClass=*)", Attributes: []string{"1.1"}, // Request no attributes SizeLimit: 1, }) return err // Works even without Bind! } // For true auth validation, use Bind explicitly func (c *Client) ValidateCredentials() error { return c.conn.Bind(c.bindDN, c.bindPassword) } ``` ### Testcontainers Pattern for LDAP Use ephemeral OpenLDAP containers for integration tests: ```go //go:build integration package ldap_test import ( "context" "testing" "github.com/testcontainers/testcontainers-go" "github.com/testcontainers/testcontainers-go/wait" ) func setupOpenLDAPContainer(t *testing.T) (host string, port int, cleanup func()) { ctx := context.Background() req := testcontainers.ContainerRequest{ Image: "osixia/openldap:1.5.0", // Pin version! ExposedPorts: []string{"389/tcp"}, Env: map[string]string{ "LDAP_ORGANISATION": "Test Org", "LDAP_DOMAIN": "example.com", "LDAP_ADMIN_PASSWORD": "admin", // OK for ephemeral test container "LDAP_BASE_DN": "dc=example,dc=com", }, WaitingFor: wait.ForListeningPort("389/tcp"), } container, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ ContainerRequest: req, Started: true, }) if err != nil { t.Fatalf("Failed to start container: %v", err) } mappedPort, _ := container.MappedPort(ctx, "389") hostIP, _ := container.Host(ctx) return hostIP, mappedPort.Int(), func() { container.Terminate(ctx) } } func TestLDAPIntegration(t *testing.T) { host, port, cleanup := setupOpenLDAPContainer(t) defer cleanup() client, err := NewClient(Config{ Host: host, Port: port, BindDN: "cn=admin,dc=example,dc=com", BindPW: "admin", BaseDN: "dc=example,dc=com", }) require.NoError(t, err) defer client.Close() // Test operations... } ``` ### CI Service Container Pattern (Without Testcontainers) For CI environments where testcontainers are not available, use GitHub Actions service containers with a `skipIfNoLDAP` pattern: ```go package web_test import ( "net" "strconv" "testing" "time" ldapv3 "github.com/go-ldap/ldap/v3" ) const ( ldapHost = "127.0.0.1" // NOT "localhost" — avoids simple-ldap-go mock detection ldapPort = 1389 ldapBaseDN = "dc=test,dc=local" ldapAdmin = "cn=admin,dc=test,dc=local" ldapPass = "admin" ) // skipIfNoLDAP skips the test if the LDAP server is not reachable. func skipIfNoLDAP(t *testing.T) { t.Helper() address := net.JoinHostPort(ldapHost, strconv.Itoa(ldapPort)) conn, err := (&net.Dialer{Timeout: 2 * time.Second}).Dial("tcp", address) if err != nil { t.Skipf("LDAP server not available at %s: %v", address, err) } _ = conn.Close() } // seedLDAPData uses go-ldap/ldap/v3 directly to create test entries. func seedLDAPData(t *testing.T) { t.Helper() address := net.JoinHostPort(ldapHost, strconv.Itoa(ldapPort)) conn, err := ldapv3.DialURL("ldap://" + address) if err != nil { t.Fatalf("failed to connect to LDAP: %v", err) } defer func() { _ = conn.Close() }() if err := conn.Bind(ldapAdmin, ldapPass); err != nil { t.Fatalf("failed to bind: %v", err) } // Add OUs, users, groups as needed addReq := ldapv3.NewAddRequest("ou=users,"+ldapBaseDN, nil) addReq.Attribute("objectClass", []string{"organizationalUnit"}) addReq.Attribute("ou", []string{"users"}) if err := conn.Add(addReq); err != nil { // It's okay if the entry already exists on re-runs. if ldapErr, ok := err.(*ldapv3.Error); !ok || ldapErr.ResultCode != ldapv3.LDAPResultEntryAlreadyExists { t.Fatalf("failed to add seed data: %v", err) } } } ``` GitHub Actions service container configuration: ```yaml services: openldap: image: osixia/openldap:1.5.0 ports: - 1389:389 env: LDAP_ORGANISATION: "Test Org" LDAP_DOMAIN: "test.local" LDAP_ADMIN_PASSWORD: "admin" LDAP_BASE_DN: "dc=test,dc=local" ``` **Key patterns:** - Use `127.0.0.1` not `localhost` to avoid simple-ldap-go mock detection - Use `net.Dialer` (not `net.DialTimeout`) to satisfy the `noctx` linter - Use `go-ldap/ldap/v3` directly for seeding test data (independent of app's LDAP library) - Make handler assertions resilient: accept success OR error when LDAP session credentials may be stale - Close connections with `defer func() { _ = conn.Close() }()` to satisfy `errcheck` ### Security Note: Test Credentials Hardcoded credentials in testcontainer setup are acceptable because: 1. Containers are ephemeral (destroyed after test) 2. Run on isolated localhost ports 3. Contain no real data **Never** use production credentials in tests. Always use dedicated test accounts with minimal permissions. -
lefthook-template.md 2.3 KB
# Lefthook Template for Go Projects Install: `go install github.com/evilmartians/lefthook@latest && lefthook install` Or add to Makefile: `make setup` ## lefthook.yml ```yaml # Go project git hooks - powered by lefthook # https://github.com/evilmartians/lefthook pre-commit: parallel: true commands: go-mod-tidy: run: go mod tidy -diff 2>/dev/null || (echo "Run: go mod tidy" && exit 1) go-vet: glob: "*.go" run: go vet ./... gofmt: glob: "*.go" run: | unformatted=$(gofmt -l $(git ls-files '*.go')) [ -z "$unformatted" ] || (echo "Unformatted: $unformatted" && exit 1) commit-msg: commands: conventional-commits: run: | msg=$(cat {1}) echo "$msg" | grep -qE "^(feat|fix|docs|style|refactor|test|chore|perf|ci|build|revert)(\(.+\))?: .+" || \ echo "Warning: not conventional commits format" signoff: run: | grep -qE "^Signed-off-by: .+ <.+>" {1} || \ (echo "Missing --signoff" && exit 1) pre-push: parallel: true commands: lint: glob: "*.go" run: golangci-lint run --timeout=3m test: # -short only keeps this fast if slow/integration tests skip via # testing.Short() guards. Set -timeout to cover your slowest # *unguarded* package, or the push fails on every push. run: go test -short -timeout=60s ./... ``` Customize per project: add security scanning (gosec), mutation testing, etc. **Pre-push timeout pitfall.** The `-short -timeout=60s` smoke-test stays fast *only if* slow/integration tests honor `-short` with a `if testing.Short() { t.Skip(...) }` guard. A package that ignores `-short` (real-Docker integration tests are the classic case) runs in full, blows the fixed timeout, and fails the hook on **every** push — which reads like a real regression but is not. Diagnose before reaching for `--no-verify`: ```bash go list -deps ./slow/pkg | grep your/changed/pkg # if this matches, the slow package depends on your change go test -short -timeout=300s ./slow/pkg # what does it actually take? ``` If the slow package is unrelated to your diff and pre-existing, `git push --no-verify` is justified — then fix the root cause separately by guarding those tests with `testing.Short()` (preferred, keeps the smoke-test fast) or raising `-timeout` to cover the slowest unguarded package. -
linting.md 14.2 KB
# Go Linting and Code Quality ## golangci-lint v2 Configuration golangci-lint v2 uses a new YAML structure. Here's a production-ready configuration: ```yaml # .golangci.yml version: "2" run: tests: true linters: default: none enable: # Bugs & Correctness (Critical) - govet # Go vet checks - staticcheck # Comprehensive static analysis - errcheck # Unchecked errors - errorlint # Error wrapping issues - bodyclose # HTTP response body close - noctx # HTTP requests without context - durationcheck # Detects time.Second * time.Second bugs - nilerr # Catches return nil when err != nil - nilnesserr # Checks err != nil but returns different nil - fatcontext # Detects nested contexts in loops - contextcheck # Non-inherited context usage - copyloopvar # Loop variable copy issues (Go 1.22+) - forcetypeassert # Unchecked type assertions (panic risk) - makezero # Slice with non-zero initial length bugs # Security - gosec # Security issues # Performance - prealloc # Slice preallocation suggestions - unconvert # Unnecessary type conversions - perfsprint # Faster sprintf alternatives # Style & Maintainability - gocyclo # Cyclomatic complexity - gocognit # Cognitive complexity - funlen # Function length limits - nestif # Nested if statement depth - ineffassign # Ineffective assignments - unused # Unused code detection - misspell # Spelling mistakes - revive # Fast, configurable linter - gocritic # Opinionated linter # Modernization (Go 1.22+) - intrange # Use for range n - usestdlibvars # Use http.StatusOK instead of 200 - modernize # Modern Go features # Testing Quality - thelper # Test helpers should call t.Helper() - tparallel # Correct t.Parallel() usage settings: gocyclo: min-complexity: 15 gocognit: min-complexity: 30 funlen: lines: 80 statements: 50 nestif: min-complexity: 4 misspell: locale: US errcheck: check-type-assertions: true check-blank: false # Allow explicit _ = err exclusions: generated: lax presets: - comments - common-false-positives - legacy - std-error-handling rules: # Exclude complexity checks in test files - linters: - gocyclo - gocognit - funlen - nestif path: _test\.go # Example: Exclude inherently complex functions # - linters: # - gocyclo # - gocognit # path: parser\.go # text: "(parse|complexFunction)" formatters: enable: - gci # Import grouping - gofumpt # Stricter gofmt settings: gci: sections: - standard - default - prefix(github.com/your-org/your-project) ``` ## Linter Selection Strategy ### By Category | Category | Linters | Priority | |----------|---------|----------| | **Bugs** | govet, staticcheck, errcheck, nilerr | Critical | | **Security** | gosec, bidichk | High | | **Performance** | prealloc, unconvert, perfsprint | Medium | | **Style** | gocyclo, funlen, revive, gocritic | Medium | | **Modernization** | intrange, modernize, usestdlibvars | Low | ### Adding Exclusions Properly When a linter flags inherently complex code that cannot be simplified: ```yaml exclusions: rules: # Document WHY the exclusion is needed # Next() and Prev() have inherent complexity due to # multi-field time calculation with wraparound logic - linters: - gocognit - gocyclo path: spec\.go text: "(Next|Prev)" ``` **Best Practice**: Always add a comment explaining why the exclusion is justified. ## Common staticcheck/revive Fixes ### ST1005: Error String Formatting Error strings should NOT be capitalized or end with punctuation: ```go // BAD - Will trigger ST1005 return errors.New("H expressions require a hash key") return fmt.Errorf("Invalid input: %s.", input) // GOOD return errors.New("h expressions require a hash key") return fmt.Errorf("invalid input: %s", input) ``` **Rationale**: Error messages are often wrapped or concatenated. Lowercase prevents awkward capitalization like `"failed: Invalid input"`. ### ST1003: Naming Conventions ```go // BAD var serverId string // Should be serverID func GetUserId() {} // Should be GetUserID type HttpClient struct // Should be HTTPClient // GOOD var serverID string func GetUserID() {} type HTTPClient struct ``` ### gosec G104: Unhandled Errors For functions that always return nil errors (like `hash.Hash.Write`): ```go // BAD - gosec G104 warning h := fnv.New64a() h.Write([]byte(key)) // Error unhandled // GOOD - Explicitly acknowledge the ignored return h := fnv.New64a() _, _ = h.Write([]byte(key)) // hash.Hash.Write never returns error ``` **When to use `_, _ =`**: - `hash.Hash.Write()` - Never returns error per spec - `bytes.Buffer.Write()` - Never returns error - `strings.Builder.WriteString()` - Never returns error ### revive: Error Naming ```go // BAD var InvalidInput = errors.New("invalid input") // Should start with Err type ValidationFailed struct{} // Should end with Error // GOOD var ErrInvalidInput = errors.New("invalid input") type ValidationError struct{} ``` ### revive: Stdlib Package Name Conflicts The `var-naming` rule flags package names that conflict with Go stdlib packages. Common conflicts and safe alternatives: | Avoid | Conflicts with | Use instead | |-------|---------------|-------------| | `rpc` | `net/rpc` | `rpchandler`, `rpcapi` | | `jsonrpc` | `net/rpc/jsonrpc` | `jsonrpchandler`, `jsonrpcapi` | | `http` | `net/http` | `httputil`, `server` | | `log` | `log` | `logger`, `logging` | Check with: `go list std | grep -w <name>` Also verify type names don't stutter after rename: ```go // BAD - stutters: rpchandler.JSONRPCResponse type JSONRPCResponse struct { ... } // GOOD - clean: rpchandler.Response type Response struct { ... } ``` ### golangci-lint: CI vs Local Version Drift When CI uses `version: latest` in the golangci-lint-action, linter behavior may differ from local runs: - gosec rules (e.g., G704 SSRF) may fire in CI but not locally due to version differences - Use `//nolint:gosec,nolintlint` to suppress in both environments - `nolintlint` complains about unused directives when the target linter doesn't fire locally ```go // Suppresses gosec in CI and nolintlint locally when gosec doesn't fire resp, err := client.Do(req) //nolint:gosec,nolintlint // G704: URL is a compile-time constant ``` **Best practice**: Pin the golangci-lint version in CI to match local, or accept the dual-nolint pattern for edge cases. ## Common Linter Gotchas ### The Default Output Cap Silently Truncates golangci-lint stops at `max-issues-per-linter: 50` and `max-same-issues: 3` by default, and says nothing about what it dropped. Any count you read off a run is a floor, not a total. This bites hardest when taking stock before a cleanup: a report of "50 findings" that is really 123 turns an afternoon's work into a week's. The tell is a run that reports exactly 50, or two runs that list different files while reporting the same total. ```bash # For an inventory, uncap it: golangci-lint run --max-issues-per-linter=0 --max-same-issues=0 ``` Consider uncapping in the config as well. A cap also lets a regression hide behind it: once a linter is at its limit, the next new finding is invisible. ```yaml issues: max-issues-per-linter: 0 max-same-issues: 0 # Findings on a line another linter already reported are dropped by default. uniq-by-line: false ``` ### A Cache Entry Outlives Its Worktree golangci-lint can report findings with paths that no longer exist — typically after a git worktree is removed, when a shared cache still holds its analysis. The giveaway is `no such file or directory` warnings naming a deleted path alongside the findings. ```bash golangci-lint cache clean ``` Treat findings whose paths you cannot open as cache artifacts, not as code problems, and re-run before acting on them. Specific linter rules that frequently trip up developers: ### dupl: Test Function Duplication The `dupl` linter flags test functions with similar structure (threshold ~30 lines). Extract shared logic into helpers: ```go // BAD - Two test functions with nearly identical structure triggers dupl func TestHandlerA(t *testing.T) { app := setupApp(t) req := httptest.NewRequest(http.MethodGet, "/a", nil) resp, err := app.Test(req) require.NoError(t, err) assert.Equal(t, 200, resp.StatusCode) // ... 30+ lines of similar setup } // GOOD - Extract common pattern into a helper func testEndpoint(t *testing.T, app *fiber.App, method, path string, wantStatus int) { t.Helper() req := httptest.NewRequest(method, path, nil) resp, err := app.Test(req) require.NoError(t, err) assert.Equal(t, wantStatus, resp.StatusCode) } ``` ### nlreturn: Blank Line Before Return The `nlreturn` linter requires a blank line before `return` statements, including inside closures and anonymous functions: ```go // BAD func example() error { result := compute() return result // nlreturn: missing blank line before return } // GOOD func example() error { result := compute() return result } ``` ### noctx: Network Dialing The `noctx` linter flags `net.DialTimeout()`. Use `net.Dialer` instead: ```go // BAD - noctx flags this conn, err := net.DialTimeout("tcp", address, 2*time.Second) // GOOD - Use Dialer struct conn, err := (&net.Dialer{Timeout: 2 * time.Second}).Dial("tcp", address) ``` ### revive unused-parameter: Underscore Convention For intentionally unused parameters, rename to `_` prefix or bare `_`: ```go // BAD - revive flags unused parameter func setup(t *testing.T) { /* t not used */ } // GOOD - Explicitly mark as unused func setup(_ *testing.T) { /* intentionally unused */ } ``` **Note:** Only use `_` for parameters that are truly unused. If you need `t.Helper()`, `t.Cleanup()`, etc., keep the parameter named. ### errcheck: Deferred Close The `errcheck` linter requires handling errors from `Close()` even in defer: ```go // BAD - errcheck flags this defer conn.Close() // GOOD - Explicitly discard the error defer func() { _ = conn.Close() }() ``` ### revive: Package Names with Underscores Go convention discourages underscores in package names. When they are intentional (e.g., test packages with build tags), suppress with a nolint comment: ```go //nolint:revive // underscore in package name is intentional for build tag isolation package integration_test ``` ## go fix — Automated Modernization (Go 1.26+) Go 1.26 ships a rewritten `go fix` with 22 built-in modernizers. Run after Go upgrades: ```bash go fix -diff ./... # Preview changes go fix ./... # Apply changes ``` Key modernizers: `any`, `rangeint`, `slicescontains`, `mapsloop`, `minmax`, `waitgroup`, `testingcontext`, `reflecttypefor`, `stringscutprefix`, `stringsseq`, `stringsbuilder`. **Always run linters after `go fix`** — it may leave unused imports, redundant variables, or gofumpt issues. See `references/modernization.md` for the full modernizer reference and manual migrations like `errors.AsType[T]`. ## Running Linters ### Development Workflow ```bash # Quick check during development golangci-lint run --fast # Full check before commit golangci-lint run # Check specific files golangci-lint run ./pkg/... # Auto-fix where possible golangci-lint run --fix ``` ### CI Configuration ```yaml # GitHub Actions - name: golangci-lint uses: golangci/golangci-lint-action@v6 with: version: v1.62 args: --timeout 5m ``` ### gci Import-Order Pre-commit (Recurring CI Friction) `gci` import ordering differences between local formats and CI are the most common blocker across Go repos. Enforce the exact ordering that matches `.golangci.yml` before commit. The lefthook block below is a **fragment to merge into an existing `lefthook.yml`** — not a standalone file. See `references/lefthook-template.md` for a complete starter config that this block slots into. ```bash # One-shot fix across the whole module gci write --skip-generated -s standard -s default -s localmodule . # Pre-commit hook fragment (merge into existing lefthook.yml under pre-commit.commands) pre-commit: parallel: true commands: gci: glob: "*.go" run: gci write --skip-generated -s standard -s default -s localmodule {staged_files} lint: glob: "*.go" run: golangci-lint run ``` The `-s standard -s default -s localmodule` ordering must match the `sections:` list under `formatters.settings.gci` in `.golangci.yml`. If the `.golangci.yml` uses `prefix(github.com/org/project)` instead of `localmodule`, the pre-commit `gci` call must match exactly. **Install**: `go install github.com/daixiang0/gci@latest` ### Pre-commit Hook (lefthook) ```yaml # .lefthook.yml pre-commit: parallel: true commands: lint: glob: "*.go" run: golangci-lint run --new-from-rev=HEAD~1 ``` ## Complexity Guidelines | Metric | Threshold | Action if Exceeded | |--------|-----------|-------------------| | Cyclomatic (gocyclo) | 15 | Refactor or document why justified | | Cognitive (gocognit) | 30 | Simplify or add exclusion with rationale | | Function Length | 80 lines | Extract helper functions | | Nesting Depth | 4 levels | Refactor with early returns | ### When Complexity is Justified Some functions have inherent complexity that cannot be reduced without fragmenting the algorithm: 1. **Parser functions** - Multiple input formats require branching 2. **Time calculations** - Multi-field wraparound (year/month/day/hour/minute/second) 3. **State machines** - Multiple states and transitions In these cases, add an exclusion with clear documentation. ## Makefile Integration ```makefile .PHONY: lint lint-full lint-fix # Quick lint for development lint: golangci-lint run --fast # Full lint for CI lint-full: golangci-lint run --timeout 5m # Auto-fix issues lint-fix: golangci-lint run --fix # Run specific linters only lint-security: golangci-lint run -E gosec,bidichk lint-bugs: golangci-lint run -E govet,staticcheck,errcheck,nilerr ``` -
logging.md 10.1 KB
# Structured Logging with log/slog ## Why slog Over logrus `log/slog` is Go's stdlib structured logging package (since Go 1.21). It replaces third-party loggers like logrus (maintenance mode since 2020) and zap. **Benefits of slog:** - Zero external dependencies - Structured key-value pairs by design - Pluggable handlers (`TextHandler`, `JSONHandler`, custom) - Runtime-mutable log levels via `slog.LevelVar` - `AddSource: true` replaces manual `runtime.Caller` hacks - Direct use as dependency — `*slog.Logger` IS the interface, no wrapper needed **Anti-pattern: Custom Logger interfaces wrapping slog.** Don't create `type Logger interface { Debug(msg string, args ...any) }` — just use `*slog.Logger` directly. It already is a clean, well-designed interface. Custom wrappers block slog's handler ecosystem and add indirection for no benefit. ## Setup ### Basic Logger with LevelVar ```go func buildLogger(level string) (*slog.Logger, *slog.LevelVar) { levelVar := &slog.LevelVar{} // Map level string to slog level switch strings.ToLower(level) { case "trace", "debug": levelVar.Set(slog.LevelDebug) case "", "info": levelVar.Set(slog.LevelInfo) case "warning", "warn": levelVar.Set(slog.LevelWarn) case "error", "fatal", "panic", "critical": levelVar.Set(slog.LevelError) default: levelVar.Set(slog.LevelInfo) } handler := slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{ AddSource: true, Level: levelVar, }) return slog.New(handler), levelVar } ``` **Key points:** - `slog.LevelVar` enables runtime level changes without rebuilding the logger - `AddSource: true` automatically adds `source=file.go:42` — no `runtime.Caller` needed - Store `levelVar` alongside logger for commands that need `ApplyLogLevel` ### Runtime Level Changes ```go func ApplyLogLevel(level string, lv *slog.LevelVar) error { if level == "" { return nil } switch strings.ToLower(level) { case "trace", "debug": lv.Set(slog.LevelDebug) case "info": lv.Set(slog.LevelInfo) case "warning", "warn": lv.Set(slog.LevelWarn) case "error", "fatal", "panic", "critical": lv.Set(slog.LevelError) default: return fmt.Errorf("invalid log level %q", level) } return nil } ``` **Backward compatibility note:** slog's `Level.UnmarshalText` only recognizes `DEBUG`, `INFO`, `WARN`, `ERROR`. If migrating from logrus, add a pre-mapping for logrus level names like `trace`, `warning`, `fatal`, `panic`, `notice`, `critical`. ## Structured Logging Patterns ### Use Structured Attributes, Not fmt.Sprintf ```go // BAD: Buries structured data in formatted string logger.Info(fmt.Sprintf("Scheduler started with %d jobs", jobCount)) // GOOD: Structured attributes enable machine parsing and filtering logger.Info("Scheduler started", "jobCount", jobCount) // BAD: Error details lost in string formatting logger.Error(fmt.Sprintf("Job %s failed: %v", name, err)) // GOOD: Each field is independently queryable logger.Error("Job failed", "job", name, "error", err) ``` ### Key Naming Conventions ```go // Use camelCase for attribute keys (Go convention) logger.Info("Request completed", "method", r.Method, "path", r.URL.Path, "statusCode", resp.StatusCode, "duration", time.Since(start), ) // Group related attributes logger.Info("Job completed", slog.Group("job", slog.String("name", job.GetName()), slog.String("type", "exec"), ), slog.Group("execution", slog.Duration("duration", d), slog.Bool("failed", false), ), ) ``` ### Passing Loggers Through Structs ```go // Use *slog.Logger directly in struct fields — it IS the interface type Scheduler struct { Logger *slog.Logger LevelVar *slog.LevelVar // Only if runtime level changes needed // ... } type Context struct { Logger *slog.Logger Execution *Execution Job Job } // Create child loggers with additional context func (s *Scheduler) runJob(job Job) { jobLogger := s.Logger.With("job", job.GetName()) jobLogger.Info("Starting job") // ... jobLogger.Info("Job completed", "duration", elapsed) } ``` ## Middleware Logging ```go // Logging middleware using slog func WithLogging(logger *slog.Logger) Middleware { return func(next Job) Job { return JobFunc(func(ctx context.Context) error { start := time.Now() logger.Info("Starting job", "job", next.GetName()) err := next.Run(ctx) attrs := []any{ "job", next.GetName(), "duration", time.Since(start), } if err != nil { logger.Error("Job failed", append(attrs, "error", err)...) } else { logger.Info("Job completed", attrs...) } return err }) } } ``` ## Web Handler Logging ```go type Handler struct { scheduler *Scheduler logger *slog.Logger } func (h *Handler) TriggerJob(w http.ResponseWriter, r *http.Request) { name := chi.URLParam(r, "name") reqLogger := h.logger.With("handler", "TriggerJob", "jobName", name) job, err := h.scheduler.GetJob(name) if err != nil { reqLogger.Warn("Job not found") http.Error(w, "Job not found", http.StatusNotFound) return } go func() { if err := job.Run(context.Background()); err != nil { reqLogger.Error("Manual job execution failed", "error", err) } }() w.WriteHeader(http.StatusAccepted) } ``` ## Testing with slog ### Discard Logger for Tests ```go // Simple: discard all logs logger := slog.New(slog.NewTextHandler(io.Discard, nil)) // With specific level (only errors logged) logger := slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{ Level: slog.LevelError, })) ``` ### Capturing Logs in Tests For tests that need to assert on log output, implement a custom `slog.Handler`: ```go type TestHandler struct { mu sync.Mutex records []slog.Record } func (h *TestHandler) Enabled(_ context.Context, _ slog.Level) bool { return true } func (h *TestHandler) Handle(_ context.Context, r slog.Record) error { h.mu.Lock() defer h.mu.Unlock() h.records = append(h.records, r.Clone()) return nil } func (h *TestHandler) WithAttrs(attrs []slog.Attr) slog.Handler { return &TestHandler{records: h.records} } func (h *TestHandler) WithGroup(_ string) slog.Handler { return &TestHandler{records: h.records} } // Query helpers func (h *TestHandler) HasMessage(msg string) bool { h.mu.Lock() defer h.mu.Unlock() for _, r := range h.records { if strings.Contains(r.Message, msg) { return true } } return false } func (h *TestHandler) HasAttr(key, value string) bool { h.mu.Lock() defer h.mu.Unlock() for _, r := range h.records { found := false r.Attrs(func(a slog.Attr) bool { if a.Key == key && strings.Contains(a.Value.String(), value) { found = true return false // found, stop iteration for this record } return true }) if found { return true } } return false } ``` **Important:** The test handler captures `r.Message` separately from attributes. If your test assertions check for values that were moved from `fmt.Sprintf` to structured attributes during a migration, you need to update assertions to check attributes, not message strings. ```go // Usage in tests func TestRetryLogging(t *testing.T) { handler := &TestHandler{} logger := slog.New(handler) retrier := NewRetrier(logger) retrier.Execute(failingFunc) // Check message text assert.True(t, handler.HasMessage("Job failed, retrying")) // Check structured attributes assert.True(t, handler.HasAttr("attempt", "1")) assert.True(t, handler.HasAttr("maxRetries", "3")) } ``` ## Migration from logrus ### Level Mapping | logrus | slog | Notes | |--------|------|-------| | `Trace` | `Debug` | slog has no Trace; use Debug | | `Debug` | `Debug` | Direct mapping | | `Info` | `Info` | Direct mapping | | `Warn` / `Warning` | `Warn` | Direct mapping | | `Error` | `Error` | Direct mapping | | `Fatal` | `Error` + `os.Exit(1)` | slog has no Fatal; log then exit | | `Panic` | `Error` + `panic()` | slog has no Panic; log then panic | ### Callsite Conversion ```go // logrus printf-style logger.Debugf("loaded config from %s", path) logger.WithField("job", name).WithError(err).Error("execution failed") logger.WithFields(logrus.Fields{"job": name, "attempt": n}).Warn("retrying") // slog structured style logger.Debug("loaded config", "file", path) logger.Error("execution failed", "job", name, "error", err) logger.Warn("retrying", "job", name, "attempt", n) ``` ### Migration Checklist 1. Replace `Logger` interface/field types with `*slog.Logger` throughout 2. Replace `logrus.New()` with `slog.New(slog.NewTextHandler(...))` 3. Convert `logger.Debugf("msg %s", x)` to `logger.Debug("msg", "key", x)` 4. Replace `logrus.Fields{...}` with inline key-value pairs 5. Replace `WithError(err)` with `"error", err` attribute 6. Replace `WithField("k", v)` with `logger.With("k", v)` for persistent context 7. Delete custom Logger interfaces — `*slog.Logger` IS the interface 8. Delete logrus adapter/wrapper code 9. Update test loggers (see Testing section above) 10. Run `go mod tidy` to remove logrus from go.mod 11. Add logrus to depguard deny list in `.golangci.yml` 12. **Update CI workflows** — remove references to deleted logging packages ### CI Gotcha When deleting a logging package, check CI workflow files for references: ```yaml # BAD: References deleted package — CI will fail with [setup failed] go test -race ./core/... ./config/... ./logging/... # GOOD: Removed deleted package go test -race ./core/... ./config/... # BAD: Matrix includes deleted package package: [cli, core, config, logging, middlewares, web] # GOOD: Removed from matrix package: [cli, core, config, middlewares, web] ``` Always grep CI workflows after deleting any package: `grep -r "package-name" .github/workflows/` -
makefile.md 6.1 KB
# Standard Makefile Interface A consistent Makefile interface enables CI/CD automation and cross-project tooling. This defines the standard targets every Go project should implement. ## Required Targets | Target | Description | Exit Code | |--------|-------------|-----------| | `make test` | Run tests with race detection and coverage | 0 on pass, 1 on fail | | `make build` | Build the application binary | 0 on success | | `make lint` | Run golangci-lint | 0 on pass, 1 on issues | ## Recommended Targets | Target | Description | |--------|-------------| | `make fuzz` | Run fuzz tests (30s per target) | | `make vuln-check` | Run govulncheck | | `make security` | Run security checks (gosec, gitleaks) | | `make all` | lint + test + build | | `make clean` | Remove build artifacts | | `make fmt` | Format code | | `make setup` | Install dependencies and tools | | `make dev-check` | Pre-commit quality gates | ## Template ```makefile # Project Makefile # Implements standard interface for CI/CD compatibility # Build configuration VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev") COMMIT ?= $(shell git rev-parse --short HEAD 2>/dev/null || echo "unknown") BUILD_TIME := $(shell date -u +"%Y-%m-%dT%H:%M:%SZ") LDFLAGS := -s -w -X 'main.version=$(VERSION)' -X 'main.commit=$(COMMIT)' BUILDFLAGS := -ldflags="$(LDFLAGS)" -trimpath # Coverage settings COVERAGE_THRESHOLD := 80 COVERAGE_FILE := coverage.out .DEFAULT_GOAL := all # ============================================================================= # STANDARD INTERFACE (Required) # ============================================================================= .PHONY: all all: lint test build .PHONY: test test: ## Run tests with race detection and coverage @go test -race -coverprofile=$(COVERAGE_FILE) -covermode=atomic ./... @go tool cover -func=$(COVERAGE_FILE) | tail -n 1 .PHONY: build build: ## Build the application binary @CGO_ENABLED=0 go build $(BUILDFLAGS) -o bin/app ./cmd/app .PHONY: lint lint: ## Run golangci-lint @golangci-lint run --timeout 5m # ============================================================================= # RECOMMENDED TARGETS # ============================================================================= .PHONY: fuzz fuzz: ## Run fuzz tests (30s per target) @for pkg in $$(go list ./... | grep -v /vendor/); do \ for fuzz in $$(go test -list='^Fuzz' $$pkg 2>/dev/null | grep '^Fuzz'); do \ echo "Fuzzing $$fuzz in $$pkg..."; \ go test -fuzz=$$fuzz -fuzztime=30s $$pkg || exit 1; \ done; \ done .PHONY: vuln-check vuln-check: ## Run govulncheck @go run golang.org/x/vuln/cmd/govulncheck@latest ./... .PHONY: security security: ## Run security checks (gosec, gitleaks) @gosec ./... @gitleaks detect .PHONY: fmt fmt: ## Format code @gofmt -w $$(git ls-files '*.go') @goimports -w $$(git ls-files '*.go') .PHONY: vet vet: ## Run go vet @go vet ./... .PHONY: clean clean: ## Remove build artifacts @rm -rf bin/ $(COVERAGE_FILE) coverage-reports/ .PHONY: setup setup: ## Install dependencies and tools @go mod download @go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest @go install golang.org/x/vuln/cmd/govulncheck@latest .PHONY: dev-check dev-check: fmt vet lint security test ## Pre-commit quality gates @echo "All checks passed!" .PHONY: help help: ## Show this help @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | \ awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-15s\033[0m %s\n", $$1, $$2}' ``` ## CI Integration The standard interface enables simple CI workflows: ```yaml jobs: quality: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: go-version-file: go.mod - run: make lint - run: make test - run: make build ``` ## Coverage Enforcement ```makefile .PHONY: test-coverage test-coverage: test @COVERAGE=$$(go tool cover -func=$(COVERAGE_FILE) | grep "^total:" | awk '{print $$3}' | sed 's/%//'); \ if [ "$${COVERAGE%.*}" -lt "$(COVERAGE_THRESHOLD)" ]; then \ echo "Coverage $${COVERAGE}% below threshold $(COVERAGE_THRESHOLD)%"; \ exit 1; \ fi ``` ### The threshold is enforced over whatever `./...` matched `test` profiles `./...`, so the `total:` line mixes the library with every other package in the module — `examples/`, `testutil/`, generated helpers. The threshold is then a number about that mixture, and the mixture usually scores *higher* than the library: demo code is short, straight-line and exercised by a smoke test, so it lifts the average. The consequence is a gate that passes while the thing it guards sits under it. Measured on one library: `./...` 80.3%, library alone 77.8%, threshold 79 — the gate reported green on a figure 2.5 points above the code it exists to guard, which had never met it. Name the population the threshold describes, and profile that — the change belongs in `test`, where the profile is written: ```makefile COVERAGE_PACKAGES := . # or ./internal/... — the code the gate is about .PHONY: test test: @go test -race -coverprofile=$(COVERAGE_FILE) -covermode=atomic $(COVERAGE_PACKAGES) ``` Check it against the tool that reports coverage elsewhere: if `codecov.yml` carries `ignore:` entries, the gate should be measuring the same set, or the two numbers will disagree and each will look authoritative. Codecov has a separate failure mode where those `ignore:` entries silently never match — see `references/reusable-workflows.md` § Codecov. Recalibrating after narrowing the population is not weakening the gate: the old number described a set nobody was guarding. Say so in the commit, with both figures. ## Docker Integration ```makefile DOCKER_IMAGE := myapp DOCKER_TAG := $(VERSION) .PHONY: docker-build docker-build: ## Build Docker image @docker build -t $(DOCKER_IMAGE):$(DOCKER_TAG) . .PHONY: docker-test docker-test: ## Run tests in Docker @docker run --rm $(DOCKER_IMAGE):$(DOCKER_TAG) make test ``` ## Related - `references/linting.md` - golangci-lint configuration - `references/testing.md` - Testing patterns - `references/fuzz-testing.md` - Fuzz testing setup - `references/mutation-testing.md` - Mutation testing setup -
modernization.md 15.7 KB
# Go Modernization Patterns ## go fix — Automated Code Modernization Go 1.26 ships a rewritten `go fix` with 22 built-in modernizers. Run it on any codebase to apply idiomatic Go patterns automatically. ### Running go fix ```bash # Apply all applicable modernizers go fix ./... # Preview changes without applying (dry run) go fix -diff ./... # Apply specific modernizer only go fix -fix=any ./... ``` **A plain `go fix ./...` does not see a file behind a build tag, and reports nothing about it.** `//go:build integration` hides a file from the run, so a repository whose whole test tier sits behind tags comes back clean while every one of those files is untouched. Measured 2026-09-21 on two libraries: `go fix ./...` reported **0** findings in each, and `-tags=integration` then produced 38 and 24. Run it once per tag set the repository uses and take the union — a file behind `!integration` is only visible to the run *without* the tag, so neither run alone is enough: ```bash go fix ./... # files with no tag, and !tag files go fix -tags=integration,e2e ./... # files behind those tags ``` The same applies to the reporting form. To see which analyzer produced each finding rather than a diff, run the fix tool through `go vet`: ```bash go vet -vettool=$(go tool -n fix) -json -tags=integration ./... ``` Two more things stop `go fix` from seeing a package at all, and both look like "nothing to modernize": a failing `//go:embed` pattern (build the frontend assets first) and generated code that is not committed (run `templ generate`, `mockgen` and friends first). `go fix` exits non-zero and prints the load error, so check the exit status rather than the empty diff. ### Modernizer Reference | Modernizer | What it does | Example | |------------|-------------|---------| | `any` | `interface{}` → `any` | `func Foo(x interface{})` → `func Foo(x any)` | | `rangeint` | C-style loops → range | `for i := 0; i < n; i++` → `for i := range n` | | `slicescontains` | Manual contains loops → `slices.Contains()` | Loop+compare → `slices.Contains(s, v)` | | `mapsloop` | Manual map copy → `maps.Copy()` | for+assign → `maps.Copy(dst, src)` | | `minmax` | if/else capping → `min()`/`max()` builtins | if/else block → `min(a, b)` | | `stringscutprefix` | `HasPrefix`+`TrimPrefix` → `CutPrefix` | Two calls → `strings.CutPrefix(s, p)` | | `stringsseq` | `range strings.Split()` → `SplitSeq()` | Avoids allocating intermediate slice | | `waitgroup` | `wg.Add(1)/go/defer wg.Done()` → `wg.Go()` | Three lines → `wg.Go(func() { ... })` | | `testingcontext` | `context.WithCancel(context.Background())` → `t.Context()` | In tests only | | `reflecttypefor` | `reflect.TypeOf((*T)(nil)).Elem()` → `reflect.TypeFor[T]()` | Cleaner generic form | | `stringsbuilder` | `output += s` → `strings.Builder` | Better performance for string concatenation | ### Proving a construct needs the new language version `go fix` only applies modernizers your `go.mod` allows, so after a directive raise some rewrites are not style choices — they are constructs the previous language version rejected. Do not assert that from the release notes. Build the same file under both directives and let the compiler say so: ```bash mkdir -p /tmp/langprobe && cd /tmp/langprobe cat > p.go <<'EOF' package p type Inner struct{ A string } type Outer struct { Inner B string } var _ = Outer{A: "x", B: "y"} EOF for v in 1.26.0 1.27.0; do { echo "module langprobe"; echo; echo "go $v"; } > go.mod out=$(go build ./... 2>&1); rc=$? if [ $rc -eq 0 ]; then printf 'go %s: compiles\n' "$v" else printf 'go %s: %s\n' "$v" "$out"; fi done ``` Capture the status; do not pipe it. `go build ./... | tail -1` reports `tail`'s exit status, so a failed build looks like a success and the branch printing the success marker never runs — which is how a plausible transcript ends up in a document without ever having been produced by the script above it. ``` go 1.26.0: # langprobe ./p.go:9:15: use of promoted field Inner.A in struct literal of type Outer requires go1.27 or later (-lang was set to go1.26; check go.mod) go 1.27.0: compiles ``` The diagnostic names the version, which turns "this is a 1.27 feature" from a claim into evidence — worth putting in the pull request, because a review bot whose model predates the toolchain will report exactly these lines as a compile error and propose undoing them. **Go 1.27: promoted fields in composite literals.** `Outer{A: "x"}` for a field promoted from an embedded struct is the rewrite most visible after raising the directive to 1.27. The modernizer is `embedlit`; on the same tree with `go 1.26.0` in `go.mod` it produces no diff at all, which is the gating in action. An embedded field whose promoted name equals its own type name cannot be flattened — `Unicode: Unicode{Unicode: "yes"}` stays wrapped, because `Unicode:` in that literal is the embedded field, not the promoted string. A mixed result is correct, not an inconsistency. Confirm a flattened literal still builds the same value before trusting it: a `reflect.DeepEqual` comparison against the wrapped form costs one throwaway test and rules out the promoted name resolving to a different field. Give that test a control case comparing deliberately different values — a comparison that cannot fail proves nothing. **`atomictypes` reaches further than the call sites it rewrites.** It changes a field's *type*, and a struct walked by reflection is walked differently afterwards — `int32` is a leaf, `atomic.Int32` is a struct with fields of its own. `embedlit` does not have this property: it rewrites composite literals and never a struct definition, so it leaves reflection untouched. Where a struct's identity is derived by reflection — a config hash, a cache key, a change detector — compare that derived value before and after on the same input rather than reasoning about it. On ofelia, `atomictypes` turned a job's `running int32` into `atomic.Int32`, and the job hash walked that struct field-by-field, recursing into any field of kind Struct *before* checking its tag; the hash happened to stay byte-identical, but nothing in the diff said so. ### go fix Best Practices 1. **Run after upgrading Go** — `go fix` detects your `go.mod` version and only applies applicable modernizers 2. **Run it once per build-tag set** — see above; a plain run reports nothing about tagged files 3. **Review the diff** — Use `go fix -diff ./...` first to understand what changes will be made 4. **Run the repo's own formatter after, not plain `gofmt`** — `go fix` may leave behind unused imports, redundant variables, or gofumpt issues. The `embedlit` rewrite in particular produces literals gofumpt rejects, which plain `gofmt` accepts. Use `golangci-lint fmt` or whatever the repository's lint job checks. 5. **Commit separately** — Keep `go fix` changes in their own commit for clean history ### Common Post-fix Cleanup After `go fix`, watch for: ```go // go fix may leave redundant loop variable copies (Go 1.22+) for field := range t.Fields() { field := field // ← delete this (copyloopvar lint) // ... } // go fix may inline helpers and leave them unused //go:fix inline func stringPtr(s string) *string { // ← delete if unused return new(s) } ``` ## errors.AsType[T] (Go 1.26) Go 1.26 adds `errors.AsType[T]` — a type-safe generic replacement for `errors.As` that eliminates pre-declared target variables. ### Before (errors.As) ```go var flagErr *flags.Error if errors.As(err, &flagErr) { if flagErr.Type == flags.ErrHelp { return } } ``` ### After (errors.AsType) ```go if flagErr, ok := errors.AsType[*flags.Error](err); ok { if flagErr.Type == flags.ErrHelp { return } } ``` ### Common Conversion Patterns **Positive check with value use:** ```go // Before var exitErr NonZeroExitError if errors.As(err, &exitErr) { log.Printf("exit code: %d", exitErr.ExitCode) } // After if exitErr, ok := errors.AsType[NonZeroExitError](err); ok { log.Printf("exit code: %d", exitErr.ExitCode) } ``` **Negative check (guard clause):** ```go // Before var validationErrors validator.ValidationErrors if !errors.As(err, &validationErrors) { return fmt.Errorf("unexpected error: %w", err) } // After validationErrors, ok := errors.AsType[validator.ValidationErrors](err) if !ok { return fmt.Errorf("unexpected error: %w", err) } ``` **Bool-only check (discard value):** ```go // Before func IsNonZeroExitError(err error) bool { var exitErr NonZeroExitError return errors.As(err, &exitErr) } // After func IsNonZeroExitError(err error) bool { _, ok := errors.AsType[NonZeroExitError](err) return ok } ``` ### Why errors.AsType is Better | `errors.As` | `errors.AsType[T]` | |---|---| | Requires pre-declared target variable | No variable declaration needed | | Type safety checked at runtime | Type checked at compile time | | `errors.As(err, &target)` — pointer indirection | `errors.AsType[T](err)` — direct generic | | Target variable leaks into outer scope | Scoped to `if` block with `:=` | ### Migration Go 1.27's `go fix` ships an `errorsastype` analyzer that performs this rewrite, so it is no longer purely manual — but it does not catch every shape. Run the tool first, then grep for what it left: ```bash go fix ./...; echo "untagged: $?" go fix -tags=integration,e2e ./...; echo "tagged: $?" grep -rn 'errors\.As(' --include='*.go' . ``` The two runs are separate statements on purpose. `go fix` exits non-zero on a package-load error, so chaining them with `&&` lets one failing run silently skip the other and leaves the union incomplete — which is the very gap this section exists to close. Read both exit statuses. Measured 2026-09-21 across seven repositories: the analyzer offered one rewrite, in a positive `if errors.As(err, &target)` check, and a grep then found seven further call sites it had not touched — every one of them in an `if !errors.As(...)` guard or a `return errors.As(...)` body. Treat a clean `go fix` as a partial pass and finish the remainder by hand. ## sync.WaitGroup.Go (Go 1.25) `sync.WaitGroup` gained a `Go` method that combines `Add(1)`, goroutine launch, and `defer Done()`: ```go // Before var wg sync.WaitGroup for range 20 { wg.Add(1) go func() { defer wg.Done() doWork() }() } wg.Wait() // After var wg sync.WaitGroup for range 20 { wg.Go(func() { doWork() }) } wg.Wait() ``` `go fix` handles this conversion automatically via the `waitgroup` modernizer. ## testing.B.Loop (Go 1.24) — and the three benchmarks that must keep b.N `b.Loop` manages the benchmark timer itself, keeps the loop body's values alive so the compiler cannot delete the measured work, and runs the benchmark function once per measurement instead of re-running it with a growing `N`. `go fix` does **not** perform this rewrite, so it survives a clean modernizer pass: ```go // Before b.ResetTimer() for i := 0; i < b.N; i++ { doWork() } // After — the ResetTimer is now redundant, b.Loop resets on its first call for b.Loop() { doWork() } ``` Where the body reads the index, declare the counter outside: ```go i := 0 for b.Loop() { doWork(i % 10) i++ } ``` **Three shapes must keep `b.N`.** Converting them is not a style regression, it is a defect — the first two were found by running the benchmarks, not by reading the code: 1. **The timer is stopped when `b.Loop()` is called.** b.Loop requires a running timer at each call and aborts otherwise with `benchmark.go:417: B.Loop called with timer stopped`. Manual timer control is *not* itself forbidden: `b.StopTimer()` / `b.StartTimer()` **balanced inside** the body, so that the timer runs again before the next `b.Loop()`, is fine and measures 185 ns/op. What fails is the older shape that stops the timer before the loop and again as the body's last statement: ```go b.StopTimer() // ← aborts: timer stopped at the first b.Loop() for b.Loop() { setup() b.StartTimer() work() b.StopTimer() // ← and stopped again at every later one } ``` 2. **More than one `b.N` loop in one benchmark scope.** There is only one loop that measures; another sizes its setup by `b.N` (pre-populate exactly `b.N` cache entries, then delete one per iteration). b.Loop cannot express that: its iteration count is decided as it runs, and `b.N` is only meaningful *after* it returns false. 3. **A body with `continue` or `goto`,** where a trailing `i++` would be skipped. Verify a conversion by executing every benchmark once, not by the unit suite — a broken conversion shows up as a benchmark that no longer runs: ```bash go test -run='^$' -bench=. -benchtime=1x ./... go test -run='^$' -bench=. -benchtime=1x -tags=integration,e2e ./... ``` **`for i := 0; b.Loop(); i++` also keeps the body alive**, so it is a legitimate way to carry an index without a separate counter. The documentation's wording ("the loop condition must be written exactly as `b.Loop()`") reads as if only `for b.Loop() { … }` qualified, and three benchmarks built to separate the two spellings could not tell them apart. The compiler settles it: `cmd/compile/internal/bloop.isTestingBLoop` accepts any `ir.OFOR` whose `Cond` is a call to `testing.(*B).Loop`, and inspects neither the loop's `Init` nor its `Post`. A three-clause loop is an `OFOR`, so it gets the same `runtime.KeepAlive` wrapping. Pick whichever of the two reads better; do not pick on a belief about keep-alive. ## new(expr) — Pointer to Value (Go 1.26) Go 1.26 extends `new()` to accept expressions (not just types), returning a pointer to a copy: ```go // Before — temporary variable needed func stringPtr(s string) *string { return &s } // After — direct construction p := new("hello") // *string pointing to "hello" n := new(42) // *int pointing to 42 ``` `go fix` can inline helper functions annotated with `//go:fix inline` that follow this pattern. ## for range n (Go 1.22) Integer range loops replace C-style counting: ```go // Before for i := 0; i < 10; i++ { fmt.Println(i) } // After for i := range 10 { fmt.Println(i) } // When index is unused for range 10 { doSomething() } ``` ## Loop Variable Capture Fix (Go 1.22) Go 1.22 fixed loop variable capture — the `tt := tt` shadow is no longer needed: ```go // Before (Go < 1.22) — required to prevent capture bug for _, tt := range tests { tt := tt // ← was needed t.Run(tt.name, func(t *testing.T) { t.Parallel() // ... }) } // After (Go 1.22+) — safe without shadow for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { t.Parallel() // ... }) } ``` The `copyloopvar` linter flags unnecessary copies. ## t.Context() (Go 1.24) Tests can use `t.Context()` instead of manually creating background contexts: ```go // Before func TestSomething(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() // use ctx... } // After func TestSomething(t *testing.T) { ctx := t.Context() // cancelled automatically when test ends // use ctx... } ``` `go fix` handles this via the `testingcontext` modernizer. ## Version-Gated Features Summary | Feature | Minimum Go | go fix? | |---------|-----------|---------| | `any` keyword | 1.18 | Yes | | Generics | 1.18 | N/A | | `for range n` | 1.22 | Yes | | Loop variable fix | 1.22 | N/A | | `min()`/`max()` builtins | 1.21 | Yes | | `slices.Contains()` | 1.21 | Yes | | `maps.Copy()` | 1.21 | Yes | | `strings.CutPrefix()` | 1.20 | Yes | | `t.Context()` | 1.24 | Yes | | `sync.WaitGroup.Go()` | 1.25 | Yes | | `strings.SplitSeq()` | 1.25 | Yes | | `errors.AsType[T]()` | 1.26 | Partly (`errorsastype`, Go 1.27; finish by hand) | | `testing.B.Loop()` | 1.24 | No (manual) | | `new(expr)` | 1.26 | Yes | | `reflect.TypeFor[T]()` | 1.22 | Yes | -
mutation-testing.md 9.6 KB
# Go Mutation Testing Mutation testing measures test quality by introducing small code changes (mutations) and verifying tests detect them. Higher scores indicate more effective tests. ## Tool: Gremlins [go-gremlins](https://github.com/go-gremlins/gremlins) is the recommended mutation testing tool for Go. ```bash # Install go install github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 # Run gremlins unleash --config=.gremlins.yaml ``` ## Configuration Create `.gremlins.yaml` in project root: ```yaml # Packages to test test-packages: - . - ./cmd/... - ./internal/... # Mutator types to enable mutators: - CONDITIONALS_BOUNDARY # Change < to <=, > to >= - CONDITIONALS_NEGATION # Negate conditions (== to !=) - INCREMENT_DECREMENT # Change ++ to -- - INVERT_LOGICAL # Invert && to || - INVERT_NEGATIVES # Remove negation operators - INVERT_LOOPCTRL # Change break to continue # Files/patterns to exclude exclude: - "**/*_test.go" # Test files - "**/test/**" # Test helpers - "**/mock/**" # Mock implementations - "**/generated/**" # Generated code # Only mutate code covered by tests coverage: true # Minimum acceptable mutation score (%) threshold: 60 # Timeout multiplier for test runs timeout-coefficient: 5 # Output reports output: json: mutation-report.json html: mutation-report.html # Test timeout test-timeout: 120s ``` ## Understanding Results | Metric | Meaning | |--------|---------| | **Killed** | Tests detected the mutation (good!) | | **Survived** | Tests missed the mutation (needs improvement) | | **Timed Out** | Tests hung on mutation (usually killed) | | **Skipped** | Excluded from analysis | **Test Efficacy** = (Killed + Timed Out) / Total Mutations Target: **60%+ for production code** ## CI Integration ### GitHub Actions Workflow ```yaml name: Mutation Testing on: push: branches: [main] pull_request: paths: - '**.go' - '.gremlins.yaml' jobs: mutation: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: go-version-file: go.mod - name: Install gremlins run: go install github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 - name: Run mutation tests run: | gremlins unleash --config=.gremlins.yaml 2>&1 | tee output.txt SCORE=$(grep -oP 'Test efficacy: \K[\d.]+' output.txt || echo "0") echo "Mutation Score: ${SCORE}%" if (( $(echo "$SCORE < 60" | bc -l) )); then echo "::warning::Mutation score below 60%" fi # The `|| echo "0"` above is load-bearing in a way that hides failure: # gremlins compiles the packages it mutates, and one it cannot build is # reported as "[build failed]" — after which it stops without printing # "Test efficacy" at all. The fallback then makes the job publish 0% and # pass, which reads as a measured result rather than a run that never # happened. Check for it explicitly. - name: Fail on packages gremlins could not build if: always() run: | [ -f output.txt ] || exit 0 grep -q '\[build failed\]' output.txt || exit 0 echo "::error::gremlins could not build these packages:" grep '\[build failed\]' output.txt exit 1 - name: Upload reports uses: actions/upload-artifact@v4 with: name: mutation-reports path: | mutation-report.json mutation-report.html ``` ### Diff-Based Testing (PRs only) For efficiency, only test mutations in changed files on PRs: ```yaml - name: Run mutation tests (diff only) if: github.event_name == 'pull_request' run: | BASE_REF="${{ github.event.pull_request.base.sha }}" gremlins unleash --config=.gremlins.yaml --diff "$BASE_REF" ``` ## A Score of 0% Means "Nothing Ran" gremlins does not partially degrade. If any package it mutates fails to compile, it prints `[build failed]` for that package and stops — no efficacy line is emitted at all. Every score-extraction idiom in the wild falls back to `0` when the line is missing, so the job publishes **0%** and passes. This is easy to ship and hard to notice, because a low score looks like a test-quality problem rather than a run that never happened. Two things cause it in practice: - **Generated sources that are not committed.** templ, sqlc, mockgen and friends produce `*.go` files that CI must generate before gremlins runs. A repo where `go test` works locally (generated files present) fails here. Pass the codegen command in a pre-build step. - **Build tags.** A package that only compiles under `integration` or `e2e` fails the default build. Put the `[build failed]` check in its **own step**. The run step usually carries `continue-on-error: true` so a low score does not fail the workflow — and that would swallow the build check too. A missing score is a threshold question; a package that does not compile is not. ## Makefile Integration ```makefile .PHONY: mutation mutation: @echo "Running mutation tests..." @gremlins unleash --config=.gremlins.yaml .PHONY: mutation-report mutation-report: mutation @echo "Opening mutation report..." @open mutation-report.html 2>/dev/null || xdg-open mutation-report.html ``` ## Improving Mutation Score ### Common Surviving Mutations 1. **Boundary conditions** - Add tests for `<` vs `<=`, `>` vs `>=` 2. **Error paths** - Test both success and failure cases 3. **Loop controls** - Verify break/continue behavior 4. **Negation** - Test both true and false conditions 5. **Increment/Decrement** - Check exact values, not just "changed" ### Example: Fixing a Survivor ```go // Original code func IsValid(x int) bool { return x > 0 // Mutation: x >= 0 survives } // Original test (insufficient) func TestIsValid(t *testing.T) { assert.True(t, IsValid(1)) // x > 0 and x >= 0 both pass assert.False(t, IsValid(-1)) // x > 0 and x >= 0 both fail } // Fixed test (kills the mutation) func TestIsValid(t *testing.T) { assert.True(t, IsValid(1)) assert.False(t, IsValid(-1)) assert.False(t, IsValid(0)) // Boundary case kills x >= 0 } ``` ## Hand-rolled mutations: a build failure is not a caught defect Gremlins is the tool, but a targeted question — "does anything actually pin this one line?" — is usually answered faster by injecting the defect yourself. That loop has a failure mode the tool does not have, and it reports the wrong answer in the reassuring direction. Removing a call often orphans its import. `go test ./...` then exits non-zero on `"slices" imported and not used`, the loop sees a non-zero exit, and prints CAUGHT for a mutation no assertion ever saw. The run measured the compiler. Two rules follow: - **Every mutation must build clean before its result counts.** Check for a compiler diagnostic (`# package` lines, `[build failed]`), not just the exit code. Where a mutation would orphan a symbol, keep it referenced — `_ = slices.Clone(x)`, `_ = uuid.New()` — so the defect is the only change. - **Restore from a copy, never `git checkout -- <file>`.** That restores the last *committed* state, discarding the guard you just wrote and have not committed. `cp file /tmp/x.bak` first, `cp` back after each mutation, and run the full suite at the end to prove the tree is the one you think it is. ```bash cp pkg/thing.go /tmp/thing.bak # ... inject, then: if ! out=$(go vet ./... 2>&1); then echo "DOES NOT BUILD — not evidence:"; echo "$out" else go test ./...; echo "exit=$?" fi cp /tmp/thing.bak pkg/thing.go ``` The guard has to come first and it has to stop. `go build ... || echo warning` prints the warning and then runs the suite anyway, so the compile failure still reaches the exit code the loop reads — the sample would demonstrate the defect it exists to prevent. Keep the diagnostic rather than discarding it to `/dev/null`: "which mutation failed to build" is the thing you need next. `go vet` is the gate rather than `go build`, because `go test` runs vet too. A mutation that builds and only trips vet — a `%d` verb given a string, say — passes a `go build` guard and then produces the identical `FAIL [build failed]` the section is about. A mutation that survives is the finding. Before writing the test that catches it, check the assertion will reach the mutated code at all: a test that rebuilds the production expression in its own body holds whatever production does, so it stays green through every mutation of the real thing. Extract the expression into a named function and have production call it. Then have the test call it for the **actual** value only. The expected value must come from somewhere the mutation cannot move: a literal, or an invariant of the result. Calling the extracted function on both sides reproduces the original defect one level up — a mutation shifts both sides together and the test stays green. "The id splits into two parts on the underscore, the first is this literal GUID, the second parses as a GUID, and two calls differ" are invariants; "equals `membershipID(group)`" is not an assertion at all. ## Best Practices 1. **Start with 60% threshold** - Increase as tests mature 2. **Exclude generated code** - Focus on hand-written logic 3. **Use coverage mode** - Only mutate tested code 4. **Run on CI** - Catch regressions early 5. **Diff mode for PRs** - Full runs on main branch only ## Related - `references/testing.md` - General testing patterns - `references/fuzz-testing.md` - Complementary input validation testing - [go-gremlins documentation](https://github.com/go-gremlins/gremlins) -
resilience.md 751 B
# Resilience Patterns in Go For scheduled jobs, retry-with-backoff, circuit breaker, and timeout wrappers are built into go-cron — don't hand-roll them. See `references/cron-scheduling.md` § Resilience Wrappers (`RetryWithBackoff`, `RetryOnError`, `CircuitBreaker`) and § Concurrency Wrappers (`Timeout`, `TimeoutWithContext`). Outside the job scheduler (HTTP clients, background workers), exponential backoff, circuit breakers, graceful shutdown, rate limiting, and health checks are standard patterns backed by well-known libraries (`golang.org/x/time/rate` for rate limiting; `context` + `sync.WaitGroup` + `os/signal` for graceful shutdown) — no Netresearch-specific convention beyond what's documented in `references/cron-scheduling.md`. -
reusable-workflows.md 9.8 KB
# Reusable Workflows for Go Repos Patterns for unifying CI/CD across multiple Go repositories by calling shared reusable workflows instead of duplicating action configuration per-repo. ## Why Four repos that each re-declare their own `golangci-lint`, `govulncheck`, build, and release jobs drift apart. Pinning actions becomes 4× the work; a new linter rule must be added in 4 places; a security bump must be chased through 4 repos. Reusable workflows fix this by making the caller a thin YAML file that passes inputs and forwards permissions. ## Caller File (Minimal) ```yaml # .github/workflows/ci.yml in a Go repo that delegates to a shared workflow name: CI on: push: branches: [main] pull_request: permissions: contents: read jobs: go-ci: uses: netresearch/shared-ci/.github/workflows/go-ci.yml@v3 with: go-version: "1.23" lint-timeout: 5m permissions: contents: read checks: write # required if the reusable workflow writes check annotations pull-requests: read # required if the reusable workflow reads PR metadata secrets: CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} ``` ## Permission Propagation (The Recurring Gotcha) Reusable workflows run under the **caller's** token, with permissions capped by the caller's `permissions:` block. Two failure modes: 1. **Caller too narrow**: Caller declares `permissions: contents: read`, but the reusable workflow needs `checks: write` to publish annotations → annotations silently don't appear. 2. **Caller uses `secrets: inherit`**: Exposes every org secret to every action in the reusable workflow, including transitively called third-party actions. High blast radius on supply chain compromise. ### Rules - **Explicit per-permission grants at the caller job level**, matching what the reusable workflow documents as required. Never just `permissions: write-all`. - **Explicit per-secret forwarding**: `secrets: { CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} }`. Never `secrets: inherit`. - **Reusable workflow documents its required permissions** in a comment block at the top of its file — treat this as contract. ### Required-permissions table (document in the reusable workflow) ```yaml # Required caller permissions: # contents: read # checkout # checks: write # annotations from golangci-lint-action, gotestsum # pull-requests: read # auto-merge eligibility check # security-events: write # govulncheck SARIF upload (only if scan-mode is enabled) # # Required caller secrets (forward explicitly): # CODECOV_TOKEN # optional; skip if code-coverage-upload: false ``` ## Exposing Release Gates via Outputs Release workflows need the SHA and tag emitted by the build job so downstream jobs (provenance, attestation, signing) can attach artifacts. Expose them as job outputs: ```yaml # shared-ci go-release.yml (reusable) on: workflow_call: outputs: image_digest: value: ${{ jobs.build.outputs.digest }} tag: value: ${{ jobs.build.outputs.tag }} jobs: build: outputs: digest: ${{ steps.buildx.outputs.digest }} tag: ${{ steps.meta.outputs.tags }} # ... ``` Caller consumes outputs: ```yaml jobs: release: uses: netresearch/shared-ci/.github/workflows/go-release.yml@v3 # ... provenance: needs: release uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0 with: image: "ghcr.io/${{ github.repository }}" digest: ${{ needs.release.outputs.image_digest }} ``` > **Cannot run under a SHA-pinning ruleset.** `generator_container_slsa3.yml` calls two nested actions by tag — `detect-workflow-js` and `generate-builder` — in `@v2.0.0` as used above and in `@v2.1.0`, the latest release. A repository or organisation with `sha_pinning_required` on rejects the run at the first of them. Pinning the `uses:` line above to a SHA does not help: the rejected references are inside the generator, and it refuses to run from a SHA anyway. Upstream [slsa-github-generator#4440](https://github.com/slsa-framework/slsa-github-generator/issues/4440) is open. The fallback there is `actions/attest-build-provenance`, which is a step action rather than a reusable workflow, so it replaces the job rather than its `uses:` line — for an image, `subject-name` plus the `subject-digest` the build emitted, in a job holding `id-token: write` and `attestations: write`. It attests to what GitHub can witness, which is not the isolated builder Level 3 denotes: state the level you reach. Without exposed outputs, release gates degrade to "find the SHA by re-querying the registry" — racy, slow, and often wrong. ## Pinning Reusable Workflows Always reference reusable workflows by a released tag (or SHA), never by `@main`: ```yaml uses: netresearch/shared-ci/.github/workflows/go-ci.yml@v3 # preferred uses: netresearch/shared-ci/.github/workflows/go-ci.yml@<40-char-SHA> # most strict uses: netresearch/shared-ci/.github/workflows/go-ci.yml@main # AVOID — moving target ``` `@main` means your CI behavior can change without a PR in the caller repo. A surprise lint failure from a reusable-workflow bump has no changelog in your repo — treat this as a supply-chain risk. ## Migrating From Per-Repo Actions to Callers Incremental migration: 1. Publish the reusable workflow in `shared-ci` with a `v0.x` tag. 2. In one pilot Go repo, swap the existing `.github/workflows/ci.yml` for a caller — open as PR, observe CI behavior for at least one merge cycle. 3. Roll out to remaining repos only after the pilot is stable for a week. 4. Delete per-repo action configs after callers are merged and CI is green on each. Do not batch steps 2-4 across all repos on day one. One pilot, then fan out. ## Codecov: Go names files by import path, so `ignore` silently never fires A Go coverage profile identifies files by import path, so every entry leaves the runner as `github.com/<org>/<repo>/users.go` rather than `users.go`. Codecov ships default path fixes that are meant to normalise this, but they do not always match — on one repository the prefix survived into the stored report untouched. **Check before prescribing anything:** pull the report and look at the names. ```bash curl -s "https://api.codecov.io/api/v2/github/<org>/repos/<repo>/commits/<sha>/" \ | jq -r '.report.files[].name' | head ``` If the entries come back prefixed, two things are already broken quietly: - **`ignore:` stops matching.** Codecov compiles `examples/**` to `(?s:examples/.*)\Z`, anchored at the start, which a prefixed path cannot satisfy. Excluded directories stay in the project total at 0% and drag it down. - **Codecov cannot map a report entry to a file in the repository**, so files can read 0% against a suite that covers them, and uploads merge unpredictably. Only then add a `fixes:` entry with the module path — it is the remedy for a prefix the defaults did not strip, not a rule to apply blind: ```yaml fixes: - "github.com/<org>/<repo>/::" # strip the import-path prefix ``` The tell is a project percentage that does not move: totals identical across commits that changed the tree — `lines` and `hits` equal to the digit — can indicate a report that is no longer being recomputed rather than coverage that happens to be stable. It is a signal, not a proof: the same totals are legitimate when the commits in between touched only files outside the uploaded profile. Confirm by checking what those commits changed against the paths the report actually carries before concluding anything. On one repository this stood at 19.36% for thirty-two commits; after the `fixes:` entry the same tree measured 88.77%, with the ignored directories gone from the report and a previously absent flag appearing in it. Validate before merging — the endpoint both checks the file and prints how each pattern compiled, which is how you confirm an `ignore` entry can match at all: ```bash curl -X POST --data-binary @codecov.yml https://codecov.io/validate ``` Two neighbouring traps once the paths are correct: - **Per-flag `paths:` filters** are worth a second look, not an automatic deletion. Path fixes are applied before entries are mapped, so a repository-relative glob such as `"**/*.go"` that could not match a prefixed entry beforehand matches the normalised path afterwards — repairing `fixes:` may be all the filter needed. Remove it only where it exists purely to compensate for the prefix; where it scopes a flag to a subproject or directory, dropping it silently widens what that flag and its status cover. - **Repairing the paths can arm a gate that was inert.** A `patch` status with `target: 100%` posts nothing while the report is broken; the moment paths resolve it starts failing pull requests against a threshold nobody chose. Set it `informational: true` in the same change and calibrate the target from the first corrected run. ## Common Anti-Patterns | Anti-pattern | Consequence | Fix | |--------------|-------------|-----| | `secrets: inherit` | All org secrets exposed to every action in the chain | Explicit `secrets: { NAME: ${{ secrets.NAME }} }` per secret | | Caller `permissions: write-all` | Overbroad; blast radius on compromised action | Enumerate only what the reusable workflow documents | | Reusable workflow pinned `@main` | Silent behavior drift | Pin to released tag or full SHA | | Missing outputs on reusable workflow | Downstream release gates can't get SHA/tag | Declare job outputs + workflow outputs | | Per-repo duplication of `golangci-lint` config | Drift across repos; 4× maintenance | Centralize in reusable workflow + `.golangci.yml` template | | `version: latest` for `golangci-lint-action` in CI | CI behavior drifts from local runs; rules fire in one but not the other | Pin the golangci-lint version in both; see `references/linting.md` → *golangci-lint: CI vs Local Version Drift* for the dual-`nolint` workaround when pinning isn't possible | -
single-build-release.md 13.2 KB
# Single-Build Release Pipeline for Go Apps Cross-compile every target once, publish the binaries as GitHub Release assets, then re-use those same binaries to assemble the container image. One `go build` per platform serves both release-page downloads AND the image push — no second compile in a Dockerfile, no drift between the tarball and the image. This is the fleet convention for every go-app repo (ofelia, raybeam, ldap-manager, ldap-selfservice-password-changer). The template for it lives in [`netresearch/.github/templates/go-app`](https://github.com/netresearch/.github/tree/main/templates/go-app) and consumers sync from it; this doc explains the moving parts so you can debug a specific repo or introduce a new one. ## Quick index | Piece | Where | |---|---| | Matrix cross-compile + release upload | [binaries job](#binaries-job) | | Assemble image from matrix output | [container job](#container-job) | | Binary-selector Dockerfile | [Dockerfile](#dockerfile) | | Version metadata convention | [ldflag convention](#ldflag-convention) | | Package entrypoint detection | [main-package: auto](#main-package-auto) | | Reliable build timestamp | [auto-build-timestamp](#auto-build-timestamp) | ## Why (in one sentence) If you publish binaries AND images, the naive setup compiles Go twice per platform per release — once in a matrix job, again inside the Dockerfile. That's wasted CI time, two slightly-different artifacts for the same tag, and a second place for ldflags/build-tags to drift. ## Pipeline shape ``` push tag v* └── create-release.yml → creates the release shell + outputs tag/sha └── binaries matrix → build-go-attest.yml × 8 targets │ (linux-{386,amd64,arm64,armv6,armv7}, │ darwin-{amd64,arm64}, windows-amd64) │ uploads every binary as a release asset └── container job → build-container.yml (5 linux platforms) │ with pre-build-command that runs │ `gh release download --pattern <name>-linux-*` │ to populate bin/ in the build context └── finalize → checksums + cosign + verification notes ``` Binaries for `linux/386` etc. land in `bin/<name>-linux-386`. The Dockerfile's binary-selector stage picks the right one per `TARGETARCH`/`TARGETVARIANT`. ## binaries job Calls `netresearch/.github/.github/workflows/build-go-attest.yml@main` from a matrix. Key inputs: ```yaml strategy: fail-fast: false matrix: include: - { target: linux-386, goos: linux, goarch: "386" } - { target: linux-amd64, goos: linux, goarch: amd64 } - { target: linux-arm64, goos: linux, goarch: arm64 } - { target: linux-armv6, goos: linux, goarch: arm, goarm: "6" } - { target: linux-armv7, goos: linux, goarch: arm, goarm: "7" } - { target: darwin-amd64, goos: darwin, goarch: amd64 } - { target: darwin-arm64, goos: darwin, goarch: arm64 } - { target: windows-amd64, goos: windows, goarch: amd64 } uses: netresearch/.github/.github/workflows/build-go-attest.yml@main with: binary-name: ${{ github.event.repository.name }}-${{ matrix.target }} main-package: auto goos: ${{ matrix.goos }} goarch: ${{ matrix.goarch }} goarm: ${{ matrix.goarm || '' }} ldflags: >- -s -w -X main.version=${{ needs.create-release.outputs.tag }} -X main.build=${{ needs.create-release.outputs.sha }} ref: ${{ needs.create-release.outputs.tag }} release-tag: ${{ needs.create-release.outputs.tag }} sbom: true setup-bun: true pre-build-command: | if [ -f package.json ]; then bun install --frozen-lockfile bun run build:assets fi auto-build-timestamp: true ``` Every matrix entry produces one binary file and uploads it to the release. `sbom: true` adds a Syft SBOM alongside each binary. ## container job Depends on `binaries` so the release has the assets ready: ```yaml container: needs: [create-release, binaries] uses: netresearch/.github/.github/workflows/build-container.yml@main permissions: contents: read packages: write security-events: write # required for trivy SARIF upload id-token: write attestations: write with: image-name: ${{ github.event.repository.name }} ref: ${{ needs.create-release.outputs.tag }} platforms: "linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64" sign: true attest: true pre-build-command: | set -euo pipefail mkdir -p bin for suffix in linux-386 linux-amd64 linux-arm64 linux-armv6 linux-armv7; do gh release download "${{ needs.create-release.outputs.tag }}" \ --pattern "${{ github.event.repository.name }}-${suffix}" --dir bin chmod +x "bin/${{ github.event.repository.name }}-${suffix}" done ``` The `pre-build-command` runs inside `build-container.yml` after checkout and before `docker buildx build`. It pulls the binaries the matrix just published and drops them into `bin/` where the Dockerfile expects them. `GH_TOKEN` is pre-populated by the reusable workflow. **5 platforms by default.** If a platform is incompatible (e.g. Fiber v3 on `linux/arm` overflows int on 32-bit), narrow the matrix AND the `platforms:` string in the caller and record the reason in the repo's `.github/template.yaml` `intentional-drift:` list. ## Dockerfile The binary-selector pattern: one stage copies all pre-built Linux binaries in, picks the right one by `TARGETARCH`/`TARGETVARIANT`, then a minimal runtime stage copies only the selected binary. ```dockerfile # Binary-selector stage — picks the pre-built binary for the target # platform. The release binaries matrix produces bin/<name>-linux-*; # the container job's pre-build-command downloads them here before # `docker buildx build`. FROM alpine:3.23 AS binary-selector ARG TARGETARCH ARG TARGETVARIANT COPY bin/<name>-linux-* /tmp/ RUN set -eux; \ case "${TARGETARCH}" in \ arm) BINARY="<name>-linux-arm${TARGETVARIANT}" ;; \ 386|amd64|arm64) BINARY="<name>-linux-${TARGETARCH}" ;; \ *) echo "Unsupported: ${TARGETARCH}" >&2; exit 1 ;; \ esac; \ cp "/tmp/${BINARY}" /usr/bin/<name>; \ chmod +x /usr/bin/<name> # Runtime stage — pick the minimal base that fits your runtime needs. # `scratch` / `distroless/static:nonroot` / `alpine:3.23` are common. FROM alpine:3.23 COPY --from=binary-selector /usr/bin/<name> /usr/bin/<name> ENTRYPOINT ["/usr/bin/<name>"] ``` **Notes:** - `TARGETARCH` is `386|amd64|arm64|arm` on buildx's standard platforms; `TARGETVARIANT` is `v6|v7` only when `TARGETARCH=arm`. - `FROM scratch` works when the Go binary is fully static. Use `distroless/static-debian12:nonroot` for a cleaner nonroot + CA bundle combo. Use `alpine` only when you need a shell for ops. - **Local `docker build` requires `bin/` to be populated first.** This is by design — the Dockerfile is a staging layer, not a compiler. Document this in the repo's README so contributors know to run `go build -o bin/<name>-linux-amd64` before `docker buildx build .`. ## ldflag convention Every go-app main package declares the same three string variables: ```go package main // Populated by release.yml's ldflags + build-go-attest's // auto-build-timestamp step. var ( version = "" build = "" buildTime = "" ) ``` The template ldflags are fixed across the fleet: ``` -X main.version=<release-tag> -X main.build=<commit-sha> -X main.buildTime=<ISO-8601 commit timestamp> # via auto-build-timestamp ``` Consumers decide what to do with the values. Three live patterns: ### Pattern 1: consume directly (ofelia) main package just uses the vars: ```go slog.Info("starting server", "version", version, "build", build, "build_time", buildTime, // slog key in snake_case, not camelCase ) ``` ### Pattern 2: forward into `internal/version` (ldap-manager) main package declares matching vars then forwards at init() time: ```go import internalversion "github.com/<org>/<repo>/internal/version" var ( version, build, buildTime = "", "", "" ) // forwardBuildMetadata is extracted so it's unit-testable (can be // called with test inputs and then the test asserts on // internalversion.* under a sync.Mutex snapshot/restore). func forwardBuildMetadata(v, b, t string) { if v != "" { internalversion.Version = v } if b != "" { internalversion.CommitHash = b } if t != "" { internalversion.BuildTimestamp = t } } func init() { forwardBuildMetadata(version, build, buildTime) } ``` Empty-string check preserves local-dev defaults ("dev", "n/a") when not injected. ### Pattern 3: forward into a package that already computed `vcs.revision` (raybeam) The internal/build package had a VCS-derived fallback. Keep it; just layer main.* on top in init(): ```go // internal/build/build.go — fallback works for non-ldflag builds. var vcsRevision = func() string { if info, ok := debug.ReadBuildInfo(); ok { for _, s := range info.Settings { if s.Key == "vcs.revision" { return s.Value } } } return "" }() var Version = func() string { if vcsRevision != "" { return vcsRevision }; return "unknown" }() var CommitHash = vcsRevision // main.go — alias the internal package to avoid shadowing 'build'. import buildpkg "github.com/<org>/<repo>/internal/build" var (version, build, buildTime = "", "", "") func init() { if version != "" { buildpkg.Version = version } if build != "" { buildpkg.CommitHash = build } _ = buildTime // reserved receiver, not surfaced yet } ``` ### Gotcha: package init order catches you if you're using cobra If `cmd.rootCmd` is declared with `Version: build.Version`, cobra captures the VALUE at package-init time. `cmd` is imported by `main`, so Go initializes `cmd` first — BEFORE main's init() runs and updates `build.Version`. Cobra's --version then reports the pre-forward value. **Fix:** defer the assignment to `Execute()`, which runs after all inits: ```go var rootCmd = &cobra.Command{ Use: "raybeam" /* no Version here */ } func Execute() { rootCmd.Version = formatVersion(build.Version, build.CommitHash) rootCmd.Execute() } ``` ### About `-X main.X` for undeclared vars Go 1.21+ linkers **silently ignore** `-X` targets for vars that don't exist in the linked package — the build succeeds and the flag is a no-op. Older toolchains can reject with "symbol not found". The template ships all three of `main.version`, `main.build`, `main.buildTime` unconditionally; consumers MUST declare matching vars so: 1. The ldflag actually lands (modern Go). 2. Older-toolchain edge cases don't break the build (belt-and-braces). ## main-package: auto `build-go-attest.yml` accepts `main-package: auto`, which resolves after checkout by scanning for a `package main` declaration: 1. Any non-test `./*.go` declares `package main` → use `.` (handles `ofelia.go`, `cmd.go`, `main.go`). 2. Else `./cmd/<repo-name>/*.go` declares `package main` → use `./cmd/<repo-name>` (handles ldap-manager's layout). 3. Else fail hard. Repo name comes from `${GITHUB_REPOSITORY##*/}` (always populated), not `github.event.repository.name` (empty on some triggers). This is why the release.yml template stays byte-identical across consumers regardless of whether they put main at root or under `cmd/`. ## auto-build-timestamp `build-go-attest.yml` accepts `auto-build-timestamp: true`, which resolves the HEAD commit timestamp from git after checkout (not from `github.event.head_commit.timestamp`, which is empty on `workflow_dispatch` backfills): ```bash # Inside build-go-attest.yml, after checkout: if ! BUILD_TS=$(git show -s --format=%cI HEAD); then echo "::error::git show failed. Check actions/checkout." exit 1 fi if [[ -z "$BUILD_TS" ]]; then echo "::error::git show returned empty." exit 1 fi LDFLAGS="${LDFLAGS} -X main.buildTime=${BUILD_TS}" ``` **Do not** pipe stderr into `BUILD_TS` via `2>&1` — a harmless warning (e.g., CRLF normalization) would end up inside the ldflag value. See [workflow-bash-patterns.md](https://github.com/netresearch/github-project-skill/blob/main/skills/github-project/references/workflow-bash-patterns.md) in the github-project skill for details. Works for both tag push and workflow_dispatch "rebuild tag" backfills, so `main.buildTime` is never empty when the feature is opted in. ## Related - [reusable-workflows.md](./reusable-workflows.md) — how caller/reusable permission propagation works - [docker.md](./docker.md) — Docker client patterns in Go code (not Dockerfile) - github-project-skill [workflow-bash-patterns.md](https://github.com/netresearch/github-project-skill/blob/main/skills/github-project/references/workflow-bash-patterns.md) — bash inside `run:` gotchas ## GoReleaser changelog: `github-native` ignores your filters With `changelog.use: github-native`, GoReleaser delegates to GitHub's generated-release-notes API and **ignores `sort` and `filters`** — the [upstream docs](https://goreleaser.com/customization/changelog/) mark both options "Disabled when using 'github-native'", so every `chore(deps)` / bot commit leaks into the release notes. Switch to an implementation where `filters.exclude` applies (`git`, `github`, `gitlab`, `gitea`): ```yaml changelog: use: git filters: exclude: - "^chore\\(deps\\)" - "^ci:" ``` -
testing.md 13.8 KB
# Go Testing Patterns ## Build Tags for Test Isolation Three-tier convention: unit tests are untagged and run by default; integration and e2e tests opt in via `//go:build`. The tag must be the first line of the file, followed by a blank line before the package clause. ```go // File: job_test.go — unit tests, no tag, run by default package core ``` ```go // File: docker_integration_test.go — require real external deps (Docker, see references/docker.md) //go:build integration package core ``` ```go // File: workflow_e2e_test.go — complete system //go:build e2e package e2e ``` ```bash go test ./... # unit only (CI default) go test -tags=integration ./... # + integration go test -tags="integration e2e" ./... # full suite ``` ### An untagged file is in *every* build, not just the unit one "unit = untagged" describes which tier a file belongs to, not which builds compile it. A file with no constraint is compiled into the untagged build **and** every tagged one, so a test in it runs in both. That is harmless until a helper has a tagged/untagged pair — the usual shape for container setup: ```go // File: test_setup.go — the tagged build gets a real container //go:build integration package core func SetupTestContainer(t *testing.T) *TestContainer { return startContainer(t) } ``` ```go // File: test_setup_stub.go — the untagged build gets a skip //go:build !integration package core func SetupTestContainer(t *testing.T) *TestContainer { t.Skip("requires the integration build tag") return nil } ``` A test in an **untagged** file that calls that helper skips in the unit tier and executes for real in the tagged one. A defect in such a test is therefore invisible to `go test ./...` — it can only fail where CI runs the tagged build. Symptom: every local run green, the integration job red, and the diff looks unrelated to the failure. So: before reporting a change as verified, run the tier CI runs, not only the default one. When a test that constructs something is edited, check which file it lives in and which helper it calls. ### A `-run` filter can only subtract from what the tag selected Pairing a build tag with a name pattern is a common way to write an "integration only" target: ```make test-integration: go test -tags=integration -run="Test.*Integration" ./... # drops tests ``` The tag already chose the files. The pattern then removes every test in them whose *name* does not match, which is silent and easy to get wrong: names like `TestBulkOperations` or `TestCacheInvalidation` live in tagged files and contain no "Integration". Select by tag and let the tagged build carry the untagged tests too; that overlap is the cost of selecting by tag. Count what a selector would run without starting anything — `-test.list` takes the same patterns as `-run`: ```bash go test -c -tags=integration -o /tmp/x.test . /tmp/x.test -test.list '.*' # everything in the tagged build /tmp/x.test -test.list 'Test.*Integration' # what the -run filter would keep ``` Compare the two counts against the number of `func Test` in the tagged files. A gap is tests compiled and never run. ### Addresses for tests that must fail to connect A test that asserts a dial fails needs an address that fails *fast* and *always*: | address | behaviour | use | |---|---|---| | `example.com`, `test.com`, `server.com` | resolve to live hosts that drop packets on most ports | never — each attempt runs to the client's connect timeout (60s measured with go-ldap's default) | | `something.invalid` | NXDOMAIN (RFC 6761), but still asks a resolver | fine for a handful of calls | | `127.0.0.1:1` | refused immediately, no name resolution at all | anything that dials repeatedly | Two traps beyond the timeout: - **Never a port something might serve.** `localhost:389` makes "the dial must fail" pass for the wrong reason on a developer running that service, and fail outright once it succeeds. - **Some hostnames are data, not addresses.** A string fed to an error formatter or a masking function may have an expected output computed from its length. Rewriting the address there breaks the expectation; check what the test does with the string before sweeping it. ## Time Control Testing time-dependent code (schedulers, caches, rate limiters) with real timers is slow and flaky. Don't hand-roll a `Clock`/`FakeClock` abstraction — go-cron ships one; see `references/cron-scheduling.md` § Testing with FakeClock. ## Resource Isolation: One Instance Per Test Give each test its own instance of any **stateful** fixture (scheduler, server, store, temp dir). Sharing a mutable instance across tests causes order-dependent pollution and flaky failures that often only reproduce under `-shuffle=on` or in CI. ```go // Bad — shared, stateful scheduler: one test's jobs leak into the next var sharedScheduler *Scheduler // package-level // Good — each test gets a fresh instance and cleans it up func TestScheduler_AddJob(t *testing.T) { s := NewScheduler(context.Background()) t.Cleanup(s.Stop) // ... assert against this isolated scheduler } ``` Exception: a **read-only** fixture that no test mutates may be shared for speed. The rule targets *mutable* state — if a test can change it, give each test its own. ## Race Detection ```bash # Run tests with race detector go test -race ./... # Build binary with race detection go build -race ./cmd/app # Run specific package go test -race -v ./core/... ``` ### Common Race Patterns and Fixes **1. Unsynchronized map access:** ```go // BAD: Race condition type Cache struct { data map[string]string } func (c *Cache) Set(k, v string) { c.data[k] = v } // Race! func (c *Cache) Get(k string) string { return c.data[k] } // Race! // GOOD: Protected with mutex type Cache struct { mu sync.RWMutex data map[string]string } func (c *Cache) Set(k, v string) { c.mu.Lock() defer c.mu.Unlock() c.data[k] = v } func (c *Cache) Get(k string) string { c.mu.RLock() defer c.mu.RUnlock() return c.data[k] } ``` **2. Goroutine capturing loop variable:** > **Note:** Go 1.22+ fixed loop variable capture. The `i := i` shadow is no longer needed. > The examples below show the modern style. ```go // Modern Go (1.22+): safe without shadow for i := range 10 { go func() { fmt.Println(i) // Safe: each iteration gets its own copy }() } } ``` **3. Check-then-act pattern:** ```go // BAD: Race between check and update if cache.Get(key) == nil { cache.Set(key, compute()) // Another goroutine might have set it! } // GOOD: Atomic operation value := cache.GetOrSet(key, func() string { return compute() }) ``` **4. RLock vs Lock - Know When to Upgrade:** ```go // BAD: RLock used when writing to a field func (c *Cache) Get(key string) ([]byte, bool) { c.mu.RLock() defer c.mu.RUnlock() entry, ok := c.entries[key] if ok { entry.accessedAt = time.Now() // RACE! Writing under RLock } return entry.data, ok } // GOOD: Use Lock when any write occurs func (c *Cache) Get(key string) ([]byte, bool) { c.mu.Lock() // Full lock needed for accessedAt update defer c.mu.Unlock() entry, ok := c.entries[key] if ok { entry.accessedAt = time.Now() // Safe } return entry.data, ok } ``` **Rule**: RLock is ONLY safe when the entire operation is read-only. Any write (including updating timestamps, counters, or "metadata") requires a full Lock. ## Common Gotchas ### Integer to String Conversion A common trap in Go: `string(rune(i))` does NOT convert an integer to its string representation: ```go // BAD - Produces unicode codepoint, not numeric string! for i := range 10 { key := "key" + string(rune(i)) // key + "\x00", "\x01", etc. } // GOOD - Correct integer to string conversion for i := range 10 { key := "key" + strconv.Itoa(i) // "key0", "key1", etc. } // Also acceptable key := fmt.Sprintf("key%d", i) ``` **Why this happens**: `string(rune(i))` interprets `i` as a Unicode code point. `string(rune(65))` produces `"A"`, not `"65"`. ### Test Assertion Precision Choose the right assertion for nil checks: ```go // BAD - assert.Empty works but is less precise assert.Empty(t, err) // Passes for nil, "", 0, empty slices... // GOOD - assert.Nil is explicit about intent assert.Nil(t, err) // Only passes for nil // For error checking, even better: assert.NoError(t, err) require.NoError(t, err) // Fails test immediately ``` ### Unused Test Parameters Always name `*testing.T` parameters to enable helper functions: ```go // BAD - Cannot use require.NotPanics or t.Helper() func TestSomething(_ *testing.T) { // ... } // GOOD - Full access to testing helpers func TestSomething(t *testing.T) { require.NotPanics(t, func() { // test code }) } ``` ### Fuzz Target Naming Fuzz targets must match the `Fuzz*` pattern exactly: ```go // BAD - Target name doesn't match function //go:build ignore func FuzzParser(f *testing.F) { f.Fuzz(func(t *testing.T, data []byte) { // ... }) } // GOOD - Target exists and matches name in fuzz command // go test -fuzz=FuzzParser func FuzzParser(f *testing.F) { f.Fuzz(func(t *testing.T, data []byte) { // ... }) } ``` ### Always Check app.Test() Errors When testing Fiber/Echo handlers, always check the error: ```go // BAD - Ignores potential test setup errors resp, _ := app.Test(req) // GOOD - Fails test if request setup fails resp, err := app.Test(req) require.NoError(t, err) defer resp.Body.Close() ``` ### Fiber v2 Testing Patterns #### Full App Setup with Cleanup Use `t.Cleanup()` to ensure goroutine teardown and prevent leaks: ```go func setupFullTestApp(t *testing.T) *App { t.Helper() app := NewApp(testConfig) app.Setup() t.Cleanup(func() { _ = app.fiber.Shutdown() }) return app } ``` #### Auth Session Cookie Generation For handlers requiring authentication, create session cookies with a separate mini Fiber app: ```go func createAuthCookie(t *testing.T, sessionStore *session.Store) *http.Cookie { t.Helper() miniApp := fiber.New() var cookie *http.Cookie miniApp.Get("/set-session", func(c *fiber.Ctx) error { sess, err := sessionStore.Get(c) if err != nil { return err } sess.Set("username", "testuser") sess.Set("dn", "cn=testuser,dc=example,dc=com") return sess.Save() }) req := httptest.NewRequest(http.MethodGet, "/set-session", nil) resp, err := miniApp.Test(req) require.NoError(t, err) defer resp.Body.Close() for _, c := range resp.Cookies() { if c.Name == "session_id" { cookie = c break } } require.NotNil(t, cookie) return cookie } ``` #### Using Auth Cookies in Tests ```go func TestProtectedHandler(t *testing.T) { app := setupFullTestApp(t) cookie := createAuthCookie(t, app.sessionStore) req := httptest.NewRequest(http.MethodGet, "/api/users", nil) req.AddCookie(cookie) resp, err := app.fiber.Test(req) require.NoError(t, err) defer resp.Body.Close() assert.Equal(t, http.StatusOK, resp.StatusCode) } ``` ### Prefer assert.ErrorAs Over errors.As in Tests Use `assert.ErrorAs` from testify for better failure messages: ```go // BAD - Manual errors.As with less informative failures var target *MyError if !errors.As(err, &target) { t.Errorf("expected *MyError, got %T", err) } // GOOD - testify provides clear diff output on failure var target *MyError assert.ErrorAs(t, err, &target) ``` ### Always Use t.Helper() in Test Helpers Mark all test helper functions with `t.Helper()` so failure messages point to the calling test, not the helper: ```go func assertUserExists(t *testing.T, store UserStore, username string) { t.Helper() // Failure will report caller's line, not this function user, err := store.Get(username) require.NoError(t, err) assert.NotNil(t, user) } ``` ### Reap a Daemonized Helper by Process Group A test that starts a helper server with `exec.Command` and kills it in cleanup looks correct and is not. `cmd.Process.Kill()` kills only the process it started — typically `sh -c "..."` — and the real server survives as an orphan, still holding the stdout it inherited from the test binary. `go test` then reports, *after every test has already passed*: ```text PASS *** Test I/O incomplete 1m0s after exiting. exec: WaitDelay expired before I/O complete FAIL example.com/pkg/test ``` `PASS` followed by `FAIL` is the signature. It is timing-dependent, so it reproduces on some runs and not others, and the orphan also holds the port — which makes the *next* run connect to a stale server from the previous one and produce a wrong diagnosis. Start the helper in its own process group and kill the group: ```go cmd := exec.Command(shPath, "-c", command) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} if err := cmd.Start(); err != nil { t.Fatalf("can't start helper: %v", err) } t.Cleanup(func() { // Negative PID = the whole group, so children die with the shell. if err := syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL); err != nil { t.Logf("can't kill process group of %d: %v", cmd.Process.Pid, err) } }) ``` `syscall.SysProcAttr.Setpgid` is Unix-only — guard the file with `//go:build !windows`, or use `os/exec`'s `Cancel` plus `WaitDelay` when the suite must also run on Windows. Verify the fix by what it leaves behind, not by the exit code alone: after a green run the port must be free (`ss -ltnp | grep :<port>`). A run that passes while leaking the daemon has not been fixed, it has been raced. ## Related - `references/fuzz-testing.md` — fuzz testing patterns, security-focused seeds - `references/mutation-testing.md` — mutation testing, test-quality measurement - `references/makefile.md` — standard Makefile test targets - `references/cron-scheduling.md` — go-cron FakeClock testing patterns
-
-
scripts
-
verify-go-project.sh 2.3 KB
#!/bin/bash # Go Project Verification Script # Validates Go project structure and quality set -e PROJECT_DIR="${1:-.}" ERRORS=0 WARNINGS=0 echo "=== Go Project Verification ===" echo "Directory: $PROJECT_DIR" echo "" # Check go.mod exists if [[ -f "$PROJECT_DIR/go.mod" ]]; then echo "✅ go.mod found" MODULE=$(grep "^module" "$PROJECT_DIR/go.mod" | awk '{print $2}') echo " Module: $MODULE" else echo "❌ go.mod not found" ((ERRORS++)) fi # Check go.sum exists if [[ -f "$PROJECT_DIR/go.sum" ]]; then echo "✅ go.sum found" else echo "⚠️ go.sum not found (run 'go mod tidy')" ((WARNINGS++)) fi # Check standard directories echo "" echo "=== Directory Structure ===" for dir in cmd core internal pkg; do if [[ -d "$PROJECT_DIR/$dir" ]]; then echo "✅ $dir/ exists" fi done # Check for main.go MAIN_FILES=$(find "$PROJECT_DIR" -name "main.go" 2>/dev/null | head -5) if [[ -n "$MAIN_FILES" ]]; then echo "✅ Entry points found:" echo "$MAIN_FILES" | while read f; do echo " - $f"; done else echo "⚠️ No main.go found" ((WARNINGS++)) fi # Run go vet echo "" echo "=== Static Analysis ===" if command -v go &> /dev/null; then cd "$PROJECT_DIR" if go vet ./... 2>&1; then echo "✅ go vet passed" else echo "❌ go vet found issues" ((ERRORS++)) fi else echo "⚠️ Go not installed, skipping vet" ((WARNINGS++)) fi # Check for tests echo "" echo "=== Test Coverage ===" TEST_FILES=$(find "$PROJECT_DIR" -name "*_test.go" 2>/dev/null | wc -l) if [[ "$TEST_FILES" -gt 0 ]]; then echo "✅ Found $TEST_FILES test files" else echo "⚠️ No test files found" ((WARNINGS++)) fi # Check for Dockerfile echo "" echo "=== Deployment ===" if [[ -f "$PROJECT_DIR/Dockerfile" ]]; then echo "✅ Dockerfile found" else echo "⚠️ No Dockerfile found" ((WARNINGS++)) fi # Check for Makefile if [[ -f "$PROJECT_DIR/Makefile" ]]; then echo "✅ Makefile found" else echo "⚠️ No Makefile found" ((WARNINGS++)) fi # Summary echo "" echo "=== Summary ===" echo "Errors: $ERRORS" echo "Warnings: $WARNINGS" if [[ $ERRORS -gt 0 ]]; then echo "❌ Verification FAILED" exit 1 else echo "✅ Verification PASSED" exit 0 fi
-
-
checkpoints.yaml 6.2 KB
# Checkpoints for go-development skill # Validates Go project structure, tooling, and best practices version: 1 skill_id: go-development preconditions: - type: file_exists target: go.mod mechanical: # === PROJECT STRUCTURE === - id: GD-01 type: file_exists target: go.mod severity: error desc: "go.mod must exist" - id: GD-02 type: file_exists target: go.sum severity: error desc: "go.sum must exist (dependencies must be tracked)" - id: GD-03 type: file_exists target: Makefile severity: warning desc: "Makefile should exist for standard build interface" - id: GD-04 type: file_exists target: .gitignore severity: warning desc: ".gitignore should exist" # === LINTING CONFIGURATION === - id: GD-05 type: command pattern: "test -f .golangci.yml || test -f .golangci.yaml || test -f .golangci.toml" severity: warning desc: "golangci-lint config should exist (.golangci.yml or similar)" # === TESTS === - id: GD-06 type: command pattern: "find . -name '*_test.go' -not -path './vendor/*' | head -1 | grep -q ." severity: error desc: "Project must have test files (*_test.go)" - id: GD-07 type: command pattern: "find . -name '*_test.go' -not -path './vendor/*' | xargs grep -l 'func Test' | head -1 | grep -q ." severity: error desc: "Test files must contain test functions" # === MAKEFILE TARGETS === - id: GD-08 type: contains target: Makefile pattern: "test:" severity: warning desc: "Makefile should have a test target" - id: GD-09 type: contains target: Makefile pattern: "build:" severity: warning desc: "Makefile should have a build target" - id: GD-10 type: contains target: Makefile pattern: "lint:" severity: warning desc: "Makefile should have a lint target" # === GO MODULE HYGIENE === - id: GD-11 type: regex target: go.mod pattern: "^go \\d+\\.\\d+" severity: error desc: "go.mod must specify Go version" - id: GD-12 type: regex target: go.mod pattern: "^module " severity: error desc: "go.mod must declare module path" # === CODE QUALITY === - id: GD-13 type: command pattern: "! find . -name '*.go' -not -path './vendor/*' | xargs grep -l 'interface{}' | head -1 | grep -q . 2>/dev/null" severity: info desc: "Prefer 'any' over 'interface{}' (Go 1.18+)" - id: GD-14 type: command pattern: "! find . -name '*.go' -not -path './vendor/*' | xargs grep -l 'sync.Map' | head -1 | grep -q . 2>/dev/null" severity: info desc: "Prefer generic typed maps over sync.Map" # === RACE DETECTION === - id: GD-15 type: regex target: Makefile pattern: "-race" severity: warning desc: "Makefile test target should include -race flag" # === DOCKER (if applicable) === - id: GD-16 type: regex target: Makefile pattern: "-trimpath" severity: info desc: "Build flags should include -trimpath for reproducibility" # === VULNERABILITY SCANNING === - id: GD-17 type: regex target: Makefile pattern: "govulncheck" severity: warning desc: "Makefile should have a govulncheck target for vulnerability scanning" # === GOLANGCI-LINT V2 CONFIG === - id: GD-18 type: command pattern: "test -f .golangci.yml && grep -q 'version:' .golangci.yml || test -f .golangci.yaml && grep -q 'version:' .golangci.yaml || true" severity: info desc: "golangci-lint config should declare version field (v2 format)" # === FUZZ TESTS === - id: GD-19 type: command pattern: "find . -name '*_test.go' -not -path './vendor/*' | xargs grep -l 'func Fuzz' 2>/dev/null | head -1 | grep -q . || true" severity: info desc: "Project should have fuzz test functions (Fuzz*) for parser/input code" # === MAKEFILE ALL TARGET === - id: GD-23 type: contains target: Makefile pattern: "all:" severity: info desc: "Makefile should have an 'all' target combining lint, test, and build" # === COVERPROFILE === - id: GD-24 type: regex target: Makefile pattern: "-coverprofile" severity: info desc: "Makefile test target should generate coverage profile" # === GIT HOOKS === - id: GD-26 type: file_exists target: lefthook.yml severity: info desc: "lefthook.yml should exist for pre-commit/pre-push git hooks" # === ERROR VARIABLE NAMING === - id: GD-25 type: command pattern: "! grep -rlE 'var [A-Z]\\w+ = errors\\.New' --include='*.go' . | xargs grep -vl 'var Err' | head -1 | grep -q ." severity: info desc: "Exported error variables should use Err prefix (e.g., ErrInvalidInput)" llm_reviews: - id: GD-20 domain: go-quality prompt: | Review the Go project for code quality and best practices: 1. Does go.mod use a reasonable Go version (not outdated)? 2. Is the golangci-lint config comprehensive (errcheck, govet, staticcheck enabled)? 3. Does the Makefile provide standard targets (test, build, lint, all)? 4. Are error messages lowercase without punctuation (Go convention)? 5. Is error wrapping used consistently with fmt.Errorf and %w? 6. Are acronyms properly cased (ID, URL, HTTP not Id, Url, Http)? severity: warning desc: "Go code quality and convention adherence" - id: GD-21 domain: go-quality prompt: | Review the Go project for testing completeness: 1. Do packages with business logic have corresponding test files? 2. Are table-driven tests used for functions with multiple input scenarios? 3. Do test helpers call t.Helper()? 4. Is t.Parallel() used where safe? 5. Are tests independent (no shared mutable state between tests)? 6. Is test coverage reasonable for critical paths? severity: warning desc: "Go testing patterns and coverage" - id: GD-22 domain: go-quality prompt: | Review the Go project for security and resilience: 1. Are HTTP requests made with context (no noctx violations)? 2. Are HTTP response bodies always closed? 3. Is graceful shutdown implemented for servers? 4. Are goroutines properly managed (no leaks)? 5. Is context propagation consistent throughout? severity: info desc: "Go security and resilience patterns" -
SKILL.md 4 KB
--- name: go-development description: "Use when developing Go applications, implementing job schedulers or cron (netresearch/go-cron, ofelia), Docker API integrations, LDAP/AD clients, building resilient services with retry logic, setting up Go test suites (unit/integration/fuzz/mutation), or running golangci-lint." license: "(MIT AND CC-BY-SA-4.0). See LICENSE-MIT and LICENSE-CC-BY-SA-4.0" compatibility: "Requires go 1.21+, golangci-lint, docker." metadata: author: Netresearch DTT GmbH version: "1.16.1" repository: https://github.com/netresearch/go-development-skill allowed-tools: Bash(go:*) Bash(make:*) Bash(docker:*) Bash(golangci-lint:*) Read Write Glob Grep --- # Go Development Patterns ## Core Principles ### Type Safety - **Avoid:** `interface{}` (use `any`), `sync.Map`, scattered type assertions, reflection - **Prefer:** Generics `[T any]`, `errors.AsType[T]` (Go 1.26), concrete types - Run `go fix ./...` after upgrades ### Consistency - One pattern per problem domain - Match existing codebase patterns - Refactor holistically or not at all - Config precedence: defaults < config file < env vars < flags ### Testing - Build tags isolate test tiers: unit (default), `integration`, `e2e` - Always use `t.Parallel()`, `t.Helper()`, table-driven subtests - Use `log/slog` directly -- never wrap it in custom Logger interfaces ### Conventions - Naming: ID, URL, HTTP (not Id, Url, Http) — not tool-enforced (ST1003 is off by default) - Error wrapping: `fmt.Errorf("failed to process: %w", err)` ## References Git hooks: `ls lefthook.yml 2>/dev/null && lefthook install || echo "Add lefthook — see references/lefthook-template.md"` Load as needed: | Reference | Purpose | |-----------|---------| | `references/architecture.md` | Package structure, state mutation completeness | | `references/logging.md` | Structured logging with log/slog, migration from logrus | | `references/cron-scheduling.md` | go-cron patterns: named jobs, runtime updates, resilience | | `references/resilience.md` | Pointer to go-cron's built-in retry/circuit-breaker/timeout wrappers | | `references/docker.md` | Docker client patterns, buffer pooling | | `references/ldap.md` | LDAP/Active Directory integration | | `references/testing.md` | Build tags, resource isolation, race gotchas | | `references/linting.md` | golangci-lint v2, staticcheck | | `references/api-design.md` | Enum/status defensive handling | | `references/fuzz-testing.md` | Go fuzzing patterns, security seeds | | `references/contracts-and-invariants.md` | Contracts, invariants, property tests | | `references/mutation-testing.md` | Gremlins configuration, test quality measurement | | `references/makefile.md` | Standard Makefile interface for CI/CD | | `references/modernization.md` | `go fix` modernizers and their build-tag trap, `errors.AsType[T]`, `b.Loop` | | `references/dependencies.md` | Upgrades: `go get -u all`, majors, build-set scoping | | `references/lefthook-template.md` | Ready-to-use lefthook.yml for Go project git hooks | | `references/branch-protection.md` | Ruleset watermark: three-ruleset gate, bypass modes | | `references/reusable-workflows.md` | Reusable Actions workflow callers, permission propagation, release-gate outputs | | `references/single-build-release.md` | Single-build release: cross-compile once, reuse for release+container | | `references/awesome-go-submission.md` | awesome-go submission: CI-parsed PR body, name collisions | ## Quality Gates Run before completing any review: ```bash golangci-lint run --timeout 5m # Linting go vet ./... # Static analysis staticcheck ./... # Additional checks govulncheck ./... # Vulnerability scan go test -race ./... # Race detection ``` ## Stdlib Vulnerability Fixes When `govulncheck` reports stdlib vulnerabilities: check fix version via `vuln.go.dev`, update `go X.Y.Z` in `go.mod`, run `go mod tidy`. --- > **Contributing:** Submit improvements to https://github.com/netresearch/go-development-skill
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.