datarobot-app-framework-cicd
Guidance for setting up CI/CD pipelines for DataRobot application templates using GitLab, GitHub Actions, and Pulumi for infrastructure as code. Use when setting up CI/CD pipelines, configuring deployments, or managing infrastructure for DataRobot application templates.
Install
npx skills add https://github.com/datarobot-oss/datarobot-agent-skills/tree/main/skills/datarobot-app-framework-cicd
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install datarobot-oss-datarobot-agent-skills@llmmart
git clone https://github.com/datarobot-oss/datarobot-agent-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole datarobot-oss/datarobot-agent-skills collection as a plugin from our marketplace. Git is the plain clone.
README
DataRobot Application Templates CI/CD Skill
A skill that provides comprehensive guidance for setting up production-grade CI/CD pipelines for DataRobot application templates.
Overview
Transform your DataRobot application templates from manual deployments to automated CI/CD workflows with:
- Automated Testing: Run linters and tests on every pull/merge request
- Review Deployments: Spin up full application stacks for PR validation
- Continuous Delivery: Automatically deploy changes when merged to main
- Infrastructure as Code: Use Pulumi for declarative infrastructure management
- Secrets Management: Secure handling of API keys and credentials
- Multi-Platform Support: Works with both GitLab and GitHub
What's Included
Documentation
- SKILL.md: Complete guidance for CI/CD setup
- Quick start guide
- Platform-specific configurations (GitLab, GitHub)
- Pulumi state management strategies
- Secrets management patterns
- Troubleshooting guide
Example Configurations
The scripts/ directory contains reference implementations that should be copied to the application template's infra/ directory:
infra-README.md: Documentation for the infra/ directory explaining structure and usage
gitlab-ci.yml: Complete GitLab CI/CD pipeline
- Automated testing and linting
- Manual review app deployments
- Continuous delivery on merge
- Azure Blob Storage backend example
github-deploy.yml: GitHub Actions deployment workflow
- Automated testing and linting
- PR-based review deployments
- GPG-encrypted secrets
- Pulumi Cloud backend example
github-destroy.yml: GitHub Actions cleanup workflow
- Manual stack destruction
- Resource cleanup for review apps
setup-github-secrets.sh: Automated GitHub secrets setup
- Uses GitHub CLI (
gh) - Interactive secret entry
- Configures all required secrets for Actions
- Uses GitHub CLI (
setup-gitlab-variables.sh: Automated GitLab variables setup
- Uses GitLab CLI (
glab) - Interactive variable entry
- Configures all required variables for CI/CD
- Uses GitLab CLI (
encrypt-secrets.sh: GPG encryption for .env files
- Encrypts root .env to .env.gpg
- Interactive encryption workflow
- GitHub Actions secrets preparation
- Step-by-step instructions
decrypt-secrets.sh: GPG decryption for local development
- Decrypts root .env.gpg to .env
- Safe local secrets management
- Testing workflow simulation
taskfile-snippets.yaml: CI/CD Task definitions
- Secrets management commands
- Pulumi deployment tasks
- CI/CD testing helpers
- Add an
includesentry in the project's rootTaskfile.ymlpointing toinfra/Taskfile.yaml(do NOT add tasks to root)
pulumi-setup.sh: Interactive Pulumi setup script
- Backend configuration (Cloud, Azure, AWS)
- Initial stack creation
- Credential management
Quick Start
When implementing CI/CD for an application template:
Choose your platform: GitLab or GitHub
Create infra directory and copy scripts:
# Create infra directory in project root and copy entire scripts folder mkdir -p infra cp -R <skill-path>/scripts infra/scripts chmod +x infra/scripts/*.sh # For GitLab - copy to root cp infra/scripts/gitlab-ci.yml .gitlab-ci.yml # For GitHub - copy to .github/workflows/ mkdir -p .github/workflows cp infra/scripts/github-deploy.yml .github/workflows/deploy.yml cp infra/scripts/github-destroy.yml .github/workflows/destroy.ymlCreate
infra/Taskfile.yaml:# Copy taskfile-snippets.yaml to infra/Taskfile.yaml cp infra/scripts/taskfile-snippets.yaml infra/Taskfile.yaml # Then add a single includes entry to root Taskfile.yml: # includes: # infra: # taskfile: ./infra/Taskfile.yaml # dir: . # ⚠️ Do NOT paste CI/CD tasks into the root Taskfile.ymlSet up Pulumi:
cd infra/scripts ./pulumi-setup.sh cd ../.. # Install GitHub CLI: brew install gh gh auth login task setup-github-secretsFor GitLab (automated):
# Install GitLab CLI: brew install glab glab auth login task setup-gitlab-varsEncrypt your secrets (for GitHub only):
# Creates .env.gpg in project root from .env task encrypt-secrets git add .env.gpg git commit -m "Add encrypted secrets"
Key Features
GitLab CI/CD
- Parallel test execution for faster feedback
- DIY backend support (Azure Blob, S3, GCS)
- MR-specific stack names for isolation
- Automatic commenting with deployment info
- Manual cleanup jobs
GitHub Actions
- GPG-encrypted secrets for better management
- PR comments with deployment URLs
- Pulumi Cloud integration
- Manual destroy workflows
- Matrix testing support
Pulumi Integration
- Centralized state management
- Stack isolation per environment
- Idempotent deployments
- Cross-machine synchronization
- Codespace compatibility
Use Cases
Development Teams
- Test infrastructure changes in isolation
- Review applications before merging
- Automatically deploy to staging/production
- Track infrastructure state across team
DevOps Engineers
- Implement IaC for AI applications
- Manage multiple environments
- Automate deployment workflows
- Monitor infrastructure changes
Data Scientists
- Deploy models with applications
- Test changes in review environments
- Collaborate on application features
- Focus on ML, not infrastructure
Example Repositories
See these live implementations:
- GitLab: demo-data-agent
- GitHub: demo-talk-to-my-data-agent
Platform Support
- ✅ GitLab CI/CD
- ✅ GitHub Actions
- ✅ Pulumi Cloud
- ✅ Azure Blob Storage
- ✅ AWS S3
- ✅ Google Cloud Storage
Resources
- SKILL.md - Complete documentation
- Task - Workflow management
- Pulumi - Infrastructure as Code
- DataRobot Application Templates
Contributing
This is the canonical upstream source for this skill at datarobot-agent-skills. Downstream repositories (such as af-component-base) bundle a local copy, when this skill is updated, those copies should be synced.
Contributions welcome via pull request.
License
See the main repository LICENSE file.
Skill manifest
DataRobot Application Templates CI/CD Skill
This skill provides comprehensive guidance for setting up production-grade CI/CD pipelines for DataRobot application templates, including automated testing, review deployments, and continuous delivery.
Quick Start
Default behavior: When a user asks to "set up CI/CD" without specifying a platform or backend, always use the Simple Path below — three workflow files, two GitHub Secrets, done. Do not create infra/scripts/, do not add CI/CD tasks to infra/Taskfile.yaml, do not involve GPG encryption unless the user explicitly asks for it.
Only deviate from the simple path when the user specifies:
- A specific Pulumi state backend (Azure Blob, S3, GCS) → use
scripts/and see Implementation Pattern - GitLab CI/CD → see GitLab CI/CD Configuration
- Many secrets to manage → consider GPG approach in
scripts/
Simple Path: Pulumi Cloud + GitHub Secrets
For most data scientists and AI engineers, this is all you need. No GPG encryption, no cloud storage account, no extra scripts.
What to create in the user's repository:
Copy the three workflow files to
.github/workflows/:Source Destination Trigger examples/github-cd-pulumi-cloud.yml.github/workflows/cd.ymlAutomatic — every merge to mainexamples/github-deploy-pulumi-cloud.yml.github/workflows/deploy-pr.ymlManual — user picks PR branch + enters stack name (e.g. pr-42)examples/github-destroy-pulumi-cloud.yml.github/workflows/destroy.ymlManual — user enters stack name to tear down Create
.github/workflows/README.mdfromexamples/workflows-README.md. This is the setup guide that tells the user exactly what secrets and variables to add and how.Tell the user to follow the setup guide in
.github/workflows/README.md.
That's it. Do not add anything to infra/Taskfile.yaml or create infra/scripts/ for this path.
Required GitHub Secrets (both required — no defaults):
| Name | Kind |
|---|---|
DATAROBOT_API_TOKEN |
Secret |
PULUMI_ACCESS_TOKEN |
Secret |
Optional GitHub Variable (defaults to ci if not set):
| Name | Kind | Default |
|---|---|---|
PULUMI_STACK_CI_NAME |
Variable | ci |
When to use the advanced approach (GPG + DIY backends) instead:
- You have many secrets (GPG encrypts all of
.envbehind a single passphrase — only one GitHub Secret needed) - Your organization prohibits Pulumi Cloud and requires a self-managed backend (Azure Blob / S3 / GCS)
- You need GitLab CI/CD
The templates and scripts for all of these are in scripts/ in this skill directory. If the skill has already been propagated to the project's infra/ directory (common in downstream templates), look in infra/scripts/ instead. See the Implementation Pattern section below for full setup guidance.
| Scenario | Key files in scripts/ |
|---|---|
| Azure Blob / S3 / GCS Pulumi backend | pulumi-setup.sh, taskfile-snippets.yaml |
| GitHub Actions + GPG secrets | github-deploy.yml, github-cd.yml, encrypt-secrets.sh, setup-github-secrets.sh |
| GitLab CI/CD | gitlab-ci.yml, setup-gitlab-variables.sh |
Adapting the deploy command
The example workflows use uv run pulumi up --yes directly. Before copying them, check infra/Taskfile.yaml — the project may already wrap the deploy command in a task:
cat infra/Taskfile.yaml # look for 'up-yes', 'deploy', or similar tasks
| What you find | What to use in CI |
|---|---|
up-yes task |
task up-yes — non-interactive, purpose-built for CI; prefer this over raw Pulumi |
deploy task (alias for up) |
Avoid — typically runs pulumi up interactively; only safe in CI if you confirm it passes -y internally |
| No Taskfile or no relevant task | Keep uv run pulumi up --yes as-is |
To use task in a workflow, add an install step and swap the run command:
- name: Install Task
run: pip install go-task-bin
- name: Deploy
working-directory: infra
env:
DATAROBOT_API_TOKEN: ${{ secrets.DATAROBOT_API_TOKEN }}
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
uv sync --all-extras
task up-yes
DataRobot API token (service account)
DATAROBOT_API_TOKEN should come from a DataRobot service account — a DataRobot user created for automation, not tied to anyone's personal login. This prevents CI/CD from breaking when the engineer who originally set it up leaves the team.
To set one up: ask your DataRobot admin to create a dedicated user (e.g. ci-bot@your-org.com). Under that account, go to Developer Tools → API Key and generate a token. Store it as the DATAROBOT_API_TOKEN secret in GitHub.
Note: This is purely a DataRobot concept — it has no relation to Pulumi state management or backend configuration. "Service account" here just means a non-personal DataRobot user.
Implementation Pattern
When implementing CI/CD for an application template, follow this structure:
Project Structure:
application-template-root/
├── infra/
│ ├── README.md # ⚠️ GENERATE THIS — tailored to the chosen CI/CD platform and Pulumi backend
│ ├── Taskfile.yaml # ⚠️ CI/CD tasks go HERE — copy from infra/scripts/taskfile-snippets.yaml
│ └── scripts/ # Copy entire scripts/ directory here
│ ├── README.md # Copy from scripts/infra-README.md
│ ├── setup-github-secrets.sh
│ ├── setup-gitlab-variables.sh
│ ├── encrypt-secrets.sh
│ ├── decrypt-secrets.sh
│ ├── pulumi-setup.sh
│ ├── gitlab-ci.yml
│ ├── github-deploy.yml
│ ├── github-cd.yml
│ ├── github-destroy.yml
│ └── taskfile-snippets.yaml
├── .env # User's secrets (never commit!)
├── .env.gpg # Encrypted secrets (commit for GitHub)
├── .gitlab-ci.yml # Copy from infra/scripts/gitlab-ci.yml
├── .github/
│ └── workflows/
│ ├── deploy.yml # Copy from infra/scripts/github-deploy.yml (PR review deploys)
│ ├── cd.yml # Copy from infra/scripts/github-cd.yml (push-to-main CD)
│ └── destroy.yml # Copy from infra/scripts/github-destroy.yml
└── Taskfile.yml # Root Taskfile — ADD ONLY one `includes` entry (see below). DO NOT add tasks here.
Key Points:
- ⚠️ ALWAYS generate
infra/README.mdtailored to the chosen platform and backend — see "Generating infra/README.md" below - All CI/CD scripts go in
infra/scripts/directory - ⚠️ CRITICAL: All CI/CD tasks go in
infra/Taskfile.yaml— NEVER add CI/CD tasks directly to the rootTaskfile.yml .envand.env.gpgstay in project root- Scripts in
infra/scripts/reference../../.env(two levels up) - Root
Taskfile.ymlgets exactly ONE addition: anincludesentry pointing to./infra/Taskfile.yaml - CI/CD configs (
.gitlab-ci.yml,.github/workflows/) are copied to standard locations
Root Taskfile.yml — the only change needed:
# Add this includes block to the existing root Taskfile.yml:
includes:
infra:
taskfile: ./infra/Taskfile.yaml
dir: infra
# Tasks are then run as: task infra:encrypt-secrets, task infra:setup-github-secrets, etc.
Generating infra/README.md
After determining the user's CI/CD platform (GitHub/GitLab) and Pulumi backend, always create infra/README.md with content tailored to their choices. It should cover:
- Architecture overview — which platform was chosen and why, and which Pulumi backend
- First-time setup — the exact sequence of
task infra:*commands needed to bootstrap - Day-to-day tasks — a table or list of the
task infra:*commands relevant to their platform - How deployments work — short description of each trigger:
- GitHub:
deploy.ymlfires on PR open/sync (review stack),cd.ymlfires on push to main (CI stack),destroy.ymlis manual - GitLab:
review_appis manual on MR,deploy_cifires on push to default branch,destroy_review_appis manual
- GitHub:
- Secrets / credentials — what variables/secrets are needed and where they live (GitHub Secrets, GitLab CI/CD variables,
.env.gpg) - Stack migration note — if backend was migrated from a local stack, document what was done so future contributors understand the history
Adjust section titles, task names, and stack-naming strategy to match what was actually configured. The README should be accurate enough that a new contributor can set up CI/CD without referring to any other document.
Workflow examples
See references/workflow-examples.md for step-by-step examples covering GitLab CI/CD, GitHub Actions with GPG secrets, and continuous delivery setup.
Using Task for workflow management
Application templates use Task to simplify local development and CI/CD workflows. Task provides a unified interface for Python and TypeScript/React components.
Example Taskfile.yaml
See references/example-taskfile.yaml for a complete example.
Using Task in CI/CD
# Install Task
pip install go-task-bin
# Install dependencies
task install
# Run linters (with fixes)
task lint
# Run linters (check only)
task lint-check
# Run tests
task test
GitLab CI/CD Configuration
The complete pipeline configuration lives in scripts/gitlab-ci.yml. Copy it to your repository root:
cp infra/scripts/gitlab-ci.yml .gitlab-ci.yml
Key pipeline jobs:
lint/test— run on every same-project MRreview_app— manual deploy per MR; stack name driven by thePULUMI_STACK_REVIEW_NAMECI/CD variabledeploy_ci— automatic deploy on merge to default branch; stack name driven byPULUMI_STACK_CI_NAMEdestroy_review_app— manual cleanup of review stacks
PULUMI_STACK_REVIEW_NAME and PULUMI_STACK_CI_NAME must be set as plain CI/CD variables in GitLab (Settings → CI/CD → Variables). The pipeline file includes sensible defaults that project-level variables override.
GitHub Actions Configuration
The complete workflow files live in scripts/:
scripts/github-deploy.yml→ copy to.github/workflows/deploy.ymlscripts/github-destroy.yml→ copy to.github/workflows/destroy.yml
mkdir -p .github/workflows
cp infra/scripts/github-deploy.yml .github/workflows/deploy.yml
cp infra/scripts/github-destroy.yml .github/workflows/destroy.yml
The deploy workflow triggers on pull requests and derives PULUMI_STACK_NAME from the PULUMI_STACK_REVIEW_NAME Actions variable and the PR number. Set PULUMI_STACK_REVIEW_NAME and PULUMI_STACK_CI_NAME as repository variables (Settings → Secrets and variables → Actions → Variables tab), not secrets.
Pulumi State Management
Pulumi Cloud Backend (Recommended)
The simplest approach for managing Pulumi state:
# Install Pulumi
curl -fsSL https://get.pulumi.com | sh
# Login to Pulumi Cloud
pulumi login
# Create/select stack
pulumi stack select --create dev
# Deploy
pulumi up
CI/CD Setup: Add PULUMI_ACCESS_TOKEN to your CI/CD secrets. Get token from Pulumi Console.
DIY Backend Options
For organizations that cannot use Pulumi Cloud:
Azure Blob Storage
# Login to Azure backend
pulumi login azblob://container-name
# Set Azure credentials
export AZURE_STORAGE_ACCOUNT=myaccount
export AZURE_STORAGE_KEY=mykey
AWS S3
# Login to S3 backend
pulumi login s3://bucket-name
# AWS credentials from environment
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
Google Cloud Storage
# Login to GCS backend
pulumi login gs://bucket-name
# GCP credentials from environment
export GOOGLE_CREDENTIALS=...
Migrating Stacks to a Different Backend
When a developer has an existing local stack (a Pulumi.<stackname>.yaml file) that was
created against a different backend than the CI/CD destination, the stack state must be
exported and re-imported before switching.
pulumi-setup.sh handles this automatically: it checks pulumi whoami --verbose for the
Backend URL and compares it with the target URL. If they differ and local stack files
exist, it offers to migrate them.
Manual migration steps (if not using the script):
# 1. Confirm current backend and stacks
pulumi whoami --verbose # note "Backend URL:"
pulumi stack ls -a # list stacks on current backend
# 2. Export each stack that exists locally (Pulumi.<name>.yaml)
pulumi stack export --stack <stackname> --file <stackname>-backup.json
# 3. Login to the new backend (set any required credentials first)
# Examples:
pulumi login # Pulumi Cloud
pulumi login azblob://my-container # Azure Blob
pulumi login s3://my-bucket # AWS S3
# 4. Create the stack in the new backend and import state
pulumi stack select --create <stackname>
pulumi stack import --file <stackname>-backup.json
# 5. Clean up the backup
rm <stackname>-backup.json
Key signals that migration is needed:
pulumi whoami --verboseshowsBackend URL: file://(local) but CI/CD uses cloud storage- Backend URL domain/scheme differs between developer machine and CI target
Managing Stacks Across Environments
# List all stacks
pulumi stack ls -a
# Output:
# NAME LAST UPDATE RESOURCE COUNT
# organization/project/prod 1 day ago 15
# organization/project/staging 2 days ago 12
# organization/project/dev 1 hour ago 10
# github-pr-repo-42 3 hours ago 13
# Select and update a stack
pulumi stack select dev
pulumi up
# View stack outputs
pulumi stack output --json
# Delete a stack
pulumi stack rm review-app-123 --yes
Secrets Management
All credentials (DataRobot API token, Pulumi access token, LLM keys, cloud storage keys) are stored in .env and committed to the repository encrypted as .env.gpg. The only secret that needs to be configured in the CI/CD system directly is CICD_SECRET_PASSPHRASE (the GPG passphrase). Non-sensitive stack name variables (PULUMI_STACK_CI_NAME, PULUMI_STACK_REVIEW_NAME) are set as plain variables, not secrets.
DataRobot API token (service account)
DATAROBOT_API_TOKEN should come from a DataRobot service account — a DataRobot user created for automation, not tied to anyone's personal login. This prevents CI/CD from breaking when the engineer who originally set it up leaves the team.
To set one up: ask your DataRobot admin to create a dedicated user (e.g. ci-bot@your-org.com). Under that account, go to Developer Tools → API Key and generate a token. Store it as the DATAROBOT_API_TOKEN secret in your CI/CD system.
Note: This is purely a DataRobot concept — it has no relation to Pulumi state management or backend configuration. "Service account" here just means a non-personal DataRobot user.
GitHub
Run scripts/setup-github-secrets.sh for interactive setup — it sets CICD_SECRET_PASSPHRASE as a repository secret and PULUMI_STACK_CI_NAME / PULUMI_STACK_REVIEW_NAME as repository variables.
To encrypt .env for CI:
task infra:encrypt-secrets
# or: ./infra/scripts/encrypt-secrets.sh
Add the resulting .env.gpg to git. For local decryption:
task infra:decrypt-secrets
# or: ./infra/scripts/decrypt-secrets.sh
GitLab
Run scripts/setup-gitlab-variables.sh for interactive setup — it sets:
CICD_SECRET_PASSPHRASE— masked, for decrypting.env.gpgGITLAB_API_TOKEN— masked, for posting MR commentsPULUMI_STACK_CI_NAME/PULUMI_STACK_REVIEW_NAME— plain variables
Alternatively configure in the UI: Project Settings → CI/CD → Variables. Mark CICD_SECRET_PASSPHRASE and GITLAB_API_TOKEN as Masked and Protected.
Best practices
CI/CD Pipeline Design
- Fast feedback: Run linting and testing in parallel
- Manual gates: Make review apps manual to save resources
- Automatic cleanup: Provide easy ways to destroy test environments
- Stack isolation: Use unique stack names per PR/MR
- Idempotent operations: Design deployments to be safely re-runnable
Pulumi State
- Use centralized backends: Enable collaboration and CI/CD
- Stack naming conventions: Use consistent patterns (e.g.,
github-pr-{repo}-{number}) - Clean up stacks: Remove unused stacks to reduce clutter
- State locking: Backends handle this automatically
- Backup state: Cloud backends provide automatic backups
Security
- Never commit secrets: Use .gitignore for .env files
- Encrypt sensitive data: Use GPG for GitHub, CI/CD variables for GitLab
- Rotate credentials: Regularly update API tokens and keys
- Scope permissions: Use least-privilege access for service accounts
- Audit access: Monitor who has access to secrets
Resource Management
- Tag resources: Use consistent tagging for tracking
- Set TTLs: Consider time-to-live for review environments
- Monitor costs: Track resource usage per environment
- Auto-cleanup: Implement automatic deletion of old review apps
- Resource limits: Set quotas to prevent runaway costs
Troubleshooting
Common Issues
Pulumi state conflicts:
- Ensure only one deployment runs at a time per stack
- Use unique stack names for concurrent deployments
- Check backend connection and credentials
Secret decryption failures:
- Verify GPG passphrase is correct
- Check .env.gpg file is in repository
- Ensure GPG is installed in CI environment
Deployment timeouts:
- Increase timeout values in workflow
- Check DataRobot API connectivity
- Verify resource provisioning isn't blocked
Stack not found:
- List stacks:
pulumi stack ls -a - Verify backend connection
- Check stack name matches pattern
Resource conflicts:
- Use unique names per stack
- Check for orphaned resources
- Review Pulumi state for inconsistencies
Example Repositories
Reference implementations:
- GitLab: demo-data-agent - Complete GitLab CI/CD setup
- GitHub: demo-talk-to-my-data-agent - Complete GitHub Actions setup
Resources
Files (datarobot-agent-skills)
-
examples
-
github-cd-pulumi-cloud.yml 1.1 KB
--- # Continuous delivery: deploy to the persistent CI stack on every push to main. # # Required GitHub Secrets (Settings → Secrets and variables → Actions → Secrets): # DATAROBOT_API_TOKEN — DataRobot service account API key # PULUMI_ACCESS_TOKEN — from https://app.pulumi.com/account/tokens # # Optional GitHub Variable (defaults to "ci" if not set): # PULUMI_STACK_CI_NAME — stack name for the CI environment (e.g. "ci", "prod") # # Pulumi Cloud uses PULUMI_ACCESS_TOKEN automatically — no explicit login step needed. name: Deploy (CD) on: push: branches: [main] permissions: contents: read jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: astral-sh/setup-uv@v5 - name: Deploy working-directory: infra env: DATAROBOT_API_TOKEN: ${{ secrets.DATAROBOT_API_TOKEN }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} PULUMI_STACK: ${{ vars.PULUMI_STACK_CI_NAME || 'ci' }} run: | uv sync --all-extras uv run pulumi stack select --create $PULUMI_STACK uv run pulumi up --yes -
github-deploy-pulumi-cloud.yml 1.3 KB
--- # PR preview deployment: manually triggered so you control when a PR stack is created. # Go to: Actions → Deploy (PR Preview) → Run workflow → select your PR branch → enter stack name. # # Tip: use the PR number as the stack name (e.g. "pr-42") so it's easy to track and clean up. # # Required GitHub Secrets (Settings → Secrets and variables → Actions → Secrets): # DATAROBOT_API_TOKEN — DataRobot service account API key # PULUMI_ACCESS_TOKEN — from https://app.pulumi.com/account/tokens name: Deploy (PR Preview) on: workflow_dispatch: inputs: stack_name: description: "Stack name for this PR preview (e.g. 'pr-42')" required: true default: "pr-preview" type: string permissions: contents: read jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: astral-sh/setup-uv@v5 - name: Deploy PR Preview Stack working-directory: infra env: DATAROBOT_API_TOKEN: ${{ secrets.DATAROBOT_API_TOKEN }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} PULUMI_STACK: ${{ github.event.inputs.stack_name }} run: | uv sync --all-extras uv run pulumi stack select --create $PULUMI_STACK uv run pulumi up --yes -
github-destroy-pulumi-cloud.yml 1.2 KB
--- # Manual stack destroy: triggered from the GitHub Actions UI (Actions → Destroy Stack → Run workflow). # Use this to clean up review stacks after a PR is merged or closed. # # Required GitHub Secrets: # DATAROBOT_API_TOKEN — DataRobot service account API key # PULUMI_ACCESS_TOKEN — from https://app.pulumi.com/account/tokens name: Destroy Stack on: workflow_dispatch: inputs: stack_name: description: "Stack name to destroy (e.g. review-pr-42)" required: true type: string permissions: contents: read jobs: destroy: runs-on: ubuntu-latest env: # Bind input to an env var — prevents it from being interpolated directly into shell. PULUMI_STACK: ${{ github.event.inputs.stack_name }} steps: - uses: actions/checkout@v4 - uses: astral-sh/setup-uv@v5 - name: Destroy Stack working-directory: infra env: DATAROBOT_API_TOKEN: ${{ secrets.DATAROBOT_API_TOKEN }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} run: | uv sync --all-extras uv run pulumi stack select $PULUMI_STACK uv run pulumi destroy --yes uv run pulumi stack rm --yes -
workflows-README.md 3 KB
# CI/CD Setup This directory contains GitHub Actions workflows for deploying to DataRobot using Pulumi Cloud. ## Workflows | File | Trigger | What it does | |------|---------|--------------| | `cd.yml` | Automatic — every merge to `main` | Deploys `main` to the persistent CI stack | | `deploy-pr.yml` | Manual — you choose branch and stack name | Deploys a preview stack from a PR branch | | `destroy.yml` | Manual — you enter the stack name | Destroys a named Pulumi stack | `cd.yml` is the only workflow that runs automatically. Everything else requires you to trigger it from the Actions tab. ## One-time setup ### 1. Create a Pulumi Cloud account Sign up at [app.pulumi.com](https://app.pulumi.com) (free tier is sufficient for most projects). Create an access token: click your **profile icon → Access Tokens → Create token**. Copy the token — you will need it in step 3. ### 2. Get a DataRobot API key Ask your DataRobot admin to create a **service account** user (e.g. `ci-bot@your-org.com`) and generate an API key under **Developer Tools → API Key**. Using a service account instead of a personal API key prevents CI/CD from breaking when people leave the team. > If you are just getting started, you can use your own personal API key temporarily and replace it later. ### 3. Add GitHub Secrets Go to: **Settings → Secrets and variables → Actions → Secrets tab → New repository secret** | Secret name | Where to get it | |-------------|----------------| | `DATAROBOT_API_TOKEN` | DataRobot → Developer Tools → API Key | | `PULUMI_ACCESS_TOKEN` | Pulumi Cloud → Profile → Access Tokens | Both secrets are required. The workflows will not run without them. ### 4. (Optional) Set a custom CI stack name The `cd.yml` workflow deploys to a Pulumi stack named `ci` by default. To use a different name, add a repository variable: **Settings → Secrets and variables → Actions → Variables tab → New repository variable** | Variable name | Value | |---------------|-------| | `PULUMI_STACK_CI_NAME` | Your preferred stack name (e.g. `ci`, `prod`, `staging`) | Skip this step if `ci` is fine. ## How to trigger deployments **Automatic deploy (merge to main):** `cd.yml` runs automatically every time code is merged to `main`. No action needed. **Manual PR preview deploy:** 1. Open a PR and get ready to test it in a live environment. 2. Go to **Actions → Deploy (PR Preview)** in the GitHub UI. 3. Click **Run workflow**. 4. Select your PR branch from the dropdown. 5. Enter a stack name — tip: use the PR number (e.g. `pr-42`) so it's easy to track. 6. Click **Run workflow**. Each stack is isolated, so multiple PRs can have their own live environment. **Destroy a stack (manual cleanup):** 1. Go to **Actions → Destroy Stack**. 2. Click **Run workflow → Run workflow**. 3. Enter the exact stack name you want to destroy (e.g. `pr-42` or `ci`). > The CI stack (`ci`) is recreated automatically on the next merge to `main`. Destroy it only if you want to tear everything down.
-
-
references
-
example-taskfile.yaml 895 B
--- version: '3' dotenv: - .env includes: react: taskfile: ./frontend_react/react_src/Taskfile.yaml dir: ./frontend_react/react_src/ tasks: install: desc: 📦 Install all dependencies cmds: - uv venv .venv - source .venv/bin/activate && uv pip install -r requirements.txt - task: react:install python-lint: desc: 🧹 Lint Python code cmds: - ruff format . - ruff check . --fix - mypy --pretty . python-lint-check: desc: 🧹 Check Python linting without fixes cmds: - ruff format --check . - ruff check . - mypy --pretty . lint: deps: - react:lint - python-lint desc: 🧹 Lint all code lint-check: deps: - react:lint-check - python-lint-check desc: 🧹 Check linting for all code test: deps: - react:test desc: 🧪 Run all tests -
workflow-examples.md 2.5 KB
# CI/CD Workflow Examples ## Example 1: Set up GitLab CI/CD with review apps **User request**: "Set up GitLab CI/CD for my application template with automated testing and manual review deployments" **Agent workflow**: 1. Create `infra/scripts/` directory in project root: `mkdir -p infra` 2. Copy entire scripts directory: `cp -R <skill-path>/scripts infra/scripts` 3. Make scripts executable: `chmod +x infra/scripts/*.sh` 4. Copy CI/CD configs to standard locations: - GitLab: `cp infra/scripts/gitlab-ci.yml .gitlab-ci.yml` - GitHub: `cp infra/scripts/github-*.yml .github/workflows/` 5. Copy tasks from `infra/scripts/taskfile-snippets.yaml` to `infra/Taskfile.yaml` Then add an `includes` entry to the root `Taskfile.yml` pointing to `./infra/Taskfile.yaml` — **do NOT paste tasks directly into root Taskfile.yml** 6. Guide user to run `task infra:setup-github-secrets` or `task infra:setup-gitlab-vars` 7. If GitHub, guide user to run `task encrypt-secrets` to encrypt `.env` file 8. **Generate `infra/README.md`** tailored to GitLab + chosen Pulumi backend 9. Test pipeline with a sample PR/MR ## Example 2: Set up GitHub Actions with encrypted secrets **User request**: "Configure GitHub Actions CI/CD with GPG-encrypted secrets and review deployments" **Agent workflow**: 1. Create `infra/scripts/` directory: `mkdir -p infra && cp -R <skill-path>/scripts infra/scripts` 2. Make scripts executable: `chmod +x infra/scripts/*.sh` 3. Copy GitHub workflows: `cp infra/scripts/github-*.yml .github/workflows/` 4. Copy `infra/scripts/taskfile-snippets.yaml` to `infra/Taskfile.yaml`: `cp infra/scripts/taskfile-snippets.yaml infra/Taskfile.yaml` Add an `includes` entry for `./infra/Taskfile.yaml` to the root `Taskfile.yml` — **do NOT paste tasks directly into root Taskfile.yml** 5. Guide user to encrypt `.env` with `task infra:encrypt-secrets` 6. Guide user to set up GitHub secrets with `task infra:setup-github-secrets` 7. Add encrypted `.env.gpg` to repository 8. **Generate `infra/README.md`** tailored to GitHub Actions + chosen Pulumi backend 9. Test workflow with a sample pull request ## Example 3: Configure continuous delivery **User request**: "Set up automatic deployment when changes are merged to main branch" **Agent workflow**: 1. Add deployment job triggered on push to main branch 2. Configure Pulumi to use persistent stack name (e.g., "ci" or "prod") 3. Set up automatic stack selection and update 4. Configure deployment to run only on successful tests 5. Add deployment status notifications 6. Document the CD process for the team
-
-
scripts
-
decrypt-secrets.sh 1.2 KB
#!/usr/bin/env bash # Copyright (c) 2026 DataRobot, Inc. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # GPG decryption script for .env files # Used for local development and testing CI/CD workflows set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" echo "🔓 Decrypt .env.gpg file" echo "========================" echo "" if [ ! -f "$REPO_ROOT/.env.gpg" ]; then echo "❌ Error: .env.gpg file not found in project root" echo "Run ./encrypt-secrets.sh first to create it" exit 1 fi if [ -f "$REPO_ROOT/.env" ]; then echo "⚠️ Warning: .env already exists" read -p "Overwrite? (y/N): " -n 1 -r echo if [[ ! $REPLY =~ ^[Yy]$ ]]; then echo "Aborted" exit 0 fi fi echo "Enter your encryption passphrase:" gpg --quiet --batch --yes --decrypt --output "$REPO_ROOT/.env" "$REPO_ROOT/.env.gpg" if [ -f "$REPO_ROOT/.env" ]; then echo "" echo "✅ Successfully decrypted .env.gpg → .env" echo "" echo "⚠️ Remember: Never commit .env to git!" echo " Make sure .env is in your .gitignore" else echo "❌ Error: Decryption failed" echo "Check your passphrase and try again" exit 1 fi -
encrypt-secrets.sh 1.9 KB
#!/usr/bin/env bash # Copyright (c) 2026 DataRobot, Inc. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # GPG encryption script for .env files # Used for GitHub Actions secrets management # See: https://docs.github.com/en/actions/security-for-github-actions/using-encrypted-secrets set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" echo "🔒 Encrypt .env file with GPG" echo "==============================" echo "" if [ ! -f "$REPO_ROOT/.env" ]; then echo "❌ Error: .env file not found in project root" echo "Create a .env file in the project root first with your secrets" exit 1 fi if [ -f "$REPO_ROOT/.env.gpg" ]; then echo "⚠️ Warning: .env.gpg already exists" read -p "Overwrite? (y/N): " -n 1 -r echo if [[ ! $REPLY =~ ^[Yy]$ ]]; then echo "Aborted" exit 0 fi fi echo "Enter a strong passphrase for encryption:" echo "(You'll need to add this to GitHub Secrets as CICD_SECRET_PASSPHRASE)" echo "" gpg --symmetric --cipher-algo AES256 --output "$REPO_ROOT/.env.gpg" "$REPO_ROOT/.env" if [ -f "$REPO_ROOT/.env.gpg" ]; then echo "" echo "✅ Successfully encrypted .env → .env.gpg" echo "" echo "Next steps:" echo "1. Add .env.gpg to git:" echo " git add .env.gpg" echo " git commit -m 'Add encrypted secrets'" echo "" echo "2. Add the passphrase to GitHub:" echo " - Go to Settings → Secrets and variables → Actions" echo " - Create new secret: CICD_SECRET_PASSPHRASE" echo " - Paste your passphrase as the value" echo "" echo "3. Ensure .env is in .gitignore (never commit plaintext secrets!)" echo "" echo "⚠️ IMPORTANT: Store your passphrase securely!" echo " If you lose it, you'll need to re-encrypt with a new passphrase." else echo "❌ Error: Encryption failed" exit 1 fi -
github-cd.yml 2 KB
--- # Example GitHub Actions continuous delivery workflow for DataRobot Application Templates # Copy this to your repository as .github/workflows/cd.yml # # Triggers on every push to the default branch and deploys to the persistent CI stack. # The stack name is controlled by the PULUMI_STACK_CI_NAME repository variable # (set via: gh variable set PULUMI_STACK_CI_NAME --body 'ci'). name: Pulumi CD on: push: branches: - main # Restrict default permissions for all jobs — each job grants only what it needs. permissions: contents: read env: PULUMI_STACK: ${{ vars.PULUMI_STACK_CI_NAME || 'ci' }} jobs: deploy: name: pulumi-deploy-ci-stack runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Decrypt Secrets run: | gpg --quiet --batch --yes --decrypt --passphrase="$CICD_SECRET_PASSPHRASE" --output .env .env.gpg # Load all .env variables into the GitHub Actions environment for subsequent steps grep -v '^#' .env | grep -v '^$' >> $GITHUB_ENV env: # Only this one secret needs to be stored in GitHub repository secrets CICD_SECRET_PASSPHRASE: ${{ secrets.CICD_SECRET_PASSPHRASE }} - uses: actions/setup-python@v5 with: python-version: 3.12 - name: Install Pulumi run: | curl -fsSL https://get.pulumi.com | sh echo "$HOME/.pulumi/bin" >> $GITHUB_PATH - name: Setup Project Dependencies run: | command -v uv >/dev/null 2>&1 || curl -LsSf https://astral.sh/uv/install.sh | sh uv venv .venv source .venv/bin/activate uv pip install -r requirements.txt - name: Deploy CI Stack run: | source .venv/bin/activate export $(grep -v '^#' .env | xargs) # PULUMI_STACK selects the persistent CI stack; --create initialises it if it doesn't exist yet pulumi stack select --create $PULUMI_STACK pulumi up --yes echo "✅ Deployed CI stack: $PULUMI_STACK" -
github-deploy.yml 5 KB
--- # Example GitHub Actions deployment workflow for DataRobot Application Templates # Copy this to your repository as .github/workflows/deploy.yml name: Pulumi Deployment on: pull_request: types: [opened, synchronize, reopened] # Restrict default permissions for all jobs — each job grants only what it needs. permissions: contents: read env: PULUMI_STACK: ${{ vars.PULUMI_STACK_REVIEW_NAME || 'review' }}-${{ github.event.number }} jobs: # Test and lint job - runs first test: name: test-and-lint runs-on: ubuntu-latest # Skip entirely for fork PRs — secrets are withheld by GitHub for forks anyway, # but this makes the intent explicit and avoids wasting CI minutes. if: github.event.pull_request.head.repo.full_name == github.repository steps: - uses: actions/checkout@v4 - name: Decrypt Secrets run: | gpg --quiet --batch --yes --decrypt --passphrase="$CICD_SECRET_PASSPHRASE" --output .env .env.gpg # Load all .env variables into the GitHub Actions environment for subsequent steps grep -v '^#' .env | grep -v '^$' >> $GITHUB_ENV env: # Only this one secret needs to be stored in GitHub repository secrets CICD_SECRET_PASSPHRASE: ${{ secrets.CICD_SECRET_PASSPHRASE }} - uses: actions/setup-python@v5 with: python-version: 3.12 - name: Install Task run: pip install go-task-bin - name: Install Dependencies run: task install - name: Lint run: task lint-check - name: Test run: task test # Deployment job - runs after tests pass deploy: name: pulumi-deploy-stack runs-on: ubuntu-latest needs: test # Same fork guard as the test job above. if: github.event.pull_request.head.repo.full_name == github.repository permissions: contents: read # required for actions/checkout on private repos pull-requests: write # required for peter-evans/create-or-update-comment steps: - uses: actions/checkout@v4 - name: Decrypt Secrets run: | gpg --quiet --batch --yes --decrypt --passphrase="$CICD_SECRET_PASSPHRASE" --output .env .env.gpg # Load all .env variables into the GitHub Actions environment for subsequent steps grep -v '^#' .env | grep -v '^$' >> $GITHUB_ENV env: # Only this one secret needs to be stored in GitHub repository secrets CICD_SECRET_PASSPHRASE: ${{ secrets.CICD_SECRET_PASSPHRASE }} - uses: actions/setup-python@v5 with: python-version: 3.12 - name: Install Pulumi run: | curl -fsSL https://get.pulumi.com | sh echo "$HOME/.pulumi/bin" >> $GITHUB_PATH - name: Setup Project Dependencies run: | command -v uv >/dev/null 2>&1 || curl -LsSf https://astral.sh/uv/install.sh | sh uv venv .venv source .venv/bin/activate uv pip install -r requirements.txt - name: Plan Pulumi Update id: plan_pulumi_update run: | source .venv/bin/activate export $(grep -v '^#' .env | xargs) # PULUMI_STACK env var selects the stack; --create initialises it if it doesn't exist yet pulumi stack select --create $PULUMI_STACK pulumi up --yes # Store JSON output once and parse it for all values PULUMI_OUTPUT=$(pulumi stack output --json) APPLICATION_URL=$(echo "$PULUMI_OUTPUT" | jq -r 'to_entries[] | select(.key | startswith("Data Analyst Application")) | .value') DEPLOYMENT_URL=$(echo "$PULUMI_OUTPUT" | jq -r 'to_entries[] | select(.key | startswith("Generative Analyst Deployment")) | .value') APP_ID=$(echo "$PULUMI_OUTPUT" | jq -r '.DATAROBOT_APPLICATION_ID // empty') LLM_ID=$(echo "$PULUMI_OUTPUT" | jq -r '.LLM_DEPLOYMENT_ID // empty') echo "application_url=${APPLICATION_URL}" >> $GITHUB_OUTPUT echo "deployment_url=${DEPLOYMENT_URL}" >> $GITHUB_OUTPUT echo "app_id=${APP_ID}" >> $GITHUB_OUTPUT echo "llm_id=${LLM_ID}" >> $GITHUB_OUTPUT # PULUMI_ACCESS_TOKEN and all other secrets are sourced from .env.gpg above - name: Comment PR with App URL uses: peter-evans/create-or-update-comment@v4 with: token: ${{ secrets.GITHUB_TOKEN }} issue-number: ${{ github.event.number }} body: | # 🚀 Your application is ready! ## Application Info - **Application URL:** [${{ steps.plan_pulumi_update.outputs.application_url }}](${{ steps.plan_pulumi_update.outputs.application_url }}) - **Application ID:** `${{ steps.plan_pulumi_update.outputs.app_id }}` ## LLM Deployment - **Deployment URL:** [${{ steps.plan_pulumi_update.outputs.deployment_url }}](${{ steps.plan_pulumi_update.outputs.deployment_url }}) - **Deployment ID:** `${{ steps.plan_pulumi_update.outputs.llm_id }}` ### Pulumi Stack - **Stack Name:** `${{ env.PULUMI_STACK }}` -
github-destroy.yml 1.9 KB
--- # Example GitHub Actions destroy workflow for DataRobot Application Templates # Copy this to your repository as .github/workflows/destroy.yml # This workflow allows manual destruction of review app stacks name: Pulumi Stack Destroy on: workflow_dispatch: inputs: stack_name: description: 'Stack name to destroy (e.g. github-pr-foobar-42)' required: true type: string jobs: destroy: name: pulumi-destroy-stack runs-on: ubuntu-latest permissions: contents: read env: # Bind the user-supplied input to PULUMI_STACK so it is never interpolated # directly into the shell script (prevents script injection attacks). PULUMI_STACK: ${{ github.event.inputs.stack_name }} steps: - uses: actions/checkout@v4 - name: Decrypt Secrets run: | gpg --quiet --batch --yes --decrypt --passphrase="$CICD_SECRET_PASSPHRASE" --output .env .env.gpg # Load all .env variables into the GitHub Actions environment for subsequent steps grep -v '^#' .env | grep -v '^$' >> $GITHUB_ENV env: # Only this one secret needs to be stored in GitHub repository secrets CICD_SECRET_PASSPHRASE: ${{ secrets.CICD_SECRET_PASSPHRASE }} - uses: actions/setup-python@v5 with: python-version: 3.12 - name: Install Pulumi run: | curl -fsSL https://get.pulumi.com | sh echo "$HOME/.pulumi/bin" >> $GITHUB_PATH - name: Setup Project Dependencies run: | command -v uv >/dev/null 2>&1 || curl -LsSf https://astral.sh/uv/install.sh | sh uv venv .venv source .venv/bin/activate uv pip install -r requirements.txt - name: Destroy Pulumi Stack run: | source .venv/bin/activate # PULUMI_STACK env var identifies the stack for all commands pulumi destroy --yes pulumi stack rm --yes -
gitlab-ci.yml 4.7 KB
--- # Example GitLab CI/CD configuration for DataRobot Application Templates # Copy this to your repository root as .gitlab-ci.yml image: cimg/python:3.11-node variables: DATAROBOT_ENDPOINT: https://app.datarobot.com/api/v2 FRONTEND_TYPE: react # ── Pulumi stack names ───────────────────────────────────────────────────── # Set PULUMI_STACK_CI_NAME and PULUMI_STACK_REVIEW_NAME as plain CI/CD # variables in GitLab (Settings → CI/CD → Variables) to override these # defaults. Project variables always take precedence over the values below. PULUMI_STACK_REVIEW_NAME: review-mr PULUMI_STACK_CI_NAME: ci # ── Secrets ──────────────────────────────────────────────────────────────── # Only one secret needs to be set in GitLab CI/CD settings: # CICD_SECRET_PASSPHRASE: "$CICD_SECRET_PASSPHRASE" (GPG passphrase to decrypt .env.gpg) # ⚠️ CICD_SECRET_PASSPHRASE MUST be set as a "Protected variable" in GitLab so it is # never exposed to pipelines triggered from forks. # All other secrets (DATAROBOT_API_TOKEN, PULUMI_ACCESS_TOKEN, LLM keys, etc.) # are stored in .env.gpg and sourced automatically in before_script. before_script: # Decrypt .env.gpg and source all secrets — only CICD_SECRET_PASSPHRASE must be set in GitLab CI/CD settings - gpg --quiet --batch --yes --decrypt --passphrase="$CICD_SECRET_PASSPHRASE" --output .env .env.gpg - export $(grep -v '^#' .env | grep -v '^$' | xargs) - pip install go-task-bin - task install - source .venv/bin/activate stages: - check - review - deploy - cleanup # Linting stage - runs on every merge request from the same project (not forks) lint: stage: check script: - task lint-check rules: - if: $CI_MERGE_REQUEST_ID && $CI_PROJECT_ID == $CI_MERGE_REQUEST_SOURCE_PROJECT_ID # Testing stage - runs on every merge request from the same project (not forks) test: stage: check script: - task test rules: - if: $CI_MERGE_REQUEST_ID && $CI_PROJECT_ID == $CI_MERGE_REQUEST_SOURCE_PROJECT_ID # Review app deployment - manual trigger for MR validation (same-project MRs only) review_app: stage: review variables: PULUMI_STACK: ${PULUMI_STACK_REVIEW_NAME}-${CI_MERGE_REQUEST_IID} script: # Install Pulumi - curl -fsSL https://get.pulumi.com | sh - export PATH="$HOME/.pulumi/bin:$PATH" # Login to Pulumi backend (DIY backend example using Azure Blob) - pulumi login --cloud-url "azblob://dr-ai-apps-pulumi" # Create the stack if it doesn't exist (PULUMI_STACK selects it for all subsequent commands) - pulumi stack select --create $PULUMI_STACK # Deploy the stack - pulumi up --yes - echo "Deploying review app for $PULUMI_STACK" # Get stack outputs and format for GitLab comment - STACK_OUTPUT="<br><br>$(pulumi stack output --shell)" - STACK_OUTPUT="${STACK_OUTPUT//$'\n'/<br>}" # Post comment to merge request with deployment info - | curl --header "PRIVATE-TOKEN: $GITLAB_API_TOKEN" \ --data "body=Review Deployment: $STACK_OUTPUT" \ "$CI_API_V4_URL/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes" rules: - if: $CI_MERGE_REQUEST_ID && $CI_PROJECT_ID == $CI_MERGE_REQUEST_SOURCE_PROJECT_ID when: manual # Cleanup review app - manual trigger to destroy resources destroy_review_app: stage: cleanup variables: PULUMI_STACK: ${PULUMI_STACK_REVIEW_NAME}-${CI_MERGE_REQUEST_IID} script: - curl -fsSL https://get.pulumi.com | sh - export PATH="$HOME/.pulumi/bin:$PATH" - pulumi login --cloud-url "azblob://dr-ai-apps-pulumi" # Destroy the stack (PULUMI_STACK identifies which stack to target) - pulumi destroy --yes # Remove the stack from Pulumi backend - pulumi stack rm --yes - echo "Destroyed $PULUMI_STACK stack" rules: - if: $CI_MERGE_REQUEST_ID && $CI_PROJECT_ID == $CI_MERGE_REQUEST_SOURCE_PROJECT_ID when: manual - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH when: manual needs: - job: review_app optional: true # Continuous delivery - automatic deployment on merge to default branch deploy_ci: stage: deploy variables: PULUMI_STACK: $PULUMI_STACK_CI_NAME script: - curl -fsSL https://get.pulumi.com | sh - export PATH="$HOME/.pulumi/bin:$PATH" - pulumi login --cloud-url "azblob://dr-ai-apps-pulumi" # Select stack, creating it if it doesn't exist yet - pulumi stack select --create $PULUMI_STACK - pulumi up --yes - echo "Deployed $PULUMI_STACK stack" rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH when: on_success -
infra-README.md 6.4 KB
# Infrastructure and CI/CD Scripts This directory contains all CI/CD and infrastructure-related scripts for the application template. ## Directory Structure ``` project-root/ ├── infra/ │ ├── Taskfile.yaml # ⚠️ CI/CD tasks go HERE (copy from infra/scripts/taskfile-snippets.yaml) │ └── scripts/ # CI/CD scripts directory │ ├── setup-github-secrets.sh # GitHub secrets setup (gh CLI) │ ├── setup-gitlab-variables.sh # GitLab variables setup (glab CLI) │ ├── encrypt-secrets.sh # Encrypt root .env → .env.gpg │ ├── decrypt-secrets.sh # Decrypt root .env.gpg → .env │ ├── pulumi-setup.sh # Pulumi backend configuration │ ├── gitlab-ci.yml # GitLab CI/CD template │ ├── github-deploy.yml # GitHub Actions deploy template │ ├── github-destroy.yml # GitHub Actions destroy template │ ├── taskfile-snippets.yaml # Source for infra/Taskfile.yaml (do NOT paste into root Taskfile.yml) │ └── README.md # This file ├── .env # Secrets (never commit!) ├── .env.gpg # Encrypted secrets (commit this) ├── .gitlab-ci.yml # Copied from infra/scripts/gitlab-ci.yml ├── .github/ │ └── workflows/ │ ├── deploy.yml # Copied from infra/scripts/github-deploy.yml │ └── destroy.yml # Copied from infra/scripts/github-destroy.yml └── Taskfile.yml # Root Taskfile — ONLY add includes entry for infra/Taskfile.yaml ``` ## Scripts Overview ### Secrets Management #### `encrypt-secrets.sh` Encrypts `../../.env` to `../../.env.gpg` using GPG for GitHub Actions. **Usage:** ```bash cd infra/scripts ./encrypt-secrets.sh # Or from root: task encrypt-secrets ``` #### `decrypt-secrets.sh` Decrypts `../../.env.gpg` to `../../.env` for local development. **Usage:** ```bash cd infra/scripts ./decrypt-secrets.sh # Or from root: task decrypt-secrets ``` #### `setup-github-secrets.sh` Interactive setup of GitHub repository secrets using `gh` CLI. **Usage:** ```bash cd infra/scripts ./setup-github-secrets.sh # Or from root: task setup-github-secrets ``` **Prerequisites:** - GitHub CLI installed: `brew install gh` - Authenticated: `gh auth login` #### `setup-gitlab-variables.sh` Interactive setup of GitLab project variables using `glab` CLI. **Usage:** ```bash cd infra/scripts ./setup-gitlab-variables.sh # Or from root: task setup-gitlab-vars ``` **Prerequisites:** - GitLab CLI installed: `brew install glab` - Authenticated: `glab auth login` ### Infrastructure Setup #### `pulumi-setup.sh` Interactive Pulumi backend configuration with support for: - Pulumi Cloud - Azure Blob Storage - AWS S3 - Google Cloud Storage **Usage:** ```bash cd infra/scripts ./pulumi-setup.sh ``` ### CI/CD Configuration Templates #### `gitlab-ci.yml` Complete GitLab CI/CD pipeline template with: - Automated linting and testing - Manual review app deployments - Continuous delivery on merge to main - Pulumi DIY backend support **Setup:** ```bash cp infra/scripts/gitlab-ci.yml .gitlab-ci.yml ``` #### `github-deploy.yml` GitHub Actions deployment workflow with: - Automated testing and linting - PR-based review deployments - GPG-encrypted secrets support - PR comments with deployment URLs **Setup:** ```bash mkdir -p .github/workflows cp infra/scripts/github-deploy.yml .github/workflows/deploy.yml ``` #### `github-destroy.yml` GitHub Actions workflow for manual stack cleanup. **Setup:** ```bash cp infra/scripts/github-destroy.yml .github/workflows/destroy.yml ``` ### Taskfile Integration #### `taskfile-snippets.yaml` Contains Task definitions to copy to `infra/Taskfile.yaml`. > ⚠️ **Do NOT paste these tasks into the root `Taskfile.yml`.** Instead: > 1. Copy this file: `cp infra/scripts/taskfile-snippets.yaml infra/Taskfile.yaml` > 2. Add a single `includes` entry to the root `Taskfile.yml`: > ```yaml > includes: > infra: > taskfile: ./infra/Taskfile.yaml > dir: . > ``` > 3. Run tasks as: `task infra:encrypt-secrets`, `task infra:setup-github-secrets`, etc. ## Quick Start ### For GitLab 1. **Setup infrastructure:** ```bash cp infra/scripts/gitlab-ci.yml .gitlab-ci.yml ``` 2. **Configure secrets:** ```bash task setup-gitlab-vars ``` 3. **Push and test:** ```bash git add .gitlab-ci.yml git commit -m "Add GitLab CI/CD" git push ``` ### For GitHub 1. **Setup infrastructure:** ```bash mkdir -p .github/workflows cp infra/scripts/github-deploy.yml .github/workflows/deploy.yml cp infra/scripts/github-destroy.yml .github/workflows/destroy.yml ``` 2. **Encrypt secrets:** ```bash task encrypt-secrets git add .env.gpg ``` 3. **Configure GitHub secrets:** ```bash task setup-github-secrets ``` 4. **Push and test:** ```bash git add .github/ .env.gpg git commit -m "Add GitHub Actions CI/CD" git push ``` ## File Locations - **Secrets**: `.env` and `.env.gpg` are in the **project root**, not in `infra/` - **Scripts**: All scripts are in `infra/scripts/` directory - **CI/CD configs**: Copied from `infra/scripts/` to their standard locations (`.gitlab-ci.yml`, `.github/workflows/`) - **Taskfile**: `infra/Taskfile.yaml` contains all CI/CD tasks. Root `Taskfile.yml` includes it via `includes: {infra: {taskfile: ./infra/Taskfile.yaml}}` ## Security Notes - **Never commit `.env`** - Add to `.gitignore` - **Do commit `.env.gpg`** - It's encrypted and safe - **Store GPG passphrase** in GitHub Secrets as `CICD_SECRET_PASSPHRASE` - **Mark GitLab variables** as "Masked" to hide in logs - **Rotate credentials** regularly ## Troubleshooting ### Scripts won't run ```bash chmod +x infra/scripts/*.sh ``` ### Can't find .env Make sure `.env` is in the project root, not in `infra/` ### GPG decryption fails Ensure you're using the same passphrase used for encryption ### gh/glab commands fail Make sure you're authenticated: ```bash gh auth login # GitHub glab auth login # GitLab ``` ## Resources - [DataRobot CI/CD Skill Documentation](../SKILL.md) - [Task Documentation](https://taskfile.dev) - [Pulumi Documentation](https://www.pulumi.com/docs/) - [GitHub Actions](https://docs.github.com/actions) - [GitLab CI/CD](https://docs.gitlab.com/ci/) -
pulumi-setup.sh 10.8 KB
#!/usr/bin/env bash # Copyright (c) 2026 DataRobot, Inc. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # Pulumi setup script for DataRobot Application Templates # This script helps initialize Pulumi with different backend options set -euo pipefail # Resolve the repo root (script lives at repo_root/infra/scripts/) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" DOTENV="$REPO_ROOT/.env" echo "🎯 DataRobot Application Template - Pulumi Setup" echo "==================================================" echo "" # Detect existing stacks on a different backend and offer to migrate them. # Must be called AFTER exporting any required cloud credentials to the environment # but BEFORE calling `pulumi login` for the new backend. migrate_stacks_if_needed() { local target_backend_url="$1" # Get the current backend URL from `pulumi whoami --verbose` local current_backend current_backend=$(pulumi whoami --verbose 2>/dev/null | grep "Backend URL:" | awk '{print $3}' || true) # Nothing to do if Pulumi isn't logged in or backend is already the target [[ -z "$current_backend" || "$current_backend" == "$target_backend_url" ]] && return 0 # Collect Pulumi.<stackname>.yaml stack config files (skip the root Pulumi.yaml) local stack_files=() while IFS= read -r -d '' f; do stack_files+=("$f") done < <(find . -maxdepth 1 -name "Pulumi.*.yaml" -not -name "Pulumi.yaml" -print0 2>/dev/null) # No local stacks — nothing to migrate [[ ${#stack_files[@]} -eq 0 ]] && return 0 echo "" echo "⚠️ Existing Pulumi stacks detected on a different backend" echo " Current backend : $current_backend" echo " Target backend : $target_backend_url" echo "" echo " Stacks found:" for f in "${stack_files[@]}"; do local sname sname=$(basename "$f" .yaml | sed 's/^Pulumi\.//') echo " - $sname" done echo "" read -rp "Migrate these stacks to the new backend? [y/N]: " MIGRATE_CHOICE if [[ "$MIGRATE_CHOICE" != "y" && "$MIGRATE_CHOICE" != "Y" ]]; then echo "⏭️ Skipping stack migration" return 0 fi # Export all stacks while still logged into the old backend local tmpdir tmpdir=$(mktemp -d) local stack_names=() for f in "${stack_files[@]}"; do local stack_name stack_name=$(basename "$f" .yaml | sed 's/^Pulumi\.//') stack_names+=("$stack_name") echo " 📤 Exporting stack '$stack_name' from $current_backend..." pulumi stack export --stack "$stack_name" --file "$tmpdir/$stack_name.json" done # Login to the new backend (cloud credentials must already be in the environment) pulumi login "$target_backend_url" # Import each stack into the new backend for stack_name in "${stack_names[@]}"; do echo " 📥 Importing stack '$stack_name' into $target_backend_url..." pulumi stack select --create "$stack_name" pulumi stack import --file "$tmpdir/$stack_name.json" echo " ✅ Migrated stack: $stack_name" done rm -rf "$tmpdir" echo "" echo "✅ Stack migration complete" } # Function to setup Pulumi Cloud backend setup_pulumi_cloud() { echo "" echo "🌐 Setting up Pulumi Cloud backend" echo "-----------------------------------" echo "1. Go to https://app.pulumi.com/account/tokens" echo "2. Create a new access token" echo "3. Enter the token below" echo "" local token_hint="" [[ -n "${PULUMI_ACCESS_TOKEN:-}" ]] && token_hint=" [already set, press Enter to keep]" read -rsp "Pulumi Access Token${token_hint}: " PULUMI_TOKEN echo "" PULUMI_TOKEN="${PULUMI_TOKEN:-${PULUMI_ACCESS_TOKEN:-}}" export PULUMI_ACCESS_TOKEN="$PULUMI_TOKEN" migrate_stacks_if_needed "https://api.pulumi.com" pulumi login echo "✅ Logged in to Pulumi Cloud" } # Function to setup Azure Blob backend setup_azure_backend() { echo "" echo "☁️ Setting up Azure Blob Storage backend" echo "----------------------------------------" local acct_default="${AZURE_STORAGE_ACCOUNT:-}" local acct_hint="" [[ -n "$acct_default" ]] && acct_hint=" [$acct_default]" read -rp "Azure Storage Account${acct_hint}: " AZURE_ACCOUNT AZURE_ACCOUNT="${AZURE_ACCOUNT:-$acct_default}" read -rp "Azure Container Name: " AZURE_CONTAINER local key_hint="" [[ -n "${AZURE_STORAGE_KEY:-}" ]] && key_hint=" [already set, press Enter to keep]" read -rsp "Azure Storage Key${key_hint}: " AZURE_KEY echo "" AZURE_KEY="${AZURE_KEY:-${AZURE_STORAGE_KEY:-}}" export AZURE_STORAGE_ACCOUNT="$AZURE_ACCOUNT" export AZURE_STORAGE_KEY="$AZURE_KEY" migrate_stacks_if_needed "azblob://$AZURE_CONTAINER" pulumi login "azblob://$AZURE_CONTAINER" echo "✅ Logged in to Azure Blob backend" # Add to .env if it exists if [[ -f "$DOTENV" ]]; then echo "AZURE_STORAGE_ACCOUNT=$AZURE_ACCOUNT" >> "$DOTENV" echo "AZURE_STORAGE_KEY=$AZURE_KEY" >> "$DOTENV" echo "📝 Added Azure credentials to $DOTENV" fi } # Function to setup AWS S3 backend setup_s3_backend() { echo "" echo "☁️ Setting up AWS S3 backend" echo "----------------------------" read -rp "S3 Bucket Name: " S3_BUCKET local aws_key_default="${AWS_ACCESS_KEY_ID:-}" local aws_key_hint="" [[ -n "$aws_key_default" ]] && aws_key_hint=" [$aws_key_default]" read -rp "AWS Access Key ID${aws_key_hint}: " AWS_KEY_ID AWS_KEY_ID="${AWS_KEY_ID:-$aws_key_default}" local aws_secret_hint="" [[ -n "${AWS_SECRET_ACCESS_KEY:-}" ]] && aws_secret_hint=" [already set, press Enter to keep]" read -rsp "AWS Secret Access Key${aws_secret_hint}: " AWS_SECRET echo "" AWS_SECRET="${AWS_SECRET:-${AWS_SECRET_ACCESS_KEY:-}}" export AWS_ACCESS_KEY_ID="$AWS_KEY_ID" export AWS_SECRET_ACCESS_KEY="$AWS_SECRET" migrate_stacks_if_needed "s3://$S3_BUCKET" pulumi login "s3://$S3_BUCKET" echo "✅ Logged in to S3 backend" # Add to .env if it exists if [[ -f "$DOTENV" ]]; then echo "AWS_ACCESS_KEY_ID=$AWS_KEY_ID" >> "$DOTENV" echo "AWS_SECRET_ACCESS_KEY=$AWS_SECRET" >> "$DOTENV" echo "📝 Added AWS credentials to $DOTENV" fi } # Function to create initial stack create_stack() { echo "" echo "📚 Creating Pulumi stack" echo "------------------------" # List stacks already in the current backend local existing_stacks existing_stacks=$(pulumi stack ls --json 2>/dev/null \ | python3 -c "import sys,json; [print(s['name']) for s in json.load(sys.stdin)]" 2>/dev/null \ || true) if [[ -n "$existing_stacks" ]]; then echo "Existing stacks in current backend:" while IFS= read -r s; do echo " - $s" done <<< "$existing_stacks" echo "" fi read -rp "Stack name (e.g., dev, staging, prod): " STACK_NAME # If the name already exists in the current backend, offer to just select it if echo "$existing_stacks" | grep -qx "$STACK_NAME" 2>/dev/null; then echo "" echo "✅ Stack '$STACK_NAME' found in the current backend" read -rp "Select and use it? [Y/n]: " USE_EXISTING if [[ "$USE_EXISTING" != "n" && "$USE_EXISTING" != "N" ]]; then pulumi stack select "$STACK_NAME" echo "✅ Selected existing stack: $STACK_NAME" return 0 fi fi # Check for a local Pulumi.<name>.yaml that doesn't exist in the current backend yet local stack_config="Pulumi.${STACK_NAME}.yaml" if [[ -f "$stack_config" ]] && ! echo "$existing_stacks" | grep -qx "$STACK_NAME" 2>/dev/null; then echo "" echo "📄 Found local stack config '$stack_config' but no matching stack in the current backend" read -rp "Import an exported stack state file into the current backend? [y/N]: " IMPORT_CHOICE if [[ "$IMPORT_CHOICE" == "y" || "$IMPORT_CHOICE" == "Y" ]]; then read -rp "Path to exported state JSON file: " STATE_FILE if [[ -n "$STATE_FILE" && -f "$STATE_FILE" ]]; then pulumi stack select --create "$STACK_NAME" pulumi stack import --file "$STATE_FILE" echo "✅ Imported stack '$STACK_NAME' into current backend" return 0 else echo "⚠️ State file not found — creating a fresh stack instead" fi fi fi pulumi stack select --create "$STACK_NAME" echo "✅ Created and selected stack: $STACK_NAME" echo "" echo "📋 Available commands:" echo " pulumi up - Deploy the stack" echo " pulumi destroy - Destroy the stack" echo " pulumi stack ls - List all stacks" echo " pulumi stack output - View stack outputs" } # Configure CI/CD secrets and variables by running the appropriate setup script. setup_cicd() { echo "" echo "🔐 Configure CI/CD secrets and variables" echo "-----------------------------------------" echo "1) GitHub Actions" echo "2) GitLab CI/CD" echo "3) Skip" read -rp "Selection [1-3]: " CICD_CHOICE case $CICD_CHOICE in 1) bash "$SCRIPT_DIR/setup-github-secrets.sh" ;; 2) bash "$SCRIPT_DIR/setup-gitlab-variables.sh" ;; 3) echo "⏭️ Skipping — run setup-github-secrets.sh or setup-gitlab-variables.sh manually" ;; *) echo "❌ Invalid selection" ;; esac } # Main setup flow main() { # Source .env so existing vars are available as defaults throughout setup if [[ -f "$DOTENV" ]]; then set -a # shellcheck source=/dev/null source "$DOTENV" set +a fi if ! command -v pulumi &> /dev/null; then echo "❌ Pulumi is not installed. Install it from https://www.pulumi.com/docs/install/ and re-run this script." exit 1 fi echo "" echo "🔧 Choose Pulumi backend:" echo "1) Pulumi Cloud (recommended)" echo "2) Azure Blob Storage" echo "3) AWS S3" echo "4) Skip (already configured)" read -rp "Selection [1-4]: " BACKEND_CHOICE case $BACKEND_CHOICE in 1) setup_pulumi_cloud ;; 2) setup_azure_backend ;; 3) setup_s3_backend ;; 4) echo "⏭️ Skipping backend setup" ;; *) echo "❌ Invalid selection" exit 1 ;; esac create_stack setup_cicd echo "" echo "🎉 Pulumi setup complete!" echo "" echo "Next steps:" echo "1. Run 'pulumi up' to deploy your application" echo "2. Push .env.gpg and your CI/CD workflow files to trigger your first pipeline" } # Run main function main -
setup-github-secrets.sh 3.6 KB
#!/usr/bin/env bash # Copyright (c) 2026 DataRobot, Inc. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # Add secrets to GitHub repository using GitHub CLI # Requires: gh CLI (https://cli.github.com/) set -euo pipefail echo "🔐 GitHub Secrets Setup" echo "=======================" echo "" # Check if gh is installed if ! command -v gh &> /dev/null; then echo "❌ GitHub CLI (gh) not installed" echo "" echo "Install with:" echo " macOS: brew install gh" echo " Linux: See https://github.com/cli/cli#installation" echo "" exit 1 fi # Check if authenticated if ! gh auth status &> /dev/null; then echo "❌ Not authenticated with GitHub CLI" echo "Run: gh auth login" exit 1 fi echo "✅ GitHub CLI authenticated" echo "" # Get repository (or use current) REPO=$(gh repo view --json nameWithOwner -q .nameWithOwner 2>/dev/null || echo "") if [ -z "$REPO" ]; then read -rp "Enter repository (owner/repo): " REPO fi echo "Repository: $REPO" echo "" # Function to add a secret (masked) add_secret() { local secret_name=$1 local secret_description=$2 local secret_value="" echo "📝 Setting up secret: $secret_name" echo " $secret_description" read -rsp " Enter value: " secret_value echo "" if [ -n "$secret_value" ]; then echo "$secret_value" | gh secret set "$secret_name" --repo "$REPO" echo " ✅ Added $secret_name" else echo " ⏭️ Skipped (empty value)" fi echo "" } # Function to add a plain (non-secret) Actions variable add_variable() { local var_name=$1 local var_description=$2 local default_value=${3:-} local var_value="" echo "📝 Setting up variable: $var_name" echo " $var_description" if [ -n "$default_value" ]; then read -rp " Enter value [$default_value]: " var_value var_value="${var_value:-$default_value}" else read -rp " Enter value: " var_value fi if [ -n "$var_value" ]; then gh variable set "$var_name" --body "$var_value" --repo "$REPO" echo " ✅ Set $var_name=$var_value" else echo " ⏭️ Skipped (empty value)" fi echo "" } # Add secrets echo "─── Secrets (encrypted) ────────────────────────────────────────────────" echo "This setup requires ONE secret." echo "All other credentials (DATAROBOT_API_TOKEN, PULUMI_ACCESS_TOKEN, LLM keys, etc.)" echo "are stored encrypted in .env.gpg and loaded automatically by the workflow." echo "" add_secret "CICD_SECRET_PASSPHRASE" "GPG passphrase for decrypting .env.gpg" # Add plain CI/CD variables (not sensitive — visible in logs) echo "─── Variables (plain text) ─────────────────────────────────────────────" echo "Pulumi stack names are not secrets — they are plain Actions variables." echo "" add_variable "PULUMI_STACK_CI_NAME" \ "Pulumi stack deployed on every merge to the default branch" \ "ci" add_variable "PULUMI_STACK_REVIEW_NAME" \ "Pulumi stack name prefix for PR review apps (PR number appended automatically in the workflow)" \ "review" echo "" echo "🎉 Secrets and variables setup complete!" echo "" echo "View all secrets: gh secret list --repo $REPO" echo "View all variables: gh variable list --repo $REPO" echo "" echo "Next steps:" echo "1. Ensure your .env.gpg is committed to the repository" echo "2. Push your .github/workflows to trigger actions" echo "3. Test with a pull request" -
setup-gitlab-variables.sh 4.4 KB
#!/usr/bin/env bash # Copyright (c) 2026 DataRobot, Inc. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # Add variables to GitLab project using GitLab CLI # Requires: glab CLI (https://gitlab.com/gitlab-org/cli) set -euo pipefail echo "🔐 GitLab CI/CD Variables Setup" echo "================================" echo "" # Check if glab is installed if ! command -v glab &> /dev/null; then echo "❌ GitLab CLI (glab) not installed" echo "" echo "Install with:" echo " macOS: brew install glab" echo " Linux: See https://gitlab.com/gitlab-org/cli#installation" echo "" exit 1 fi # Check if authenticated if ! glab auth status &> /dev/null; then echo "❌ Not authenticated with GitLab CLI" echo "Run: glab auth login" exit 1 fi echo "✅ GitLab CLI authenticated" echo "" # Get project (or use current) PROJECT=$(glab repo view --output json 2>/dev/null | jq -r '.path_with_namespace' || echo "") if [ -z "$PROJECT" ]; then read -rp "Enter project (group/project): " PROJECT fi echo "Project: $PROJECT" echo "" # Function to add a variable add_variable() { local var_name=$1 local var_description=$2 local var_value="" local mask_flag="--masked" # Mask by default for security echo "📝 Setting up: $var_name" echo " $var_description" read -rsp " Enter value: " var_value echo "" if [ -n "$var_value" ]; then # GitLab CLI command to set variable glab variable set "$var_name" "$var_value" --scope="*" $mask_flag --repo "$PROJECT" 2>/dev/null || \ glab variable update "$var_name" "$var_value" --scope="*" $mask_flag --repo "$PROJECT" 2>/dev/null echo " ✅ Added $var_name" else echo " ⏭️ Skipped (empty value)" fi echo "" } # Function to add a plain (non-masked) variable add_plain_variable() { local var_name=$1 local var_description=$2 local default_value=${3:-} local var_value="" echo "📝 Setting up variable: $var_name" echo " $var_description" if [ -n "$default_value" ]; then read -rp " Enter value [$default_value]: " var_value var_value="${var_value:-$default_value}" else read -rp " Enter value: " var_value fi if [ -n "$var_value" ]; then glab variable set "$var_name" "$var_value" --scope="*" --repo "$PROJECT" 2>/dev/null || \ glab variable update "$var_name" "$var_value" --scope="*" --repo "$PROJECT" 2>/dev/null echo " ✅ Set $var_name=$var_value" else echo " ⏭️ Skipped (empty value)" fi echo "" } # Add variables echo "─── Masked variables (secrets) ─────────────────────────────────────────" echo "This setup requires ONE masked CI/CD variable for the GPG passphrase." echo "All other credentials (DATAROBOT_API_TOKEN, PULUMI_ACCESS_TOKEN, LLM keys, etc.)" echo "are stored encrypted in .env.gpg and loaded automatically by the pipeline." echo "" # Core variable: the GPG passphrase to decrypt .env.gpg add_variable "CICD_SECRET_PASSPHRASE" "GPG passphrase for decrypting .env.gpg" # GitLab API token for commenting on MRs (not in .env since it's GitLab-specific) echo "" echo "📝 GitLab API Token for MR comments" echo " Create at: https://gitlab.com/-/profile/personal_access_tokens" echo " Required scopes: api" add_variable "GITLAB_API_TOKEN" "GitLab personal access token (for posting MR comments)" # Add plain (non-masked) Pulumi stack name variables echo "" echo "─── Plain variables (not sensitive) ────────────────────────────────────" echo "Pulumi stack names are not secrets — they are plain CI/CD variables." echo "" add_plain_variable "PULUMI_STACK_CI_NAME" \ "Pulumi stack deployed on every merge to the default branch" \ "ci" add_plain_variable "PULUMI_STACK_REVIEW_NAME" \ "Pulumi stack name prefix for MR review apps (MR IID appended automatically in the pipeline)" \ "review" echo "" echo "🎉 Variables setup complete!" echo "" echo "View all variables: glab variable list --repo $PROJECT" echo "Manage in UI: https://gitlab.com/$PROJECT/-/settings/ci_cd#js-cicd-variables-settings" echo "" echo "Next steps:" echo "1. Ensure your .env.gpg is committed to the repository" echo "2. Push your .gitlab-ci.yml to trigger pipelines" echo "3. Test with a merge request" -
taskfile-snippets.yaml 5.6 KB
--- # Taskfile snippets for CI/CD # ⚠️ COPY THIS FILE TO infra/Taskfile.yaml — do NOT paste tasks into the root Taskfile.yml # # Step 1: cp infra/scripts/taskfile-snippets.yaml infra/Taskfile.yaml # Step 2: Add this includes block to root Taskfile.yml (the ONLY change to root): # # includes: # infra: # taskfile: ./infra/Taskfile.yaml # dir: ./infra # # Tasks are then run as: task infra:encrypt-secrets, task infra:setup-github-secrets, etc. # Scripts referenced here are in infra/scripts/ and .env is assumed to be in project root (../../.env) version: '3' tasks: # =========================================================================== # Secrets Management Tasks # =========================================================================== encrypt-secrets: desc: 🔒 Encrypt .env file for GitHub Actions dir: scripts cmds: - ./encrypt-secrets.sh preconditions: - test -f ../../.env decrypt-secrets: desc: 🔓 Decrypt .env.gpg file for local use dir: scripts cmds: - ./decrypt-secrets.sh preconditions: - test -f ../../.env.gpg verify-secrets: desc: 🔍 Verify .env file has required variables cmds: - | REQUIRED_VARS="DATAROBOT_API_TOKEN DATAROBOT_ENDPOINT" MISSING="" for var in $REQUIRED_VARS; do if ! grep -q "^${var}=" ../.env 2>/dev/null; then MISSING="$MISSING $var" fi done if [ -n "$MISSING" ]; then echo "❌ Missing required variables:$MISSING" exit 1 fi echo "✅ All required variables present" setup-github-secrets: desc: 🔐 Setup GitHub secrets interactively dir: scripts cmds: - ./setup-github-secrets.sh setup-gitlab-vars: desc: 🔐 Setup GitLab variables interactively dir: scripts cmds: - ./setup-gitlab-variables.sh # =========================================================================== # CI/CD Deployment Tasks # =========================================================================== pulumi-login-cloud: desc: 🌐 Login to Pulumi Cloud cmds: - pulumi login preconditions: - sh: command -v pulumi msg: "Pulumi not installed. Run: curl -fsSL https://get.pulumi.com | sh" pulumi-login-azure: desc: ☁️ Login to Azure Blob backend cmds: - pulumi login azblob://{{.CONTAINER}} vars: CONTAINER: '{{.CONTAINER | default "pulumi-state"}}' preconditions: - sh: command -v pulumi msg: "Pulumi not installed" - sh: test -n "$AZURE_STORAGE_ACCOUNT" msg: "AZURE_STORAGE_ACCOUNT not set" - sh: test -n "$AZURE_STORAGE_KEY" msg: "AZURE_STORAGE_KEY not set" pulumi-login-s3: desc: ☁️ Login to S3 backend cmds: - pulumi login s3://{{.BUCKET}} vars: BUCKET: '{{.BUCKET | default "pulumi-state"}}' preconditions: - sh: command -v pulumi msg: "Pulumi not installed" - sh: test -n "$AWS_ACCESS_KEY_ID" msg: "AWS_ACCESS_KEY_ID not set" pulumi-stack-create: desc: 📚 Create new Pulumi stack cmds: - pulumi stack select --create {{.STACK}} vars: STACK: '{{.STACK | default "dev"}}' preconditions: - sh: command -v pulumi msg: "Pulumi not installed" pulumi-deploy: desc: 🚀 Deploy with Pulumi cmds: - source .venv/bin/activate && pulumi up {{.FLAGS}} vars: FLAGS: '{{.FLAGS | default "--yes"}}' preconditions: - test -d .venv - sh: command -v pulumi msg: "Pulumi not installed" pulumi-destroy: desc: 🗑️ Destroy Pulumi stack cmds: - source .venv/bin/activate && pulumi destroy {{.FLAGS}} vars: FLAGS: '{{.FLAGS | default "--yes"}}' preconditions: - test -d .venv - sh: command -v pulumi msg: "Pulumi not installed" pulumi-output: desc: 📋 Show Pulumi stack outputs cmds: - pulumi stack output {{.FORMAT}} vars: FORMAT: '{{.FORMAT | default "--json"}}' # =========================================================================== # CI/CD Workflow Testing # =========================================================================== ci-test-local: desc: 🧪 Test CI pipeline locally cmds: - task: lint-check - task: test - echo "✅ CI checks passed" ci-simulate-deploy: desc: 🎭 Simulate GitHub Actions deployment locally cmds: - task: decrypt-secrets - task: install - task: ci-test-local - task: pulumi-deploy - echo "✅ Deployment simulation complete" # ============================================================================= # Usage Examples # ============================================================================= # # Secrets management: # task infra:encrypt-secrets # Encrypt .env for GitHub # task infra:decrypt-secrets # Decrypt for local use # task infra:verify-secrets # Check required vars # # Pulumi operations: # task infra:pulumi-login-cloud # Login to Pulumi Cloud # task infra:pulumi-login-azure # Login to Azure backend # task infra:pulumi-stack-create STACK=dev # Create new stack # task infra:pulumi-deploy # Deploy current stack # task infra:pulumi-deploy FLAGS="--preview-only" # Preview changes # task infra:pulumi-destroy # Destroy current stack # task infra:pulumi-output # Show outputs # # Testing CI/CD: # task infra:ci-test-local # Run all CI checks # task infra:ci-simulate-deploy # Simulate full deployment
-
-
QUICK_REFERENCE.md 4 KB
# CI/CD Setup Quick Reference ## GitHub Secrets Setup ### Using GitHub CLI (Automated) ```bash # Install gh CLI brew install gh # macOS # or: https://cli.github.com/ # Authenticate gh auth login # Run setup script ./infra/scripts/setup-github-secrets.sh ``` ### Manual Commands ```bash # Add individual secrets echo "your-value" | gh secret set SECRET_NAME # Only ONE secret is required — all other credentials live in .env.gpg echo "your-gpg-passphrase" | gh secret set CICD_SECRET_PASSPHRASE # List secrets gh secret list # Delete a secret gh secret remove SECRET_NAME ``` ## GitLab Variables Setup ### Using GitLab CLI (Automated) ```bash # Install glab CLI brew install glab # macOS # or: https://gitlab.com/gitlab-org/cli # Authenticate glab auth login # Run setup script ./infra/scripts/setup-gitlab-variables.sh ``` ### Manual Commands ```bash # Add individual variables (masked for security) glab variable set VAR_NAME "your-value" --masked # Only TWO variables are required — all other credentials live in .env.gpg # CICD_SECRET_PASSPHRASE: GPG passphrase to decrypt .env.gpg glab variable set CICD_SECRET_PASSPHRASE "your-gpg-passphrase" --masked # GITLAB_API_TOKEN: needed for posting MR comments (GitLab-specific, not in .env) glab variable set GITLAB_API_TOKEN "your-gitlab-token" --masked # List variables glab variable list # Update a variable glab variable update VAR_NAME "new-value" --masked # Delete a variable glab variable delete VAR_NAME ``` ## Secrets Management ### Encrypt .env for GitHub ```bash # Automated ./infra/scripts/encrypt-secrets.sh # Manual gpg --symmetric --cipher-algo AES256 .env git add .env.gpg git commit -m "Add encrypted secrets" ``` ### Decrypt .env Locally ```bash # Automated ./infra/scripts/decrypt-secrets.sh # Manual gpg --quiet --batch --yes --decrypt --output .env .env.gpg ``` ## Pulumi Setup ### Interactive Setup ```bash ./infra/scripts/pulumi-setup.sh ``` ### Manual Setup ```bash # Pulumi Cloud pulumi login # Azure Blob export AZURE_STORAGE_ACCOUNT=myaccount export AZURE_STORAGE_KEY=mykey pulumi login azblob://container-name # AWS S3 export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=... pulumi login s3://bucket-name ``` ## Migrating Stacks to a Different Backend Run `pulumi-setup.sh` — it detects backend mismatches automatically. For manual migration: ```bash # 1. Check current backend and local stacks pulumi whoami --verbose # note "Backend URL:" ls Pulumi.*.yaml # stack files present locally # 2. Export each stack (while still on the old backend) pulumi stack export --stack <stackname> --file <stackname>-backup.json # 3. Login to the new backend (set credentials first) pulumi login azblob://my-container # or s3://, or pulumi login for Cloud # 4. Import into new backend pulumi stack select --create <stackname> pulumi stack import --file <stackname>-backup.json rm <stackname>-backup.json ``` > Migration is needed when `pulumi whoami --verbose` shows a different Backend URL than > the CI/CD target (e.g., local `file://` vs CI `azblob://`). ## Common Task Commands ### From taskfile-snippets.yaml > ⚠️ Tasks are namespaced under `infra:` (included from `infra/Taskfile.yaml`) ```bash # Secrets task infra:encrypt-secrets task infra:decrypt-secrets task infra:verify-secrets # Pulumi task infra:pulumi-login-cloud task infra:pulumi-login-azure task infra:pulumi-deploy task infra:pulumi-destroy task infra:pulumi-output # Testing task infra:ci-test-local task infra:ci-simulate-deploy ``` ## Quick Start Workflow ### GitHub ```bash # 1. Install tools brew install gh # 2. Setup gh auth login ./infra/scripts/encrypt-secrets.sh # encrypts .env → .env.gpg ./infra/scripts/setup-github-secrets.sh # sets only CICD_SECRET_PASSPHRASE git add .env.gpg .github/ # 3. Push and test git commit -m "Add CI/CD" git push ``` ### GitLab ```bash # 1. Install tools brew install glab # 2. Setup glab auth login ./infra/scripts/setup-gitlab-variables.sh git add .gitlab-ci.yml # 3. Push and test git commit -m "Add CI/CD" git push ``` -
README.md 6.6 KB
# DataRobot Application Templates CI/CD Skill A skill that provides comprehensive guidance for setting up production-grade CI/CD pipelines for DataRobot application templates. ## Overview Transform your DataRobot application templates from manual deployments to automated CI/CD workflows with: - **Automated Testing**: Run linters and tests on every pull/merge request - **Review Deployments**: Spin up full application stacks for PR validation - **Continuous Delivery**: Automatically deploy changes when merged to main - **Infrastructure as Code**: Use Pulumi for declarative infrastructure management - **Secrets Management**: Secure handling of API keys and credentials - **Multi-Platform Support**: Works with both GitLab and GitHub ## What's Included ### Documentation - **SKILL.md**: Complete guidance for CI/CD setup - Quick start guide - Platform-specific configurations (GitLab, GitHub) - Pulumi state management strategies - Secrets management patterns - Troubleshooting guide ### Example Configurations The `scripts/` directory contains reference implementations that should be copied to the application template's `infra/` directory: - **infra-README.md**: Documentation for the infra/ directory explaining structure and usage - **gitlab-ci.yml**: Complete GitLab CI/CD pipeline - Automated testing and linting - Manual review app deployments - Continuous delivery on merge - Azure Blob Storage backend example - **github-deploy.yml**: GitHub Actions deployment workflow - Automated testing and linting - PR-based review deployments - GPG-encrypted secrets - Pulumi Cloud backend example - **github-destroy.yml**: GitHub Actions cleanup workflow - Manual stack destruction - Resource cleanup for review apps - **setup-github-secrets.sh**: Automated GitHub secrets setup - Uses GitHub CLI (`gh`) - Interactive secret entry - Configures all required secrets for Actions - **setup-gitlab-variables.sh**: Automated GitLab variables setup - Uses GitLab CLI (`glab`) - Interactive variable entry - Configures all required variables for CI/CD - **encrypt-secrets.sh**: GPG encryption for .env files - Encrypts root .env to .env.gpg - Interactive encryption workflow - GitHub Actions secrets preparation - Step-by-step instructions - **decrypt-secrets.sh**: GPG decryption for local development - Decrypts root .env.gpg to .env - Safe local secrets management - Testing workflow simulation - **taskfile-snippets.yaml**: CI/CD Task definitions - Secrets management commands - Pulumi deployment tasks - CI/CD testing helpers - Add an `includes` entry in the project's root `Taskfile.yml` pointing to `infra/Taskfile.yaml` (do NOT add tasks to root) - **pulumi-setup.sh**: Interactive Pulumi setup script - Backend configuration (Cloud, Azure, AWS) - Initial stack creation - Credential management ## Quick Start When implementing CI/CD for an application template: 1. **Choose your platform**: GitLab or GitHub 2. **Create infra directory and copy scripts**: ```bash # Create infra directory in project root and copy entire scripts folder mkdir -p infra cp -R <skill-path>/scripts infra/scripts chmod +x infra/scripts/*.sh # For GitLab - copy to root cp infra/scripts/gitlab-ci.yml .gitlab-ci.yml # For GitHub - copy to .github/workflows/ mkdir -p .github/workflows cp infra/scripts/github-deploy.yml .github/workflows/deploy.yml cp infra/scripts/github-destroy.yml .github/workflows/destroy.yml ``` 3. **Create `infra/Taskfile.yaml`**: ```bash # Copy taskfile-snippets.yaml to infra/Taskfile.yaml cp infra/scripts/taskfile-snippets.yaml infra/Taskfile.yaml # Then add a single includes entry to root Taskfile.yml: # includes: # infra: # taskfile: ./infra/Taskfile.yaml # dir: . # ⚠️ Do NOT paste CI/CD tasks into the root Taskfile.yml ``` 4. **Set up Pulumi**: ```bash cd infra/scripts ./pulumi-setup.sh cd ../.. # Install GitHub CLI: brew install gh gh auth login task setup-github-secrets ``` For GitLab (automated): ```bash # Install GitLab CLI: brew install glab glab auth login task setup-gitlab-vars ``` 6. **Encrypt your secrets** (for GitHub only): ```bash # Creates .env.gpg in project root from .env task encrypt-secrets git add .env.gpg git commit -m "Add encrypted secrets" ``` ## Key Features ### GitLab CI/CD - Parallel test execution for faster feedback - DIY backend support (Azure Blob, S3, GCS) - MR-specific stack names for isolation - Automatic commenting with deployment info - Manual cleanup jobs ### GitHub Actions - GPG-encrypted secrets for better management - PR comments with deployment URLs - Pulumi Cloud integration - Manual destroy workflows - Matrix testing support ### Pulumi Integration - Centralized state management - Stack isolation per environment - Idempotent deployments - Cross-machine synchronization - Codespace compatibility ## Use Cases ### Development Teams - Test infrastructure changes in isolation - Review applications before merging - Automatically deploy to staging/production - Track infrastructure state across team ### DevOps Engineers - Implement IaC for AI applications - Manage multiple environments - Automate deployment workflows - Monitor infrastructure changes ### Data Scientists - Deploy models with applications - Test changes in review environments - Collaborate on application features - Focus on ML, not infrastructure ## Example Repositories See these live implementations: - **GitLab**: [demo-data-agent](https://gitlab.com/datarobot-oss/demo-data-agent) - **GitHub**: [demo-talk-to-my-data-agent](https://github.com/datarobot-forks/demo-talk-to-my-data-agent) ## Platform Support - ✅ GitLab CI/CD - ✅ GitHub Actions - ✅ Pulumi Cloud - ✅ Azure Blob Storage - ✅ AWS S3 - ✅ Google Cloud Storage ## Resources - [SKILL.md](SKILL.md) - Complete documentation - [Task](https://taskfile.dev) - Workflow management - [Pulumi](https://www.pulumi.com/docs/) - Infrastructure as Code - [DataRobot Application Templates](https://docs.datarobot.com/en/docs/wb-apps/app-templates/index.html) ## Contributing This is the **canonical upstream source** for this skill at [datarobot-agent-skills](https://github.com/datarobot-oss/datarobot-agent-skills/tree/main/skills/datarobot-app-framework-cicd). Downstream repositories (such as [af-component-base](https://github.com/datarobot-community/af-component-base)) bundle a local copy, when this skill is updated, those copies should be synced. Contributions welcome via pull request. ## License See the main repository LICENSE file. -
SKILL.md 19.5 KB
--- name: datarobot-app-framework-cicd description: Guidance for setting up CI/CD pipelines for DataRobot application templates using GitLab, GitHub Actions, and Pulumi for infrastructure as code. Use when setting up CI/CD pipelines, configuring deployments, or managing infrastructure for DataRobot application templates. context-tokens: "~6 000 (SKILL.md) + ~2 000 (scripts/*) + ~400 per examples/* file" --- # DataRobot Application Templates CI/CD Skill This skill provides comprehensive guidance for setting up production-grade CI/CD pipelines for DataRobot application templates, including automated testing, review deployments, and continuous delivery. ## Quick Start **Default behavior:** When a user asks to "set up CI/CD" without specifying a platform or backend, always use the [Simple Path](#simple-path-pulumi-cloud--github-secrets) below — three workflow files, two GitHub Secrets, done. Do not create `infra/scripts/`, do not add CI/CD tasks to `infra/Taskfile.yaml`, do not involve GPG encryption unless the user explicitly asks for it. Only deviate from the simple path when the user specifies: - A specific Pulumi state backend (Azure Blob, S3, GCS) → use `scripts/` and see [Implementation Pattern](#implementation-pattern) - GitLab CI/CD → see [GitLab CI/CD Configuration](#gitlab-cicd-configuration) - Many secrets to manage → consider GPG approach in `scripts/` ## Simple Path: Pulumi Cloud + GitHub Secrets For most data scientists and AI engineers, this is all you need. No GPG encryption, no cloud storage account, no extra scripts. **What to create in the user's repository:** 1. Copy the three workflow files to `.github/workflows/`: | Source | Destination | Trigger | |--------|-------------|---------| | `examples/github-cd-pulumi-cloud.yml` | `.github/workflows/cd.yml` | Automatic — every merge to `main` | | `examples/github-deploy-pulumi-cloud.yml` | `.github/workflows/deploy-pr.yml` | Manual — user picks PR branch + enters stack name (e.g. `pr-42`) | | `examples/github-destroy-pulumi-cloud.yml` | `.github/workflows/destroy.yml` | Manual — user enters stack name to tear down | 2. Create `.github/workflows/README.md` from `examples/workflows-README.md`. This is the setup guide that tells the user exactly what secrets and variables to add and how. 3. Tell the user to follow the setup guide in `.github/workflows/README.md`. That's it. Do **not** add anything to `infra/Taskfile.yaml` or create `infra/scripts/` for this path. **Required GitHub Secrets** (both required — no defaults): | Name | Kind | |------|------| | `DATAROBOT_API_TOKEN` | Secret | | `PULUMI_ACCESS_TOKEN` | Secret | **Optional GitHub Variable** (defaults to `ci` if not set): | Name | Kind | Default | |------|------|---------| | `PULUMI_STACK_CI_NAME` | Variable | `ci` | **When to use the advanced approach (GPG + DIY backends) instead:** - You have many secrets (GPG encrypts all of `.env` behind a single passphrase — only one GitHub Secret needed) - Your organization prohibits Pulumi Cloud and requires a self-managed backend (Azure Blob / S3 / GCS) - You need GitLab CI/CD The templates and scripts for all of these are in `scripts/` in this skill directory. If the skill has already been propagated to the project's `infra/` directory (common in downstream templates), look in `infra/scripts/` instead. See the [Implementation Pattern](#implementation-pattern) section below for full setup guidance. | Scenario | Key files in `scripts/` | |----------|------------------------| | Azure Blob / S3 / GCS Pulumi backend | `pulumi-setup.sh`, `taskfile-snippets.yaml` | | GitHub Actions + GPG secrets | `github-deploy.yml`, `github-cd.yml`, `encrypt-secrets.sh`, `setup-github-secrets.sh` | | GitLab CI/CD | `gitlab-ci.yml`, `setup-gitlab-variables.sh` | ### Adapting the deploy command The example workflows use `uv run pulumi up --yes` directly. Before copying them, check `infra/Taskfile.yaml` — the project may already wrap the deploy command in a task: ```bash cat infra/Taskfile.yaml # look for 'up-yes', 'deploy', or similar tasks ``` | What you find | What to use in CI | |---------------|-------------------| | `up-yes` task | `task up-yes` — non-interactive, purpose-built for CI; prefer this over raw Pulumi | | `deploy` task (alias for `up`) | Avoid — typically runs `pulumi up` interactively; only safe in CI if you confirm it passes `-y` internally | | No Taskfile or no relevant task | Keep `uv run pulumi up --yes` as-is | To use `task` in a workflow, add an install step and swap the run command: ```yaml - name: Install Task run: pip install go-task-bin - name: Deploy working-directory: infra env: DATAROBOT_API_TOKEN: ${{ secrets.DATAROBOT_API_TOKEN }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} run: | uv sync --all-extras task up-yes ``` ### DataRobot API token (service account) `DATAROBOT_API_TOKEN` should come from a **DataRobot service account** — a DataRobot user created for automation, not tied to anyone's personal login. This prevents CI/CD from breaking when the engineer who originally set it up leaves the team. To set one up: ask your DataRobot admin to create a dedicated user (e.g. `ci-bot@your-org.com`). Under that account, go to **Developer Tools → API Key** and generate a token. Store it as the `DATAROBOT_API_TOKEN` secret in GitHub. > **Note:** This is purely a DataRobot concept — it has no relation to Pulumi state management or backend configuration. "Service account" here just means a non-personal DataRobot user. ## Implementation Pattern When implementing CI/CD for an application template, follow this structure: **Project Structure:** ``` application-template-root/ ├── infra/ │ ├── README.md # ⚠️ GENERATE THIS — tailored to the chosen CI/CD platform and Pulumi backend │ ├── Taskfile.yaml # ⚠️ CI/CD tasks go HERE — copy from infra/scripts/taskfile-snippets.yaml │ └── scripts/ # Copy entire scripts/ directory here │ ├── README.md # Copy from scripts/infra-README.md │ ├── setup-github-secrets.sh │ ├── setup-gitlab-variables.sh │ ├── encrypt-secrets.sh │ ├── decrypt-secrets.sh │ ├── pulumi-setup.sh │ ├── gitlab-ci.yml │ ├── github-deploy.yml │ ├── github-cd.yml │ ├── github-destroy.yml │ └── taskfile-snippets.yaml ├── .env # User's secrets (never commit!) ├── .env.gpg # Encrypted secrets (commit for GitHub) ├── .gitlab-ci.yml # Copy from infra/scripts/gitlab-ci.yml ├── .github/ │ └── workflows/ │ ├── deploy.yml # Copy from infra/scripts/github-deploy.yml (PR review deploys) │ ├── cd.yml # Copy from infra/scripts/github-cd.yml (push-to-main CD) │ └── destroy.yml # Copy from infra/scripts/github-destroy.yml └── Taskfile.yml # Root Taskfile — ADD ONLY one `includes` entry (see below). DO NOT add tasks here. ``` **Key Points:** - **⚠️ ALWAYS generate `infra/README.md`** tailored to the chosen platform and backend — see "Generating infra/README.md" below - All CI/CD scripts go in `infra/scripts/` directory - **⚠️ CRITICAL: All CI/CD tasks go in `infra/Taskfile.yaml` — NEVER add CI/CD tasks directly to the root `Taskfile.yml`** - `.env` and `.env.gpg` stay in project root - Scripts in `infra/scripts/` reference `../../.env` (two levels up) - Root `Taskfile.yml` gets exactly ONE addition: an `includes` entry pointing to `./infra/Taskfile.yaml` - CI/CD configs (`.gitlab-ci.yml`, `.github/workflows/`) are copied to standard locations **Root Taskfile.yml — the only change needed:** ```yaml # Add this includes block to the existing root Taskfile.yml: includes: infra: taskfile: ./infra/Taskfile.yaml dir: infra # Tasks are then run as: task infra:encrypt-secrets, task infra:setup-github-secrets, etc. ``` ### Generating infra/README.md After determining the user's CI/CD platform (GitHub/GitLab) and Pulumi backend, **always create `infra/README.md`** with content tailored to their choices. It should cover: 1. **Architecture overview** — which platform was chosen and why, and which Pulumi backend 2. **First-time setup** — the exact sequence of `task infra:*` commands needed to bootstrap 3. **Day-to-day tasks** — a table or list of the `task infra:*` commands relevant to their platform 4. **How deployments work** — short description of each trigger: - GitHub: `deploy.yml` fires on PR open/sync (review stack), `cd.yml` fires on push to main (CI stack), `destroy.yml` is manual - GitLab: `review_app` is manual on MR, `deploy_ci` fires on push to default branch, `destroy_review_app` is manual 5. **Secrets / credentials** — what variables/secrets are needed and where they live (GitHub Secrets, GitLab CI/CD variables, `.env.gpg`) 6. **Stack migration note** — if backend was migrated from a local stack, document what was done so future contributors understand the history Adjust section titles, task names, and stack-naming strategy to match what was actually configured. The README should be accurate enough that a new contributor can set up CI/CD without referring to any other document. ## Workflow examples See [`references/workflow-examples.md`](references/workflow-examples.md) for step-by-step examples covering GitLab CI/CD, GitHub Actions with GPG secrets, and continuous delivery setup. ## Using Task for workflow management Application templates use [Task](https://taskfile.dev) to simplify local development and CI/CD workflows. Task provides a unified interface for Python and TypeScript/React components. ### Example Taskfile.yaml See [`references/example-taskfile.yaml`](references/example-taskfile.yaml) for a complete example. ### Using Task in CI/CD ```bash # Install Task pip install go-task-bin # Install dependencies task install # Run linters (with fixes) task lint # Run linters (check only) task lint-check # Run tests task test ``` ## GitLab CI/CD Configuration The complete pipeline configuration lives in `scripts/gitlab-ci.yml`. Copy it to your repository root: ```bash cp infra/scripts/gitlab-ci.yml .gitlab-ci.yml ``` Key pipeline jobs: - `lint` / `test` — run on every same-project MR - `review_app` — manual deploy per MR; stack name driven by the `PULUMI_STACK_REVIEW_NAME` CI/CD variable - `deploy_ci` — automatic deploy on merge to default branch; stack name driven by `PULUMI_STACK_CI_NAME` - `destroy_review_app` — manual cleanup of review stacks `PULUMI_STACK_REVIEW_NAME` and `PULUMI_STACK_CI_NAME` must be set as plain CI/CD variables in GitLab (Settings → CI/CD → Variables). The pipeline file includes sensible defaults that project-level variables override. ## GitHub Actions Configuration The complete workflow files live in `scripts/`: - `scripts/github-deploy.yml` → copy to `.github/workflows/deploy.yml` - `scripts/github-destroy.yml` → copy to `.github/workflows/destroy.yml` ```bash mkdir -p .github/workflows cp infra/scripts/github-deploy.yml .github/workflows/deploy.yml cp infra/scripts/github-destroy.yml .github/workflows/destroy.yml ``` The deploy workflow triggers on pull requests and derives `PULUMI_STACK_NAME` from the `PULUMI_STACK_REVIEW_NAME` Actions variable and the PR number. Set `PULUMI_STACK_REVIEW_NAME` and `PULUMI_STACK_CI_NAME` as repository **variables** (Settings → Secrets and variables → Actions → **Variables** tab), not secrets. ## Pulumi State Management ### Pulumi Cloud Backend (Recommended) The simplest approach for managing Pulumi state: ```bash # Install Pulumi curl -fsSL https://get.pulumi.com | sh # Login to Pulumi Cloud pulumi login # Create/select stack pulumi stack select --create dev # Deploy pulumi up ``` **CI/CD Setup**: Add `PULUMI_ACCESS_TOKEN` to your CI/CD secrets. Get token from [Pulumi Console](https://app.pulumi.com/account/tokens). ### DIY Backend Options For organizations that cannot use Pulumi Cloud: #### Azure Blob Storage ```bash # Login to Azure backend pulumi login azblob://container-name # Set Azure credentials export AZURE_STORAGE_ACCOUNT=myaccount export AZURE_STORAGE_KEY=mykey ``` #### AWS S3 ```bash # Login to S3 backend pulumi login s3://bucket-name # AWS credentials from environment export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=... ``` #### Google Cloud Storage ```bash # Login to GCS backend pulumi login gs://bucket-name # GCP credentials from environment export GOOGLE_CREDENTIALS=... ``` ### Migrating Stacks to a Different Backend When a developer has an existing local stack (a `Pulumi.<stackname>.yaml` file) that was created against a different backend than the CI/CD destination, the stack state must be exported and re-imported before switching. `pulumi-setup.sh` handles this automatically: it checks `pulumi whoami --verbose` for the **Backend URL** and compares it with the target URL. If they differ and local stack files exist, it offers to migrate them. **Manual migration steps** (if not using the script): ```bash # 1. Confirm current backend and stacks pulumi whoami --verbose # note "Backend URL:" pulumi stack ls -a # list stacks on current backend # 2. Export each stack that exists locally (Pulumi.<name>.yaml) pulumi stack export --stack <stackname> --file <stackname>-backup.json # 3. Login to the new backend (set any required credentials first) # Examples: pulumi login # Pulumi Cloud pulumi login azblob://my-container # Azure Blob pulumi login s3://my-bucket # AWS S3 # 4. Create the stack in the new backend and import state pulumi stack select --create <stackname> pulumi stack import --file <stackname>-backup.json # 5. Clean up the backup rm <stackname>-backup.json ``` **Key signals that migration is needed:** - `pulumi whoami --verbose` shows `Backend URL: file://` (local) but CI/CD uses cloud storage - Backend URL domain/scheme differs between developer machine and CI target ### Managing Stacks Across Environments ```bash # List all stacks pulumi stack ls -a # Output: # NAME LAST UPDATE RESOURCE COUNT # organization/project/prod 1 day ago 15 # organization/project/staging 2 days ago 12 # organization/project/dev 1 hour ago 10 # github-pr-repo-42 3 hours ago 13 # Select and update a stack pulumi stack select dev pulumi up # View stack outputs pulumi stack output --json # Delete a stack pulumi stack rm review-app-123 --yes ``` ## Secrets Management All credentials (DataRobot API token, Pulumi access token, LLM keys, cloud storage keys) are stored in `.env` and committed to the repository encrypted as `.env.gpg`. The only secret that needs to be configured in the CI/CD system directly is `CICD_SECRET_PASSPHRASE` (the GPG passphrase). Non-sensitive stack name variables (`PULUMI_STACK_CI_NAME`, `PULUMI_STACK_REVIEW_NAME`) are set as plain variables, not secrets. ### DataRobot API token (service account) `DATAROBOT_API_TOKEN` should come from a **DataRobot service account** — a DataRobot user created for automation, not tied to anyone's personal login. This prevents CI/CD from breaking when the engineer who originally set it up leaves the team. To set one up: ask your DataRobot admin to create a dedicated user (e.g. `ci-bot@your-org.com`). Under that account, go to **Developer Tools → API Key** and generate a token. Store it as the `DATAROBOT_API_TOKEN` secret in your CI/CD system. > **Note:** This is purely a DataRobot concept — it has no relation to Pulumi state management or backend configuration. "Service account" here just means a non-personal DataRobot user. ### GitHub Run `scripts/setup-github-secrets.sh` for interactive setup — it sets `CICD_SECRET_PASSPHRASE` as a repository secret and `PULUMI_STACK_CI_NAME` / `PULUMI_STACK_REVIEW_NAME` as repository variables. To encrypt `.env` for CI: ```bash task infra:encrypt-secrets # or: ./infra/scripts/encrypt-secrets.sh ``` Add the resulting `.env.gpg` to git. For local decryption: ```bash task infra:decrypt-secrets # or: ./infra/scripts/decrypt-secrets.sh ``` ### GitLab Run `scripts/setup-gitlab-variables.sh` for interactive setup — it sets: - `CICD_SECRET_PASSPHRASE` — masked, for decrypting `.env.gpg` - `GITLAB_API_TOKEN` — masked, for posting MR comments - `PULUMI_STACK_CI_NAME` / `PULUMI_STACK_REVIEW_NAME` — plain variables Alternatively configure in the UI: Project Settings → CI/CD → Variables. Mark `CICD_SECRET_PASSPHRASE` and `GITLAB_API_TOKEN` as **Masked** and **Protected**. ## Best practices ### CI/CD Pipeline Design 1. **Fast feedback**: Run linting and testing in parallel 2. **Manual gates**: Make review apps manual to save resources 3. **Automatic cleanup**: Provide easy ways to destroy test environments 4. **Stack isolation**: Use unique stack names per PR/MR 5. **Idempotent operations**: Design deployments to be safely re-runnable ### Pulumi State 1. **Use centralized backends**: Enable collaboration and CI/CD 2. **Stack naming conventions**: Use consistent patterns (e.g., `github-pr-{repo}-{number}`) 3. **Clean up stacks**: Remove unused stacks to reduce clutter 4. **State locking**: Backends handle this automatically 5. **Backup state**: Cloud backends provide automatic backups ### Security 1. **Never commit secrets**: Use .gitignore for .env files 2. **Encrypt sensitive data**: Use GPG for GitHub, CI/CD variables for GitLab 3. **Rotate credentials**: Regularly update API tokens and keys 4. **Scope permissions**: Use least-privilege access for service accounts 5. **Audit access**: Monitor who has access to secrets ### Resource Management 1. **Tag resources**: Use consistent tagging for tracking 2. **Set TTLs**: Consider time-to-live for review environments 3. **Monitor costs**: Track resource usage per environment 4. **Auto-cleanup**: Implement automatic deletion of old review apps 5. **Resource limits**: Set quotas to prevent runaway costs ## Troubleshooting ### Common Issues **Pulumi state conflicts:** - Ensure only one deployment runs at a time per stack - Use unique stack names for concurrent deployments - Check backend connection and credentials **Secret decryption failures:** - Verify GPG passphrase is correct - Check .env.gpg file is in repository - Ensure GPG is installed in CI environment **Deployment timeouts:** - Increase timeout values in workflow - Check DataRobot API connectivity - Verify resource provisioning isn't blocked **Stack not found:** - List stacks: `pulumi stack ls -a` - Verify backend connection - Check stack name matches pattern **Resource conflicts:** - Use unique names per stack - Check for orphaned resources - Review Pulumi state for inconsistencies ## Example Repositories Reference implementations: - **GitLab**: [demo-data-agent](https://gitlab.com/datarobot-oss/demo-data-agent) - Complete GitLab CI/CD setup - **GitHub**: [demo-talk-to-my-data-agent](https://github.com/datarobot-forks/demo-talk-to-my-data-agent) - Complete GitHub Actions setup ## Resources - [Task Documentation](https://taskfile.dev) - [Pulumi Documentation](https://www.pulumi.com/docs/) - [Pulumi State and Backends](https://www.pulumi.com/docs/iac/concepts/state-and-backends/) - [GitLab CI/CD](https://docs.gitlab.com/ci/) - [GitHub Actions](https://docs.github.com/actions) - [DataRobot Application Templates](https://docs.datarobot.com/en/docs/wb-apps/app-templates/index.html) - [DataRobot Codespaces](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html)
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.