CircleCI â Configuration & Optimization Guide
In this tutorial, you'll learn about CircleCI. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
CircleCI is a cloud-native CI/CD platform that uses a .circleci/config.yml file to define pipeline workflows, jobs, and steps with powerful Caching, parallelism, and orb ecosystems.
What You'll Learn
Why It Matters
CircleCI's parallelism and Caching features dramatically reduce CI run times compared to traditional single-node pipelines. Its orb marketplace provides pre-built configurations for common tools (AWS, Docker, Slack), eliminating boilerplate configuration. DodaTech reduced CI feedback time from 35 minutes to 9 minutes by leveraging CircleCI parallelism and dependency Caching across 30+ Microservices.
Real-World Use
DodaZIP's build pipeline runs on CircleCI with 8-way parallelism â unit tests split across containers, integration tests running in dedicated service containers, and build artifacts cached for deploy jobs. The pipeline completes in under 10 minutes for a codebase with 5,000+ tests.
flowchart LR
A[Git Push] --> B[.circleci/config.yml]
B --> C[Workflow: CI]
C --> D[Job: lint]
C --> E[Job: build]
C --> F[Job: test - 8x parallelism]
D --> G[Job: docker-build]
E --> F
F --> G
G --> H[Job: deploy-staging]
H --> I[Job: approval]
I --> J[Job: deploy-prod]
style B fill:#343434,color:#fff
Prerequisites: A GitHub or Bitbucket account. CircleCI account linked to your Repository.
Basic Configuration
# .circleci/config.yml
version: 2.1
orbs:
node: circleci/node@5.1.0
docker: circleci/docker@2.5.0
slack: circleci/slack@4.12.5
executors:
node-executor:
docker:
- image: cimg/node:22.0
resource_class: medium
commands:
install-deps:
steps:
- node/install-packages:
pkg-manager: npm
cache-version: v1
- run:
name: Check dependencies
command: npm audit --audit-level=high
jobs:
lint:
executor: node-executor
steps:
- checkout
- install-deps
- run:
name: Run linter
command: npm run lint
build:
executor: node-executor
steps:
- checkout
- install-deps
- run:
name: Build application
command: npm run build
- persist_to_workspace:
root: .
paths:
- dist/
- node_modules/
test:
executor: node-executor
parallelism: 8
steps:
- checkout
- attach_workspace:
at: .
- run:
name: Run tests with parallelism
command: |
circleci tests glob "tests/**/*.test.ts" | \
circleci tests split --split-by=timings > /tmp/test-files
npx jest --listTests --passWithNoTests | \
xargs npx jest --runInBand
- store_test_results:
path: test-results/
- store_artifacts:
path: test-results/
destination: test-output
docker-build:
executor: docker/docker
steps:
- checkout
- docker/check
- docker/build:
image: dodatech/user-service
tag: $CIRCLE_SHA1
- docker/push:
image: dodatech/user-service
tag: $CIRCLE_SHA1
deploy-staging:
executor: node-executor
steps:
- run:
name: Deploy to staging
command: |
kubectl set image deployment/user-service \
user-service=dodatech/user-service:$CIRCLE_SHA1 \
-n staging
deploy-production:
executor: node-executor
steps:
- run:
name: Deploy to production
command: |
kubectl set image deployment/user-service \
user-service=dodatech/user-service:$CIRCLE_SHA1 \
-n production
workflows:
version: 2
ci-pipeline:
jobs:
- lint
- build:
requires:
- lint
- test:
requires:
- build
- docker-build:
requires:
- test
filters:
branches:
only: main
- deploy-staging:
requires:
- docker-build
filters:
branches:
only: main
- hold-deploy-production:
type: approval
requires:
- deploy-staging
filters:
branches:
only: main
- deploy-production:
requires:
- hold-deploy-production
filters:
branches:
only: main
Parallelism and Test Splitting
jobs:
test:
docker:
- image: cimg/node:22.0
- image: postgres:16-alpine
environment:
POSTGRES_DB: myapp_test
POSTGRES_USER: myapp
POSTGRES_PASSWORD: testpass
- image: redis:7-alpine
parallelism: 4
environment:
DATABASE_URL: postgres://myapp:testpass@localhost:5432/myapp_test
REDIS_URL: redis://localhost:6379
steps:
- checkout
- run: npm ci
- run:
name: Split and run tests
command: |
circleci tests glob "tests/**/*.test.ts" | \
circleci tests split --split-by=timings --timings-type=filename > /tmp/tests-to-run
npx jest $(cat /tmp/tests-to-run) --ci --runInBand
- store_test_results:
path: test-results/
Caching Strategies
jobs:
install:
docker:
- image: cimg/node:22.0
steps:
- checkout
- restore_cache:
keys:
- npm-deps-{{ checksum "package-lock.json" }}
- npm-deps-
- run: npm ci
- save_cache:
key: npm-deps-{{ checksum "package-lock.json" }}
paths:
- ~/.npm
- node_modules/
- persist_to_workspace:
root: .
paths:
- node_modules/
Context-based Environment Management
workflows:
ci-pipeline:
jobs:
- deploy-staging:
context: stg-dodatech
filters:
branches:
only: main
- hold-deploy-production:
type: approval
requires:
- deploy-staging
- deploy-production:
context: prod-dodatech
requires:
- hold-deploy-production
Using Orbs
orbs:
aws-ecs: circleci/aws-ecs@3.2.0
aws-ecr: circleci/aws-ecr@8.2.0
jobs:
deploy-ecs:
executor: aws-ecs/default
steps:
- aws-ecr/build-and-push-image:
repo: ${AWS_ECR_REPO_NAME}
tag: ${CIRCLE_SHA1}
- aws-ecs/deploy-service-update:
family: user-service
cluster-name: production
container-image-name-updates: "container=${AWS_ECR_REPO_NAME},tag=${CIRCLE_SHA1}"
Common Configuration Mistakes
Not using workspaces to share artifacts between jobs: Each job runs in a fresh container. Use
persist_to_workspaceandattach_workspaceto pass build output between jobs without rebuilding.Ignoring parallelism for test jobs: Without
parallelism: N, all tests run on one container. Increasing parallelism splits tests across multiple containers, reducing total test time proportionally.Hardcoding resource class sizes: Over-provisioning resources wastes credits, under-provisioning slows builds. Use
resource_class: mediumfor linting,medium+for builds, andlargefor parallel test execution.Not using CircleCI contexts for secrets: Secrets defined in project settings are tedious to manage across multiple projects. Contexts provide named sets of environment variables shared across projects.
Missing cache key fallbacks: Without fallback keys (
restore_cache: keys: - npm-deps-{{ ... }} - npm-deps-), cache misses force full re-downloads. Fallbacks restore the nearest matching cache.
Practice Questions
What is the purpose of workspaces in CircleCI? Answer: Workspaces pass files between jobs in a workflow.
persist_to_workspacesaves files from one job,attach_workspacemakes them available in downstream jobs.How does CircleCI test splitting work? Answer:
circleci tests splitdivides test files across parallel containers based on file timings or name hashing, ensuring each container runs a balanced subset of the test suite.What are orbs and how do they help? Answer: Orbs are reusable configuration packages (like Docker or Slack) that provide pre-built jobs, commands, and executors. They eliminate boilerplate configuration.
What is the difference between contexts and project environment variables? Answer: Project env vars are scoped to a single project. Contexts are shared across projects and organizations, managed in CircleCI's organization settings.
Challenge
Create a full CircleCI configuration for a Node.js microservice that: uses the Node and Docker orbs, runs linting and security audit, builds the application, tests with 8-way parallelism using test splitting against PostgreSQL and Redis service containers, builds and pushes a Docker image, deploys to staging, requires manual approval for production, uses contexts for environment-specific secrets, and sends Slack notifications on pipeline status.
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro