JMeter in the Pipeline + On-Demand — How to Run Performance Testing Both Automatically and On Command

Why should performance testing be done both automatically and on demand?

The most common issue in Thai organizations is: "Performance testing only happens after a problem occurs." The system slows down, customers complain, and only then does the team start investigating—only to discover the issue came from code deployed the previous week.

Research across mid-sized and large enterprises shows that teams embedding performance testing into the pipeline can detect regressions 5–10 times faster than teams that test only when issues arise, while also reducing Mean Time to Recovery (MTTR) by more than 60%.

But that does not mean automated testing alone is enough. Some situations require on-demand testing, such as preparing systems before a major campaign or investigating root causes after an incident.

JMeter is a tool that supports both approaches exceptionally well—automated execution in the pipeline and scenario-based execution through a self-service portal.

Part 1: JMeter in the CI/CD Pipeline — Automated testing on every deployment

Why embed performance testing in the pipeline?

The idea behind Shift-Left Testing is to move testing as early as possible in the development process. Instead of waiting until staging or production, testing begins as soon as code is pushed to the repository.

The clearest advantage is regression detection—when a developer introduces code that increases response time by 200ms, the system can catch it immediately in the pull request rather than after customers start complaining.

Key benefits of embedding JMeter in the pipeline:

• Detect performance regressions immediately — no need to wait for manual testing
• Establish performance standards so every release is validated against the same criteria
• Reduce the cost of defects — fixing issues in development is 10–100 times less expensive than fixing them in production
• Build team confidence with every deployment

Integration patterns with major CI/CD platforms

JMeter can integrate with all major CI/CD platforms commonly used in the market. Each platform has different integration methods, with distinct strengths and trade-offs to consider:

GitHub Actions

GitHub Actions is currently one of the most widely adopted platforms. For JMeter-based performance testing, teams can use ready-made actions from the Marketplace or build custom workflows.

A key consideration is the use of self-hosted runners for resource-intensive testing, because GitHub-hosted runners have CPU and memory limitations that can make test results inconsistent.

GitLab CI/CD

GitLab has a strong advantage in artifact management—JMeter test results can be stored as artifacts and easily compared across pipelines. It also includes built-in capabilities for performance testing reports.

Jenkins

Jenkins offers the highest level of flexibility—it includes performance plugins that directly support JMeter and can display response time and throughput trends across hundreds of builds.

One important best practice is that JMeter should run on a Jenkins agent, not on the controller—this follows Jenkins official documentation and prevents load tests from affecting the Jenkins server itself.

Azure DevOps

Microsoft provides Azure Load Testing Service, built specifically to support JMeter. It allows teams to run JMeter test plans directly in the pipeline, with built-in threshold evaluation and automatic quality gates.

For organizations already using Azure, this is often the lowest-complexity option because there is no need to manage load generation infrastructure internally.

Performance Budget & Quality Gate — The criteria that determine whether a build passes or fails

The core of pipeline-based performance testing is the performance budget—predefined thresholds that automatically fail a build if test results exceed acceptable limits.

Below is a recommended minimum performance budget:

The critical point is to tailor thresholds to the context of each system—an API expected to respond within 200ms should not be judged by the same criteria as a report that takes 5 seconds to generate.

A strong quality gate should not look only at individual values; it should also evaluate change relative to the baseline. For example, if response time increases by 20% compared with the previous build, it should trigger a warning even if it still falls within the threshold.

Test selection strategy by level

Not every test needs to run every time. Choosing the right test for the right context is a critical discipline:

Every Pull Request (Smoke Test):

• Test only the critical path (login, core data views, primary transaction flow)
• Use a small number of virtual users (5–10 users)
• Keep runtime within 2–3 minutes
• Goal: detect whether new code breaks the system or significantly degrades performance

Nightly Build (Load Test):

• Test all major business flows
• Use a number of virtual users aligned with expected production traffic
• Run for 15–30 minutes
• Goal: verify that performance remains within defined thresholds

Weekly (Stress/Soak Test):

• Test maximum system capacity under peak load
• Use 2–5 times the normal number of virtual users
• Run for 1–2 hours
• Goal: identify the breaking point and uncover memory leaks

Result retention and cross-build comparison

Good test results must be stored and comparable—not just marked pass or fail, but tracked over time to show whether performance has improved or degraded over the past 30–90 days.

What should be retained:

• JMeter HTML Report — stored as an artifact for each build for historical review
• Summary Metrics — stored in structured form for trend analysis
• Raw Data — retained for deep analysis when issues occur
• Comparison Report — automatically generated against the previous build

Good data retention enables teams to answer questions such as "From which release did this API start becoming slower?" in just a few minutes.

Alerting when tests fail

Pipeline-based performance testing is useless if no one knows a test has failed. An effective alerting system should include:

• Slack / Microsoft Teams — immediate notifications to the engineering channel when the performance gate fails
• LINE Notify — for Thai teams that primarily use LINE
• Email Digest — daily summaries for managers and stakeholders
• Dashboard — real-time visibility into the status of all pipelines

Good alerts must provide enough context—not just "Test Failed," but for example: "Response Time P95 increased from 300ms to 850ms on endpoint /api/checkout starting from commit ABC123."

Artifact and dashboard management

JMeter generates highly detailed reports—HTML dashboards with response time graphs, throughput, error distribution, and much more.

Items that require planning:

• Retention Policy — how long should reports be kept? We recommend 90 days for summary data and 30 days for full reports
• Storage Strategy — use CI/CD artifact storage or move reports to object storage
• Access Control — who needs access to the reports? Developers, QA, managers, or stakeholders
• Dashboard Integration — consolidate JMeter metrics into the organization’s main dashboards

Part 2: JMeter On-Demand Testing — Testing based on the situation

When is on-demand testing necessary?

While the pipeline covers routine daily testing, some scenarios require purpose-specific testing outside the normal cycle:

Before a major release (Pre-Release)
When launching a new feature with significant architectural impact, testing needs to be more detailed than standard pipeline checks—including edge cases and failure scenarios not covered in nightly tests.

After an incident (Post-Incident)
When the system crashes or slows down enough to affect users, the team must be able to recreate the same conditions immediately to confirm that the implemented fix actually resolves the issue.

Capacity Planning
Before festivals or major campaigns such as 11.11, year-end sales, or back-to-school periods, the team needs to verify whether the system can handle traffic increases of 3–10 times normal volume.

Infrastructure Comparison (Vendor Comparison)
When migrating to a new cloud provider or upgrading servers, the organization must compare whether the new infrastructure genuinely performs better than the current one.

On-demand testing workflows by scenario

Pre-Launch Load Testing

Before launching a campaign or new feature, the workflow should be:

1. Define the traffic model — estimate concurrent users, usage patterns, and peak periods
2. Create realistic test scenarios — not just repeated requests to the same URL
3. Run a baseline test to measure current performance
4. Run the load test based on the projected traffic model
5. Run a stress test at 150–200% of projected traffic
6. Analyze results and tune the system — scale up or out if necessary
7. Run again to confirm that tuning improvements are effective

Post-Incident Root Cause Analysis

When an incident occurs, JMeter can support analysis with the following process:

1. Recreate the incident conditions — use real access logs as test input
2. Gradually increase load until the system begins to fail
3. Identify the bottleneck — determine which component is the constraint
4. Test the fix — apply the fix and rerun the same test
5. Turn the scenario into a regression test and add it to the pipeline

Capacity Planning for Seasonal Traffic

For businesses with seasonal traffic patterns:

1. Analyze historical data — review traffic patterns from the past 1–2 years
2. Create a traffic model that reflects real patterns, including ramp-up and ramp-down
3. Test step by step — normal load → expected peak → 2x peak → 3x peak
4. Define auto-scaling rules based on test results
5. Test auto-scaling to confirm it responds quickly enough in practice

Vendor/Infrastructure Comparison

When comparing infrastructure options:

1. Define a standard test suite to be used across all options
2. Run the same tests on every infrastructure being evaluated
3. Collect metrics in a consistent format — response time, throughput, error rate, cost
4. Create a comparison report showing the pros and cons of each option
5. Evaluate the cost-performance ratio, not performance alone