CI CD Continuous Performance Test Framework with LoadRunner/Jenkins/Bitbucket

Framework Requirements

Followings are the hardware/software were used to setup the framework. 

(Updated 24 Aug 2020 - See appendix for automated email alerts using office 365 email address)

Tool/Application Name

Version

Windows laptop

10 (64 bit)

Java Development Kit

8

LoadRunner Full Setup

12.62

Jenkins for Windows

2.235.3

Git

2.28.0

Bitbucket Web Repository

N/A

Jenkins Plugin – Microfocus Application Automation Tool

6.3

Jenkins Plugin – Git

4.3.0

Jenkins Plugin – Bitbucket

1.1.16

Assumptions

Following assumptions are applicable.

-          - It is assumed that LoadRunner setup is already existing and working

-          - Existing test repository available in Bitbucket/Git

-          - Sample LoadRunner scripts/scenarios are available

-          - LoadRunner scenario must have an SLA (e.g. response time, hits/sec, user load, throughput etc.) defined

-          - The performance report works only in IE if opened from Jenkins. From the results folder, all reports can be opened in any browser.  

Framework Architecture


1          Installation and Configuration

 

1.1       Install JDK

·       This is required in the machine where Jenkins will be installed

·       Go to path https://www.oracle.com/technetwork/java/javase/downloads/index.html

·       Depending on version requirement (java 8 was used for the framework), click Download JDK link

·       From the next page, download the windows 64 bit jdk (.exe) file

Follow the on screen instructions and install from the JDK file

1.2       Install Git

·       Git should be installed on the LoadRunner Controller machine

·       Download and install 64 bit Git for windows from the official site https://git-scm.com/download/win (Take prior approval before installing)

1.3     Install Jenkins

·       Download Jenkins (LTS version) for windows from the official site: https://www.jenkins.io/download/

·       Follow the on screen instructions and install from the downloaded file (Get prior approvals before downloading/installing)

Note – Jenkins can be setup in the same machine as LoadRunner Controller. If setup in another machine, firewall should be opened so that Jenkins can talk to LR (LoadRunner) machine.

·       When installation is completed, check that you can see the login screen by going to URL: http://localhost:8080/

Note – Don’t install any plugins suggested by default.


1.4       Install Plugins

·       Login to Jenkins and go to Manage Jenkins->Manage Plugins

·       Search for “Micro Focus Automation” and select the checkbox against the plugin and choose download and install after restart.

·       After installation and restart is done, again repeat the above to install some additional plugins i.e. Git and Bitbucket

1.5       Jenkins Configuration

Followings are some key settings to be used for Jenkins:

1)      Jenkins -> Configure System:

o   Set Jenkins URL as <Jenkin machine’s full host name>:<port number>

 

Sample screenshot attached for ref.



2)      Jenkins -> Global Tool Configuration:

o   Set Git installations as Name: Git and Path: C:\Program Files\Git\bin\git.exe

Sample screenshot attached:

 

3)      Jenkins -> Configure Global Security:

o   If Jenkins and LoadRunner are on the same machine, skip this step.

o   If different, then LoadRunner machine must be able to talk to Jenkins server using a port defined in this settings.

In Configure Global Security, under agents, select TCP port for incoming agents as Fixed and set any available port.

 

              Sample screenshot attached below:


                

1.6      Setup Nodes

This step is required if Jenkins and LoadRunner machines are different. A node is basically a machine where you want to trigger the test. For our purpose, this is the LR Controller machine. Note that the Controller can invoke lots of Load Generators to run the test but those aren’t Nodes in this case.

Note – By default the local machine where Jenkins is installed is treated as the “master”.

To setup a node, go to Manage Jenkins -> Manage Nodes and Clouds -> New Node and then enter any name and select permanent agent. Enter the details similar to the below:

Change name as required, specify a root directory where the agent machine will store the test results (ensure that path exists on the Node machine), enter a label using which you can identify the Node. 


After adding all details, click SAVE. Now in the main screen, the newly added Node should be listed in red colour.

Click the node and in the next screen, it should give you options how to enable the Node. Click “Launch” to download a JNLP java file. The file basically contains some jar commands to connect the Agent with the Master.


Copy the JNLP file to the remote Node machine. Next step is to run the JNLP file but before that the agent port (8282 in above screenshot under Configure Global Security) needs to be added to the Inbound Rules in the Node machine using which it can listen to the master.

Once the inbound rule is updated, run the JNLP file and it should be now connected to the Jenkins server. The Node will then appear as Online in the Jenkins.


1.7      Verify Repository

·       Ensure you have the access to the Bitbucket repository which the dev team uses as source code repository.

·       Login to https://bitbucket.org using your credentials (check with release manager for access) and get the repository details.

1.8       Jenkins Project Setup

After all the above pre-requisites are done, it’s time to setup your first CI CD project. The project will be a Freestyle project where some key settings are to be setup. After defining all the below settings, you can save the project and then it will automatically run the test.

A snapshot of the settings is also attached.

·       General Settings:

o   If you have multiple nodes/agents and you want to use specific machines to run the test, then in General settings, specify the label for the Node (Controller machine) you want to use. If Jenkins and Controller are on same machine then skip this.

·       Source Code Management:

o   Select Git repository and add the repo details. Click the Add button to setup the login details. Once login details are added, you can select it from the credentials dropdown.

o   Depending on which branch you want to check for new changes to trigger the test, enter the details */<branch name> e.g. */master, */dev, */QA etc. If you want to scan all branches in the repo, leave it blank.

o   If you want to check only certain files or paths for changes, then select additional behaviours as “Polling ignores commits in certain paths” and set the “Included Region”. E.g. if you mention test.txt then under the repo, Jenkins looks for change in file test.txt. If change is found in the file then performance test is triggered.

·       Build Triggers:

o   Select Poll SCM and specify the schedule/frequency at which Jenkins will scan the repo for changes. E.g. if you mention */15 * * * * this means, every 15 minutes, Jenkins will search for changes.

 

·       Build:

o   If Jenkins finds changes, then as per Build settings the text execution will be done.

o   Select “Execute Micro Focus tests from file system”

o   Enter the path in Node machine where the LoadRunner scenarios are available. The .lrs (LoadRunner scenario) file present in the path will be used for the test execution

o   Enter the path in the Controller machine/Node where you want to store the test result files

o   In LoadRunner settings, you can specify various times. If you want to see runtime summary, then select vuser states, error count, transactions. Every time a test is running, you can see this summary in Jenkins console output.

·       Post build action:

o   In this setting, you should define how you want to handle the results/reports.

                        For performance test, select Always archive and publish reports (LR only)

        


2          Execution

The execution will be triggered by a change to the configured repository. To make some changes, login to the repo and open the required branch/file that is configured. Make any change and commit.

Note – In reality, developers may use Git tool to push/commit change to the repo. For test, we can directly edit a file in Bitbucket using a browser.



    In the polling log, check that after the configured frequency, Jenkins will find changes and will automatically create a new build that runs the performance test.


Click the new build just created and from Console Output you can see the runtime summary


When the test is completed, you can see the below reports/stats from the project’s home page. A trend comparing all past results is displayed. The trend includes the key transactions for which you have defined the SLA while building the Controller scenario. Any transaction which is within the SLA is marked as pass and anything above is marked in red as fail.

            In the below screenshot, 3 transactions had target SLA of avg response time as 3 seconds. Of the 3                         transactions, you can see how many met the 3 seconds avg response time in each build.

            

If you click the Project Performance Report, you can see the actual SLA values of the transactions across each build.

            

                If you want to see more detailed results of a build, click the build and you can find the Performance                     Report, Transaction Summary and Rich Report

            

          

Performance Report looks as below:


    

NOTE#1 – In Performance Report, there are various links to other graphs which are not working from Jenkins. These work fine if the report is opened directly from the results directory.

NOTE#2 - Transaction Summary currently doesn’t open from Jenkins and Rich report is not generated even if you have the default template defined in Analysis in the Controller machine.

After each test, all the result files are copied from Controller machine to the Jenkins machine. To view the detailed reports for each test, you can go to the Jenkins installation folder and find the reports for each build as shown below.




3  Known Issues/Limitations

Some known issues are:

-          Reports may not open in Jenkins. Workaround is to access the reports directly from Jenkins installation path as shown above

-          Sometimes the test execution may hang as Jenkins is not be able to close the Controller

-          This framework can’t run with ALM (as Performance Center license is required for that)

This framework will work only where Git repository is used for version control

Comments

Post a Comment

Popular posts from this blog

LoadRunner integration with HP QC/ALM

Hi, If your organization is using LoadRunner for performance testing and also using ALM/QC for test management then you can leverage ALM/QC as a repository for performance testing. It cannot replace Performance Center; however, features like storing scripts, execution, store results, access reports etc. can be done with some restrictions. See below steps to integrate LR with ALM/QC  and let me know if any queries. 1) Install ALM/QC desktop client. If you are using only IE client of ALM and ALM has timeout enabled then your test results won't update from Controller to ALM. 2) Open Vugen and from version control, connect to ALM project by connecting to your ALM URL (https://XXXX:<port>/qcbin/) 3) Open your script from local drive and do a Save As and select ALM/QC. This will save the script in to Test Plan folder 4) In ALM->test plan, create a new test and choose type as LR-SCENARIO. This will create a load runner scenario. 5) To design the scenario, select th...
Read more

Android native app scripting with LoadRunner

May 4 2019 There are a lot of ways to generate vugen script for your native android app.  I was using an app which had ssl encryption and used json requests and I followed the below to create script in Vugen. THIS DOESN'T NEED TO ROOT YOUR DEVICE. 1) Install app on a real android device 2) Install HTTPCanary app from playstore 3) Follow the onscreen steps to install the app and install its certificate and allow all the required access 4) From settings define the filter to select the app you want to capture packets 5) Start the recording and leave it to run. This will capture all packets from the test app even when minimized 6) Open the test app and perform transactions. 7) Open HTTPCanary and stop the recording. This will generate multiple captures for the test app. These will be sorted as per time so that helps you to understand the sequence. Each capture contains one request and the corresponding response. 8) Open each capture which will show you the request/response/...
Read more