Monday, 17 July 2017

Jenkins, Bitbucket and Static Analysis

Static Analysis

So you are running static analysis on your builds, great. However who actually views this after the first few weeks? We get all excited about driving home the gains of automatic potential bug finding but who actaully goes and fixes all the issues, cannot say I've seen many sprints with a ticket to fix a particular issue across the whole code base. But there is a better way of presenting this data, why not put those comments on a code review in Bitbucket and have them reviewed along with the code. This way in with the review you can get feedback on what your tooling suggests. I think this is by far the best place to have the information, still have the full info on Jenkins or CI. This builds on the workflow that we have already in Jenkins and Bitbucket back in Jenkins pipeline + Bitbucket pull request, so if you haven't done that part yet please go back as I am building on that process.

 Process Requirements:

1.     User creates a pull request for his branch.
2.     Jenkins builds the pull request merged with the target branch.
3.     Static Analysis is done on the code during the build.
4.     Comments on the pull request are reported back to Bitbucket.

This is the basic requirements that we need to satisfy and we had some dependencies.

System requirements

Jenkins:

Jenkins (2.19.1)
Violation Comments to Bitbucket Server Plugin (1.51)

Starting in Jenkins 2.0 there are some security changes for parameters given via a URL and these parameters need to be whitelisted before Jenkins runs.

Special startup parameters:
To Whitelist the parameters you will need to add the parameters to the end of this system property.
“-Dhudson.plugins.git.GitStatus.safeParameters= PULL_REQUEST_TO_REPO_PROJECT_KEY
PULL_REQUEST_TO_REPO_SLUG”

For windows this can be done using the Jenkins.xml in the Jenkins home DIR. These need to be added before the -jar term as parameters after this are ignored.
Ubuntu:
File: /etc/default/Jenkins
Property: JAVA_ARGS
RedHat:
File: /etc/sysconfig/Jenkins
Property: JENKINS_JAVA_OPTIONS


General Jenkins Configuration


In the manage Jenkins GUI you can add to the default
Bitbucket Server Violations Server Defaults
    Username/password :  username and password to use for bitbucket
    Base URL: url of bitbucket in your organisation


Job Specific configuration


Add the following parameters to the build:

PULL_REQUEST_TO_REPO_PROJECT_KEY
PULL_REQUEST_TO_REPO_SLUG

For an example we can use pylint on python code in windows using this windows batch script:
mkdir pylint
for /R %%i in (*.py) do pylint -rn -f parseable "%%i" >> pylint\\pylint.log

In the post-build actions set the following:
Report violations to Bitbucket Server:

Project key: ${PULL_REQUEST_TO_REPO_PROJECT_KEY}
Repo_slug: ${PULL_REQUEST_TO_REPO_SLUG}
Pull Request ID: ${PULL_REQUEST_ID}

I set to comment one per violation as it's a little easier to see the comments on the line.
The file search for violations searches the absolute path, it's not relative to the workspace:
.*/*pylint\.log$

Now the way the plugin works is that the violations are uploaded from the node using the account in the job, I would use a service account for this, so it's easy to see that it has come from Jenkins. If you are using self-signed certificates, you will need to add the certificate to the Java keystore which the agent is using to connect.

Obtain the certificates that you need.
Use the keytool.exe in JAVA_HOME\bin\keytool.exe
Then use the keytool with commands similar to the following

"keytool -import -alias server_name -file cert.cer -keystore JAVA_HOME\lib\security\cacerts -storepass changeit -noprompt"


BitBucket Configuration


Building on the Jenkins git commit notify that we already have from Jenkins pull request building, we have a full build trigger of:

 (Jenkins_url)/git/notifyCommit?url=${PULL_REQUEST_TO_SSH_CLONE_URL}&branches=pr&sha1=${PULL_REQUEST_FROM_HASH}&PULL_REQUEST_URL=${PULL_REQUEST_URL}&PULL_REQUEST_ID=${PULL_REQUEST_ID}&PULL_REQUEST_TO_BRANCH=${PULL_REQUEST_TO_BRANCH}&PULL_REQUEST_TO_REPO_PROJECT_KEY=${PULL_REQUEST_TO_REPO_PROJECT_KEY}

I have only done this for a FreeStyle Job so far but there is a good example on the plugin page


Wednesday, 8 February 2017

Pandas wheel building

Pandas is a great library for data analysis but working with it on windows with a set version of numpy can be problematic. We deploy packages internally as wheels, and our code relies on a later version of numpy.

So what's the solution, build a wheel with the version of numpy that we use, this can be done if you have the python compiler for windows.

We use Python 2.7 for which you can find the installer here

virtualenv pandas
cd pandas\scripts
activate
cd ../..
pip install numpy==1.9.3+mkl --no-index --find-links=z:\PythonWheels
cd Source
python setup.py build_ext --inplace --force
python setup.py bdist_wheel

Voila, one wheel built with numpy==1.9.3+mkl, this can then be installed by pip on various environments.

Friday, 6 January 2017

Jenkins pipeline + Bitbucket pull request

Bitbucket and Jenkins pull requests

So we are using Bitbucket and Jenkins and moving from GitHub, GitHub and Jenkins have a great integration feature in the pull requests can be setup to build on Jenkins and return the build status to GitHub, this is what we would like to do with Bitbucket.

I found this which details pretty much what we were required to do, thanks goes to Christian Galsterer for doing this in the first place, but I came across several differences and we needed to extend the behaviour to working with pipeline builds as well.

Process Requirements:


1.     User creates a new branch (e.g. feature, bugfix).
2.     After completing his development work and pushing his changes to Bitbucket the user creates a pull request.
3.     In order to approve a pull request we require at least one successful Jenkins build. Thereby we would like to get not only the build result of the code checked in for the pull request but get the build status after the code has been merged with the target branch.
4.     When a pull request is created/updated Jenkins shall be triggered automatically for real continuous integration.
5.     The source of the pull request shall be automatically merged with the target branch.
6.     Set the build description with the pull request ID and a link back to the Bitbucket pull request.
7.     The build result shall be reported back to Bitbucket.
8.     Only if the build was successful and the number of successful builds configured in Bitbucket is reached the pull request can be approved and merged.

This is the basic requirements that we need to satisfy and we had some dependencies.

System requirements

Jenkins:

Jenkins (2.19.1)
With the following plugins:
Pre SCM build step (0.3)
Groovy plugin (1.29)
Groovy installed and on the system path.
Git Plugin(3.0.0)
Stash Notifier (1.11.4)
                This requires a username and password for bitbucket to update the pull request.
Pipeline plugin

Starting in Jenkins 2.0 there are some security changes for parameters, and these need to be white listed at the Jenkins Master.

Special startup parameters:
To make the parameters passable from the commit notify to the jobs the parameters need to be added to the start-up parameters of Jenkins this is to do with security changes in Jenkins.
“-Dhudson.plugins.git.GitStatus.safeParameters= PULL_REQUEST_URL, PULL_REQUEST_ID, PULL_REQUEST_FROM_HASH, PULL_REQUEST_TO_BRANCH”
For windows this can be done using the Jenkins.xml in the Jenkins home DIR. These need to be added before the -jar term as parameters after this are ignored.
Ubuntu:
File: /etc/default/Jenkins
Property: JAVA_ARGS
RedHat:
File: /etc/sysconfig/Jenkins
Property: JENKINS_JAVA_OPTIONS

BitBucket Plugins:


Configuration

Once installed the plugin needs to be configured for use. First we are going to do a straight commitnotify, they work by taking the GIT repo url and then matching it to a job in Jenkins that isa configured for polling (there does not need to be a poll interval it's just the url that it will match with)

BitBucket Configuration

Pull request notifier in bitbucket is done using the following:
1.       Install the Pull Request Notifier for Bitbucket add-on via the Universal Plugin Manager or manually by downloading from the Atlassian Marketplace.
2.       Administration > Manage Add-ons > Pull Request Notifier > Configure
3.       Select trigger only if there are no conflicts.
4.       Select the following triggers:
a.       OPENED
b.      REOPENED
c.       RESCOPED
d.      RESCOPED_FROM
e.      RESCOPED_TO
5.       Enter Jenkins URL
6.       Choose GET as the HTTP method
7.       Use the following URL:
a.       (Jenkins_url)/git/notifyCommit?url=${PULL_REQUEST_TO_SSH_CLONE_URL}&branches=pr&sha1=${PULL_REQUEST_FROM_HASH}&PULL_REQUEST_URL=${PULL_REQUEST_URL}&PULL_REQUEST_ID=${PULL_REQUEST_ID}& PULL_REQUEST_TO_BRANCH=${ PULL_REQUEST_TO_BRANCH}
8.       Save the trigger using save (note the view does not refresh)
The notifier plug in exposes certain MACROS
PULL_REQUEST_TO_SSH_CLONE_URL : this is the url of the GIT repo
PULL_REQUEST_TO_BRANCH: name of the pull request branch destination
PULL_REQUEST_FROM_HASH: SHA1 of the commit to merge into the branch
PULL_REQUEST_URL: Pull request URL in bitbucket
PULL_REQUEST_ID: pull request ID


General Jenkins Configuration


SCM
                Git needs to have a username and password in the global configuration so that it can finish some merge requests, this is required if the tip of master is updated and the GIT on the agent cannot fast forward to the tip of the branch to be merged.

Global Security settings:
Change the Markup Formatter to be HTML safe rather than plain text so that this can work.

Stash Notifier Plugin:
Jenkins > Configure System > Stash Notifier
Enter Root URL, Stash user and the stash password

If you use self-signed SSL certs you might need to set ignore SSL for testing, this issue should be solved before deploying live.


Job Specific configuration

Parameters

Parameterize the build with the following options:
PULL_REQUEST_URL : string
PULL_REQUEST_ID: string
PULL_REQUEST_TO_BRANCH: string

SCM

SCM GIT configuration, the GIT url must be accessed by SSH, this is important due to the commit notify is only done if the repository url is the same.
The branch specifier must be of the form: pr
This is so only pull requests are built by this job
Tick the option merge before build use the following options:
Name: origin
Branch: ${PULL_REQUEST_TO_BRANCH}
Merge: default
FF mode : --ff

Triggers

Polling must be enabled, no polling period needs to be set. This is due to the way in which the commitnotify has been done within the git plugin.

Pre-build step

“Run build step before SCM” system Groovy script to set the build description:

 def currentBuild = Thread.currentThread().executable  
 def PULL_REQUEST_URL = build.buildVariableResolver.resolve('PULL_REQUEST_URL')  
 def PULL_REQUEST_ID = build.buildVariableResolver.resolve('PULL_REQUEST_ID')  
 def description = "<a href='$PULL_REQUEST_URL'>PR #$PULL_REQUEST_ID</a>"  
 currentBuild.setDescription(description)  
  



Pipeline jobs

Pipeline job needs to be configured slightly differently as commit notify does not operate on these. Instead we use buildwithparameters to trigger the job in Jenkins in any case configuration is detailed below.

BitBucket


For the Pull request trigger the following needs to be configured:
Injection Url:
http://JENKINS_URL/crumbIssuer/api/xml?xpath=//crumb
Injection regexp:
<crumb>([^<]*)</crumb>
Basic authentication:
username and password
URL:
This needs to be a POST action
A header needs to be added:
    Header: Jenkins-Crumb
    Value: ${INJECTION_URL_VALUE}

This trigger will now trigger the job in Jenkins to be built with parameters that are required to merge the pull request in the target repository.
Below is an example of a script to merge the two and the notify stash.


 stage 'merge'  
 node {  
   def description = "<a href='$PULL_REQUEST_URL'>PR #$PULL_REQUEST_ID</a>"  
   currentBuild.setDescription(description)  
   sh 'git config --global user.name "Jenkins"'  
   sh 'git config --global user.email "jenkins@jenkins.com"'  
    checkout changelog: true, poll: true, scm:  
   [$class: 'GitSCM',  
     branches: [[name: PULL_REQUEST_FROM_HASH ]],  
     doGenerateSubmoduleConfigurations: false,  
     extensions: [[ $class: 'PreBuildMerge',  
             options: [mergeStrategy: 'MergeCommand.Strategy', fastForwardMode: 'NO_FF', mergeRemote: 'origin', mergeTarget: PULL_REQUEST_TO_BRANCH]]],  
             submoduleCfg: [],  
             userRemoteConfigs: [[credentialsId: 'jdengel', url: 'ssh://git@172.17.23.33:7999/pul/branch_test.git']]]  
    echo 'done'  
 }  
  stage 'notify'  
 node {  
   step([$class: 'StashNotifier'])     // Notifies the Stash Instance of an INPROGRESS build  
    try {  
     // Do stuff  
      currentBuild.result = 'SUCCESS'   // Set result of currentBuild !Important!  
   } catch(err) {  
     currentBuild.result = 'FAILED'   // Set result of currentBuild !Important!  
   }  
    step([$class: 'StashNotifier'])     // Notifies the Stash Instance of the build result  
 }  

Tuesday, 6 December 2016

Getting Credentials into Jenkins scripts

So you need a password or key to be hidden but used inside a Jenkins script.

The Credentials Binding Plugin is what you need to use. This can inject these as environmental variables in your Jenkins job scripts:


This way you can then use them later in a script to clone a repository without giving out your password to a Jenkins slave, like so:


git clone https://$SECRET@bitbucket/scm/project/repository.git

This works with files for ssh keys as well, so you can now script your password and keys into your scripts. A service account with access only to what it needs to would be the best way to support this and you have those access credentials as safe as your jenkins master is.,



Friday, 25 November 2016

Triggering bamboo, with only specific file change in source control

Another small issue while working on bamboo today, I wanted to only trigger a bamboo plan from a specific subset of a SVN or GIT repository, so we can only trigger full test builds when we change real code and not on documentation updates.

So you think this would be easy right ? Not exactly.

There is a menu option in Bamboo under the repositories menu as follows :



So input a small change like the above for change just on XML files. Change the files in the test repository and boom, nothing.

So it turns out after some digging that Bamboo only matches on the full file path, some examples are here


After knowing this it becomes a little easier to get right, but it's not obviously clear and as a user there is no trigger log immediately available it's not clear how to proceed, hopefully this clears things up.

Tuesday, 27 September 2016

Silent Install Pycharm with shortcut


So I was having an Issue where an installer was not creating a shortcut on the desktop, turns out it was a installer issue but due to the fact we SCCM to deploy we can run a simple powershell script to solve this issue.

Below is the script, that needs to be run as admin.


$InstallPath = "${env:ProgramFiles(x86)}\JetBrains\PyCharm Community Edition 5.0.4\"
$InstallPath
.\pycharm-community-5.0.4.exe /S /D=$InstallPath
# Create a Shortcut with Windows PowerShell
$TargetFile = "$InstallPath\bin\pycharm.exe"
$ShortcutFile = "$env:Public\Desktop\Pycharm.lnk"
$WScriptShell = New-Object -ComObject WScript.Shell
$Shortcut = $WScriptShell.CreateShortcut($ShortcutFile)
$Shortcut.TargetPath = $TargetFile
$Shortcut.Save()


UPDATE 10/07/2017:

I have found the real reason why this is required, when using silent install the installer will always default to non-admin and as such does not do the following:

-          No shortcut gets created in the Public\Desktop
-          No Start Menu entries are created
-          The software is not listed in the “Programs and Features” list in Control Panel
-          No registry key is added to the HKLM:\..\Uninstall branch
-          The uninstall registry key is created in the HKCU branch

this is tracked here:
IDEA-146215

Tuesday, 12 April 2016

SSH Keys and Windows, basic no passphrase

For most people GIT access is controlled via username and password, this is fine when working only a small project, when rarely doing push and pull from your remote repos online. However the next step is to use an SSH key.

This mean that you share some information between your PC and the remote server, these are called keys, for the uninitiated. Using the Git for Windows installer and OpenSSH you can generate and configure an SSH key for the connection.

Disclaimer - This is not the most secure method but is meant to be a stepping stone to passphrase based SSH with an agent.

To create an SSHKey look to use the ssh-keygen application that comes with GIT for windows.
The SSH-keygen application is normally stored in the C:\Program Files\Git\usr\bin\ and a key can be generated with the following command
"ssh-keygen" -t rsa -f bitbucket_rsa -N ""
Details:
-t type RSA
-f filename butbucket_rsa
-N passphrase

The command above creates a key pair in the current directory called bitbucket_rsa and bitbucket_rsa .pub with no passphrase. You will need to copy the bibucket_ras to the ~.ssh folder in your user directory.
To make the PC use the private key (bitbucket_rsa) when making the connection, you can use the config file in the ~/.ssh directory. This file is formatted in the following way.
host bitbucket.blah.com
IdentityFile ~/.ssh/bitbucket_rsa
Details
host then the URL of the server that you are making the connection to.
Identify file this is the location of the private key file.

To add a ppublic key into your bitbucket account you need to do the following:
  • Sign in to bitbucket
  • click the top right hand profile button
  • click the manage account button
  • click SSH keys
  • click on the add key button
  • copy and paste the contents of the bitbucket_rsa.pub into the text box and then click add key.
Your key is now on the server.

When you make the first connection over command line to the server, you will be asked if you accept the host key the server and a finger print will be shown much like below:



[bitbucket.com]:3456,[0.0.0.123]:9090 ssh-rsa RRRB3NzaC1yc2EAAAADAQABAAABAQCch1FyAnxrvKPw7AkhVyfR9FOJNVzIAUfgxlhHDR7rafYgmJkSwUCZtsYKRv6wNUEeRHh5EU/yYr/2zn0+5l2A/AG6X95SwrBBSzpeelFo2hoIjIZmnHv93k5VK0OXfpAfKqBRDMY5s1tVcdyKa6qCirHydyYbViM3Ob3mBhrhY/w87jXZqaMLQlNUbMfH6iGdaVBNeqgqSPeclgHge7kwSoIukMY0pqiQAinS0bb3aPzl4dwNCJBrghV1eYKGHL7/ESX+Mj8PrsJDzWv1IxG6Ey5jjcY+zw/386sKY2QbmB55m91lg50LfEZP/1OorbLBxDqThAW5maU7J27knGaR
Once verified this will be contained in a knownhosts file in the .ssh directory, the same location as the config file and the private key.

Now you should have a very simple SSH connection available.