MDCMS GitLab Interface Manual

MDCMS-GitLab Interface Guide

Published March 18, 2022

Overview

The primary objective of the GitLab interface is to invoke pipeline jobs defined on GitLab servers to build, test, deploy, review and rollback artifacts on remote platforms from within an MDCMS RFP and to then capture the result and job details from those jobs. This way, MDCMS acts as a centralized deployment manager across all platforms for an organization.

Prerequisites

• MDCMS v8.4 or higher must be installed and licensed on at least one IBM i partition

• MDOpen v8.4 or higher must be licensed on the same IBM i partitions

• MDWorkflow Base + Pipeline must be licensed on the same IBM i partitions

• The user(s) responsible for configuring the interface must have MDOpen v8.4 or higher installed in a version of the Eclipse IDE, such as the standard, free Eclipse IDE from Eclipse.org or Rational Developer for i from IBM

• The user(s) responsible for configuring the interface in MDOpen must be authorized to MDSEC Code 5 (Attribute Maintenance) and MDSEC Code 10 (Server Location Maintenance).

• The user(s) responsible for configuring the interface in GitLab must have access to the GitLab Dashboard and sufficient knowledge and privileges in GitLab to:

  • create service users and add API tokens
  • add and update GitLab Projects

• The network firewall must allow bi-directional http(s) traffic between the MDCMS partitions and GitLab servers. The port to allow on the GitLab side is the port that the GitLab server listens to. The port to allow on the MDCMS side is either the port defined for the MD REST API server or the port defined for the http server that forwards requests to the MD REST API server.

Configure Connection to Pipeline Servers

Before MDCMS can communicate with a GitLab server, the location and credentials must be defined. The user that will configure this must have MDSEC authority to code md/10 – Server Location Maintenance.

Take the following steps to add a Pipeline Server connection:

1. Within MDOpen, expand the repository for a partition and then expand Settings

2. Left-click on Pipeline Servers. The Pipeline Servers view will open and list any already defined servers

3. Within the view, right-click and select option Add (or Copy if a similar item already exists)

4. A dialog will pop up with the following fields:

1. Once the field values have been entered, click the Save button

Test Connection to Pipeline Server

The connection can be tested by right-clicking on the row in the Pipeline Server view and selecting option Test Connection. The results of the test, and any interaction from MDCMS to the server, are available in the pipeline logs.

To view the Pipeline logs, right-click on a row in the Pipeline Server view and select option Logs. Then left-click on the log for the current date to view the contents.

The pipeline consumer logs are stored as IFS files (one per day per server) in folder /MDCMS/logs/<instance>/pipeline. A log file is deleted after n days, based on the retention settings in the MDCMS Log maintenance screen. The default is to retain these IFS files for 10 days.

Define Pipeline Webhooks

In order for MDCMS to know when a Pipeline is complete, a webhook from GitLab to MDCMS is required.

See the pre-requisites section at the beginning of this tutorial to ensure that the communication from GitLab to MDCMS will be feasible.

In order to create, update or delete the MDCMS webhook for a GitLab project, right-click on the Pipeline Server entry and select option Project Webhooks.

This brings up a view with pipeline event type, and information about the currently registered Webhook, if applicable.

Press the Create/Update Webhooks to create the webhook for the event or to update it if the MDCMS URL changes due to a new address of the partition.

If the webhook is no longer needed, then click the Delete Webhooks button.

In order to verify that the webhook is functioning, do the following:

1. go into the Project in GitLab

2. click on Settings->Webhooks

3. scroll to the bottom to see the list of recent deliveries

Configure *PIPE Attributes

There is a special MDCMS Object Type, *PIPE, that can be assigned to an attribute. Attributes of this type indicate to MDCMS that 0 or more Pipeline jobs will be invoked for object requests assigned to the attribute when an RFP runs for a target level.

The same attribute ID should be defined for each level in a migration path so that the object requests will migrate from level to level. Then, for each level, Attribute Pipelines will be defined if any pipeline jobs should be invoked for that level. If no pipeline execution is required for a specific level, MDCMS will simply migrate the request records in the RFP without performing activity on them.

If certain artifacts will require a different set of Pipeline jobs then other artifacts, then a separate attribute should be created for them.

The user that will configure an attribute must have MDSEC authority to code 5 – Attribute Maintenance for the target Appl/Lvl.

Take the following steps to add a *PIPE Attribute:

• Within MDOpen, expand the repository for a partition and then expand Settings

• Left-click on Attributes. The Attributes view will open and list any already defined attributes based on the filter values.

• Within the view, right-click and select option Add (or Copy if a similar item already exists)

• A dialog will pop up with many fields. Only the following fields are relevant for *PIPE attributes:

• Once the field values have been entered, click the OK button

Configure Attribute Pipelines

Once a Pipeline Server and a *PIPE attribute is defined, Attribute Pipelines can be defined to specify which Pipeline Jobs should run for which attribute at each promotion level.

The user that will configure an Attribute Pipeline must have MDSEC authority to code 5 – Attribute Maintenance for the target Appl/Lvl.

Take the following steps to add an Attribute Pipeline:

• Within MDOpen, expand the repository for a partition and then expand Settings

• Left-click on Attribute Pipelines. The Attribute Pipelines will open and list any already defined pipelines based on the filter values. Alternatively, right-click on a Pipeline Server and Select option Attribute Pipelines to list all Pipelines for that server or right-click on a *PIPE attribute and Select option Attribute Pipelines to list all Pipelines for that attribute.

• Within the view, right-click and select option Add (or Copy if a similar item already exists)

• A dialog will pop up with the following fields:

Once the field values have been entered, click the OK button

Attribute Pipeline Parameters

Left-Click on a row in the Attribute Pipelines view to edit the configuration of that Attribute Pipeline.

Besides the fields described in the Add dialog, there is also a listing of any parameters defined for the Attribute Pipeline.
For example:

Attribute Pipeline Parameters are passed as Parameter/Value pairs to the job when it is requested to be executed.
These coincide with parameters that are defined for the Pipeline job that are then used within the Pipeline script instead of hardcoded values.

There are 2 String Parameters that are always passed by MDCMS and MUST NOT be additionally defined within MDOpen in the list of parameters for the Attribute Pipeline. Those 2 parameters are:

• MD_TRANS

• MD_CALLBACK

Any other parameters that are required by your script should be defined in GitLab and in MDOpen.

To add a parameter in MDOpen, right-click within the Parameter List in the Attribute Pipeline editor and select option Add Attribute Pipeline Parameter. The following fields will appear:

Options to manage Parameters across Attribute Pipelines

Apply attribute pipeline parameter – when this option is selected, every Attribute Pipeline will be listed where the Parameter doesn’t exist or the Parameter exists, but with a different description or value. Check the description or value checkbox for each Attribute Pipeline that you wish to apply the values of the parameter to.

Delete attribute pipeline parameter – when this option is selected (multi-select is enabled for this option), the current Attribute Pipeline will be shown at the top of the parameter deletion list with the Delete checkbox checked by default. Below that entry will be every other Attribute Pipeline where the parameter(s) are defined and the option to delete them from those entries as well.

Configure MDCMS REST API Server

When an MDCMS RFP invokes a Pipeline job, it passes along as parameter MD_CALLBACK the MDCMS Webhook to invoke when the Pipeline job is complete. In order for the Webhook to be accepted by MDCMS, a HTTP server for MDCMS must be configured on the partition where the RFP runs. Only one server can be defined per instance of MDCMS per partition and it only needs to be configured once. If your instance of the MDCMS http server isn’t yet configured, perform the instructions in the next 2 sections.

Set/Change URL

Within a 5250 Emulator, take the following steps to get to the URL configuration screen:

1. Command MDCMS from a command line

2. Option 1=MDCMS Setup Menu

3. Option 10=Interface Settings

4. Option 9=MD REST API Server

5. Option 1=Set/Change URL

MDLRURL COMPANY NAME 7.04.19
SCRN1 Set MD REST API URL Endpoint 13:04:38

Current Endpoint: https://mysystem.mycompany.com

New Endpoint....: https://mysystem.mycompany.com___________________

Example API URL.: https://mysystem.mycompany.com/mdcms/applications

WARNING: Remember to update any webhook definitions when modifying a previously active URL Endpoint

Enter=Confirm F12=Cancel

The Endpoint is the URL for the partition where the MDCMS REST server is hosted. If connecting directly to the server, then include the port number in the URL (example: https://mysystem.mycompany.com:2121)

Otherwise, if the using a forward proxy from the default server, the port numbers can be excluded.

The services themselves use the context path of <Endpoint>/<mdcms instance>/<resource> and the instance name and resource are appended automatically, so do not include them in the definition of the Endpoint on this screen.

Generate MDCMS REST API Server

Within a 5250 Emulator, take the following steps to get to the REST API server generation screen:

1. Command MDCMS from a command line

2. Option 1=MDCMS Setup Menu

3. Option 10=Interface Settings

4. Option 9=MD REST API Server

5. Option 2=Generate Server

Server Name

Only one server is to be generated for an instance of MDCMS. By default, the name of the server is the same as the name of the instance (example = MDCMS or MDCMSTEST).

The http server can’t already exist when requesting to generate it.

Regardless of the name of the server, the context path on the URL is always the name of the mdcms instance in lower case and the server configuration is stored in folder /www/<mdcms instance>.

Port Number

The port number is required and should be set to a number not already used by something else.

Starting/Stopping the MDCMS REST API Server

The server can be started with command STRTCPSVR SERVER(*HTTP) HTTPSVR(MDCMS)
assuming MDCMS is the name of the server.

It’s recommended to configure the server to autostart or to add the command to the QSTRUPPGM instructions.

The server can be stopped with command ENDTCPSVR SERVER(*HTTP) HTTPSVR(MDCMS)
assuming MDCMS is the name of the server.

Testing/Troubleshooting the MDCMS REST API Server

The server can be tested from a browser using the example API URL that is displayed in the Set/Change URL screen described at the beginning of this chapter. If a timeout occurs, verify that the firewall allows the request through.

You can also check the http logs in /www/mdcms/logs.

If the request gets through but it doesn’t respond correctly, check log file LXERRLOG in library MDCMS.

If a GitLab callback fails, verify the MD_CALLBACK parameter is defined for the GitLab job and inspect the job’s Console Output for detailed information and to verify that the job’s script actually tries to invoke the Callback.

Defining Callback for Passthru Variables in GitLab Pipeline

If variable values need to be returned to MDCMS for use in downstream pipeline jobs, In order for MDCMS to know when a triggered GitLab Project plan is finished, the following can be defined:

1. Within GitLab, navigate to the Project

2. Click on CI/CD->Editor

3. Define an after_script, which runs after a job completes

4. The script statement should be similar to the following:

   "curl -X POST -H 'content-type: application/json' -data '{\md_passthru\:\$MD_TRANS\,\passthru_name_1\:\ARTIFACT_DIRECTORY\,\passthru_value_1\:\REPODIR\,\passthru_name_2\:\MY_PARM\,\passthru_value_2\:\something useful\}' $MD_CALLBACK"

In the above example, 2 passthru name/value pairs are sent to the MDCMS webhook for storage and potential use in later pipeline jobs. The $MD_TRANS and $MD_CALLBACK variables are always sent by MDCMS when the pipeline is triggered and identify what and how the MDCMS API should be invoked.

Requesting *PIPE Object Requests

Now that everything is configured, it’s time to start requesting items and having them built and deployed via GitLab.

The most common way for files and folders to get requested is through Continuous Integration triggered by a push to a Git or SVN repository. This process is defined in MDOpen and full instructions on that configuration is available in the Continuous Integration tutorials.

If the Continuous Integration definition is set to create *PIPE request records, then those records will be created automatically for each new/changed/deleted file if for diffs, or for all folders and files if for contents.

Each record will contain the Repository type (*GIT or *SVN), Repository ID defined in MDOpen and the revision hash for Git or the revision number for SVN.

If manually creating *PIPE Object Requests, you can manually add the repository information to the Object Request Detail record in the Repository fields near the bottom of the Object Request editor.

If the pipeline job requires information about the SCM repository in order to pull artifacts from it, you can pass that information dynamically as wildcards on Attribute Pipeline Parameters.

SCM Wildcards:

MDCMS does NOT physically store and migrate actual files for *PIPE object requests, keeping the MDCMS process very light and fast. Instead, it utilizes best practices to expect the pipeline jobs to retrieve the files from the SCM themselves when they need it.

If using Continuous Integration or Cross-Platform impact analysis, MDCMS will clone the SCM repository onto the IBM i partition to perform tree-walking or diff-walking, but those files don’t proceed further in the promotion process.

Other Object Request-Specific Wildcards that may be useful as parameters: