Persisting Custom Implementations on WSO2 Products in Kubernetes Deployments

Hi Mates,

In this story, Um going to discuss on the usage of the custom implementations like jar files, patches and any other format of a file and how to persist them.

There can be multiple occations where it is necessary to add sum custom files to the WSO2 APIM distribution. For an example let’s say there is a pre QA patch given in order to confirm the suggested fix.

As you may already know that the files that we are adding/copying to containers will not be persisted and will be lost when the container/pod rollout. Most of the time WSO2 APIM server restart is necessary in order to reflect the changes. hence it is necessary to persist the custom implementations.

There are several ways to persist the data in Kuberenetes.

1. Using Configmaps
2. Using Persistent Volumes
3. Downloading files from the internet at the point of server startup or container initialization.

Lets evaluate these options one by one seperately

1. Using Configmaps

“A ConfigMap is an API object used to store non-confidential data in key-value pairs. Pods can consume ConfigMaps as environment variables, command-line arguments, or as configuration files in a volume.”

Normally we use configmaps to mount the relevant configurations which are needed to done on the WSO2 docker images. You can easily identify the configs that are available in the WSO2 Kubernetes artifacts.

Configmaps also can be used to persist custom implementations in our deployments. But there is a limitation on the size of the configmap that we are trying to create. Because the data stored in a ConfigMap cannot exeed 1 MB. So we will not be able to create configmaps using files which are larger than 1MB.

Please follow the below steps to achieve this using a configmap. Lets there is a custom jar file which should be copied to the <WSO2_APIM_HOME>/repository/components/lib directory.

Step 1: Create the configmap using the below command. We need to execute the below command in a kubectl client terminal by selecting the relevant namespace.

kubectl create configmap apim-custom-jar-cm — from-file=<Location>/custom.jar -n <namespace>

<Location>: file location

<namespace>: namespace of the deployment.

If you are using the helm charts, Please create configmap inside the template folder with the below content.

apiVersion: v1
kind: ConfigMap
metadata:
  name: apim-custom-jar-cm
  namespace : <namespace>
binaryData:
  custom.jar: |-
    {{ .Files.Get "<Location>/custom.jar" | b64enc }}

Here please keep the original jar file outside of the template folder and refer it in the configmap yaml. Eg:

{{ .Files.Get “artifacts/am/repository/components/lib/custom.jar” | b64enc }}

Step 2: Create a volume using the created configmap. For this we need to create a new volume in the relevant deployment.yaml file.

volumes:
- name: apim-custom-jar-cm
  configMap:
    name: apim-custom-jar-cm

Step 3: Mount the created volume under the Volume Mounts in deployment.yaml

volumeMounts:
- name: apim-custom-jar-cm
  mountPath: /home/wso2carbon/wso2-config-volume/repository/components/lib/custom.jar
  subPath: custom.jar

Note: Here I have mounted the config map to a different location which is outside the product distribution. There is a logic inside the docker-entrypoint.sh file of the WSO2 docker images to copy all the file mounted inside this “/home/wso2carbon/wso2-config-volume” directory to the relevant locations. Please refer the below lines inside the docker-entrypoint.sh file.

# volume mounts
config_volume=${WORKING_DIRECTORY}/wso2-config-volume
...............
# copy any configuration changes mounted to config_volume
test -d ${config_volume} && [[ "$(ls -A ${config_volume})" ]] && cp -RL ${config_volume}/* ${WSO2_SERVER_HOME}/
# copy any artifact changes mounted to artifact_volume
test -d ${artifact_volume} && [[ "$(ls -A ${artifact_volume})" ]] && cp -RL ${artifact_volume}/* ${WSO2_SERVER_HOME}/

So all the files inside this wso2-config-volume directory will be copied to the relevant locations. But we need to create the file structure based on the relative path inside this directory. Eg: If we want to put a file inside the /home/wso2carbon/wso2am-3.1.0/repository/conf directory, we need to keep the file inside the /home/wso2carbon/wso2-config-volume/repository/conf directory.

Like wise there is a seperate directory for adding patches as well. That is the /home/wso2carbon/wso2-patch-volume directory. So we can add the necessary patches similarly.

Please refer the below lines inside the docker-entrypoint.sh file.

# volume mounts
patch_volume=${WORKING_DIRECTORY}/wso2-patch-volume
...............
# copy any patches mounted to patch_volume
if [[ ! `test -d ${WSO2_SERVER_HOME}/updates/wum` ]]
then
test -d ${patch_volume} && [[ "$(ls -A ${patch_volume})" ]] \
&& echo "Copying critical patch fixes to GA pack" \
&& cp -RL ${patch_volume}/* ${WSO2_SERVER_HOME}/
fi

There are several advantages of doing this in this manner.

• Owner of this file be the wso2carbon user instead of the root user. If we directly mount the file to the relevant location owner of the file will be the root.
• Sever will be able to modify the file dynamically.

2. Using Persistent Volumes

“A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes”

You may already have experianced with the persistent volumes created in the WSO2 Kuberenets artifacts. Here Um going to setup a EFS server as a PV in our deployment.

Lets create a PV which can be used to do the same task which we did using the configmap earlier.

Step 1: Create the wso2am-custom-volume.yaml and add the below configs.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: wso2am-custom-volume
  labels:
    purpose: wso2am-custom-volume
spec:
  capacity:
    storage: 
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: "testing.amazonaws.com"
    path: "/custom"

Step 2: Create the wso2am-custom-volume-claim.yaml as below.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wso2am-custom-volume-claim
  namespace : <namespace>
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10M
  storageClassName: ""
  selector:
    matchLabels:
      purpose: wso2am-custom-volume

Step 3: Create a volume using the created PVC. For this we need to create a new volume in the relevant deployment.yaml file.

volumes:
- name: wso2am-custom-volume-claim
  persistentVolumeClaim:
    claimName: wso2am-custom-volume-claim

Step 4: Mount the created volume under the Volume Mounts in deployment.yaml

volumeMounts:
- name: wso2am-custom-volume-claim
  mountPath: /home/wso2carbon/wso2-config-volume/repository/components/lib

Step 5: Provide the necessary permissions for the shared folder from the EFS side

• sudo chown nobody:nogroup /mount-directory
• sudo chmod 777 /mount-directory

Step 6: Copy the custom.jar file to the shared location and deploy the environment.

3. Downloading files from the internet at the point of server startup or container initialization.

There may be some occations where it is not possible to follow the above 2 options. So we can add the following 2 lines to the docker-entrypoint.sh file or to the wso2server.sh file which is inside the <WSO2_APIM_HOME>/bin directory. But please note that this jar/file will be downloadded at every container initialization and server startup.

Lets say we need to put a MS SQL driver to the lib folder. We can add the below 3 lines to the docker-entrypoint.sh or to the top of the wso2server.sh file.

wget --content-disposition "https://go.microsoft.com/fwlink/?linkid=2122433" -P /home/wso2carbon/wso2-config-volume/tempDir
unzip /home/wso2carbon/wso2-config-volume/tempDir/sqljdbc_8.2.2.0_enu.zip -d /home/wso2carbon/wso2-config-volume
cp /home/wso2carbon/wso2-config-volume/tempDir/sqljdbc_8.2\\enu/mssql-jdbc-8.2.2.jre8.jar /home/wso2carbon/wso2-config-volume/repository/components/lib

So this jar will be copied to the lib directory as expected.

There can be a lot of ways to achieve the same. We may not know until you try out and see. 😉

Congratulations!!!

Now You Know how to persist data in Kuberenetes 💪🎊🎉🥳

Hope this may help you at any point of your career.

“The Deeper You Dig, The Deeper You get”.“The Tired You Feel, The Success You Achieve”- Shiran