RCE Endeavors 😅

Table of Contents:

• Introduction (1/11)
• Goals and Architecture (2/11)
• The inotify API (3/11)
• File Watcher, C++ (4/11)
• File Watcher, Bash (5/11)
• Execution Engine (6/11)
• Containerization (7/11)
• Kubernetes (8/11)
• Demo (9/11)
• Conclusion (10/11)
• System Setup (11/11)
• Source code

This post will cover how to add Kubernetes as an orchestration layer for the Dockerfiles that were created in the previous post. Right now, each language has its own dedicated container environment to run it. However, launching these Dockerfiles is a manual process, and these files do not automatically scale with user input. If we have a C++ compiler system running and provide it with a constant stream of input files, the system will eventually back up if the rate of compilation and execution is greater than the rate of input files being added. It would be nice to automatically launch Docker containers to handle this increased load, and alternatively it would be nice to automatically kill off extra Docker containers that are idle for long periods of time. This is where Kubernetes comes in to the picture.

I won’t go into much detail about Kubernetes; their interactive tutorials cover more ground than I ever could in a blog post. All that I will mention is that it is an exceptionally feature rich and complex platform for automating deployments, scaling, and general orchestration of containers. As part of this multi-language compiler system, the only features that we care about are autoscaling and mounting a partition to the host.

Deployment configuration

The Kubernetes deployments for each language is provided in a deployment.yaml file. This file provides a spec for the containers for CPU and memory limits, as well as volume information for where the shared mount volume. This shared mount volume is needed as an easy way to provide input source files to the running containers, as containers cannot have access to the host filesystem. Exposing mounted volumes to containers can be a security risk and was explicitly chosen to keep the system simple, as opposed to fully secure. A deployment.yaml file is shown below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-compiler-c
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      compiler: c
  template:
    metadata:
      labels:
        compiler: c
    spec:
      containers:
      - name: compiler-base-c
        image: localhost:32000/compiler-base-c:latest
        resources:
          requests:
            memory: "512Mi"
            cpu: "100m"
          limits:
            memory: "1024Mi"
            cpu: "1000m"
        volumeMounts:
        - mountPath: /home/user/code/share
          name: io-shared-volume
        lifecycle:
          preStop:
            exec:
              command: ['sh', '-c', './shutdown.sh']
      volumes:
      - name: io-shared-volume
        hostPath:
          path: /home/{{USER}}/Desktop/share
          type: Directory
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: test-compiler-c
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test-compiler-c
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 800Mi
  

This particular file is for the C compiler environment. The container compiler-base-c will be launched under this deployment configuration. This container will have the resources specified under the resources section and have a mounted volume that will be visible as /home/user/code/share on the container. This mounted volume will map to the host at path /home/{{USER}}/Desktop/share where {{USER}} will be replaced by the current username when the deployment script is run. When the container is set to shut down, it will invoke the “sh -c ./shutdown.sh” command to clean up.

The latter half of the deployment.yaml file specifies the autoscaling configurations. These containers will scale based on two resource limmits: CPU and memory. If the CPU utilization averages above 75% then Kubernetes will scale in another container pod. Likewise, if the average memory usage is above 800 MB then the same will happen. The minReplicas and maxReplicas specify the minimum and maximum number of container pods that may be active at any given time. Kubernetes will continue to scale the number of container pods up to a maximum of ten if the memory and/or CPU usage is consistently high. If pods become underutilized then they will be automatically taken out of service and killed until there is one pod remaining.

Deployment script

The script to perform a deployment is shown below. The script will perform a deployment for each supported language. There is some logic to substitute the {{USER}} field in the deployment file, but the rest is a straightforward call to the Kubernetes command-line tool kubectl to do the deployment.

#!/bin/bash

LANGUAGES="c cpp cs java py"

for LANGUAGE in $LANGUAGES; do
    CURRENT_USER=${USER}
    REPLACE_USER=`cat "Deployments/${LANGUAGE}/deployment.yaml" | sed "s/{{USER}}/$CURRENT_USER/g"`
    echo "Deploying containers for ${LANGUAGE}"
    echo "$REPLACE_USER" > Deployments/${LANGUAGE}/deployment.runtime.yaml
    sudo microk8s.kubectl apply -f Deployments/${LANGUAGE}/deployment.runtime.yaml
    echo "Deployed containers for ${LANGUAGE}"
done

This fully covers it for the design and implementation of the multi-language compiler system! As described throughout these posts, the system has support for multiple languages, interactive input, multi-threading, and a scalable production deployment in a containerized environment. The next post will show off a demo of the system in action.