{"id":2039,"date":"2024-07-03T20:15:11","date_gmt":"2024-07-03T20:15:11","guid":{"rendered":"https:\/\/www.w3computing.com\/articles\/?p=2039"},"modified":"2024-07-03T20:15:16","modified_gmt":"2024-07-03T20:15:16","slug":"use-kubernetes-with-apache-flink-for-real-time-data-processing","status":"publish","type":"post","link":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/","title":{"rendered":"Use Kubernetes with Apache Flink for Real-Time Data Processing"},"content":{"rendered":"\n

Introduction<\/h2>\n\n\n\n

Apache Flink is a powerful stream processing framework used for processing large volumes of data in real-time. Kubernetes, on the other hand, is a container orchestration platform that automates the deployment, scaling, and management of containerized applications. Combining Apache Flink with Kubernetes provides a robust solution for managing and scaling real-time data processing applications. This tutorial aims to provide an in-depth guide on how to deploy and manage Apache Flink applications on Kubernetes for real-time data processing. We assume that the reader has a basic understanding of both Apache Flink and Kubernetes.<\/p>\n\n\n\n

Prerequisites<\/h2>\n\n\n\n

Before we dive into the details, ensure you have the following prerequisites:<\/p>\n\n\n\n

    \n
  1. A working Kubernetes cluster (Minikube, GKE, EKS, or AKS can be used for development and testing).<\/li>\n\n\n\n
  2. kubectl command-line tool configured to interact with your Kubernetes cluster.<\/li>\n\n\n\n
  3. Apache Flink binaries or a Docker image of Apache Flink.<\/li>\n\n\n\n
  4. Basic understanding of Kubernetes concepts such as Pods, Services, Deployments, and ConfigMaps.<\/li>\n<\/ol>\n\n\n\n

    Setting Up Kubernetes Cluster<\/h2>\n\n\n\n

    If you do not have a Kubernetes cluster set up, you can use Minikube for local development and testing. Here is a quick guide to set up Minikube:<\/p>\n\n\n\n

    Install Minikube<\/h3>\n\n\n\n

    Follow the installation guide for your operating system from the official Minikube documentation<\/a>.<\/p>\n\n\n\n

    Start Minikube<\/h3>\n\n\n
    minikube start --cpus 4 --memory 8192<\/code><\/span>Code language:<\/span> Shell Session<\/span> (<\/span>shell<\/span>)<\/span><\/small><\/pre>\n\n\n
    This command starts a Minikube cluster with 4 CPUs and 8GB of memory, which should be sufficient for running Apache Flink.<\/p>\n\n\n\n
    Verify Minikube<\/h3>\n\n\n\n
    Ensure Minikube is running correctly:<\/p>\n\n\n
    kubectl get nodes<\/code><\/span>Code language:<\/span> Shell Session<\/span> (<\/span>shell<\/span>)<\/span><\/small><\/pre>\n\n\n
    You should see a node named minikube<\/code> in the Ready<\/code> state.<\/p>\n\n\n\n
    Deploying Apache Flink on Kubernetes<\/h2>\n\n\n\n
    Now that we have a Kubernetes cluster up and running, let’s deploy Apache Flink. We will use a Docker image of Apache Flink for this purpose.<\/p>\n\n\n\n
    Create a Namespace<\/h3>\n\n\n\n
    First, create a namespace for Flink to keep the resources isolated:<\/p>\n\n\n
    kubectl<\/span> create<\/span> namespace<\/span> flink<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
    Flink Docker Image<\/h3>\n\n\n\n
    We will use the official Apache Flink Docker image. You can also build your own custom image if needed. The official Flink Docker images are available on Docker Hub.<\/p>\n\n\n\n
    Deploy Flink JobManager<\/h3>\n\n\n\n
    The JobManager is the central coordinator of a Flink cluster, responsible for scheduling tasks, managing checkpoints, and handling job lifecycle events. We will create a Deployment and a Service for the JobManager.<\/p>\n\n\n\n
    JobManager Deployment<\/h4>\n\n\n\n
    Create a file named jobmanager-deployment.yaml<\/code> with the following content:<\/p>\n\n\n
    apiVersion:<\/span> apps\/v1<\/span>\nkind:<\/span> Deployment<\/span>\nmetadata:<\/span>\n  name:<\/span> flink-jobmanager<\/span>\n  namespace:<\/span> flink<\/span>\nspec:<\/span>\n  replicas:<\/span> 1<\/span>\n  selector:<\/span>\n    matchLabels:<\/span>\n      app:<\/span> flink<\/span>\n      component:<\/span> jobmanager<\/span>\n  template:<\/span>\n    metadata:<\/span>\n      labels:<\/span>\n        app:<\/span> flink<\/span>\n        component:<\/span> jobmanager<\/span>\n    spec:<\/span>\n      containers:<\/span>\n        -<\/span> name:<\/span> jobmanager<\/span>\n          image:<\/span> flink:latest<\/span>\n          args:<\/span> [\"jobmanager\"]<\/span>\n          ports:<\/span>\n            -<\/span> containerPort:<\/span> 6123<\/span>\n              name:<\/span> rpc<\/span>\n            -<\/span> containerPort:<\/span> 8081<\/span>\n              name:<\/span> web<\/span>\n          env:<\/span>\n            -<\/span> name:<\/span> JOB_MANAGER_RPC_ADDRESS<\/span>\n              value:<\/span> flink-jobmanager<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
    JobManager Service<\/h4>\n\n\n\n
    Create a file named jobmanager-service.yaml<\/code> with the following content:<\/p>\n\n\n
    apiVersion:<\/span> v1<\/span>\nkind:<\/span> Service<\/span>\nmetadata:<\/span>\n  name:<\/span> flink-jobmanager<\/span>\n  namespace:<\/span> flink<\/span>\nspec:<\/span>\n  ports:<\/span>\n    -<\/span> port:<\/span> 6123<\/span>\n      name:<\/span> rpc<\/span>\n    -<\/span> port:<\/span> 8081<\/span>\n      name:<\/span> web<\/span>\n  selector:<\/span>\n    app:<\/span> flink<\/span>\n    component:<\/span> jobmanager<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
    Deploy the JobManager resources:<\/p>\n\n\n
    kubectl apply -f jobmanager-deployment.yaml\nkubectl apply -f jobmanager-service.yaml<\/code><\/span>Code language:<\/span> Shell Session<\/span> (<\/span>shell<\/span>)<\/span><\/small><\/pre>\n\n\n
    Deploy Flink TaskManagers<\/h3>\n\n\n\n
    TaskManagers are the worker nodes in a Flink cluster. They execute the tasks assigned by the JobManager.<\/p>\n\n\n\n
    TaskManager Deployment<\/h4>\n\n\n\n
    Create a file named taskmanager-deployment.yaml<\/code> with the following content:<\/p>\n\n\n
    apiVersion:<\/span> apps\/v1<\/span>\nkind:<\/span> Deployment<\/span>\nmetadata:<\/span>\n  name:<\/span> flink-taskmanager<\/span>\n  namespace:<\/span> flink<\/span>\nspec:<\/span>\n  replicas:<\/span> 2<\/span>\n  selector:<\/span>\n    matchLabels:<\/span>\n      app:<\/span> flink<\/span>\n      component:<\/span> taskmanager<\/span>\n  template:<\/span>\n    metadata:<\/span>\n      labels:<\/span>\n        app:<\/span> flink<\/span>\n        component:<\/span> taskmanager<\/span>\n    spec:<\/span>\n      containers:<\/span>\n        -<\/span> name:<\/span> taskmanager<\/span>\n          image:<\/span> flink:latest<\/span>\n          args:<\/span> [\"taskmanager\"]<\/span>\n          ports:<\/span>\n            -<\/span> containerPort:<\/span> 6121<\/span>\n              name:<\/span> data<\/span>\n            -<\/span> containerPort:<\/span> 6122<\/span>\n              name:<\/span> rpc<\/span>\n          env:<\/span>\n            -<\/span> name:<\/span> JOB_MANAGER_RPC_ADDRESS<\/span>\n              value:<\/span> flink-jobmanager<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
    Deploy the TaskManager resources:<\/p>\n\n\n
    kubectl apply -f taskmanager-deployment.yaml<\/code><\/span>Code language:<\/span> Shell Session<\/span> (<\/span>shell<\/span>)<\/span><\/small><\/pre>\n\n\n
    Verify the Deployment<\/h3>\n\n\n\n
    Check the status of the deployments:<\/p>\n\n\n
    kubectl get deployments -n flink<\/code><\/span>Code language:<\/span> Shell Session<\/span> (<\/span>shell<\/span>)<\/span><\/small><\/pre>\n\n\n
    You should see both the flink-jobmanager<\/code> and flink-taskmanager<\/code> deployments in the Available<\/code> state.<\/p>\n\n\n\n
    Check the status of the pods:<\/p>\n\n\n
    kubectl get pods -n flink<\/code><\/span>Code language:<\/span> Shell Session<\/span> (<\/span>shell<\/span>)<\/span><\/small><\/pre>\n\n\n
    You should see one flink-jobmanager<\/code> pod and two flink-taskmanager<\/code> pods running.<\/p>\n\n\n\n
    Accessing Flink Web Dashboard<\/h2>\n\n\n\n
    To access the Flink web dashboard, you need to expose the JobManager service. You can use a port-forward for local development:<\/p>\n\n\n
    kubectl port-forward svc\/flink-jobmanager 8081:8081 -n flink<\/code><\/span>Code language:<\/span> Shell Session<\/span> (<\/span>shell<\/span>)<\/span><\/small><\/pre>\n\n\n
    Now, you can access the Flink web dashboard at http:\/\/localhost:8081<\/code>.<\/p>\n\n\n\n
    Running a Flink Job on Kubernetes<\/h2>\n\n\n\n
    With the Flink cluster up and running, let’s run a Flink job. For demonstration purposes, we will use a sample Flink job that performs word count on a stream of text data.<\/p>\n\n\n\n
    Sample Flink Job<\/h3>\n\n\n\n
    Save the following Flink job code in a file named WordCount.java<\/code>:<\/p>\n\n\n
    import<\/span> org.apache.flink.api.common.functions.FlatMapFunction;\nimport<\/span> org.apache.flink.api.java.tuple.Tuple2;\nimport<\/span> org.apache.flink.streaming.api.datastream.DataStream;\nimport<\/span> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;\nimport<\/span> org.apache.flink.util.Collector;\n\npublic<\/span> class<\/span> WordCount<\/span> <\/span>{\n    public<\/span> static<\/span> void<\/span> main<\/span>(String[] args)<\/span> throws<\/span> Exception <\/span>{\n        \/\/ set up the execution environment<\/span>\n        final<\/span> StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();\n\n        \/\/ create a DataStream from socket text stream<\/span>\n        DataStream<String> text = env.socketTextStream(\"localhost\"<\/span>, 9999<\/span>);\n\n        \/\/ parse the data, group it, window it, and aggregate the counts<\/span>\n        DataStream<Tuple2<String, Integer>> counts = text\n                .flatMap(new<\/span> Tokenizer())\n                .keyBy(value -> value.f0)\n                .sum(1<\/span>);\n\n        \/\/ print the results<\/span>\n        counts.print();\n\n        \/\/ execute program<\/span>\n        env.execute(\"Streaming WordCount\"<\/span>);\n    }\n\n    \/\/ Tokenizer function to split text into words<\/span>\n    public<\/span> static<\/span> final<\/span> class<\/span> Tokenizer<\/span> implements<\/span> FlatMapFunction<\/span><String<\/span>, Tuple2<\/span><String<\/span>, Integer<\/span>>> <\/span>{\n        @Override<\/span>\n        public<\/span> void<\/span> flatMap<\/span>(String value, Collector<Tuple2<String, Integer>> out)<\/span> <\/span>{\n            for<\/span> (String word : value.split(\"\\\\s\"<\/span>)) {\n                if<\/span> (word.length() > 0<\/span>) {\n                    out.collect(new<\/span> Tuple2<>(word, 1<\/span>));\n                }\n            }\n        }\n    }\n}<\/code><\/span>Code language:<\/span> Java<\/span> (<\/span>java<\/span>)<\/span><\/small><\/pre>\n\n\n
    Build and Package the Flink Job<\/h3>\n\n\n\n
    Compile and package the Flink job into a JAR file using Maven or any other build tool. Ensure you have Maven installed, then run the following command in the directory containing the pom.xml<\/code> file:<\/p>\n\n\n
    mvn clean package<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
    This command will generate a JAR file in the target<\/code> directory.<\/p>\n\n\n\n
    Submit the Flink Job<\/h3>\n\n\n\n
    Submit the Flink job to the Flink cluster running on Kubernetes using the Flink web dashboard or the Flink CLI.<\/p>\n\n\n\n
    Using Flink Web Dashboard<\/h4>\n\n\n\n
      \n
    1. Open the Flink web dashboard at http:\/\/localhost:8081<\/code>.<\/li>\n\n\n\n
    2. Navigate to the “Submit new job” page.<\/li>\n\n\n\n
    3. Upload the JAR file and submit the job.<\/li>\n<\/ol>\n\n\n\n
      Using Flink CLI<\/h4>\n\n\n\n
      You can also use the Flink CLI to submit the job. First, copy the JAR file to the JobManager pod:<\/p>\n\n\n
      kubectl cp target\/your-flink-job.jar flink\/flink-jobmanager:\/your-flink-job.jar<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Then, submit the job using the Flink CLI:<\/p>\n\n\n
      kubectl exec<\/span> -it $(kubectl get pods -n flink -l app=flink,component=jobmanager -o jsonpath='{.items[0].metadata.name}'<\/span>) -n flink -- \/bin\/sh\nflink run \/your-flink-job.jar<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Replace \/your-flink-job.jar<\/code> with the path to your JAR file in the JobManager pod.<\/p>\n\n\n\n
      Verifying the Job<\/h3>\n\n\n\n
      Once the job is submitted, you can verify its execution through the Flink web dashboard. You should see the job listed under “Running Jobs” with details about its execution.<\/p>\n\n\n\n
      Sending Data to the Flink Job<\/h3>\n\n\n\n
      The sample Flink job expects a text stream from a socket. To send data to the Flink job, you can use nc<\/code> (netcat) to create a socket server:<\/p>\n\n\n
      nc -lk 9999<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Now, type some text into the terminal, and you<\/p>\n\n\n\n
      should see the word counts appearing in the Flink web dashboard.<\/p>\n\n\n\n
      Scaling Flink Cluster on Kubernetes<\/h2>\n\n\n\n
      One of the key benefits of running Apache Flink on Kubernetes is the ease of scaling the cluster. You can scale the number of TaskManager replicas to increase the processing capacity.<\/p>\n\n\n\n
      Scaling TaskManagers<\/h3>\n\n\n\n
      To scale the TaskManagers, update the replicas<\/code> field in the taskmanager-deployment.yaml<\/code> file and apply the changes:<\/p>\n\n\n
      spec:<\/span>\n  replicas:<\/span> 4<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Apply the changes:<\/p>\n\n\n
      kubectl apply -f taskmanager-deployment.yaml<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Verify the scaling operation:<\/p>\n\n\n
      kubectl get pods -n flink<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      You should see four flink-taskmanager<\/code> pods running.<\/p>\n\n\n\n
      Auto-scaling with Kubernetes<\/h3>\n\n\n\n
      Kubernetes supports Horizontal Pod Autoscaling (HPA) based on CPU utilization or custom metrics. To enable auto-scaling for Flink TaskManagers, you need to set up metrics and create an HPA resource.<\/p>\n\n\n\n
      Metrics Server<\/h4>\n\n\n\n
      First, install the Kubernetes Metrics Server:<\/p>\n\n\n
      kubectl apply -f https:\/\/github.com\/kubernetes-sigs\/metrics-server\/releases\/latest\/download\/components.yaml<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Verify the Metrics Server:<\/p>\n\n\n
      kubectl get deployment metrics-server -n kube-system<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Create HPA for TaskManagers<\/h4>\n\n\n\n
      Create a file named taskmanager-hpa.yaml<\/code> with the following content:<\/p>\n\n\n
      apiVersion:<\/span> autoscaling\/v1<\/span>\nkind:<\/span> HorizontalPodAutoscaler<\/span>\nmetadata:<\/span>\n  name:<\/span> flink-taskmanager-hpa<\/span>\n  namespace:<\/span> flink<\/span>\nspec:<\/span>\n  scaleTargetRef:<\/span>\n    apiVersion:<\/span> apps\/v1<\/span>\n    kind:<\/span> Deployment<\/span>\n    name:<\/span> flink-taskmanager<\/span>\n  minReplicas:<\/span> 2<\/span>\n  maxReplicas:<\/span> 10<\/span>\n  targetCPUUtilizationPercentage:<\/span> 50<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Apply the HPA resource:<\/p>\n\n\n
      kubectl apply -f taskmanager-hpa.yaml<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Verify the HPA:<\/p>\n\n\n
      kubectl get hpa -n flink<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      The HPA will automatically scale the number of TaskManager replicas based on the CPU utilization.<\/p>\n\n\n\n
      Advanced Configuration and Monitoring<\/h2>\n\n\n\n
      Configuring Flink Properties<\/h3>\n\n\n\n
      You can configure Flink properties using ConfigMaps in Kubernetes. Create a ConfigMap with the desired Flink configuration and mount it as a volume in the JobManager and TaskManager pods.<\/p>\n\n\n\n
      Create a file named flink-configmap.yaml<\/code> with the following content:<\/p>\n\n\n
      apiVersion:<\/span> v1<\/span>\nkind:<\/span> ConfigMap<\/span>\nmetadata:<\/span>\n  name:<\/span> flink-config<\/span>\n  namespace:<\/span> flink<\/span>\ndata:<\/span>\n  flink-conf.yaml:<\/span> |\n    jobmanager.rpc.address: flink-jobmanager\n    taskmanager.numberOfTaskSlots: 2\n    state.backend: filesystem\n    state.checkpoints.dir: s3:\/\/your-bucket\/checkpoints\/\n    state.savepoints.dir: s3:\/\/your-bucket\/savepoints\/<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Apply the ConfigMap:<\/p>\n\n\n
      kubectl<\/span> apply<\/span> -f<\/span> flink-configmap.yaml<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Update the JobManager and TaskManager deployments to mount the ConfigMap:<\/p>\n\n\n
      spec:<\/span>\n  containers:<\/span>\n    -<\/span> name:<\/span> jobmanager<\/span>\n      image:<\/span> flink:latest<\/span>\n      volumeMounts:<\/span>\n        -<\/span> name:<\/span> flink-config-volume<\/span>\n          mountPath:<\/span> \/opt\/flink\/conf<\/span>\n  volumes:<\/span>\n    -<\/span> name:<\/span> flink-config-volume<\/span>\n      configMap:<\/span>\n        name:<\/span> flink-config<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Apply the updated deployments:<\/p>\n\n\n
      kubectl apply -f jobmanager-deployment.yaml\nkubectl apply -f taskmanager-deployment.yaml<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Monitoring Flink with Prometheus and Grafana<\/h3>\n\n\n\n
      Monitoring is essential for managing a Flink cluster in production. You can use Prometheus and Grafana for monitoring Flink metrics.<\/p>\n\n\n\n
      Deploy Prometheus and Grafana<\/h4>\n\n\n\n
      You can deploy Prometheus and Grafana using Helm charts:<\/p>\n\n\n
      helm repo add prometheus-community https:\/\/prometheus-community.github.io\/helm-charts\nhelm repo add grafana https:\/\/grafana.github.io\/helm-charts\nhelm repo update\n\n# Deploy Prometheus<\/span>\nhelm install prometheus prometheus-community\/prometheus\n\n# Deploy Grafana<\/span>\nhelm install grafana grafana\/grafana<\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
      Configure Flink for Prometheus<\/h4>\n\n\n\n
      Update the Flink configuration to enable Prometheus metrics:<\/p>\n\n\n
      data:<\/span>\n  flink-conf.yaml:<\/span> |\n    metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter\n    metrics.reporter.prom.port: 9250<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Apply the ConfigMap and restart the JobManager and TaskManager pods.<\/p>\n\n\n\n
      Create Prometheus Scrape Config<\/h4>\n\n\n\n
      Add a scrape configuration for Flink in Prometheus:<\/p>\n\n\n
      scrape_configs:<\/span>\n  -<\/span> job_name:<\/span> 'flink'<\/span>\n    static_configs:<\/span>\n      -<\/span> targets:<\/span> ['<jobmanager-pod-ip>:9250',<\/span> '<taskmanager-pod-ip>:9250'<\/span>]<\/span><\/code><\/span>Code language:<\/span> YAML<\/span> (<\/span>yaml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Replace <jobmanager-pod-ip><\/code> and <taskmanager-pod-ip><\/code> with the actual IP addresses of the JobManager and TaskManager pods.<\/p>\n\n\n\n
      Import Grafana Dashboard<\/h4>\n\n\n\n
      Import a Flink dashboard into Grafana to visualize the metrics. You can find pre-built dashboards on the Grafana website or create your own.<\/p>\n\n\n\n
      Conclusion<\/h2>\n\n\n\n
      In this tutorial, we have covered the complete process of deploying Apache Flink on Kubernetes for real-time data processing. We discussed setting up a Kubernetes cluster, deploying Flink components, running a Flink job, scaling the cluster, configuring Flink properties, and monitoring the Flink cluster using Prometheus and Grafana. By following this guide, you should be able to leverage the power of Kubernetes to manage and scale your Apache Flink applications effectively.<\/p>\n","protected":false},"excerpt":{"rendered":"
      Introduction Apache Flink is a powerful stream processing framework used for processing large volumes of data in real-time. Kubernetes, on the other hand, is a container orchestration platform that automates the deployment, scaling, and management of containerized applications. Combining Apache Flink with Kubernetes provides a robust solution for managing and scaling real-time data processing applications. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[21],"tags":[],"class_list":["post-2039","post","type-post","status-publish","format-standard","category-containers","entry"],"yoast_head":"\nUse Kubernetes with Apache Flink for Real-Time Data Processing<\/title>\n<meta name=\"description\" content=\"Apache Flink is a powerful stream processing framework used for processing large volumes of data in real-time.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Use Kubernetes with Apache Flink for Real-Time Data Processing\" \/>\n<meta property=\"og:description\" content=\"Apache Flink is a powerful stream processing framework used for processing large volumes of data in real-time.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-07-03T20:15:11+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-07-03T20:15:16+00:00\" \/>\n<meta name=\"author\" content=\"w3compadmin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"w3compadmin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"TechArticle\",\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/use-kubernetes-with-apache-flink-for-real-time-data-processing\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/use-kubernetes-with-apache-flink-for-real-time-data-processing\\\/\"},\"author\":{\"name\":\"w3compadmin\",\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/#\\\/schema\\\/person\\\/a550b3e20d78bb4f79b7c6b7b53f0561\"},\"headline\":\"Use Kubernetes with Apache Flink for Real-Time Data Processing\",\"datePublished\":\"2024-07-03T20:15:11+00:00\",\"dateModified\":\"2024-07-03T20:15:16+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/use-kubernetes-with-apache-flink-for-real-time-data-processing\\\/\"},\"wordCount\":1154,\"articleSection\":[\"Containers\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/use-kubernetes-with-apache-flink-for-real-time-data-processing\\\/\",\"url\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/use-kubernetes-with-apache-flink-for-real-time-data-processing\\\/\",\"name\":\"Use Kubernetes with Apache Flink for Real-Time Data Processing\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/#website\"},\"datePublished\":\"2024-07-03T20:15:11+00:00\",\"dateModified\":\"2024-07-03T20:15:16+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/#\\\/schema\\\/person\\\/a550b3e20d78bb4f79b7c6b7b53f0561\"},\"description\":\"Apache Flink is a powerful stream processing framework used for processing large volumes of data in real-time.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/use-kubernetes-with-apache-flink-for-real-time-data-processing\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/use-kubernetes-with-apache-flink-for-real-time-data-processing\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/use-kubernetes-with-apache-flink-for-real-time-data-processing\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Articles Home\",\"item\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Containers\",\"item\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/containers\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Use Kubernetes with Apache Flink for Real-Time Data Processing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/#website\",\"url\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/\",\"name\":\"Developer Articles Hub\",\"description\":\"\",\"alternateName\":\"Developer Articles\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/#\\\/schema\\\/person\\\/a550b3e20d78bb4f79b7c6b7b53f0561\",\"name\":\"w3compadmin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/wp-content\\\/litespeed\\\/avatar\\\/bd481d404e42caa2763662a3bfe825f8.jpg?ver=1779536135\",\"url\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/wp-content\\\/litespeed\\\/avatar\\\/bd481d404e42caa2763662a3bfe825f8.jpg?ver=1779536135\",\"contentUrl\":\"https:\\\/\\\/www.w3computing.com\\\/articles\\\/wp-content\\\/litespeed\\\/avatar\\\/bd481d404e42caa2763662a3bfe825f8.jpg?ver=1779536135\",\"caption\":\"w3compadmin\"},\"sameAs\":[\"http:\\\/\\\/w3computing.com\\\/articles\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Use Kubernetes with Apache Flink for Real-Time Data Processing","description":"Apache Flink is a powerful stream processing framework used for processing large volumes of data in real-time.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/","og_locale":"en_US","og_type":"article","og_title":"Use Kubernetes with Apache Flink for Real-Time Data Processing","og_description":"Apache Flink is a powerful stream processing framework used for processing large volumes of data in real-time.","og_url":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/","article_published_time":"2024-07-03T20:15:11+00:00","article_modified_time":"2024-07-03T20:15:16+00:00","author":"w3compadmin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"w3compadmin","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"TechArticle","@id":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/#article","isPartOf":{"@id":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/"},"author":{"name":"w3compadmin","@id":"https:\/\/www.w3computing.com\/articles\/#\/schema\/person\/a550b3e20d78bb4f79b7c6b7b53f0561"},"headline":"Use Kubernetes with Apache Flink for Real-Time Data Processing","datePublished":"2024-07-03T20:15:11+00:00","dateModified":"2024-07-03T20:15:16+00:00","mainEntityOfPage":{"@id":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/"},"wordCount":1154,"articleSection":["Containers"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/","url":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/","name":"Use Kubernetes with Apache Flink for Real-Time Data Processing","isPartOf":{"@id":"https:\/\/www.w3computing.com\/articles\/#website"},"datePublished":"2024-07-03T20:15:11+00:00","dateModified":"2024-07-03T20:15:16+00:00","author":{"@id":"https:\/\/www.w3computing.com\/articles\/#\/schema\/person\/a550b3e20d78bb4f79b7c6b7b53f0561"},"description":"Apache Flink is a powerful stream processing framework used for processing large volumes of data in real-time.","breadcrumb":{"@id":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.w3computing.com\/articles\/use-kubernetes-with-apache-flink-for-real-time-data-processing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Articles Home","item":"https:\/\/www.w3computing.com\/articles\/"},{"@type":"ListItem","position":2,"name":"Containers","item":"https:\/\/www.w3computing.com\/articles\/containers\/"},{"@type":"ListItem","position":3,"name":"Use Kubernetes with Apache Flink for Real-Time Data Processing"}]},{"@type":"WebSite","@id":"https:\/\/www.w3computing.com\/articles\/#website","url":"https:\/\/www.w3computing.com\/articles\/","name":"Developer Articles Hub","description":"","alternateName":"Developer Articles","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.w3computing.com\/articles\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.w3computing.com\/articles\/#\/schema\/person\/a550b3e20d78bb4f79b7c6b7b53f0561","name":"w3compadmin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.w3computing.com\/articles\/wp-content\/litespeed\/avatar\/bd481d404e42caa2763662a3bfe825f8.jpg?ver=1779536135","url":"https:\/\/www.w3computing.com\/articles\/wp-content\/litespeed\/avatar\/bd481d404e42caa2763662a3bfe825f8.jpg?ver=1779536135","contentUrl":"https:\/\/www.w3computing.com\/articles\/wp-content\/litespeed\/avatar\/bd481d404e42caa2763662a3bfe825f8.jpg?ver=1779536135","caption":"w3compadmin"},"sameAs":["http:\/\/w3computing.com\/articles"]}]}},"featured_image_src":null,"featured_image_src_square":null,"author_info":{"display_name":"w3compadmin","author_link":"https:\/\/www.w3computing.com\/articles\/author\/w3compadmin\/"},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/posts\/2039","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/comments?post=2039"}],"version-history":[{"count":4,"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/posts\/2039\/revisions"}],"predecessor-version":[{"id":2043,"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/posts\/2039\/revisions\/2043"}],"wp:attachment":[{"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/media?parent=2039"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/categories?post=2039"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.w3computing.com\/articles\/wp-json\/wp\/v2\/tags?post=2039"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}