Distance Calculator

Dell EMC Streaming Data Analytics - Hands-on

Read Post

This post demos how to make a distance calculator on Streaming Data Platform


  • Local deployment of InfluxDB+Grafana+Pravega+IoT GW
  • Optional: Connect Raspberry Pi 4 and run the app to collect the data in json format via a distance sensor
  • Using local IDE (Or local Flink cluster) to run Flink Distance Calculation Job
    • Fixed window

      Calculate the average value of each fixed time window (3s)

    • Distance Calculation logical

      Normal: <=1m A little Far: > 1m && <= 3m Far: > 3m

Demo environment

Demo env

Workload Flow

  1. We fully utilize SDP platform from start to end
  2. Collect Distance sensor data via a MQTT Sensor or Use simulator to generate the data from one CSV file and write into one MQTT topic
  3. Which further writes it into the Pravega Stream via Pravega Writer API
  4. Flink Job read the data from stream using Pravega Reader API
  5. Computing and processing in each fixed 3 time window
  6. Sink to Influx DB
  7. Grafana can Visualize the Output with custom dashboard Workflow


Setup distance calculator demo using Helm

  • The Helm charts for distance calculator demo is in
    cd sdp-starter-kit/distance-calculator/charts/
  • Get all available runtime images using the below command.
    kubectl get runtimeimages -A


    Update flinkImage variable in calculator/values.yaml with the runtime image name available for the flink version 1.15.2.

  • Get spec.values.image.tag using below command.
    kubectl get projectfeature pravegamqttbroker --output="jsonpath={.spec.templates.pravegaMQTTBroker}"


    Update chartVersion variable in features/values.yaml with spec.values.image.tag.

  • Deploy distance-calculator and mqtt-writer by running below script.
    ./deploy.sh <name_of_the_application>

    ex:- ./deploy.sh distance-calculator

    Wait for few minutes until flink application starts

  • To visualize dashboard navigate to AnalyticsYour ProjectFeatures
  • Open Metrics endpoint setup11
  • Hover on Dashboard and Click on Manage setup12
  • Click on Distance Calculataor Dashboard setup17
  • Below view can be observed setup14

    Note:- In case if you don’t see any graph or button to Zoom to Data, Apply the time range of 10:37:00 to 11:00:00 for the current date setup18

Setup distance calculator demo manually

1. Compile distance-calculator project
  • Use below command to package and create jar file
      cd distance-calculator
      mvn clean package
2. Setup SDP
  • Navigate to Analytics
  • Click on New Project
  • Select Artifact Repository, Metrics, Pravega MQTT Broker, Zookeeper from the Features dropdown setup1
  • Click on Save button and wait for the project to deploy
  • Navigate to AnalyticsYour ProjectFeatures
  • IP in the “Endpoint” section corresponding to “Pravega MQTT Broker” with port 8883 forms MQTT_BROKER_URL. ex: setup2
  • Click on pravega-mqtt-broker in the “Secrets” section to get MQTT_USERNAME and MQTT_PASSWORD setup3

  • Upload Jar file to Artifacts
  • Navigate to AnalyticsYour ProjectArtifactsFiles setup5
  • Upload the jar file from –> //distance-calculator/calculator/target/calculator-1.0.0.jar
  • Navigate to AnalyticsYour ProjectFlinkCreate Flink Cluster setup6
  • Select Flink version 1.15.2 and Click Save to create Flink Cluster setup7
  • Click File
  • Select Main Application from dropdown
  • Type main class as io.pravega.flinkapp.DistanceCalculator
  • Select Flink Version – 1.15.2
  • Click Create and wait for the app to Start setup8
5. Set the Environment variables
  • Note: - Edit Distance.csv and replace with current dates (To avoid retention policy error– DATA is only valid for 7 days) Below Image shows replacing of dates to Today’s date setup4
  • In Windows
      set MQTT_BROKER_URL=tls://<MQTT_BROKER_URL>:8883
      set MQTT_ALLOW_INSECURE=true
      set MQTT_DATA_FILE=C:\\<Project_Path>\\mqtt-writer\\Distance.csv
      set MQTT_USE_AUTH=true
  • In Linux/Ubuntu
      export MQTT_BROKER_URL=tls://<MQTT_BROKER_URL>:8883
      export MQTT_ALLOW_INSECURE=true
      export MQTT_DATA_FILE=/<Project_Path>/mqtt-writer/Distance.csv
      export MQTT_USE_AUTH=true
6. Simulate the MQTT writer
  • Use below command to run MQTT writer simulator
      java -jar mqtt-writer\target\mqtt-writer-1.0.0.jar
7. Get Influxdb Data Source
  • Navigate to AnalyticsYour ProjectFeatures setup9
  • Go to Setting → Data Source and Get Influxdb Data Source name and copy data_souce_name (disatanceflink_flnk in below image) setup10
  • Edit dashboard.json and replace datasource value from test_flnk to data_souce_name
8. Adding Dashboard in Grafana
  • Navigate to AnalyticsYour ProjectFeatures
  • Open Metrics endpoint setup11
  • Hover on Dashboard and Click on Manage setup12
  • Click on Import setup13
  • Click Upload Json File → Select dashboard.json
  • Below view can be observed setup14



Post on 11 Nov 2022