Skip to content

Latest commit

 

History

History
201 lines (124 loc) · 5.75 KB

README.md

File metadata and controls

201 lines (124 loc) · 5.75 KB

Trucking Data Simulator

About the Simulator

The trucking data simulator allows you to do the following:

  • Generate streaming events for different sensors on a truck
  • Control the number of trucks/drivers on the road
  • Control the number of events generated by each truck
  • Use real trucking routes with real lat/long locations for the truck
  • Control the output of the event (csv, json)
  • Control what metadata goes into the event (schema metadata)
  • Control where the event is generated (in a file, as an event into Kafka)

Out of the box, it supports two event types representing 2 different sesnsors that would be on a truck:

  • Truck Geo Event

    • Sample Raw Event:

2016-10-19 14:54:38.18|truck_geo_event|40|23|Jeff Markham|1090292248|Peoria to Ceder Rapids Route 2|Normal|40.7|-89.52|1|

Architecture Diagram

  • Truck Speed Event

2016-10-19 14:54:38.197|truck_speed_event|40|23|Jeff Markham|1090292248|Peoria to Ceder Rapids Route 2|73|

Architecture Diagram

Building the Simulator

Pre-requisites

  1. Java 8
  2. setup your JAVA_HOME to point to java 8
  3. Put JAVA_HOME on your PATH
  4. Install Maven
  5. Put Maven on your PATH
  6. Git

Build the Simulator

  • Clone the repo and the build the libraries
git clone https://github.com/georgevetticaden/hdp.git
cd hdp/app-utils/hdp-app-utils/
mvn clean install -DskipTests=true
cd ../hdp-app-utils/../../reference-apps/iot-trucking-app/
mvn clean install -DskipTests=true
cd trucking-data-simulator
mvn clean compile assembly:single
cd target

Running the Simulator

  1. Download the Data-Loader.zip.
  2. Unzip it to the location where you want to run the simulator. Lets call the directory you unzip it to as: $DATA_LOADER_HOME.
  3. untar the route.tar.gz file (tar -zxvf $DATA_LOADER_HOME/routes.tar.gz)

The simulator has a number of different runners based on the type of output you want generated. The below walks through some of the scenarios

Example 1: Generate Trucking Events with SchemaId Embedded in Event

  • Command:
nohup java -cp \
data-loader-jar-with-dependencies.jar \
hortonworks.hdf.sam.refapp.simulator.SimulationRegistrySerializerRunnerApp \
20000 \
hortonworks.hdf.sam.refapp.simulator.impl.domain.transport.Truck \
hortonworks.hdf.sam.refapp.simulator.impl.collectors.FileEventWithSchemaIdCollector \
1 \
$DATA_LOADER_HOME/routes/midwest/ \
10000 \
/tmp/truck-sensor-data/all-streams-with-schemaid-embedded.txt \
$REST_URL_TO_SCHEMA_REGISTRY \
ALL_STREAMS &

  • Details

    • Each Truck (13 trucks total) will generate 20000 events.
    • Each event will be prepended with schema-id and schema-version
    • Each truck will wait 10 seconds before its route is finished before going on the road again.
    • All of these events will be streamed into the following file: /tmp/truck-sensor-data/all-streams-with-schemaid-embedded.txt

  • Sample Output

<schema-id>9<schema-id><schema-version>1<schema-version>|2017-05-01 12:30:42.086|truck_geo_event|69|13|Suresh Srinivas|6|Des Moines to Chicago|Lane Departure|41.62|-93.58|1|

<schema-id>10<schema-id><schema-version>1<schema-version>|2017-05-01 12:30:42.099|truck_speed_event|69|13|Suresh Srinivas|6|Des Moines to Chicago|104|

Example 2: Generate Trucking Events in CSV Format

  • Command:
nohup java -cp \
data-loader-jar-with-dependencies.jar \
hortonworks.hdf.sam.refapp.simulator.SimulationRunnerApp \
20000 \
hortonworks.hdf.sam.refapp.simulator.impl.domain.transport.Truck \
hortonworks.hdf.sam.refapp.simulator.impl.collectors.FileEventCollector \
1 \
$DATA_LOADER_HOME/routes/midwest/ \
10000 \
/tmp/truck-sensor-data/all-streams.txt \
ALL_STREAMS &

  • Details

    • Each Truck (13 trucks total) will generate 20000 events.
    • Each truck will wait 10 seconds before its route is finished before going on the road again.
    • All of these events will be streamed into the following file: /tmp/truck-sensor-data/all-streams.txt

  • Sample Output

2017-05-01 12:38:42.747|truck_geo_event|101|13|Suresh Srinivas|12|Des Moines to Chicago|Unsafe tail distance|41.62|-93.58|1|

2017-05-01 12:38:42.799|truck_speed_event|101|13|Suresh Srinivas|12|Des Moines to Chicago|80|

2017-05-01 12:38:42.8|truck_geo_event|102|10|George Vetticaden|4|Saint Louis to Tulsa|Normal|38.64|-90.18|1|

2017-05-01 12:38:42.801|truck_speed_event|102|10|George Vetticaden|4|Saint Louis to Tulsa|65|

Example 3: Generate Trucking Events Serialized by HWX Schema Registry

nohup java -cp \
data-loader-jar-with-dependencies.jar \
hortonworks.hdf.sam.refapp.simulator.SimulationRegistrySerializerRunnerApp \
20000 \
hortonworks.hdf.sam.refapp.simulator.impl.domain.transport.Truck \
hortonworks.hdf.sam.refapp.simulator.impl.collectors.FileEventSerializedWithRegistryCollector \
1 \
$DATA_LOADER_HOME/routes/midwest/ \
10000 \
/tmp/truck-sensor-data/all-streams-serialized-with-hwx-registry.txt \
$REST_URL_TO_SCHEMA_REGISTRY \
ALL_STREAMS &

  • Details

    • Each Truck (13 trucks total) will generate 20000 events.
    • Each event is serialized into Avro using the HWX Schema Registry
    • Each truck will wait 10 seconds before its route is finished before going on the road again.
    • All of these events will be streamed into the following file: /tmp/truck-sensor-data/all-streams-serialized-with-hwx-registry.txt

  • Sample Output

^A^@^@^@^@^@^@^@        ^@^@^@^A^@.2017-05-01 13:13:16.499^^truck_geo_eventL^X^PJoe Witt^B,Saint Louis to Memphis^\Lane Departure<8f>Âõ(\OC@<9a><99><99><99><99><89>VÀ^B^A^@^@^@^@^@^@^@
^@^@^@^A^@.2017-05-01 13:13:17.119"truck_speed_eventL^X^PJoe Witt^B,Saint Louis to Memphis¦^A^A^@^@^@^@^@^@^@   ^@^@^@^A^@