This post introduces connectors to read and write Pravega Streams with Apache Nifi stream processing framework.

The connectors can be used to build end-to-end stream processing pipelines (see Samples) that use Pravega as the stream storage and message bus, and Apache Flink for computation over the streams.

Features & Highlights

PublishPravega: This processor writes incoming FlowFiles to a Pravega stream. It uses Pravega transactions to provide at-least-one guarantees.

PublishPravegaRecord: This is similar to PublishPravega but it uses a NiFi Record Reader to parse the incoming FlowFiles as CSV, JSON, or Avro. Each record will be written as a separate event to a Pravega stream. Events written to Pravega will be serialized with a Record Writer and can be CSV, JSON, or Avro.

ConsumePravega: This processor reads events from a Pravega stream and produces FlowFiles. This processor stores the most recent successful Pravega checkpoint in the NiFi cluster state to allow it to resume when restarting the processor or node. It provides at-least-once guarantees.

All of these processors allow for concurrency in a NiFi cluster or a standalone NiFi node.

Source

https://github.com/pravega/nifi-pravega

Documentation

To learn more about how to build and use the NiFi Connector library, follow the connector documentation here.

More information on NiFi can be found in the NiFi User’s Guide.

License

NiFi connectors for Pravega are 100% open source and community-driven. All components are available under Apache 2 License on GitHub.