Depending on the choices you make, Flink possiable outcomes:

  • Flink makes no effort to recover from failures (at most once)
  • Nothing is lost, but you may experience duplicated results (at least once)
  • Nothing is lost or duplicated (exactly once)

Given that Flink recovers from faults by rewinding and replaying the source data streams, when the ideal situation is described as exactly once this does not mean that every event will be processed exactly once. Instead, it means that every event will affect the state being managed by Flink exactly once.

Exactly Once End-to-end

To achieve exactly once end-to-end, so that every event from the sources affects the sinks exactly once, the following must be true:

  1. your sources must be replayable, and
  2. your sinks must be transactional (or idempotent)