This post is about Kafka Connect, Mirror Maker 2, how they manage offsets, and how to deal with them.
When a consumer starts consuming messages from Kafka, it will probably use a consumer-group and Kafka will store the
offset of the last message consumed by that consumer-group. This offset is stored in a Kafka topic called __consumer_offsets.
Big Data Types is a library that can safely convert types between different Big Data systems.
The library implements a few abstract types that can hold any kind of structure, and using type-class derivations, it can convert between multiple types without having any code relating them. In other words, there is no need to implement a transformation between type A to type B, the library will do it for you.
As an example, let's say we have a generic type called Generic. Now we want to convert from type A to type B.
If we implement the conversion from A to Generic and the conversion from Generic to B, automatically we can convert from A to B although there is no single line of code mixing A and B.