I am trying to understand the benefits of fan-out design pattern. In the below blog, it is mentioned that "there is overhead associated with categorizing streams into topics, and producer throughput is sensitive to that overhead"
Can someone explain what is the overhead mentioned here ? Is it always necessary to consider a Fan-out pattern ? If I know the actual topic names (mapping a field in the input payload to a topic under a stream), do I need to implement a fan-out pattern where data is first received at the raw topic and categorized into a bunch of topics based on some field in the input payload? Please share your thoughts and real world experience.
To give a background, I am trying to stream the data from a cloud system to MapR stream. I will know the actual target topic name by parsing the payload at the gateway(An ESB where the Producer runs). I have considered the fan-out design pattern for now. But trying to get some thoughts/suggestions around it before I implement it in production.