Streams in NodeJs — Our lifesavers

No, I will not talk about some fancy video streams. I will talk about much more boring stuff. If you are NodeJs developer, you probably got in touch with stream api. When I first saw it, I was like: “Why do we need such complicated and relatively hard to maintain pattern? I can just get object, do some operations and throw it to function that consumes it!” I was partly right. In fact if you are new to streams, it takes some learning curve to understand how they work, which types do what, which one to use, how to pipe multiple streams etc. But this article is not about telling you all this theoretical stuff about them. There is a lot of other great articles for that. I will talk more about how and why we used them in our project and hopefully you will see some place in your project too (if they are not there yet!).

If you never saw stream, then streams are basically pipes, where you can throw data, you can read data from it and if you need, you can do some transformations on this data. What is needed to say is, that you process data one by one. You do not get the whole data set that was sent to stream. This takes us to its main advantage of streams and gives us huge advantages.

What advantages? Well let me tell you a little story. This story talks about one application thats main purpose was to take some .csv files and import them to database. Pretty standard…