This was our second a craft conference, the place to be if you are connected to IT and want to be a better professional.
The main topic of the conference was the art of craftsmanship. It was held in Budapest at a nice venue called Varkert Bazar which is right below the fortress at the bank of the Danube river, which made coffee breaks feel nice and cozy.
There were 4 tracks, one in an actual tent within the fortress and other in a garage (parking garage adapted for conference talks) on lazy bags. Buzzwords from mostly all talks were microservices, distributed systems, functional programming, big data. Here are some talks which we found to be the most interesting.
Scaling Micro-Services at Gilt - Adrian Trenaman
Glit is a high volume traffic website in USA which has loads in peaks (the promotion is at 12 o'clock so load is expected at that time). The talk was about explaining how they had turned from a huge monolith (ruby on rails + postgresql) to microservice architecture with hundreds of microservices. During their transition period, they built microservices on top of the existing service layer and migrated functionalists gradually until they migrated all and they could replace the old service layer with new microservice. They also talked about testing, and why it had been hard and expensive to have a staging environment. They are performing testing in production, they have microservice both in frontend and backend so when a new feature is developed it is deployed on one application server but it is not visible. Testers know how to navigate to a hidden functionality and test it and when they are confident it is working, they deploy it on other servers with visibility to customers. Here is a video.
Building Reliable Distributed Data Systems - Jeremy Edberg (Netflix)
This talk was mainly interesting because of the mentioned tools that are in use in Netflix. The talk also put an emphasis on problems with good monitoring, Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, it has nice monitoring capabilities which can be checked in screenshots. Simian Army represents tools for keeping your cloud operating in top form. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures. They have tools which randomly destroy nodes, servers and data centers and you can test how your application would react. There is also latency monkey which adds artificial latency to endpoint to simulate load. Here is a video.
NoSQL Key Value Stores - the Key to Velocity - Peter Milne (Aerospike)
We did not attend this talk but we spoke with them at company booth. AerospikeDB is a pretty fresh DB on the horizon, they are of course advertised as cassandra killer, just like cassandra was a mongo DB killer. This DB is definitely something worth checking out. It is fast, keeps data in memory (has an option to save to data stores in hybrid mode), data is processed on nodes so you can do aggregations and filtering directly on nodes. This accelerates some application filtering since it filters data on each node and aggregates results when returning to application. We are planning to watch this talk to catch more insight, here is a video.
Using Logs To Build a Solid Data Infrastructure - Martin Kleppmann (Linkedin)
A great talk putting emphasis on why you should use append only logs and how you can benefit from them. Essentially, the idea is to place all messages in logs which are append only, and not to mutate data. The log is re-playable, everyone can subscribe to read from it and use that information to sync data. Martin provided an example of how to keep data in DB and elasticsearch without thinking of sync issues. Basically there is a log in which data is written and two workers save that data, one in DB and one indexed in elasticsearch. He talked a lot about Apache Kafka project, which is a distributed messaging broker built on top of distributed log idea. The nice thing about it is that you can mirror logs, it saves messages on disks, replays the whole log history from disks (so when you add new data source you can replay messages and have the whole history in the new data source). Kafka also enables a lot of prototyping and ideas to be tested, you just mirror your log, attach new subscriber which will, for example, write to elasticsearch and you test against it, if it is ok, use it, if not, just drop the mirror of queues. Here is a video. Logical next step was the following talk How Do You Scale a Logging - Paul Stack (OpenTable) which also talked about kafka, mistakes they had made and some of take-aways from those mistakes, so you might watch that video as well, not that good as Martin Kleppmann but good for kafka experience.
Microservice AntiPatterns - Tammer Saleh (Pivotal)
The last talk we would recommend, Tammer from pivotal, provided a nice overview of 10 things that hurt when you do microservice architecture, and how to solve them (at least from his perspective). One of the things we liked from this talk is the explanation how to develop server and client at same time. The basic idea is to create an integration client (http client) with all endpoints calls for each language which you plan to support. This is like a contract, which has mock mode and from that point on, both client and server sides can be developed. Client can test and develop against contract (mock response) and server side must be developed to adhere to contract. When server side is ready, mock mode is switched to real mode. This way you can speed up development and provide sort of documentation right away. Here is a video.
Food for thought
Microservice architecture is definitely the way to go forward but it is accompanied with its problems (monitoring, testing, deployment). Big data is the next big thing, systems should be built with that in mind (a rule of thumb is always build for 3, meaning 3 application servers, 3 nodes, etc.). Scalability and fault tolerance is a new norm and each system must have the ability to grow and must be resilient to errors
In relation to craftsmanship, the speakers reflected on the topic of learning, how hard it is to cope with technology and be good in all things required within a project. In the past, we had only a couple of tools, frameworks, languages which had to be understood in order to do your job well and to finish projects. Now, microservice architecture has enabled an easy usage of different DB, different languages and frameworks, deployment tools, monitoring tools but it is quite hard to follow everything. The idea is to delegate, and implement some sort of microservice architecture in the organization as well. You should have specialized teams in your organization, know just enough to delegate and to manage to remain on top level but provide some features to these micro teams (two pizza teams) to work on them from bottom to top independently and give full control to them regarding the technology and tooling choices.