When should you use Apache Camel?

Updated 20 November 2020

Tags: Apache Camel

Comments

Someone asked me a question the other day. They had heard about Apache Camel, but weren’t sure whether they should be using it. The person asked me “When should you use Apache Camel?”. What a great question! So I wanted to summarise what I know: when you should use Camel? When should you not use it? And how do people and companies use Apache Camel in the real world?

Apache Camel is an integration framework for Java. It’s most suited for situations where you want to fetch data from files or applications, process and combine it with other data, and then move it to another application.

Apache Camel is a great choice when you’re working with data that needs to be shared between systems. This happens when you have data stored in different applications. For example, personnel files might be stored in an HR system, but need need to be shared with Finance to be able to process the monthly payroll. Camel acts as this kind of programmatic glue between applications, implemented in Java.

Are you wondering whether you should adopt Apache Camel in your own projects? Read on to find out more, and which kind of projects it’s right for.

What is Apache Camel?

Apache Camel is an integration toolkit or framework, written in Java.

Camel comes as a set of libraries and components, and a language (called a DSL) for describing how you want to move data between applications. You can add it to your existing Java application or you can run it in a standalone Java application.

Whenever you need to pull data from an application, remix it, merge it, and route it somewhere else, you can use Camel. Camel does this by providing:

• a language (DSL) for writing your data flows, which are called routes in Camel

• a set of patterns for implementing common things like error handling and transformation, which can be added to your Camel routes

• a set of 300+ components for connecting to hundreds of different applications and protocols

• an embeddable runtime (the “Camel context”) which runs your integrations

Camel is distributed as a set of libraries (JAR files) and is released under the open source Apache License, under the Apache Software Foundation.

When should you use Apache Camel?

I found there are a few major drivers when you would want to use Apache Camel:

Riding with the Camel

• Integrating applications together: Camel is intended for situations where you need to move data between different protocols and applications (like files, emails, APIs or web apps).

  You can use Apache Camel when you want to move data between any one of the apps and protocols supported by its 200+ components. Components in Camel generally all work in a similar way. So, once you understand how to use one component, you will find it easier to use others. Camel includes components for many different apps, from Facebook and Twitter, to Salesforce and Workday. You can also write your own custom component.

• Pattern-based development: Many frequent requirements for integration – like support for transactions, or transformation – would usually be complicated to plan and write in code. But Camel provides many of these, and can often be enabled with just the flick of a switch (OK, by just changing a variable!). Camel provides patterns and functionality for things like:

  • routing data based on its content, using Camel’s content-based routing

  • handling errors, transactions and rollbacks

  • transforming data

  • caching frequently-accessed data

  • encryption and authentication

  These are examples of just some of the things that Camel can do.

  These requirements are made easier in Camel, because it provides these features as a set of patterns, called enterprise integration patterns (after the book of the same name). You can pull any of these enterprise integration patterns “off the shelf”, and use them in your code, without having to write your own solution every time you need these capabilities.

  This toolbox of patterns can make Camel a very productive way to write integration glue code, when you need to connect distributed systems.

  

  Once you’ve learned to drive…

• One high-level style for many integrations: Once you’ve learned the basic patterns, and how to work with Camel components, you’ll find that it becomes easy to churn out many integrations in Camel.

  This is an advantage of Camel: the ability to create many integrations fairly quickly. Camel is ideal if you are developing a set of integrations, and you would like them all to developed in a similar way. This can be really attractive option in larger companies where it helps to pick one approach which is shared and understood by the development team.

• Working with data, and especially Java objects: As it’s a Java framework, it’s especially good at working with Java objects. So if you’re working with a file format that can be de-serialised into a Java object (many file formats can be converted into Java objects, like XML, JSON….) then it will be handled easily by Camel.

How do people use Camel in the real world?

What sort of real-world examples are there? How is Camel used in the wild?

Here’s just a smattering of projects that I’ve either seen or been involved with. In each of these projects, companies are using Camel to achieve a certain goal. I’ve also included the related technologies in brackets:

The wide world of Camel

• Process and route data: Process customer orders and route them on to a database (Camel on Spring Boot with ActiveMQ Artemis and Qpid Dispatch Router)

• Process web-submitted data: Receive and process/transform agricultural survey forms, and then insert into a database (Camel on Apache Karaf)

• Financial transaction processing using message queues: Process financial transactions and route them to the correct department (Camel on Apache Karaf)

• Put a gateway in front of your APIs: Implement a lightweight API Gateway, which authenticates and routes messages to the right API (Apache Camel on Spring Boot)

• Data distribution: Poll for changes in HR data in a SaaS application, and distribute the changes to many downstream apps and files (Apache Camel on Spring Boot on Kubernetes)

• Back-end order processing: Process car orders and car service requests (Apache Camel on Spring Boot, on Kubernetes)

• Modernising legacy APIs: Expose data from a legacy ERP system as a REST API, so that clients can consume it (Apache Camel on Spring Boot, on Kubernetes)

• Ad hoc data processing: Generate a sales report every day from data in a database (Apache Camel on Spring Boot)

Generally the best use cases for Camel are where you have a source of data that you want to consume from – e.g. incoming messages on a queue, or fetching data from an API – and a target, where you want to send the data to.

When shouldn’t you use it?

So when shouldn’t you use Camel? There’s a time and a place for everything, and I think Camel is great but it can’t do it all:

• For heavy data transformation: Although Camel is great at connecting to lots of different applications, it’s not designed for heavy data transformation and analysis. If you have a really data-heavy workflow, where you need to do lots of intensive merging and processing of data – for example like batch processing or ETL – then I think there are other tools which are better for the job.

  But, Camel is still great at orchestrating several steps together into a flow. So you might still consider using Camel as the orchestrator for your data transformation processes.

• If you only need to write one single integration ‘flow’..: Camel is fairly lightweight, and it has got even more lightweight in recent releases (3.x onwards). But it’s probably not worth learning all the patterns and the whole Camel approach to integration development, if you just need it for writing one integration. If you just need to write one integration flow, you’re maybe better off writing your integration code yourself.

• If you don’t have any Java skills in the team: As Camel is a Java framework, it requires some Java knowledge to use it. It doesn’t require much if you use the XML based DSL (language) for configuring it, but it still helps to have some Java knowledge to understand the concepts, because it usually runs on a Java Virtual Machine (JVM). This knowledge is especially useful so that you can troubleshoot when things go wrong: knowing how to read a stack trace always comes in handy!

How do you create apps with Apache Camel?

Apache Camel is free and open source. Getting started with Camel usually involves either creating a new Java application containing Camel, or adding it to your existing Java application.

Camel can be deployed in lots of different ways:

• Embedded in Spring Boot applications and microservices

• Embedded in Quarkus applications, for serverless and containers

• Embedded in Apache Tomcat

• Deployed into Apache Karaf, the OSGi container

• Deployed into WildFly, the Java application server

• Run natively on Kubernetes using Camel-K

• As a simple, standalone Java app with a main() method

To get started, you add the Apache Camel dependencies to your application, create a Camel Context, write your integration routes in Camel’s DSL (either Java or XML), and then start the Context.

A good way to create and run a simple Camel application is to read my Camel tutorial, and then check out the Camel examples repository on GitHub, which is packed with useful example apps for you to learn from.

Once you’ve developed your Camel-based integration app, you can deploy it to your server, virtual machine or a cloud environment.

Final thoughts on using Camel

Camel is a fantastic integration framework that I honestly love working with. It’s especially suited to application integration – that means it’s good if you need to consume, transform and route data between applications.

Apache Camel is even more relevant today in the cloud era, where the number of applications in a typical enterprise just seems to be growing bigger and bigger.

So I hope you’ve found this guide useful, and now you know when to use Apache Camel! When you need to think about how to integrate and process data from applications, why not give Camel a try?