InfluxDB users can now ingest data from third-party brokers without coding

InfluxData, the company behind InfluxDB time series database, is aiming to simplify data collection for its customers. The company today announced it is launching native collectors, a tool that offers a one-click way to allow developers to process, transform and store real-time data from third-party brokers in InfluxDB Cloud. 

Time series data stems from a variety of sources and widely distributed assets and applications. In order to make sense of this information, developers are required to consolidate everything in a single place. The process of consolidating has been fairly complex, with teams facing the task of building pipelines with resource-intensive customizations. Basically, they have to create an intermediary layer to transform and transfer the data to the cloud database in use. 

Developers analyzing time series data with InfluxDB Cloud also face similar challenges, having to rope in additional software agents or write code to immediately process and ingest data.

Native Collectors

With native collectors, InfluxData is removing the intermediary layer, enabling developers to connect sources (devices, applications or third-party public/private brokers) directly to InfluxDB Cloud. This gives them a simple, no-code way to collect, transform and store time series data in cloud environments. In other words, users can get faster ingestion without engaging in complex customization, orchestration, or additional hosting services. 

“Rather than writing custom collectors and other scripts, this model allows a user to provide a connection configuration, set of topic subscriptions, and define a set of parsing rules to take what is typically very unstructured ’in flight’ data and turn it into consumable time series,” said Brian Gilmore, director of IoT and emerging technology at InfluxData.

“All of this is configured through our product’s web user interface, typically by an admin, and after this point, any new message, metric or other data posted to the broker as part of normal operations becomes immediately available to users through our data explorer and notebook interfaces as well as our developer APIs,” he added.

The information is enriched, formatted and processed for InfluxDB Cloud before ingestion.

The offering can easily offload the complexity around building pipelines for time series data ingestion. All one has to do is simply set up service-to-service integrations between third-party brokers and InfluxDB Cloud and then initiate the transfer. As Gilmore noted, the process works in one step and as a managed service, without ongoing developer or administrative overhead. 

In addition, the tool also consolidates what could potentially be thousands of lines of customer-managed custom code run in serverless or containers into three simple configurations.

Availability 

As of now, InfluxDB native collectors are available for MQTT – a standard messaging protocol used for the Internet of Things (IoT). The company has confirmed that it will be expanding the solution and launching additional collectors for Apache Kafka and AMQP, as well. However, those will not be available until later this year. 

“This is a new multi-tenant cloud service offering, with dedicated computing and other resources available in an elastic manner. We have advanced the ability to mine key time-series components from highly unstructured JSON and text message bodies passed through these brokers,”Gilmore noted. “While we expect users to leverage InfluxDB analytics, visualizations, and APIs, just our method for subscribing to topics, receiving updates, and assigning time stamps to what are typically real-time topic updates turns InfluxDB Cloud into a very compelling persistence model for broker-based communication which is far beyond what is available in MQTT alone or with SQL-based persistence today.”

Within the time series data handling arena, InfluxData competes against players such as Druid, Amazon Timestream, QuestDB, Prometheus and Time Scale. InfluxData itself has raised close to $120 million across multiple rounds. It counts Norwest Venture Partners, Sapphire Ventures and Battery Ventures among its investors.