Appearance

Writing Transformations ​

Datable provides a code editor to write custom transformations against your telemetry in vanilla JavaScript. This gives you complete control over the quality and quantity of the telemetry you forward downstream for analytical use cases.

Adding a transformation ​

  1. To add a transformation step, first navigate to the Pipelines page and click "+ New pipeline", or select an existing pipeline. This opens the pipeline builder page.
  2. From the pipeline builder, you can add transformations and export steps to modify and route your telemetry.
  3. Click on "Add step" and select "Transformation" to enter the interactive Code editor.

The code editor ​

There are four components to the code editor.

  • JavaScript: For writing and editing JavaScript code.
  • Console Log: For printing results to the console.
  • Input: For viewing a sample record. Use this to understand what your data looks like before applying any transformation.
  • Output: For previewing what your data looks like as it leaves the current transformation step.

By default, the JavaScript editor is populated with the following code.

javascript
/***
* You have access to the following inputs:
*  - `metadata`:  { timestamp, datatype }
*    -> datatype is a string, and can be 'logs' or 'traces'
*  - `record`:    { resource, body, ... }
*/

// These are the key attributes of an opentelemetry formatted record
const { attributes, resource, body } = record
const { timestamp, datatype } = metadata

// Here we only allow records tagged as 'logs' to pass through,
// we set the return value to null, we effectively filter out non-log data. 
if (datatype !== 'log') return null;
  
return record
/***
* You have access to the following inputs:
*  - `metadata`:  { timestamp, datatype }
*    -> datatype is a string, and can be 'logs' or 'traces'
*  - `record`:    { resource, body, ... }
*/

// These are the key attributes of an opentelemetry formatted record
const { attributes, resource, body } = record
const { timestamp, datatype } = metadata

// Here we only allow records tagged as 'logs' to pass through,
// we set the return value to null, we effectively filter out non-log data. 
if (datatype !== 'log') return null;
  
return record

Returning ​

Returning a record sends it to the next step

Records ​

The record object represents an individual event entity (log or span) processed by Datable. Datable follows OpenTelemetry specifications wherever possible. In OpenTelemetry, a resource represents the entity that produced the telemetry, while the "attributes" field is used as a grab-bag for information about the event. The attributes object will vary depending on your data source and the type of data ingested.

Datable automatically maps the event values generated by compatible telemetry sources to their OpenTelemetry equivalent. By sharing this common standard, Datable maximizes the number of downstream resources that can ingest and make sense of your telemetry. When an event field cannot be mapped to a corresponding OpenTelemetry field, it is assigned to attributes.

Metadata ​

The metadata object is generated by and internal to Datable. It is immutable, and contains information like the time an event was received, and its datatype ("logs" or "traces").

Body ​

For log events, the body field contains the log's payload. It may be semi-structured (plain-text) or structured (JSON).

Recipes ​

You can use these values to build complex, contextual transformations on your ingested telemetry data. Check out our recipes section to learn how you can reduce costs, enforce standards, and govern your telemetry.

-- Expects it to be returned