DTL - Data Transformation Language!

NEWS:

Editor Support Update - Starting with DTL 5.0.4, there is now extensive editor support with syntax highlighting and snippets for all DTL functions. If you use VS Code, you can find the DTL extension in the extension marketplace! (look for the one by jk0ne / DTL Team) If you use vim, Sublime Text or any other tmLanguage compatible editor, you can get the extensions from the releases screen in gitlab.
Doing Math right - Beginning with version 5.0.0, DTL performs numeric operations using arbitrary percision arithmetic by default. This means the math calculations you perform in DTL are going to be exactly correct. Say goodbye to floating-point math errors. Not sure what this means? Try entering 0.1 + 0.2 in your browser console or in node.js.

What is DTL?

DTL is a powerful data manipulation toolset that allows you to extract, manipulate, and transform data on the command line, in your node.js code, on the server, or even in the browser. You can use DTL to describe how to transform one set of data into another, validate input data, or perform complex calculations.

You can try DTL by simply installing it with npm or yarn:

$ yarn global add dtl-js ## or $ npm install -g dtl-js

OR you can try right now in your browser below.

DTL also includes powerful CLI tools for transforming data in bulk (dtl) and for exploring the syntax (dtlr).

The dtl command line tool lets you process data in bulk, sifting, sorting, grouping and performing complex data manipulation. It's great for things like data migration, batch processing or cross-application bulk data transformation.

The dtl tool is like having all the *nix text tools (like grep, sort awk, sed, cut etc.) for your complex data structures, and works with JSON, yaml, and csv data.

Finally, the dtlr REPL tool has built-in help and is great for exploring your data and trying things out in the terminal.

TL;DR

Try it out!

You can try DTL below. Select an example to auto-fill with sample data and related DTL transform, then click the button to apply the DTL transform. Your result will be displayed in the Result box. (Pro tip: Try editing the transform or input data to change the result.)

Load an example:

Input data

→↓

DTL Transform

↓ ↓

Result data

Why DTL?

One of the key advantages of DTL is its compatibility with JavaScript, which means you can use it anywhere you can use JavaScript. DTL is also incredibly versatile and can be used to manipulate data in a variety of scenarios, including handling input data from forms or API calls and converting between data formats.

DTL is a secure and easy-to-use toolset that lets you define how to manipulate data without writing code. Because DTL has no access to the larger environment, no filesystem access, and no library access, it's a fundamentally safe way to implement complicated data transformations and calculations, even if those calculations come from a third party.

As of 5.0.0 of DTL, all math is done using arbitrary precision arithmetic (using bignumber.js) This means your math calculations in DTL are accurate (more accurate than standard Javascript) and will produce exactly the correct results.

Compared to raw code, DTL is more portable, secure, and easy to use. It offers a more secure approach to data manipulation by restricting its access to the larger environment. This feature makes it an ideal choice for developers who need to handle sensitive data or work with data from untrusted sources. Additionally, DTL's portability allows you to put your calculations wherever you need them. They can be transfered between server and browser safely, they can be stored in a database or anywhere that JSON can be stored, giving you maximum flexibility and consistency across your application.

DTL also gives you easy ways to perform complex tasks which would require significantly more complex and potentially bug-prone code. It's grep() function, for example, makes it supremely easy to extract the specific data you're interested in regardless of how complex the rules are to find it.

In short, It's better than raw code because it's portable, secure, and easy to use.

The DTL CLI's versatility and compatibility with multiple data formats, such as JSON, YAML, and CSV, make it an excellent tool for data extraction, transformation, and loading (ETL) tasks. Its CLI tool, dtl, allows you to work with complex data structures as easily as text tools like grep and awk allow you to work with text.

If you're looking for a powerful data manipulation toolset that works seamlessly with JavaScript and provides CLI tools for batch processing and real-time data manipulation, DTL is the perfect choice for you.

DTL Docs

How do I get DTL?

Installing DTL is easy...

$ yarn global add dtl-js ## or $ npm install -g dtl-js

To use DTL in your site or browser-based app, simply use the dtl-transform-browser.min.js from the npm package, or include it from the jsdelivr.net CDN:

Usage

To use DTL to experiment, you can load the dtlr REPL environment by issuing the following command:

$ dtlr

Once the REPL environment is loaded, you can enter the command .help to get general help, or .help followed by the name of a helper function to get help on that function, for example:.help grep will show you the documentation for the grep function.

How do I use it in my code?

To use DTL in your code, simply require the DTL module in your code:

const DTL = require("dtl-js"); let transform = { "out": "(: $. :)" }; let input = { /* your input data */ }; let dtl = new DTL(); let result = dtl.apply(input, transform);

You can browse the DTL npm page for a longer example.

How do I use the dtl CLI tool

The dtl command line tool allows you to apply complex (or simple) DTL transforms to data on the command line without writing any code. It understands JSON, JSONLines, YAML and CSV data for both input and output. Issue the command below for help:

$ dtl -h

To convert data from yaml to JSON:

$ dtl -o data.json data.yaml

To convert data from JSON to yaml:

$ dtl -o new.yaml data.json

Take JSON input on stdin, and output YAML on stdout

$ dtl -I json -O yaml

To extract names from a list of people

$ dtl -e 'map($people "(: $item.name :)")' astros.json

More DTL docs

DTL has extensive documentation on its syntax and helpers. You can find information on the expression syntax here and information about the helper functions within DTL in the helper reference. Some docs on some more advanced topics exists. You can also try out dtl and get help in the dtlr command line tool.

Talk to us!

If you would like to chat about DTL and related topics, you can find us on Discord here.

Who made this thing?!?

That'd be mainly this guy, Jay Kuri with a little help from his friends