Skip to main content

JSONata Masterclass: Writing Transforms That Stay Readable

The Transform component is where most workflows do their real work. It builds request bodies before a REST API call, cleans up responses afterwards, and turns cryptic M3 field codes into readable JSON.

Short expressions are easy. Expressions grow, though, and a 60-line Transform written as one nested expression is very hard to debug six months later. This guide shows how experienced builders structure their JSONata so it stays readable, predictable and safe.

What you'll learn​

  • How a Transform sees its input, and how to reach data from earlier steps
  • How to break an expression into named steps with blocks and variable declarations ($name := …)
  • How to write small helper functions and reuse them
  • The defensive patterns that prevent the most common surprises: missing fields, single values instead of arrays, and numbers stored as text
  • How to group and total data, and how to join large lists quickly with an index and $lookup()

Prerequisites​

  • A workflow with a Transform step, typically placed after a REST API or M3 API step.
  • Basic familiarity with JSONata paths such as body.results and filters such as [price > 20]. The JSONata documentation covers these well.

Where a Transform gets its data​

A Transform expression is evaluated against the output of the previous step. That output is available as $. Paths without a prefix, such as body.results, start from there too.

After a REST API step, the input has the shape { "statusCode": …, "body": …, "headers": … }, so the data you want is usually under body.

When you need data from somewhere else in the workflow, use $context. The most common paths are below. The $context Reference lists everything it holds.

You wantWrite
The data the workflow was started with$context.data.trigger.body
The output of a specific earlier step$context.data.all.<step id>, for example $context.data.all.rest_api_1.body
A tag on the user who started the workflow$context.user.tags.<tag key>[0]

Tags are key–value pairs set on each user. For example, $context.user.tags.warehouse[0] lets one workflow behave differently depending on who runs it. A tag's value is usually stored as a one-item list, such as ["100"], and the [0] takes the value out of it. See Tags in the $context Reference for why it matters.

Recommended: refer to steps by ID

Use $context.data.all.<step id> instead of $, even for the previous step. The expression then keeps working when steps are added or moved, and it gives the same result when you test it in the Transform editor, where $ holds the whole loaded run instead of the previous step. The examples in this guide do this, with rest_api_1 as the ID of the REST API step. See How step IDs are formed to work out the ID of your own step.

The example used in this guide​

Every section below works on the same input: the response from a REST API step, rest_api_1, that listed the lines of an M3 customer order. As with most M3 data, every value is text, and some values have trailing spaces:

{
  "statusCode": 200,
  "body": {
    "results": [
      {
        "transaction": "LstLine",
        "records": [
          { "ORNO": "1000123", "PONR": "1", "POSX": "0", "ITNO": "BOLT-M8 ", "ITDS": "Bolt M8x40", "WHLO": "100", "ORQA": "250", "DLQT": "250", "SAPR": "0.45", "DWDT": "20261015", "ORST": "66" },
          { "ORNO": "1000123", "PONR": "2", "POSX": "0", "ITNO": "NUT-M8", "ITDS": "Nut M8", "WHLO": "100", "ORQA": "250", "DLQT": "100", "SAPR": "0.12", "DWDT": "20261015", "ORST": "44" },
          { "ORNO": "1000123", "PONR": "3", "POSX": "0", "ITNO": "WASHER-M8", "ITDS": "Washer M8", "WHLO": "200", "ORQA": "500", "DLQT": "0", "SAPR": "0.05", "DWDT": "20261020", "ORST": "33" }
        ]
      }
    ]
  }
}

The goal is a clean order summary: readable line fields, real numbers, ISO dates, open quantities per warehouse, and an order total.

Step 1 — Recognise the one-liner problem​

A first attempt usually looks like this:

$context.data.all.rest_api_1.body.results[transaction="LstLine"].records.{"item": ITNO, "qty": $number(ORQA), "open": $number(ORQA) - $number(DLQT), "date": $substring(DWDT,0,4) & "-" & $substring(DWDT,4,2) & "-" & $substring(DWDT,6,2)}

It works, but it has three problems that only get worse as the expression grows:

  • The long path to the records is buried at the front. If you need the records again (for a total, say), you have to repeat it.
  • The date logic sits inline. The next date field means copying it again, and a fix has to be made in every copy.
  • The output keeps the trailing space in "BOLT-M8 ", and nothing in the expression makes that easy to spot.

Step 2 — Use a block with variable declarations​

Wrap the expression in parentheses to make a block. Inside a block, you write a sequence of statements separated by semicolons. Each $name := … declares a variable. The value of the last statement is the result of the whole Transform.

(
  /* 1. Pick out the raw records once */
  $lines := $context.data.all.rest_api_1.body.results[transaction = "LstLine"].records;

  /* 2. Reshape each line */
  $lines.{
    "item": $trim(ITNO),
    "orderedQty": $number(ORQA),
    "openQty": $number(ORQA) - $number(DLQT)
  }
)

Output:

[
  { "item": "BOLT-M8", "orderedQty": 250, "openQty": 0 },
  { "item": "NUT-M8", "orderedQty": 250, "openQty": 150 },
  { "item": "WASHER-M8", "orderedQty": 500, "openQty": 500 }
]

The expression now reads top to bottom: first find the data, then shape it. A few habits make blocks easier to read:

  • Name intermediate results after what they are ($lines, $openLines, $byWarehouse), not after where they came from ($x, $tmp, $data2).
  • Declare each long path once. Every later statement uses the short name.
  • Comment the intent with /* … */, especially for business rules ("status 66 means fully delivered").
tip

Variables declared inside a block exist only inside that block. A Transform built this way never leaks names into other steps, so short, local names are fine.

Step 3 — Extract helper functions​

Anything you do more than once becomes a function. Declare it like a variable:

(
  /* helpers */
  $toIsoDate := function($d) {
    $length($d) = 8
      ? $substring($d, 0, 4) & "-" & $substring($d, 4, 2) & "-" & $substring($d, 6, 2)
      : null
  };
  $num := function($v) { $exists($v) and $trim($string($v)) != "" ? $number($trim($string($v))) : 0 };

  $lines := $context.data.all.rest_api_1.body.results[transaction = "LstLine"].records;

  $lines.{
    "lineNumber": $num(PONR) * 1000 + $num(POSX),
    "item": $trim(ITNO),
    "description": ITDS,
    "warehouse": WHLO,
    "orderedQty": $num(ORQA),
    "openQty": $num(ORQA) - $num(DLQT),
    "lineAmount": $round($num(ORQA) * $num(SAPR), 2),
    "requestedDate": $toIsoDate(DWDT),
    "status": $num(ORST)
  }[]
)

Output (first line shown):

[
  {
    "lineNumber": 1000,
    "item": "BOLT-M8",
    "description": "Bolt M8x40",
    "warehouse": "100",
    "orderedQty": 250,
    "openQty": 0,
    "lineAmount": 112.5,
    "requestedDate": "2026-10-15",
    "status": 66
  }
]

Two helpers do a lot of work here:

  • $toIsoDate converts M3's YYYYMMDD dates and returns null instead of a broken string when the value is empty.
  • $num turns text into a number and returns 0 for missing or blank values. Plain $number() raises an error for an empty string, and also for a padded value such as " 5". That's why the helper trims the value first.

Put helpers at the top of the block, so a reader sees the vocabulary before the logic that uses it.

Step 4 — Always get an array when you expect a list​

This is the most common JSONata surprise. When a path matches exactly one item, JSONata returns that item on its own, not a one-item array.

With the expression from Step 2, an order with a single line returns this:

{ "item": "BOLT-M8", "orderedQty": 250, "openQty": 0 }

It does not return [ { … } ]. Anything downstream that expects a list breaks: a REST API For each, an Email {{#each}} loop, a Table Updater, or the system receiving your request.

The fix is the [] suffix. It forces the result to stay an array, which is why the expression in Step 3 ends with }[]. Use it on every list you build:

"lines": $lines.{ "item": ITNO }[],
"items": $wl.item[]

The same applies to filters. $lines[openQty > 0] returns a single object when only one line is open. Write $lines[openQty > 0][] if the result must be a list.

Step 5 — Compare numbers as numbers​

M3 and many other systems return numbers as text. Text compares alphabetically, so the results look plausible but are wrong:

ExpressionResultWhy
"10" > "9"falseText comparison: "1" sorts before "9"
"250" = 250falseText is never equal to a number
["9","10","100"]^($)["10","100","9"]Sorted as text
["9","10","100"]^($number($))["9","10","100"]Sorted as numbers

Convert values with $number() (or a helper like $num) before you compare, sort, sum or do arithmetic. Convert back with $string() when the receiving system expects text. Use $type($v) while debugging to check what you actually have.

Step 6 — Group and total​

To group, put an object constructor directly after a path: path{ key: value }. For each distinct key, the value expression is evaluated over the matching items:

$lines{ warehouse: $sum(openQty) }
{ "100": 150, "200": 500 }

Most receiving systems want a list rather than an object keyed by warehouse. A clean pattern is to group the full lines first and then build one summary per key:

$byWarehouse := $lines{ warehouse: $ };
$warehouses := $keys($byWarehouse).(
  $wl := $lookup($byWarehouse, $)[];
  {
    "warehouse": $,
    "lineCount": $count($wl),
    "openQty": $sum($wl.openQty),
    "items": $wl.item[]
  }
);
  • $keys() gives the list of warehouse codes.
  • $lookup() fetches the lines for one code.
  • The [] after $lookup(…) keeps a warehouse with a single line from turning into a lone object (see Step 4).

For simple key/value results, $each() converts an object straight into a list:

$each($lines{ warehouse: $sum(openQty) }, function($v, $k) { { "warehouse": $k, "openQty": $v } })
[ { "warehouse": "100", "openQty": 150 }, { "warehouse": "200", "openQty": 500 } ]

Step 7 — Join large lists with an index, not a filter​

A common task is to combine two lists, for example enriching each order line with item master data that an earlier step fetched. The obvious way is a filter inside the loop:

$items := $context.data.all.rest_api_2.body.results[transaction = "LstItems"].records;

$lines.(
  $line := $;
  $item := $items[$trim(ITNO) = $line.item];
  $merge([$line, { "unit": $item.UNMS, "itemGroup": $item.ITGR }])
)

This gives the right result, and with a handful of records it is fine. It just doesn't scale.

Why the filter gets slow​

A filter like $items[…] has no shortcut. It checks the condition against every item in the list, every time it runs. Inside $lines.( … ) it runs once per line, so the total work is lines × items:

LinesItemsComparisons with a filter
5020010,000
2,00010,00020,000,000
10,00050,000500,000,000

The data only grew by a factor of 200 or so, but the work grew by a factor of 50,000. A Transform that took milliseconds in testing can take minutes, or time out, on a production run.

$lookup() on an object works differently. An object's keys are indexed, so fetching a value by its key is a direct jump to that entry, and it costs about the same whether the object has 10 keys or 100,000. If you build that object once and look up each line in it, the total work is roughly lines + items: 60,000 steps instead of 500 million in the last row above.

Refactor the list into an object keyed by the join field​

$lookup() needs an object, but REST and M3 responses give you an array of objects. So the first step is to turn the array into an object where each key is the join field and each value is the full original record:

[
  { "ITNO": "BOLT-M8", "ITDS": "Bolt M8x40", "UNMS": "PCS", "ITGR": "FAST" },
  { "ITNO": "NUT-M8", "ITDS": "Nut M8", "UNMS": "PCS", "ITGR": "FAST" }
]

becomes:

{
  "BOLT-M8": { "ITNO": "BOLT-M8", "ITDS": "Bolt M8x40", "UNMS": "PCS", "ITGR": "FAST" },
  "NUT-M8": { "ITNO": "NUT-M8", "ITDS": "Nut M8", "UNMS": "PCS", "ITGR": "FAST" }
}

This is the same grouping syntax as Step 6. $ as the value keeps the whole record, so nothing is lost and every field is still available after the lookup:

/* one pass over the items: build the index */
$itemsByNo := $items{ $trim(ITNO): $ };

/* one cheap lookup per line */
$lines.(
  $item := $lookup($itemsByNo, item);
  $merge([$, { "unit": $item.UNMS, "itemGroup": $item.ITGR }])
)

Three details matter when you build the index:

  • Keys must be text. An object key that evaluates to a number raises an error, so wrap numeric fields: $records{ $string(CUNO): $ }. Look up with the same $string(…) value.
  • Normalise the key the same way on both sides. If you trim or uppercase when you build the index, do the same when you look up. "BOLT-M8 " and "BOLT-M8" are different keys, and the lookup quietly returns nothing.
  • Duplicate keys collect into an array. If two records share a key, the value becomes an array of both records instead of a single object. That is what you want for one-to-many joins, as with $byWarehouse in Step 6. Add [] after the lookup there (see Step 4). For a key you expect to be unique, duplicates usually point to a data problem in the source.
tip

Reach for an index whenever a filter on one list runs inside a loop over another list, and either list can hold more than a few hundred records. The same applies to $filter() inside $map(). A single filter that isn't inside a loop, like $lines[openQty > 0], scans the list only once and needs no index.

Step 8 — Put it all together​

Here is the complete Transform. It is organised in labelled sections: helpers, input, reshape, group, result.

(
  /* ---------- helpers ---------- */
  $toIsoDate := function($d) {
    $length($d) = 8
      ? $substring($d, 0, 4) & "-" & $substring($d, 4, 2) & "-" & $substring($d, 6, 2)
      : null
  };
  $num := function($v) { $exists($v) and $trim($string($v)) != "" ? $number($trim($string($v))) : 0 };

  /* ---------- input ---------- */
  $records := $context.data.all.rest_api_1.body.results[transaction = "LstLine"].records;

  /* ---------- reshape ---------- */
  $lines := $records.{
    "lineNumber": $num(PONR) * 1000 + $num(POSX),
    "item": $trim(ITNO),
    "warehouse": WHLO,
    "orderedQty": $num(ORQA),
    "openQty": $num(ORQA) - $num(DLQT),
    "lineAmount": $round($num(ORQA) * $num(SAPR), 2),
    "requestedDate": $toIsoDate(DWDT),
    "status": $num(ORST)
  }[];

  /* ---------- group by warehouse ---------- */
  $byWarehouse := $lines{ warehouse: $ };
  $warehouses := $keys($byWarehouse).(
    $wl := $lookup($byWarehouse, $)[];
    {
      "warehouse": $,
      "lineCount": $count($wl),
      "openQty": $sum($wl.openQty),
      "items": $wl.item[]
    }
  );

  /* ---------- result ---------- */
  {
    "orderNumber": $records[0].ORNO,
    "lineCount": $count($lines),
    "totalAmount": $round($sum($lines.lineAmount), 2),
    "fullyDelivered": $count($lines[openQty > 0]) = 0,
    "earliestRequestedDate": $min($lines.$toMillis(requestedDate)) ~> $fromMillis("[Y0001]-[M01]-[D01]"),
    "warehouses": $warehouses[],
    "lines": $lines
  }
)

Output:

{
  "orderNumber": "1000123",
  "lineCount": 3,
  "totalAmount": 167.5,
  "fullyDelivered": false,
  "earliestRequestedDate": "2026-10-15",
  "warehouses": [
    { "warehouse": "100", "lineCount": 2, "openQty": 150, "items": ["BOLT-M8", "NUT-M8"] },
    { "warehouse": "200", "lineCount": 1, "openQty": 500, "items": ["WASHER-M8"] }
  ],
  "lines": [
    { "lineNumber": 1000, "item": "BOLT-M8", "warehouse": "100", "orderedQty": 250, "openQty": 0, "lineAmount": 112.5, "requestedDate": "2026-10-15", "status": 66 },
    { "lineNumber": 2000, "item": "NUT-M8", "warehouse": "100", "orderedQty": 250, "openQty": 150, "lineAmount": 30, "requestedDate": "2026-10-15", "status": 44 },
    { "lineNumber": 3000, "item": "WASHER-M8", "warehouse": "200", "orderedQty": 500, "openQty": 500, "lineAmount": 25, "requestedDate": "2026-10-20", "status": 33 }
  ]
}

The ~> operator in earliestRequestedDate passes the value on its left into the function on its right. It reads as "take the smallest date, then format it", which is often clearer than nesting one function call inside another.

Function cheat sheet​

These functions and paths cover most of what real-world Transforms need:

NeedFunctionsExample
Read a step's output$context.data.all$context.data.all.rest_api_1.body
Read the workflow's input$context.data.trigger$context.data.trigger.body.orderNumber
Who ran it, and which run$context.user, $context.workflowExecution$context.user.email, $context.workflowExecution.executionId
Environment-specific values$context.runtime$context.runtime.connections.erp
Does a field exist?$exists$exists(note) ? note : "—"
How many?$count$count($lines[openQty > 0])
Convert type$number, $string, $boolean$number(ORQA)
Clean up text$trim, $uppercase, $lowercase, $replace$trim(ITNO)
Cut text$substring, $substringBefore, $substringAfter$substring(DWDT, 0, 4)
Split and join$split, $join$join($split("A;B;C", ";"), ", ") → "A, B, C"
Look up a value by key$lookup$lookup({"100": "Main warehouse"}, WHLO)
Combine objects$merge$merge([{"a": 1}, {"b": 2}]) → {"a": 1, "b": 2}
Unique values$distinct$distinct($lines.warehouse)
Object keys / object → list$keys, $each$keys($byWarehouse)
Loop with logic$map, $filter, $reduce$map(items, function($v, $i) { { "pos": $i + 1, "item": $v } })
Totals$sum, $max, $min, $average, $round$round($sum($lines.lineAmount), 2)
Dates and times$now, $millis, $toMillis, $fromMillis$fromMillis($toMillis("2026-10-02T09:30:00Z"), "[D01].[M01].[Y0001] [H01]:[m01]") → "02.10.2026 09:30"
note

$now() and $millis() return the current time in UTC. Allow for this when you compare against local business dates.

Anti-patterns to avoid​

AvoidDo instead
One giant nested expressionA block with named steps, read top to bottom
Reading the previous step with $$context.data.all.<step id>, so it's clear which step you mean
Repeating $context.data.all.rest_api_1.body.results[transaction="LstLine"].recordsDeclare it once: $records := …
Copy-pasting the same date or number logicA helper function at the top of the block
Assuming a path always returns a listAdd [] wherever a list is expected
Comparing or summing text valuesConvert with $number() first
Filtering one list inside a loop over anotherIndex it once with list{ key: $ }, then $lookup()
Business rules without explanation/* status 66 = fully delivered */

Testing your expressions​

  • Test in the Transform editor: Load a test run, and Output shows the result as you type, including the error message if the expression fails. Expressions that refer to steps by ID work on the loaded run as they are. See Testing with real data for what differs from a real run.
  • The JSONata exerciser is handy for trying out standard JSONata, but it has no $context or Vince-specific functions. While testing there, replace $context.data.all.rest_api_1 with a variable declared at the top of the block (for example $order := { … sample … };), and swap it back afterwards.
  • Test the edge cases on purpose: an empty list, a single item (Step 4), a missing field, and a value with extra spaces.

Further reading​

Last updated on