Transform Data
Using a transformation step in the builder to reformat raw data inputs
Last updated
Using a transformation step in the builder to reformat raw data inputs
Last updated
The 'Transform' step will likely be one of the first steps you add to your build flow after setting up your model's inputs.
It will allow you to apply some simple transformation to data inputs, ensuring you get the variables you need, in the correct format, that you'll then use later in your model.
How might you use the Transform step?
If you were building a pricing model for car insurance, it's likely you'd be hitting third-party APIs with data you've gathered from your customers (such as the DVLA's database of vehicle registrations).
You could use the transform step to normalise the case of characters for a registration plate to ensure they were all capitalised, or to remove spaces. Doing this before you hit that API will ensure all of your registration plate data is standardised and won’t cause errors.
Once you've added the 'Transform' step and named it, like you will with any other step in the Build tool (more on that here), you'll need to start adding and setting up your transformations.
To get started, click the cog symbol in the top right of the 'Transform' step.
If this is the first transformation you're adding, you'll see an empty table. If you're building on a template, then there may already been some transformations listed in the table.
To add a new your new transformation, click the 'ADD' button in the top right of the table and complete the details of your transformation.
The next step is to set up and format your new transformation.
This labels the new variable you're creating. Whether you're doing a calculation to calculate someone's age from their date of birth, or reformatting it - keep the description short and simple so you can recognise it easily in future steps.
This is output name that will be passed to the next step, and how you'll reference the input in any syntax in future steps.
This will be auto-generated based on the label you've given, but you can edit it at any time.
This the data type of the data you want to transform the input into.
You can choose from 5 data types: Boolean, Decimal, Integer, String or Date.
This is the expression that will tell the builder which data input you want to transform, and how you want to transform it.
In the most simple terms, your expression should be formatted as INPUT, then FUNCTION, then the desired TRANSFORMATION.
Here's an example of using Swallow's 'Replace' function to reformat a car's registration plate, removing spaces and forcing the whole input to be capitalised:
Output key of the INPUT you want to transform formatted in brackets e.g. {{vehicle_reg}}
Transformation you want to apply. You can use standard operators or choose from a range of pre-existing Swallow functions e.g. .replace
Desired format of the transformed data output e.g. (/\s/g, '').toUpperCase()
To learn how to structure your expression in more detail and to see the full list, check out our full page on Syntax. Here you can learn how to structure your expression in more detail, debug and see a full glossary.
This is the default value you want to set.
This is the data the model will default to if the value is not defined by the customer or the transformation fails for any reason.
It's really simple to test your transformation before you save it.
Simply click the 'Test' button in the bottom right of the 'Expression' form field, and you'll see either a green tick or a red cross next to a label reading 'View'.
This will tell you whether your syntax is written and parsing correctly or not.
Clicking the 'View' label will open up a debug menu that will allow you to see how your expression is transforming test inputs to evaluate if it's doing what you expect it to.
For more information on how testing works in detail, head over to the Running Tests page.
Once you're happy your transformation is working correctly, just press the 'Save' button in the bottom right of the modal and you're ready to start the process and add more transformations if you want to.
Once you're happy with all of your transformations, just press the 'Save' button in the top right of the screen. You're now ready to head back to the 'Build' screen.
