# Normalize data

This guide provides an overview of data normalization in TQL. Normalization transforms raw, inconsistent data into a clean, standardized format that’s ready for analysis, storage, and sharing.

## What is normalization?

Normalization involves several key transformations:

1. **Clean up values** - Replace nulls, normalize sentinels, fix types
2. **Map to schemas** - Translate fields to a standard schema like OCSF
3. **Package mappings** - Create reusable, tested mapping operators

Each step builds on the previous. Start with clean data, then map to your target schema, and finally package your mappings for production use.

## Why normalize?

Raw data from different sources varies in:

* **Field names**: `src_ip` vs `source_address` vs `client.ip`
* **Value formats**: `"true"` vs `true` vs `1` vs `"yes"`
* **Missing values**: `null` vs `""` vs `"-"` vs `"N/A"`
* **Timestamps**: Unix epochs vs ISO strings vs custom formats

Normalization solves these inconsistencies, enabling:

* Unified queries across data sources
* Reliable enrichment and correlation
* Consistent analytics and dashboards
* Interoperability with external tools

## The normalization pipeline

A typical normalization pipeline follows this structure:

```tql
// 1. Collect raw data
from_kafka "raw-events"


// 2. Parse into structured events
this = message.parse_json()


// 3. Clean up values
replace what="N/A", with=null
replace what="-", with=null


// 4. Map to target schema
my_source::ocsf::map


// 5. Output normalized events
publish "normalized-events"
```

## Normalization guides

Work through these guides in order for a complete normalization workflow:

### Clean up values

[Clean up values](https://preview.docs.tenzir.com/375/375/guides/normalization/clean-up-values.md) — Start by fixing data quality issues:

* Replace null placeholders (`"None"`, `"N/A"`, `"-"`)
* Normalize sentinel values
* Fix types (strings to timestamps, IPs, numbers)
* Provide default values for missing fields

### Map to OCSF

[Map to OCSF](https://preview.docs.tenzir.com/375/375/guides/normalization/map-to-ocsf.md) — Learn the comprehensive approach to OCSF mapping:

* Identify the correct event class
* Map fields by attribute group
* Handle unmapped fields
* Validate with `ocsf::cast`

### Map to UDM

[Map to UDM](https://preview.docs.tenzir.com/375/375/guides/normalization/map-to-udm.md) — Learn how to map events to Google SecOps UDM records:

* Choose the correct UDM event type
* Populate metadata and participant nouns
* Convert source values to UDM enums
* Preserve unmapped fields in `additional`

### Map to ASIM

[Map to ASIM](https://preview.docs.tenzir.com/375/375/guides/normalization/map-to-asim.md) — Learn how to map events to Microsoft Sentinel ASIM records:

* Choose the correct ASIM event or entity schema
* Populate schema, product, and event metadata
* Map role-prefixed source, destination, actor, target, and device fields
* Preserve unmapped fields in `AdditionalFields`

### Map to other schemas

[Map to other schemas](https://preview.docs.tenzir.com/375/375/guides/normalization/map-to-other-schemas.md) — Brief guidance on alternative schemas that don’t have a dedicated guide:

* Elastic Common Schema (ECS)

## When to normalize

Normalize data at the ingestion point in your pipeline:

```plaintext
Collection → Parsing → Normalization → Storage/Forwarding
              ↑
        You are here
```

Normalizing early ensures all downstream consumers work with consistent data. Avoid normalizing the same data multiple times by storing normalized events.

## See also

* [Parse string fields](https://preview.docs.tenzir.com/375/375/guides/parsing/parse-string-fields.md)
* [Create a package](https://preview.docs.tenzir.com/375/375/guides/packages/create-a-package.md)
* [Write tests](https://preview.docs.tenzir.com/375/375/guides/testing/write-tests.md)
* [Map data to OCSF](https://preview.docs.tenzir.com/375/375/tutorials/map-data-to-ocsf.md)

## Contents

- [Clean-up-values](https://preview.docs.tenzir.com/375/guides/normalization/clean-up-values.md)
- [Map-to-ocsf](https://preview.docs.tenzir.com/375/guides/normalization/map-to-ocsf.md)
- [Map-to-udm](https://preview.docs.tenzir.com/375/guides/normalization/map-to-udm.md)
- [Map-to-asim](https://preview.docs.tenzir.com/375/guides/normalization/map-to-asim.md)
- [Map-to-other-schemas](https://preview.docs.tenzir.com/375/guides/normalization/map-to-other-schemas.md)