How to Clean and Normalize Text Data: Pipelines & Examples

Messy text is everywhere: copy-pasted content from web pages, log files with inconsistent formatting, CSV exports with trailing spaces, code snippets with mixed line endings. Before you can process, analyze, or display text reliably, you need to clean it up. This guide covers the most common text cleaning tasks with practical solutions.

The Cleaning Pipeline

Text cleaning works best as a pipeline — a sequence of small, focused operations. Here’s a recommended order:

1. Normalize line endings (CRLF → LF)
2. Trim whitespace (leading/trailing per line)
3. Collapse spaces (multiple → single)
4. Remove empty lines (if desired)
5. Remove duplicates (if desired)
6. Normalize Unicode (NFC form)

Each step is independent and composable. Skip what you don’t need.

Removing Duplicate Lines

Duplicates appear frequently in log files, database exports, and copy-pasted lists.

Preserving order

const unique = [...new Set(text.split('\n'))];

Set keeps insertion order in JavaScript (ES2015+), so the first occurrence of each line is preserved.

unique = list(dict.fromkeys(text.splitlines()))

Python’s dict preserves insertion order since 3.7+.

awk '!seen[$0]++' filename

The awk one-liner is the gold standard for order-preserving deduplication in the terminal.

When sorted order is acceptable

sort -u filename

Faster than awk for huge files because sort can use disk-backed merge sort, but the output order changes.

Try our Duplicate Line Remover for instant deduplication in the browser.

Removing Empty Lines

Empty lines serve as paragraph separators in prose, but they’re noise in data files, logs, and lists.

const cleaned = text.split('\n')
  .filter(line => line.trim() !== '')
  .join('\n');

cleaned = '\n'.join(line for line in text.splitlines() if line.strip())

grep -v '^[[:space:]]*$' filename
# or
sed '/^\s*$/d' filename

Tip: Decide whether “empty” means truly empty ("") or whitespace-only (" "). The examples above treat whitespace-only lines as empty. If you want to keep lines with only spaces, use line !== '' instead of line.trim() !== ''.

Use our Empty Line Remover for a one-click solution.

Trimming Whitespace

Trailing whitespace causes diff noise in version control, and leading whitespace can break parsers that expect clean field values.

Trim every line

const trimmed = text.split('\n').map(line => line.trim()).join('\n');

Trim trailing only (preserve indentation)

const trimmed = text.split('\n')
  .map(line => line.replace(/\s+$/, ''))
  .join('\n');

sed 's/[[:space:]]*$//' filename

Collapse multiple spaces

const collapsed = text.replace(/ {2,}/g, ' ');

This replaces runs of 2+ spaces with a single space. It doesn’t touch tabs or newlines — only space characters.

Our Whitespace Cleaner combines all these operations with checkboxes.

Fixing Line Endings

Windows uses CRLF (\r\n), Unix/macOS uses LF (\n). Mixing them causes:

• Git diffs showing every line as changed
• Scripts failing with \r: command not found
• Parsers splitting on \n but leaving \r attached to field values

Normalize to LF

const normalized = text.replace(/\r\n/g, '\n').replace(/\r/g, '\n');

sed -i 's/\r$//' filename     # GNU sed
tr -d '\r' < input > output   # portable
dos2unix filename              # dedicated tool

Prevent the problem

Add a .gitattributes file to your repo:

* text=auto eol=lf

This tells Git to normalize line endings to LF in the repository and convert to the OS-native format on checkout.

Adding and Removing Line Numbers

Line numbers are essential for code reviews and bug reports, but they need to be stripped before the code can be used.

Adding numbers

const numbered = text.split('\n')
  .map((line, i) => `${String(i + 1).padStart(3)} | ${line}`)
  .join('\n');

nl -ba filename        # number all lines
cat -n filename        # number non-empty lines

Removing numbers

const clean = text.split('\n')
  .map(line => line.replace(/^\s*\d+\s*[|:.)\-\]]\s?/, ''))
  .join('\n');

The regex handles common formats: 1 | , 1: , 1. , 1) , 1] , and 1- .

Use our Add Line Numbers and Remove Line Numbers tools for this.

Wrapping Lines with Prefix/Suffix

Bulk-editing lines is a common task: wrapping in quotes, HTML tags, or SQL syntax.

const wrapped = text.split('\n')
  .map(line => `'${line}',`)
  .join('\n');
// "apple" → "'apple',"

sed "s/.*/'&',/" filename

Common patterns:

• SQL IN clause: prefix ', suffix ',
• HTML list: prefix <li>, suffix </li>
• CSV quoting: prefix ", suffix "
• Markdown list: prefix -

Our Add Prefix & Suffix to Lines tool handles this with separate prefix and suffix fields.

Normalizing Unicode

The same visible character can have multiple Unicode representations. The letter “é” can be:

• Precomposed (NFC): U+00E9 — one code point
• Decomposed (NFD): U+0065 + U+0301 — two code points (e + combining accent)

They look identical but fail string comparison:

'é' === 'é'  // might be false!
'é'.normalize('NFC') === 'é'.normalize('NFC')  // always true

Always normalize to NFC before comparing, searching, or storing text:

const normalized = text.normalize('NFC');

import unicodedata
normalized = unicodedata.normalize('NFC', text)

Removing Invisible Characters

Text from the web often contains invisible Unicode characters:

const clean = text.replace(/[\u200B-\u200F\u2028-\u202F\uFEFF]/g, '');

BOM tip: The byte order mark (U+FEFF) often appears at the start of files saved by Windows editors. It can break JSON parsing and shell scripts. Strip it with: text.replace(/^\uFEFF/, '').

Complete Cleaning Function

Here’s a reusable function that combines the most common operations:

function cleanText(text, options = {}) {
  const {
    trimLines = true,
    collapseSpaces = true,
    removeEmpty = false,
    removeDuplicates = false,
    normalizeLf = true,
    normalizeUnicode = true,
  } = options;

  let lines = normalizeLf
    ? text.replace(/\r\n/g, '\n').split('\n')
    : text.split(/\r?\n/);

  if (normalizeUnicode) lines = lines.map(l => l.normalize('NFC'));
  if (trimLines) lines = lines.map(l => l.trim());
  if (collapseSpaces) lines = lines.map(l => l.replace(/ {2,}/g, ' '));
  if (removeEmpty) lines = lines.filter(l => l !== '');
  if (removeDuplicates) lines = [...new Set(lines)];

  return lines.join('\n');
}

Summary

For more text techniques, see Text Processing Tips for Developers and the Regex Cheat Sheet with Examples.

Further Reading

• Unicode Normalization Forms — the Unicode standard for NFC, NFD, NFKC, NFKD
• The Great Newline Schism — why Windows and Unix use different line endings
• MDN: String.prototype.normalize() — JavaScript Unicode normalization reference