# `LeXtract.Error.Processing.Tokenization`
[🔗](https://github.com/YgorCastor/lextract.git/blob/main/lib/lextract/error/processing/tokenization.ex#L1)

Error for tokenization failures.

Raised when text cannot be tokenized, typically due to encoding
issues or invalid character sequences.

## Examples

    iex> error = LeXtract.Error.Processing.Tokenization.exception(
    ...>   reason: "encoding failed"
    ...> )
    iex> Exception.message(error)
    "Tokenization failed: encoding failed"

    iex> error = LeXtract.Error.Processing.Tokenization.exception(
    ...>   reason: "invalid UTF-8 sequence",
    ...>   text_sample: "test \xFF invalid"
    ...> )
    iex> String.contains?(Exception.message(error), "invalid UTF-8")
    true

# `t`

```elixir
@type t() :: %LeXtract.Error.Processing.Tokenization{
  __exception__: term(),
  bread_crumbs: term(),
  class: term(),
  path: term(),
  reason: String.t(),
  splode: term(),
  stacktrace: term(),
  text_sample: String.t() | nil,
  vars: term()
}
```

# `exception`

```elixir
@spec exception(opts :: Keyword.t()) :: %LeXtract.Error.Processing.Tokenization{
  __exception__: term(),
  bread_crumbs: term(),
  class: term(),
  path: term(),
  reason: term(),
  splode: term(),
  stacktrace: term(),
  text_sample: term(),
  vars: term()
}
```

Create an `Elixir.LeXtract.Error.Processing.Tokenization` without raising it.

## Keys

- :reason
- :text_sample

# `message`

```elixir
@spec message(t()) :: String.t()
```

Formats the error message for tokenization failures.

---

*Consult [api-reference.md](api-reference.md) for complete listing*