While exploring foundation models for time series, I found it impossible to understand them without first grasping transformers. Rather than treating these models as black boxes, I traced the concepts backward—from foundation models to transformers, and from transformers to self-attention. What made this journey intriguing is that, although transformers were originally designed for language, the core idea transfers naturally to time series. The two modalities differ greatly, yet they share something essential: both are sequences, and in both, order changes meaning.
Consider language: dog bites man tells a completely different story than man bites dog.
Time series follow the same principle. A temperature of 30°C yesterday and 20°C today conveys a different narrative than 20°C yesterday and 30°C today. The sequence itself—its temporal ordering—determines the interpretation, whether we’re forecasting energy demand, detecting anomalies in sensor data, or predicting stock prices.
The Rise of Transformers in Time Series
Since their introduction in 2017, transformers have dominated natural language processing and have increasingly become the backbone of modern time series forecasting. By 2026, architectures such as Temporal Fusion Transformers and PatchTST have pushed the boundaries of what’s possible with sequence data, achieving state-of-the-art results across finance, healthcare, and industrial IoT applications.
Yet, there’s an inherent challenge when using transformers for time series: the architecture itself is permutation-invariant. Without any awareness of order, self-attention treats the input as a set, not a sequence.
Self-Attention Without Order
At the heart of a transformer lies self-attention, which computes weighted relationships between every pair of elements in the input. This mechanism is powerful—it captures dependencies regardless of distance, making it highly effective for both language and time series.
But if we feed a transformer a time series without any positional information, the model sees all time steps as equal. It has no built-in way to know that the value at time t-1 is more relevant to time t than a value far in the past. This omission destroys the very signal that makes time series meaningful.
How Positional Encoding Restores Order
Positional encoding addresses this by injecting information about each element’s position into the input representation. In the vanilla transformer, sinusoidal encodings—using sine and cosine functions of varying frequencies—are added to the token embeddings. For time series, these encodings allow the self-attention mechanism to distinguish between recent observations and older ones, effectivelyreintroducing temporal structure.
Over the years, researchers have developed more sophisticated variants, including learned positional embeddings, relative positional encodings, and time-aware decoders. As of 2026, newer approaches like rotary position embeddings (RoPE) have gained popularity for their ability to handle longer sequences and improve generalization, which is particularly useful in multivariate time series with thousands of steps.
A Visual Guide to Positional Encoding
To see why positional encoding matters, consider a simple sine wave dataset. If we plot two identical subsequences—one shifted in time—and feed them to a transformer without positional encoding, the model produces identical attention scores, rendering it unable to differentiate between the two. With positional encoding, however, the model assigns distinct attention patterns, enabling it to recognize that the second subsequence occurs later and should be treated accordingly.
This visual intuition carries over to real-world applications. For instance, in energy load forecasting, the pattern of electricity consumption on a Tuesday morning differs from a Sunday afternoon, even if the raw values were exactly the same. Positional encoding lets the transformer capture such context.
Beyond the Basics: Why This Matters for Practitioners
Understanding positional encoding isn’t just an academic exercise. It profoundly affects model design and performance. Choosing the right encoding scheme can mean the difference between a model that fails to capture seasonal patterns and one that achieves top-tier accuracy. For practitioners entering the time series space in 2026, this knowledge is essential—no longer optional.
Moreover, it clarifies why certain models, like Informer or Autoformer, have introduced mechanism modifications specifically to preserve temporal dependencies. Without positional encoding, even the most advanced attention variants would remain, in essence, order-insensitive.
Final Thoughts
The journey from language to time series reveals a deep parallel: both domains hinge on sequence order, and both require transformers to be made order-aware through positional encoding. Whether you’re building forecasting pipelines or anomaly detection systems, appreciating this fundamental mechanism will empower you to move beyond black-box usage and design models that genuinely respect the temporal nature of your data.
As the field progresses, positional encoding continues to evolve—becoming more expressive, more efficient, and more attuned to the complexities of time. For anyone serious about transformers for time series, understanding this glass bead game is not just a step; it’s the essence of the work.