Large language models (LLMs) excel at reasoning across text and other modalities, yet they struggle with continuous temporal signals such as time-series data. This limitation is critical, as many real-world domains—from healthcare to finance—depend on reasoning about evolving trajectories.
Recent work by our group, in collaboration with Stanford and Google researchers, has developed and validated a new multimodal LLM architecture, OpenTSLM, which extends LLaMA and Gemma with time-series as an additional modality. Using soft prompting and cross-attention (Flamingo-style), these Text–Time-Series LLMs were shown to reason effectively over longitudinal health data, achieving state-of-the-art performance in tasks like ECG-based question answering, activity recognition, and sleep stage detection.
Building on this foundation, we aim to extend these architectures beyond healthcare. We hypothesize that OpenTSLM models can be adapted to reason about time-series trajectories from new text input across a wide range of domains—explaining why certain events may shift temporal patterns, what alternative scenarios are plausible, and how new signals relate to historical dynamics. Once such reasoning capabilities are established, the same models could also predict future trajectories, combining causal and contextual insights with temporal forecasting.
Beyond healthcare, we see broad potential applications:
Finance: Explaining and forecasting stock movements and market anomalies from news and reports.
Data sources: GDELT, Bloomberg, Refinitiv, FactSet, Yahoo Finance, SEC filings.
Robotics: We are envisioning Text-Time Series Vision Language Models (TS-VLMs) to effectively combine ****sensor data (time-series), video and text to enable Instruction following over time, i.e., executing multi-step tasks from a single natural-language command.
Data sources: RT‑X Datasets, SmolVLA
Industrial Monitoring: Diagnosing system behavior and anticipating failures by combining machine sensor streams with technician logs.
Data sources: NASA turbofan engine datasets, industrial IoT sensor streams, Kaggle predictive maintenance benchmarks.
Supply Chains: Reasoning about demand shocks from policy, logistics, or environmental events.
Data sources: UN Comtrade, World Bank logistics indices, maritime AIS data, corporate shipment datasets.
Elections & Public Opinion: Interpreting news cycles and social signals to explain and project polling trends.
Data sources: FiveThirtyEight, Ipsos, Pew Research, Twitter/X streams, GDELT event data.
Climate & Environment: Reasoning about the impact of extreme weather events on energy consumption or carbon emissions.
Data sources: NOAA, ECMWF/ERA5 reanalysis, Copernicus Climate Data Store, IEA emissions datasets.
Macroeconomics: Linking central bank statements or policy news to shifts in inflation, interest rates, or unemployment.
Data sources: Federal Reserve FRED database, IMF, ECB, BIS, OECD economic indicators.
Energy Markets: Explaining how geopolitical decisions (e.g., OPEC actions) influence oil and gas trajectories.
Data sources: EIA, IEA, OPEC reports, Platts, Reuters energy desk.
Retail & Consumer Behavior: Reasoning about how marketing campaigns, reviews, or regulations drive shifts in sales and demand.
Data sources: Nielsen retail scanner data, Amazon reviews, Google Trends, Kaggle e-commerce datasets.
Our objective is to develop and validate novel Time Series Language Model (TSLM) architectures beyond healthcare, demonstrating their ability to reason over and, where appropriate, predict temporal data streams. If successful, this will enable a new class of multimodal reasoning agents that explain not only what is happening, but also why systems evolve as they do—bridging text and time-series signals in critical domains such as finance, healthcare, and robotics.