Signals in motion: Using multimodal AI to interpret brand engagement in real time

Today’s digital experiences are more layered, emotional, and fast moving than ever before. Marketers and analysts can no longer afford to rely solely on clickstream or time-on-page data to understand what’s truly resonating. Customers interact through multiple modes – tapping, scrolling, hovering, watching and each of these actions carries subtle intent.

At eClerx, we’re helping brands transform customer experiences by monitoring these interactions and allowing our clients to scale digital commerce through real time personalization.

Beyond the click: Why single-source clickstream data isn’t enough anymore

Clickstream data is a great starting point—it shows where users click, which pages they visit, and how long they stay. But it’s only one piece of a much bigger puzzle.

To truly understand your audience, you need to go deeper. Here’s why:

1. Discover the “why” behind the click: A click alone doesn’t reveal intent. Was the user confident and engaged, or confused and uncertain? Did they click by choice or by accident?
2. See the moments between clicks: Scroll patterns, cursor movement, hover pauses, and even “rage clicks” reveal emotion and engagement levels that clickstream alone can’t capture.
3. Replace assumptions with clarity: When data lacks depth, it’s tempting to fill the gaps with guesswork. Richer behavioral insights prevent bias, ensuring decisions are grounded in reality—not assumptions.

When you combine clickstream data with deeper behavioral signals, you move from simply tracking actions to truly understanding experiences – and that’s where transformation happens.

For example: eClerx partnered with a banking organization to help reduce the 65% bounce rate on its account comparison page. Click data alone didn’t reveal the issue. When we added scroll depth and rage click tracking, it became clear users were repeatedly interacting with a non-clickable element mid-page, then leaving. Once CTAs were repositioned and clarified with sub-texts, the bounce rate dropped and engagement improved significantly.

Signals in motion: Real-time experience mapping

What if your analytics could detect frustration before a user even bounces? Multimodal AI lets you build real-time experience maps that combine scroll depth, dwell time, video watch behavior, cursor velocity, and eye tracking. These signals feed into models that predict drop-off likelihood, interest level, or conversion probability – while the user is still active on-site.

This isn’t just attribution, it’s intervention.

Imagine being able to suppress exit modals for high-intent users, but proactively offer support to those showing signs of confusion.

A durable goods retailer used GA4 scroll depth and video engagement data to optimize their product category landing page. By identifying key sections where users paused or dropped off, they restructured content flow and repositioned video explainers higher on the page. The result? A 17% drop in bounce rate and a 22% lift in click-throughs to product detail pages.

Predictive CX models: A new marketing arsenal turning insights to action

Predictive modelling becomes more accurate when data types contextualize each other. For example, a user scrolling deeply but never clicking could signal informational intent, while video engagement paired with gaze analysis might reveal emotional resonance with a product.

Multimodal AI isn’t just interpretive—it’s predictive. The goal is to move from “what happened” to “what will happen” and “what should we do about it?” If a user has a history of fast-scrolling, rage clicking, and bouncing within 30 seconds, AI can flag them in real time and trigger smoother experiences—be it simplified UI, or escalated chat support

We have seen multiple use cases across user journey where businesses can implement these models and make multi-million-dollar ROI

• Churn predictors: Users who repeatedly pause videos and revisit pricing pages
• Conversion likelihood scores: Based on depth of interaction across modalities
• Personalization engines: Adapt UI, content, offers in real time

From integration to impact: Putting multimodal AI to work

Typically, a multimodal engine aimed at real time personalization works by unifying all behavioral data in BigQuery or a CDP, then enriching it with session replays or chat logs.

From there, machine-learning models are trained to segment users based on engagement quality and trigger timely interventions—like personalizing product banners or reordering content blocks or triggering exit modals based on intent prediction.

Operationalizing multimodal AI

With the right setup, multimodal AI stops just being a dashboard feature, enabling marketers to shift from reactive reporting to proactive personalization, offering support or content dynamically, based on micro-signals.

For example, a B2C client used scroll + click + chatbot data to identify and recover 18% of high-exit users from the cart page.

To get started, a business can begin by identifying clear use cases before scaling up for optimal ROI, like reducing cart abandonment or improving lead form completions.

The next step would be to feed in a mix of engagement signals—such as video completion rates, scroll velocity, and chatbot queries.

Finally, apply the AI outputs to personalize journeys.

Why this matters now: The competitive edge

Experience has overtaken price and product as the key battleground. AI-driven engagement insights are helping brands deliver emotionally resonant and in-the-moment responses.

Leading brands are embedding AI into decision systems—fueling faster learning cycles and better personalization. Waiting means falling behind.

And with the end of third-party cookies, deeper interpretation of on-site behavior—like gaze, scroll depth, or video replays—will be the holy grail for intent modeling.

Making it actionable: Cross-team buy-in for smarter engagement

To make multimodal AI actionable, cross-functional collaboration is key. Product, marketing, and analytics teams must work together to:  

1. Align on key engagement signals
2. Define what ‘good’ engagement looks like 
3. Build real-time triggers and feedback loops

If you’re still relying on single-source engagement metrics, you’re missing the full picture. Multimodal AI is not about replacing marketers—it is about empowering them with better foresight and ability to make smarter decisions faster.

Ready to turn micro-signals into macro impact?

Let’s chat. Book a free signal health check consult to see how we can apply these frameworks and get the most out of your investment.

Book a free consult

Lets talk