Agent Monitor (CDN-Based Data)

See which AI crawlers access your website and how their activity changes over time.

Agent Monitor helps you understand how AI crawlers interact with your website. After you connect your CDN, Limy analyzes CDN request data to identify known AI bots, the pages they request, request volume, and activity patterns over time.

CDN data is important because crawler activity may not appear in standard website analytics. Connecting your CDN provides the request data Limy uses to power Agent Monitor.

A crawler request shows that a bot accessed a URL. It does not confirm that the page was indexed or used in an AI-generated answer.

What Agent Monitor Shows

After your CDN integration is connected, Agent Monitor can show:

• Which AI agents and crawlers access your website

• Which pages they request

• Request volume by agent and page

• Changes in activity over time

• Important pages receiving little or no observed crawler activity

This helps you understand how AI agents interact with your website and identify activity that may need further investigation.

How to Set Up Agent Monitor

1

Identify your CDN Provider

This information is typically available from your DevOps or infrastructure team. Common providers include:

• Cloudflare

• Akamai

• Fastly

• Netlify

• CloudFront

2

Follow the CDN integration instructions.

3

Confirm the CDN integration is completed

After the integration is complete

Open your Limy account and select Agent Monitor under Monitoring. Data will appear as agents and crawlers make requests to your website.

FAQs

Does Agent Monitor track users or personal data?

Agent Monitor tracks automated crawlers and agents rather than human user sessions. However, Limy may process IP addresses as a data processor when the CDN integration is used.

Does this integration affect site performance?

No. The integration is read-only and does not impact site speed or availability.

Do I need engineering support to set this up?

Yes. CDN access is typically managed by DevOps or IT teams.

How long does it take to see data?

Data appears as AI crawlers revisit your site. Timing depends on crawl frequency and AI platform behavior.

What if I don’t see important pages being crawled?

This may indicate crawl-ability or accessibility issues, which can be addressed through Recommendations or technical adjustments.

---

Related Content

AI Traffic MonitorMonitoringHow Limy Generates PromptsAttribution

Still have questions? We’re here to help. Contact us at [e-mail] - we’d be happy to assist.