AI Chatbot Features: How It Works Under the Hood

AI chatbot features like automatic learning are what set modern chatbots apart from traditional rule-based bots. When you connect your website to Boei, a sophisticated process transforms your content into something an AI can understand and search through. These features are perfect for ecommerce websites looking to automate support. Here's what happens under the hood:

1. Content Extraction & Cleaning

Our crawler visits every page of your website and extracts the meaningful content. This isn't just copying HTML — we intelligently remove:

• Navigation menus, footers, and sidebars
• Advertisements, forms, scripts, and styles
• Comments, pagination, and breadcrumbs
• Duplicate content and boilerplate text

What remains is the actual content your visitors care about: product descriptions, FAQs, policies, articles, and documentation. We also extract metadata like page titles, H1 headings, and descriptions.

2. Intelligent Chunking

Long pages are split into smaller, semantic chunks that preserve meaning. Our chunking algorithm:

• Preserves code blocks and tables as complete units
• Respects paragraph and section boundaries
• Keeps related information together
• Optimizes chunk size for retrieval accuracy

3. Creating Embeddings

Embeddings are numerical representations of text that capture semantic meaning. Think of them as coordinates in a multi-dimensional space where similar concepts are close together. The sentence "What's your return policy?" and "Can I send items back?" have different words but nearly identical embeddings because they mean the same thing.

We use state-of-the-art embedding models to convert each chunk of your content into these numerical vectors — typically 1,536 dimensions that capture nuance, context, and meaning.

4. Vector Storage

These embeddings are stored in a specialized vector database optimized for similarity search. Unlike traditional databases that match exact keywords, vector databases find content based on meaning. This is why the chatbot understands questions even when visitors don't use the exact words from your website. See these features in action: 18 chatbot use cases. All features included in our simple pricing. Easy setup on WordPress.

One of the most important AI chatbot features is intelligent question handling. When a visitor types a question, a multi-stage process ensures they get the most accurate answer possible:

Step 1: Query Pre-Processing

The raw question is analyzed and enhanced before searching. This includes:

• Identifying the intent behind the question
• Expanding abbreviations and fixing typos
• Generating alternative phrasings for better matching
• Extracting key entities (product names, features, etc.)

Step 2: Hybrid Search

We don't rely on just one search method. Instead, we combine:

• Vector search — Find content with similar meaning using embeddings
• BM25 keyword search — Traditional text matching for exact terms

This hybrid approach catches both semantic matches ("refund" matches "return policy") and exact matches (specific product names, model numbers).

Step 3: Re-Ranking

Results from both search methods are combined and re-ranked based on:

• Content type relevance (FAQs weighted higher for questions)
• Page importance and freshness
• Match quality across both search methods
• Custom rules (e.g., pricing pages for price questions)

Step 4: Answer Generation

The top-ranked content chunks are sent to the LLM (GPT-5 or Claude) along with the original question. The AI synthesizes an answer using only the provided content — never its general training data. This is called Retrieval-Augmented Generation (RAG).

Step 5: Source Attribution

Every answer includes links to the source pages used. Visitors can verify information themselves, and you can see exactly what content informed each response.

Flexible knowledge sources are among the most powerful AI chatbot features available. Your AI chatbot can learn from multiple types of content, all processed through the same embedding pipeline:

Website Content

• Sitemap scraping — Parse XML sitemaps to discover all URLs automatically
• Domain crawling — Intelligent crawler discovers pages even without a sitemap
• JavaScript support — Fetch dynamic content from SPAs and JS-rendered pages using our custom scraper

Uploaded Documents

• PDF files — Product manuals, guides, policies, brochures
• Word documents — Internal documentation, procedures
• Excel spreadsheets — Product catalogs, specifications, pricing
• PowerPoint presentations — Training materials, sales decks
• Text files — Any plain text content

Manual Content

• FAQ management — Add question/answer pairs directly
• Custom text blocks — Paste any content you want the bot to know
• Custom rules — Define special handling for specific content types

All sources are combined into a unified knowledge base. The bot seamlessly searches across everything when answering questions.