AI Crawler
A bot that reads websites on behalf of an AI company, either to build an index or to answer a live question. They arrive under their own names and obey robots.txt, so you can decide which are welcome.
Related terms
Training Data
The text, images or code a model learned from. It shapes everything the model knows and every bias it carries, and it is why what your website said two years ago may still be what an assistant repeats.
Google-Extended
The robots.txt setting that controls whether Google may use your content for Gemini and AI Overviews. It is separate from ordinary Google Search, so you can allow one and refuse the other.
Fine-Tuning
Taking a general model and training it further on your own examples so it follows your tone or handles your particular task. Useful when prompting alone keeps producing nearly-right answers.
Explainability
How well a system can show why it reached a decision. It matters most where the decision affects somebody — a loan, a diagnosis, a job application — and “the model said so” is not an answer.
Entity SEO
Optimising to be recognised as a specific, real thing rather than for particular keywords. In practice it means consistent naming, structured data, and being mentioned in enough other places to be corroborated.
Knowledge Graph
A map of things and how they relate — this company, its founder, its address, its services. Search engines and AI systems use one to know that two mentions of a name refer to the same business.