Scrapebento journal
Websites Built For Humans AI Needs A Different Layer
A website can be easy for people to use and still leave software guessing about prices, availability, and other important details. A machine-readable layer makes those details explicit through structured data, clear content boundaries, stable identifiers, documented state, and access rules. The goal is not to build a second website. It is to provide a dependable way to retrieve the same information without reconstructing it from layouts and clicks. This guide explains what that looks like, with example requests and responses for product data, pagination, and rate limits.
A customer opens a product page, selects blue, chooses size 10, and checks whether the item can be shipped to their country.
An automated service needs the same answers. It should not have to reconstruct the entire shopping experience just to find them.
The difficulty is not that websites contain no useful data. It is that the information a task needs can be spread across page content, interactive controls, and responses that only arrive after a particular action.
A machine-readable layer gives software a more direct route. It states what the information describes, which conditions apply, and how to request the next piece.
That does not necessarily require a separate website or a new database. Existing standards already offer useful building blocks: semantic HTML can identify content, JSON-LD can embed structured data in a page, and OpenAPI can describe an HTTP API’s operations and responses.
The practical change is to treat machine-readable access as a supported interface. Define what it returns, keep it consistent with the human-facing site, and test it when the underlying product changes.
What the Human Web Leaves Agents to Guess
Imagine a product page with a large price, a smaller crossed-out price, a promotional banner, and a note about member discounts.
Which number should an agent report?
The answer depends on more than finding a currency symbol. It needs to distinguish the current selling price from the previous price, determine whether membership is required, and establish whether the selected product variant changes the amount.
Now add a delivery-country selector and a stock message that updates after the customer chooses a size. A client that only retrieves the initial HTML has not necessarily received the information needed to answer the question.
These are different problems, but they share a cause: the relationship between the values is not explicit enough for the task.
HTML already has meaningful elements. An<article>can identify a self-contained piece of content, while<nav>identifies a major navigation section. The problem is not that the web lacks semantics altogether. Those semantics simply do not explain every product-specific rule, pricing condition, or inventory state.
Consider a page that displays “In stock” before a size is selected. That message might mean at least one variant is available. It does not necessarily answer whether the blue version in size 10 can be purchased.
A useful machine-readable response should make that distinction directly. The client should not need to infer it from badge colors, disabled buttons, or the order in which options appear.
Why This Matters Now
The problem becomes more noticeable when a task moves from an occasional manual check to a recurring automated job.
Checking one price by hand may involve a few clicks. Checking hundreds of prices repeatedly turns every hidden condition into something the collection process must handle.
Suppose a price-monitoring job reads the same product page every hour. If its extraction logic depends on a particular layout, a redesign creates another dependency to test. If it consumes a documented price field, the presentation can change without necessarily changing that interface.
That is the value of separating presentation from the data contract.
For a site owner, ask which facts should remain accessible even when the page design changes. For a scraper developer, ask whether there is a supported structured interface before rebuilding the browser workflow.
A dedicated API is one option, not an automatic requirement. JSON-LD can be embedded directly in HTML, so structured information can travel with the existing page. Where clients need filters, pagination, or more detailed responses, a documented API may be a better fit.
Whichever approach you choose, avoid maintaining two competing versions of the truth. The visible price and the machine-readable price should come from the same underlying rules, with any differences in market, account, or product variant made explicit.
What a Machine-Readable Layer Actually Contains
Five components provide a useful starting point. They cover not just the values an agent receives, but also what those values mean and how the client should use them.
1. Structured data that names the facts
A value such as 79.00 is not enough on its own. Is it a price, a discount, or a shipping threshold? Which currency applies? What item does it describe?
Structured data attaches meaning to the value. Existing vocabularies include properties for price, currency, and availability. A custom API can also expose these details, provided its field definitions are documented.
Consider a request for one product variant in a specific market:
GET /api/v1/variants/shoe-104-blue-10?market=US HTTP/1.1
Host: shop.example
Accept: application/json
Authorization: Bearer <redacted>A useful response could look like this:
HTTP/1.1 200 OK
Content-Type: application/json
{
"product_id": "shoe-104",
"variant_id": "shoe-104-blue-10",
"name": "Road running shoe",
"url": "https://shop.example/products/road-running-shoe",
"options": {
"color": "blue",
"size": "10",
"size_system": "US"
},
"market": "US",
"price": {
"amount": "79.00",
"currency": "USD",
"unit": "pair",
"tax_included": false,
"shipping_included": false
},
"availability": "in_stock",
"observed_at": "2026-10-27T04:00:00Z"
}The endpoints, values, and JSON fields in these examples are illustrative. They describe a fictional API contract, not a universal standard. Unrelated HTTP headers are omitted.
This response answers a specific question: the blue shoe in US size 10 is listed as available in the US market at $79 per pair, excluding tax and shipping, at the stated observation time.
The client does not have to decide which price belongs to which size. The response makes that relationship explicit.
It also avoids pretending to answer more than it does. Availability at an observation time is not a stock reservation, and a price excluding shipping is not the final checkout total.
2. Explicit content boundaries
Not every task needs an API. Sometimes the most useful improvement is making the existing page easier to interpret.
Use semantic markup to distinguish the main content from navigation and supporting material. The<main>element identifies the page’s dominant content, while elements such as<article>,<nav>, and<aside>describe different roles within the document.
For a documentation page, keep the instructions clearly separated from related articles, promotional messages, and navigation links. For a product page, distinguish the product description from recommendations for other items.
These boundaries do not guarantee perfect extraction. They give a client better evidence about which content belongs to the requested page and which content is merely nearby.
Treat this as an improvement to the existing website, not an inferior substitute for building something new.
3. Stable URLs and identifiers
Give clients a dependable way to refer to the same entity over time.
In the example above, product_id identifies the shoe model, while variant_id identifies the particular color and size. The page URL provides a human-readable reference.
Those identifiers serve different purposes. A product title may change. A page may move. Neither should automatically make the underlying item appear to be a completely new product in a client’s database.
As an interface design rule, keep entity identifiers stable, preserve useful page addresses where possible, and document how moved or removed resources are handled.
This matters when comparing observations. A monitoring system needs to distinguish “the price changed” from “this is a different item with a similar name.”
Do not make clients use display text as the primary identity when you can provide a stable identifier instead.
4. Machine-readable state
A useful interface should describe the conditions behind its answer.
For inventory, that may include the variant and market. For pricing, it may include billing frequency, quantity, or account eligibility. For search results, it includes the filters applied and whether more results remain.
Pagination is a good example. A “Load more” button makes sense to a visitor, but a data client needs a documented way to continue.
A request might look like this:
GET /api/v1/products?category=running-shoes&market=US&limit=2 HTTP/1.1
Host: shop.example
Accept: application/json
Authorization: Bearer <redacted>The response can declare both the current selection and the next request:
HTTP/1.1 200 OK
Content-Type: application/json
{
"items": [
{
"id": "shoe-104",
"name": "Road running shoe"
},
{
"id": "shoe-205",
"name": "Trail running shoe"
}
],
"filters": {
"category": "running-shoes",
"market": "US"
},
"next": "/api/v1/products?category=running-shoes&market=US&limit=2&cursor=cursor_2"
}Under this example’s contract, the client follows next to continue. The final page returns"next": null. The filters remain part of the request, rather than disappearing when the client moves to another page.
Freshness needs the same clarity. Define whether a timestamp records when the source was checked, when a value changed, or when the response was generated. Those are not interchangeable.
In the earlier product example, observed_at means when the represented price and availability were checked. Generating a new response should not automatically make an old observation look fresh.
Also distinguish missing information from a negative result. Design the interface so “inventory could not be checked” does not become “out of stock.” An explicit unknown state is more useful than a confident answer the system cannot support.
5. Published access rules and predictable errors
A machine-readable interface should explain how to use it.
Document authentication requirements, supported operations, request limits, and expected error responses. OpenAPI can describe operations, parameters, response structures, and security requirements in a machine-readable document.
Keep crawler instructions separate from access control. robots.txt communicates crawling rules, but its standard explicitly states that those rules are not authorization. Protect restricted data through server-side access controls rather than relying on a published instruction alone.
Errors are part of the interface too. A 429 Too Many Requests response can explain that a limit was exceeded and may include a Retry-After header.
For example, another request to the product endpoint might receive:
GET /api/v1/variants/shoe-104-blue-10?market=US HTTP/1.1
Host: shop.example
Accept: application/json
Authorization: Bearer <redacted>HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 30
{
"error": {
"code": "rate_limit_exceeded",
"message": "This API token has reached its request limit.",
"scope": "api_token"
}
}Here, the client is asked to wait at least 30 seconds before another attempt. The header communicates the waiting time; the example’s error body identifies the affected quota scope.
If several workers share that token, they should coordinate their cooldown rather than let each worker continue independently.
That is a much more useful contract than returning a generic HTML error page and leaving the client to guess what happened.
What Remains Uncertain
The useful question is not whether structured formats exist. They do. The decisions are how much information to expose, how clients discover it, and how the interface stays aligned with the website.
Three questions deserve attention.
How much of the existing site can be reused?
Start with the task. An article may only need clearer HTML and structured metadata. A product comparison may need explicit pricing conditions and pagination. A personalized workflow may require an authenticated interface.
Do not build a separate system simply because the audience includes agents. Build the smallest supported interface that answers the intended questions reliably.
How should access rules be enforced?
Document what clients are allowed to request, but keep enforcement in the service. A published description and a working permission check solve different problems.
Also distinguish reading information from taking action. An endpoint that returns availability should not silently imply permission to reserve stock or place an order.
Could the new interface create lock-in?
It could, particularly if essential data can only be retrieved through a proprietary runtime or undocumented behavior.
Prefer formats your team can inspect, export, and validate. Keep the underlying data model separate from any one client integration, and make breaking changes deliberate rather than accidental.
The goal is not to predict which tooling will remain popular. It is to avoid making the usefulness of your information depend entirely on one tool.
FAQ
What is the difference between the human web and a machine-readable layer?
A human-facing page presents information through text, layout, and interaction. A machine-readable layer makes the entities, values, and conditions explicit enough for software to process.
The two can coexist in the same page. JSON-LD, for example, can be embedded in HTML rather than served through a separate website.
Why does this matter for readers searching for trustworthy answers?
It reduces the amount of interpretation needed to answer a question.
A response that identifies the exact product variant, market, currency, and observation time is easier to check than an extracted number with no context.
Structure alone does not make information accurate. The source still needs to maintain it, and the client still needs to check whether it is current and relevant to the question.
What should teams watch next?
Watch whether clients can use the information you already publish without relying on undocumented behavior.
Can they retrieve the right variant? Follow pagination? Distinguish unknown availability from unavailable stock? Recognize a rate limit? Trace the result back to its source?
Those are useful acceptance tests for any format or integration you adopt. A new protocol is worth evaluating when it helps with those tasks, not merely because it is described as agent-ready.
Conclusion
A readable page and a dependable data interface serve different needs. You do not have to choose between them.
Start with one task your site should support reliably. Define the information needed to complete it, make the relevant conditions explicit, and return clear errors when the answer is unavailable.
Keep identifiers stable. Explain timestamps. Document access. Test that the machine-readable response agrees with the human-facing page under the same conditions.
The aim is straightforward: when an agent asks for a price, availability, or status, it should receive an answer it can use, along with enough context to know what that answer means.