# Dynamic sites

A dynamic site stores data. Start a new one from `luan/examples/blog`, the skeleton, and look at `chat` for anything blog doesn't cover, above all users and logins. Clone chat for any dynamic site.

From blog, a new site keeps `serve.sh` (unchanged: it serves on the default port), `src/`, and the admin pages in `src/private/tools/`. It drops the push scripts and `serve_and_backup.luan`, which are only for Franklin's own testing.

## Files

```
src/
	lib/Db.luan         the database
	lib/Post.luan       a module per kind of data
	lib/Utils.luan      general helpers, including check_referer()
	lib/Shared.luan     what the site's pages share
	site.js             ajax(), ajaxForm() and error reporting
	error_log.js.luan   logs browser errors on the server
	X.html.luan         pages
	X.js.luan           AJAX handlers
	private/tools/      admin pages, password protected when hosted
	private/local/      the database, never pushed
```

## The database

The database is a Lucene index (see [library/lucene/Lucene.md](library/lucene/Lucene.md)), opened in `lib/Db.luan`:

```lua
local dir = uri("site:/private/local/lucene")
local Db = Lucene.index( dir, {
	default_type = Lucene.type.english
	default_fields = {"subject","content"}
	log_dir = uri("site:/private/local/lucene_log")
} )

Db.indexed_fields.post_date = Lucene.type.long

function Db.not_in_transaction()
	logger.error(new_error("not in transaction"))
end

Db.restore_from_log()

Db.update{
	[1] = function()
		...
	end
}

if Http.is_serving then
	Thread.schedule( Db.check, { delay=0, repeating_delay=Time.period{hours=1} } )
end

return Db
```

- It lives under `private/local/`, so pushes to hosting never touch it.
- `log_dir` keeps a log of every change. `restore_from_log()` rebuilds the index from the log if the index was lost, and `check()`, scheduled hourly, checks that index and log agree.
- `Db.update` runs database migrations, numbered steps that each run once.
- `Db.not_in_transaction` logs an error when something saves outside a transaction.
- `Db.check` is scheduled only `if Http.is_serving`, so a command-line script that loads `Db` doesn't start a repeating thread.
- Every kind of document shares the one index, kept apart by a `type` field. `Db.indexed_fields` applies to the whole index, so indexed field names are prefixed with the type: `post_date`, `user_email`, `chat_user_ids`. Fields that aren't indexed, like `content`, aren't prefixed.
- Ids come from the index. A new index gives id 1 to the `type:version` document that `Db.update` keeps, so a lookup by id must check the type of what it finds.
- Store ids and dates in indexed fields as longs. Chat's `to_doc` converts them with `Number.long`: `post_chat_id = long(post.chat_id)`.

## A model module

A module per kind of data, like `lib/Post.luan`:

```lua
local Post = {}

local function from_doc(doc)
	doc.type == "post" or error "wrong type"
	return Post.new {
		id = doc.id
		subject = doc.subject
		content = doc.content
		date = doc.date
	}
end

local function to_doc(post)
	return {
		type = "post"
		id = post.id
		subject = post.subject or error()
		content = post.content or error()
		date = post.date or error()
	}
end

function Post.new(post)
	post.date = post.date or now()

	function post.save()
		local doc = to_doc(post)
		Db.save(doc)
		post.id = doc.id
	end

	function post.reload()
		return Post.get_by_id(post.id) or error(post.id)
	end

	function post.delete()
		Db.delete("id:"..post.id)
	end

	return post
end

function Post.get_by_id(id)
	local doc = db_get_document("id:"..id)
	return doc and doc.type=="post" and from_doc(doc) or nil
end

return Post
```

- `to_doc` checks the required fields with `or error()`, so a save with a missing field fails at once.
- Lookups return `nil` for a missing id or a document of another type.
- Changes go through methods on an object found by a lookup, so access checks can live on the object (see Security).
- Newest first is `Db.search("type:post",1,n,{sort="id desc"})`, since ids only increase.
- Modules that use each other require each other inside the functions that use them: `post.get_chat()` does `local Chat = require "site:/lib/Chat.luan"`.

## Transactions

Every change runs inside `Db.run_in_transaction(fn)`, deletes included. It holds the index's write lock while `fn` runs, so changes from different requests never interleave, and rolls everything back if `fn` raises an error. A nested call joins the outer transaction.

To change a stored object, get it inside the transaction: look it up there, or reload it there and use the copy.

```lua
run_in_transaction( function()
	local post = Post.get_by_id(post_id) or error "post not found"
	post.subject = parameters.subject or error()
	post.save()
end )
```

Deleting works the same way: blog's `delete.js` looks the post up and deletes it inside one transaction. Only saves outside a transaction are caught by `Db.not_in_transaction`; a delete outside one isn't, so don't rely on the log to catch it.

An object found before the transaction may be stale, and saving it would overwrite another request's change. `reload()` returns a new object, so a bare `post.reload()` does nothing; write `post = post.reload()`.

## Forms and AJAX

- A page `X.html.luan` shows a form, and the form sends itself to a handler `X.js.luan` with `ajaxForm` from `site.js`:
	```html
	<form onsubmit="ajaxForm('/new.js',this)" action="javascript:">
	```
- `ajax(url,postData)` in `site.js` sends a request with `XMLHttpRequest` (POST if there is data, otherwise GET) and `eval`s the JavaScript that comes back. `ajaxForm(url,form)` sends a form's named fields through `ajax`.
- A handler does its work and returns JavaScript:
	```lua
	return function()
		check_referer()
		local parameters = Http.request.parameters
		run_in_transaction( function()
			local post = Post.new{
				subject = parameters.subject or error()
				content = parameters.content or error()
			}
			post.save()
		end )
		Io.stdout = Http.response.text_writer()
	%>
		location = '/';
	<%
	end
	```
	A handler can also report a problem without reloading the page, by returning JavaScript that shows a message, as chat's `login.js` does for "Login failed".
- A failed request, and any JavaScript error caught by `window.onerror`, is posted to `/error_log.js`, which logs it on the server. So browser errors show up in the server log. Every site with nontrivial JavaScript has this.
- Use `XMLHttpRequest` and explicit callbacks, never `fetch`, promises or `async`/`await`.
- A dialog that edits a value holds a form. Chat's `account.html.luan` does it like this: `editX()` fills the form through `form.elements.name.value` (which also sets a radio group) and calls `dialog.showModal()`. The form's `onsubmit="saveX(this)"` calls `closeModal(form)`, updates the page, and sends the form with `ajaxForm`. Field names match the parameters the handler reads.
- For confirmations, chat's `confirm2(message,fn)` in `site.js` creates one `<dialog confirm2>` on first use, with `autofocus` on Cancel, and calls `fn` on OK.

## Live updates

Chat pushes JavaScript from the server to open pages with server-sent events. The page opens `new EventSource('/user/<id>')` (or `/chat/<id>`), and the server calls `Http.push(url,js)`, which sends the JavaScript to every page listening on that URL; the page evaluates it. The page also calls `heartbeat.js` every 10 seconds, which records that the user is online and catches up on anything missed.

## Users and logins

Follow chat's login. It has no account signup and no password the user chooses.

- `register.html` asks only for an email. `register.js` creates the user if needed, with a generated password, and mails a login link, `login.html?email=...&password=...`. Inviting someone sends the same link.
- `login.html` has email and password fields filled from the URL, so the mailed link opens a filled-in form and the user just presses Submit. Values from the URL are escaped with `Html.encode`.
- The form posts through `ajaxForm` to `login.js`. On failure it returns JavaScript that puts "Login failed" in `<p failed>`. On success, `user.login()` sets persistent `user` and `password` cookies, the password one `HttpOnly` (JavaScript never needs it), and the JavaScript sets `location` to the next page.
- `User.current()` compares the cookies with the database once per request, caching the result for that request with `Luan.set_local_only`.
- Logout is done on the server (`logout.js`), because JavaScript can't remove an `HttpOnly` cookie.
- Show password: a `<div password>` with `position: relative` holds the password input and two absolutely positioned icons, `<img show>` and `<img hide>`, with `title` text. CSS picks the icon from the input's type, as in `[password]:has(input[type=password]) img[show]`, so the input's `type` is the only state. `setPassword(img,type)` sets the type and focuses the input. On submit, `doSubmit(form)` sets the type back to `password` before calling `ajaxForm`, so browser password managers still recognize the login and offer to save it.
- Sites that send mail read the mail server from `src/private/Config.luan`, which is never committed. Chat has `Config_sample.luan` to copy. `Config.luan` also has `mail_from_domain`, because the `From` address must match the domain the site runs on, which differs between production and testing. The rest of the address is in the code: chat's `Shared.luan` has `local default_from = title.." <chat@"..mail_from_domain..">"`.

## Security

- Every request that changes the database must call `check_referer()` from `lib/Utils.luan`, whether or not anyone is logged in, so `register.js` calls it too. A request that changes a user's data must also check the logged-in user.
	```lua
	function Utils.check_referer()
		local headers = Http.request.headers
		local referer = headers["Referer"] or error "no referer"
		local host = headers["Host"] or error()
		starts_with(referer,"https://"..host.."/") \
			or starts_with(referer,"http://"..host.."/") \
			or error "bad referer"
	end
	```
	It requires a Referer from the site itself, and rejects a missing one, since another site can turn its Referer off. That stops forged requests from other sites for GET and POST alike, so handlers that change data may use GET. Blog calls it in each handler that changes data. Chat calls it in `User.current()` for every path ending in `.js`, which covers every handler that uses the login, and also in `register.js`, which changes the database before anyone is logged in.
- Access control: every handler that takes an id must check that the current user may use that thing. Put the check on the model object: chat has `chat.check_current_user()` (the user is in the chat) and `post.check_current_user()` (the user wrote the post). Ids are sequential, so guessing them is trivial.
- Escape everything written into HTML with `Html.encode`, and prefer values from the database to values from the request. See the Security section of [websites.md](websites.md).