#+STARTUP: showall hidestars
#+TITLE: Example of making and managing a website with emacs org-mode
#+LINK_UP: ../index.html
#+LINK_HOME: ../index.html
* Comments.
[[http://danamlund.wordpress.com/2009/12/23/danamlund-dk-org-mode-source/][Comments]].
* News
- *2 Mar 2012*: Fixed bug in upload newer-file detection.
- *17 Nov 2011*: Better progress indication and error detection for
uploading published html files through curl and ftp.
- *9 Sep 2011*: New method of uploading changed files to an ftp
server.
- *20 Mar 2010*: Published.
* Description
I explain how I make and manage this website using /org-mode/ and a bit
of other /emacs/ features. I talk about the problems I needed to solve
in order to use /org-mode/ to host this specific website. This means
setting /org-mode/ up to convert a set of files to html, how to write
documents in /org-mode/, and how to automatically update the website
using some /elisp shell scripting/ and /lftp/ to mirror the ftp server.
* Problems
1. Setting /org-mode/ up to convert to html (publishing).
2. Writing documents using /org-mode/.
1. Code highlighting used weird colors because of emacs
color-theme different from website colors.
3. Automatically update and upload the website.
1. Generate a directory listing html file (sitemap).
2. Don't upload /svn/ '.svn' directories.
* org-mode publishing setup
I followed [[http://orgmode.org/worg/org-tutorials/org-publish-html-tutorial.php][Worg's org-mode publishing tutorial]] when setting up my
website. My text here will be less complete as I will only talk about
the features I used.
In =~/data/danmalund.dk/org= I have my source files (meaning org, css
and any other files that should be processed or simply copied to the
website directory).
In =~/data/danamludn.dk= I have my published files, meaning the
html-files that make up the website.
So to set this up in org-mode I have the elisp code:
#+BEGIN_SRC elisp
(org-publish-projects
'(
("danamlund.dk-notes"
:base-directory "~/data/danamlund.dk/org"
:base-extension "org"
:publishing-directory "~/data/danamlund.dk"
:recursive t
:publishing-function org-publish-org-to-html
:headline-levels 999
:auto-preamble t
:sub-superscript nil
:author "Dan Amlund Thomsen"
:email "danamlund@gmail.com"
)
("danamlund.dk-static"
:base-directory "~/data/danamlund.dk/org"
:base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
:publishing-directory "~/data/danamlund.dk"
:recursive t
:publishing-function org-publish-attachment
)))
#+END_SRC
There are 2 "projects", one that converts org-files to html-files. And
one that just copies every file over. I will explain some of these.
* =base-directory= is where the source-files (org-files in this
case) are located.
* =publishing-directory= is where to copy the processed input
files.
* =publishing-function= is how to process the input files (convert
/org/ to /html/ in the first case, just plain copy in the second).
* =headline-levels= depth (amount of =*='s) to threat as new
sections. (don't see why anyone would need less than infinite)
The rest (=headline-level=, =section-numbers=, =sub_superscript=,
=style=, =author=, =email=) are default values for settings related to
org files, they can be overwritten by chaning the value in each
orgfiles.
Note that this a call to the org-mode function =org-publish-projects=
which publishes the input projects. So executing this function (move
cursor after last =)= and C-x C-e) will create the website files.
Note that my org-files are located in a sub-directory of where the
published website is located. This doesn't cause any errors as long as
I don't overwrite any files by having a /org/ directory inside my
source directory.
* org-mode documents
[[http://orgmode.org/worg/org-tutorials/org-publish-html-tutorial.php][Worg's org-mode publishing tutorial]] also talks about how to organize
org files in a published project. I will discuss how I set it up for
this website.
As an example I will use the org files describing this html file which
is called
#+HTML: orgsite.org
.
At the very beginning of orgsite.org we have
#+BEGIN_SRC org
#+STARTUP: showall hidestars
#+TITLE: Example of making and managing a website with emacs org-mode
#+LINK_UP: index.html
#+LINK_HOME: index.html
#+OPTIONS: toc:nil
#+END_SRC org
These are options to org-mode specifying meta-data like the title, how
the file is shown in emacs like the startup, and how the published
page should look like the options. The =title= defines the title of
the document and is used to set the title of the html document. The
=startup= defines the =showall= and =hidestars= options, which shows
all text and hides all but the last star (in section definitions). The
=options= sets the value of =toc= to true, meaning a table of contents
will be created when converting the file to html.
Some meta-data (like author and email) are left out as default values
for those are defined in the org-mode publishing project.
The actual page setup of orgsite.org consist of sections and
subsections which are defined by 1 or more =*= followed by a
title. Links are defined inside a bunch of square brackets. These
brackets are hidden when viewing the file in emacs and can be edited
by moving the cursor over the link and C-c C-l. Fonts are defined by
surrounding text by various symbols (* for bold, / for italic, = for
fixed-width).
I define code highlighting with the same syntax as in the beginning of
the file (=#+BEGIN_SRC elisp= and =#+END_SRC=). The =elisp= means that
the highlighting colors will be fetched from =elisp-mode=, later i use
=org= which highlights according to =org-mode=.
In the end of orgsite.org I have an empty section which closes any
sub-sections before printing the last part of the html-file which is
the author and timestamp. I also add a horizontal line to seperate the
page from the author and timestamp part of the html. Note that I use
=#+HTML:=, if I didn't then the < and > signs would be converted to
printable characters and not be parsed by the browser.
** Fixing code highlighting when using colors different from the website.
My emacs have a black background and white text. If I convert
org-files to html code-highlighting will probably look weird, meaning
having black background color for some keywords and generally using
colors with a low contrast to the white background of th website.
To fix this I make some elisp that automatically changes the colors to
white background and black text, then convert org-files to html and
then change the colors back.
#+BEGIN_SRC elisp
(progn
(set-background-color "white")
(set-foreground-color "black")
(org-publish-projects
'(
("danamlund.dk-notes"
:base-directory "~/data/danamlund.dk/org"
:base-extension "org"
:publishing-directory "~/data/danamlund.dk"
:recursive t
:publishing-function org-publish-org-to-html
:headline-levels 4
:auto-preamble t
:sub-superscript nil
:author "Dan Amlund Thomsen"
:email "danamlund@gmail.com"
)
("danamlund.dk-static"
:base-directory "~/data/danamlund.dk/org"
:base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
:publishing-directory "~/data/danamlund.dk"
:recursive t
:publishing-function org-publish-attachment
)))
(set-background-color "black")
(set-foreground-color "white"))
#+END_SRC
* Automatically update and upload
To automatically update and upload my website I convert org-files to
html, create a directory index, and upload the changed file to a ftp
server. I will describe what the elisp part of
#+HTML: index.org
.
** Easily converting org-files to html
I use the same elisp I showed earlier to do this. I keep it in my
index.org file under the section named /COMMENT/ which isn't included
in the html output. I then simply execute the elisp (C-c C-e) when I
need to generate the html files.
** Uploading changed files to an ftp
After publishing the files to a local directory I upload the changed
files to an ftp.
In order to get the files that was changed I keep a clone of the ftp
in a separate directory (in this case: =danamlund.dk_ftp=). And to get
the list of different files I use /rsync/. To upload the changed files
I use /curl/. To get this to work together I have to fiddle a bit with
other shell commands. My upload-changed-files code looks like this:
#+BEGIN_SRC elisp
(let* ((dir-root "~/data")
(dir-publish "danamlund.dk")
(dir-ftp-clone "danamlund.dk_ftp")
(ftp-host "danamlund.dk")
(ftp-user "danamlund.dk")
(ftp-pass (read-passwd "ftp password: "))
(ftp-dir "")
(ftp-url (format "ftp://%s:%s@%s/%s"
ftp-user ftp-pass ftp-host ftp-dir)))
(defun get-newer-files (dir1 dir2 &optional filter)
(let ((filter- (or filter (lambda (f) (or (equal f ".")
(equal f "..")
(equal f ".svn")))))
(f-newer (lambda (f1 f2)
(let ((a1 (file-attributes f1))
(a2 (file-attributes f2)))
(or (null a2) (> (car (nth 5 a1)) (car (nth 5 a2)))
(> (cadr (nth 5 a1)) (cadr (nth 5 a2)))))))
(output '())
(inputs (directory-files dir1)))
(while (not (null inputs))
(let* ((f (car inputs))
(f1 (concat dir1 "/" f))
(f2 (concat dir2 "/" f)))
(setq inputs (cdr inputs))
(cond
((funcall filter- (file-name-nondirectory f)) t)
((file-directory-p f1)
(if (not (file-exists-p f2))
(setq output (cons f output)))
(setq inputs (append (mapcar (lambda (f-) (concat f "/" f-))
(directory-files f1))
inputs)))
((funcall f-newer f1 f2)
(setq output (cons f output))))))
(reverse output)))
(let* ((changed-files (get-newer-files (concat dir-root "/" dir-publish)
(concat dir-root "/" dir-ftp-clone)))
(files (length changed-files))
(cur-file 1)
(max-mini-window-height-temp max-mini-window-height))
(cd dir-root)
(dolist (f changed-files)
(message (format "Uploading file %d/%d: '%s'" cur-file files f))
(setq max-mini-window-height 0)
(if (not (file-directory-p f))
(if (not (= 0 (shell-command
(concat "curl --ftp-create-dirs "
"-T \"" dir-publish "/" f "\""
" \"" ftp-url (file-name-directory f)
"\""))))
(error "Error uploading '%s'" (concat dir-publish "/" f))))
(if (not (= 0 (shell-command
(if (file-directory-p f)
(concat "mkdir " dir-ftp-clone "/" f "\n")
(concat "cp \"" dir-publish "/" f "\" "
"\"" dir-ftp-clone "/" f "\"")))))
(error "error copying '%s'" (concat dir-publish "/" f)))
(setq max-mini-window-height max-mini-window-height-temp)
(setq cur-file (+ 1 cur-file)))
(switch-to-buffer "*Uploaded files*")
(delete-region (point-min) (point-max))
(dolist (f changed-files)
(insert (concat f "\n")))
changed-files))
#+END_SRC
Let us explain what this code does. The function =read-passwd= asks
the user to type in a password in the minibuffer. The function
=get-newer-files= returns a list of file names that have been modified
more recently in =dir1= than their equally-named file in =dir2=.
In the final part we process each of the outputted files names from
=get-newer-files=. Processing these files means uploading them to our
ftp server through curl and copying them to our local ftp-clone
directory.
During the processing we print which file is being uploaded and how
many files are left. We also stop in case a file couldn't be uploaded
to the ftp or couldn't be copied to the ftp-clone.
** Putting it all together
As can be seen in index.org I have combined these features into two
operations, one to generate the website and one to upload it. It is
usually a good idea to check the website before uploading it to the
public.
#+BEGIN_SRC elisp
(progn
(require 'org-publish)
(set-background-color "white")
(set-foreground-color "black")
(org-publish-projects
'(
("danamlund.dk-notes"
:base-directory "~/data/danamlund.dk/org"
:base-extension "org"
:publishing-directory "~/data/danamlund.dk"
:recursive t
:publishing-function org-publish-org-to-html
:headline-levels 4
:auto-preamble t
:sub-superscript nil
:author "Dan Amlund Thomsen"
:email "danamlund@gmail.com"
)
("danamlund.dk-static"
:base-directory "~/data/danamlund.dk/org"
:base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
:publishing-directory "~/data/danamlund.dk"
:recursive t
:publishing-function org-publish-attachment
)))
(with-temp-file "~/data/danamlund.dk/org/index.html"
(insert (my-ls-html "~/data/danamlund.dk/org")))
(set-background-color "black")
(set-foreground-color "white"))
(let* ((dir-root "~/data")
(dir-publish "danamlund.dk")
(dir-ftp-clone "danamlund.dk_ftp")
(ftp-host "danamlund.dk")
(ftp-user "danamlund.dk")
(ftp-pass (read-passwd "ftp password: "))
(ftp-dir "")
(ftp-url (format "ftp://%s:%s@%s/%s"
ftp-user ftp-pass ftp-host ftp-dir)))
(defun get-newer-files (dir1 dir2 &optional filter)
(let ((filter- (or filter (lambda (f) (or (equal f ".")
(equal f "..")
(equal f ".svn")))))
(f-newer (lambda (f1 f2)
(let ((a1 (file-attributes f1))
(a2 (file-attributes f2)))
(or (null a2)
(let ((t1 (nth 5 a1))
(t2 (nth 5 a2)))
(or (> (car t1) (car t2))
(and (= (car t1) (car t2))
(> (cadr t1) (cadr t2)))))))))
(output '())
(inputs (directory-files dir1)))
(while (not (null inputs))
(let* ((f (car inputs))
(f1 (concat dir1 "/" f))
(f2 (concat dir2 "/" f)))
(setq inputs (cdr inputs))
(cond
((funcall filter- (file-name-nondirectory f)) t)
((file-directory-p f1)
(if (not (file-exists-p f2))
(setq output (cons f output)))
(setq inputs (append (mapcar (lambda (f-) (concat f "/" f-))
(directory-files f1))
inputs)))
((funcall f-newer f1 f2)
(setq output (cons f output))))))
(reverse output)))
(let* ((changed-files (get-newer-files (concat dir-root "/" dir-publish)
(concat dir-root "/" dir-ftp-clone)))
(files (length changed-files))
(cur-file 1)
(max-mini-window-height-temp max-mini-window-height))
(cd dir-root)
(dolist (f changed-files)
(message (format "Uploading file %d/%d: '%s'" cur-file files f))
(setq max-mini-window-height 0)
(if (not (file-directory-p f))
(if (not (= 0 (shell-command
(concat "curl --ftp-create-dirs "
"-T \"" dir-publish "/" f "\""
" \"" ftp-url (file-name-directory f)
"\""))))
(error "Error uploading '%s'" (concat dir-publish "/" f))))
(if (not (= 0 (shell-command
(if (file-directory-p f)
(concat "mkdir " dir-ftp-clone "/" f "\n")
(concat "cp \"" dir-publish "/" f "\" "
"\"" dir-ftp-clone "/" f "\"")))))
(error "error copying '%s'" (concat dir-publish "/" f)))
(setq max-mini-window-height max-mini-window-height-temp)
(setq cur-file (+ 1 cur-file)))
(switch-to-buffer "*Uploaded files*")
(delete-region (point-min) (point-max))
(dolist (f changed-files)
(insert (concat f "\n")))
changed-files))
#+END_SRC