Example of making and managing a website with emacs org-mode Example of making and managing a website with emacs org-mode

1 Comments.

2 News

2 Mar 2012: Fixed bug in upload newer-file detection.
17 Nov 2011: Better progress indication and error detection for uploading published html files through curl and ftp.
9 Sep 2011: New method of uploading changed files to an ftp server.
20 Mar 2010: Published.

3 Description

I explain how I make and manage this website using org-mode and a bit of other emacs features. I talk about the problems I needed to solve in order to use org-mode to host this specific website. This means setting org-mode up to convert a set of files to html, how to write documents in org-mode, and how to automatically update the website using some elisp shell scripting and lftp to mirror the ftp server.

4 Problems

Setting org-mode up to convert to html (publishing).
Writing documents using org-mode.
1. Code highlighting used weird colors because of emacs color-theme different from website colors.
Automatically update and upload the website.
1. Generate a directory listing html file (sitemap).
2. Don't upload svn '.svn' directories.

5 org-mode publishing setup

I followed Worg's org-mode publishing tutorial when setting up my website. My text here will be less complete as I will only talk about the features I used.

In ~/data/danmalund.dk/org I have my source files (meaning org, css and any other files that should be processed or simply copied to the website directory).

In ~/data/danamludn.dk I have my published files, meaning the html-files that make up the website.

So to set this up in org-mode I have the elisp code:

(org-publish-projects
     '(
       ("danamlund.dk-notes"
        :base-directory "~/data/danamlund.dk/org"
        :base-extension "org"
        :publishing-directory "~/data/danamlund.dk"
        :recursive t
        :publishing-function org-publish-org-to-html
        :headline-levels 999
        :auto-preamble t

        :sub-superscript nil
        :author "Dan Amlund Thomsen"
        :email "danamlund@gmail.com"
        )
       ("danamlund.dk-static"
        :base-directory "~/data/danamlund.dk/org"
        :base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
        :publishing-directory "~/data/danamlund.dk"
        :recursive t
        :publishing-function org-publish-attachment
        )))

There are 2 "projects", one that converts org-files to html-files. And one that just copies every file over. I will explain some of these.

base-directory is where the source-files (org-files in this case) are located.
publishing-directory is where to copy the processed input files.
publishing-function is how to process the input files (convert org to html in the first case, just plain copy in the second).
headline-levels depth (amount of *'s) to threat as new sections. (don't see why anyone would need less than infinite)

The rest (headline-level, section-numbers, sub_superscript, style, author, email) are default values for settings related to org files, they can be overwritten by chaning the value in each orgfiles.

Note that this a call to the org-mode function org-publish-projects which publishes the input projects. So executing this function (move cursor after last ) and C-x C-e) will create the website files.

Note that my org-files are located in a sub-directory of where the published website is located. This doesn't cause any errors as long as I don't overwrite any files by having a org directory inside my source directory.

6 org-mode documents

Worg's org-mode publishing tutorial also talks about how to organize org files in a published project. I will discuss how I set it up for this website.

As an example I will use the org files describing this html file which is called

orgsite.org

At the very beginning of orgsite.org we have #+BEGIN_SRC org

#+END_SRC org

These are options to org-mode specifying meta-data like the title, how the file is shown in emacs like the startup, and how the published page should look like the options. The title defines the title of the document and is used to set the title of the html document. The startup defines the showall and hidestars options, which shows all text and hides all but the last star (in section definitions). The options sets the value of toc to true, meaning a table of contents will be created when converting the file to html.

Some meta-data (like author and email) are left out as default values for those are defined in the org-mode publishing project.

The actual page setup of orgsite.org consist of sections and subsections which are defined by 1 or more * followed by a title. Links are defined inside a bunch of square brackets. These brackets are hidden when viewing the file in emacs and can be edited by moving the cursor over the link and C-c C-l. Fonts are defined by surrounding text by various symbols (* for bold, / for italic, = for fixed-width).

I define code highlighting with the same syntax as in the beginning of the file (#+BEGIN_SRC elisp and #+END_SRC). The elisp means that the highlighting colors will be fetched from elisp-mode, later i use org which highlights according to org-mode.

In the end of orgsite.org I have an empty section which closes any sub-sections before printing the last part of the html-file which is the author and timestamp. I also add a horizontal line to seperate the page from the author and timestamp part of the html. Note that I use #+HTML:, if I didn't then the < and > signs would be converted to printable characters and not be parsed by the browser.

6.1 Fixing code highlighting when using colors different from the website.

My emacs have a black background and white text. If I convert org-files to html code-highlighting will probably look weird, meaning having black background color for some keywords and generally using colors with a low contrast to the white background of th website.

To fix this I make some elisp that automatically changes the colors to white background and black text, then convert org-files to html and then change the colors back.

(progn
  (set-background-color "white")
  (set-foreground-color "black")
  (org-publish-projects
   '(
     ("danamlund.dk-notes"
      :base-directory "~/data/danamlund.dk/org"
      :base-extension "org"
      :publishing-directory "~/data/danamlund.dk"
      :recursive t
      :publishing-function org-publish-org-to-html
      :headline-levels 4
      :auto-preamble t

      :sub-superscript nil
      :author "Dan Amlund Thomsen"
      :email "danamlund@gmail.com"
      )
     ("danamlund.dk-static"
      :base-directory "~/data/danamlund.dk/org"
      :base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
      :publishing-directory "~/data/danamlund.dk"
      :recursive t
      :publishing-function org-publish-attachment
      )))
  (set-background-color "black")
  (set-foreground-color "white"))

7 Automatically update and upload

To automatically update and upload my website I convert org-files to html, create a directory index, and upload the changed file to a ftp server. I will describe what the elisp part of

index.org

7.1 Easily converting org-files to html

I use the same elisp I showed earlier to do this. I keep it in my index.org file under the section named COMMENT which isn't included in the html output. I then simply execute the elisp (C-c C-e) when I need to generate the html files.

7.2 Uploading changed files to an ftp

After publishing the files to a local directory I upload the changed files to an ftp.

In order to get the files that was changed I keep a clone of the ftp in a separate directory (in this case: danamlund.dk_ftp). And to get the list of different files I use rsync. To upload the changed files I use curl. To get this to work together I have to fiddle a bit with other shell commands. My upload-changed-files code looks like this:

(let* ((dir-root "~/data")
       (dir-publish "danamlund.dk")
       (dir-ftp-clone "danamlund.dk_ftp")
       (ftp-host "danamlund.dk")
       (ftp-user "danamlund.dk")
       (ftp-pass (read-passwd "ftp password: "))
       (ftp-dir "")
       (ftp-url (format "ftp://%s:%s@%s/%s"
                        ftp-user ftp-pass ftp-host ftp-dir)))

  (defun get-newer-files (dir1 dir2 &optional filter)
    (let ((filter- (or filter (lambda (f) (or (equal f ".")
                                              (equal f "..")
                                              (equal f ".svn")))))
          (f-newer (lambda (f1 f2)
                     (let ((a1 (file-attributes f1))
                           (a2 (file-attributes f2)))
                       (or (null a2) (> (car (nth 5 a1)) (car (nth 5 a2)))
                           (> (cadr (nth 5 a1)) (cadr (nth 5 a2)))))))
          (output '())
          (inputs (directory-files dir1)))
      (while (not (null inputs))
        (let* ((f (car inputs))
               (f1 (concat dir1 "/" f))
               (f2 (concat dir2 "/" f)))
          (setq inputs (cdr inputs))
          (cond
           ((funcall filter- (file-name-nondirectory f)) t)
           ((file-directory-p f1)
            (if (not (file-exists-p f2))
                (setq output (cons f output)))
            (setq inputs (append (mapcar (lambda (f-) (concat f "/" f-))
                                         (directory-files f1)) 
                                 inputs)))
           ((funcall f-newer f1 f2)
            (setq output (cons f output))))))
      (reverse output)))

    (let* ((changed-files (get-newer-files (concat dir-root "/" dir-publish)
                                           (concat dir-root "/" dir-ftp-clone)))
           (files (length changed-files))
           (cur-file 1)
           (max-mini-window-height-temp max-mini-window-height))
      (cd dir-root)
      (dolist (f changed-files)
        (message (format "Uploading file %d/%d: '%s'" cur-file files f))
        (setq max-mini-window-height 0)
        (if (not (file-directory-p f))
          (if (not (= 0 (shell-command 
                         (concat "curl --ftp-create-dirs "
                                 "-T \"" dir-publish "/" f "\""
                                 " \"" ftp-url (file-name-directory f) 
                                 "\""))))
              (error "Error uploading '%s'" (concat dir-publish "/" f))))
        (if (not (= 0 (shell-command
                       (if (file-directory-p f)
                           (concat "mkdir " dir-ftp-clone "/" f "\n")
                         (concat "cp \"" dir-publish "/" f "\" "
                                 "\"" dir-ftp-clone "/" f "\"")))))
            (error "error copying '%s'" (concat dir-publish "/" f)))
        (setq max-mini-window-height max-mini-window-height-temp)
        (setq cur-file (+ 1 cur-file)))
      (switch-to-buffer "*Uploaded files*")
      (delete-region (point-min) (point-max))
      (dolist (f changed-files)
        (insert (concat f "\n")))
      changed-files))

Let us explain what this code does. The function read-passwd asks the user to type in a password in the minibuffer. The function get-newer-files returns a list of file names that have been modified more recently in dir1 than their equally-named file in dir2.

In the final part we process each of the outputted files names from get-newer-files. Processing these files means uploading them to our ftp server through curl and copying them to our local ftp-clone directory.

During the processing we print which file is being uploaded and how many files are left. We also stop in case a file couldn't be uploaded to the ftp or couldn't be copied to the ftp-clone.

7.3 Putting it all together

As can be seen in index.org I have combined these features into two operations, one to generate the website and one to upload it. It is usually a good idea to check the website before uploading it to the public.

(progn 
  (require 'org-publish)
  (set-background-color "white")
  (set-foreground-color "black")
  (org-publish-projects
   '(
     ("danamlund.dk-notes"
      :base-directory "~/data/danamlund.dk/org"
      :base-extension "org"
      :publishing-directory "~/data/danamlund.dk"
      :recursive t
      :publishing-function org-publish-org-to-html
      :headline-levels 4
      :auto-preamble t

      :sub-superscript nil
      :author "Dan Amlund Thomsen"
      :email "danamlund@gmail.com"
      )
     ("danamlund.dk-static"
      :base-directory "~/data/danamlund.dk/org"
      :base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
      :publishing-directory "~/data/danamlund.dk"
      :recursive t
      :publishing-function org-publish-attachment
      )))
  (with-temp-file "~/data/danamlund.dk/org/index.html"
    (insert (my-ls-html "~/data/danamlund.dk/org")))
  (set-background-color "black")
  (set-foreground-color "white"))

(let* ((dir-root "~/data")
       (dir-publish "danamlund.dk")
       (dir-ftp-clone "danamlund.dk_ftp")
       (ftp-host "danamlund.dk")
       (ftp-user "danamlund.dk")
       (ftp-pass (read-passwd "ftp password: "))
       (ftp-dir "")
       (ftp-url (format "ftp://%s:%s@%s/%s"
                        ftp-user ftp-pass ftp-host ftp-dir)))

  (defun get-newer-files (dir1 dir2 &optional filter)
    (let ((filter- (or filter (lambda (f) (or (equal f ".")
                                              (equal f "..")
                                              (equal f ".svn")))))
          (f-newer (lambda (f1 f2)
                       (let ((a1 (file-attributes f1))
                             (a2 (file-attributes f2)))
                         (or (null a2) 
                             (let ((t1 (nth 5 a1))
                                   (t2 (nth 5 a2)))
                               (or (> (car t1) (car t2))
                                   (and (= (car t1) (car t2))
                                        (> (cadr t1) (cadr t2)))))))))
          (output '())
          (inputs (directory-files dir1)))
      (while (not (null inputs))
        (let* ((f (car inputs))
               (f1 (concat dir1 "/" f))
               (f2 (concat dir2 "/" f)))
          (setq inputs (cdr inputs))
          (cond
           ((funcall filter- (file-name-nondirectory f)) t)
           ((file-directory-p f1)
            (if (not (file-exists-p f2))
                (setq output (cons f output)))
            (setq inputs (append (mapcar (lambda (f-) (concat f "/" f-))
                                         (directory-files f1)) 
                                 inputs)))
           ((funcall f-newer f1 f2)
            (setq output (cons f output))))))
      (reverse output)))

    (let* ((changed-files (get-newer-files (concat dir-root "/" dir-publish)
                                           (concat dir-root "/" dir-ftp-clone)))
           (files (length changed-files))
           (cur-file 1)
           (max-mini-window-height-temp max-mini-window-height))
      (cd dir-root)
      (dolist (f changed-files)
        (message (format "Uploading file %d/%d: '%s'" cur-file files f))
        (setq max-mini-window-height 0)
        (if (not (file-directory-p f))
          (if (not (= 0 (shell-command 
                         (concat "curl --ftp-create-dirs "
                                 "-T \"" dir-publish "/" f "\""
                                 " \"" ftp-url (file-name-directory f) 
                                 "\""))))
              (error "Error uploading '%s'" (concat dir-publish "/" f))))
        (if (not (= 0 (shell-command
                       (if (file-directory-p f)
                           (concat "mkdir " dir-ftp-clone "/" f "\n")
                         (concat "cp \"" dir-publish "/" f "\" "
                                 "\"" dir-ftp-clone "/" f "\"")))))
            (error "error copying '%s'" (concat dir-publish "/" f)))
        (setq max-mini-window-height max-mini-window-height-temp)
        (setq cur-file (+ 1 cur-file)))
      (switch-to-buffer "*Uploaded files*")
      (delete-region (point-min) (point-max))
      (dolist (f changed-files)
        (insert (concat f "\n")))
      changed-files))