Author: Kovid Goyal
Date: 2012-12-28T18:30:00+00:00

Navigation

- next

- Start »

calibre User Manual
calibre is an ebook library manager. It can view, convert and catalog ebooks in most of the major ebook formats. It can also talk to many ebook reader devices. It can go out to the Internet and fetch metadata for your books. It can download newspapers and convert them into ebooks for convenient reading. It is cross platform, running on Linux, Windows and OS X.

You’ve just started calibre. What do you do now? Before calibre can do anything with your ebooks, it first has to know about them. Drag and drop a few ebook files into calibre, or click the “Add books” button and browse for the ebooks you want to work with. Once you’ve added the books, they will show up in the main view looking something like this:

Once you’ve admired the list of books you just added to your heart’s content, you’ll probably want to read one. In order to do that you’ll have to convert the book to a format your reader understands. When first running calibre, the Welcome Wizard starts and will set up calibre for your reader device. Conversion is a breeze. Just select the book you want to convert then click the “Convert books” button. Ignore all the options for now and click “OK”. The little icon in the bottom right corner will start spinning. Once it’s finished spinning, your converted book is ready. Click the “View” button to read the book.

If you want to read the book on your reader, connect it to the computer, wait till calibre detects it (10-20 seconds) and then click the “Send to device” button. Once the icon stops spinning again, disconnect your reader and read away! If you didn’t convert the book in the previous step, calibre will auto convert it to the format your reader device understands.

To get started with more advanced usage, you should read about the Graphical User Interface. For even more power and versatility, learn the Command Line Interface. You will find the list of Frequently Asked Questions useful as well.

Sections

- The Graphical User Interface
- Adding your favorite news website
- The Ebook Viewer
- Ebook Conversion
- Editing Ebook Metadata
- Frequently Asked Questions
- Tutorials
- Customizing calibre
- Command Line Interface

- Setting up a calibre development environment

- Glossary

The main calibre user interface

- The Graphical User Interface

- Actions
- Preferences
- Catalogs
- Search & Sort
- The Search Interface
- Saving searches
- Book Details
- Tag Browser
- Jobs
- Keyboard Shortcuts

Adding your favorite news website to calibre

- Adding your favorite news website

- Completely automatic fetching
- Customizing the fetch process
- Tips for developing new recipes
- Further reading
- API documentation

The calibre ebook viewer

- The Ebook Viewer

- Starting the viewer

- Navigating around an ebook

- Customizing the look and feel of your reading experience

- Dictionary lookup

- Copying text and images

Customizing calibre‘s ebook conversion

- Ebook Conversion

- Introduction
- Look & Feel
- Page Setup
- Heuristic Processing
- Search & Replace
- Structure Detection
- Table of Contents

- Using images as chapter titles when converting HTML input documents

- How options are set/saved for Conversion

- Format specific tips

Editing ebook metadata

- Editing Ebook Metadata

- Editing the metadata of one book at a time

- Editing the metadata of many books at a time

Frequently Asked Questions

- Frequently Asked Questions

- Ebook Format Conversion
- Device Integration
- Library Management
- Content From The Web
- Miscellaneous

Tutorials

- Tutorials

- Adding your favorite news website

- Managing subgroups of books, for example “genre”

- XPath Tutorial

- The calibre template language

- All about using regular expressions in calibre

- Integrating the calibre content server into other servers

- Writing your own plugins to extend calibre‘s functionality

- Typesetting Math in ebooks

- Creating AZW3 • EPUB • MOBI Catalogs

Customizing calibre

- Customizing calibre

- Environment variables
- Tweaks
- Overriding icons, templates, et cetera
- Customizing calibre with plugins

The Command Line Interface

- Command Line Interface

- Documented Commands

- Undocumented Commands

Setting up a calibre development environment

- Setting up a calibre development environment

- Design philosophy
- Getting the code
- Windows development environment
- OS X development environment
- Linux development environment

- Having separate “normal” and “development” calibre installs on the same computer

- Debugging tips

- Using calibre in your projects

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

The Graphical User Interface

The Graphical User Interface (GUI) provides access to all library management and ebook format conversion features. The basic workflow for using calibre is to first add books to the library from your hard disk.
calibre will automatically try to read metadata from the books and add them to its internal database. Once they are in the database, you can perform various Actions on them that include conversion from one format to another, transfer to the reading device, viewing on your computer, and editing metadata.
The latter includes modifying the cover, description, and tags among other details.
Note that calibre creates copies of the files you add to it. Your original files are left untouched.

The interface is divided into various sections:

- Actions
- Preferences
- Catalogs
- Search & Sort
- The Search Interface
- Saving searches
- Book Details
- Tag Browser
- Jobs
- Keyboard Shortcuts

Actions

The actions toolbar provides convenient shortcuts to commonly used actions. If you right-click the buttons, you can perform variations on the default action.
Please note that the actions toolbar will look slightly different depending on whether you have an ebook reader attached to your computer.

- Add books
- Edit metadata
- Convert books
- View
- Send to device
- Fetch news
- Library
- Device
- Save to disk
- Connect/Share
- Remove books

Add books

The Add books action has six variations accessed by doing a right-click on the button.

1. Add books from a single directory: Opens a file chooser dialog and allows you to specify which books in a directory should be added. This action is context sensitive, i.e. it depends on which catalog you have selected. If you have selected the Library, books will be added to the library. If you have selected the ebook reader device, the books will be uploaded to the device, and so on.

2. Add books from directories, including sub-directories (One book per directory, assumes every ebook file is the same book in a different format): Allows you to choose a directory. The directory and all its sub-directories are scanned recursively, and any ebooks found are added to the library. calibre assumes that each directory contains a single book. All ebook files in a directory are assumed to be the same book in different formats. This action is the inverse of the Save to disk action, i.e. you can Save to disk, delete the books and re-add them with no lost information except for the date (this assumes you have not changed any of the setting for the Save to disk action).

3. Add books from directories, including sub-directories (Multiple books per directory, assumes every ebook file is a different book): Allows you to choose a directory. The directory and all its sub-directories are scanned recursively and any ebooks found are added to the library. calibre assumes that each directory contains many books. All ebook files with the same name in a directory are assumed to be the same book in different formats. Ebooks with different names are added as different books.

4. Add empty book. (Book Entry with no formats): Allows you to create a blank book record. This can be used to then manually fill out the information about a book that you may not have yet in your collection.

5. Add from ISBN: Allows you to add one or more books by entering their ISBNs.

6. Add files to selected book records: Allows you to add or update the files associated with an existing book in your library.

The Add books action can read metadata from a wide variety of ebook formats. In addition, it tries to guess metadata from the filename.
See the Guessing metadata from file names section, to learn how to configure this.

To add an additional format for an existing book use the Edit metadata action.

Edit metadata

The Edit metadata action has four variations which can be accessed by doing a right-click on the button.

1. Edit metadata individually: Allows you to edit the metadata of books one-by-one with the option of fetching metadata, including covers, from the Internet. It also allows you to add or remove particular ebook formats from a book.

2. Edit metadata in bulk: Allows you to edit common metadata fields for large numbers of books simulataneously. It operates on all the books you have selected in the Library view.

3. Download metadata and covers: Downloads metadata and covers (if available) for the books that are selected in the book list.

4. Merge book records: Gives you the capability of merging the metadata and formats of two or more book records. You can choose to either delete or keep the records that were not clicked first.

For more details see Editing Ebook Metadata.

Convert books

Ebooks can be converted from a number of formats into whatever format your ebook reader prefers.
Many ebooks available for purchase will be protected by Digital Rights Management [http://drmfree.calibre-ebook.com/about#drm] (DRM) technology.
calibre will not convert these ebooks. It is easy to remove the DRM from many formats, but as this may be illegal, you will have to find tools to liberate your books yourself and then use calibre to convert them.

For most people, conversion should be a simple one-click affair. If you want to learn more about the conversion process, see Ebook Conversion.

The Convert books action has three variations, accessed by doing a right-click on the button.

1. Convert individually: Allows you to specify conversion options to customize the conversion of each selected ebook.

2. Bulk convert: Allows you to specify options only once to convert a number of ebooks in bulk.

3. Create a catalog of the books in your calibre library: Allows you to generate a complete listing of the books in your library, including all metadata, in several formats such as XML, CSV, BiBTeX, EPUB and MOBI. The catalog will contain all the books currently showing in the library view.
This allows you to use the search features to limit the books to be catalogued. In addition, if you select multiple books using the mouse, only those books will be added to the catalog. If you generate the catalog in an ebook format such as EPUB, MOBI or AZW3, the next time you connect your ebook reader the catalog will be automatically sent to the device.
For more information on how catalogs work, read the Creating AZW3 • EPUB • MOBI Catalogs.

View

The View action displays the book in an ebook viewer program. calibre has a built-in viewer for many ebook formats.
For other formats it uses the default operating system application. You can configure which formats should open with the internal viewer via
Preferences->Behavior. If a book has more than one format, you can view a particular format by doing a right-click on the button.

Send to device

The Send to device action has eight variations, accessed by doing a right-click on the button.

1. Send to main memory: The selected books are transferred to the main memory of the ebook reader.

2. Send to card (A): The selected books are transferred to the storage card (A) on the ebook reader.

3. Send to card (B): The selected books are transferred to the storage card (B) on the ebook reader.

4. Send specific format to: The selected books are transferred to the selected storage location on the device, in the format that you specify.

5. Eject device: Detaches the device from calibre.

6. Set default send to device action: Allows you to specify which of the options, 1 through 5 above or 7 below, will be the default action when you click the main button.

7. Send and delete from library: The selected books are transferred to the selected storage location on the device and then deleted from the Library.

8. Fetch Annotations (experimental): Transfers annotations you may have made on an ebook on your device to the comments metadata of the book in the calibre library.

You can control the file name and folder structure of files sent to the device by setting up a template in Preferences->Import/Export->Sending books to devices. Also see The calibre template language.

Fetch news

The Fetch news action downloads news from various websites and converts it into an ebook that can be read on your ebook reader. Normally, the newly created ebook is added to your ebook library, but if an ebook reader is connected at the time the download finishes, the news is also uploaded to the reader automatically.

The Fetch news action uses simple recipes (10-15 lines of code) for each news site. To learn how to create recipes for your own news sources, see Adding your favorite news website.

The Fetch news action has three variations, accessed by doing a right-click on the button.

1. Schedule news download: Allows you to schedule the download of of your selected news sources from a list of hundreds available. Scheduling can be set individually for each news source you select and the scheduling is flexible allowing you to select specific days of the week or a frequency of days between downloads.

2. Add a custom news source: Allows you to create a simple recipe for downloading news from a custom news site that you wish to access. Creating the recipe can be as simple as specifying an RSS news feed URL, or you can be more prescriptive by creating Python-based code for the task. For more information see Adding your favorite news website.

3. Download all scheduled news sources: Causes calibre to immediately begin downloading all news sources that you have scheduled.

Library

The Library action allows you to create, switch between, rename or remove a Library. calibre allows you to create as many libraries as you wish. You could, for instance, create a fiction library, a non-fiction library, a foreign language library, a project library, or any structure that suits your needs. Libraries are the highest organizational structure within calibre. Each library has its own set of books, tags, categories and base storage location.

1. Switch/create library...: Allows you to; a) connect to a pre-existing calibre library at another location, b) create an empty library at a new location or, c) move the current library to a newly specified location.

2. Quick switch: Allows you to switch between libraries that have been registered or created within calibre.

3. Rename library: Allows you to rename a Library.

4. Delete library: Allows you to unregister a library from calibre.

5. <library name>: Actions 5, 6 etc... give you immediate switch access between multiple libraries that you have created or attached to. This list contains only the 5 most frequently used libraries. For the complete list, use the Quick Switch menu.

6. Library maintenance: Allows you to check the current library for data consistency issues and restore the current library’s database from backups.

Note

Metadata about your ebooks, e.g. title, author, and tags, is stored in a single file in your calibre library folder called metadata.db. If this file gets corrupted (a very rare event), you can lose the metadata. Fortunately, calibre automatically backs up the metadata for every individual book in the book’s folder as an OPF file. By using the Restore Library action under Library Maintenance described above, you can have calibre rebuild the metadata.db file from the individual OPF files for you.

You can copy or move books between different libraries (once you have more than one library setup) by right clicking on the book and selecting the action Copy to library.

Device

The Device action allows you to view the books in the main memory or storage cards of your device, or to eject the device (detach it from calibre).
This icon shows up automatically on the main calibre toolbar when you connect a supported device. You can click on it to see the books on your device. You can also drag and drop books from your calibre library onto the icon to transfer them to your device. Conversely, you can drag and drop books from your device onto the library icon on the toolbar to transfer books from your device to the calibre library.

Save to disk

The Save to disk action has five variations, accessed by doing a right-click on the button.

1. Save to disk: Saves the selected books to disk organized in directories. The directory structure looks like:

Author_(sort)
Title
Book Files

You can control the file name and folder structure of files saved to disk by setting up a template in Preferences->Import/Export->Saving books to disk. Also see The calibre template language.

2. Save to disk in a single directory: Saves the selected books to disk in a single directory.

For 1. and 2., all available formats, as well as metadata, are stored to disk for each selected book. Metadata is stored in an OPF file. Saved books can be re-imported to the library without any loss of information by using the Add books action.

3. Save only *<your preferred>* format to disk: Saves the selected books to disk in the directory structure as shown in (1.) but only in your preferred ebook format. You can set your preferred format in Preferences->Behaviour->Preferred output format

4. Save only *<your preferred>* format to disk in a single directory: Saves the selected books to disk in a single directory but only in your preferred ebook format. You can set your preferred format in Preferences->Behaviour->Preferred output format

5. Save single format to disk...: Saves the selected books to disk in the directory structure as shown in (1.) but only in the format you select from the pop-out list.

Connect/Share

The Connect/Share action allows you to manually connect to a device or folder on your computer. It also allows you to set up you calibre library for access via a web browser or email.

The Connect/Share action has four variations, accessed by doing a right-click on the button.

1. Connect to folder: Allows you to connect to any folder on your computer as though it were a device and use all the facilities calibre has for devices with that folder. Useful if your device cannot be supported by calibre but is available as a USB disk.

2. Connect to iTunes: Allows you to connect to your iTunes books database as though it were a device. Once the books are sent to iTunes, you can use iTunes to make them available to your various iDevices.

3. Start Content Server: Starts calibre‘s built-in web server. When started, your calibre library will be accessible via a web browser from the Internet (if you choose). You can configure how the web server is accessed by setting preferences at Preferences->Sharing->Sharing over the net

4. Setup email based sharing of books: Allows sharing of books and news feeds by email. After setting up email addresses for this option, calibre will send news updates and book updates to the entered email addresses. You can configure how calibre sends email by setting preferences at Preferences->Sharing->Sharing books by email. Once you have set up one or more email addresses, this menu entry will be replaced by menu entries to send books to the configured email addresses.

Remove books

The Remove books action deletes books permanently, so use it with care. It is context sensitive, i.e. it depends on which catalog you have selected. If you have selected the Library, books will be removed from the library. If you have selected the ebook reader device, books will be removed from the device. To remove only a particular format for a given book use the Edit metadata action. Remove books also has five variations which can be accessed by doing a right-click on the button.

1. Remove selected books: Allows you to permanently remove all books that are selected in the book list.

2. Remove files of a specific format from selected books...: Allows you to permanently remove ebook files of a specified format from books that are selected in the book list.

3. Remove all formats from selected books, except...: Allows you to permanently remove ebook files of any format except a specified format from books that are selected in the book list.

3. Remove all formats from selected books: Allows you to permanently remove all ebook files from books that are selected in the book list. Only the metadata will remain.

4. Remove covers from selected books: Allows you to permanently remove cover image files from books that are selected in the book list.

5. Remove matching books from device: Allows you to remove ebook files from a connected device that match the books that are selected in the book list.

Note

Note that when you use Remove books to delete books from your calibre library, the book record is permanently deleted, but on Windows and OS X the files are placed into the recycle bin. This allows you to recover them if you change your mind.

Preferences

The Preferences action allows you to change the way various aspects of calibre work. It has four variations, accessed by doing a right-click on the button.

1. Preferences: Allows you to change the way various aspects of calibre work. Clicking the button also performs this action.

2. Run welcome wizard: Allows you to start the Welcome Wizard which appeared the first time you started calibre.

3. Get plugins to enhance |app|: Opens a new windows that shows plugins for calibre. These plugins are developed by third parties to extend calibre‘s functionality.

4. Restart in debug mode: Allows you to enable a debugging mode that can assist the calibre developers in solving problems you encounter with the program. For most users this should remain disabled unless instructed by a developer to enable it.

Catalogs

A catalog is a collection of books. calibre can manage two types of different catalogs:

1. Library: This is a collection of books stored in your calibre library on your computer.

2. Device: This is a collection of books stored in your ebook reader. It will be available when you connect the reader to your computer.

Many operations, such as adding books, deleting, viewing, etc., are context sensitive. So, for example, if you click the View button when you have the Device catalog selected, calibre will open the files on the device to view. If you have the Library catalog selected, files in your calibre library will be opened instead.

Search & Sort

The Search & Sort section allows you to perform several powerful actions on your book collections.

- You can sort them by title, author, date, rating, etc. by clicking on the column titles. You can also sub-sort, i.e. sort on multiple columns. For example, if you click on the title column and then the author column, the book will be sorted by author and then all the entries for the same author will be sorted by title.

- You can search for a particular book or set of books using the search bar. More on that below.

- You can quickly and conveniently edit metadata by double-clicking the entry you want changed in the list.

- You can perform Actions on sets to books. To select multiple books you can either:

- Keep the Ctrl key pressed and click on the books you want selected.

- Keep the Shift key pressed and click on the starting and ending book of a range of books you want selected.

- You can configure which fields you want displayed by using the Preferences dialog.

The Search Interface

You can search all the metadata by entering search terms in the search bar. Searches are case insensitive. For example:

Asimov Foundation format:lrf

This will match all books in your library that have Asimov and Foundation in their metadata and are available in the LRF format. Some more examples:

author:Asimov and not series:Foundation
title:"The Ring" or "This book is about a ring"
format:epub publisher:feedbooks.com

Searches are by default ‘contains’. An item matches if the search string appears anywhere in the indicated metadata.
Two other kinds of searches are available: equality search and search using regular expressions [http://en.wikipedia.org/wiki/Regular_expression].

Equality searches are indicated by prefixing the search string with an equals sign (=). For example, the query tag:"=science" will match “science”, but not “science fiction” or “hard science”. Regular expression searches are indicated by prefixing the search string with a tilde (~). Any python-compatible regular expression [http://docs.python.org/library/re.html] can be used. Note that backslashes used to escape special characters in reqular expressions must be doubled because single backslashes will be removed during query parsing. For example, to match a literal parenthesis you must enter \\(. Regular expression searches are ‘contains’ searches unless the expression contains anchors.

Should you need to search for a string with a leading equals or tilde, prefix the string with a backslash.

Enclose search strings with quotes (”) if the string contains parenthesis or spaces. For example, to search for the tag Science Fiction you would need to search for tag:"=science fiction". If you search for tag:=science fiction you will find all books with the tag ‘science’ and containing the word ‘fiction’ in any metadata.

You can build advanced search queries easily using the Advanced Search Dialog accessed by clicking the button .

Available fields for searching are: tag, title, author, publisher, series, series_index, rating, cover, comments, format, identifiers, date, pubdate, search, size and custom columns. If a device is plugged in, the ondevice field becomes available, when searching the calibre library view. To find the search name (actually called the lookup name) for a custom column, hover your mouse over the column header in the library view.

The syntax for searching for dates is:

pubdate:>2000-1 Will find all books published after Jan, 2000
date:<=2000-1-3 Will find all books added to calibre before 3 Jan, 2000
pubdate:=2009 Will find all books published in 2009

If the date is ambiguous, the current locale is used for date comparison. For example, in an mm/dd/yyyy locale 2/1/2009 is interpreted as 1 Feb 2009. In a dd/mm/yyyy locale it is interpreted as 2 Jan 2009. Some special date strings are available. The string today translates to today’s date, whatever it is. The strings yesterday and thismonth (or the translated equivalent in the current language) also work.
In addition, the string daysago (also translated) can be used to compare to a date some number of days ago.
For example:

date:>10daysago
date:<=45daysago

You can search for books that have a format of a certain size like this:

size:>1.1M Will find books with a format larger than 1.1MB
size:<=1K Will find books with a format smaller than 1KB

Dates and numeric fields support the relational operators = (equals), > (greater than), >= (greater than or equal to), < (less than), <= (less than or equal to), and != (not equal to).
Rating fields are considered to be numeric. For example, the search rating:>=3 will find all books rated 3 or higher.

You can search for the number of items in multiple-valued fields such as tags. These searches begin with the character #, then use the same syntax as numeric fields. For example, to find all books with more than 4 tags use tags:#>4. To find all books with exactly 10 tags use tags:#=10.

Series indices are searchable. For the standard series, the search name is ‘series_index’. For custom series columns, use the column search name followed by _index. For example, to search the indices for a custom series column named #my_series, you would use the search name #my_series_index.
Series indices are numbers, so you can use the relational operators described above.

The special field search is used for saved searches. So if you save a search with the name
“My spouse’s books” you can enter search:"My spouse's books" in the search bar to reuse the saved search. More about saving searches below.

You can search for the absence or presence of a field using the special “true” and “false” values. For example:

cover:false will give you all books without a cover
series:true will give you all books that belong to a series
comments:false will give you all books with an empty comment
format:false will give you all books with no actual files (empty records)

Yes/no custom columns are searchable. Searching for false, empty, or blank will find all books with undefined values in the column. Searching for true will find all books that do not have undefined values in the column. Searching for yes or checked will find all books with Yes in the column.
Searching for no or unchecked will find all books with No in the column. Note that the words yes, no, blank, empty, checked and unchecked are translated; you must use the current language’s equivalent word. The words true and false and the special values _yes and _no are not translated.

Hierarchical items (e.g. A.B.C) use an extended syntax to match initial parts of the hierarchy. This is done by adding a period between the exact match indicator (=) and the text. For example, the query tags:=.A will find the tags A and A.B, but will not find the tags AA or AA.B. The query tags:=.A.B will find the tags A.B and A.B.C, but not the tag A.

Identifiers (e.g., isbn, doi, lccn etc) also use an extended syntax. First, note that an identifier has the form type:value, as in isbn:123456789. The extended syntax permits you to specify independently which type and value to search for. Both the type and the value parts of the query can use equality, contains, or regular expression matches. Examples:

- identifiers:true will find books with any identifier.

- identifiers:false will find books with no identifier.

- identifiers:123 will search for books with any type having a value containing 123.

- identifiers:=123456789 will search for books with any type having a value equal to 123456789.

- identifiers:=isbn: and identifiers:isbn:true will find books with a type equal to isbn having any value

- identifiers:=isbn:false will find books with no type equal to isbn.

- identifiers:=isbn:123 will find books with a type equal to isbn having a value containing 123.

- identifiers:=isbn:=123456789 will find books with a type equal to isbn having a value equal to 123456789.

- identifiers:i:1 will find books with a type containing an i having a value containing a 1.

Advanced Search Dialog

Saving searches
calibre allows you to save a frequently used search under a special name and then reuse that search with a single click. To do this, create your search either by typing it in the search bar or using the Tag Browser. Then type the name you would like to give to the search in the Saved Searches box next to the search bar. Click the plus icon next to the saved searches box to save the search.

Now you can access your saved search in the Tag Browser under “Searches”. A single click will allow you to reuse any arbitrarily complex search easily, without needing to re-create it.

Guessing metadata from file names

In the Add/Save section of the configuration dialog, you can specify a regular expression that calibre will use to try and guess metadata from the names of ebook files that you add to the library. The default regular expression is:

title - author

i.e., it assumes that all characters up to the first - are the title of the book and subsequent characters are the author of the book. For example, the filename:

Foundation and Earth - Isaac Asimov.txt

will be interpreted to have the title: Foundation and Earth and author: Isaac Asimov

Tip

If the filename does not contain the hyphen, the regular expression will fail.

Book Details

The Book Details display shows extra information and the cover for the currently selected book.

Tag Browser

The Tag Browser allows you to easily browse your collection by Author/Tags/Series/etc. If you click on any item in the Tag Browser, for example the author name Isaac Asimov, then the list of books to the right is restricted to showing books by that author. You can click on category names as well. For example, clicking on “Series” will show you all books in any series.

The first click on an item will restrict the list of books to those that contain or match the item. Continuing the above example, clicking on Isaac Asimov will show books by that author. Clicking again on the item will change what is shown, depending on whether the item has children (see sub-categories and hierarchical items below). Continuing the Isaac Asimov example, clicking again on Isaac Asimov will restrict the list of books to those not by Isaac Asimov. A third click will remove the restriction, showing all books. If you hold down the Ctrl or Shift keys and click on multiple items, then restrictions based on multiple items are created. For example you could hold Ctrl and click on the tags History and Europe for finding books on European history. The Tag Browser works by constructing search expressions that are automatically entered into the Search bar. Looking at what the Tag Browser generates is a good way to learn how to construct basic search expressions.

Items in the Tag browser have their icons partially colored. The amount of color depends on the average rating of the books in that category. So for example if the books by Isaac Asimov have an average of four stars, the icon for Isaac Asimov in the Tag Browser will be 4/5th colored. You can hover your mouse over the icon to see the average rating.

The outer-level items in the tag browser, such as Authors and Series, are called categories. You can create your own categories, called User Categories, which are useful for organizing items. For example, you can use the User Categories Editor (click the Manage User Categories button) to create a user category called Favorite Authors, then put the items for your favorites into the category. User categories can have sub-categories. For example, the user category Favorites.Authors is a sub-category of Favorites. You might also have Favorites.Series, in which case there will be two sub-categories under Favorites. Sub-categories can be created by right-clicking on a user category, choosing “Add sub-category to ...”, and entering the sub-category name; or by using the User Categories Editor by entering names like the Favorites example above.

You can search user categories in the same way as built-in categories, by clicking on them. There are four different searches cycled through by clicking:

1. “everything matching an item in the category” indicated by a single green plus sign.

2. “everything matching an item in the category or its sub-categories” indicated by two green plus signs.

3. “everything not matching an item in the category” shown by a single red minus sign.

4. “everything not matching an item in the category or its sub-categories” shown by two red minus signs.

It is also possible to create hierarchies inside some of the text categories such as tags, series, and custom columns. These hierarchies show with the small triangle, permitting the sub-items to be hidden. To use hierarchies of items in a category, you must first go to Preferences->Look & Feel and enter the category name(s) into the “Categories with hierarchical items” box. Once this is done, items in that category that contain periods will be shown using the small triangle. For example, assume you create a custom column called “Genre” and indicate that it contains hierarchical items. Once done, items such as Mystery.Thriller and Mystery.English will display as Mystery with the small triangle next to it. Clicking on the triangle will show Thriller and English as sub-items. See Managing subgroups of books, for example “genre” for more information.

Hierarchical items (items with children) use the same four ‘click-on’ searches as user categories. Items that do not have children use two of the searches: “everything matching” and “everything not matching”.

You can drag and drop items in the Tag browser onto user categories to add them to that category. If the source is a user category, holding the shift key while dragging will move the item to the new category. You can also drag and drop books from the book list onto items in the Tag Browser; dropping a book on an item causes that item to be automatically applied to the dropped books. For example, dragging a book onto Isaac Asimov will set the author of that book to Isaac Asimov. Dropping it onto the tag History will add the tag History to the book’s tags.

There is a search bar at the top of the Tag Browser that allows you to easily find any item in the Tag Browser. In addition, you can right click on any item and choose one of several operations. Some examples are to hide the it, rename it, or open a “Manage x” dialog that allows you to manage items of that kind. For example, the “Manage Authors” dialog allows you to rename authors and control how their names are sorted.

You can control how items are sorted in the Tag browser via the box at the bottom of the Tag Browser. You can choose to sort by name, average rating or popularity (popularity is the number of books with an item in your library; for example, the popularity of Isaac Asimov is the number of books in your library by Isaac Asimov).

Jobs

The Jobs panel shows the number of currently running jobs. Jobs are tasks that run in a separate process. They include converting ebooks and talking to your reader device. You can click on the jobs panel to access the list of jobs. Once a job has completed you can see a detailed log from that job by double-clicking it in the list. This is useful to debug jobs that may not have completed successfully.

Keyboard Shortcuts

Calibre has several keyboard shortcuts to save you time and mouse movement. These shortcuts are active in the book list view (when you’re not editing the details of a particular book), and most of them affect the title you have selected. The calibre ebook viewer has its own shortcuts which can be customised by clicking the Preferences button in the viewer.

Note

Note: The Calibre keyboard shortcuts do not require a modifier key (Command, Option, Control, etc.), unless specifically noted. You only need to press the letter key, e.g. E to edit.

Keyboard Shortcuts

Keyboard Shortcut | Action
F2 (Enter in OS X) | Edit the metadata of the currently selected field in the book list.
A | Add Books
Shift+A | Add Formats to the selected books
C | Convert selected Books
D | Send to device
Del | Remove selected Books
E | Edit metadata of selected books
G | Get Books
I | Show book details
M | Merge selected records
Alt+M | Merge selected records, keeping originals
O | Open containing folder
S | Save to Disk
V | View
Alt+V/Cmd+V in OS X | View specific format
Alt+Shift+J | Toggle jobs list
Alt+Shift+B | Toggle Cover Browser
Alt+Shift+D | Toggle Book Details panel
Alt+Shift+T | Toggle Tag Browser
Alt+A | Show books by the same author as the current book
Alt+T | Show books with the same tags as current book
Alt+P | Show books by the same publisher as current book
Alt+Shift+S | Show books in the same series as current book
/, Ctrl+F | Focus the search bar
Shift+Ctrl+F | Open the advanced search dialog
Esc | Clear the current search

N or F3

Find the next book that matches the current search (only works if the highlight checkbox next to the search bar is checked)

Shift+N or Shift+F3

Find the next book that matches the current search (only works if the highlight checkbox next to the search bar is checked)

Ctrl+D | Download metadata and shortcuts
Ctrl+R | Restart calibre
Ctrl+Shift+R | Restart calibre in debug mode
Shift+Ctrl+E | Add empty books to calibre
Ctrl+Q | Quit calibre

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

Adding your favorite news website
calibre has a powerful, flexible and easy-to-use framework for downloading news from the Internet and converting it into an ebook. The following will show you, by means of examples, how to get news from various websites.

To gain an understanding of how to use the framework, follow the examples in the order listed below:

- Completely automatic fetching

- portfolio.com

- bbc.co.uk

- Customizing the fetch process

- Using the print version of bbc.co.uk
- Replacing article styles
- Slicing and dicing
- Real life example

- Tips for developing new recipes
- Further reading
- API documentation

Completely automatic fetching

If your news source is simple enough, calibre may well be able to fetch it completely automatically, all you need to do is provide the URL. calibre gathers all the information needed to download a news source into a recipe. In order to tell calibre about a news source, you have to create a recipe for it. Let’s see some examples:

portfolio.com
portfolio.com is the website for Condé Nast Portfolio, a business related magazine. In order to download articles from the magazine and convert them to ebooks, we rely on the RSS feeds of portfolio.com. A list of such feeds is available at http://www.portfolio.com/rss/.

Lets pick a couple of feeds that look interesting:

1. Business Travel: http://feeds.portfolio.com/portfolio/businesstravel

2. Tech Observer: http://feeds.portfolio.com/portfolio/thetechobserver

I got the URLs by clicking the little orange RSS icon next to each feed name. To make calibre download the feeds and convert them into an ebook, you should right click the Fetch news button and then the Add a custom news source menu item. A dialog similar to that shown below should open up.

First enter Portfolio into the Recipe title field. This will be the title of the ebook that will be created from the articles in the above feeds.

The next two fields (Oldest article and Max. number of articles) allow you some control over how many articles should be downloaded from each feed, and they are pretty self explanatory.

To add the feeds to the recipe, enter the feed title and the feed URL and click the Add feed button. Once you have added both feeds, simply click the Add/update recipe button and you’re done! Close the dialog.

To test your new recipe, click the Fetch news button and in the Custom news sources sub-menu click Portfolio. After a couple of minutes, the newly downloaded Portfolio ebook will appear in the main library view (if you have your reader connected, it will be put onto the reader instead of into the library). Select it and hit the View button to read!

The reason this worked so well, with so little effort is that portfolio.com provides full-content RSS feeds, i.e., the article content is embedded in the feed itself. For most news sources that provide news in this fashion, with full-content feeds, you don’t need any more effort to convert them to ebooks. Now we will look at a news source that does not provide full content feeds. In such feeds, the full article is a webpage and the feed only contains a link to the webpage with a short summary of the article.

bbc.co.uk

Lets try the following two feeds from The BBC:

1. News Front Page: http://newsrss.bbc.co.uk/rss/newsonline_world_edition/front_page/rss.xml

2. Science/Nature: http://newsrss.bbc.co.uk/rss/newsonline_world_edition/science/nature/rss.xml

Follow the procedure outlined in portfolio.com to create a recipe for The BBC (using the feeds above). Looking at the downloaded ebook, we see that calibre has done a creditable job of extracting only the content you care about from each article’s webpage. However, the extraction process is not perfect. Sometimes it leaves in undesirable content like menus and navigation aids or it removes content that should have been left alone, like article headings. In order, to have perfect content extraction, we will need to customize the fetch process, as described in the next section.

Customizing the fetch process

When you want to perfect the download process, or download content from a particularly complex website, you can avail yourself of all the power and flexibility of the recipe framework. In order to do that, in the Add custom news sources dialog, simply click the Switch to Advanced mode button.

The easiest and often most productive customization is to use the print version of the online articles. The print version typically has much less cruft and translates much more smoothly to an ebook. Let’s try to use the print version of the articles from The BBC.

Using the print version of bbc.co.uk

The first step is to look at the ebook we downloaded previously from bbc.co.uk. At the end of each article, in the ebook is a little blurb telling you where the article was downloaded from. Copy and paste that URL into a browser. Now on the article webpage look for a link that points to the “Printable version”. Click it to see the print version of the article. It looks much neater! Now compare the two URLs. For me they were:

Article URL
http://news.bbc.co.uk/2/hi/science/nature/7312016.stm

Print version URL
http://newsvote.bbc.co.uk/mpapps/pagetools/print/news.bbc.co.uk/2/hi/science/nature/7312016.stm

So it looks like to get the print version, we need to prefix every article URL with:

newsvote.bbc.co.uk/mpapps/pagetools/print/

Now in the Advanced Mode of the Custom news sources dialog, you should see something like (remember to select The BBC recipe before switching to advanced mode):

You can see that the fields from the Basic mode have been translated to python code in a straightforward manner. We need to add instructions to this recipe to use the print version of the articles. All that’s needed is to add the following two lines:

def print_version(self, url):
return url.replace('http://', 'http://newsvote.bbc.co.uk/mpapps/pagetools/print/')

This is python, so indentation is important. After you’ve added the lines, it should look like:

In the above, def print_version(self, url) defines a method that is called by calibre for every article. url is the URL of the original article. What print_version does is take that url and replace it with the new URL that points to the print version of the article. To learn about python [http://www.python.org] see the tutorial [http://docs.python.org/tut/].

Now, click the Add/update recipe button and your changes will be saved. Re-download the ebook. You should have a much improved ebook. One of the problems with the new version is that the fonts on the print version webpage are too small. This is automatically fixed when converting to an ebook, but even after the fixing process, the font size of the menus and navigation bar to become too large relative to the article text. To fix this, we will do some more customization, in the next section.

Replacing article styles

In the previous section, we saw that the font size for articles from the print version of The BBC was too small. In most websites, The BBC included, this font size is set by means of CSS stylesheets. We can disable the fetching of such stylesheets by adding the line:

no_stylesheets = True

The recipe now looks like:

The new version looks pretty good. If you’re a perfectionist, you’ll want to read the next section, which deals with actually modifying the downloaded content.

Slicing and dicing
calibre contains very powerful and flexible abilities when it comes to manipulating downloaded content. To show off a couple of these, let’s look at our old friend the The BBC recipe again. Looking at the source code (HTML) of a couple of articles (print version), we see that they have a footer that contains no useful information, contained in

<div class="footer">
...
</div>

This can be removed by adding:

remove_tags = [dict(name='div', attrs={'class':'footer'})]

to the recipe. Finally, lets replace some of the CSS that we disabled earlier, with our own CSS that is suitable for conversion to an ebook:

extra_css = '.headline {font-size: x-large;} \n .fact { padding-top: 10pt }'

With these additions, our recipe has become “production quality”, indeed it is very close to the actual recipe used by calibre for the BBC, shown below:

##
## Title: BBC News, Sport, and Blog Calibre Recipe
## Contact: mattst - jmstanfield@gmail.com
##
## License: GNU General Public License v3 - http://www.gnu.org/copyleft/gpl.html
## Copyright: mattst - jmstanfield@gmail.com
##
## Written: November 2011
## Last Edited: 2011-11-19
##

__license__ = 'GNU General Public License v3 - http://www.gnu.org/copyleft/gpl.html'
__copyright__ = 'mattst - jmstanfield@gmail.com'

'''
BBC News, Sport, and Blog Calibre Recipe
'''

# Import the regular expressions module.
import re

# Import the BasicNewsRecipe class which this class extends.
from calibre.web.feeds.recipes import BasicNewsRecipe class BBCNewsSportBlog(BasicNewsRecipe):

#
# **** IMPORTANT USERS READ ME ****
#
# First select the feeds you want then scroll down below the feeds list
# and select the values you want for the other user preferences, like
# oldest_article and such like.
#
#
# Select the BBC rss feeds which you want in your ebook.
# Selected feed have NO '#' at their start, de-selected feeds begin with a '#'.
#
# Eg. ("News Home", "http://feeds.bbci.co.uk/... - include feed.
# Eg. #("News Home", "http://feeds.bbci.co.uk/... - do not include feed.
#
# There are 68 feeds below which constitute the bulk of the available rss
# feeds on the BBC web site. These include 5 blogs by editors and
# correspondants, 16 sports feeds, 15 'sub' regional feeds (Eg. North West
# Wales, Scotland Business), and 7 Welsh language feeds.
#
# Some of the feeds are low volume (Eg. blogs), or very low volume (Eg. Click)
# so if "oldest_article = 1.5" (only articles published in the last 36 hours)
# you may get some 'empty feeds' which will not then be included in the ebook.
#
# The 15 feeds currently selected below are simply my default ones.
#
# Note: With all 68 feeds selected, oldest_article set to 2,
# max_articles_per_feed set to 100, and simultaneous_downloads set to 10,
# the ebook creation took 29 minutes on my speedy 100 mbps net connection,
# fairly high-end desktop PC running Linux (Ubuntu Lucid-Lynx).
# More realistically with 15 feeds selected, oldest_article set to 1.5,
# max_articles_per_feed set to 100, and simultaneous_downloads set to 20,
# it took 6 minutes. If that's too slow increase 'simultaneous_downloads'.
#
# Select / de-select the feeds you want in your ebook.
# feeds = [
("News Home", "http://feeds.bbci.co.uk/news/rss.xml"),
("UK", "http://feeds.bbci.co.uk/news/uk/rss.xml"),
("World", "http://feeds.bbci.co.uk/news/world/rss.xml"),
#("England", "http://feeds.bbci.co.uk/news/england/rss.xml"),
#("Scotland", "http://feeds.bbci.co.uk/news/scotland/rss.xml"),
#("Wales", "http://feeds.bbci.co.uk/news/wales/rss.xml"),
#("N. Ireland", "http://feeds.bbci.co.uk/news/northern_ireland/rss.xml"),
#("Africa", "http://feeds.bbci.co.uk/news/world/africa/rss.xml"),
#("Asia", "http://feeds.bbci.co.uk/news/world/asia/rss.xml"),
#("Europe", "http://feeds.bbci.co.uk/news/world/europe/rss.xml"),
#("Latin America", "http://feeds.bbci.co.uk/news/world/latin_america/rss.xml"),
#("Middle East", "http://feeds.bbci.co.uk/news/world/middle_east/rss.xml"),
("US & Canada", "http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml"),
("Politics", "http://feeds.bbci.co.uk/news/politics/rss.xml"),
("Science/Environment", "http://feeds.bbci.co.uk/news/science_and_environment/rss.xml"),
("Technology", "http://feeds.bbci.co.uk/news/technology/rss.xml"),
("Magazine", "http://feeds.bbci.co.uk/news/magazine/rss.xml"),
("Entertainment/Arts", "http://feeds.bbci.co.uk/news/entertainment_and_arts/rss.xml"),
#("Health", "http://feeds.bbci.co.uk/news/health/rss.xml"),
#("Education/Family", "http://feeds.bbci.co.uk/news/education/rss.xml"),
("Business", "http://feeds.bbci.co.uk/news/business/rss.xml"),
("Special Reports", "http://feeds.bbci.co.uk/news/special_reports/rss.xml"),
("Also in the News", "http://feeds.bbci.co.uk/news/also_in_the_news/rss.xml"),
#("Newsbeat", "http://www.bbc.co.uk/newsbeat/rss.xml"),
#("Click", "http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/programmes/click_online/rss.xml"),
("Blog: Nick Robinson (Political Editor)", "http://feeds.bbci.co.uk/news/correspondents/nickrobinson/rss.sxml"),
#("Blog: Mark D'Arcy (Parliamentary Correspondent)", "http://feeds.bbci.co.uk/news/correspondents/markdarcy/rss.sxml"),
#("Blog: Robert Peston (Business Editor)", "http://feeds.bbci.co.uk/news/correspondents/robertpeston/rss.sxml"),
#("Blog: Stephanie Flanders (Economics Editor)", "http://feeds.bbci.co.uk/news/correspondents/stephanieflanders/rss.sxml"),
("Blog: Rory Cellan-Jones (Technology correspondent)", "http://feeds.bbci.co.uk/news/correspondents/rorycellanjones/rss.sxml"),
("Sport Front Page", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/front_page/rss.xml"),
#("Football", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/football/rss.xml"),
#("Cricket", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/cricket/rss.xml"),
#("Rugby Union", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/rugby_union/rss.xml"),
#("Rugby League", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/rugby_league/rss.xml"),
#("Tennis", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/tennis/rss.xml"),
#("Golf", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/golf/rss.xml"),
#("Motorsport", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/motorsport/rss.xml"),
#("Boxing", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/boxing/rss.xml"),
#("Athletics", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/athletics/rss.xml"),
#("Snooker", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/snooker/rss.xml"),
#("Horse Racing", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/horse_racing/rss.xml"),
#("Cycling", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/cycling/rss.xml"),
#("Disability Sport", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/disability_sport/rss.xml"),
#("Other Sport", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/rss.xml"),
#("Olympics 2012", "http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/olympics_2012/rss.xml"),
#("N. Ireland Politics", "http://feeds.bbci.co.uk/news/northern_ireland/northern_ireland_politics/rss.xml"),
#("Scotland Politics", "http://feeds.bbci.co.uk/news/scotland/scotland_politics/rss.xml"),
#("Scotland Business", "http://feeds.bbci.co.uk/news/scotland/scotland_business/rss.xml"),
#("E. Scotland, Edinburgh & Fife", "http://feeds.bbci.co.uk/news/scotland/edinburgh_east_and_fife/rss.xml"),
#("W. Scotland & Glasgow", "http://feeds.bbci.co.uk/news/scotland/glasgow_and_west/rss.xml"),
#("Highlands & Islands", "http://feeds.bbci.co.uk/news/scotland/highlands_and_islands/rss.xml"),
#("NE. Scotland, Orkney & Shetland", "http://feeds.bbci.co.uk/news/scotland/north_east_orkney_and_shetland/rss.xml"),
#("South Scotland", "http://feeds.bbci.co.uk/news/scotland/south_scotland/rss.xml"),
#("Central Scotland & Tayside", "http://feeds.bbci.co.uk/news/scotland/tayside_and_central/rss.xml"),
#("Wales Politics", "http://feeds.bbci.co.uk/news/wales/wales_politics/rss.xml"),
#("NW. Wales", "http://feeds.bbci.co.uk/news/wales/north_west_wales/rss.xml"),
#("NE. Wales", "http://feeds.bbci.co.uk/news/wales/north_east_wales/rss.xml"),
#("Mid. Wales", "http://feeds.bbci.co.uk/news/wales/mid_wales/rss.xml"),
#("SW. Wales", "http://feeds.bbci.co.uk/news/wales/south_west_wales/rss.xml"),
#("SE. Wales", "http://feeds.bbci.co.uk/news/wales/south_east_wales/rss.xml"),
#("Newyddion - News in Welsh", "http://feeds.bbci.co.uk/newyddion/rss.xml"),
#("Gwleidyddiaeth", "http://feeds.bbci.co.uk/newyddion/gwleidyddiaeth/rss.xml"),
#("Gogledd-Ddwyrain", "http://feeds.bbci.co.uk/newyddion/gogledd-ddwyrain/rss.xml"),
#("Gogledd-Orllewin", "http://feeds.bbci.co.uk/newyddion/gogledd-orllewin/rss.xml"),
#("Canolbarth", "http://feeds.bbci.co.uk/newyddion/canolbarth/rss.xml"),
#("De-Ddwyrain", "http://feeds.bbci.co.uk/newyddion/de-ddwyrain/rss.xml"),
#("De-Orllewin", "http://feeds.bbci.co.uk/newyddion/de-orllewin/rss.xml"),
]

# **** SELECT YOUR USER PREFERENCES ****

# Title to use for the ebook.
# title = 'BBC News'

# A brief description for the ebook.
# description = u'BBC web site ebook created using rss feeds.'

# The max number of articles which may be downloaded from each feed.
# I've never seen more than about 70 articles in a single feed in the
# BBC feeds.
# max_articles_per_feed = 100

# The max age of articles which may be downloaded from each feed. This is
# specified in days - note fractions of days are allowed, Eg. 2.5 (2 and a
# half days). My default of 1.5 days is the last 36 hours, the point at
# which I've decided 'news' becomes 'old news', but be warned this is not
# so good for the blogs, technology, magazine, etc., and sports feeds.
# You may wish to extend this to 2-5 but watch out ebook creation time will
# increase as well. Setting this to 30 will get everything (AFAICT) as long
# as max_articles_per_feed remains set high (except for 'Click' which is
# v. low volume and its currently oldest article is 4th Feb 2011).
# oldest_article = 1.5

# Number of simultaneous downloads. 20 is consistantly working fine on the
# BBC News feeds with no problems. Speeds things up from the defualt of 5.
# If you have a lot of feeds and/or have increased oldest_article above 2
# then you may wish to try increasing simultaneous_downloads to 25-30,
# Or, of course, if you are in a hurry. [I've not tried beyond 20.]
# simultaneous_downloads = 20

# Timeout for fetching files from the server in seconds. The default of
# 120 seconds, seems somewhat excessive.
# timeout = 30

# The format string for the date shown on the ebook's first page.
# List of all values: http://docs.python.org/library/time.html
# Default in news.py has a leading space so that's mirrored here.
# As with 'feeds' select/de-select by adding/removing the initial '#',
# only one timefmt should be selected, here's a few to choose from.
# timefmt = ' [%a, %d %b %Y]' # [Fri, 14 Nov 2011] (Calibre default)
#timefmt = ' [%a, %d %b %Y %H:%M]' # [Fri, 14 Nov 2011 18:30]
#timefmt = ' [%a, %d %b %Y %I:%M %p]' # [Fri, 14 Nov 2011 06:30 PM]
#timefmt = ' [%d %b %Y]' # [14 Nov 2011]
#timefmt = ' [%d %b %Y %H:%M]' # [14 Nov 2011 18.30]
#timefmt = ' [%Y-%m-%d]' # [2011-11-14]
#timefmt = ' [%Y-%m-%d-%H-%M]' # [2011-11-14-18-30]

#
# **** IMPORTANT ****
#
# DO NOT EDIT BELOW HERE UNLESS YOU KNOW WHAT YOU ARE DOING.
#
# DO NOT EDIT BELOW HERE UNLESS YOU KNOW WHAT YOU ARE DOING.
#
# I MEAN IT, YES I DO, ABSOLUTELY, AT YOU OWN RISK. :)
#
# **** IMPORTANT ****
#

# Author of this recipe.
__author__ = 'mattst'

# Specify English as the language of the RSS feeds (ISO-639 code).
language = 'en_GB'

# Set tags.
tags = 'news, sport, blog'

# Set publisher and publication type.
publisher = 'BBC'
publication_type = 'newspaper'

# Disable stylesheets from site.
no_stylesheets = True

# Specifies an override encoding for sites that have an incorrect charset
# specified. Default of 'None' says to auto-detect. Some other BBC recipes
# use 'utf8', which works fine (so use that if necessary) but auto-detecting
# with None is working fine, so stick with that for robustness.
encoding = None

# Sets whether a feed has full articles embedded in it. The BBC feeds do not.
use_embedded_content = False

# Removes empty feeds - why keep them!?
remove_empty_feeds = True

# Create a custom title which fits nicely in the Kindle title list.
# Requires "import time" above class declaration, and replacing
# title with custom_title in conversion_options (right column only).
# Example of string below: "BBC News - 14 Nov 2011"
#
# custom_title = "BBC News - " + time.strftime('%d %b %Y')

'''
# Conversion options for advanced users, but don't forget to comment out the
# current conversion_options below. Avoid setting 'linearize_tables' as that
# plays havoc with the 'old style' table based pages.
# conversion_options = { 'title' : title,
'comments' : description,
'tags' : tags,
'language' : language,
'publisher' : publisher,
'authors' : publisher,
'smarten_punctuation' : True
}
'''

conversion_options = { 'smarten_punctuation' : True }

# Specify extra CSS - overrides ALL other CSS (IE. Added last).
extra_css = 'body { font-family: verdana, helvetica, sans-serif; } \
.introduction, .first { font-weight: bold; } \
.cross-head { font-weight: bold; font-size: 125%; } \
.cap, .caption { display: block; font-size: 80%; font-style: italic; } \
.cap, .caption, .caption img, .caption span { display: block; text-align: center; margin: 5px auto; } \
.byl, .byd, .byline img, .byline-name, .byline-title, .author-name, .author-position, \
.correspondent-portrait img, .byline-lead-in, .name, .bbc-role { display: block; \ text-align: center; font-size: 80%; font-style: italic; margin: 1px auto; } \
.story-date, .published { font-size: 80%; } \ table { width: 100%; } \ td img { display: block; margin: 5px auto; } \ ul { padding-top: 10px; } \ ol { padding-top: 10px; } \ li { padding-top: 5px; padding-bottom: 5px; } \ h1 { text-align: center; font-size: 175%; font-weight: bold; } \ h2 { text-align: center; font-size: 150%; font-weight: bold; } \ h3 { text-align: center; font-size: 125%; font-weight: bold; } \ h4, h5, h6 { text-align: center; font-size: 100%; font-weight: bold; }'

# Remove various tag attributes to improve the look of the ebook pages.
remove_attributes = [ 'border', 'cellspacing', 'align', 'cellpadding', 'colspan',
'valign', 'vspace', 'hspace', 'alt', 'width', 'height' ]

# Remove the (admittedly rarely used) line breaks, "<br />", which sometimes
# cause a section of the ebook to start in an unsightly fashion or, more
# frequently, a "<br />" will muck up the formatting of a correspondant's byline.
# "<br />" and "<br clear/>" are far more frequently used on the table formatted
# style of pages, and really spoil the look of the ebook pages.
preprocess_regexps = [(re.compile(r'<br[ ]*/>', re.IGNORECASE), lambda m: ''),
(re.compile(r'<br[ ]*clear.*/>', re.IGNORECASE), lambda m: '')]

# Create regular expressions for tag keeping and removal to make the matches more
# robust against minor changes and errors in the HTML, Eg. double spaces, leading
# and trailing spaces, missing hyphens, and such like.
# Python regular expression ('re' class) page: http://docs.python.org/library/re.html

# ***************************************
# Regular expressions for keep_only_tags:
# ***************************************

# The BBC News HTML pages use variants of 'storybody' to denote the section of a HTML
# page which contains the main text of the article. Match storybody variants: 'storybody',
# 'story-body', 'story body','storybody ', etc.
storybody_reg_exp = '^.*story[_ -]*body.*$'

# The BBC sport and 'newsbeat' (features) HTML pages use 'blq_content' to hold the title
# and published date. This is one level above the usual news pages which have the title
# and date within 'story-body'. This is annoying since 'blq_content' must also be kept,
# resulting in a lot of extra things to be removed by remove_tags.
blq_content_reg_exp = '^.*blq[_ -]*content.*$'

# The BBC has an alternative page design structure, which I suspect is an out-of-date
# design but which is still used in some articles, Eg. 'Click' (technology), 'FastTrack'
# (travel), and in some sport pages. These alternative pages are table based (which is
# why I think they are an out-of-date design) and account for -I'm guesstimaking- less
# than 1% of all articles. They use a table class 'storycontent' to hold the article
# and like blq_content (above) have required lots of extra removal by remove_tags.
story_content_reg_exp = '^.*story[_ -]*content.*$'

# Keep the sections of the HTML which match the list below. The HTML page created by
# Calibre will fill <body> with those sections which are matched. Note that the
# blq_content_reg_exp must be listed before storybody_reg_exp in keep_only_tags due to
# it being the parent of storybody_reg_exp, that is to say the div class/id 'story-body'
# will be inside div class/id 'blq_content' in the HTML (if 'blq_content' is there at
# all). If they are the other way around in keep_only_tags then blq_content_reg_exp
# will end up being discarded.
keep_only_tags = [ dict(name='table', attrs={'class':re.compile(story_content_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(blq_content_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'id':re.compile(blq_content_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(storybody_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'id':re.compile(storybody_reg_exp, re.IGNORECASE)}) ]

# ************************************
# Regular expressions for remove_tags:
# ************************************

# Regular expression to remove share-help and variant tags. The share-help class
# is used by the site for a variety of 'sharing' type links, Eg. Facebook, delicious,
# twitter, email. Removed to avoid page clutter.
share_help_reg_exp = '^.*share[_ -]*help.*$'

# Regular expression to remove embedded-hyper and variant tags. This class is used to
# display links to other BBC News articles on the same/similar subject.
embedded_hyper_reg_exp = '^.*embed*ed[_ -]*hyper.*$'

# Regular expression to remove hypertabs and variant tags. This class is used to
# display a tab bar at the top of an article which allows the user to switch to
# an article (viewed on the same page) providing further info., 'in depth' analysis,
# an editorial, a correspondant's blog entry, and such like. The ability to handle
# a tab bar of this nature is currently beyond the scope of this recipe and
# possibly of Calibre itself (not sure about that - TO DO - check!).
hypertabs_reg_exp = '^.*hyper[_ -]*tabs.*$'

# Regular expression to remove story-feature and variant tags. Eg. 'story-feature',
# 'story-feature related narrow', 'story-feature wide', 'story-feature narrow'.
# This class is used to add additional info. boxes, or small lists, outside of
# the main story. TO DO: Work out a way to incorporate these neatly.
story_feature_reg_exp = '^.*story[_ -]*feature.*$'

# Regular expression to remove video and variant tags, Eg. 'videoInStoryB',
# 'videoInStoryC'. This class is used to embed video.
video_reg_exp = '^.*video.*$'

# Regular expression to remove audio and variant tags, Eg. 'audioInStoryD'.
# This class is used to embed audio.
audio_reg_exp = '^.*audio.*$'

# Regular expression to remove pictureGallery and variant tags, Eg. 'pictureGallery'.
# This class is used to embed a photo slideshow. See also 'slideshow' below.
picture_gallery_reg_exp = '^.*picture.*$'

# Regular expression to remove slideshow and variant tags, Eg. 'dslideshow-enclosure'.
# This class is used to embed a slideshow (not necessarily photo) but both
# 'slideshow' and 'pictureGallery' are used for slideshows.
slideshow_reg_exp = '^.*slide[_ -]*show.*$'

# Regular expression to remove social-links and variant tags. This class is used to
# display links to a BBC bloggers main page, used in various columnist's blogs
# (Eg. Nick Robinson, Robert Preston).
social_links_reg_exp = '^.*social[_ -]*links.*$'

# Regular expression to remove quote and (multi) variant tags, Eg. 'quote',
# 'endquote', 'quote-credit', 'quote-credit-title', etc. These are usually
# removed by 'story-feature' removal (as they are usually within them), but
# not always. The quotation removed is always (AFAICT) in the article text
# as well but a 2nd copy is placed in a quote tag to draw attention to it.
# The quote class tags may or may not appear in div's.
quote_reg_exp = '^.*quote.*$'

# Regular expression to remove hidden and variant tags, Eg. 'hidden'.
# The purpose of these is unclear, they seem to be an internal link to a
# section within the article, but the text of the link (Eg. 'Continue reading
# the main story') never seems to be displayed anyway. Removed to avoid clutter.
# The hidden class tags may or may not appear in div's.
hidden_reg_exp = '^.*hidden.*$'

# Regular expression to remove comment and variant tags, Eg. 'comment-introduction'.
# Used on the site to display text about registered users entering comments.
comment_reg_exp = '^.*comment.*$'

# Regular expression to remove form and variant tags, Eg. 'comment-form'.
# Used on the site to allow registered BBC users to fill in forms, typically
# for entering comments about an article.
form_reg_exp = '^.*form.*$'

# Extra things to remove due to the addition of 'blq_content' in keep_only_tags.

#<div class="story-actions"> Used on sports pages for 'email' and 'print'.
story_actions_reg_exp = '^.*story[_ -]*actions.*$'

#<div class="bookmark-list"> Used on sports pages instead of 'share-help' (for
# social networking links).
bookmark_list_reg_exp = '^.*bookmark[_ -]*list.*$'

#<div id="secondary-content" class="content-group">
# NOTE: Don't remove class="content-group" that is needed.
# Used on sports pages to link to 'similar stories'.
secondary_content_reg_exp = '^.*secondary[_ -]*content.*$'

#<div id="featured-content" class="content-group">
# NOTE: Don't remove class="content-group" that is needed.
# Used on sports pages to link to pages like 'tables', 'fixtures', etc.
featured_content_reg_exp = '^.*featured[_ -]*content.*$'

#<div id="navigation">
# Used on sports pages to link to pages like 'tables', 'fixtures', etc.
# Used sometimes instead of "featured-content" above.
navigation_reg_exp = '^.*navigation.*$'

#<a class="skip" href="#blq-container-inner">Skip to top</a>
# Used on sports pages to link to the top of the page.
skip_reg_exp = '^.*skip.*$'

# Extra things to remove due to the addition of 'storycontent' in keep_only_tags,
# which are the alterative table design based pages. The purpose of some of these
# is not entirely clear from the pages (which are a total mess!).

# Remove mapping based tags, Eg. <map id="world_map">
# The dynamic maps don't seem to work during ebook creation. TO DO: Investigate.
map_reg_exp = '^.*map.*$'

# Remove social bookmarking variation, called 'socialBookMarks'.
social_bookmarks_reg_exp = '^.*social[_ -]*bookmarks.*$'

# Remove page navigation tools, like 'search', 'email', 'print', called 'blq-mast'.
blq_mast_reg_exp = '^.*blq[_ -]*mast.*$'

# Remove 'sharesb', I think this is a generic 'sharing' class. It seems to appear
# alongside 'socialBookMarks' whenever that appears. I am removing it as well
# under the assumption that it can appear alone as well.
sharesb_reg_exp = '^.*sharesb.*$'

# Remove class 'o'. The worst named user created css class of all time. The creator
# should immediately be fired. I've seen it used to hold nothing at all but with
# 20 or so empty lines in it. Also to hold a single link to another article.
# Whatever it was designed to do it is not wanted by this recipe. Exact match only.
o_reg_exp = '^o$'

# Remove 'promotopbg' and 'promobottombg', link lists. Have decided to
# use two reg expressions to make removing this (and variants) robust.
promo_top_reg_exp = '^.*promotopbg.*$'
promo_bottom_reg_exp = '^.*promobottombg.*$'

# Remove 'nlp', provides heading for link lists. Requires an exact match due to
# risk of matching those letters in something needed, unless I see a variation
# of 'nlp' used at a later date.
nlp_reg_exp = '^nlp$'

# Remove 'mva', provides embedded floating content of various types. Variant 'mvb'
# has also now been seen. Requires an exact match of 'mva' or 'mvb' due to risk of
# matching those letters in something needed.
mva_or_mvb_reg_exp = '^mv[ab]$'

# Remove 'mvtb', seems to be page navigation tools, like 'blq-mast'.
mvtb_reg_exp = '^mvtb$'

# Remove 'blq-toplink', class to provide a link to the top of the page.
blq_toplink_reg_exp = '^.*blq[_ -]*top[_ -]*link.*$'

# Remove 'products and services' links, Eg. desktop tools, alerts, and so on.
# Eg. Class="servicev4 ukfs_services" - what a mess of a name. Have decided to
# use two reg expressions to make removing this (and variants) robust.
prods_services_01_reg_exp = '^.*servicev4.*$'
prods_services_02_reg_exp = '^.*ukfs[_ -]*services.*$'

# Remove -what I think is- some kind of navigation tools helper class, though I am
# not sure, it's called: 'blq-rst blq-new-nav'. What I do know is it pops up
# frequently and it is not wanted. Have decided to use two reg expressions to make
# removing this (and variants) robust.
blq_misc_01_reg_exp = '^.*blq[_ -]*rst.*$'
blq_misc_02_reg_exp = '^.*blq[_ -]*new[_ -]*nav.*$'

# Remove 'puffbox' - this may only appear inside 'storyextra', so it may not
# need removing - I have no clue what it does other than it contains links.
# Whatever it is - it is not part of the article and is not wanted.
puffbox_reg_exp = '^.*puffbox.*$'

# Remove 'sibtbg' and 'sibtbgf' - some kind of table formatting classes.
sibtbg_reg_exp = '^.*sibtbg.*$'

# Remove 'storyextra' - links to relevant articles and external sites.
storyextra_reg_exp = '^.*story[_ -]*extra.*$'

remove_tags = [ dict(name='div', attrs={'class':re.compile(story_feature_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(share_help_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(embedded_hyper_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(hypertabs_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(video_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(audio_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(picture_gallery_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(slideshow_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(quote_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(hidden_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(comment_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(story_actions_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(bookmark_list_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'id':re.compile(secondary_content_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'id':re.compile(featured_content_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'id':re.compile(navigation_reg_exp, re.IGNORECASE)}), dict(name='form', attrs={'id':re.compile(form_reg_exp, re.IGNORECASE)}), dict(attrs={'class':re.compile(quote_reg_exp, re.IGNORECASE)}), dict(attrs={'class':re.compile(hidden_reg_exp, re.IGNORECASE)}), dict(attrs={'class':re.compile(social_links_reg_exp, re.IGNORECASE)}), dict(attrs={'class':re.compile(comment_reg_exp, re.IGNORECASE)}), dict(attrs={'class':re.compile(skip_reg_exp, re.IGNORECASE)}), dict(name='map', attrs={'id':re.compile(map_reg_exp, re.IGNORECASE)}), dict(name='map', attrs={'name':re.compile(map_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'id':re.compile(social_bookmarks_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'id':re.compile(blq_mast_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(sharesb_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(o_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(promo_top_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(promo_bottom_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(nlp_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(mva_or_mvb_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(mvtb_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(blq_toplink_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(prods_services_01_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(prods_services_02_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(blq_misc_01_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(blq_misc_02_reg_exp, re.IGNORECASE)}), dict(name='div', attrs={'class':re.compile(puffbox_reg_exp, re.IGNORECASE)}), dict(attrs={'class':re.compile(sibtbg_reg_exp, re.IGNORECASE)}), dict(attrs={'class':re.compile(storyextra_reg_exp, re.IGNORECASE)})
]

# Uses url to create and return the 'printer friendly' version of the url.
# In other words the 'print this page' address of the page.
#
# There are 3 types of urls used in the BBC site's rss feeds. There is just
# 1 type for the standard news while there are 2 used for sports feed urls.
# Note: Sports urls are linked from regular news feeds (Eg. 'News Home') when
# there is a major story of interest to 'everyone'. So even if no BBC sports
# feeds are added to 'feeds' the logic of this method is still needed to avoid
# blank / missing / empty articles which have an index title and then no body.
def print_version(self, url):

# Handle sports page urls type 01:
if (url.find("go/rss/-/sport1/") != -1):
temp_url = url.replace("go/rss/-/", "")

# Handle sports page urls type 02:
elif (url.find("go/rss/int/news/-/sport1/") != -1):
temp_url = url.replace("go/rss/int/news/-/", "")

# Handle regular news page urls:
else:
temp_url = url.replace("go/rss/int/news/-/", "")

# Always add "?print=true" to the end of the url.
print_url = temp_url + "?print=true"

return print_url

# Remove articles in feeds based on a string in the article title or url.
#
# Code logic written by: Starson17 - posted in: "Recipes - Re-usable code"
# thread, in post with title: "Remove articles from feed", see url:
# http://www.mobileread.com/forums/showpost.php?p=1165462&postcount=6
# Many thanks and all credit to Starson17.
#
# Starson17's code has obviously been altered to suite my requirements.
def parse_feeds(self):

# Call parent's method.
feeds = BasicNewsRecipe.parse_feeds(self)

# Loop through all feeds.
for feed in feeds:

# Loop through all articles in feed.
for article in feed.articles[:]:

# Match key words and remove article if there's a match.

# Most BBC rss feed video only 'articles' use upper case 'VIDEO'
# as a title prefix. Just match upper case 'VIDEO', so that
# articles like 'Video game banned' won't be matched and removed.
if 'VIDEO' in article.title:
feed.articles.remove(article)

# Most BBC rss feed audio only 'articles' use upper case 'AUDIO'
# as a title prefix. Just match upper case 'AUDIO', so that
# articles like 'Hi-Def audio...' won't be matched and removed.
elif 'AUDIO' in article.title:
feed.articles.remove(article)

# Most BBC rss feed photo slideshow 'articles' use 'In Pictures',
# 'In pictures', and 'in pictures', somewhere in their title.
# Match any case of that phrase.
elif 'IN PICTURES' in article.title.upper():
feed.articles.remove(article)

# As above, but user contributed pictures. Match any case.
elif 'YOUR PICTURES' in article.title.upper():
feed.articles.remove(article)

# 'Sportsday Live' are articles which contain a constantly and
# dynamically updated 'running commentary' during a live sporting
# event. Match any case.
elif 'SPORTSDAY LIVE' in article.title.upper():
feed.articles.remove(article)

# Sometimes 'Sportsday Live' (above) becomes 'Live - Sport Name'.
# These are being matched below using 'Live - ' because removing all
# articles with 'live' in their titles would remove some articles
# that are in fact not live sports pages. Match any case.
elif 'LIVE - ' in article.title.upper():
feed.articles.remove(article)

# 'Quiz of the week' is a Flash player weekly news quiz. Match only
# the 'Quiz of the' part in anticipation of monthly and yearly
# variants. Match any case.
elif 'QUIZ OF THE' in article.title.upper():
feed.articles.remove(article)

# Remove articles with 'scorecards' in the url. These are BBC sports
# pages which just display a cricket scorecard. The pages have a mass
# of table and css entries to display the scorecards nicely. Probably
# could make them work with this recipe, but might take a whole day
# of work to sort out all the css - basically a formatting nightmare.
elif 'scorecards' in article.url:
feed.articles.remove(article)

return feeds

# End of class and file.

This recipe explores only the tip of the iceberg when it comes to the power of calibre. To explore more of the abilities of calibre we’ll examine a more complex real life example in the next section.

Real life example

A reasonably complex real life example that exposes more of the API of BasicNewsRecipe is the recipe for The New York Times
import string, re from calibre import strftime from calibre.web.feeds.recipes import BasicNewsRecipe from calibre.ebooks.BeautifulSoup import BeautifulSoup class NYTimes(BasicNewsRecipe):

title = 'The New York Times'
__author__ = 'Kovid Goyal'
description = 'Daily news from the New York Times'
timefmt = ' [%a, %d %b, %Y]'
needs_subscription = True remove_tags_before = dict(id='article')
remove_tags_after = dict(id='article')
remove_tags = [dict(attrs={'class':['articleTools', 'post-tools', 'side_tool', 'nextArticleLink clearfix']}), dict(id=['footer', 'toolsRight', 'articleInline', 'navigation', 'archive', 'side_search', 'blog_sidebar', 'side_tool', 'side_index']), dict(name=['script', 'noscript', 'style'])]
encoding = 'cp1252'
no_stylesheets = True extra_css = 'h1 {font: sans-serif large;}\n.byline {font:monospace;}'

def get_browser(self):
br = BasicNewsRecipe.get_browser()
if self.username is not None and self.password is not None:
br.open('http://www.nytimes.com/auth/login')
br.select_form(name='login')
br['USERID'] = self.username br['PASSWORD'] = self.password br.submit()
return br def parse_index(self):
soup = self.index_to_soup('http://www.nytimes.com/pages/todayspaper/index.html')

def feed_title(div):
return ''.join(div.findAll(text=True, recursive=False)).strip()

articles = {}
key = None ans = []
for div in soup.findAll(True, attrs={'class':['section-headline', 'story', 'story headline']}):

if div['class'] == 'section-headline':
key = string.capwords(feed_title(div))
articles[key] = []
ans.append(key)

elif div['class'] in ['story', 'story headline']:
a = div.find('a', href=True)
if not a:
continue url = re.sub(r'\?.*', '', a['href'])
url += '?pagewanted=all'
title = self.tag_to_string(a, use_alt=True).strip()
description = ''
pubdate = strftime('%a, %d %b')
summary = div.find(True, attrs={'class':'summary'})
if summary:
description = self.tag_to_string(summary, use_alt=False)

feed = key if key is not None else 'Uncategorized'
if not articles.has_key(feed):
articles[feed] = []
if not 'podcasts' in url:
articles[feed].append( dict(title=title, url=url, date=pubdate, description=description, content=''))
ans = self.sort_index_by(ans, {'The Front Page':-1, 'Dining In, Dining Out':1, 'Obituaries':2})
ans = [(key, articles[key]) for key in ans if articles.has_key(key)]
return ans def preprocess_html(self, soup):
refresh = soup.find('meta', {'http-equiv':'refresh'})
if refresh is None:
return soup content = refresh.get('content').partition('=')[2]
raw = self.browser.open('http://www.nytimes.com'+content).read()
return BeautifulSoup(raw.decode('cp1252', 'replace'))

We see several new features in this recipe. First, we have:

timefmt = ' [%a, %d %b, %Y]'

This sets the displayed time on the front page of the created ebook to be in the format, Day, Day_Number Month, Year. See timefmt.

Then we see a group of directives to cleanup the downloaded HTML:

remove_tags_before = dict(name='h1')
remove_tags_after = dict(id='footer')
remove_tags = ...

These remove everything before the first <h1> tag and everything after the first tag whose id is footer. See remove_tags, remove_tags_before, remove_tags_after.

The next interesting feature is:

needs_subscription = True
...
def get_browser(self):
...

needs_subscription = True tells calibre that this recipe needs a username and password in order to access the content. This causes, calibre to ask for a username and password whenever you try to use this recipe. The code in calibre.web.feeds.news.BasicNewsRecipe.get_browser() actually does the login into the NYT website. Once logged in, calibre will use the same, logged in, browser instance to fetch all content. See mechanize [http://wwwsearch.sourceforge.net/mechanize/] to understand the code in get_browser.

The next new feature is the calibre.web.feeds.news.BasicNewsRecipe.parse_index() method. Its job is to go to http://www.nytimes.com/pages/todayspaper/index.html and fetch the list of articles that appear in todays paper. While more complex than simply using RSS, the recipe creates an ebook that corresponds very closely to the days paper. parse_index makes heavy use of BeautifulSoup [http://www.crummy.com/software/BeautifulSoup/documentation.html] to parse the daily paper webpage.

The final new feature is the calibre.web.feeds.news.BasicNewsRecipe.preprocess_html() method. It can be used to perform arbitrary transformations on every downloaded HTML page. Here it is used to bypass the ads that the nytimes shows you before each article.

Tips for developing new recipes

The best way to develop new recipes is to use the command line interface. Create the recipe using your favorite python editor and save it to a file say myrecipe.recipe. The .recipe extension is required. You can download content using this recipe with the command:

ebook-convert myrecipe.recipe .epub --test -vv --debug-pipeline debug

The command ebook-convert will download all the webpages and save them to the EPUB file myrecipe.epub. The -vv makes ebook-convert spit out a lot of information about what it is doing. The --test makes it download only a couple of articles from at most two feeds. In addition, ebook-convert will put the downloaded HTML into the debug/input directory, where debug is the directory you specified in the --debug-pipeline option.

Once the download is complete, you can look at the downloaded HTML by opening the file debug/input/index.html in a browser. Once you’re satisfied that the download and preprocessing is happening correctly, you can generate ebooks in different formats as shown below:

ebook-convert myrecipe.recipe myrecipe.epub
ebook-convert myrecipe.recipe myrecipe.mobi
...

If you’re satisfied with your recipe, and you feel there is enough demand to justify its inclusion into the set of built-in recipes, post your recipe in the calibre recipes forum [http://www.mobileread.com/forums/forumdisplay.php?f=228] to share it with other calibre users.

Note

On OS X, the ebook-convert command will not be available by default. Go to Preferences->Miscellaneous and click the install command line tools button to make it available.

See also

ebook-convert

The command line interface for all ebook conversion.

Further reading

To learn more about writing advanced recipes using some of the facilities, available in BasicNewsRecipe you should consult the following sources:

API Documentation

Documentation of the BasicNewsRecipe class and all its important methods and fields.

BasicNewsRecipe [http://bazaar.launchpad.net/~kovid/calibre/trunk/annotate/head:/src/calibre/web/feeds/news.py]

The source code of BasicNewsRecipe

Built-in recipes [http://bazaar.launchpad.net/~kovid/calibre/trunk/files/head:/recipes/]

The source code for the built-in recipes that come with calibre

The calibre recipes forum [http://www.mobileread.com/forums/forumdisplay.php?f=228]

Lots of knowledgeable calibre recipe writers hang out here.

API documentation

- API Documentation for recipes

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »
- Adding your favorite news website »

API Documentation for recipes

The API for writing recipes is defined by the BasicNewsRecipe
class calibre.web.feeds.news.BasicNewsRecipe(options, log, progress_reporter)[source]

Base class that contains logic needed in all recipes. By overriding progressively more of the functionality in this class, you can make progressively more customized/powerful recipes. For a tutorial introduction to creating recipes, see Adding your favorite news website.

abort_recipe_processing(msg)[source]

Causes the recipe download system to abort the download of this recipe, displaying a simple feedback message to the user.

add_toc_thumbnail(article, src)[source]

Call this from populate_article_metadata with the src attribute of an
<img> tag from the article that is appropriate for use as the thumbnail representing the article in the Table of Contents. Whether the thumbnail is actually used is device dependent (currently only used by the Kindles). Note that the referenced image must be one that was successfully downloaded, otherwise it will be ignored.

classmethod adeify_images(soup)[source]

If your recipe when converted to EPUB has problems with images when viewed in Adobe Digital Editions, call this method from within postprocess_html().

cleanup()[source]

Called after all articles have been download. Use it to do any cleanup like logging out of subscription sites, etc.

clone_browser(br)[source]

Clone the browser br. Cloned browsers are used for multi-threaded downloads, since mechanize is not thread safe. The default cloning routines should capture most browser customization, but if you do something exotic in your recipe, you should override this method in your recipe and clone manually.

Cloned browser instances use the same, thread-safe CookieJar by default, unless you have customized cookie handling.

default_cover(cover_file)[source]

Create a generic cover for recipes that dont have a cover

download()[source]

Download and pre-process all articles from the feeds in this recipe.
This method should be called only once on a particular Recipe instance.
Calling it more than once will lead to undefined behavior.
:return: Path to index.html

extract_readable_article(html, url)[source]

Extracts main article content from ‘html’, cleans up and returns as a (article_html, extracted_title) tuple.
Based on the original readability algorithm by Arc90.

get_article_url(article)[source]

Override in a subclass to customize extraction of the URL that points to the content for each article. Return the article URL. It is called with article, an object representing a parsed article from a feed. See feedparser [http://packages.python.org/feedparser/].
By default it looks for the original link (for feeds syndicated via a service like feedburner or pheedo) and if found, returns that or else returns article.link [http://packages.python.org/feedparser/reference-entry-link.html].

classmethod get_browser(*args, **kwargs)[source]

Return a browser instance used to fetch documents from the web. By default it returns a mechanize [http://wwwsearch.sourceforge.net/mechanize/] browser instance that supports cookies, ignores robots.txt, handles refreshes and has a mozilla firefox user agent.

If your recipe requires that you login first, override this method in your subclass. For example, the following code is used in the New York
Times recipe to login for full access:

def get_browser(self):
br = BasicNewsRecipe.get_browser()
if self.username is not None and self.password is not None:
br.open('http://www.nytimes.com/auth/login')
br.select_form(name='login')
br['USERID'] = self.username
br['PASSWORD'] = self.password
br.submit()
return br

get_cover_url()[source]

Return a URL to the cover image for this issue or None.
By default it returns the value of the member self.cover_url which is normally None. If you want your recipe to download a cover for the e-book override this method in your subclass, or set the member variable self.cover_url before this method is called.

get_feeds()[source]

Return a list of RSS feeds to fetch for this profile. Each element of the list must be a 2-element tuple of the form (title, url). If title is None or an empty string, the title from the feed is used. This method is useful if your recipe needs to do some processing to figure out the list of feeds to download. If so, override in your subclass.

get_masthead_title()[source]

Override in subclass to use something other than the recipe title

get_masthead_url()[source]

Return a URL to the masthead image for this issue or None.
By default it returns the value of the member self.masthead_url which is normally None. If you want your recipe to download a masthead for the e-book override this method in your subclass, or set the member variable self.masthead_url before this method is called.
Masthead images are used in Kindle MOBI files.

get_obfuscated_article(url)[source]

If you set articles_are_obfuscated this method is called with every article URL. It should return the path to a file on the filesystem that contains the article HTML. That file is processed by the recursive
HTML fetching engine, so it can contain links to pages/images on the web.

This method is typically useful for sites that try to make it difficult to access article content automatically.

classmethod image_url_processor(baseurl, url)[source]

Perform some processing on image urls (perhaps removing size restrictions for dynamically generated images, etc.) and return the precessed URL.

index_to_soup(url_or_raw, raw=False)[source]

Convenience method that takes an URL to the index page and returns a BeautifulSoup [http://www.crummy.com/software/BeautifulSoup/documentation.html] of it.

url_or_raw: Either a URL or the downloaded index page as a string

is_link_wanted(url, tag)[source]

Return True if the link should be followed or False otherwise. By default, raises NotImplementedError which causes the downloader to ignore it.

Parameters:

- url – The URL to be followed

- tag – The Tag from which the URL was derived

parse_feeds()[source]

Create a list of articles from the list of feeds returned by BasicNewsRecipe.get_feeds().
Return a list of Feed objects.

parse_index()[source]

This method should be implemented in recipes that parse a website instead of feeds to generate a list of articles. Typical uses are for news sources that have a “Print Edition” webpage that lists all the articles in the current print edition. If this function is implemented, it will be used in preference to BasicNewsRecipe.parse_feeds().

It must return a list. Each element of the list must be a 2-element tuple of the form ('feed title', list of articles).

Each list of articles must contain dictionaries of the form:

{
'title' : article title,
'url' : URL of print version,
'date' : The publication date of the article as a string,
'description' : A summary of the article
'content' : The full article (can be an empty string). Obsolete
do not use, instead save the content to a temporary
file and pass a file:///path/to/temp/file.html as
the URL.
}

For an example, see the recipe for downloading The Atlantic.
In addition, you can add ‘author’ for the author of the article.

If you want to abort processing for some reason and have calibre show the user a simple message instead of an error, call abort_recipe_processing().

populate_article_metadata(article, soup, first)[source]

Called when each HTML page belonging to article is downloaded.
Intended to be used to get article metadata like author/summary/etc.
from the parsed HTML (soup).
:param article: A object of class calibre.web.feeds.Article.
If you change the summary, remember to also change the text_summary
:param soup: Parsed HTML belonging to this article
:param first: True iff the parsed HTML is the first page of the article.

postprocess_book(oeb, opts, log)[source]

Run any needed post processing on the parsed downloaded e-book.

Parameters:

- oeb – An OEBBook object

- opts – Conversion options

postprocess_html(soup, first_fetch)[source]

This method is called with the source of each downloaded HTML file, after it is parsed for links and images.
It can be used to do arbitrarily powerful post-processing on the HTML.
It should return soup after processing it.

Parameters:

- soup –

A BeautifulSoup [http://www.crummy.com/software/BeautifulSoup/documentation.html] instance containing the downloaded HTML.

- first_fetch – True if this is the first page of an article.

preprocess_html(soup)[source]

This method is called with the source of each downloaded HTML file, before it is parsed for links and images. It is called after the cleanup as specified by remove_tags etc.
It can be used to do arbitrarily powerful pre-processing on the HTML.
It should return soup after processing it.

soup: A BeautifulSoup [http://www.crummy.com/software/BeautifulSoup/documentation.html] instance containing the downloaded HTML.

preprocess_raw_html(raw_html, url)[source]

This method is called with the source of each downloaded HTML file, before it is parsed into an object tree. raw_html is a unicode string representing the raw HTML downloaded from the web. url is the URL from which the HTML was downloaded.

Note that this method acts before preprocess_regexps.

This method must return the processed raw_html as a unicode object.

classmethod print_version(url)[source]

Take a url pointing to the webpage with article content and return the URL pointing to the print version of the article. By default does nothing. For example:

def print_version(self, url):
return url + '?&pagewanted=print'

skip_ad_pages(soup)[source]

This method is called with the source of each downloaded HTML file, before any of the cleanup attributes like remove_tags, keep_only_tags are applied. Note that preprocess_regexps will have already been applied.
It is meant to allow the recipe to skip ad pages. If the soup represents an ad page, return the HTML of the real page. Otherwise return
None.

soup: A BeautifulSoup [http://www.crummy.com/software/BeautifulSoup/documentation.html] instance containing the downloaded HTML.

sort_index_by(index, weights)[source]

Convenience method to sort the titles in index according to weights. index is sorted in place. Returns index.

index: A list of titles.

weights: A dictionary that maps weights to titles. If any titles in index are not in weights, they are assumed to have a weight of 0.

classmethod tag_to_string(tag, use_alt=True, normalize_whitespace=True)[source]

Convenience method to take a BeautifulSoup [http://www.crummy.com/software/BeautifulSoup/documentation.html]
Tag and extract the text from it recursively, including any CDATA sections and alt tag attributes. Return a possibly empty unicode string.

use_alt: If True try to use the alt attribute for tags that don’t have any textual content
tag: BeautifulSoup [http://www.crummy.com/software/BeautifulSoup/documentation.html]
Tag

articles_are_obfuscated = False

Set to True and implement get_obfuscated_article() to handle websites that try to make it difficult to scrape content.

auto_cleanup = False

Automatically extract all the text from downloaded article pages. Uses the algorithms from the readability project. Setting this to True, means that you do not have to worry about cleaning up the downloaded HTML manually (though manual cleanup will always be superior).

auto_cleanup_keep = None

Specify elements that the auto cleanup algorithm should never remove
The syntax is a XPath expression. For example:

auto_cleanup_keep = '//div[@id="article-image"]' will keep all divs with
id="article-image"
auto_cleanup_keep = '//*[@class="important"]' will keep all elements
with class="important"
auto_cleanup_keep = '//div[@id="article-image"]|//span[@class="important"]'
will keep all divs with id="article-image" and spans
with class="important"

center_navbar = True

If True the navigation bar is center aligned, otherwise it is left aligned

conversion_options = {}

Recipe specific options to control the conversion of the downloaded content into an e-book. These will override any user or plugin specified values, so only use if absolutely necessary. For example:

conversion_options = {
'base_font_size' : 16,
'tags' : 'mytag1,mytag2',
'title' : 'My Title',
'linearize_tables' : True,
}

cover_margins = (0, 0, '#ffffff')

By default, the cover image returned by get_cover_url() will be used as the cover for the periodical. Overriding this in your recipe instructs calibre to render the downloaded cover into a frame whose width and height are expressed as a percentage of the downloaded cover.
cover_margins = (10, 15, ‘#ffffff’) pads the cover with a white margin
10px on the left and right, 15px on the top and bottom.
Color names defined at http://www.imagemagick.org/script/color.php Note that for some reason, white does not always work on windows. Use
#ffffff instead

delay = 0

Delay between consecutive downloads in seconds. The argument may be a floating point number to indicate a more precise time.

description = u''

A couple of lines that describe the content this recipe downloads.
This will be used primarily in a GUI that presents a list of recipes.

encoding = None

Specify an override encoding for sites that have an incorrect charset specification. The most common being specifying latin1 and using cp1252. If None, try to detect the encoding. If it is a callable, the callable is called with two arguments: The recipe object and the source to be decoded. It must return the decoded source.

extra_css = None

Specify any extra CSS that should be added to downloaded HTML files
It will be inserted into <style> tags, just before the closing </head> tag thereby overriding all CSS except that which is declared using the style attribute on individual HTML tags.
For example:

extra_css = '.heading { font: serif x-large }'

feeds = None

List of feeds to download
Can be either [url1, url2, ...] or [('title1', url1), ('title2', url2),...]

filter_regexps = []

List of regular expressions that determines which links to ignore
If empty it is ignored. Used only if is_link_wanted is not implemented. For example:

filter_regexps = [r'ads\.doubleclick\.net']

will remove all URLs that have ads.doubleclick.net in them.

Only one of BasicNewsRecipe.match_regexps or BasicNewsRecipe.filter_regexps should be defined.

ignore_duplicate_articles = None

Ignore duplicates of articles that are present in more than one section.
A duplicate article is an article that has the same title and/or URL.
To ignore articles with the same title, set this to:
ignore_duplicate_articles = {‘title’}
To use URLs instead, set it to:
ignore_duplicate_articles = {‘url’}
To match on title or URL, set it to:
ignore_duplicate_articles = {‘title’, ‘url’}

keep_only_tags = []

Keep only the specified tags and their children.
For the format for specifying a tag see BasicNewsRecipe.remove_tags.
If this list is not empty, then the <body> tag will be emptied and re-filled with the tags that match the entries in this list. For example:

keep_only_tags = [dict(id=['content', 'heading'])]

will keep only tags that have an id attribute of “content” or “heading”.

language = 'und'

The language that the news is in. Must be an ISO-639 code either two or three characters long

masthead_url = None

By default, calibre will use a default image for the masthead (Kindle only).
Override this in your recipe to provide a url to use as a masthead.

match_regexps = []

List of regular expressions that determines which links to follow
If empty, it is ignored. Used only if is_link_wanted is not implemented. For example:

match_regexps = [r'page=[0-9]+']

will match all URLs that have page=some number in them.

Only one of BasicNewsRecipe.match_regexps or BasicNewsRecipe.filter_regexps should be defined.

max_articles_per_feed = 100

Maximum number of articles to download from each feed. This is primarily useful for feeds that don’t have article dates. For most feeds, you should use BasicNewsRecipe.oldest_article

needs_subscription = False

If True the GUI will ask the user for a username and password to use while downloading
If set to “optional” the use of a username and password becomes optional

no_stylesheets = False

Convenient flag to disable loading of stylesheets for websites that have overly complex stylesheets unsuitable for conversion to ebooks formats
If True stylesheets are not downloaded and processed

oldest_article = 7.0

Oldest article to download from this news source. In days.

preprocess_regexps = []

List of regexp substitution rules to run on the downloaded HTML.
Each element of the list should be a two element tuple. The first element of the tuple should be a compiled regular expression and the second a callable that takes a single match object and returns a string to replace the match. For example:

preprocess_regexps = [
(re.compile(r'<!--Article ends here-->.*</body>', re.DOTALL|re.IGNORECASE),
lambda match: '</body>'),
]

will remove everythong from <!–Article ends here–> to </body>.

publication_type = 'unknown'

Publication type
Set to newspaper, magazine or blog. If set to None, no publication type metadata will be written to the opf file.

recipe_disabled = None

Set to a non empty string to disable this recipe
The string will be used as the disabled message

recursions = 0

Number of levels of links to follow on article webpages

remove_attributes = []

List of attributes to remove from all tags
For example:

remove_attributes = ['style', 'font']

remove_empty_feeds = False

If True empty feeds are removed from the output.
This option has no effect if parse_index is overridden in the sub class. It is meant only for recipes that return a list of feeds using feeds or get_feeds(). It is also used if you use the ignore_duplicate_articles option.

remove_javascript = True

Convenient flag to strip all javascript tags from the downloaded HTML

remove_tags = []

List of tags to be removed. Specified tags are removed from downloaded HTML.
A tag is specified as a dictionary of the form:

{
name : 'tag name', #e.g. 'div'
attrs : a dictionary, #e.g. {class: 'advertisment'}
}

All keys are optional. For a full explanantion of the search criteria, see Beautiful Soup [http://www.crummy.com/software/BeautifulSoup/documentation.html#Thebasicfindmethod:findAll(name,attrs,recursive,text,limit,**kwargs)] A common example:

remove_tags = [dict(name='div', attrs={'class':'advert'})]

This will remove all <div class=”advert”> tags and all their children from the downloaded HTML.

remove_tags_after = None

Remove all tags that occur after the specified tag.
For the format for specifying a tag see BasicNewsRecipe.remove_tags.
For example:

remove_tags_after = [dict(id='content')]

will remove all tags after the first element with id=”content”.

remove_tags_before = None

Remove all tags that occur before the specified tag.
For the format for specifying a tag see BasicNewsRecipe.remove_tags.
For example:

remove_tags_before = dict(id='content')

will remove all tags before the first element with id=”content”.

requires_version = (0, 6, 0)

Minimum calibre version needed to use this recipe

reverse_article_order = False

Reverse the order of articles in each feed

simultaneous_downloads = 5

Number of simultaneous downloads. Set to 1 if the server is picky.
Automatically reduced to 1 if BasicNewsRecipe.delay > 0

summary_length = 500

Max number of characters in the short description

template_css = u'\n .article_date {\n color: gray; font-family: monospace;\n }\n\n .article_description {\n text-indent: 0pt;\n }\n\n a.article {\n font-weight: bold; text-align:left;\n }\n\n a.feed {\n font-weight: bold;\n }\n\n .calibre_navbar {\n font-family:monospace;\n }\n '

The CSS that is used to style the templates, i.e., the navigation bars and the Tables of Contents. Rather than overriding this variable, you should use extra_css in your recipe to customize look and feel.

timefmt = ' [%a, %d %b %Y]'

The format string for the date shown on the first page.
By default: Day_Name, Day_Number Month_Name Year

timeout = 120.0

Timeout for fetching files from server in seconds

title = u'Unknown News Source'

The title to use for the ebook

use_embedded_content = None

Normally we try to guess if a feed has full articles embedded in it based on the length of the embedded content. If None, then the default guessing is used. If True then the we always assume the feeds has embedded content and if False we always assume the feed does not have embedded content.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

The Ebook Viewer
calibre includes a built-in ebook viewer that can view all the major ebook formats.
The viewer is highly customizable and has many advanced features.

- Starting the viewer

- Navigating around an ebook

- Customizing the look and feel of your reading experience

- Dictionary lookup

- Copying text and images

Starting the viewer

You can view any of the books in your calibre library by selecting the book and pressing the View button. This will open up the book in the ebook viewer. You can also launch the viewer by itself from the Start menu in Windows or using the command ebook-viewer in Linux and OS X (you have to install the command line tools on OS X first by going to Preferences->Advanced->Miscellaneous).

Navigating around an ebook

You can “turn pages” in a book by using the Page Next and Page Previous buttons , or by pressing the Page Down/Page Up keys. Unlike most ebook viewers, calibre does not force you to view books in paged mode. You can scroll by amounts less than a page by using the scroll bar or various customizable keyboard shortcuts.

Bookmarks

When you are in the middle of a book and close the viewer, it will remember where you stopped reading and return there the next time you open the book. You can also set bookmarks in the book by using the Bookmark button . When viewing EPUB format books, these bookmarks are actually saved in the EPUB file itself. You can add bookmarks, then send the file to a friend.
When they open the file, they will be able to see your bookmarks.

Table of Contents

If the book you are reading defines a Table of Contents, you can access it by pressing the Table of Contents button .
This will bring up a list of sections in the book. You can click on any of them to jump to that portion of the book.

Navigating by location

Ebooks, unlike paper books, have no concept of pages. Instead, as you read through the book, you will notice that your position in the book is displayed in the upper left corner in a box like this . This is both your current position and the total length of the book. These numbers are independent of the screen size and font size you are viewing the book at, and they play a similar role to page numbers in paper books.
You can enter any number you like to go to the corresponding location in the book.

calibre also has a very handy reference mode. You can turn it on by clicking the Reference Mode button . Once you do this, every time you move your mouse over a paragraph, calibre will display a unique number made up of the section and paragraph numbers.

You can use this number to unambiguously refer to parts of the books when discussing it with friends or referring to it in other works. You can enter these numbers in the box marked Go to at the top of the window to go to a particular reference location.

If you click on links inside the ebook to take you to different parts of the book, such as an endnote, you can use the back and forward buttons in the top left corner to return to where you were. These buttons behave just like those in a web browser.

Customizing the look and feel of your reading experience

You can change font sizes on the fly by using the font size buttons . You can also make the viewer full screen by pressing the Full Screen button . By clicking the Preferences button , you can change the default fonts used by the viewer to ones you like as well as the default font size when the viewer starts up.

More advanced customization can be achieved by the User Stylesheet setting. This is a stylesheet you can set that will be applied to every book. Using it you can do things like have white text on a black background, change paragraph styles, text justification, etc.
For examples of custom stylesheets used by calibre‘s users, see the forums [http://www.mobileread.com/forums/showthread.php?t=51500].

Dictionary lookup

You can look up the meaning of words in the current book by right clicking on a word. calibre uses the publicly available dictionary server at dict.org to look up words. The definition is displayed in a small box at the bottom of the screen.

Copying text and images

You can select text and images by dragging the content with your mouse and then right clicking to copy to the clipboard.
The copied material can be pasted into another application as plain text and images.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

Ebook Conversion
calibre has a conversion system that is designed to be very easy to use. Normally, you just add a book to calibre, click convert and calibre will try hard to generate output that is as close as possible to the input. However, calibre accepts a very large number of input formats, not all of which are as suitable as others for conversion to ebooks. In the case of such input formats, or if you just want greater control over the conversion system, calibre has a lot of options to fine tune the conversion process. Note however that calibre‘s conversion system is not a substitute for a full blown ebook editor. To edit ebooks, I would recommend first converting them to EPUB using calibre and then using a dedicated EPUB editor, like Sigil [http://code.google.com/p/sigil/] to get the book into perfect shape. You can then use the edited EPUB as input for conversion into other formats in calibre.

This document will refer mainly to the conversion settings as found in the conversion dialog, pictured below. All these settings are also available via command line interface to conversion, documented at ebook-convert. In calibre, you can obtain help on any individual setting by holding your mouse over it, a tooltip will appear describing the setting.

Contents

- Introduction
- Look & Feel
- Page Setup
- Heuristic Processing
- Search & Replace
- Structure Detection
- Table of Contents

- Using images as chapter titles when converting HTML input documents

- How options are set/saved for Conversion

- Format specific tips

Introduction

The first thing to understand about the conversion system is that it is designed as a pipeline.
Schematically, it looks like this:

The input format is first converted to XHTML by the appropriate Input Plugin.
This HTML is then transformed. In the last step, the processed XHTML is converted to the specified output format by the appropriate Output Plugin. The results of the conversion can vary greatly, based on the input format. Some formats convert much better than others. A list of the best source formats for conversion is available here.

The transforms that act on the XHTML output are where all the work happens. There are various transforms, for example, to insert book metadata as a page at the start of the book, to detect chapter headings and automatically create a Table of Contents, to proportionally adjust font sizes, et cetera. It is important to remeber that all the transforms act on the
XHTML output by the Input Plugin, not on the input file itself. So, for example, if you ask calibre to convert an RTF file to EPUB, it will first be converted to XHTML internally, the various transforms will be applied to the XHTML and then the Output Plugin will create the EPUB file, automatically generating all metadata, Table of Contents, et cetera.

You can see this process in action by using the debug option . Just specify the path to a directory for the debug output. During conversion, calibre will place the XHTML generated by the various stages of the conversion pipeline in different sub-directories.
The four sub-directories are:

Stages of the conversion pipeline

Directory | Description
input | This contains the HTML output by the Input Plugin. Use this to debug the Input Plugin.
parsed | The result of pre-processing and converting to XHTML the output from the Input Plugin. Use to debug structure detection.

structure

Post structure detection, but before CSS flattening and font size conversion. Use to debug font size conversion and CSS transforms.

processed | Just before the ebook is passed to the output plugin. Use to debug the Output Plugin.

If you want to edit the input document a little before having calibre convert it, the best thing to do is edit the files in the input sub-directory, then zip it up, and use the zip file as the input format for subsequent conversions. To do this use the Edit meta information dialog to add the zip file as a format for the book and then, in the top left corner of the conversion dialog, select ZIP as the input format.

This document will deal mainly with the various transforms that operate on the intermediate XHTML and how to control them. At the end are some tips specific to each Input/Output format.

Look & Feel

Contents

- Font size rescaling
- Paragraph spacing
- Extra CSS
- Miscellaneous

This group of options controls various aspects of the look and feel of the converted ebook.

Font size rescaling

One of the nicest features of the e-reading experience is the ability to easily adjust font sizes to suit individual needs and lighting conditions. calibre has sophisticated algorithms to ensure that all the books it outputs have a consistent font sizes, no matter what font sizes are specified in the input document.

The base font size of a document is the most common font size in that document, i.e., the size of the bulk of text in that document. When you specify a Base font size, calibre automatically rescales all font sizes in the document proportionately, so that the most common font size becomes the specified base font size and other font sizes are rescaled appropriately. By choosing a larger base font size, you can make the fonts in the document larger and vice versa. When you set the base font size, for best results, you should also set the font size key.

Normally, calibre will automatically choose a base font size appropriate to the Output Profile you have chosen (see Page Setup). However, you can override this here in case the default is not suitable for you.

The Font size key option lets you control how non-base font sizes are rescaled.
The font rescaling algorithm works using a font size key, which is simply a comma-separated list of font sizes. The font size key tells calibre how many “steps” bigger or smaller a given font size should be compared to the base font size. The idea is that there should be a limited number of font sizes in a document. For example, one size for the body text, a couple of sizes for different levels of headings and a couple of sizes for super/sub scripts and footnotes. The font size key allows calibre to compartmentalize the font sizes in the input documents into separate “bins” corresponding to the different logical font sizes.

Let’s illustrate with an example.
Suppose the source document we are converting was produced by someone with excellent eyesight and has a base font size of 8pt. That means the bulk of the text in the document is sized at 8pts, while headings are somewhat larger (say 10 and 12pt) and footnotes somewhat smaller at 6pt.
Now if we use the following settings:

Base font size : 12pt
Font size key : 7, 8, 10, 12, 14, 16, 18, 20

The output document will have a base font size of 12pt, headings of 14 and 16pt and footnotes of 8pt.
Now suppose we want to make the largest heading size stand out more and make the footnotes a little larger as well. To achieve this, the font key should be changed to:

New font size key : 7, 9, 12, 14, 18, 20, 22

The largest headings will now become 18pt, while the footnotes will become 9pt. You can play with these settings to try and figure out what would be optimum for you by using the font rescaling wizard, which can be accessed by clicking the little button next to the Font size key setting.

All the font size rescaling in the conversion can also be disabled here, if you would like to preserve the font sizes in the input document.

A related setting is Line height. Line height controls the vertical height of lines. By default, (a line height of 0), no manipulation of line heights is performed. If you specify a non-default value, line heights will be set in all locations that don’t specify their own line heights. However, this is something of a blunt weapon and should be used sparingly.
If you want to adjust the line heights for some section of the input, it’s better to use the Extra CSS.

Paragraph spacing

Normally, paragraphs in XHTML are rendered with a blank line between them and no leading text indent. calibre has a couple of options to control this. Remove spacing between paragraphs forcefully ensure that all paragraphs have no inter paragraph spacing. It also sets the text indent to 1.5em (can be changed) to mark the start of every paragraph. Insert blank line does the opposite, guaranteeing that there is exactly one blank line between each pair of paragraphs.
Both these options are very comprehensive, removing spacing, or inserting it for all paragraphs
(technically <p> and <div> tags). This is so that you can just set the option and be sure that it performs as advertised, irrespective of how messy the input file is. The one exception is when the input file uses hard line breaks to implement inter-paragraph spacing.

If you want to remove the spacing between all paragraphs, except a select few, don’t use these options. Instead add the following CSS code to Extra CSS:

p, div { margin: 0pt; border: 0pt; text-indent: 1.5em }
.spacious { margin-bottom: 1em; text-indent: 0pt; }

Then, in your source document, mark the paragraphs that need spacing with class=”spacious”.
If your input document is not in HTML, use the Debug option, described in the Introduction to get HTML
(use the input sub-directory).

Extra CSS

This option allows you to specify arbitrary CSS that will be applied to all HTML files in the input. This CSS is applied with very high priority and so should override most CSS present in the input document itself. You can use this setting to fine tune the presentation/layout of your document. For example, if you want all paragraphs of class endnote to be right aligned, just add:

.endnote { text-align: right }

or if you want to change the indentation of all paragraphs:

p { text-indent: 5mm; }

Extra CSS is a very powerful option, but you do need an understanding of how CSS works to use it to its full potential. You can use the debug pipeline option described above to see what
CSS is present in your input document.

Miscellaneous

There are a few more options in this section.

No text justification

Normally, if the output format supports it, calibre will force the output ebook to have justified text (i.e., a smooth right margin). This option will turn off this behavior, in which case whatever justification is specified in the input document will be used instead.

Linearize tables

Some badly designed documents use tables to control the layout of text on the page.
When converted these documents often have text that runs off the page and other artifacts.
This option will extract the content from the tables and present it in a linear fashion.
Note that this option linearizes all tables, so only use it if you are sure the input document does not use tables for legitimate purposes, like presenting tabular information.

Transliterate unicode characters

Transliterate unicode characters to an ASCII representation. Use with care because this will replace unicode characters with ASCII. For instance it will replace “Михаил Горбачёв”
with “Mikhail Gorbachiov”. Also, note that in cases where there are multiple representations of a character (characters shared by Chinese and Japanese for instance) the representation used by the largest number of people will be used (Chinese in the previous example).
This option is mainly useful if you are going to view the ebook on a device that does not have support for unicode.

Input character encoding

Older documents sometimes don’t specify their character encoding. When converted, this can result in non-English characters or special characters like smart quotes being corrupted.
calibre tries to auto-detect the character encoding of the source document, but it does not’
always succeed. You can force it to assume a particular character encoding by using this setting. cp1252 is a common encoding for documents produced using windows software. You should also read How do I convert my file containing non-English characters, or smart quotes? for more on encoding issues.

Page Setup

The Page Setup options are for controlling screen layout, like margins and screen sizes. There are options to setup page margins, which will be used by the Output Plugin, if the selected Output Format supports page margins. In addition, you should choose an Input profile and an Output profile. Both sets of profiles basically deal with how to interpret measurements in the input/output documents, screen sizes and default font rescaling keys.

If you know that the file you are converting was intended to be used on a particular device/software platform, choose the corresponding input profile, otherwise just choose the default input profile. If you know the files you are producing are meant for a particular device type, choose the corresponding Output profile. In particular, for MOBI Output files, you should choose the Kindle, for LIT the Microsoft Reader and for EPUB the Sony Reader. In the case of EPUB, the Sony Reader profile will result in EPUB files that will work everywhere. However, it has some side effects, like inserting artificial section breaks to keep internal components below the size threshold, needed for SONY devices. In particular for the iPhone/Android phones, choose the SONY output profile. If you know your EPUB files will not be read on a SONY or similar device, use the default output profile. If you are producing MOBI files that are not intended for the Kindle, choose the Mobipocket books output profile.

The Output profile also controls the screen size. This will cause, for example, images to be auto-resized to be fit to the screen in some output formats. So choose a profile of a device that has a screen size similar to your device.

Heuristic Processing

Heuristic Processing provides a variety of functions which can be used to try and detect and correct common problems in poorly formatted input documents. Use these functions if your input document suffers from poor formatting. Because these functions rely on common patterns, be aware that in some cases an option may lead to worse results, so use with care. As an example, several of these options will remove all non-breaking-space entities, or may include false positive matches relating to the function.

Enable heuristic processing

This option activates calibre‘s Heuristic Processing stage of the conversion pipeline.
This must be enabled in order for various sub-functions to be applied

Unwrap lines

Enabling this option will cause calibre to attempt to detect and correct hard line breaks that exist within a document using punctuation clues and line length. calibre will first attempt to detect whether hard line breaks exist, if they do not appear to exist calibre will not attempt to unwrap lines. The line-unwrap factor can be reduced if you want to ‘force’ calibre to unwrap lines.

Line-unwrap factor

This option controls the algorithm calibre uses to remove hard line breaks. For example, if the value of this option is 0.4, that means calibre will remove hard line breaks from the end of lines whose lengths are less than the length of 40% of all lines in the document. If your document only has a few line breaks which need correction, then this value should be reduced to somewhere between 0.1 and 0.2.

Detect and markup unformatted chapter headings and sub headings

If your document does not have chapter headings and titles formatted differently from the rest of the text, calibre can use this option to attempt detection them and surround them with heading tags. <h2> tags are used for chapter headings; <h3> tags are used for any titles that are detected.

This function will not create a TOC, but in many cases it will cause calibre‘s default chapter detection settings to correctly detect chapters and build a TOC. Adjust the XPath under Structure Detection if a TOC is not automatically created. If there are no other headings used in the document then setting “//h:h2” under Structure Detection would be the easiest way to create a TOC for the document.

The inserted headings are not formatted, to apply formatting use the Extra CSS option under the Look and Feel conversion settings. For example, to center heading tags, use the following:

h2, h3 { text-align: center }

Renumber sequences of <h1> or <h2> tags

Some publishers format chapter headings using multiple <h1> or <h2> tags sequentially.
calibre‘s default conversion settings will cause such titles to be split into two pieces. This option will re-number the heading tags to prevent splitting.

Delete blank lines between paragraphs

This option will cause calibre to analyze blank lines included within the document. If every paragraph is interleaved with a blank line, then calibre will remove all those blank paragraphs. Sequences of multiple blank lines will be considered scene breaks and retained as a single paragraph. This option differs from the ‘Remove Paragraph Spacing’
option under ‘Look and Feel’ in that it actually modifies the HTML content, while the other option modifies the document styles. This option can also remove paragraphs which were inserted using calibre‘s ‘Insert blank line’ option.

Ensure scene breaks are consistently formatted

With this option calibre will attempt to detect common scene-break markers and ensure that they are center aligned.
‘Soft’ scene break markers, i.e. scene breaks only defined by extra white space, are styled to ensure that they will not be displayed in conjunction with page breaks.

Replace scene breaks

If this option is configured then calibre will replace scene break markers it finds with the replacement text specified by the user. Please note that some ornamental characters may not be supported across all reading devices.

In general you should avoid using html tags, calibre will discard any tags and use pre-defined markup. <hr /> tags, i.e. horizontal rules, and <img> tags are exceptions. Horizontal rules can optionally be specified with styles, if you choose to add your own style be sure to include the ‘width’ setting, otherwise the style information will be discarded. Image tags can used, but calibre does not provide the ability to add the image during conversion, this must be done after the fact using the ‘Tweak Book’ feature, or Sigil.

Example image tag (place the image within an ‘Images’ folder inside the epub after conversion):

<img style=”width:10%” src=”../Images/scenebreak.png” />

Example horizontal rule with styles:

<hr style=”width:20%;padding-top: 1px;border-top: 2px ridge black;border-bottom: 2px groove black;”/>

Remove unnecessary hyphens
calibre will analyze all hyphenated content in the document when this option is enabled. The document itself is used as a dictionary for analysis. This allows calibre to accurately remove hyphens for any words in the document in any language, along with made-up and obscure scientific words. The primary drawback is words appearing only a single time in the document will not be changed. Analysis happens in two passes, the first pass analyzes line endings. Lines are only unwrapped if the word exists with or without a hyphen in the document. The second pass analyzes all hyphenated words throughout the document, hyphens are removed if the word exists elsewhere in the document without a match.

Italicize common words and patterns

When enabled, calibre will look for common words and patterns that denote italics and italicize them. Examples are common text conventions such as ~word~ or phrases that should generally be italicized, e.g. latin phrases like ‘etc.’ or ‘et cetera’.

Replace entity indents with CSS indents

Some documents use a convention of defining text indents using non-breaking space entities. When this option is enabled calibre will attempt to detect this sort of formatting and convert them to a 3% text indent using css.

Search & Replace

These options are useful primarily for conversion of PDF documents or OCR conversions, though they can also be used to fix many document specific problems. As an example, some conversions can leaves behind page headers and footers in the text. These options use regular expressions to try and detect headers, footers, or other arbitrary text and remove or replace them. Remember that they operate on the intermediate XHTML produced by the conversion pipeline. There is a wizard to help you customize the regular expressions for your document. Click the magic wand beside the expression box, and click the ‘Test’ button after composing your search expression. Successful matches will be highlighted in Yellow.

The search works by using a python regular expression. All matched text is simply removed from the document or replaced using the replacement pattern. The replacement pattern is optional, if left blank then text matching the search pattern will be deleted from the document. You can learn more about regular expressions and their syntax at All about using regular expressions in calibre.

Structure Detection

Structure detection involves calibre trying its best to detect structural elements in the input document, when they are not properly specified. For example, chapters, page breaks, headers, footers, etc. As you can imagine, this process varies widely from book to book. Fortunately, calibre has very powerful options to control this. With power comes complexity, but if once you take the time to learn the complexity, you will find it well worth the effort.

Chapters and page breaks
calibre has two sets of options for chapter detection and inserting page breaks. This can sometimes be slightly confusing, as by default, calibre will insert page breaks before detected chapters as well as the locations detected by the page breaks option.
The reason for this is that there are often location where page breaks should be inserted that are not chapter boundaries.
Also, detected chapters can be optionally inserted into the auto generated Table of Contents.

calibre uses XPath, a powerful language to allow the user to specify chapter boundaries/page breaks. XPath can seem a little daunting to use at first, fortunately, there is a XPath tutorial in the User Manual. Remember that Structure Detection operates on the intermediate XHTML produced by the conversion pipeline. Use the debug option described in the Introduction to figure out the appropriate settings for your book. There is also a button for a XPath wizard to help with the generation of simple XPath expressions.

By default, calibre uses the following expression for chapter detection:

//*[((name()='h1' or name()='h2') and re:test(., 'chapter|book|section|part\s+', 'i')) or @class = 'chapter']

This expression is rather complex, because it tries to handle a number of common cases simulataneously. What it means is that calibre will assume chapters start at either <h1> or <h2> tags that have any of the words (chapter, book, section or part) in them or that have the class=”chapter” attribute.

A related option is Chapter mark, which allows you to control what calibre does when it detects a chapter. By default, it will insert a page break before the chapter. You can have it insert a ruled line instead of, or in addition to the page break.
You can also have it do nothing.

The default setting for detecting page breaks is:

//*[name()='h1' or name()='h2']

which means that calibre will insert page breaks before every <h1> and <h2> tag by default.

Note

The default expressions may change depending on the input format you are converting.

Miscellaneous

There are a few more options in this section.

Insert metadata as page at start of book

One of the great things about calibre is that it allows you to maintain very complete metadata about all of your books, for example, a rating, tags, comments, etc. This option will create a single page with all this metadata and insert it into the converted ebook, typically just after the cover. Think of it as a way to create your own customised book jacket.

Remove first image

Sometimes, the source document you are converting includes the cover as part of the book, instead of as a separate cover. If you also specify a cover in calibre, then the converted book will have two covers. This option will simply remove the first image from the source document, thereby ensuring that the converted book has only one cover, the one specified in calibre.

Table of Contents

When the input document has a Table of Contents in its metadata, calibre will just use that. However, a number of older formats either do not support a metadata based Table of Contents, or individual documents do not have one. In these cases, the options in this section can help you automatically generate a Table of Contents in the converted ebook, based on the actual content in the input document.

The first option is Force use of auto-generated Table of Contents. By checking this option you can have calibre override any Table of Contents found in the metadata of the input document with the auto generated one.

The default way that the creation of the auto generated Table of Contents works is that, calibre will first try to add any detected chapters to the generated table of contents. You can learn how to customize the detection of chapters in the Structure Detection section above. If you do not want to include detected chapters in the generated table of contents, check the Do not add detected chapters option.

If less than the Chapter threshold number of chapters were detected, calibre will then add any hyperlinks it finds in the input document to the Table of Contents. This often works well many input documents include a hyperlinked Table of Contents right at the start. The Number of links option can be used to control this behavior. If set to zero, no links are added. If set to a number greater than zero, at most that number of links is added.

calibre will automatically filter duplicates from the generated Table of Contents. However, if there are some additional undesirable entries, you can filter them using the TOC Filter option. This is a regular expression that will match the title of entries in the generated table of contents. Whenever a match is found, it will be removed.
For example, to remove all entries titles “Next” or “Previous” use:

Next|Previous

Finally, the Level 1,2,3 TOC options allow you to create a sophisticated multi-level Table of Contents.
They are XPath expressions that match tags in the intermediate XHTML produced by the conversion pipeline. See the Introduction for how to get access to this XHTML. Also read the XPath Tutorial, to learn how to construct XPath expressions. Next to each option is a button that launches a wizard to help with the creation of basic XPath expressions. The following simple example illustrates how to use these options.

Suppose you have an input document that results in XHTML that look like this:

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Sample document</title>
</head>
<body>
<h1>Chapter 1</h1>
...
<h2>Section 1.1</h2>
...
<h2>Section 1.2</h2>
...
<h1>Chapter 2</h1>
...
<h2>Section 2.1</h2>
...
</body>
</html>

Then, we set the options as:

Level 1 TOC : //h:h1
Level 2 TOC : //h:h2

This will result in an automatically generated two level Table of Contents that looks like:

Chapter 1
Section 1.1
Section 1.2
Chapter 2
Section 2.1

Warning

Not all output formats support a multi level Table of Contents. You should first try with EPUB Output. If that works, then try your format of choice.

Using images as chapter titles when converting HTML input documents

Suppose you want to use an image as your chapter title, but still want calibre to be able to automatically generate a Table of Contents for you from the chapter titles.
Use the following HTML markup to achieve this

<html>
<body>
<h2>Chapter 1</h2>
<p>chapter 1 text...</p>
<h2 title="Chapter 2"><img src="chapter2.jpg" /></h2>
<p>chapter 2 text...</p>
</body>
</html>

Set the Level 1 TOC setting to //h:h2. Then, for chapter two, calibre will take the title from the value of the title attribute on the <h2> tag, since the tag has no text.

How options are set/saved for Conversion

There are two places where conversion options can be set in calibre. The first is in Preferences->Conversion. These settings are the defaults for the conversion options. Whenever you try to convert a new book, the settings set here will be used by default.

You can also change settings in the conversion dialog for each book conversion. When you convert a book, calibre remembers the settings you used for that book, so that if you convert it again, the saved settings for the individual book will take precedence over the defaults set in Preferences. You can restore the individual settings to defaults by using the Restore to defaults button in the individual book conversion dialog.

When you Bulk Convert a set of books, settings are taken in the following order:

- From the defaults set in Preferences->Conversion

- From the saved conversion settings for each book being converted (if any). This can be turned off by the option in the top left corner of the Bulk Conversion dialog.

- From the settings set in the Bulk conversion dialog

Note that the final settings for each book in a Bulk Conversion will be saved and re-used if the book is converted again. Since the highest priority in Bulk Conversion is given to the settings in the Bulk Conversion dialog, these will override any book specific settings. So you should only bulk convert books together that need similar settings. The exceptions are metadata and input format specific settings. Since the Bulk Conversion dialog does not have settings for these two categories, they will be taken from book specific settings (if any) or the defaults.

Note

You can see the actual settings used during any conversion by clicking the rotating icon in the lower right corner and then double clicking the individual conversion job. This will bring up a conversion log that will contain the actual settings used, near the top.

Format specific tips

Here you will find tips specific to the conversion of particular formats. Options specific to particular format, whether input or output are available in the conversion dialog under their own section, for example TXT Input or EPUB Output.

Convert Microsoft Word documents
calibre does not directly convert .doc/.docx files from Microsoft Word. However, in Word, you can save the document as HTML and then convert the resulting HTML file with calibre. When saving as HTML, be sure to use the
“Save as Web Page, Filtered” option as this will produce clean HTML that will convert well. Note that Word produces really messy HTML, converting it can take a long time, so be patient. Another alternative is to use the free OpenOffice. Open your .doc file in OpenOffice and save it in OpenOffice’s format .odt. calibre can directly convert .odt files.

There is a Word macro package that can automate the conversion of Word documents using calibre. It also makes generating the Table of Contents much simpler. It is called BookCreator and is available for free at mobileread [http://www.mobileread.com/forums/showthread.php?t=28313].

An easy way to generate a Table of Contents when converting a Word document is:

1. Mark your Chapters and sub-Chapters in the doc file with one of the MS built-in styles called ‘Heading 1’, ‘Heading 2’, ..., ‘Heading 6’. ‘Heading 1’ equates to the HTML tag <h1>, ‘Heading 2’ to <h2> etc

2. Save the doc as Webpage-filtered (rather than Webpage) and import the html file into calibre

3. When you convert in calibre you use what you did in step 1 to set the box called ‘Detect chapters at’ on the Convert - Structure Detection page. For example:

- If you mark Chapters with style ‘Heading 2’ then set the ‘Detect chapters at’ box to //h:h2 This will give you a proper external metadata TOC in the converted epub.

- A slightly more complex example...if your book has Sections and Chapters and you want a 2-level nested metadata TOC. Mark the doc Sections with style ‘Heading 2’ and the Chapters with style ‘Heading 3’. When you convert set the ‘Detect chapters at’ box to //h:h2|//h:h3. On the Convert - TOC page set the ‘Level 1 TOC’ box to //h:h2 and the ‘Level 2 TOC’ box to //h:h3.

Convert TXT documents

TXT documents have no well defined way to specify formatting like bold, italics, etc, or document structure like paragraphs, headings, sections and so on, but there are a variety of conventions commonly used. By default calibre attempts automatic detection of the correct formatting and markup based on those conventions.

TXT input supports a number of options to differentiate how paragraphs are detected.

Paragraph Style: Auto

Analyzes the text file and attempts to automatically determine how paragraphs are defined. This option will generally work fine, if you achieve undesirable results try one of the manual options.

Paragraph Style: Block

Assumes one or more blank lines are a paragraph boundary:

This is the first.

This is the
second paragraph.

Paragraph Style: Single

Assumes that every line is a paragraph:

This is the first.
This is the second.
This is the third.

Paragraph Style: Print

Assumes that every paragraph starts with an indent (either a tab or 2+ spaces). Paragraphs end when the next line that starts with an indent is reached:

This is the
first.
This is the second.

This is the
third.

Paragraph Style: Unformatted

Assumes that the document has no formatting, but does use hard line breaks. Punctuation and median line length are used to attempt to re-create paragraphs.

Formatting Style: Auto

Attemtps to detect the type of formatting markup being used. If no markup is used then heuristic formatting will be applied.

Formatting Style: Heuristic

Analyzes the document for common chapter headings, scene breaks, and italicized words and applies the appropriate html markup during conversion.

Formatting Style: Markdown
calibre also supports running TXT input though a transformation preprocessor known as markdown. Markdown allows for basic formatting to be added to TXT documents, such as bold, italics, section headings, tables, lists, a Table of Contents, etc. Marking chapter headings with a leading # and setting the chapter XPath detection expression to “//h:h1” is the easiest way to have a proper table of contents generated from a TXT document.
You can learn more about the markdown syntax at daringfireball [http://daringfireball.net/projects/markdown/syntax].

Formatting Style: None

Applies no special formatting to the text, the document is converted to html with no other changes.

Convert PDF documents

PDF documents are one of the worst formats to convert from. They are a fixed page size and text placement format.
Meaning, it is very difficult to determine where one paragraph ends and another begins. calibre will try to unwrap paragraphs using a configurable, Line Un-Wrapping Factor. This is a scale used to determine the length at which a line should be unwrapped. Valid values are a decimal between 0 and 1. The default is 0.45, just under the median line length. Lower this value to include more text in the unwrapping. Increase to include less. You can adjust this value in the conversion settings under PDF Input.

Also, they often have headers and footers as part of the document that will become included with the text.
Use the Search and Replace panel to remove headers and footers to mitigate this issue. If the headers and footers are not removed from the text it can throw off the paragraph unwrapping. To learn how to use the header and footer removal options, read All about using regular expressions in calibre.

Some limitations of PDF input are:

- Complex, multi-column, and image based documents are not supported.

- Extraction of vector images and tables from within the document is also not supported.

- Some PDFs use special glyphs to represent ll or ff or fi, etc. Conversion of these may or may not work depending on just how they are represented internally in the PDF.

- Links and Tables of Contents are not supported

- PDFs that use embedded non-unicode fonts to represent non-English characters will result in garbled output for those characters

- Some PDFs are made up of photographs of the page with OCRed text behind them. In such cases calibre uses the OCRed text, which can be very different from what you see when you view the PDF file

To re-iterate PDF is a really, really bad format to use as input. If you absolutely must use PDF, then be prepared for an output ranging anywhere from decent to unusable, depending on the input PDF.

Comic Book Collections

A comic book collection is a .cbc file. A .cbc file is a zip file that contains other CBZ/CBR files. In addition the
.cbc file must contain a simple text file called comics.txt, encoded in UTF-8. The comics.txt file must contain a list of the comics files inside the .cbc file, in the form filename:title, as shown below:

one.cbz:Chapter One
two.cbz:Chapter Two
three.cbz:Chapter Three

The .cbc file will then contain:

comics.txt
one.cbz
two.cbz
three.cbz

calibre will automatically convert this .cbc file into a ebook with a Table of Contents pointing to each entry in comics.txt.

EPUB advanced formatting demo

Various advanced formatting for EPUB files is demonstrated in this demo file [http://calibre-ebook.com/downloads/demos/demo.epub].
The file was created from hand coded HTML using calibre and is meant to be used as a template for your own EPUB creation efforts.

The source HTML it was created from is available demo.zip [http://calibre-ebook.com/downloads/demos/demo.zip]. The settings used to create the
EPUB from the ZIP file are:

ebook-convert demo.zip .epub -vv --authors "Kovid Goyal" --language en --level1-toc '//*[@class="title"]' --disable-font-rescaling --page-breaks-before / --no-default-epub-cover

Note that because this file explores the potential of EPUB, most of the advanced formatting is not going to work on readers less capable than calibre‘s built-in EPUB viewer.

Convert ODT documents
calibre can directly convert ODT (OpenDocument Text) files. You should use styles to format your document and minimize the use of direct formatting.
When inserting images into your document you need to anchor them to the paragraph, images anchored to a page will all end up in the front of the conversion.

To enable automatic detection of chapters, you need to mark them with the build-in styles called ‘Heading 1’, ‘Heading 2’, ..., ‘Heading 6’ (‘Heading 1’ equates to the HTML tag <h1>, ‘Heading 2’ to <h2> etc). When you convert in calibre you can enter which style you used into the ‘Detect chapters at’ box. Example:

- If you mark Chapters with style ‘Heading 2’, you have to set the ‘Detect chapters at’ box to //h:h2

- For a nested TOC with Sections marked with ‘Heading 2’ and the Chapters marked with ‘Heading 3’ you need to enter //h:h2|//h:h3. On the Convert - TOC page set the ‘Level 1 TOC’ box to //h:h2 and the ‘Level 2 TOC’ box to //h:h3.

Well-known document properties (Title, Keywords, Description, Creator) are recognized and calibre will use the first image (not to small, and with good aspect-ratio) as the cover image.

There is also an advanced property conversion mode, which is activated by setting the custom property opf.metadata (‘Yes or No’ type) to Yes in your ODT document (File->Properties->Custom Properties).
If this property is detected by calibre, the following custom properties are recognized (opf.authors overrides document creator):

opf.titlesort
opf.authors
opf.authorsort
opf.publisher
opf.pubdate
opf.isbn
opf.language
opf.series
opf.seriesindex

In addition to this, you can specify the picture to use as the cover by naming it opf.cover (right click, Picture->Options->Name) in the ODT. If no picture with this name is found, the ‘smart’ method is used.
As the cover detection might result in double covers in certain output formats, the process will remove the paragraph (only if the only content is the cover!) from the document. But this works only with the named picture!

To disable cover detection you can set the custom property opf.nocover (‘Yes or No’ type) to Yes in advanced mode.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

Editing Ebook Metadata

Contents

- Editing the metadata of one book at a time

- Downloading metadata
- Managing book formats
- All about covers

- Editing the metadata of many books at a time

- Search and replace

- Bulk downloading of metadata

Ebooks come in all shapes and sizes and more often than not, their metadata (things like title/author/series/publisher) is incomplete or incorrect.
The simplest way to change metadata in calibre is to simply double click on an entry and type in the correct replacement.
For more sophisticated, “power editing” use the edit metadata tools discussed below.

Editing the metadata of one book at a time

Click the book you want to edit and then click the Edit metadata button or press the E key. A dialog opens that allows you to edit all aspects of the metadata. It has various features to make editing faster and more efficient. A list of the commonly used tips:

- You can click the button in between title and authors to swap them automatically.

- You can click the button next to author sort to have calibre automatically fill it in using the sort values stored with each author. Use the Manage authors dialog to see and change the authors’ sort values. This dialog can be opened by clicking and holding the button next to author sort.

- You can click the button next to tags to use the Tag Editor to manage the tags associated with the book.

- The ISBN box will have a red background if you enter an invalid ISBN. It will be green for valid ISBNs

- The author sort box will be red if the author sort value differs from what calibre thinks it should be.

Downloading metadata

The nicest feature of the edit metadata dialog is its ability to automatically fill in many metadata fields by getting metadata from various websites. Currently, calibre uses isbndb.com, Google Books, Amazon and Library Thing. The metadata download can fill in Title, author, series, tags, rating, description and ISBN for you.

To use the download, fill in the title and author fields and click the Fetch metadata button. calibre will present you with a list of books that most closely match the title and author. If you fill in the ISBN field first, it will be used in preference to the title and author. If no matches are found, try making your search a little less specific by including only some key words in the title and only the author last name.

Managing book formats

In calibre, a single book entry can have many different formats associated with it. For example you may have obtained the Complete Works of Shakespeare in EPUB format and later converted it to MOBI to read on your Kindle. calibre automatically manages multiple formats for you. In the Available formats section of the Edit metadata dialog, you can manage these formats. You can add a new format, delete an existing format and also ask calibre to set the metadata and cover for the book entry from the metadata in one of the formats.

All about covers

You can ask calibre to download book covers for you, provided the book has a known ISBN. Alternatively you can specify a file on your computer to use as the cover. calibre can even generate a default cover with basic metadata on it for you. You can drag and drop images onto the cover to change it and also right click to copy/paste cover images.

In addition, there is a button to automatically trim borders from the cover, in case your cover image has an ugly border.

Editing the metadata of many books at a time

First select the books you want to edit by holding Ctrl or Shift and clicking on them. If you select more than one book, clicking the Edit metadata button will cause a new Bulk metadata edit dialog to open. Using this dialog, you can quickly set the author/publisher/rating/tags/series etc of a bunch of books to the same value. This is particularly useful if you have just imported a number of books that have some metadata in common. This dialog is very powerful, for example, it has a Search and Replace tab that you can use to perform bulk operations on metadata and even copy metadata from one column to another.

The normal edit metadata dialog also has Next and Previous buttons that you can use to edit the metadata of several books one after the other.

Search and replace

The Bulk metadata edit dialog allows you to perform arbitrarily powerful search and replace operations on the selected books. By default it uses a simple text search and replace, but it also support regular expressions. For more on regular expressions, see All about using regular expressions in calibre.

As noted above, there are two search and replace modes: character match and regular expression. Character match will look in the Search field you choose for the characters you type in the search for box and replace those characters with what you type in the replace with box. Each occurance of the search characters in the field will be replaced. For example, assume the field being searched contains a bad cat. if you search for a to be replaced with HELLO, then the result will be HELLO bHELLOd cHELLOt.

If the field you are searching on is a multiple field like tags, then each tag is treated separately. For example, if your tags contain Horror, Scary, the search expression r, will not match anything because the expression will first be applied to Horror and then to Scary.

If you want the search to ignore upper/lowercase differences, uncheck the Case sensitive box.

You can have calibre change the case of the result (information after the replace has happened) by choosing one of the functions from the Apply function after replace box. The operations available are:

- Lower case – change all the characters in the field to lower case

- Upper case – change all the characters in the field to upper case

- Title case – capitalize each word in the result.

The Your test box is provided for you to enter text to check that search/replace is doing what you want. In the majority of cases the book test boxes will be sufficient, but it is possible that there is a case you want to check that isn’t shown in these boxes. Enter that case into Your test.

Regular expression mode has some differences from character mode, beyond (of course) using regular expressions. The first is that functions are applied to the parts of the string matched by the search string, not the entire field. The second is that functions apply to the replacement string, not to the entire field.

The third and most important is that the replace string can make reference to parts of the search string by using backreferences. A backreference is \\n where n is an integer that refers to the n’th parenthesized group in the search expression. For example, given the same example as above, a bad cat, a search expression a (...) (...), and a replace expression a \2 \1, the result will be a cat bad. Please see the All about using regular expressions in calibre for more information on backreferences.

One useful pattern: assume you want to change the case of an entire field. The easiest way to do this is to use character mode, but lets further assume you want to use regular expression mode. The search expression should be (.*) the replace expression should be \1, and the desired case change function should be selected.

Finally, in regular expression mode you can copy values from one field to another. Simply make the source and destination field different. The copy can replace the destination field, prepend to the field (add to the front), or append to the field (add at the end). The ‘use comma’ checkbox tells calibre to (or not to) add a comma between the text and the destination field in prepend and append modes. If the destination is multiple (e.g., tags), then you cannot uncheck this box.

Search and replace is done after all the other metadata changes in the other tabs are applied. This can lead to some confusion, because the test boxes will show the information before the other changes, but the operation will be applied after the other changes. If you have any doubts about what is going to happen, do not mix search/replace with other changes.

Bulk downloading of metadata

If you want to download the metadata for multiple books at once, right-click the Edit metadata button and select Download metadata. You can choose to download only metadata, only covers, or both.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

Frequently Asked Questions

Contents

- Ebook Format Conversion
- Device Integration
- Library Management
- Content From The Web
- Miscellaneous

Ebook Format Conversion

Contents

- What formats does calibre support conversion to/from?

- What are the best source formats to convert?

- I converted a PDF file, but the result has various problems?

- How do I convert my file containing non-English characters, or smart quotes?

- What’s the deal with Table of Contents in MOBI files?

- The covers for my MOBI files have stopped showing up in Kindle for PC/Kindle for Android/iPad etc.

- How do I convert a collection of HTML files in a specific order?

- The EPUB I produced with calibre is not valid?

- How do I use some of the advanced features of the conversion tools?

What formats does calibre support conversion to/from?

calibre supports the conversion of many input formats to many output formats.
It can convert every input format in the following list, to every output format.

Input Formats: CBZ, CBR, CBC, CHM, DJVU, EPUB, FB2, HTML, HTMLZ, LIT, LRF, MOBI, ODT, PDF, PRC, PDB, PML, RB, RTF, SNB, TCR, TXT, TXTZ

Output Formats: AZW3, EPUB, FB2, OEB, LIT, LRF, MOBI, HTMLZ, PDB, PML, RB, PDF, RTF, SNB, TCR, TXT, TXTZ

Note

PRC is a generic format, calibre supports PRC files with TextRead and MOBIBook headers.
PDB is also a generic format. calibre supports eReder, Plucker, PML and zTxt PDB files.
DJVU support is only for converting DJVU files that contain embedded text. These are typically generated by OCR software.
MOBI books can be of two types Mobi6 and KF8. calibre fully supports both. MOBI files often have .azw or .azw3 file extensions

What are the best source formats to convert?

In order of decreasing preference: LIT, MOBI, AZW, EPUB, AZW3, FB2, HTML, PRC, RTF, PDB, TXT, PDF

I converted a PDF file, but the result has various problems?

PDF is a terrible format to convert from. For a list of the various issues you will encounter when converting PDF, see: Convert PDF documents.

How do I convert my file containing non-English characters, or smart quotes?

There are two aspects to this problem:

1. Knowing the encoding of the source file: calibre tries to guess what character encoding your source files use, but often, this is impossible, so you need to tell it what encoding to use. This can be done in the GUI via the Input character encoding field in the Look & Feel section. The command-line tools all have an --input-encoding option.

2. When adding HTML files to calibre, you may need to tell calibre what encoding the files are in. To do this go to Preferences->Advanced->Plugins->File Type plugins and customize the HTML2Zip plugin, telling it what encoding your HTML files are in. Now when you add HTML files to calibre they will be correctly processed. HTML files from different sources often have different encodings, so you may have to change this setting repeatedly. A common encoding for many files from the web is cp1252 and I would suggest you try that first. Note that when converting HTML files, leave the input encoding setting mentioned above blank. This is because the HTML2ZIP plugin automatically converts the HTML files to a standard encoding (utf-8).

3. Embedding fonts: If you are generating an LRF file to read on your SONY Reader, you are limited by the fact that the Reader only supports a few non-English characters in the fonts it comes pre-loaded with. You can work around this problem by embedding a unicode-aware font that supports the character set your file uses into the LRF file. You should embed atleast a serif and a sans-serif font. Be aware that embedding fonts significantly slows down page-turn speed on the reader.

What’s the deal with Table of Contents in MOBI files?

The first thing to realize is that most ebooks have two tables of contents. One is the traditional Table of Contents, like the TOC you find in paper books. This Table of Contents is part of the main document flow and can be styled however you like. This TOC is called the content TOC.

Then there is the metadata TOC. A metadata TOC is a TOC that is not part of the book text and is typically accessed by some special button on a reader. For example, in the calibre viewer, you use the Show Table of Contents button to see this TOC. This TOC cannot be styled by the book creator. How it is represented is up to the viewer program.

In the MOBI format, the situation is a little confused. This is because the MOBI format, alone amongst mainstream ebook formats, does not have decent support for a metadata TOC. A MOBI book simulates the presence of a metadata TOC by putting an extra content TOC at the end of the book. When you click Goto Table of Contents on your Kindle, it is to this extra content TOC that the Kindle takes you.

Now it might well seem to you that the MOBI book has two identical TOCs. Remember that one is semantically a content TOC and the other is a metadata TOC, even though both might have exactly the same entries and look the same. One can be accessed directly from the Kindle’s menus, the other cannot.

When converting to MOBI, calibre detects the metadata TOC in the input document and generates an end-of-file TOC in the output MOBI file. You can turn this off by an option in the MOBI Output settings. You can also tell calibre whether to put it and the start or the end of the book via an option in the MOBI Output settings. Remember this TOC is semantically a metadata TOC, in any format other than MOBI it cannot not be part of the text. The fact that it is part of the text in MOBI is an accident caused by the limitations of MOBI. If you want a TOC at a particular location in your document text, create one by hand. So we strongly recommend that you leave the default as it is, i.e. with the metadata TOC at the end of the book.

If you have a hand edited TOC in the input document, you can use the TOC detection options in calibre to automatically generate the metadata TOC from it. See the conversion section of the User Manual for more details on how to use these options.

Finally, I encourage you to ditch the content TOC and only have a metadata TOC in your ebooks. Metadata TOCs will give the people reading your ebooks a much superior navigation experience (except on the Kindle, where they are essentially the same as a content TOC).

The covers for my MOBI files have stopped showing up in Kindle for PC/Kindle for Android/iPad etc.

This is caused by a bug in the Amazon software. You can work around it by going to Preferences->Output Options->MOBI output and setting the “Enable sharing of book content” option. If you are reconverting a previously converted book, you will also have to enable the option in the conversion dialog for that individual book (as per book conversion settings are saved and take precedence).

Note that doing this will mean that the generated MOBI will show up under personal documents instead of Books on the Kindle Fire and Amazon whispersync will not work, but the covers will. It’s your choice which functionality is more important to you. I encourage you to contact Amazon and ask them to fix this bug.

How do I convert a collection of HTML files in a specific order?

In order to convert a collection of HTML files in a specific oder, you have to create a table of contents file. That is, another HTML file that contains links to all the other files in the desired order. Such a file looks like:

<html>
<body>
<h1>Table of Contents</h1>
<p style="text-indent:0pt">
<a href="file1.html">First File</a><br/>
<a href="file2.html">Second File</a><br/>
.
.
.
</p>
</body>
</html>

Then just add this HTML file to the GUI and use the convert button to create your ebook.

Note

By default, when adding HTML files, calibre follows links in the files in depth first order. This means that if file A.html links to B.html and C.html and D.html, but B.html also links to D.html, then the files will be in the order A.html, B.html, D.html, C.html. If instead you want the order to be A.html, B.html, C.html, D.html then you must tell calibre to add your files in breadth first order. Do this by going to Preferences->Plugins and customizing the HTML to ZIP plugin.

The EPUB I produced with calibre is not valid?

calibre does not guarantee that an EPUB produced by it is valid. The only guarantee it makes is that if you feed it valid XHTML 1.1 + CSS 2.1 it will output a valid EPUB. calibre is designed for ebook consumers, not producers. It tries hard to ensure that EPUBs it produces actually work as intended on a wide variety of devices, a goal that is incompatible with producing valid EPUBs, and one that is far more important to the vast majority of its users. If you need a tool that always produces valid EPUBs, calibre is not for you.

How do I use some of the advanced features of the conversion tools?

You can get help on any individual feature of the converters by mousing over it in the GUI or running ebook-convert dummy.html .epub -h at a terminal. A good place to start is to look at the following demo files that demonstrate some of the advanced features:

- html-demo.zip [http://calibre-ebook.com/downloads/html-demo.zip]

Device Integration

Contents

- What devices does calibre support?

- How can I help get my device supported in calibre?

- My device is not being detected by calibre?

- My device is non-standard or unusual. What can I do to connect to it?

- How does calibre manage collections on my SONY reader?

- Can I use both calibre and the SONY software to manage my reader?

- How do I use calibre with my iPad/iPhone/iTouch?

- How do I use calibre with my Android phone/tablet or Kindle Fire HD?

- Can I access my calibre books using the web browser in my Kindle or other reading device?

- I get the error message “Failed to start content server: Port 8080 not free on ‘0.0.0.0’”?

- I cannot send emails using calibre?

- Why is my device not detected in linux?

- My device is getting mounted read-only in linux, so calibre cannot connect to it?

- Why does calibre not support collections on the Kindle or shelves on the Nook?

- I am getting an error when I try to use calibre with my Kobo Touch?

What devices does calibre support?

calibre can directly connect to all the major (and most of the minor) ebook reading devices, smarthphones, tablets, etc.
In addition, using the Connect to folder function you can use it with any ebook reader that exports itself as a USB disk.
You can even connect to Apple devices (via iTunes), using the Connect to iTunes function.

How can I help get my device supported in calibre?

If your device appears as a USB disk to the operating system, adding support for it to calibre is very easy.
We just need some information from you:

- Complete list of ebook formats that your device supports.

- Is there a special directory on the device in which all ebook files should be placed? Also does the device detect files placed in sub directories?

- We also need information about your device that calibre will collect automatically. First, if your device supports SD cards, insert them. Then connect your device to the computer. In calibre go to Preferences->Advanced->Miscellaneous and click the “Debug device detection” button. This will create some debug output. Copy it to a file and repeat the process, this time with your device disconnected from your computer.

- Send both the above outputs to us with the other information and we will write a device driver for your device.

Once you send us the output for a particular operating system, support for the device in that operating system will appear in the next release of calibre. To send us the output, open a bug report and attach the output to it.
See calibre bugs [http://calibre-ebook.com/bugs].

My device is not being detected by calibre?

Follow these steps to find the problem:

- Make sure that you are connecting only a single device to your computer at a time. Do not have another calibre supported device like an iPhone/iPad etc. at the same time.

- If you are connecting an Apple iDevice (iPad, iPod Touch, iPhone), use the ‘Connect to iTunes’ method in the ‘Getting started’ instructions in Calibre + Apple iDevices: Start here [http://www.mobileread.com/forums/showthread.php?t=118559].

- Make sure you are running the latest version of calibre. The latest version can always be downloaded from the calibre website [http://calibre-ebook.com/download].

- Ensure your operating system is seeing the device. That is, the device should show up in Windows Explorer (in Windows) or Finder (in OS X).

- In calibre, go to Preferences->Ignored Devices and check that your device is not being ignored

- In calibre, go to Preferences->Plugins->Device Interface plugin and make sure the plugin for your device is enabled, the plugin icon next to it should be green when it is enabled.

- If all the above steps fail, go to Preferences->Miscellaneous and click debug device detection with your device attached and post the output as a ticket on the calibre bug tracker [http://bugs.calibre-ebook.com].

My device is non-standard or unusual. What can I do to connect to it?

In addition to the Connect to Folder function found under the Connect/Share button, calibre provides a User Defined device plugin that can be used to connect to any USB device that shows up as a disk drive in your operating system. Note: on windows, the device must have a drive letter for calibre to use it. See the device plugin Preferences -> Plugins -> Device Plugins -> User Defined and Preferences -> Miscellaneous -> Get information to setup the user defined device for more information. Note that if you are using the user defined plugin for a device normally detected by a builtin calibre plugin, you must disable the builtin plugin first, so that your user defined plugin is used instead.

How does calibre manage collections on my SONY reader?

When calibre connects with the reader, it retrieves all collections for the books on the reader. The collections of which books are members are shown on the device view.

When you send a book to the reader, calibre will add the book to collections based on the metadata for that book. By default, collections are created from tags and series. You can control what metadata is used by going to Preferences->Advanced->Plugins->Device Interface plugins and customizing the SONY device interface plugin. If you remove all values, calibre will not add the book to any collection.

Collection management is largely controlled by the ‘Metadata management’ option found at Preferences->Import/Export->Sending books to devices. If set to ‘Manual’ (the default), managing collections is left to the user; calibre will not delete already existing collections for a book on your reader when you resend the book to the reader, but calibre will add the book to collections if necessary. To ensure that the collections for a book are based only on current calibre metadata, first delete the books from the reader, then resend the books. You can edit collections directly on the device view by double-clicking or right-clicking in the collections column.

If ‘Metadata management’ is set to ‘Only on send’, then calibre will manage collections more aggressively.
Collections will be built using calibre metadata exclusively. Sending a book to the reader will correct the collections for that book so its collections exactly match the book’s metadata, adding and deleting collections as necessary. Editing collections on the device view is not permitted, because collections not in the metadata will be removed automatically.

If ‘Metadata management’ is set to ‘Automatic management’, then calibre will update metadata and collections both when the reader is connected and when books are sent. When calibre detects the reader and generates the list of books on the reader, it will send metadata from the library to the reader for all books on the reader that are in the library (On device is True), adding and removing books from collections as indicated by the metadata and device customization. When a book is sent, calibre corrects the metadata for that book, adding and deleting collections. Manual editing of metadata on the device view is not allowed. Note that this option specifies sending metadata, not books. The book files on the reader are not changed.

In summary, choose ‘manual management’ if you want to manage collections yourself. Collections for a book will never be removed by calibre, but can be removed by you by editing on the device view. Choose ‘Only on send’ if you want calibre to manage collections when you send a book, adding books to and removing books from collections as needed. Choose ‘Automatic management’ if you want calibre to keep collections up to date whenever the reader is connected.

If you use multiple installations of calibre to manage your reader, then option ‘Automatic management’ may not be what you want. Connecting the reader to one library will reset the metadata to what is in that library.
Connecting to the other library will reset the metadata to what is in that other library. Metadata in books found in both libraries will be flopped back and forth.

Can I use both calibre and the SONY software to manage my reader?

Yes, you can use both, provided you do not run them at the same time. That is, you should use the following sequence:
Connect reader->Use one of the programs->Disconnect reader. Reconnect reader->Use the other program->disconnect reader.

The underlying reason is that the Reader uses a single file to keep track of ‘meta’ information, such as collections, and this is written to by both calibre and the Sony software when either updates something on the Reader.
The file will be saved when the Reader is (safely) disconnected, so using one or the other is safe if there’s a disconnection between them, but if you’re not the type to remember this, then the simple answer is to stick to one or the other for the transfer and just export/import from/to the other via the computers hard disk.

If you do need to reset your metadata due to problems caused by using both at the same time, then just delete the media.xml file on the Reader using your PC’s file explorer and it will be recreated after disconnection.

With recent reader iterations, SONY, in all its wisdom has decided to try to force you to use their software. If you install it, it auto-launches whenever you connect the reader.
If you don’t want to uninstall it altogether, there are a couple of tricks you can use. The simplest is to simply re-name the executable file that launches the library program. More detail in the forums [http://www.mobileread.com/forums/showthread.php?t=65809].

How do I use calibre with my iPad/iPhone/iTouch?

Over the air

The easiest way to browse your calibre collection on your Apple device (iPad/iPhone/iPod) is by using the calibre content server, which makes your collection available over the net. First perform the following steps in calibre

- Set the Preferred Output Format in calibre to EPUB (The output format can be set under Preferences->Interface->Behavior)

- Set the output profile to iPad (this will work for iPhone/iPods as well), under Preferences->Conversion->Common Options->Page Setup

- Convert the books you want to read on your iPhone to EPUB format by selecting them and clicking the Convert button.

- Turn on the Content Server in calibre‘s preferences and leave calibre running.

Now on your iPad/iPhone you have two choices, use either iBooks (version 1.2 and later) or Stanza (version 3.0 and later). Both are available free from the app store.

Using Stanza

Now you should be able to access your books on your iPhone by opening Stanza. Go to “Get Books” and then click the “Shared” tab. Under Shared you will see an entry “Books in calibre”. If you don’t, make sure your iPad/iPhone is connected using the WiFi network in your house, not 3G. If the calibre catalog is still not detected in Stanza, you can add it manually in Stanza. To do this, click the “Shared” tab, then click the “Edit” button and then click “Add book source” to add a new book source. In the Add Book Source screen enter whatever name you like and in the URL field, enter the following:

http://192.168.1.2:8080/

Replace 192.168.1.2 with the local IP address of the computer running calibre. If you have changed the port the calibre content server is running on, you will have to change 8080 as well to the new port. The local IP address is the IP address you computer is assigned on your home network. A quick Google search will tell you how to find out your local IP address. Now click “Save” and you are done.

If you get timeout errors while browsing the calibre catalog in Stanza, try increasing the connection timeout value in the stanza settings. Go to Info->Settings and increase the value of Download Timeout.

Using iBooks

Start the Safari browser and type in the IP address and port of the computer running the calibre server, like this:

http://192.168.1.2:8080/

Replace 192.168.1.2 with the local IP address of the computer running calibre. If you have changed the port the calibre content server is running on, you will have to change 8080 as well to the new port. The local IP address is the IP address you computer is assigned on your home network. A quick Google search will tell you how to find out your local IP address.

You will see a list of books in Safari, just click on the epub link for whichever book you want to read, Safari will then prompt you to open it with iBooks.

With the USB cable + iTunes

Use the ‘Connect to iTunes’ method in the ‘Getting started’ instructions in Calibre + Apple iDevices: Start here [http://www.mobileread.com/forums/showthread.php?t=118559].

This method only works on Windows XP and higher, and OS X 10.5 and higher. Linux is not supported (iTunes is not available in linux) and OS X 10.4 is not supported.

How do I use calibre with my Android phone/tablet or Kindle Fire HD?

There are two ways that you can connect your Android device to calibre. Using a USB cable – or wirelessly, over the air.
The first step to using an Android device is installing an ebook reading application on it. There are many free and paid ebook reading applications for
Android: Some examples (in no particular order): FBReader [https://play.google.com/store/apps/details?id=org.geometerplus.zlibrary.ui.android&hl=en], Moon+ [https://play.google.com/store/apps/details?id=com.flyersoft.moonreader&hl=en], Mantano [https://play.google.com/store/apps/details?id=com.mantano.reader.android.lite&hl=en], Aldiko [https://play.google.com/store/apps/details?id=com.aldiko.android&hl=en], Kindle [https://play.google.com/store/apps/details?id=com.amazon.kindle&feature=related_apps].

Using a USB cable

Simply plug your device into the computer with a USB cable. calibre should automatically detect the device and then you can transfer books to it by clicking the Send to Device button. calibre does not have support for every single android device out there, so if your device is not automatically detected, follow the instructions at How can I help get my device supported in calibre? to get your device supported in calibre.

Note

With newer Android devices, the USB connection is only supported on
Windows Vista and newer and Linux. If you are on Windows XP or OS X, you should use one of the wireless connection methods.

Over the air

The easiest way to transfer books wirelessly to your Android device is to use the Calibre Companion [http://www.multipie.co.uk/calibre-companion/] Android app. This app is maintained by a core calibre developer and allows calibre to connect to your Android device wirelessly, just as though you plugged in the device with a USB cable. You can browse files on the device in calibre and use the Send to device button to transfer files to your device wirelessly.

calibre also has a builtin web server, the Content Server.
You can browse your calibre collection on your Android device is by using the calibre content server, which makes your collection available over the net.
First perform the following steps in calibre

- Set the Preferred Output Format in calibre to EPUB for normal Android devices or MOBI for Kindles (The output format can be set under Preferences->Interface->Behavior)

- Convert the books you want to read on your device to EPUB/MOBI format by selecting them and clicking the Convert button.

- Turn on the Content Server in calibre‘s preferences and leave calibre running.

Now on your Android device, open the browser and browse to

http://192.168.1.2:8080/

Replace 192.168.1.2 with the local IP address of the computer running calibre. If your local network supports the use of computer names, you can replace the IP address with the network name of the computer. If you have changed the port the calibre content server is running on, you will have to change 8080 as well to the new port.

The local IP address is the IP address you computer is assigned on your home network. A quick Google search will tell you how to find out your local IP address. You can now browse your book collection and download books from calibre to your device to open with whatever ebook reading software you have on your android device.

Some reading programs support browsing the Calibre library directly. For example, in Aldiko, click My Catalogs, then + to add a catalog, then give the catalog a title such as “Calibre” and provide the URL listed above. You can now browse the Calibre library and download directly into the reading software.

Can I access my calibre books using the web browser in my Kindle or other reading device?

calibre has a Content Server that exports the books in calibre as a web page. You can turn it on under Preferences->Network->Sharing over the net. Then just point the web browser on your device to the computer running the Content Server and you will be able to browse your book collection. For example, if the computer running the server has IP address 63.45.128.5, in the browser, you would type:

http://63.45.128.5:8080

Some devices, like the Kindle (1/2/DX), do not allow you to access port 8080 (the default port on which the content server runs. In that case, change the port in the calibre Preferences to 80. (On some operating systems, you may not be able to run the server on a port number less than 1024 because of security settings. In this case the simplest solution is to adjust your router to forward requests on port 80 to port 8080).

I get the error message “Failed to start content server: Port 8080 not free on ‘0.0.0.0’”?

The most likely cause of this is your antivirus program. Try temporarily disabling it and see if it does the trick.

I cannot send emails using calibre?

Because of the large amount of spam in email, sending email can be tricky, as different mail servers use different strategies to block email.
The most common problem is if you are sending email directly (without a mail relay) in calibre. Many servers (for example, Amazon) block email that does not come from a well known relay. The most robust way to setup email sending in calibre is to do the following:

- Create a free GMail account at Google [http://www.gmail.com].

- Goto Preferences->Email in calibre and click the “Use Gmail” button and fill in the information asked for.

- calibre will then use GMail to send the mail.

- If you are sending to your Kindle, remember to update the email preferences on your Amazon Kindle page to allow email sent from your GMail email address.

Even after doing this, you may have problems. One common source of problems is that some poorly designed antivirus programs block calibre from opening a connection to send email. Try adding an exclusion for calibre in your antivirus program.

Note

Google can disable your account if you use it to send large amounts of email. So, when using GMail to send mail calibre automatically restricts itself to sending one book every five minutes. If you don’t mind risking your account being blocked you can reduce this wait interval by going to Preferences->Tweaks in calibre.

Why is my device not detected in linux?

calibre needs your linux kernel to have been setup correctly to detect devices. If your devices are not detected, perform the following tests:

grep SYSFS_DEPRECATED /boot/config-`uname -r`

You should see something like CONFIG_SYSFS_DEPRECATED_V2 is not set.
Also,

grep CONFIG_SCSI_MULTI_LUN /boot/config-`uname -r`

must return CONFIG_SCSI_MULTI_LUN=y. If you don’t see either, you have to recompile your kernel with the correct settings.

My device is getting mounted read-only in linux, so calibre cannot connect to it?

Linux kernels mount devices read-only when their filesystems have errors. You can repair the filesystem with:

sudo fsck.vfat -y /dev/sdc

Replace /dev/sdc with the path to the device node of your device. You can find the device node of your device, which will always be under /dev by examining the output of:

mount

Why does calibre not support collections on the Kindle or shelves on the Nook?

Neither the Kindle nor the Nook provide any way to manipulate collections over a USB connection.
If you really care about using collections, I would urge you to sell your Kindle/Nook and get a SONY.
Only SONY seems to understand that life is too short to be entering collections one by one on an e-ink screen :)

Note that in the case of the Kindle, there is a way to manipulate collections via USB, but it requires that the Kindle be rebooted every time it is disconnected from the computer, for the changes to the collections to be recognized. As such, it is unlikely that any calibre developers will ever feel motivated enough to support it. There is however, a calibre plugin that allows you to create collections on your Kindle from the calibre metadata. It is available from here [http://www.mobileread.com/forums/showthread.php?t=118635].

Note

Amazon have removed the ability to manipulate collections completely in their newer models, like the Kindle Touch and Kindle Fire, making even the above plugin useless. If you really want the ability to manage collections on your Kindle via a USB connection, we encourage you to complain to Amazon about it, or get a reader where this is supported, like the SONY or Kobo Readers.

I am getting an error when I try to use calibre with my Kobo Touch?

The Kobo Touch has very buggy firmware. Connecting to it has been known to fail at random. Certain combinations of motherboard, USB ports/cables/hubs can exacerbate this tendency to fail. If you are getting an error when connecting to your touch with calibre try the following, each of which has solved the problem for some calibre users.

- Connect the Kobo directly to your computer, not via USB Hub

- Try a different USB cable and a different USB port on your computer

- Try a different computer (preferably an older model)

- Try upgrading the firmware on your Kobo Touch to the latest

- Try resetting the Kobo (sometimes this cures the problem for a little while, but then it re-appears, in which case you have to reset again and again)

- Try only putting one or two books onto the Kobo at a time and do not keep large collections on the Kobo

Library Management

Contents

- What formats does calibre read metadata from?

- Where are the book files stored?

- How does calibre manage author names and sorting?

- Why doesn’t calibre let me store books in my own directory structure?

- Why doesn’t calibre have a column for foo?

- Can I have a column showing the formats or the ISBN?

- How do I move my calibre library from one computer to another?

- The list of books in calibre is blank!

- I am getting errors with my calibre library on a networked drive/NAS?

What formats does calibre read metadata from?

calibre reads metadata from the following formats: CHM, LRF, PDF, LIT, RTF, OPF, MOBI, PRC, EPUB, FB2, IMP, RB, HTML. In addition it can write metadata to: LRF, RTF, OPF, EPUB, PDF, MOBI

Where are the book files stored?

When you first run calibre, it will ask you for a folder in which to store your books. Whenever you add a book to calibre, it will copy the book into that folder. Books in the folder are nicely arranged into sub-folders by Author and Title. Note that the contents of this folder are automatically managed by calibre, do not add any files/folders manually to this folder, as they may be automatically deleted. If you want to add a file associated to a particular book, use the top right area of Edit metadata dialog to do so. Then, calibre will automatically put that file into the correct folder and move it around when the title/author changes.

Metadata about the books is stored in the file metadata.db at the top level of the library folder This file is is a sqlite database. When backing up your library make sure you copy the entire folder and all its sub-folders.

The library folder and all it’s contents make up what is called a calibre library. You can have multiple such libraries. To manage the libraries, click the calibre icon on the toolbar. You can create new libraries, remove/rename existing ones and switch between libraries easily.

You can copy or move books between different libraries (once you have more than one library setup) by right clicking on a book and selecting the Copy to library action.

How does calibre manage author names and sorting?

Author names are complex, especially across cultures. calibre has a very flexible strategy for managing author names. The first thing to understand is that books and authors are separate entities in calibre. A book can have more than one author, and an author can have more than one book. You can manage the authors of a book by the edit metadata dialog. You can manage individual authors by right clicking on the author in the Tag Browser on the left of the main calibre screen and selecting Manage authors. Using this dialog you can change the name of an author and also how that name is sorted. This will automatically change the name of the author in all the books of that author. When a book has multiple authors, separate their names using the & character.

Now coming to author name sorting:

- When a new author is added to calibre (this happens whenever a book by a new author is added), calibre automatically computes a sort string for both the book and the author.

- Authors in the Tag Browser are sorted by the sort value for the authors. Remember that this is different from the Author sort field for a book.

- By default, this sort algorithm assumes that the author name is in First name Last name format and generates a Last name, First name sort value.

- You can change this algorithm by going to Preferences->Tweaks and setting the author_sort_copy_method tweak.

- You can force calibre to recalculate the author sort values for every author by right clicking on any author and selecting Manage authors, then pushing the Recalculate all author sort values button. Do this after you have set the author_sort_copy_method tweak to what you want.

- You can force calibre to recalculate the author sort values for all books by using the bulk metadata edit dialog (select all books and click edit metadata, check the Automatically set author sort checkbox, then press OK.)

- When recalculating the author sort values for books, calibre uses the author sort values for each individual author. Therefore, ensure that the individual author sort values are correct before recalculating the books’ author sort values.

- You can control whether the Tag Browser display authors using their names or their sort values by setting the categories_use_field_for_author_name tweak in Preferences->Tweaks

Note that you can set an individual author’s sort value to whatever you want using Manage authors. This is useful when dealing with names that calibre will not get right, such as complex multi-part names like Miguel de Cervantes Saavedra or when dealing with Asian names like Sun Tzu.

With all this flexibility, it is possible to have calibre manage your author names however you like. For example, one common request is to have calibre display author names LN, FN. To do this, and if the note below does not apply to you, then:

- Set the author_sort_copy_method tweak to copy as described above.

- Restart calibre. Do not change any book metadata before doing the remaining steps.

- Change all author names to LN, FN using the Manage authors dialog.

- After you have changed all the authors, press the Recalculate all author sort values button.

- Press OK, at which point calibre will change the authors in all your books. This can take a while.

Note

When changing from FN LN to LN, FN, it is often the case that the values in author_sort are already in LN, FN format. If this is your case, then do the following:

- set the author_sort_copy_method tweak to copy as described above.

- restart calibre. Do not change any book metadata before doing the remaining steps.

- open the Manage authors dialog. Press the copy all author sort values to author button.

- Check through the authors to be sure you are happy. You can still press Cancel to abandon the changes. Once you press OK, there is no undo.

- Press OK, at which point calibre will change the authors in all your books. This can take a while.

Why doesn’t calibre let me store books in my own directory structure?

The whole point of calibre‘s library management features is that they provide a search and sort based interface for locating books that is much more efficient than any possible directory scheme you could come up with for your collection. Indeed, once you become comfortable using calibre‘s interface to find, sort and browse your collection, you wont ever feel the need to hunt through the files on your disk to find a book again. By managing books in its own directory structure of Author -> Title -> Book files, calibre is able to achieve a high level of reliability and standardization. To illustrate why a search/tagging based interface is superior to folders, consider the following. Suppose your book collection is nicely sorted into folders with the following scheme:

Genre -> Author -> Series -> ReadStatus

Now this makes it very easy to find for example all science fiction books by Isaac Asimov in the Foundation series. But suppose you want to find all unread science fiction books. There’s no easy way to do this with this folder scheme, you would instead need a folder scheme that looks like:

ReadStatus -> Genre -> Author -> Series

In calibre, you would instead use tags to mark genre and read status and then just use a simple search query like tag:scifi and not tag:read. calibre even has a nice graphical interface, so you don’t need to learn its search language instead you can just click on tags to include or exclude them from the search.

To those of you that claim that you need access to the filesystem to so that you can have access to your books over the network, calibre has an excellent content server that gives you access to your calibre library over the net.

If you are worried that someday calibre will cease to be developed, leaving all your books marooned in its folder structure, explore the powerful “Save to Disk” feature in calibre that lets you export all your files into a folder structure of arbitrary complexity based on their metadata.

Finally, the reason there are numbers at the end of every title folder, is for robustness. That number is the id number of the book record in the calibre database. The presence of the number allows you to have multiple records with the same title and author names. It is also part of what allows calibre to magically regenerate the database with all metadata if the database file gets corrupted. Given that calibre‘s mission is to get you to stop storing metadata in filenames and stop using the filesystem to find things, the increased robustness afforded by the id numbers is well worth the uglier folder names.

If you are still not convinced, then I’m afraid calibre is not for you. Look elsewhere for your book cataloguing needs. Just so we’re clear, this is not going to change. Kindly do not contact us in an attempt to get us to change this.

Why doesn’t calibre have a column for foo?

calibre is designed to have columns for the most frequently and widely used fields. In addition, you can add any columns you like. Columns can be added via Preferences->Interface->Add your own columns.
Watch the tutorial UI Power tips [http://calibre-ebook.com/demo#tutorials] to learn how to create your own columns.

You can also create “virtual columns” that contain combinations of the metadata from other columns. In the add column dialog use the Quick create links to easily create columns to show the book ISBN, formats or the time the book was last modified. For more details, see The calibre template language.

Can I have a column showing the formats or the ISBN?

Yes, you can. Follow the instructions in the answer above for adding custom columns.

How do I move my calibre library from one computer to another?

Simply copy the calibre library folder from the old to the new computer. You can find out what the library folder is by clicking the calibre icon in the toolbar. The very first item is the path to the library folder. Now on the new computer, start calibre for the first time. It will run the Welcome Wizard asking you for the location of the calibre library. Point it to the previously copied folder. If the computer you are transferring to already has a calibre installation, then the Welcome wizard wont run. In that case, click the calibre icon in the tooolbar and point it to the newly copied directory. You will now have two calibre libraries on your computer and you can switch between them by clicking the calibre icon on the toolbar.

Note that if you are transferring between different types of computers (for example Windows to OS X) then after doing the above you should also right-click the calibre icon on the tool bar, select Library Maintenance and run the Check Library action. It will warn you about any problems in your library, which you should fix by hand.

Note

A calibre library is just a folder which contains all the book files and their metadata. All the metadata is stored in a single file called metadata.db, in the top level folder. If this file gets corrupted, you may see an empty list of books in calibre. In this case you can ask calibre to restore your books by doing a right-click on the calibre icon in the toolbar and selecting Library Maintenance->Restore Library.

The list of books in calibre is blank!

In order to understand why that happened, you have to understand what a calibre library is. At the most basic level, a calibre library is just a folder. Whenever you add a book to calibre, that book’s files are copied into this folder (arranged into sub folders by author and title). Inside the calibre library folder, at the top level, you will see a file called metadata.db. This file is where calibre stores the metadata like title/author/rating/tags etc. for every book in your calibre library. The list of books that calibre displays is created by reading the contents of this metadata.db file.

There can be two reasons why calibre is showing a empty list of books:

- Your calibre library folder changed its location. This can happen if it was on an external disk and the drive letter for that disk changed. Or if you accidentally moved the folder. In this case, calibre cannot find its library and so starts up with an empty library instead. To remedy this, do a right-click on the calibre icon in the calibre toolbar (it will say 0 books underneath it) and select Switch/create library. Click the little blue icon to select the new location of your calibre library and click OK.

- Your metadata.db file was deleted/corrupted. In this case, you can ask calibre to rebuild the metadata.db from its backups. Right click the calibre icon in the calibre toolbar (it will say 0 books underneath it) and select Library maintenance->Restore database. calibre will automatically rebuild metadata.db.

I am getting errors with my calibre library on a networked drive/NAS?

Do not put your calibre library on a networked drive.

A filesystem is a complex beast. Most network filesystems lack various filesystem features that calibre uses. Some dont support file locking, some dont support hardlinking, some are just flaky. Additionally, calibre is a single user application, if you accidentally run two copies of calibre on the same networked library, bad things will happen. Finally, different OSes impose different limitations on filesystems, so if you share your networked drive across OSes, once again, bad things will happen.

Consider using the calibre Content Server to make your books available on other computers. Run calibre on a single computer and access it via the Content Server or a Remote Desktop solution.

If you must share the actual library, use a file syncing tool like
DropBox or rsync or Microsoft SkyDrive instead of a networked drive. Even with these tools there is danger of data corruption/loss, so only do this if you are willing to live with that risk.

Content From The Web

Contents

- I obtained a recipe for a news site as a .py file from somewhere, how do I use it?

- I want calibre to download news from my favorite news website.

- Can I use web2disk to download an arbitrary website?

I obtained a recipe for a news site as a .py file from somewhere, how do I use it?

Start the Add custom news sources dialog (from the Fetch news menu) and click the Switch to advanced mode button. Delete everything in the box with the recipe source code and copy paste the contents of your .py file into the box. Click Add/update recipe.

I want calibre to download news from my favorite news website.

If you are reasonably proficient with computers, you can teach calibre to download news from any website of your choosing. To learn how to do this see Adding your favorite news website.

Otherwise, you can request a particular news site by posting in the calibre Recipes forum [http://www.mobileread.com/forums/forumdisplay.php?f=228].

Can I use web2disk to download an arbitrary website?

web2disk http://mywebsite.com

Miscellaneous

Contents

- Why the name calibre?

- Why does calibre show only some of my fonts on OS X?

- calibre is not starting on Windows?

- calibre freezes/crashes occasionally?

- calibre is not starting on OS X?

- I downloaded the installer, but it is not working?

- My antivirus program claims calibre is a virus/trojan?

- How do I backup calibre?

- How do I use purchased EPUB books with calibre (or what do I do with .acsm files)?

- I am getting a “Permission Denied” error?

- Can I have the comment metadata show up on my reader?

- How do I get calibre to use my HTTP proxy?

- I want some feature added to calibre. What can I do?

- Why doesn’t calibre have an automatic update?

- How is calibre licensed?

- How do I run calibre from my USB stick?

- How do I run parts of calibre like news download and the content server on my own linux server?

Why the name calibre?

Take your pick:

- Convertor And LIBRary for Ebooks

- A high calibre product

- A tribute to the SONY Librie which was the first e-ink based ebook reader

- My wife chose it ;-)

calibre is pronounced as cal-i-ber not ca-li-bre. If you’re wondering, calibre is the British/commonwealth spelling for caliber. Being Indian, that’s the natural spelling for me.

Why does calibre show only some of my fonts on OS X?

calibre embeds fonts in ebook files it creates. Ebook files support embedding only TrueType (.ttf) fonts. Most fonts on OS X systems are in .dfont format, thus they cannot be embedded. calibre shows only TrueType fonts found on your system. You can obtain many TrueType fonts on the web. Simply download the .ttf files and add them to the Library/Fonts directory in your home directory.

calibre is not starting on Windows?

There can be several causes for this:

- If you get an error about calibre not being able to open a file because it is in use by another program, do the following:

- Uninstall calibre

- Reboot your computer

- Re-install calibre. But do not start calibre from the installation wizard.

- Temporarily disable your antivirus program (disconnect from the Internet before doing so, to be safe)

- Look inside the folder you chose for your calibre library. If you see a file named metadata.db, delete it.

- Start calibre

- From now on you should be able to start calibre normally.

- If you get an error about a Python function terminating unexpectedly after upgrading calibre, first uninstall calibre, then delete the folders (if they exists) C:\Program Files\Calibre and C:\Program Files\Calibre2. Now re-install and you should be fine.

- If you get an error in the welcome wizard on an initial run of calibre, try choosing a folder like C:\library as the calibre library (calibre sometimes has trouble with library locations if the path contains non-English characters, or only numbers, etc.)

- Try running it as Administrator (Right click on the icon and select “Run as Administrator”)

- Windows Vista: If the folder C:\Users\Your User Name\AppData\Local\VirtualStore\Program Files\calibre exists, delete it. Uninstall calibre. Reboot. Re-install.

- Any windows version: Try disabling any antivirus program you have running and see if that fixes it. Also try disabling any firewall software that prevents connections to the local computer.

If it still wont launch, start a command prompt (press the windows key and R; then type cmd.exe in the Run dialog that appears). At the command prompt type the following command and press Enter:

calibre-debug -g

Post any output you see in a help message on the Forum [http://www.mobileread.com/forums/forumdisplay.php?f=166].

calibre freezes/crashes occasionally?

There are three possible things I know of, that can cause this:

- You recently connected an external monitor or TV to your computer. In this case, whenever calibre opens a new window like the edit metadata window or the conversion dialog, it appears on the second monitor where you dont notice it and so you think calibre has frozen. Disconnect your second monitor and restart calibre.

- You are using a Wacom branded mouse. There is an incompatibility between
Wacom mice and the graphics toolkit calibre uses. Try using a non-Wacom mouse.

- If you use RoboForm, it is known to cause calibre to crash. Add calibre to the blacklist of programs inside RoboForm to fix this. Or uninstall
RoboForm.

calibre is not starting on OS X?

One common cause of failures on OS X is the use of accessibility technologies that are incompatible with the graphics toolkit calibre uses.
Try turning off VoiceOver if you have it on. Also go to System Preferences->System->Universal Access and turn off the setting for enabling access for assistive devices in all the tabs.

You can obtain debug output about why calibre is not starting by running Console.app. Debug output will be printed to it. If the debug output contains a line that looks like:

Qt: internal: -108: Error ATSUMeasureTextImage text/qfontengine_mac.mm

then the problem is probably a corrupted font cache. You can clear the cache by following these instructions [http://www.macworld.com/article/139383/2009/03/fontcacheclear.html]. If that doesn’t solve it, look for a corrupted font file on your system, in ~/Library/Fonts or the like. An easy way to check for corrupted fonts in OS X is to start the “Font Book” application, select all fonts and then in the File menu, choose “Validate fonts”.

I downloaded the installer, but it is not working?

Downloading from the Internet can sometimes result in a corrupted download. If the calibre installer you downloaded is not opening, try downloading it again. If re-downloading it does not work, download it from an alternate location [http://sourceforge.net/projects/calibre/files/]. If the installer still doesn’t work, then something on your computer is preventing it from running.

- Try temporarily disabling your antivirus program (Microsoft Security Essentials, or Kaspersky or Norton or McAfee or whatever). This is most likely the culprit if the upgrade process is hanging in the middle.

- Try rebooting your computer and running a registry cleaner like Wise registry cleaner [http://www.wisecleaner.com].

- Try downloading the installer with an alternate browser. For example if you are using Internet Explorer, try using Firefox or Chrome instead.

If you still cannot get the installer to work and you are on windows, you can use the calibre portable install [http://calibre-ebook.com/download_portable], which does not need an installer (it is just a zip file).

My antivirus program claims calibre is a virus/trojan?

The first thing to check is that you are downloading calibre from the official website: http://calibre-ebook.com/download. calibre is a very popular program and unscrupulous people try to setup websites offering it for download to fool the unwary.

If you have the official download and your antivirus program is still claiming calibre is a virus, then, your antivirus program is wrong. Antivirus programs use heuristics, patterns of code that “look suspicious” to detect viruses. It’s rather like racial profiling. calibre is a completely open source product. You can actually browse the source code yourself (or hire someone to do it for you)
to verify that it is not a virus. Please report the false identification to whatever company you buy your antivirus software from. If the antivirus program is preventing you from downloading/installing calibre, disable it temporarily, install calibre and then re-enable it.

How do I backup calibre?

The most important thing to backup is the calibre library folder, that contains all your books and metadata. This is the folder you chose for your calibre library when you ran calibre for the first time. You can get the path to the library folder by clicking the calibre icon on the main toolbar. You must backup this complete folder with all its files and sub-folders.

You can switch calibre to using a backed up library folder by simply clicking the calibre icon on the toolbar and choosing your backup library folder. A backed up library folder backs up your custom columns and saved searches as well as all your books and metadata.

If you want to backup the calibre configuration/plugins, you have to backup the config directory. You can find this config directory via Preferences->Miscellaneous. Note that restoring configuration directories is not officially supported, but should work in most cases. Just copy the contents of the backup directory into the current configuration directory to restore.

How do I use purchased EPUB books with calibre (or what do I do with .acsm files)?

Most purchased EPUB books have DRM [http://drmfree.calibre-ebook.com/about#drm]. This prevents calibre from opening them. You can still use calibre to store and transfer them to your ebook reader. First, you must authorize your reader on a windows machine with Adobe Digital Editions. Once this is done, EPUB books transferred with calibre will work fine on your reader. When you purchase an epub book from a website, you will get an ”.acsm” file. This file should be opened with Adobe Digital Editions, which will then download the actual ”.epub” ebook. The ebook file will be stored in the folder “My Digital Editions”, from where you can add it to calibre.

I am getting a “Permission Denied” error?

A permission denied error can occur because of many possible reasons, none of them having anything to do with calibre.

- You can get permission denied errors if you are using an SD card with write protect enabled.

- If you, or some program you used changed the file permissions of the files in question to read only.

- If there is a filesystem error on the device which caused your operating system to mount the filesystem in read only mode or mark a particular file as read only pending recovery.

- If the files have their owner set to a user other than you.

- If your file is open in another program.

- If the file resides on a device, you may have reached the limit of a maximum of 256 files in the root of the device. In this case you need to reformat the device/sd card referered to in the error message with a FAT32 filesystem, or delete some files from the SD card/device memory.

You will need to fix the underlying cause of the permissions error before resuming to use calibre. Read the error message carefully, see what file it points to and fix the permissions on that file.

Can I have the comment metadata show up on my reader?

Most readers do not support this. You should complain to the manufacturer about it and hopefully if enough people complain, things will change. In the meantime, you can insert the metadata, including comments into a “Jacket page” at the start of the ebook, by using the option to “Insert metadata as page at start of book” during conversion. The option is found in the Structure Detection section of the conversion settings. Note that for this to have effect you have to convert the book. If your book is already in a format that does not need conversion, you can convert from that format to the same format.

Another alternative is to create a catalog in ebook form containing a listing of all the books in your calibre library, with their metadata. Click-and-hold the convert button to access the catalog creation tool. And before you ask, no you cannot have the catalog “link directly to” books on your reader.

How do I get calibre to use my HTTP proxy?

By default, calibre uses whatever proxy settings are set in your OS. Sometimes these are incorrect, for example, on windows if you don’t use Internet Explorer then the proxy settings may not be up to date. You can tell calibre to use a particular proxy server by setting the http_proxy environment variable. The format of the variable is: http://username:password@servername you should ask your network admin to give you the correct value for this variable. Note that calibre only supports HTTP proxies not SOCKS proxies. You can see the current proxies used by calibre in Preferences->Miscellaneous.

I want some feature added to calibre. What can I do?

You have two choices:

1. Create a patch by hacking on calibre and send it to me for review and inclusion. See Development [http://calibre-ebook.com/get-involved].

2. Open a bug requesting the feature [http://calibre-ebook.com/bugs] . Remember that while you may think your feature request is extremely important/essential, calibre developers might not agree. Fortunately, calibre is open source, which means you always have the option of implementing your feature yourself, or hiring someone to do it for you. Furthermore, calibre has a comprehensive plugin architecture, so you might be able to develop your feature as a plugin, see Writing your own plugins to extend calibre’s functionality.

Why doesn’t calibre have an automatic update?

For many reasons:

- There is no need to update every week. If you are happy with how calibre works turn off the update notification and be on your merry way. Check back to see if you want to update once a year or so.

- Pre downloading the updates for all users in the background would mean require about 80TB of bandwidth every week. That costs thousands of dollars a month. And calibre is currently growing at 300,000 new users every month.

- If I implement a dialog that downloads the update and launches it, instead of going to the website as it does now, that would save the most ardent calibre updater, at most five clicks a week. There are far higher priority things to do in calibre development.

- If you really, really hate downloading calibre every week but still want to be up to the latest, I encourage you to run from source, which makes updating trivial. Instructions are available here.

How is calibre licensed?

calibre is licensed under the GNU General Public License v3 (an open source license). This means that you are free to redistribute calibre as long as you make the source code available. So if you want to put calibre on a CD with your product, you must also put the calibre source code on the CD. The source code is available for download from googlecode [http://code.google.com/p/calibre-ebook/downloads/list]. You are free to use the results of conversions from calibre however you want. You cannot use code, libraries from calibre in your software without making your software open source. For details, see The GNU GPL v3 [http://www.gnu.org/licenses/gpl.html].

How do I run calibre from my USB stick?

A portable version of calibre is available here [http://calibre-ebook.com/download_portable].

How do I run parts of calibre like news download and the content server on my own linux server?

First, you must install calibre onto your linux server. If your server is using a modern linux distro, you should have no problems installing calibre onto it.

Note

If you bought into the notion that a real server must run a decade old version of Debian, then you will have to jump through a few hoops. First, compile a newer version of glibc (>= 2.10) on your server from source. Then get the calibre linux binary tarball from the calibre google code page for your server architecture. Extract it into /opt/calibre. Put your previously compiled glibc into /opt/calibre as libc.so.6. You can now run the calibre binaries from /opt/calibre.

You can run the calibre server via the command:

/opt/calibre/calibre-server --with-library /path/to/the/library/you/want/to/share

You can download news and convert it into an ebook with the command:

/opt/calibre/ebook-convert "Title of news source.recipe" outputfile.epub

If you want to generate MOBI, use outputfile.mobi instead and use --output-profile kindle.

You can email downloaded news with the command:

/opt/calibre/calibre-smtp

I leave figuring out the exact command line as an exercise for the reader.

Finally, you can add downloaded news to the calibre library with:

/opt/calibre/calibredb add --with-library /path/to/library outfile.epub

Remember to read the command line documentation section of the calibre User Manual to learn more about these, and other commands.

Note

Some parts of calibre require a X server. If you’re lucky, nothing you do will fall into this category, if not, you will have to look into using xvfb.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

Tutorials

Here you will find tutorials to get you started using calibre‘s more advanced features, such as XPath and templates.

- Adding your favorite news website

- Managing subgroups of books, for example “genre”

- XPath Tutorial

- The calibre template language

- All about using regular expressions in calibre

- Integrating the calibre content server into other servers

- Writing your own plugins to extend calibre‘s functionality

- Typesetting Math in ebooks

- Creating AZW3 • EPUB • MOBI Catalogs

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

Managing subgroups of books, for example “genre”

Some people wish to organize the books in their library into subgroups, similar to subfolders. The most commonly provided reason is to create genre hierarchies, but there are many others. One user asked for a way to organize textbooks by subject and course number. Another wanted to keep track of gifts by subject and recipient. This tutorial will use the genre example for the rest of this post.

Before going on, please note that we are not talking about folders on the hard disk. Subgroups are not file folders. Books will not be copied anywhere. Calibre’s library file structure is not affected. Instead, we are presenting a way to organize and display subgroups of books within a calibre library.

- Setup
- Searching
- Restrictions
- Useful Template Functions

The commonly-provided requirements for subgroups such as genres are:

- A subgroup (e.g., a genre) must contain (point to) books, not categories of books. This is what distinguishes subgroups from calibre user categories.

- A book can be in multiple subgroups (genres). This distinguishes subgroups from physical file folders.

- Subgroups (genres) must form a hierarchy; subgroups can contain subgroups.

Tags give you the first two. If you tag a book with the genre then you can use the tag browser (or search) for find the books with that genre, giving you the first. Many books can have the same tag(s), giving you the second. The problem is that tags don’t satisfy the third requirement. They don’t provide a hierarchy.

Calibre’s hierarchy feature gives you the third, the ability to see the genres in a ‘tree’ and the ability to easily search for books in genre or sub-genre. For example, assume that your genre structure is similar to the following:

Genre
. History
.. Japanese
.. Military
.. Roman
. Mysteries
.. English
.. Vampire
. Science Fiction
.. Alternate History
.. Military
.. Space Opera
. Thrillers
.. Crime
.. Horror
etc.

By using the hierarchy feature, you can see these genres in the tag browser in tree form, as shown in the screen image. In this example the outermost level (Genre) is a custom column that contains the genres. Genres containing sub-genres appear with a small triangle next to them. Clicking on that triangle will open the item and show the sub-genres, as you can see with History and Science Fiction.

Clicking on a genre can search for all books with that genre or children of that genre. For example, clicking on Science Fiction can give all three of the child genres, Alternate History, Military, and Space Opera. Clicking on Alternate History will give books in that genre, ignoring those in Military and Space Opera. Of course, a book can have multiple genres. If a book has both Space Opera and Military genres, then you will see that book if you click on either genre. Searching is discussed in more detail below.

Another thing you can see from the image is that the genre Military appears twice, once under History and once under Science Fiction. Because the genres are in a hierarchy, these are two separate genres. A book can be in one, the other, or (doubtfully in this case) both. For example, the books in Winston Churchill’s “The Second World War” could be in “History.Military”. David Weber’s Honor Harrington books could be in “Science Fiction.Military”, and for that matter also in “Science Fiction.Space Opera.”

Once a genre exists, that is at least one book has that genre, you can easily apply it to other books by dragging the books from the library view onto the genre you want the books to have. You can also apply genres in the metadata editors; more on this below.

Setup

By now, your question might be “How was all of this up?” There are three steps: 1) create the custom column, 2) tell calibre that the new column is to be treated as a hierarchy, and 3) add genres.

You create the custom column in the usual way, using Preferences -> Add your own columns. This example uses “#genre” as the lookup name and “Genre” as the column heading. The column type is “Comma-separated text, like tags, shown in the tag browser.”

Then after restarting calibre, you must tell calibre that the column is to be treated as a hierarchy. Go to Preferences -> Look and Feel -> Tag Browser and enter the lookup name “#genre” into the “Categories with hierarchical items” box. Press Apply, and you are done with setting up.

At the point there are no genres in the column. We are left with the last step: how to apply a genre to a book. A genre does not exist in calibre until it appears on at least one book. To learn how to apply a genre for the first time, we must go into some detail about what a genre looks like in the metadata for a book.

A hierarchy of ‘things’ is built by creating an item consisting of phrases separated by periods. Continuing the genre example, these items would “History.Military”, “Mysteries.Vampire”, “Science Fiction.Space Opera”, etc. Thus to create a new genre, you pick a book that should have that genre, edit its metadata, and enter the new genre into the column you created. Continuing our example, if you want to assign a new genre “Comics” with a sub-genre “Superheroes” to a book, you would ‘edit metadata’ for that (comic) book, choose the Custom metadata tab, and then enter “Comics.Superheroes” as shown in the following (ignore the other custom columns):

After doing the above, you see in the tag browser:

From here on, to apply this new genre to a book (a comic book, presumably), you can either drag the book onto the genre, or add it to the book using edit metadata in exactly the same way as done above.

Searching

The easiest way to search for genres is using the tag browser, clicking on the genre you wish to see. Clicking on a genre with children will show you books with that genre and all child genres. However, this might bring up a question. Just because a genre has children doesn’t mean that it isn’t a genre in its own right. For example, a book can have the genre “History” but not “History.Military”. How do you search for books with only “History”?

The tag browser search mechanism knows if an item has children. If it does, clicking on the item cycles through 5 searches instead of the normal three. The first is the normal green plus, which shows you books with that genre only (e.g., History). The second is a doubled plus (shown above), which shows you books with that genre and all sub-genres (e.g., History and History.Military). The third is the normal red minus, which shows you books without that exact genre. The fourth is a doubled minus, which shows you books without that genre or sub-genres. The fifth is back to the beginning, no mark, meaning no search.

Restrictions

If you search for a genre then create a saved search for it, you can use the ‘restrict to’ box to create a virtual library of books with that genre. This is useful if you want to do other searches within the genre or to manage/update metadata for books in the genre. Continuing our example, you can create a saved search named ‘History.Japanese’ by first clicking on the genre Japanese in the tag browser to get a search into the search box, entering History.Japanese into the saved search box, then pushing the “save search” button (the green box with the white plus, on the right-hand side).

After creating the saved search, you can use it as a restriction.

Useful Template Functions

You might want to use the genre information in a template, such as with save to disk or send to device. The question might then be “How do I get the outermost genre name or names?” A calibre template function, subitems, is provided to make doing this easier.

For example, assume you want to add the outermost genre level to the save-to-disk template to make genre folders, as in “History/The Gathering Storm - Churchill, Winston”. To do this, you must extract the first level of the hierarchy and add it to the front along with a slash to indicate that it should make a folder. The template below accomplishes this:

{#genre:subitems(0,1)||/}{title} - {authors}

See The template language for more information templates and the subitems() function.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

XPath Tutorial

In this tutorial, you will be given a gentle introduction to XPath [http://en.wikipedia.org/wiki/XPath], a query language that can be used to select arbitrary parts of HTML [http://en.wikipedia.org/wiki/HTML] documents in calibre. XPath is a widely used standard, and googling it will yield a ton of information. This tutorial, however, focuses on using XPath for ebook related tasks like finding chapter headings in an unstructured HTML document.

Contents

- Selecting by tagname
- Selecting by attributes
- Selecting by tag content
- Sample ebook
- XPath built-in functions

Selecting by tagname

The simplest form of selection is to select tags by name. For example, suppose you want to select all the <h2> tags in a document. The XPath query for this is simply:

//h:h2 (Selects all <h2> tags)

The prefix // means search at any level of the document. Now suppose you want to search for <span> tags that are inside <a> tags. That can be achieved with:

//h:a/h:span (Selects <span> tags inside <a> tags)

If you want to search for tags at a particular level in the document, change the prefix:

/h:body/h:div/h:p (Selects <p> tags that are children of <div> tags that are
children of the <body> tag)

This will match only <p>A very short ebook to demonstrate the use of XPath.</p> in the Sample ebook but not any of the other <p> tags. The h: prefix in the above examples is needed to match XHTML tags. This is because internally, calibre represents all content as XHTML. In XHTML tags have a namespace, and h: is the namespace prefix for HTML tags.

Now suppose you want to select both <h1> and <h2> tags. To do that, we need a XPath construct called predicate. A predicate is simply a test that is used to select tags. Tests can be arbitrarily powerful and as this tutorial progresses, you will see more powerful examples. A predicate is created by enclosing the test expression in square brackets:

//*[name()='h1' or name()='h2']

There are several new features in this XPath expression. The first is the use of the wildcard *. It means match any tag. Now look at the test expression name()='h1' or name()='h2'. name() is an example of a built-in function.
It simply evaluates to the name of the tag. So by using it, we can select tags whose names are either h1 or h2. Note that the name() function ignores namespaces so that there is no need for the h: prefix.
XPath has several useful built-in functions. A few more will be introduced in this tutorial.

Selecting by attributes

To select tags based on their attributes, the use of predicates is required:

//*[@style] (Select all tags that have a style attribute)
//*[@class="chapter"] (Select all tags that have class="chapter")
//h:h1[@class="bookTitle"] (Select all h1 tags that have class="bookTitle")

Here, the @ operator refers to the attributes of the tag. You can use some of the XPath built-in functions to perform more sophisticated matching on attribute values.

Selecting by tag content

Using XPath, you can even select tags based on the text they contain. The best way to do this is to use the power of regular expressions via the built-in function re:test():

//h:h2[re:test(., 'chapter|section', 'i')] (Selects <h2> tags that contain the words chapter or
section)

Here the . operator refers to the contents of the tag, just as the @ operator referred to its attributes.

Sample ebook

<html>
<head>
<title>A very short ebook</title>
<meta name="charset" value="utf-8" />
</head>
<body>
<h1 class="bookTitle">A very short ebook</h1>
<p style="text-align:right">Written by Kovid Goyal</p>
<div class="introduction">
<p>A very short ebook to demonstrate the use of XPath.</p>
</div>

<h2 class="chapter">Chapter One</h2>
<p>This is a truly fascinating chapter.</p>

<h2 class="chapter">Chapter Two</h2>
<p>A worthy continuation of a fine tradition.</p>
</body>
</html>

XPath built-in functions

name()
The name of the current tag.
contains()

contains(s1, s2) returns true if s1 contains s2.

re:test()

re:test(src, pattern, flags) returns true if the string src matches the regular expression pattern. A particularly useful flag is i, it makes matching case insensitive. A good primer on the syntax for regular expressions can be found at regexp syntax [http://docs.python.org/lib/re-syntax.html]

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

The calibre template language

The calibre template language is used in various places. It is used to control the folder structure and file name when saving files from the calibre library to the disk or eBook reader.
It is also used to define “virtual” columns that contain data from other columns and so on.

The basic template language is very simple, but has very powerful advanced features. The basic idea is that a template consists of text and names in curly brackets that are then replaced by the corresponding metadata from the book being processed. So, for example, the default template used for saving books to device in calibre is:

{author_sort}/{title}/{title} - {authors}

For the book “The Foundation” by “Isaac Asimov” it will become:

Asimov, Isaac/The Foundation/The Foundation - Isaac Asimov

The slashes are text, which is put into the template where it appears. For example, if your template is:

{author_sort} Some Important Text {title}/{title} - {authors}

For the book “The Foundation” by “Isaac Asimov” it will become:

Asimov, Isaac Some Important Text The Foundation/The Foundation - Isaac Asimov

You can use all the various metadata fields available in calibre in a template, including any custom columns you have created yourself. To find out the template name for a column simply hover your mouse over the column header. Names for custom fields (columns you have created yourself) always have a # as the first character. For series type custom fields, there is always an additional field named #seriesname_index that becomes the series index for that series. So if you have a custom series field named #myseries, there will also be a field named #myseries_index.

In addition to the column based fields, you also can use:

{formats} - A list of formats available in the calibre library for a book
{identifiers:select(isbn)} - The ISBN number of the book

If a particular book does not have a particular piece of metadata, the field in the template is automatically removed for that book. Consider, for example:

{author_sort}/{series}/{title} {series_index}

If a book has a series, the template will produce:

Asimov, Isaac/Foundation/Second Foundation 3

and if a book does not have a series:

Asimov, Isaac/Second Foundation

(calibre automatically removes multiple slashes and leading or trailing spaces).

Advanced formatting

You can do more than just simple substitution with the templates. You can also conditionally include text and control how the substituted data is formatted.

First, conditionally including text. There are cases where you might want to have text appear in the output only if a field is not empty. A common case is series and series_index, where you want either nothing or the two values with a hyphen between them. Calibre handles this case using a special field syntax.

For example, assume you want to use the template:

{series} - {series_index} - {title}

If the book has no series, the answer will be - - title. Many people would rather the result be simply title, without the hyphens. To do this, use the extended syntax {field:|prefix_text|suffix_text}. When you use this syntax, if field has the value SERIES then the result will be prefix_textSERIESsuffix_text. If field has no value, then the result will be the empty string (nothing); the prefix and suffix are ignored. The prefix and suffix can contain blanks. Do not use subtemplates (`{ ... }`) or functions (see below) as the prefix or the suffix.

Using this syntax, we can solve the above series problem with the template:

{series}{series_index:| - | - }{title}

The hyphens will be included only if the book has a series index, which it will have only if it has a series.

Notes: you must include the : character if you want to use a prefix or a suffix. You must either use no | characters or both of them; using one, as in {field:| - }, is not allowed. It is OK not to provide any text for one side or the other, such as in {series:|| - }. Using {title:||} is the same as using {title}.

Second: formatting. Suppose you wanted to ensure that the series_index is always formatted as three digits with leading zeros. This would do the trick:

{series_index:0>3s} - Three digits with leading zeros

If instead of leading zeros you want leading spaces, use:

{series_index:>3s} - Three digits with leading spaces

For trailing zeros, use:

{series_index:0<3s} - Three digits with trailing zeros

If you use series indices with sub values (e.g., 1.1), you might want to ensure that the decimal points line up. For example, you might want the indices 1 and 2.5 to appear as 01.00 and 02.50 so that they will sort correctly. To do this, use:

{series_index:0>5.2f} - Five characters, consisting of two digits with leading zeros, a decimal point, then 2 digits after the decimal point

If you want only the first two letters of the data, use:

{author_sort:.2} - Only the first two letter of the author sort name

The calibre template language comes from python and for more details on the syntax of these advanced formatting operations, look at the Python documentation [http://docs.python.org/library/string.html#format-string-syntax].

Advanced features

Using templates in custom columns

There are sometimes cases where you want to display metadata that calibre does not normally display, or to display data in a way different from how calibre normally does. For example, you might want to display the ISBN, a field that calibre does not display. You can use custom columns for this by creating a column with the type ‘column built from other columns’ (hereafter called composite columns), and entering a template. Result: calibre will display a column showing the result of evaluating that template. To display the ISBN, create the column and enter {identifiers:select(isbn)} into the template box. To display a column containing the values of two series custom columns separated by a comma, use {#series1:||,}{#series2}.

Composite columns can use any template option, including formatting.

You cannot change the data contained in a composite column. If you edit a composite column by double-clicking on any item, you will open the template for editing, not the underlying data. Editing the template on the GUI is a quick way of testing and changing composite columns.

Using functions in templates - single-function mode

Suppose you want to display the value of a field in upper case, when that field is normally in title case. You can do this (and many more things) using the functions available for templates. For example, to display the title in upper case, use {title:uppercase()}. To display it in title case, use {title:titlecase()}.

Function references appear in the format part, going after the : and before the first | or the closing }. If you have both a format and a function reference, the function comes after another :. Functions must always end with (). Some functions take extra values (arguments), and these go inside the ().

Functions are always applied before format specifications. See further down for an example of using both a format and a function, where this order is demonstrated.

The syntax for using functions is {field:function(arguments)}, or {field:function(arguments)|prefix|suffix}. Arguments are separated by commas. Commas inside arguments must be preceeded by a backslash ( ‘\’ ). The last (or only) argument cannot contain a closing parenthesis ( ‘)’ ). Functions return the value of the field used in the template, suitably modified.

Important: If you have programming experience, please note that the syntax in this mode (single function) is not what you might expect. Strings are not quoted. Spaces are significant. All arguments must be constants; there is no sub-evaluation. Do not use subtemplates (`{ ... }`) as function arguments. Instead, use template program mode and general program mode.

Many functions use regular expressions. In all cases, regular expression matching is case-insensitive.

The functions available are listed below. Note that the definitive documentation for functions is available in the section Function classification:

lowercase() – return value of the field in lower case.

uppercase() – return the value of the field in upper case.

titlecase() – return the value of the field in title case.

capitalize() – return the value with the first letter upper case and the rest lower case.

contains(pattern, text if match, text if not match) – checks if field contains matches for the regular expression pattern. Returns text if match if matches are found, otherwise it returns text if no match.

count(separator) – interprets the value as a list of items separated by separator, returning the number of items in the list. Most lists use a comma as the separator, but authors uses an ampersand. Examples: {tags:count(,)}, {authors:count(&)}

format_number(template) – interprets the value as a number and format that number using a python formatting template such as “{0:5.2f}” or “{0:,d}” or “${0:5,.2f}”. The field_name part of the template must be a 0 (zero) (the “{0:” in the above examples). See the template language and python documentation for more examples. Returns the empty string if formatting fails.

human_readable() – expects the value to be a number and returns a string representing that number in KB, MB, GB, etc.

ifempty(text) – if the field is not empty, return the value of the field. Otherwise return text.

in_list(separator, pattern, found_val, not_found_val) – interpret the field as a list of items separated by separator, comparing the pattern against each value in the list. If the pattern matches a value, return found_val, otherwise return not_found_val.

language_codes(lang_strings) – return the language codes for the strings passed in lang_strings. The strings must be in the language of the current locale. Lang_strings is a comma-separated list.

language_strings(lang_codes, localize) – return the strings for the language codes passed in lang_codes. If localize is zero, return the strings in English. If localize is not zero, return the strings in the language of the current locale. Lang_codes is a comma-separated list.

list_item(index, separator) – interpret the field as a list of items separated by separator, returning the index`th item. The first item is number zero. The last item can be returned using `list_item(-1,separator). If the item is not in the list, then the empty value is returned. The separator has the same meaning as in the count function.

re(pattern, replacement) – return the field after applying the regular expression. All instances of pattern are replaced with replacement. As in all of calibre, these are python-compatible regular expressions.

shorten(left chars, middle text, right chars) – Return a shortened version of the field, consisting of left chars characters from the beginning of the field, followed by middle text, followed by right chars characters from the end of the string. Left chars and right chars must be integers. For example, assume the title of the book is Ancient English Laws in the Times of Ivanhoe, and you want it to fit in a space of at most 15 characters. If you use {title:shorten(9,-,5)}, the result will be Ancient E-nhoe. If the field’s length is less than left chars + right chars + the length of middle text, then the field will be used intact. For example, the title The Dome would not be changed.

swap_around_comma(val) `` -- given a value of the form ``B, A, return A B. This is most useful for converting names in LN, FN format to FN LN. If there is no comma, the function returns val unchanged.

switch(pattern, value, pattern, value, ..., else_value) – for each pattern, value pair, checks if the field matches the regular expression pattern and if so, returns that value. If no pattern matches, then else_value is returned. You can have as many pattern, value pairs as you want.

lookup(pattern, field, pattern, field, ..., else_field) – like switch, except the arguments are field (metadata) names, not text. The value of the appropriate field will be fetched and used. Note that because composite columns are fields, you can use this function in one composite field to use the value of some other composite field. This is extremely useful when constructing variable save paths (more later).

select(key) – interpret the field as a comma-separated list of items, with the items being of the form “id:value”. Find the pair with the id equal to key, and return the corresponding value. This function is particularly useful for extracting a value such as an isbn from the set of identifiers for a book.

str_in_list(val, separator, string, found_val, not_found_val) – treat val as a list of items separated by separator, comparing the string against each value in the list. If the string matches a value, return found_val, otherwise return not_found_val. If the string contains separators, then it is also treated as a list and each value is checked.

subitems(val, start_index, end_index) – This function is used to break apart lists of tag-like hierarchical items such as genres. It interprets the value as a comma-separated list of tag-like items, where each item is a period-separated list. Returns a new list made by first finding all the period-separated tag-like items, then for each such item extracting the components from start_index to end_index, then combining the results back together. The first component in a period-separated list has an index of zero. If an index is negative, then it counts from the end of the list. As a special case, an end_index of zero is assumed to be the length of the list. Examples:

Assuming a #genre column containing "A.B.C":
{#genre:subitems(0,1)} returns "A"
{#genre:subitems(0,2)} returns "A.B"
{#genre:subitems(1,0)} returns "B.C"
Assuming a #genre column containing "A.B.C, D.E":
{#genre:subitems(0,1)} returns "A, D"
{#genre:subitems(0,2)} returns "A.B, D.E"

sublist(val, start_index, end_index, separator) – interpret the value as a list of items separated by separator, returning a new list made from the items from start_index`to `end_index. The first item is number zero. If an index is negative, then it counts from the end of the list. As a special case, an end_index of zero is assumed to be the length of the list. Examples assuming that the tags column (which is comma-separated) contains “A, B ,C”:

{tags:sublist(0,1,\,)} returns "A"
{tags:sublist(-1,0,\,)} returns "C"
{tags:sublist(0,-1,\,)} returns "A, B"

test(text if not empty, text if empty) – return text if not empty if the field is not empty, otherwise return text if empty.

Now, what about using functions and formatting in the same field. Suppose you have an integer custom column called #myint that you want to see with leading zeros, as in 003. To do this, you would use a format of 0>3s. However, by default, if a number (integer or float) equals zero then the field produces the empty value, so zero values will produce nothing, not 000. If you really want to see 000 values, then you use both the format string and the ifempty function to change the empty value back to a zero. The field reference would be:

{#myint:0>3s:ifempty(0)}

Note that you can use the prefix and suffix as well. If you want the number to appear as [003] or [000], then use the field:

{#myint:0>3s:ifempty(0)|[|]}

Using functions in templates - template program mode

The template language program mode differs from single-function mode in that it permits you to write template expressions that refer to other metadata fields, modify values, and do arithmetic. It is a reasonably complete programming language.

You can use the functions documented above in template program mode. See below for details.

Beginning with an example, assume that you want your template to show the series for a book if it has one, otherwise show the value of a custom field #genre. You cannot do this in the basic language because you cannot make reference to another metadata field within a template expression. In program mode, you can. The following expression works:

{#series:'ifempty($, field('#genre'))'}

The example shows several things:

- program mode is used if the expression begins with :' and ends with '. Anything else is assumed to be single-function.

- the variable $ stands for the field the expression is operating upon, #series in this case.

- functions must be given all their arguments. There is no default value. For example, the standard built-in functions must be given an additional initial parameter indicating the source field, which is a significant difference from single-function mode.

- white space is ignored and can be used anywhere within the expression.

- constant strings are enclosed in matching quotes, either ' or ".

The language is similar to functional languages in that it is built almost entirely from functions. A statement is a function. An expression is a function. Constants and identifiers can be thought of as functions returning the value indicated by the constant or stored in the identifier.

The syntax of the language is shown by the following grammar:

constant ::= " string " | ' string ' | number
identifier ::= sequence of letters or ``_`` characters
function ::= identifier ( statement [ , statement ]* )
expression ::= identifier | constant | function | assignment
assignment ::= identifier '=' expression
statement ::= expression [ ; expression ]*
program ::= statement

Comments are lines with a ‘#’ character at the beginning of the line.

An expression always has a value, either the value of the constant, the value contained in the identifier, or the value returned by a function. The value of a statement is the value of the last expression in the sequence of statements. As such, the value of the program (statement):

1; 2; 'foobar'; 3

is 3.

Another example of a complex but rather silly program might help make things clearer:

{series_index:'
substr(
strcat($, '->',
cmp(divide($, 2), 1,
assign(c, 1); substr('lt123', c, 0),
'eq', 'gt')),
0, 6)
'| prefix | suffix}

This program does the following:

- specify that the field being looked at is series_index. This sets the value of the variable $.

- calls the substr function, which takes 3 parameters (str, start, end). It returns a string formed by extracting the start through end characters from string, zero-based (the first character is character zero). In this case the string will be computed by the strcat function, the start is 0, and the end is 6. In this case it will return the first 6 characters of the string returned by strcat, which must be evaluated before substr can return.

- calls the strcat (string concatenation) function. Strcat accepts 1 or more arguments, and returns a string formed by concatenating all the values. In this case there are three arguments. The first parameter is the value in $, which here is the value of series_index. The second paremeter is the constant string '->'. The third parameter is the value returned by the cmp function, which must be fully evaluated before strcat can return.

- The cmp function takes 5 arguments (x, y, lt, eq, gt). It compares x and y and returns the third argument lt if x < y, the fourth argument eq if x == y, and the fifth argument gt if x > y. As with all functions, all of the parameters can be statements. In this case the first parameter (the value for x) is the result of dividing the series_index by 2. The second parameter y is the constant 1. The third parameter lt is a statement (more later). The fourth parameter eq is the constant string 'eq'. The fifth parameter is the constant string 'gt'.

- The third parameter (the one for lt) is a statement, or a sequence of expressions. Remember that a statement (a sequence of semicolon-separated expressions) is also an expression, returning the value of the last expression in the list. In this case, the program first assigns the value 1 to a local variable c, then returns a substring made by extracting the c’th character to the end. Since c always contains the constant 1, the substring will return the second through end’th characters, or 't123'.

- Once the statement providing the value to the third parameter is executed, cmp can return a value. At that point, strcat` can return a value, then ``substr can return a value. The program then terminates.

For various values of series_index, the program returns:

- series_index == undefined, result = prefix ->t123 suffix

- series_index == 0.5, result = prefix 0.50-> suffix

- series_index == 1, result = prefix 1->t12 suffix

- series_index == 2, result = prefix 2->eq suffix

- series_index == 3, result = prefix 3->gt suffix

All the functions listed under single-function mode can be used in program mode. To do so, you must supply the value that the function is to act upon as the first parameter, in addition to the parameters documented above. For example, in program mode the parameters of the test function are test(x, text_if_not_empty, text_if_empty). The x parameter, which is the value to be tested, will almost always be a variable or a function call, often field().

The following functions are available in addition to those described in single-function mode. Remember from the example above that the single-function mode functions require an additional first parameter specifying the field to operate on. With the exception of the id parameter of assign, all parameters can be statements (sequences of expressions). Note that the definitive documentation for functions is available in the section Function classification:

- and(value, value, ...) – returns the string “1” if all values are not empty, otherwise returns the empty string. This function works well with test or first_non_empty. You can have as many values as you want.

- add(x, y) – returns x + y. Throws an exception if either x or y are not numbers.

- assign(id, val) – assigns val to id, then returns val. id must be an identifier, not an expression

- approximate_formats() – return a comma-separated list of formats that at one point were associated with the book. There is no guarantee that the list is correct, although it probably is. This function can be called in template program mode using the template {:'approximate_formats()'}. Note that format names are always uppercase, as in EPUB.

- booksize() – returns the value of the calibre ‘size’ field. Returns ‘’ if there are no formats.

- cmp(x, y, lt, eq, gt) – compares x and y after converting both to numbers. Returns lt if x < y. Returns eq if x == y. Otherwise returns gt.

- current_library_name() -- `` return the last name on the path to the current calibre library. This function can be called in template program mode using the template ``{:'current_library_name()'}.

- current_library_path() -- `` eturn the path to the current calibre library. This function can be called in template program mode using the template ``{:'current_library_path()'}..

- days_between(date1, date2) – return the number of days between date1 and date2. The number is positive if date1 is greater than date2, otherwise negative. If either date1 or date2 are not dates, the function returns the empty string.

- divide(x, y) – returns x / y. Throws an exception if either x or y are not numbers.

- eval(string) – evaluates the string as a program, passing the local variables (those assign ed to). This permits using the template processor to construct complex results from local variables. Because the { and } characters are special, you must use [[ for the { character and ]] for the ‘}’ character; they are converted automatically. Note also that prefixes and suffixes (the |prefix|suffix syntax) cannot be used in the argument to this function when using template program mode.

- field(name) – returns the metadata field named by name.

- first_non_empty(value, value, ...) – returns the first value that is not empty. If all values are empty, then the empty value is returned. You can have as many values as you want.

- format_date(x, date_format) – format_date(val, format_string) – format the value, which must be a date field, using the format_string, returning a string. The formatting codes are:

d : the day as number without a leading zero (1 to 31)
dd : the day as number with a leading zero (01 to 31)
ddd : the abbreviated localized day name (e.g. "Mon" to "Sun").
dddd : the long localized day name (e.g. "Monday" to "Sunday").
M : the month as number without a leading zero (1 to 12).
MM : the month as number with a leading zero (01 to 12)
MMM : the abbreviated localized month name (e.g. "Jan" to "Dec").
MMMM : the long localized month name (e.g. "January" to "December").
yy : the year as two digit number (00 to 99).
yyyy : the year as four digit number.
h : the hours without a leading 0 (0 to 11 or 0 to 23, depending on am/pm)
hh : the hours with a leading 0 (00 to 11 or 00 to 23, depending on am/pm)
m : the minutes without a leading 0 (0 to 59)
mm : the minutes with a leading 0 (00 to 59)
s : the seconds without a leading 0 (0 to 59)
ss : the seconds with a leading 0 (00 to 59)
ap : use a 12-hour clock instead of a 24-hour clock, with 'ap' replaced by the localized string for am or pm.
AP : use a 12-hour clock instead of a 24-hour clock, with 'AP' replaced by the localized string for AM or PM.
iso : the date with time and timezone. Must be the only format present.

You might get unexpected results if the date you are formatting contains localized month names, which can happen if you changed the format tweaks to contain MMMM. In this case, instead of using something like {pubdate:format_date(yyyy)}, write the template using template program mode as in {:'format_date(raw_field('pubdate'),'yyyy')'}.

- finish_formatting(val, fmt, prefix, suffix) – apply the format, prefix, and suffix to a value in the same way as done in a template like {series_index:05.2f| - |- }. This function is provided to ease conversion of complex single-function- or template-program-mode templates to general program mode (see below) to take advantage of GPM template compilation. For example, the following program produces the same output as the above template:

program: finish_formatting(field("series_index"), "05.2f", " - ", " - ")

Another example: for the template {series:re(([^\s])[^\s]+(\s|$),\1)}{series_index:0>2s| - | - }{title} use:

program:
strcat(
re(field('series'), '([^\s])[^\s]+(\s|$)', '\1'),
finish_formatting(field('series_index'), '0>2s', ' - ', ' - '),
field('title')
)

formats_modtimes(date_format) – return a comma-separated list of colon_separated items representing modification times for the formats of a book. The date_format parameter specifies how the date is to be formatted. See the date_format function for details. You can use the select function to get the mod time for a specific format. Note that format names are always uppercase, as in EPUB.

formats_paths() – return a comma-separated list of colon_separated items representing full path to the formats of a book. You can use the select function to get the path for a specific format. Note that format names are always uppercase, as in EPUB.

formats_sizes() – return a comma-separated list of colon_separated items representing sizes in bytes of the formats of a book. You can use the select function to get the size for a specific format. Note that format names are always uppercase, as in EPUB.

has_cover() – return Yes if the book has a cover, otherwise return the empty string
not(value) – returns the string “1” if the value is empty, otherwise returns the empty string. This function works well with test or first_non_empty. You can have as many values as you want.

list_difference(list1, list2, separator) – return a list made by removing from list1 any item found in list2, using a case-insensitive compare. The items in list1 and list2 are separated by separator, as are the items in the returned list.

list_equals(list1, sep1, list2, sep2, yes_val, no_val) – return yes_val if list1 and list2 contain the same items, otherwise return no_val. The items are determined by splitting each list using the appropriate separator character (sep1 or sep2). The order of items in the lists is not relevant. The compare is case insensitive.

list_intersection(list1, list2, separator) – return a list made by removing from list1 any item not found in list2, using a case-insensitive compare. The items in list1 and list2 are separated by separator, as are the items in the returned list.

list_re(src_list, separator, search_re, opt_replace) – Construct a list by first separating src_list into items using the separator character. For each item in the list, check if it matches search_re. If it does, then add it to the list to be returned. If opt_replace is not the empty string, then apply the replacement before adding the item to the returned list.

list_sort(list, direction, separator) – return list sorted using a case-insensitive sort. If direction is zero, the list is sorted ascending, otherwise descending. The list items are separated by separator, as are the items in the returned list.

list_union(list1, list2, separator) – return a list made by merging the items in list1 and list2, removing duplicate items using a case-insensitive compare. If items differ in case, the one in list1 is used. The items in list1 and list2 are separated by separator, as are the items in the returned list.

multiply(x, y) – returns x * y. Throws an exception if either x or y are not numbers.

ondevice() – return the string “Yes” if ondevice is set, otherwise return the empty string
or(value, value, ...) – returns the string “1” if any value is not empty, otherwise returns the empty string. This function works well with test or first_non_empty. You can have as many values as you want.

print(a, b, ...) – prints the arguments to standard output. Unless you start calibre from the command line (calibre-debug -g), the output will go to a black hole.

raw_field(name) – returns the metadata field named by name without applying any formatting.

series_sort() – returns the series sort value.

strcat(a, b, ...) – can take any number of arguments. Returns a string formed by concatenating all the arguments.

strcat_max(max, string1, prefix2, string2, ...) – Returns a string formed by concatenating the arguments. The returned value is initialized to string1. Prefix, string pairs are added to the end of the value as long as the resulting string length is less than max. String1 is returned even if string1 is longer than max. You can pass as many prefix, string pairs as you wish.

strcmp(x, y, lt, eq, gt) – does a case-insensitive comparison x and y as strings. Returns lt if x < y. Returns eq if x == y. Otherwise returns gt.

strlen(a) – Returns the length of the string passed as the argument.

substr(str, start, end) – returns the start‘th through the end‘th characters of str. The first character in str is the zero’th character. If end is negative, then it indicates that many characters counting from the right. If end is zero, then it indicates the last character. For example, substr('12345', 1, 0) returns '2345', and substr('12345', 1, -1) returns '234'.

subtract(x, y) – returns x - y. Throws an exception if either x or y are not numbers.

today() – return a date string for today. This value is designed for use in format_date or days_between, but can be manipulated like any other string. The date is in ISO format.

template(x) – evaluates x as a template. The evaluation is done in its own context, meaning that variables are not shared between the caller and the template evaluation. Because the { and } characters are special, you must use [[ for the { character and ]] for the ‘}’ character; they are converted automatically. For example, template('[[title_sort]]') will evaluate the template ``{title_sort} and return its value. Note also that prefixes and suffixes (the |prefix|suffix syntax) cannot be used in the argument to this function when using template program mode.

Function classification

Reference for all built-in template language functions

Arithmetic

- add(x, y)
- divide(x, y)
- multiply(x, y)
- subtract(x, y)

Boolean

- and(value, value, ...)

- not(value)

- or(value, value, ...)

Date functions

- days_between(date1, date2)

- today()

Formatting values

- finish_formatting(val, fmt, prefix, suffix)
- format_date(val, format_string)
- format_number(v, template)
- human_readable(v)

Get values from metadata

- approximate_formats()
- booksize()
- current_library_name()
- current_library_path()
- field(name)
- formats_modtimes(date_format)
- formats_paths()
- formats_sizes()
- has_cover()
- language_codes(lang_strings)
- language_strings(lang_codes, localize)
- ondevice()
- raw_field(name)
- series_sort()

If-then-else

- contains(val, pattern, text if match, text if not match)

- ifempty(val, text if empty)

- test(val, text if not empty, text if empty)

Iterating over values

- first_non_empty(value, value, ...)

- lookup(val, pattern, field, pattern, field, ..., else_field)

- switch(val, pattern, value, pattern, value, ..., else_value)

List lookup

- identifier_in_list(val, id, found_val, not_found_val)

- in_list(val, separator, pattern, found_val, not_found_val)

- list_item(val, index, separator)

- select(val, key)

- str_in_list(val, separator, string, found_val, not_found_val)

List manipulation

- count(val, separator)

- list_difference(list1, list2, separator)

- list_equals(list1, sep1, list2, sep2, yes_val, no_val)

- list_intersection(list1, list2, separator)

- list_re(src_list, separator, search_re, opt_replace)

- list_sort(list, direction, separator)
- list_union(list1, list2, separator)
- subitems(val, start_index, end_index)

- sublist(val, start_index, end_index, separator)

Other

- assign(id, val)

- print(a, b, ...)

Recursion

- eval(template)

- template(x)

Relational

- cmp(x, y, lt, eq, gt)

- strcmp(x, y, lt, eq, gt)

String case changes

- capitalize(val)
- lowercase(val)
- titlecase(val)
- uppercase(val)

String manipulation

- re(val, pattern, replacement)

- shorten(val, left chars, middle text, right chars)

- strcat(a, b, ...)

- strcat_max(max, string1, prefix2, string2, ...)

- strlen(a)
- substr(str, start, end)
- swap_around_comma(val)

API of the Metadata objects

Using general program mode

For more complicated template programs, it is sometimes easier to avoid template syntax (all the { and } characters), instead writing a more classical-looking program. You can do this in calibre by beginning the template with program:. In this case, no template processing is done. The special variable $ is not set. It is up to your program to produce the correct results.

One advantage of program: mode is that the brackets are no longer special. For example, it is not necessary to use [[ and ]] when using the template() function. Another advantage is that program mode templates are compiled to Python and can run much faster than templates in the other two modes. Speed improvement depends on the complexity of the templates; the more complicated the template the more the improvement. Compilation is turned off or on using the tweak compile_gpm_templates (Compile General Program Mode templates to Python). The main reason to turn off compilation is if a compiled template does not work, in which case please file a bug report.

The following example is a program: mode implementation of a recipe on the MobileRead forum: “Put series into the title, using either initials or a shortened form. Strip leading articles from the series name (any).” For example, for the book The Two Towers in the Lord of the Rings series, the recipe gives LotR [02] The Two Towers. Using standard templates, the recipe requires three custom columns and a plugboard, as explained in the following:

The solution requires creating three composite columns. The first column is used to remove the leading articles. The second is used to compute the ‘shorten’ form. The third is to compute the ‘initials’ form. Once you have these columns, the plugboard selects between them. You can hide any or all of the three columns on the library view.

First column:
Name: #stripped_series.
Template: {series:re(^(A|The|An)s+,)||}

Second column (the shortened form):
Name: #shortened.
Template: {#stripped_series:shorten(4,-,4)}

Third column (the initials form):
Name: #initials.
Template: {#stripped_series:re(([^s])[^s]+(s|$),1)}

Plugboard expression:
Template:{#stripped_series:lookup(.s,#initials,.,#shortened,series)}{series_index:0>2.0f| [|] }{title}
Destination field: title

This set of fields and plugboard produces:
Series: The Lord of the Rings
Series index: 2
Title: The Two Towers
Output: LotR [02] The Two Towers

Series: Dahak
Series index: 1
Title: Mutineers Moon
Output: Dahak [01] Mutineers Moon

Series: Berserkers
Series Index: 4
Title: Berserker Throne
Output: Bers-kers [04] Berserker Throne

Series: Meg Langslow Mysteries
Series Index: 3
Title: Revenge of the Wrought-Iron Flamingos
Output: MLM [03] Revenge of the Wrought-Iron Flamingos

The following program produces the same results as the original recipe, using only one custom column to hold the results of a program that computes the special title value:

Custom column:
Name: #special_title
Template: (the following with all leading spaces removed)
program:
# compute the equivalent of the composite fields and store them in local variables
stripped = re(field('series'), '^(A|The|An)\s+', '');
shortened = shorten(stripped, 4, '-' ,4);
initials = re(stripped, '[^\w]*(\w?)[^\s]+(\s|$)', '\1');

# Format the series index. Ends up as empty if there is no series index.
# Note that leading and trailing spaces will be removed by the formatter,
# so we cannot add them here. We will do that in the strcat below.
# Also note that because we are in 'program' mode, we can freely use
# curly brackets in strings, something we cannot do in template mode.
s_index = template('{series_index:0>2.0f}');

# print(stripped, shortened, initials, s_index);

# Now concatenate all the bits together. The switch picks between
# initials and shortened, depending on whether there is a space
# in stripped. We then add the brackets around s_index if it is
# not empty. Finally, add the title. As this is the last function in
# the program, its value will be returned.
strcat(
switch( stripped,
'.\s', initials,
'.', shortened,
field('series')),
test(s_index, strcat(' [', s_index, '] '), ''),
field('title'));

Plugboard expression:
Template:{#special_title}
Destination field: title

It would be possible to do the above with no custom columns by putting the program into the template box of the plugboard. However, to do so, all comments must be removed because the plugboard text box does not support multi-line editing. It is debatable whether the gain of not having the custom column is worth the vast increase in difficulty caused by the program being one giant line.

User-defined Template Functions

You can add your own functions to the template processor. Such functions are written in python, and can be used in any of the three template programming modes. The functions are added by going to Preferences -> Advanced -> Template Functions. Instructions are shown in that dialog.

Special notes for save/send templates

Special processing is applied when a template is used in a save to disk or send to device template. The values of the fields are cleaned, replacing characters that are special to file systems with underscores, including slashes. This means that field text cannot be used to create folders. However, slashes are not changed in prefix or suffix strings, so slashes in these strings will cause folders to be created. Because of this, you can create variable-depth folder structure.

For example, assume we want the folder structure series/series_index - title, with the caveat that if series does not exist, then the title should be in the top folder. The template to do this is:

{series:||/}{series_index:|| - }{title}

The slash and the hyphen appear only if series is not empty.

The lookup function lets us do even fancier processing. For example, assume that if a book has a series, then we want the folder structure series/series index - title.fmt. If the book does not have a series, then we want the folder structure genre/author_sort/title.fmt. If the book has no genre, we want to use ‘Unknown’. We want two completely different paths, depending on the value of series.

To accomplish this, we:

1. Create a composite field (call it AA) containing {series}/{series_index} - {title'}. If the series is not empty, then this template will produce series/series_index - title.

2. Create a composite field (call it BB) containing {#genre:ifempty(Unknown)}/{author_sort}/{title}. This template produces genre/author_sort/title, where an empty genre is replaced wuth Unknown.

3. Set the save template to {series:lookup(.,AA,BB)}. This template chooses composite field AA if series is not empty, and composite field BB if series is empty. We therefore have two completely different save paths, depending on whether or not series is empty.

Templates and Plugboards

Plugboards are used for changing the metadata written into books during send-to-device and save-to-disk operations. A plugboard permits you to specify a template to provide the data to write into the book’s metadata. You can use plugboards to modify the following fields: authors, author_sort, language, publisher, tags, title, title_sort. This feature helps people who want to use different metadata in books on devices to solve sorting or display issues.

When you create a plugboard, you specify the format and device for which the plugboard is to be used. A special device is provided, save_to_disk, that is used when saving formats (as opposed to sending them to a device). Once you have chosen the format and device, you choose the metadata fields to change, providing templates to supply the new values. These templates are connected to their destination fields, hence the name plugboards. You can, of course, use composite columns in these templates.

When a plugboard might apply (content server, save to disk, or send to device), calibre searches the defined plugboards to choose the correct one for the given format and device. For example, to find the appropriate plugboard for an EPUB book being sent to an ANDROID device, calibre searches the plugboards using the following search order:

- a plugboard with an exact match on format and device, e.g., EPUB and ANDROID

- a plugboard with an exact match on format and the special any device choice, e.g., EPUB and any device

- a plugboard with the special any format choice and an exact match on device, e.g., any format and ANDROID

- a plugboard with any format and any device

The tags and authors fields have special treatment, because both of these fields can hold more than one item. A book can have many tags and many authors. When you specify that one of these two fields is to be changed, the template’s result is examined to see if more than one item is there. For tags, the result is cut apart wherever calibre finds a comma. For example, if the template produces the value Thriller, Horror, then the result will be two tags, Thriller and Horror. There is no way to put a comma in the middle of a tag.

The same thing happens for authors, but using a different character for the cut, a & (ampersand) instead of a comma. For example, if the template produces the value Blogs, Joe&Posts, Susan, then the book will end up with two authors, Blogs, Joe and Posts, Susan. If the template produces the value Blogs, Joe;Posts, Susan, then the book will have one author with a rather strange name.

Plugboards affect the metadata written into the book when it is saved to disk or written to the device. Plugboards do not affect the metadata used by save to disk and send to device to create the file names. Instead, file names are constructed using the templates entered on the appropriate preferences window.

Helpful Tips

You might find the following tips useful.

- Create a custom composite column to test templates. Once you have the column, you can change its template simply by double-clicking on the column. Hide the column when you are not testing.

- Templates can use other templates by referencing a composite custom column.

- In a plugboard, you can set a field to empty (or whatever is equivalent to empty) by using the special template {}. This template will always evaluate to an empty string.

- The technique described above to show numbers even if they have a zero value works with the standard field series_index.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »
- The calibre template language »

Reference for all built-in template language functions

Here, we document all the built-in functions available in the calibre template language. Every function is implemented as a class in python and you can click the source links to see the source code, in case the documentation is insufficient. The functions are arranged in logical groups by type.

Arithmetic

- add(x, y)
- divide(x, y)
- multiply(x, y)
- subtract(x, y)

Boolean

- and(value, value, ...)

- not(value)

- or(value, value, ...)

Date functions

- days_between(date1, date2)

- today()

Formatting values

- finish_formatting(val, fmt, prefix, suffix)
- format_date(val, format_string)
- format_number(v, template)
- human_readable(v)

Get values from metadata

- approximate_formats()
- booksize()
- current_library_name()
- current_library_path()
- field(name)
- formats_modtimes(date_format)
- formats_paths()
- formats_sizes()
- has_cover()
- language_codes(lang_strings)
- language_strings(lang_codes, localize)
- ondevice()
- raw_field(name)
- series_sort()

If-then-else

- contains(val, pattern, text if match, text if not match)

- ifempty(val, text if empty)

- test(val, text if not empty, text if empty)

Iterating over values

- first_non_empty(value, value, ...)

- lookup(val, pattern, field, pattern, field, ..., else_field)

- switch(val, pattern, value, pattern, value, ..., else_value)

List lookup

- identifier_in_list(val, id, found_val, not_found_val)

- in_list(val, separator, pattern, found_val, not_found_val)

- list_item(val, index, separator)

- select(val, key)

- str_in_list(val, separator, string, found_val, not_found_val)

List manipulation

- count(val, separator)

- list_difference(list1, list2, separator)

- list_equals(list1, sep1, list2, sep2, yes_val, no_val)

- list_intersection(list1, list2, separator)

- list_re(src_list, separator, search_re, opt_replace)

- list_sort(list, direction, separator)
- list_union(list1, list2, separator)
- subitems(val, start_index, end_index)

- sublist(val, start_index, end_index, separator)

Other

- assign(id, val)

- print(a, b, ...)

Recursion

- eval(template)

- template(x)

Relational

- cmp(x, y, lt, eq, gt)

- strcmp(x, y, lt, eq, gt)

String case changes

- capitalize(val)
- lowercase(val)
- titlecase(val)
- uppercase(val)

String manipulation

- re(val, pattern, replacement)

- shorten(val, left chars, middle text, right chars)

- strcat(a, b, ...)

- strcat_max(max, string1, prefix2, string2, ...)

- strlen(a)
- substr(str, start, end)
- swap_around_comma(val)

API of the Metadata objects

Arithmetic

add(x, y)

class calibre.utils.formatter_functions.BuiltinAdd[source]

add(x, y) – returns x + y. Throws an exception if either x or y are not numbers.

divide(x, y)

class calibre.utils.formatter_functions.BuiltinDivide[source]

divide(x, y) – returns x / y. Throws an exception if either x or y are not numbers.

multiply(x, y)

class calibre.utils.formatter_functions.BuiltinMultiply[source]

multiply(x, y) – returns x * y. Throws an exception if either x or y are not numbers.

subtract(x, y)

class calibre.utils.formatter_functions.BuiltinSubtract[source]

subtract(x, y) – returns x - y. Throws an exception if either x or y are not numbers.

Boolean

and(value, value, ...)

class calibre.utils.formatter_functions.BuiltinAnd[source]

and(value, value, ...) – returns the string “1” if all values are not empty, otherwise returns the empty string. This function works well with test or first_non_empty. You can have as many values as you want.

not(value)

class calibre.utils.formatter_functions.BuiltinNot[source]

not(value) – returns the string “1” if the value is empty, otherwise returns the empty string. This function works well with test or first_non_empty. You can have as many values as you want.

or(value, value, ...)

class calibre.utils.formatter_functions.BuiltinOr[source]

or(value, value, ...) – returns the string “1” if any value is not empty, otherwise returns the empty string. This function works well with test or first_non_empty. You can have as many values as you want.

Date functions

days_between(date1, date2)

class calibre.utils.formatter_functions.BuiltinDaysBetween[source]

days_between(date1, date2) – return the number of days between date1 and date2. The number is positive if date1 is greater than date2, otherwise negative. If either date1 or date2 are not dates, the function returns the empty string.

today()

class calibre.utils.formatter_functions.BuiltinToday[source]

today() – return a date string for today. This value is designed for use in format_date or days_between, but can be manipulated like any other string. The date is in ISO format.

Formatting values

finish_formatting(val, fmt, prefix, suffix)

class calibre.utils.formatter_functions.BuiltinFinishFormatting[source]

finish_formatting(val, fmt, prefix, suffix) – apply the format, prefix, and suffix to a value in the same way as done in a template like {series_index:05.2f| - |- }. For example, the following program produces the same output as the above template: program: finish_formatting(field(“series_index”), “05.2f”, ” - ”, ” - ”)

format_date(val, format_string)

class calibre.utils.formatter_functions.BuiltinFormatDate[source]

format_date(val, format_string) – format the value, which must be a date, using the format_string, returning a string. The formatting codes are: d : the day as number without a leading zero (1 to 31) dd : the day as number with a leading zero (01 to 31) ddd : the abbreviated localized day name (e.g. “Mon” to “Sun”). dddd : the long localized day name (e.g. “Monday” to “Sunday”). M : the month as number without a leading zero (1 to 12). MM : the month as number with a leading zero (01 to 12) MMM : the abbreviated localized month name (e.g. “Jan” to “Dec”). MMMM : the long localized month name (e.g. “January” to “December”). yy : the year as two digit number (00 to 99). yyyy : the year as four digit number. h : the hours without a leading 0 (0 to 11 or 0 to 23, depending on am/pm) hh : the hours with a leading 0 (00 to 11 or 00 to 23, depending on am/pm) m : the minutes without a leading 0 (0 to 59) mm : the minutes with a leading 0 (00 to 59) s : the seconds without a leading 0 (0 to 59) ss : the seconds with a leading 0 (00 to 59) ap : use a 12-hour clock instead of a 24-hour clock, with “ap” replaced by the localized string for am or pm AP : use a 12-hour clock instead of a 24-hour clock, with “AP” replaced by the localized string for AM or PM iso : the date with time and timezone. Must be the only format present

format_number(v, template)

class calibre.utils.formatter_functions.BuiltinFormatNumber[source]

format_number(v, template) – format the number v using a python formatting template such as “{0:5.2f}” or “{0:,d}” or “${0:5,.2f}”. The field_name part of the template must be a 0 (zero) (the “{0:” in the above examples). See the template language and python documentation for more examples. Returns the empty string if formatting fails.

human_readable(v)

class calibre.utils.formatter_functions.BuiltinHumanReadable[source]

human_readable(v) – return a string representing the number v in KB, MB, GB, etc.

Get values from metadata

approximate_formats()

class calibre.utils.formatter_functions.BuiltinApproximateFormats[source]

approximate_formats() – return a comma-separated list of formats that at one point were associated with the book. There is no guarantee that this list is correct, although it probably is. This function can be called in template program mode using the template “{:’approximate_formats()’}”. Note that format names are always uppercase, as in EPUB.

booksize()

class calibre.utils.formatter_functions.BuiltinBooksize[source]

booksize() – return value of the size field

current_library_name()

class calibre.utils.formatter_functions.BuiltinCurrentLibraryName[source]

current_library_name() – return the last name on the path to the current calibre library. This function can be called in template program mode using the template “{:’current_library_name()’}”.

current_library_path()

class calibre.utils.formatter_functions.BuiltinCurrentLibraryPath[source]

current_library_path() – return the path to the current calibre library. This function can be called in template program mode using the template “{:’current_library_path()’}”.

field(name)

class calibre.utils.formatter_functions.BuiltinField[source]

field(name) – returns the metadata field named by name

formats_modtimes(date_format)

class calibre.utils.formatter_functions.BuiltinFormatsModtimes[source]

formats_modtimes(date_format) – return a comma-separated list of colon_separated items representing modification times for the formats of a book. The date_format parameter specifies how the date is to be formatted. See the date_format function for details. You can use the select function to get the mod time for a specific format. Note that format names are always uppercase, as in EPUB.

formats_paths()

class calibre.utils.formatter_functions.BuiltinFormatsPaths[source]

formats_paths() – return a comma-separated list of colon_separated items representing full path to the formats of a book. You can use the select function to get the path for a specific format. Note that format names are always uppercase, as in EPUB.

formats_sizes()

class calibre.utils.formatter_functions.BuiltinFormatsSizes[source]

formats_sizes() – return a comma-separated list of colon_separated items representing sizes in bytes of the formats of a book. You can use the select function to get the size for a specific format. Note that format names are always uppercase, as in EPUB.

has_cover()

class calibre.utils.formatter_functions.BuiltinHasCover[source]

has_cover() – return Yes if the book has a cover, otherwise return the empty string

language_codes(lang_strings)

class calibre.utils.formatter_functions.BuiltinLanguageCodes[source]

language_codes(lang_strings) – return the language codes for the strings passed in lang_strings. The strings must be in the language of the current locale. Lang_strings is a comma-separated list.

language_strings(lang_codes, localize)

class calibre.utils.formatter_functions.BuiltinLanguageStrings[source]

language_strings(lang_codes, localize) – return the strings for the language codes passed in lang_codes. If localize is zero, return the strings in English. If localize is not zero, return the strings in the language of the current locale. Lang_codes is a comma-separated list.

ondevice()

class calibre.utils.formatter_functions.BuiltinOndevice[source]

ondevice() – return Yes if ondevice is set, otherwise return the empty string

raw_field(name)

class calibre.utils.formatter_functions.BuiltinRawField[source]

raw_field(name) – returns the metadata field named by name without applying any formatting.

series_sort()

class calibre.utils.formatter_functions.BuiltinSeriesSort[source]

series_sort() – return the series sort value

If-then-else

contains(val, pattern, text if match, text if not match)

class calibre.utils.formatter_functions.BuiltinContains[source]

contains(val, pattern, text if match, text if not match) – checks if field contains matches for the regular expression pattern. Returns text if match if matches are found, otherwise it returns text if no match

ifempty(val, text if empty)

class calibre.utils.formatter_functions.BuiltinIfempty[source]

ifempty(val, text if empty) – return val if val is not empty, otherwise return text if empty

test(val, text if not empty, text if empty)

class calibre.utils.formatter_functions.BuiltinTest[source]

test(val, text if not empty, text if empty) – return text if not empty if the field is not empty, otherwise return text if empty

Iterating over values

first_non_empty(value, value, ...)

class calibre.utils.formatter_functions.BuiltinFirstNonEmpty[source]

first_non_empty(value, value, ...) – returns the first value that is not empty. If all values are empty, then the empty value is returned. You can have as many values as you want.

lookup(val, pattern, field, pattern, field, ..., else_field)

class calibre.utils.formatter_functions.BuiltinLookup[source]

lookup(val, pattern, field, pattern, field, ..., else_field) – like switch, except the arguments are field (metadata) names, not text. The value of the appropriate field will be fetched and used. Note that because composite columns are fields, you can use this function in one composite field to use the value of some other composite field. This is extremely useful when constructing variable save paths

switch(val, pattern, value, pattern, value, ..., else_value)

class calibre.utils.formatter_functions.BuiltinSwitch[source]

switch(val, pattern, value, pattern, value, ..., else_value) – for each pattern, value pair, checks if the field matches the regular expression pattern and if so, returns that value. If no pattern matches, then else_value is returned. You can have as many pattern, value pairs as you want

List lookup

identifier_in_list(val, id, found_val, not_found_val)

class calibre.utils.formatter_functions.BuiltinIdentifierInList[source]

identifier_in_list(val, id, found_val, not_found_val) – treat val as a list of identifiers separated by commas, comparing the string against each value in the list. An identifier has the format “identifier:value”. The id parameter should be either “id” or “id:regexp”. The first case matches if there is any identifier with that id. The second case matches if the regexp matches the identifier’s value. If there is a match, return found_val, otherwise return not_found_val.

in_list(val, separator, pattern, found_val, not_found_val)

class calibre.utils.formatter_functions.BuiltinInList[source]

in_list(val, separator, pattern, found_val, not_found_val) – treat val as a list of items separated by separator, comparing the pattern against each value in the list. If the pattern matches a value, return found_val, otherwise return not_found_val.

list_item(val, index, separator)

class calibre.utils.formatter_functions.BuiltinListitem[source]

list_item(val, index, separator) – interpret the value as a list of items separated by separator, returning the index`th item. The first item is number zero. The last item can be returned using `list_item(-1,separator). If the item is not in the list, then the empty value is returned. The separator has the same meaning as in the count function.

select(val, key)

class calibre.utils.formatter_functions.BuiltinSelect[source]

select(val, key) – interpret the value as a comma-separated list of items, with the items being “id:value”. Find the pair with the id equal to key, and return the corresponding value.

str_in_list(val, separator, string, found_val, not_found_val)

class calibre.utils.formatter_functions.BuiltinStrInList[source]

str_in_list(val, separator, string, found_val, not_found_val) – treat val as a list of items separated by separator, comparing the string against each value in the list. If the string matches a value, return found_val, otherwise return not_found_val. If the string contains separators, then it is also treated as a list and each value is checked.

List manipulation

count(val, separator)

class calibre.utils.formatter_functions.BuiltinCount[source]

count(val, separator) – interprets the value as a list of items separated by separator, returning the number of items in the list. Most lists use a comma as the separator, but authors uses an ampersand. Examples: {tags:count(,)}, {authors:count(&)}

list_difference(list1, list2, separator)

class calibre.utils.formatter_functions.BuiltinListDifference[source]

list_difference(list1, list2, separator) – return a list made by removing from list1 any item found in list2, using a case-insensitive compare. The items in list1 and list2 are separated by separator, as are the items in the returned list.

list_equals(list1, sep1, list2, sep2, yes_val, no_val)

class calibre.utils.formatter_functions.BuiltinListEquals[source]

list_equals(list1, sep1, list2, sep2, yes_val, no_val) – return yes_val if list1 and list2 contain the same items, otherwise return no_val. The items are determined by splitting each list using the appropriate separator character (sep1 or sep2). The order of items in the lists is not relevant. The compare is case insensitive.

list_intersection(list1, list2, separator)

class calibre.utils.formatter_functions.BuiltinListIntersection[source]

list_intersection(list1, list2, separator) – return a list made by removing from list1 any item not found in list2, using a case-insensitive compare. The items in list1 and list2 are separated by separator, as are the items in the returned list.

list_re(src_list, separator, search_re, opt_replace)

class calibre.utils.formatter_functions.BuiltinListRe[source]

list_re(src_list, separator, search_re, opt_replace) – Construct a list by first separating src_list into items using the separator character. For each item in the list, check if it matches search_re. If it does, then add it to the list to be returned. If opt_replace is not the empty string, then apply the replacement before adding the item to the returned list.

list_sort(list, direction, separator)

class calibre.utils.formatter_functions.BuiltinListSort[source]

list_sort(list, direction, separator) – return list sorted using a case-insensitive sort. If direction is zero, the list is sorted ascending, otherwise descending. The list items are separated by separator, as are the items in the returned list.

list_union(list1, list2, separator)

class calibre.utils.formatter_functions.BuiltinListUnion[source]

list_union(list1, list2, separator) – return a list made by merging the items in list1 and list2, removing duplicate items using a case-insensitive compare. If items differ in case, the one in list1 is used. The items in list1 and list2 are separated by separator, as are the items in the returned list.

subitems(val, start_index, end_index)

class calibre.utils.formatter_functions.BuiltinSubitems[source]

subitems(val, start_index, end_index) – This function is used to break apart lists of items such as genres. It interprets the value as a comma-separated list of items, where each item is a period-separated list. Returns a new list made by first finding all the period-separated items, then for each such item extracting the start_index to the end_index components, then combining the results back together. The first component in a period-separated list has an index of zero. If an index is negative, then it counts from the end of the list. As a special case, an end_index of zero is assumed to be the length of the list. Example using basic template mode and assuming a #genre value of “A.B.C”: {#genre:subitems(0,1)} returns “A”. {#genre:subitems(0,2)} returns “A.B”. {#genre:subitems(1,0)} returns “B.C”. Assuming a #genre value of “A.B.C, D.E.F”, {#genre:subitems(0,1)} returns “A, D”. {#genre:subitems(0,2)} returns “A.B, D.E”

sublist(val, start_index, end_index, separator)

class calibre.utils.formatter_functions.BuiltinSublist[source]

sublist(val, start_index, end_index, separator) – interpret the value as a list of items separated by separator, returning a new list made from the start_index to the end_index item. The first item is number zero. If an index is negative, then it counts from the end of the list. As a special case, an end_index of zero is assumed to be the length of the list. Examples using basic template mode and assuming that the tags column (which is comma-separated) contains “A, B, C”: {tags:sublist(0,1,,)} returns “A”. {tags:sublist(-1,0,,)} returns “C”. {tags:sublist(0,-1,,)} returns “A, B”.

Other

assign(id, val)

class calibre.utils.formatter_functions.BuiltinAssign[source]

assign(id, val) – assigns val to id, then returns val. id must be an identifier, not an expression

print(a, b, ...)

class calibre.utils.formatter_functions.BuiltinPrint[source]

print(a, b, ...) – prints the arguments to standard output. Unless you start calibre from the command line (calibre-debug -g), the output will go to a black hole.

Recursion

eval(template)

class calibre.utils.formatter_functions.BuiltinEval[source]

eval(template) – evaluates the template, passing the local variables (those ‘assign’ed to) instead of the book metadata. This permits using the template processor to construct complex results from local variables. Because the { and } characters are special, you must use [[ for the { character and ]] for the } character; they are converted automatically. Note also that prefixes and suffixes (the |prefix|suffix syntax) cannot be used in the argument to this function when using template program mode.

template(x)

class calibre.utils.formatter_functions.BuiltinTemplate[source]

template(x) – evaluates x as a template. The evaluation is done in its own context, meaning that variables are not shared between the caller and the template evaluation. Because the { and } characters are special, you must use [[ for the { character and ]] for the } character; they are converted automatically. For example, template(‘[[title_sort]]’) will evaluate the template {title_sort} and return its value. Note also that prefixes and suffixes (the |prefix|suffix syntax) cannot be used in the argument to this function when using template program mode.

Relational

cmp(x, y, lt, eq, gt)

class calibre.utils.formatter_functions.BuiltinCmp[source]

cmp(x, y, lt, eq, gt) – compares x and y after converting both to numbers. Returns lt if x < y. Returns eq if x == y. Otherwise returns gt.

strcmp(x, y, lt, eq, gt)

class calibre.utils.formatter_functions.BuiltinStrcmp[source]

strcmp(x, y, lt, eq, gt) – does a case-insensitive comparison of x and y as strings. Returns lt if x < y. Returns eq if x == y. Otherwise returns gt.

String case changes

capitalize(val)

class calibre.utils.formatter_functions.BuiltinCapitalize[source]

capitalize(val) – return value of the field capitalized

lowercase(val)

class calibre.utils.formatter_functions.BuiltinLowercase[source]

lowercase(val) – return value of the field in lower case

titlecase(val)

class calibre.utils.formatter_functions.BuiltinTitlecase[source]

titlecase(val) – return value of the field in title case

uppercase(val)

class calibre.utils.formatter_functions.BuiltinUppercase[source]

uppercase(val) – return value of the field in upper case

String manipulation

re(val, pattern, replacement)

class calibre.utils.formatter_functions.BuiltinRe[source]

re(val, pattern, replacement) – return the field after applying the regular expression. All instances of pattern are replaced with replacement. As in all of calibre, these are python-compatible regular expressions

shorten(val, left chars, middle text, right chars)

class calibre.utils.formatter_functions.BuiltinShorten[source]

shorten(val, left chars, middle text, right chars) – Return a shortened version of the field, consisting of left chars characters from the beginning of the field, followed by middle text, followed by right chars characters from the end of the string. Left chars and right chars must be integers. For example, assume the title of the book is Ancient English Laws in the Times of Ivanhoe, and you want it to fit in a space of at most 15 characters. If you use {title:shorten(9,-,5)}, the result will be Ancient E-nhoe. If the field’s length is less than left chars + right chars + the length of middle text, then the field will be used intact. For example, the title The Dome would not be changed.

strcat(a, b, ...)

class calibre.utils.formatter_functions.BuiltinStrcat[source]

strcat(a, b, ...) – can take any number of arguments. Returns a string formed by concatenating all the arguments

strcat_max(max, string1, prefix2, string2, ...)

class calibre.utils.formatter_functions.BuiltinStrcatMax[source]

strcat_max(max, string1, prefix2, string2, ...) – Returns a string formed by concatenating the arguments. The returned value is initialized to string1. Prefix, string pairs are added to the end of the value as long as the resulting string length is less than max. String1 is returned even if string1 is longer than max. You can pass as many prefix, string pairs as you wish.

strlen(a)

class calibre.utils.formatter_functions.BuiltinStrlen[source]

strlen(a) – Returns the length of the string passed as the argument

substr(str, start, end)

class calibre.utils.formatter_functions.BuiltinSubstr[source]

substr(str, start, end) – returns the start’th through the end’th characters of str. The first character in str is the zero’th character. If end is negative, then it indicates that many characters counting from the right. If end is zero, then it indicates the last character. For example, substr(‘12345’, 1, 0) returns ‘2345’, and substr(‘12345’, 1, -1) returns ‘234’.

swap_around_comma(val)

class calibre.utils.formatter_functions.BuiltinSwapAroundComma[source]

swap_around_comma(val) – given a value of the form “B, A”, return “A B”. This is most useful for converting names in LN, FN format to FN LN. If there is no comma, the function returns val unchanged

API of the Metadata objects

The python implementation of the template functions is passed in a Metadata object. Knowing it’s API is useful if you want to define your own template functions.

class calibre.ebooks.metadata.book.base.Metadata(title, authors=(u'Unknown', ), other=None, template_cache=None)[source]

A class representing all the metadata for a book. The various standard metadata fields are available as attributes of this object. You can also stick arbitrary attributes onto this object.

Metadata from custom columns should be accessed via the get() method, passing in the lookup name for the column, for example: “#mytags”.

Use the is_null() method to test if a field is null.

This object also has functions to format fields into strings.

The list of standard metadata fields grows with time is in STANDARD_METADATA_FIELDS.

Please keep the method based API of this class to a minimum. Every method becomes a reserved field name.

is_null(field)[source]

Return True if the value of field is null in this object.
‘null’ means it is unknown or evaluates to False. So a title of
_(‘Unknown’) is null or a language of ‘und’ is null.

Be careful with numeric fields since this will return True for zero as well as None.

Also returns True if the field does not exist.

get_identifiers()[source]

Return a copy of the identifiers dictionary.
The dict is small, and the penalty for using a reference where a copy is needed is large. Also, we don’t want any manipulations of the returned dict to show up in the book.

set_identifiers(identifiers)[source]

Set all identifiers. Note that if you previously set ISBN, calling this method will delete it.

set_identifier(typ, val)[source]

If val is empty, deletes identifier of type typ

standard_field_keys()[source]

return a list of all possible keys, even if this book doesn’t have them

custom_field_keys()[source]

return a list of the custom fields in this book

all_field_keys()[source]

All field keys known by this instance, even if their value is None

metadata_for_field(key)[source]

return metadata describing a standard or custom field.

all_non_none_fields()[source]

Return a dictionary containing all non-None metadata fields, including the custom ones.

get_standard_metadata(field, make_copy)[source]

return field metadata from the field if it is there. Otherwise return
None. field is the key name, not the label. Return a copy if requested, just in case the user wants to change values in the dict.

get_all_standard_metadata(make_copy)[source]

return a dict containing all the standard field metadata associated with the book.

get_all_user_metadata(make_copy)[source]

return a dict containing all the custom field metadata associated with the book.

get_user_metadata(field, make_copy)[source]

return field metadata from the object if it is there. Otherwise return
None. field is the key name, not the label. Return a copy if requested, just in case the user wants to change values in the dict.

set_all_user_metadata(metadata)[source]

store custom field metadata into the object. Field is the key name not the label

set_user_metadata(field, metadata)[source]

store custom field metadata for one column into the object. Field is the key name not the label

template_to_attribute(other, ops)[source]

Takes a list [(src,dest), (src,dest)], evaluates the template in the context of other, then copies the result to self[dest]. This is on a best-efforts basis. Some assignments can make no sense.

smart_update(other, replace_metadata=False)[source]

Merge the information in other into self. In case of conflicts, the information in other takes precedence, unless the information in other is NULL.

format_field(key, series_with_index=True)[source]

Returns the tuple (display_name, formatted_value)

to_html()[source]

A HTML representation of this object.

calibre.ebooks.metadata.book.base.STANDARD_METADATA_FIELDS

The set of standard metadata fields.

'''
All fields must have a NULL value represented as None for simple types, an empty list/dictionary for complex types and (None, None) for cover_data
'''

SOCIAL_METADATA_FIELDS = frozenset([
'tags', # Ordered list
'rating', # A floating point number between 0 and 10
'comments', # A simple HTML enabled string
'series', # A simple string
'series_index', # A floating point number
# Of the form { scheme1:value1, scheme2:value2}
# For example: {'isbn':'123456789', 'doi':'xxxx', ... }
'identifiers',
])

'''
The list of names that convert to identifiers when in get and set.
'''

TOP_LEVEL_IDENTIFIERS = frozenset([
'isbn',
])

PUBLICATION_METADATA_FIELDS = frozenset([
'title', # title must never be None. Should be _('Unknown')
# Pseudo field that can be set, but if not set is auto generated
# from title and languages
'title_sort',
'authors', # Ordered list. Must never be None, can be [_('Unknown')]
'author_sort_map', # Map of sort strings for each author
# Pseudo field that can be set, but if not set is auto generated
# from authors and languages
'author_sort',
'book_producer',
'timestamp', # Dates and times must be timezone aware
'pubdate',
'last_modified',
'rights',
# So far only known publication type is periodical:calibre
# If None, means book
'publication_type',
'uuid', # A UUID usually of type 4
'languages', # ordered list of languages in this publication
'publisher', # Simple string, no special semantics
# Absolute path to image file encoded in filesystem_encoding
'cover',
# Of the form (format, data) where format is, for e.g. 'jpeg', 'png', 'gif'...
'cover_data',
# Either thumbnail data, or an object with the attribute
# image_path which is the path to an image file, encoded
# in filesystem_encoding
'thumbnail',
])

BOOK_STRUCTURE_FIELDS = frozenset([
# These are used by code, Null values are None.
'toc', 'spine', 'guide', 'manifest',
])

USER_METADATA_FIELDS = frozenset([
# A dict of dicts similar to field_metadata. Each field description dict
# also contains a value field with the key #value#.
'user_metadata',
])

DEVICE_METADATA_FIELDS = frozenset([
'device_collections', # Ordered list of strings
'lpath', # Unicode, / separated
'size', # In bytes
'mime', # Mimetype of the book file being represented

])

CALIBRE_METADATA_FIELDS = frozenset([
'application_id', # An application id, currently set to the db_id.
'db_id', # the calibre primary key of the item.
'formats', # list of formats (extensions) for this book
# a dict of user category names, where the value is a list of item names
# from the book that are in that category
'user_categories',
# a dict of author to an associated hyperlink
'author_link_map',

]
)

ALL_METADATA_FIELDS = SOCIAL_METADATA_FIELDS.union(
PUBLICATION_METADATA_FIELDS).union(
BOOK_STRUCTURE_FIELDS).union(
USER_METADATA_FIELDS).union(
DEVICE_METADATA_FIELDS).union(
CALIBRE_METADATA_FIELDS)

# All fields except custom fields
STANDARD_METADATA_FIELDS = SOCIAL_METADATA_FIELDS.union(
PUBLICATION_METADATA_FIELDS).union(
BOOK_STRUCTURE_FIELDS).union(
DEVICE_METADATA_FIELDS).union(
CALIBRE_METADATA_FIELDS)

# Metadata fields that smart update must do special processing to copy.
SC_FIELDS_NOT_COPIED = frozenset(['title', 'title_sort', 'authors',
'author_sort', 'author_sort_map',
'cover_data', 'tags', 'languages',
'identifiers'])

# Metadata fields that smart update should copy only if the source is not None
SC_FIELDS_COPY_NOT_NULL = frozenset(['lpath', 'size', 'comments', 'thumbnail'])

# Metadata fields that smart update should copy without special handling
SC_COPYABLE_FIELDS = SOCIAL_METADATA_FIELDS.union(
PUBLICATION_METADATA_FIELDS).union(
BOOK_STRUCTURE_FIELDS).union(
DEVICE_METADATA_FIELDS).union(
CALIBRE_METADATA_FIELDS) - \ SC_FIELDS_NOT_COPIED.union(
SC_FIELDS_COPY_NOT_NULL)

SERIALIZABLE_FIELDS = SOCIAL_METADATA_FIELDS.union(
USER_METADATA_FIELDS).union(
PUBLICATION_METADATA_FIELDS).union(
CALIBRE_METADATA_FIELDS).union(
DEVICE_METADATA_FIELDS) - \ frozenset(['device_collections', 'formats',
'cover_data'])
# these are rebuilt when needed

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

All about using regular expressions in calibre

Regular expressions are features used in many places in calibre to perform sophisticated manipulation of ebook content and metadata. This tutorial is a gentle introduction to getting you started with using regular expressions in calibre.

Contents

- First, a word of warning and a word of courage

- Where in calibre can you use regular expressions?

- What on earth is a regular expression?

- Care to explain?

- That doesn’t sound too bad. What’s next?

- Hey, neat! This is starting to make sense!

- Well, these special characters are very neat and all, but what if I wanted to match a dot or a question mark?

- So, what are the most useful sets?

- But if I had a few varying strings I wanted to match, things get complicated?

- You missed...

- In the beginning, you said there was a way to make a regular expression case insensitive?

- I think I’m beginning to understand these regular expressions now... how do I use them in calibre?

- Conversions
- Adding books
- Bulk editing metadata

- Credits

First, a word of warning and a word of courage

This is, inevitably, going to be somewhat technical- after all, regular expressions are a technical tool for doing technical stuff. I’m going to have to use some jargon and concepts that may seem complicated or convoluted. I’m going to try to explain those concepts as clearly as I can, but really can’t do without using them at all. That being said, don’t be discouraged by any jargon, as I’ve tried to explain everything new. And while regular expressions themselves may seem like an arcane, black magic (or, to be more prosaic, a random string of mumbo-jumbo letters and signs), I promise that they are not all that complicated. Even those who understand regular expressions really well have trouble reading the more complex ones, but writing them isn’t as difficult- you construct the expression step by step. So, take a step and follow me into the rabbit hole.

Where in calibre can you use regular expressions?

There are a few places calibre uses regular expressions. There’s the Search & Replace in conversion options, metadata detection from filenames in the import settings and Search & Replace when editing the metadata of books in bulk.

What on earth is a regular expression?

A regular expression is a way to describe sets of strings. A single regular expression cat match a number of different strings. This is what makes regular expression so powerful – they are a concise way of describing a potentially large number of variations.

Note

I’m using string here in the sense it is used in programming languages: a string of one or more characters, characters including actual characters, numbers, punctuation and so-called whitespace (linebreaks, tabulators etc.). Please note that generally, uppercase and lowercase characters are not considered the same, thus “a” being a different character from “A” and so forth. In calibre, regular expressions are case insensitive in the search bar, but not in the conversion options. There’s a way to make every regular expression case insensitive, but we’ll discuss that later. It gets complicated because regular expressions allow for variations in the strings it matches, so one expression can match multiple strings, which is why people bother using them at all. More on that in a bit.

Care to explain?

Well, that’s why we’re here. First, this is the most important concept in regular expressions: A string by itself is a regular expression that matches itself. That is to say, if I wanted to match the string "Hello, World!" using a regular expression, the regular expression to use would be Hello, World!. And yes, it really is that simple. You’ll notice, though, that this only matches the exact string "Hello, World!", not e.g. "Hello, wOrld!" or "hello, world!" or any other such variation.

That doesn’t sound too bad. What’s next?

Next is the beginning of the really good stuff. Remember where I said that regular expressions can match multiple strings? This is were it gets a little more complicated. Say, as a somewhat more practical exercise, the ebook you wanted to convert had a nasty footer counting the pages, like “Page 5 of 423”. Obviously the page number would rise from 1 to 423, thus you’d have to match 423 different strings, right? Wrong, actually: regular expressions allow you to define sets of characters that are matched: To define a set, you put all the characters you want to be in the set into square brackets. So, for example, the set [abc] would match either the character “a”, “b” or “c”. Sets will always only match one of the characters in the set. They “understand” character ranges, that is, if you wanted to match all the lower case characters, you’d use the set [a-z] for lower- and uppercase characters you’d use [a-zA-Z] and so on. Got the idea? So, obviously, using the expression Page [0-9] of 423 you’d be able to match the first 9 pages, thus reducing the expressions needed to three: The second expression Page [0-9][0-9] of 423 would match all two-digit page numbers, and I’m sure you can guess what the third expression would look like. Yes, go ahead. Write it down.

Hey, neat! This is starting to make sense!

I was hoping you’d say that. But brace yourself, now it gets even better! We just saw that using sets, we could match one of several characters at once. But you can even repeat a character or set, reducing the number of expressions needed to handle the above page number example to one. Yes, ONE! Excited? You should be! It works like this: Some so-called special characters, “+”, ”?” and “*”, repeat the single element preceding them. (Element means either a single character, a character set, an escape sequence or a group (we’ll learn about those last two later)- in short, any single entity in a regular expression.) These characters are called wildcards or quantifiers. To be more precise, ”?” matches 0 or 1 of the preceding element, “*” matches 0 or more of the preceding element and “+” matches 1 or more of the preceding element. A few examples: The expression a? would match either “” (which is the empty string, not strictly useful in this case) or “a”, the expression a* would match “”, “a”, “aa” or any number of a’s in a row, and, finally, the expression a+ would match “a”, “aa” or any number of a’s in a row (Note: it wouldn’t match the empty string!). Same deal for sets: The expression [0-9]+ would match every integer number there is! I know what you’re thinking, and you’re right: If you use that in the above case of matching page numbers, wouldn’t that be the single one expression to match all the page numbers? Yes, the expression Page [0-9]+ of 423 would match every page number in that book!

Note

A note on these quantifiers: They generally try to match as much text as possible, so be careful when using them. This is called “greedy behaviour”- I’m sure you get why. It gets problematic when you, say, try to match a tag. Consider, for example, the string "<p class="calibre2">Title here</p>" and let’s say you’d want to match the opening tag (the part between the first pair of angle brackets, a little more on tags later). You’d think that the expression <p.*> would match that tag, but actually, it matches the whole string! (The character ”.” is another special character. It matches anything except linebreaks, so, basically, the expression .* would match any single line you can think of.) Instead, try using <p.*?> which makes the quantifier "*" non-greedy. That expression would only match the first opening tag, as intended.
There’s actually another way to accomplish this: The expression <p[^>]*> will match that same opening tag- you’ll see why after the next section. Just note that there quite frequently is more than one way to write a regular expression.

Well, these special characters are very neat and all, but what if I wanted to match a dot or a question mark?

You can of course do that: Just put a backslash in front of any special character and it is interpreted as the literal character, without any special meaning. This pair of a backslash followed by a single character is called an escape sequence, and the act of putting a backslash in front of a special character is called escaping that character. An escape sequence is interpreted as a single element. There are of course escape sequences that do more than just escaping special characters, for example "\t" means a tabulator. We’ll get to some of the escape sequences later. Oh, and by the way, concerning those special characters: Consider any character we discuss in this introduction as having some function to be special and thus needing to be escaped if you want the literal character.

So, what are the most useful sets?

Knew you’d ask. Some useful sets are [0-9] matching a single number, [a-z] matching a single lowercase letter, [A-Z] matching a single uppercase letter, [a-zA-Z] matching a single letter and [a-zA-Z0-9] matching a single letter or number. You can also use an escape sequence as shorthand:

\d is equivalent to [0-9]
\w is equivalent to [a-zA-Z0-9_]
\s is equivalent to any whitespace

Note

“Whitespace” is a term for anything that won’t be printed. These characters include space, tabulator, line feed, form feed and carriage return.

As a last note on sets, you can also define a set as any character but those in the set. You do that by including the character "^" as the very first character in the set. Thus, [^a] would match any character excluding “a”. That’s called complementing the set. Those escape sequence shorthands we saw earlier can also be complemented: "\D" means any non-number character, thus being equivalent to [^0-9]. The other shorthands can be complemented by, you guessed it, using the respective uppercase letter instead of the lowercase one. So, going back to the example <p[^>]*> from the previous section, now you can see that the character set it’s using tries to match any character except for a closing angle bracket.

But if I had a few varying strings I wanted to match, things get complicated?

Fear not, life still is good and easy. Consider this example: The book you’re converting has “Title” written on every odd page and “Author” written on every even page. Looks great in print, right? But in ebooks, it’s annoying. You can group whole expressions in normal parentheses, and the character "|" will let you match either the expression to its right or the one to its left. Combine those and you’re done. Too fast for you? Okay, first off, we group the expressions for odd and even pages, thus getting (Title)(Author) as our two needed expressions. Now we make things simpler by using the vertical bar ("|" is called the vertical bar character): If you use the expression (Title|Author) you’ll either get a match for “Title” (on the odd pages) or you’d match “Author” (on the even pages). Well, wasn’t that easy?

You can, of course, use the vertical bar without using grouping parentheses, as well. Remember when I said that quantifiers repeat the element preceding them? Well, the vertical bar works a little differently: The expression “Title|Author” will also match either the string “Title” or the string “Author”, just as the above example using grouping. The vertical bar selects between the entire expression preceding and following it. So, if you wanted to match the strings “Calibre” and “calibre” and wanted to select only between the upper- and lowercase “c”, you’d have to use the expression (c|C)alibre, where the grouping ensures that only the “c” will be selected. If you were to use c|Calibre, you’d get a match on the string “c” or on the string “Calibre”, which isn’t what we wanted. In short: If in doubt, use grouping together with the vertical bar.

You missed...

... wait just a minute, there’s one last, really neat thing you can do with groups. If you have a group that you previously matched, you can use references to that group later in the expression: Groups are numbered starting with 1, and you reference them by escaping the number of the group you want to reference, thus, the fifth group would be referenced as \5. So, if you searched for ([^ ]+) \1 in the string “Test Test”, you’d match the whole string!

In the beginning, you said there was a way to make a regular expression case insensitive?

Yes, I did, thanks for paying attention and reminding me. You can tell calibre how you want certain things handled by using something called flags. You include flags in your expression by using the special construct (?flags go here) where, obviously, you’d replace “flags go here” with the specific flags you want. For ignoring case, the flag is i, thus you include (?i) in your expression. Thus, test(?i) would match “Test”, “tEst”, “TEst” and any case variation you could think of.

Another useful flag lets the dot match any character at all, including the newline, the flag s. If you want to use multiple flags in an expression, just put them in the same statement: (?is) would ignore case and make the dot match all. It doesn’t matter which flag you state first, (?si) would be equivalent to the above. By the way, good places for putting flags in your expression would be either the very beginning or the very end. That way, they don’t get mixed up with anything else.

I think I’m beginning to understand these regular expressions now... how do I use them in calibre?

Conversions

Let’s begin with the conversion settings, which is really neat. In the Search and Replace part, you can input a regexp (short for regular expression) that describes the string that will be replaced during the conversion. The neat part is the wizard. Click on the wizard staff and you get a preview of what calibre “sees” during the conversion process. Scroll down to the string you want to remove, select and copy it, paste it into the regexp field on top of the window. If there are variable parts, like page numbers or so, use sets and quantifiers to cover those, and while you’re at it, remember to escape special characters, if there are some. Hit the button labeled Test and calibre highlights the parts it would replace were you to use the regexp. Once you’re satisfied, hit OK and convert. Be careful if your conversion source has tags like this example:

Maybe, but the cops feel like you do, Anita. What's one more dead vampire?
New laws don't change that. </p>
<p class="calibre4"> <b class="calibre2">Generated by ABC Amber LIT Conv
<a href="http://www.processtext.com/abclit.html" class="calibre3">erter,
http://www.processtext.com/abclit.html</a></b></p>
<p class="calibre4"> It had only been two years since Addison v. Clark.
The court case gave us a revised version of what life was

(shamelessly ripped out of this thread [http://www.mobileread.com/forums/showthread.php?t=75594”]). You’d have to remove some of the tags as well. In this example, I’d recommend beginning with the tag <b class="calibre2">, now you have to end with the corresponding closing tag (opening tags are <tag>, closing tags are </tag>), which is simply the next </b> in this case. (Refer to a good HTML manual or ask in the forum if you are unclear on this point.) The opening tag can be described using <b.*?>, the closing tag using </b>, thus we could remove everything between those tags using <b.*?>.*?</b>. But using this expression would be a bad idea, because it removes everything enclosed by <b>- tags (which, by the way, render the enclosed text in bold print), and it’s a fair bet that we’ll remove portions of the book in this way. Instead, include the beginning of the enclosed string as well, making the regular expression <b.*?>\s*Generated\s+by\s+ABC\s+Amber\s+LIT.*?</b> The \s with quantifiers are included here instead of explicitly using the spaces as seen in the string to catch any variations of the string that might occur. Remember to check what calibre will remove to make sure you don’t remove any portions you want to keep if you test a new expression. If you only check one occurrence, you might miss a mismatch somewhere else in the text. Also note that should you accidentally remove more or fewer tags than you actually wanted to, calibre tries to repair the damaged code after doing the removal.

Adding books

Another thing you can use regular expressions for is to extract metadata from filenames. You can find this feature in the “Adding books” part of the settings. There’s a special feature here: You can use field names for metadata fields, for example (?P<title>) would indicate that calibre uses this part of the string as book title. The allowed field names are listed in the windows, together with another nice test field. An example: Say you want to import a whole bunch of files named like Classical Texts: The Divine Comedy by Dante Alighieri.mobi.
(Obviously, this is already in your library, since we all love classical italian poetry) or Science Fiction epics: The Foundation Trilogy by Isaac Asimov.epub. This is obviously a naming scheme that calibre won’t extract any meaningful data out of - its standard expression for extracting metadata is (?P<title>.+) - (?P<author>[^_]+). A regular expression that works here would be [a-zA-Z]+: (?P<title>.+) by (?P<author>.+). Please note that, inside the group for the metadata field, you need to use expressions to describe what the field actually matches. And also note that, when using the test field calibre provides, you need to add the file extension to your testing filename, otherwise you won’t get any matches at all, despite using a working expression.

Bulk editing metadata

The last part is regular expression search and replace in metadata fields. You can access this by selecting multiple books in the library and using bulk metadata edit. Be very careful when using this last feature, as it can do Very Bad Things to your library! Doublecheck that your expressions do what you want them to using the test fields, and only mark the books you really want to change! In the regular expression search mode, you can search in one field, replace the text with something and even write the result into another field. A practical example: Say your library contained the books of Frank Herbert’s Dune series, named after the fashion Dune 1 - Dune, Dune 2 - Dune Messiah and so on. Now you want to get Dune into the series field. You can do that by searching for (.*?) \d+ - .* in the title field and replacing it with \1 in the series field. See what I did there? That’s a reference to the first group you’re replacing the series field with. Now that you have the series all set, you only need to do another search for .*? - in the title field and replace it with "" (an empty string), again in the title field, and your metadata is all neat and tidy. Isn’t that great? By the way, instead of replacing the entire field, you can also append or prepend to the field, so, if you wanted the book title to be prepended with series info, you could do that as well. As you by now have undoubtedly noticed, there’s a checkbox labeled Case sensitive, so you won’t have to use flags to select behaviour here.

Well, that just about concludes the very short introduction to regular expressions. Hopefully I’ll have shown you enough to at least get you started and to enable you to continue learning by yourself- a good starting point would be the Python documentation for regexps [http://docs.python.org/library/re.html].

One last word of warning, though: Regexps are powerful, but also really easy to get wrong. calibre provides really great testing possibilities to see if your expressions behave as you expect them to. Use them. Try not to shoot yourself in the foot. (God, I love that expression...) But should you, despite the warning, injure your foot (or any other body parts), try to learn from it.

Credits

Thanks for helping with tips, corrections and such:

- ldolse
- kovidgoyal
- chaley
- dwanthny
- kacir
- Starson17

For more about regexps see The Python User Manual [http://docs.python.org/library/re.html].

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

Integrating the calibre content server into other servers

Here, we will show you how to integrate the calibre content server into another server. The most common reason for this is to make use of SSL or more sophisticated authentication. There are two main techniques: Running the calibre content server as a standalone process and using a reverse proxy to connect it with your main server or running the content server in process in your main server with WSGI. The examples below are all for Apache 2.x on linux, but should be easily adaptable to other platforms.

Contents

- Using a reverse proxy

- In process

Note

This only applies to calibre releases >= 0.7.25

Using a reverse proxy

A reverse proxy is when your normal server accepts incoming requests and passes them onto the calibre server. It then reads the response from the calibre server and forwards it to the client. This means that you can simply run the calibre server as normal without trying to integrate it closely with your main server, and you can take advantage of whatever authentication systems you main server has in place. This is the simplest approach as it allows you to use the binary calibre install with no external dependencies/system integration requirements. Below, is an example of how to achieve this with Apache as your main server, but it will work with any server that supports Reverse Proxies.

First start the calibre content server as shown below:

calibre-server --url-prefix /calibre --port 8080

The key parameter here is --url-prefix /calibre. This causes the content server to serve all URLs prefixed by calibre. To see this in action, visit http://localhost:8080/calibre in your browser. You should see the normal content server website, but now it will run under /calibre.

Now suppose you are using Apache as your main server. First enable the proxy modules in apache, by adding the following to httpd.conf:

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

The exact technique for enabling the proxy modules will vary depending on your Apache installation. Once you have the proxy modules enabled, add the following rules to httpd.conf (or if you are using virtual hosts to the conf file for the virtual host in question:

RewriteEngine on
RewriteRule ^/calibre/(.*) http://localhost:8080/calibre/$1 [proxy]
RewriteRule ^/calibre http://localhost:8080 [proxy]
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

That’s all, you will now be able to access the calibre Content Server under the /calibre URL in your apache server. The above rules pass all requests under /calibre to the calibre server running on port 8080 and thanks to the –url-prefix option above, the calibre server handles them transparently.

Note

If you are willing to devote an entire VirtualHost to the content server, then there is no need to use –url-prefix and RewriteRule, instead just use the ProxyPass directive.

Note

The server engine calibre uses, CherryPy, can have trouble with proxying and KeepAlive requests, so turn them off in Apache, with the SetEnv directives shown above.

In process

The calibre content server can be run directly, in process, inside a host server like Apache using the WSGI framework.

Note

For this to work, all the dependencies needed by calibre must be installed on your system. On linux, this can be achieved fairly easily by installing the distribution provided calibre package (provided it is up to date).

First, we have to create a WSGI adapter for the calibre content server. Here is a template you can use for the purpose. Replace the paths as directed in the comments

# WSGI script file to run calibre content server as a WSGI app

import sys, os

# You can get the paths referenced here by running
# calibre-debug --paths
# on your server

# The first entry from CALIBRE_PYTHON_PATH
sys.path.insert(0, '/home/kovid/work/calibre/src')

# CALIBRE_RESOURCES_PATH
sys.resources_location = '/home/kovid/work/calibre/resources'

# CALIBRE_EXTENSIONS_PATH
sys.extensions_location = '/home/kovid/work/calibre/src/calibre/plugins'

# Path to directory containing calibre executables
sys.executables_location = '/usr/bin'

# Path to a directory for which the server has read/write permissions
# calibre config will be stored here
os.environ['CALIBRE_CONFIG_DIRECTORY'] = '/var/www/localhost/calibre-config'

del sys
del os

from calibre.library.server.main import create_wsgi_app
application = create_wsgi_app(
# The mount point of this WSGI application (i.e. the first argument to
# the WSGIScriptAlias directive). Set to empty string is mounted at /
prefix='/calibre',

# Path to the calibre library to be served
# The server process must have write permission for all files/dirs
# in this directory or BAD things will happen
path_to_library='/home/kovid/documents/demo library'
)

del create_wsgi_app

Save this adapter as calibre-wsgi-adpater.py somewhere your server will have access to it.

Let’s suppose that we want to use WSGI in Apache. First enable WSGI in Apache by adding the following to httpd.conf:

LoadModule proxy_module modules/mod_wsgi.so

The exact technique for enabling the wsgi module will vary depending on your Apache installation. Once you have the proxy modules enabled, add the following rules to httpd.conf (or if you are using virtual hosts to the conf file for the virtual host in question:

WSGIScriptAlias /calibre /var/www/localhost/cgi-bin/calibre-wsgi-adapter.py

Change the path to calibre-wsgi-adapter.py to wherever you saved it previously (make sure Apache has access to it).

That’s all, you will now be able to access the calibre Content Server under the /calibre URL in your apache server.

Note

For more help with using mod_wsgi in Apache, see mod_wsgi [http://code.google.com/p/modwsgi/wiki/WhereToGetHelp].

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

Writing your own plugins to extend calibre‘s functionality
calibre has a very modular design. Almost all functionality in calibre comes in the form of plugins. Plugins are used for conversion, for downloading news (though these are called recipes), for various components of the user interface, to connect to different devices, to process files when adding them to calibre and so on. You can get a complete list of all the built-in plugins in calibre by going to Preferences->Plugins.

Here, we will teach you how to create your own plugins to add new features to calibre.

Contents

- Anatomy of a calibre plugin

- A User Interface plugin

- __init__.py
- ui.py
- main.py
- Getting resources from the plugin zip file
- Enabling user configuration of your plugin

- The plugin API
- Debugging plugins
- More plugin examples
- Sharing your plugins with others

Note

This only applies to calibre releases >= 0.8.60

Anatomy of a calibre plugin

A calibre plugin is very simple, it’s just a zip file that contains some python code and any other resources like image files needed by the plugin. Without further ado, let’s see a basic example.

Suppose you have an installation of calibre that you are using to self publish various e-documents in EPUB and MOBI formats. You would like all files generated by calibre to have their publisher set as “Hello world”, here’s how to do it.
Create a file named __init__.py (this is a special name and must always be used for the main file of your plugin)
and enter the following Python code into it:

import os
from calibre.customize import FileTypePlugin

class HelloWorld(FileTypePlugin):

name = 'Hello World Plugin' # Name of the plugin
description = 'Set the publisher to Hello World for all new conversions'
supported_platforms = ['windows', 'osx', 'linux'] # Platforms this plugin will run on
author = 'Acme Inc.' # The author of this plugin
version = (1, 0, 0) # The version number of this plugin
file_types = set(['epub', 'mobi']) # The file types that this plugin will be applied to
on_postprocess = True # Run this plugin after conversion is complete
minimum_calibre_version = (0, 7, 53)

def run(self, path_to_ebook):
from calibre.ebooks.metadata.meta import get_metadata, set_metadata
file = open(path_to_ebook, 'r+b')
ext = os.path.splitext(path_to_ebook)[-1][1:].lower()
mi = get_metadata(file, ext)
mi.publisher = 'Hello World'
set_metadata(file, mi, ext)
return path_to_ebook

That’s all. To add this code to calibre as a plugin, simply run the following in the directory in which you created __init__.py:

calibre-customize -b .

Note

On OS X you have to first install the calibre command line tools, by going to Preferences->Miscellaneous and clicking the Install command line tools button.

You can download the Hello World plugin from helloworld_plugin.zip [http://calibre-ebook.com/downloads/helloworld_plugin.zip].

Every time you use calibre to convert a book, the plugin’s run() method will be called and the converted book will have its publisher set to “Hello World”. This is a trivial plugin, lets move on to a more complex example that actually adds a component to the user interface.

A User Interface plugin

This plugin will be spread over a few files (to keep the code clean). It will show you how to get resources
(images or data files) from the plugin zip file, allow users to configure your plugin, how to create elements in the calibre user interface and how to access and query the books database in calibre.

You can download this plugin from interface_demo_plugin.zip [http://calibre-ebook.com/downloads/interface_demo_plugin.zip]

The first thing to note is that this zip file has a lot more files in it, explained below, pay particular attention to plugin-import-name-interface_demo.txt.

plugin-import-name-interface_demo.txt

An empty text file used to enable the multi-file plugin magic. This file must be present in all plugins that use more than one .py file. It should be empty and its filename must be of the form: plugin-import-name-some_name.txt
The presence of this file allows you to import code from the .py files present inside the zip file, using a statement like:

from calibre_plugins.some_name.some_module import some_object

The prefix calibre_plugins must always be present. some_name comes from the filename of the empty text file. some_module refers to some_module.py file inside the zip file. Note that this importing is just as powerful as regular python imports. You can create packages and subpackages of .py modules inside the zip file, just like you would normally (by defining __init__.py in each sub directory), and everything should Just Work.

The name you use for some_name enters a global namespace shared by all plugins, so make it as unique as possible.
But remember that it must be a valid python identifier (only alphabets, numbers and the underscore).

__init__.py

As before, the file that defines the plugin class
main.py

This file contains the actual code that does something useful
ui.py

This file defines the interface part of the plugin
images/icon.png
The icon for this plugin
about.txt
A text file with information about the plugin

Now let’s look at the code.

__init__.py

First, the obligatory __init__.py to define the plugin metadata:

# The class that all Interface Action plugin wrappers must inherit from
from calibre.customize import InterfaceActionBase

class InterfacePluginDemo(InterfaceActionBase):
'''
This class is a simple wrapper that provides information about the actual
plugin class. The actual interface plugin class is called InterfacePlugin
and is defined in the ui.py file, as specified in the actual_plugin field
below.

The reason for having two classes is that it allows the command line
calibre utilities to run without needing to load the GUI libraries.
'''
name = 'Interface Plugin Demo'
description = 'An advanced plugin demo'
supported_platforms = ['windows', 'osx', 'linux']
author = 'Kovid Goyal'
version = (1, 0, 0)
minimum_calibre_version = (0, 7, 53)

#: This field defines the GUI plugin class that contains all the code
#: that actually does something. Its format is module_path:class_name
#: The specified class must be defined in the specified module.
actual_plugin = 'calibre_plugins.interface_demo.ui:InterfacePlugin'

def is_customizable(self):
'''
This method must return True to enable customization via
Preferences->Plugins
'''
return True

def config_widget(self):
'''
Implement this method and :meth:`save_settings` in your plugin to
use a custom configuration dialog.

This method, if implemented, must return a QWidget. The widget can have
an optional method validate() that takes no arguments and is called
immediately after the user clicks OK. Changes are applied if and only
if the method returns True.

If for some reason you cannot perform the configuration at this time,
return a tuple of two strings (message, details), these will be
displayed as a warning dialog to the user and the process will be
aborted.

The base class implementation of this method raises NotImplementedError
so by default no user configuration is possible.
'''
# It is important to put this import statement here rather than at the
# top of the module as importing the config class will also cause the
# GUI libraries to be loaded, which we do not want when using calibre
# from the command line
from calibre_plugins.interface_demo.config import ConfigWidget
return ConfigWidget()

def save_settings(self, config_widget):
'''
Save the settings specified by the user with config_widget.

:param config_widget: The widget returned by :meth:`config_widget`.
'''
config_widget.save_settings()

# Apply the changes
ac = self.actual_plugin_
if ac is not None:
ac.apply_settings()

The only noteworthy feature is the field actual_plugin. Since calibre has both command line and GUI interfaces,
GUI plugins like this one should not load any GUI libraries in __init__.py. The actual_plugin field does this for you, by telling calibre that the actual plugin is to be found in another file inside your zip archive, which will only be loaded in a GUI context.

Remember that for this to work, you must have a plugin-import-name-some_name.txt file in your plugin zip file, as discussed above.

Also there are a couple of methods for enabling user configuration of the plugin. These are discussed below.

ui.py

Now let’s look at ui.py which defines the actual GUI plugin. The source code is heavily commented and should be self explanatory:

# The class that all interface action plugins must inherit from
from calibre.gui2.actions import InterfaceAction
from calibre_plugins.interface_demo.main import DemoDialog

class InterfacePlugin(InterfaceAction):

name = 'Interface Plugin Demo'

# Declare the main action associated with this plugin
# The keyboard shortcut can be None if you dont want to use a keyboard
# shortcut. Remember that currently calibre has no central management for
# keyboard shortcuts, so try to use an unusual/unused shortcut.
action_spec = ('Interface Plugin Demo', None,
'Run the Interface Plugin Demo', 'Ctrl+Shift+F1')

def genesis(self):
# This method is called once per plugin, do initial setup here

# Set the icon for this interface action
# The get_icons function is a builtin function defined for all your
# plugin code. It loads icons from the plugin zip file. It returns
# QIcon objects, if you want the actual data, use the analogous
# get_resources builtin function.
#
# Note that if you are loading more than one icon, for performance, you
# should pass a list of names to get_icons. In this case, get_icons
# will return a dictionary mapping names to QIcons. Names that
# are not found in the zip file will result in null QIcons.
icon = get_icons('images/icon.png')

# The qaction is automatically created from the action_spec defined
# above
self.qaction.setIcon(icon)
self.qaction.triggered.connect(self.show_dialog)

def show_dialog(self):
# The base plugin object defined in __init__.py
base_plugin_object = self.interface_action_base_plugin
# Show the config dialog
# The config dialog can also be shown from within
# Preferences->Plugins, which is why the do_user_config
# method is defined on the base plugin class
do_user_config = base_plugin_object.do_user_config

# self.gui is the main calibre GUI. It acts as the gateway to access
# all the elements of the calibre user interface, it should also be the
# parent of the dialog
d = DemoDialog(self.gui, self.qaction.icon(), do_user_config)
d.show()

def apply_settings(self):
from calibre_plugins.interface_demo.config import prefs
# In an actual non trivial plugin, you would probably need to
# do something based on the settings in prefs
prefs

main.py

The actual logic to implement the Interface Plugin Demo dialog.

from PyQt4.Qt import QDialog, QVBoxLayout, QPushButton, QMessageBox, QLabel from calibre_plugins.interface_demo.config import prefs class DemoDialog(QDialog):

def __init__(self, gui, icon, do_user_config):
QDialog.__init__(self, gui)
self.gui = gui self.do_user_config = do_user_config

# The current database shown in the GUI
# db is an instance of the class LibraryDatabase2 from database.py
# This class has many, many methods that allow you to do a lot of
# things.
self.db = gui.current_db self.l = QVBoxLayout()
self.setLayout(self.l)

self.label = QLabel(prefs['hello_world_msg'])
self.l.addWidget(self.label)

self.setWindowTitle('Interface Plugin Demo')
self.setWindowIcon(icon)

self.about_button = QPushButton('About', self)
self.about_button.clicked.connect(self.about)
self.l.addWidget(self.about_button)

self.marked_button = QPushButton(
'Show books with only one format in the calibre GUI', self)
self.marked_button.clicked.connect(self.marked)
self.l.addWidget(self.marked_button)

self.view_button = QPushButton(
'View the most recently added book', self)
self.view_button.clicked.connect(self.view)
self.l.addWidget(self.view_button)

self.update_metadata_button = QPushButton(
'Update metadata in a book\'s files', self)
self.update_metadata_button.clicked.connect(self.update_metadata)
self.l.addWidget(self.update_metadata_button)

self.conf_button = QPushButton(
'Configure this plugin', self)
self.conf_button.clicked.connect(self.config)
self.l.addWidget(self.conf_button)

self.resize(self.sizeHint())

def about(self):
# Get the about text from a file inside the plugin zip file
# The get_resources function is a builtin function defined for all your
# plugin code. It loads files from the plugin zip file. It returns
# the bytes from the specified file.
#
# Note that if you are loading more than one file, for performance, you
# should pass a list of names to get_resources. In this case,
# get_resources will return a dictionary mapping names to bytes. Names that
# are not found in the zip file will not be in the returned dictionary.
text = get_resources('about.txt')
QMessageBox.about(self, 'About the Interface Plugin Demo', text.decode('utf-8'))

def marked(self):
''' Show books with only one format '''
fmt_idx = self.db.FIELD_MAP['formats']
matched_ids = set()
for record in self.db.data.iterall():
# Iterate over all records fmts = record[fmt_idx]
# fmts is either None or a comma separated list of formats if fmts and ',' not in fmts:
matched_ids.add(record[0])
# Mark the records with the matching ids self.db.set_marked_ids(matched_ids)

# Tell the GUI to search for all marked records self.gui.search.setEditText('marked:true')
self.gui.search.do_search()

def view(self):
''' View the most recently added book '''
most_recent = most_recent_id = None timestamp_idx = self.db.FIELD_MAP['timestamp']

for record in self.db.data:
# Iterate over all currently showing records timestamp = record[timestamp_idx]
if most_recent is None or timestamp > most_recent:
most_recent = timestamp most_recent_id = record[0]

if most_recent_id is not None:
# Get the row number of the id as shown in the GUI row_number = self.db.row(most_recent_id)
# Get a reference to the View plugin view_plugin = self.gui.iactions['View']
# Ask the view plugin to launch the viewer for row_number view_plugin._view_books([row_number])

def update_metadata(self):
'''
Set the metadata in the files in the selected book's record to match the current metadata in the database.
'''
from calibre.ebooks.metadata.meta import set_metadata from calibre.gui2 import error_dialog, info_dialog

# Get currently selected books rows = self.gui.library_view.selectionModel().selectedRows()
if not rows or len(rows) == 0:
return error_dialog(self.gui, 'Cannot update metadata',
'No books selected', show=True)
# Map the rows to book ids ids = list(map(self.gui.library_view.model().id, rows))
for book_id in ids:
# Get the current metadata for this book from the db mi = self.db.get_metadata(book_id, index_is_id=True, get_cover=True, cover_as_data=True)
fmts = self.db.formats(book_id, index_is_id=True)
if not fmts: continue for fmt in fmts.split(','):
fmt = fmt.lower()
# Get a python file object for the format. This will be either
# an in memory file or a temporary on disk file ffile = self.db.format(book_id, fmt, index_is_id=True, as_file=True)
# Set metadata in the format set_metadata(ffile, mi, fmt)
ffile.seek(0)
# Now replace the file in the calibre library with the updated
# file. We dont use add_format_with_hooks as the hooks were
# already run when the file was first added to calibre.
ffile.name = 'xxx' # add_format() will not work if the file
# path of the file being added is the same
# as the path of the file being replaced self.db.add_format(book_id, fmt, ffile, index_is_id=True)

info_dialog(self, 'Updated files',
'Updated the metadata in the files of %d book(s)'%len(ids), show=True)

def config(self):
self.do_user_config(parent=self)
# Apply the changes self.label.setText(prefs['hello_world_msg'])

Getting resources from the plugin zip file
calibre‘s plugin loading system defines a couple of built-in functions that allow you to conveniently get files from the plugin zip file.

get_resources(name_or_list_of_names)

This function should be called with a list of paths to files inside the zip file. For example to access the file icon.png in the directory images in the zip file, you would use: images/icon.png. Always use a forward slash as the path separator, even on windows. When you pass in a single name, the function will return the raw bytes of that file or None if the name was not found in the zip file. If you pass in more than one name then it returns a dict mapping the names to bytes.
If a name is not found, it will not be present in the returned dict.

get_icons(name_or_list_of_names)

A convenience wrapper for get_resources() that creates QIcon objects from the raw bytes returned by get_resources.
If a name is not found in the zip file the corresponding QIcon will be null.

Enabling user configuration of your plugin

To allow users to configure your plugin, you must define three methods in your base plugin class, ‘is_customizable, config_widget and save_settings as shown below:

def is_customizable(self):
'''
This method must return True to enable customization via
Preferences->Plugins
'''
return True

def config_widget(self):
'''
Implement this method and :meth:`save_settings` in your plugin to
use a custom configuration dialog.

This method, if implemented, must return a QWidget. The widget can have
an optional method validate() that takes no arguments and is called
immediately after the user clicks OK. Changes are applied if and only
if the method returns True.

If for some reason you cannot perform the configuration at this time,
return a tuple of two strings (message, details), these will be
displayed as a warning dialog to the user and the process will be
aborted.

The base class implementation of this method raises NotImplementedError
so by default no user configuration is possible.
'''
# It is important to put this import statement here rather than at the
# top of the module as importing the config class will also cause the
# GUI libraries to be loaded, which we do not want when using calibre
# from the command line
from calibre_plugins.interface_demo.config import ConfigWidget
return ConfigWidget()

def save_settings(self, config_widget):
'''
Save the settings specified by the user with config_widget.

:param config_widget: The widget returned by :meth:`config_widget`.
'''
config_widget.save_settings()

# Apply the changes
ac = self.actual_plugin_
if ac is not None:
ac.apply_settings()

calibre has many different ways to store configuration data (a legacy of its long history). The recommended way is to use the JSONConfig class, which stores your configuration information in a .json file.

The code to manage configuration data in the demo plugin is in config.py:

from PyQt4.Qt import QWidget, QHBoxLayout, QLabel, QLineEdit

from calibre.utils.config import JSONConfig

# This is where all preferences for this plugin will be stored
# Remember that this name (i.e. plugins/interface_demo) is also
# in a global namespace, so make it as unique as possible.
# You should always prefix your config file name with plugins/,
# so as to ensure you dont accidentally clobber a calibre config file
prefs = JSONConfig('plugins/interface_demo')

# Set defaults
prefs.defaults['hello_world_msg'] = 'Hello, World!'

class ConfigWidget(QWidget):

def __init__(self):
QWidget.__init__(self)
self.l = QHBoxLayout()
self.setLayout(self.l)

self.label = QLabel('Hello world &message:')
self.l.addWidget(self.label)

self.msg = QLineEdit(self)
self.msg.setText(prefs['hello_world_msg'])
self.l.addWidget(self.msg)
self.label.setBuddy(self.msg)

def save_settings(self):
prefs['hello_world_msg'] = unicode(self.msg.text())

The prefs object is now available throughout the plugin code by a simple:

from calibre_plugins.interface_demo.config import prefs

You can see the prefs object being used in main.py:

def config(self):
self.do_user_config(parent=self)
# Apply the changes
self.label.setText(prefs['hello_world_msg'])

The plugin API

As you may have noticed above, a plugin in calibre is a class. There are different classes for the different types of plugins in calibre.
Details on each class, including the base class of all plugins can be found in API Documentation for plugins.

Your plugin is almost certainly going to use code from calibre. To learn how to find various bits of functionality in the calibre code base, read the section on the calibre Code layout.

Debugging plugins

The first, most important step is to run calibre in debug mode. You can do this from the command line with:

calibre-debug -g

Or from within calibre by right-clicking the preferences button or using the Ctrl+Shift+R keyboard shortcut.

When running from the command line, debug output will be printed to the console, when running from within calibre the output will go to a txt file.

You can insert print statements anywhere in your plugin code, they will be output in debug mode. Remember, this is python, you really shouldn’t need anything more than print statements to debug ;) I developed all of calibre using just this debugging technique.

You can quickly test changes to your plugin by using the following command line:

calibre-debug -s; calibre-customize -b /path/to/your/plugin/directory; calibre

This will shutdown a running calibre, wait for the shutdown to complete, then update your plugin in calibre and relaunch calibre.

More plugin examples

You can find a list of many, sophisticated calibre plugins here [http://www.mobileread.com/forums/showthread.php?t=118764].

Sharing your plugins with others

If you would like to share the plugins you have created with other users of calibre, post your plugin in a new thread in the calibre plugins forum [http://www.mobileread.com/forums/forumdisplay.php?f=237].

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

Typesetting Math in ebooks

The calibre ebook viewer has the ability to display math embedded in ebooks (ePub and HTML files). You can typeset the math directly with TeX or MathML or
AsciiMath. The calibre viewer uses the excellent MathJax [http://www.mathjax.org] library to do this. This is a brief tutorial on creating ebooks with math in them that work well with the calibre viewer.

Note

This only applies to calibre version 0.8.66 and newer

A simple HTML file with mathematics

You can write mathematics inline inside a simple HTML file and the calibre viewer will render it into properly typeset mathematics. In the example below, we use
TeX notation for mathematics. You will see that you can use normal TeX commands, with the small caveat that ampersands and less than and greater than signs have to be written as &amp; &lt; and &gt; respectively.

The first step is to tell calibre that this will contains maths. You do this by adding the following snippet of code to the <head> section of the HTML file:

<script type="text/x-mathjax-config"></script>

That’s it, now you can type mathematics just as you would in a .tex file. For example, here are Lorentz’s equations:

<h2>The Lorenz Equations</h2>

<p>
\begin{align}
\dot{x} &amp; = \sigma(y-x) \\
\dot{y} &amp; = \rho x - y - xz \\
\dot{z} &amp; = -\beta z + xy
\end{align}
</p>

This snippet looks like the following screen shot in the calibre viewer.

The Lorenz Equations

The complete HTML file, with more equations and inline mathematics is reproduced below. You can convert this HTML file to EPUB in calibre to end up with an ebook you can distribute easily to other people.

<!DOCTYPE html>
<html>
<!-- Copyright (c) 2012 Design Science, Inc. -->
<head>
<title>Math Test Page</title>
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />

<!-- This script tag is needed to make calibre's ebook-viewer recpgnize that this file needs math typesetting -->
<script type="text/x-mathjax-config">
// This line adds numbers to all equations automatically, unless explicitly suppressed.
MathJax.Hub.Config({ TeX: { equationNumbers: {autoNumber: "all"} } });
</script>

<style>
h1 {text-align:center}
h2 {
font-weight: bold;
background-color: #DDDDDD;
padding: .2em .5em;
margin-top: 1.5em;
border-top: 3px solid #666666;
border-bottom: 2px solid #999999;
}
</style>
</head>
<body>

<h1>Sample Equations</h1>

<h2>The Lorenz Equations</h2>

<p>
\begin{align}
\dot{x} &amp; = \sigma(y-x) \label{lorenz}\\
\dot{y} &amp; = \rho x - y - xz \\
\dot{z} &amp; = -\beta z + xy
\end{align}
</p>

<h2>The Cauchy-Schwarz Inequality</h2>

<p>\[
\left( \sum_{k=1}^n a_k b_k \right)^{\!\!2} \leq
\left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)
\]</p>

<h2>A Cross Product Formula</h2>

<p>\[
\mathbf{V}_1 \times \mathbf{V}_2 =
\begin{vmatrix}
\mathbf{i} &amp; \mathbf{j} &amp; \mathbf{k} \\
\frac{\partial X}{\partial u} &amp; \frac{\partial Y}{\partial u} &amp; 0 \\
\frac{\partial X}{\partial v} &amp; \frac{\partial Y}{\partial v} &amp; 0 \\
\end{vmatrix}
\]</p>

<h2>The probability of getting \(k\) heads when flipping \(n\) coins is:</h2>

<p>\[P(E) = {n \choose k} p^k (1-p)^{ n-k} \]</p>

<h2>An Identity of Ramanujan</h2>

<p>\[
\frac{1}{(\sqrt{\phi \sqrt{5}}-\phi) e^{\frac25 \pi}} =
1+\frac{e^{-2\pi}} {1+\frac{e^{-4\pi}} {1+\frac{e^{-6\pi}}
{1+\frac{e^{-8\pi}} {1+\ldots} } } }
\]</p>

<h2>A Rogers-Ramanujan Identity</h2>

<p>\[
1 + \frac{q^2}{(1-q)}+\frac{q^6}{(1-q)(1-q^2)}+\cdots =
\prod_{j=0}^{\infty}\frac{1}{(1-q^{5j+2})(1-q^{5j+3})},
\quad\quad \text{for $|q|&lt;1$}.
\]</p>

<h2>Maxwell's Equations</h2>

<p>
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} &amp; = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} &amp; = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} &amp; = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} &amp; = 0
\end{align}
</p>

<h2>In-line Mathematics</h2>

<p>While display equations look good for a page of samples, the
ability to mix math and text in a paragraph is also important. This
expression \(\sqrt{3x-1}+(1+x)^2\) is an example of an inline equation. As
you see, equations can be used this way as well, without unduly
disturbing the spacing between lines.</p>

<h2>References to equations</h2>

<p>Here is a reference to the Lorenz Equations (\ref{lorenz}). Clicking on the equation number will take you back to the equation.</p>

</body>
</html>

More information

Since the calibre viewer uses the MathJax library to render mathematics, the best place to find out more about math in ebooks and get help is the MathJax website [http://www.mathjax.org].

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Tutorials »

Creating AZW3 • EPUB • MOBI Catalogs
calibre‘s Create catalog feature enables you to create a catalog of your library in a variety of formats. This help file describes cataloging options when generating a catalog in AZW3, EPUB and MOBI formats.

- Selecting books to catalog
- Included sections
- Prefixes
- Excluded books
- Excluded genres
- Other options
- Custom catalog covers
- Additional help resources

Selecting books to catalog

If you want all of your library cataloged, remove any search or filtering criteria in the main window. With a single book selected, all books in your library will be candidates for inclusion in the generated catalog. Individual books may be excluded by various criteria; see the Excluded genres section below for more information.

If you want only some of your library cataloged, you have two options:

- Create a multiple selection of the books you want cataloged. With more than one book selected in calibre‘s main window, only the selected books will be cataloged.

- Use the Search field or the Tag Browser to filter the displayed books. Only the displayed books will be cataloged.

To begin catalog generation, select the menu item Convert books > Create a catalog of the books in your calibre library. You may also add a Create Catalog button to a toolbar in Preferences > Interface > Toolbars for easier access to the Generate catalog dialog.

In Catalog options, select AZW3, EPUB or MOBI as the Catalog format. In the Catalog title field, provide a name that will be used for the generated catalog. If a catalog of the same name and format already exists, it will be replaced with the newly-generated catalog.

Enabling Send catalog to device automatically will download the generated catalog to a connected device upon completion.

Included sections

Sections enabled by a checkmark will be included in the generated catalog:

- Authors - all books, sorted by author, presented in a list format. Non-series books are listed before series books.

- Titles - all books, sorted by title, presented in a list format.

- Series - all books that are part of a series, sorted by series, presented in a list format.

- Genres - individual genres presented in a list, sorted by Author and Series.

- Recently Added - all books, sorted in reverse chronological order. List includes books added in the last 30 days, then a month-by-month listing of added books.

- Descriptions - detailed description page for each book, including a cover thumbnail and comments. Sorted by author, with non-series books listed before series books.

Prefixes

Prefix rules allow you to add a prefix to book listings when certain criteria are met. For example, you might want to mark books you’ve read with a checkmark, or books on your wishlist with an X.

The checkbox in the first column enables the rule. Name is a rule name that you provide. Field is either Tags or a custom column from your library. Value is the content of Field to match. When a prefix rule is satisfied, the book will be marked with the selected Prefix.

Three prefix rules have been specified in the example above:

1. Read book specifies that a book with any date in a custom column named Last read will be prefixed with a checkmark symbol.

2. Wishlist item specifies that any book with a Wishlist tag will be prefixed with an X symbol.

3. Library books specifies that any book with a value of True (or Yes) in a custom column Available in Library will be prefixed with a double arrow symbol.

The first matching rule supplies the prefix. Disabled or incomplete rules are ignored.

Excluded books

Exclusion rules allow you to specify books that will not be cataloged.

The checkbox in the first column enables the rule. Name is a rule name that you provide. Field is either Tags or a custom column in your library. Value is the content of Field to match. When an exclusion rule is satisfied, the book will be excluded from the generated catalog.

Two exclusion rules have been specified in the example above:

1. The Catalogs rule specifies that any book with a Catalog tag will be excluded from the generated catalog.

2. The Archived Books rule specifies that any book with a value of Archived in the custom column Status will be excluded from the generated catalog.

All rules are evaluated for every book. Disabled or incomplete rules are ignored.

Excluded genres

When the catalog is generated, tags in your database are used as genres. For example, you may use the tags Fiction and Nonfiction. These tags become genres in the generated catalog, with books listed under their respective genre lists based on their assigned tags. A book will be listed in every genre section for which it has a corresponding tag.

You may be using certain tags for other purposes, perhaps a + to indicate a read book, or a bracketed tag like [Amazon Freebie] to indicate a book’s source. The Excluded genres regex allows you to specify tags that you don’t want used as genres in the generated catalog. The default exclusion regex pattern \[.+\]\+ excludes any tags of the form [tag], as well as excluding +, the default tag for read books, from being used as genres in the generated catalog.

You can also use an exact tag name in a regex. For example, [Amazon Freebie] or [Project Gutenberg]. If you want to list multiple exact tags for exclusion, put a pipe (vertical bar) character between them: [Amazon Freebie]|[Project Gutenberg].

Results of regex shows you which tags will be excluded when the catalog is built, based on the tags in your database and the regex pattern you enter. The results are updated as you modify the regex pattern.

Other options

Catalog cover specifies whether to generate a new cover or use an existing cover. It is possible to create a custom cover for your catalogs - see Custom catalog covers for more information. If you have created a custom cover that you want to reuse, select Use existing cover. Otherwise, select Generate new cover.

Extra Description note specifies a custom column’s contents to be inserted into the Description page, next to the cover thumbnail. For example, you might want to display the date you last read a book using a Last Read custom column. For advanced use of the Description note feature, see this post in the calibre forum [http://www.mobileread.com/forums/showpost.php?p=1335767&postcount=395].

Thumb width specifies a width preference for cover thumbnails included with Descriptions pages. Thumbnails are cached to improve performance.To experiment with different widths, try generating a catalog with just a few books until you’ve determined your preferred width, then generate your full catalog. The first time a catalog is generated with a new thumbnail width, performance will be slower, but subsequent builds of the catalog will take advantage of the thumbnail cache.

Merge with Comments specifies a custom column whose content will be non-destructively merged with the Comments metadata during catalog generation. For example, you might have a custom column Author Bio that you’d like to append to the Comments metadata. You can choose to insert the custom column contents before or after the Comments section, and optionally separate the appended content with a horizontal rule separator. Eligible custom column types include text, comments, and composite.

Custom catalog covers

With the Generate Cover plugin [http://www.mobileread.com/forums/showthread.php?t=124219] installed, you can create custom covers for your catalog.
To install the plugin, go to Preferences > Advanced > Plugins > Get new plugins.

Additional help resources

For more information on calibre‘s Catalog feature, see the MobileRead forum sticky Creating Catalogs - Start here [http://www.mobileread.com/forums/showthread.php?t=118556], where you can find information on how to customize the catalog templates, and how to submit a bug report.

To ask questions or discuss calibre’s Catalog feature with other users, visit the MobileRead forum Calibre Catalogs [http://www.mobileread.com/forums/forumdisplay.php?f=238].

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

Customizing calibre
calibre has a highly modular design. Various parts of it can be customized. You can learn how to create recipes to add new sources of online content to calibre in the Section Adding your favorite news website. Here, you will learn, first, how to use environment variables and tweaks to customize calibre‘s behavior, and then how to specify your own static resources like icons and templates to override the defaults and finally how to use plugins to add functionality to calibre.

- Environment variables
- Tweaks
- Overriding icons, templates, et cetera
- Customizing calibre with plugins

Environment variables

- CALIBRE_CONFIG_DIRECTORY - sets the directory where configuration files are stored/read.

- CALIBRE_TEMP_DIR - sets the temporary directory used by calibre

- CALIBRE_OVERRIDE_DATABASE_PATH - allows you to specify the full path to metadata.db. Using this variable you can have metadata.db be in a location other than the library folder. Useful if your library folder is on a networked drive that does not support file locking.

- CALIBRE_DEVELOP_FROM - Used to run from a calibre development environment. See Setting up a calibre development environment.

- CALIBRE_OVERRIDE_LANG - Used to force the language used by the interface (ISO 639 language code)

- CALIBRE_NO_NATIVE_FILEDIALOGS - Causes calibre to not use native file dialogs for selecting files/directories.

- SYSFS_PATH - Use if sysfs is mounted somewhere other than /sys

- http_proxy - Used on linux to specify an HTTP proxy

Tweaks

Tweaks are small changes that you can specify to control various aspects of calibre‘s behavior. You can change them by going to Preferences->Advanced->Tweaks.
The default values for the tweaks are reproduced below

#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
__license__ = 'GPL v3'
__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'

'''
Contains various tweaks that affect calibre behavior. Only edit this file if you know what you are doing. If you delete this file, it will be recreated from defaults.
'''

#: Auto increment series index
# The algorithm used to assign a book added to an existing series a series number.
# New series numbers assigned using this tweak are always integer values, except
# if a constant non-integer is specified.
# Possible values are:
# next - First available integer larger than the largest existing number
# first_free - First available integer larger than 0
# next_free - First available integer larger than the smallest existing number
# last_free - First available integer smaller than the largest existing number
# Return largest existing + 1 if no free number is found
# const - Assign the number 1 always
# no_change - Do not change the series index
# a number - Assign that number always. The number is not in quotes. Note that
# 0.0 can be used here.
# Examples:
# series_index_auto_increment = 'next'
# series_index_auto_increment = 'next_free'
# series_index_auto_increment = 16.5
#
# Set the use_series_auto_increment_tweak_when_importing tweak to True to
# use the above values when importing/adding books. If this tweak is set to
# False (the default) then the series number will be set to 1 if it is not
# explicitly set to during the import. If set to True, then the
# series index will be set according to the series_index_auto_increment setting.
# Note that the use_series_auto_increment_tweak_when_importing tweak is used
# only when a value is not provided during import. If the importing regular
# expression produces a value for series_index, or if you are reading metadata
# from books and the import plugin produces a value, than that value will
# be used irrespective of the setting of the tweak.
series_index_auto_increment = 'next'
use_series_auto_increment_tweak_when_importing = False

#: Add separator after completing an author name
# Should the completion separator be append
# to the end of the completed text to
# automatically begin a new completion operation
# for authors.
# Can be either True or False authors_completer_append_separator = False

#: Author sort name algorithm
# The algorithm used to copy author to author_sort
# Possible values are:
# invert: use "fn ln" -> "ln, fn"
# copy : copy author to author_sort without modification
# comma : use 'copy' if there is a ',' in the name, otherwise use 'invert'
# nocomma : "fn ln" -> "ln fn" (without the comma)
# When this tweak is changed, the author_sort values stored with each author
# must be recomputed by right-clicking on an author in the left-hand tags pane,
# selecting 'manage authors', and pressing 'Recalculate all author sort values'.
# The author name suffixes are words that are ignored when they occur at the
# end of an author name. The case of the suffix is ignored and trailing
# periods are automatically handled. The same is true for prefixes.
# The author name copy words are a set of words which if they occur in an
# author name cause the automatically generated author sort string to be
# identical to the author name. This means that the sort for a string like Acme
# Inc. will be Acme Inc. instead of Inc., Acme author_sort_copy_method = 'comma'
author_name_suffixes = ('Jr', 'Sr', 'Inc', 'Ph.D', 'Phd',
'MD', 'M.D', 'I', 'II', 'III', 'IV',
'Junior', 'Senior')
author_name_prefixes = ('Mr', 'Mrs', 'Ms', 'Dr', 'Prof')
author_name_copywords = ('Corporation', 'Company', 'Co.', 'Agency', 'Council',
'Committee', 'Inc.', 'Institute', 'Society', 'Club', 'Team')

#: Splitting multiple author names
# By default, calibre splits a string containing multiple author names on
# ampersands and the words "and" and "with". You can customize the splitting
# by changing the regular expression below. Strings are split on whatever the
# specified regular expression matches.
# Default: r'(?i),?\s+(and|with)\s+'
authors_split_regex = r'(?i),?\s+(and|with)\s+'

#: Use author sort in Tag Browser
# Set which author field to display in the tags pane (the list of authors,
# series, publishers etc on the left hand side). The choices are author and
# author_sort. This tweak affects only what is displayed under the authors
# category in the tags pane and content server. Please note that if you set this
# to author_sort, it is very possible to see duplicate names in the list because
# although it is guaranteed that author names are unique, there is no such
# guarantee for author_sort values. Showing duplicates won't break anything, but
# it could lead to some confusion. When using 'author_sort', the tooltip will
# show the author's name.
# Examples:
# categories_use_field_for_author_name = 'author'
# categories_use_field_for_author_name = 'author_sort'
categories_use_field_for_author_name = 'author'

#: Control partitioning of Tag Browser
# When partitioning the tags browser, the format of the subcategory label is
# controlled by a template: categories_collapsed_name_template if sorting by
# name, categories_collapsed_rating_template if sorting by average rating, and
# categories_collapsed_popularity_template if sorting by popularity. There are
# two variables available to the template: first and last. The variable 'first'
# is the initial item in the subcategory, and the variable 'last' is the final
# item in the subcategory. Both variables are 'objects'; they each have multiple
# values that are obtained by using a suffix. For example, first.name for an
# author category will be the name of the author. The sub-values available are:
# name: the printable name of the item
# count: the number of books that references this item
# avg_rating: the average rating of all the books referencing this item
# sort: the sort value. For authors, this is the author_sort for that author
# category: the category (e.g., authors, series) that the item is in.
# Note that the "r'" in front of the { is necessary if there are backslashes
# (\ characters) in the template. It doesn't hurt anything to leave it there
# even if there aren't any backslashes.
categories_collapsed_name_template = r'{first.sort:shorten(4,,0)} - {last.sort:shorten(4,,0)}'
categories_collapsed_rating_template = r'{first.avg_rating:4.2f:ifempty(0)} - {last.avg_rating:4.2f:ifempty(0)}'
categories_collapsed_popularity_template = r'{first.count:d} - {last.count:d}'

#: Control order of categories in the tag browser
# Change the following dict to change the order that categories are displayed in
# the tag browser. Items are named using their lookup name, and will be sorted
# using the number supplied. The lookup name '*' stands for all names that
# otherwise do not appear. Two names with the same value will be sorted
# using the default order; the one used when the dict is empty.
# Example: tag_browser_category_order = {'series':1, 'tags':2, '*':3}
# resulting in the order series, tags, then everything else in default order.
tag_browser_category_order = {'*':1}

#: Specify columns to sort the booklist by on startup
# Provide a set of columns to be sorted on when calibre starts
# The argument is None if saved sort history is to be used
# otherwise it is a list of column,order pairs. Column is the
# lookup/search name, found using the tooltip for the column
# Order is 0 for ascending, 1 for descending
# For example, set it to [('authors',0),('title',0)] to sort by
# title within authors.
sort_columns_at_startup = None

#: Control how dates are displayed
# Format to be used for publication date and the timestamp (date).
# A string controlling how the publication date is displayed in the GUI
# d the day as number without a leading zero (1 to 31)
# dd the day as number with a leading zero (01 to 31)
# ddd the abbreviated localized day name (e.g. 'Mon' to 'Sun').
# dddd the long localized day name (e.g. 'Monday' to 'Qt::Sunday').
# M the month as number without a leading zero (1-12)
# MM the month as number with a leading zero (01-12)
# MMM the abbreviated localized month name (e.g. 'Jan' to 'Dec').
# MMMM the long localized month name (e.g. 'January' to 'December').
# yy the year as two digit number (00-99)
# yyyy the year as four digit number
# h the hours without a leading 0 (0 to 11 or 0 to 23, depending on am/pm) '
# hh the hours with a leading 0 (00 to 11 or 00 to 23, depending on am/pm) '
# m the minutes without a leading 0 (0 to 59) '
# mm the minutes with a leading 0 (00 to 59) '
# s the seconds without a leading 0 (0 to 59) '
# ss the seconds with a leading 0 (00 to 59) '
# ap use a 12-hour clock instead of a 24-hour clock, with "ap"
# replaced by the localized string for am or pm '
# AP use a 12-hour clock instead of a 24-hour clock, with "AP"
# replaced by the localized string for AM or PM '
# iso the date with time and timezone. Must be the only format present
# For example, given the date of 9 Jan 2010, the following formats show
# MMM yyyy ==> Jan 2010 yyyy ==> 2010 dd MMM yyyy ==> 09 Jan 2010
# MM/yyyy ==> 01/2010 d/M/yy ==> 9/1/10 yy ==> 10
# publication default if not set: MMM yyyy
# timestamp default if not set: dd MMM yyyy
# last_modified_display_format if not set: dd MMM yyyy gui_pubdate_display_format = 'MMM yyyy'
gui_timestamp_display_format = 'dd MMM yyyy'
gui_last_modified_display_format = 'dd MMM yyyy'

#: Control sorting of titles and series in the library display
# Control title and series sorting in the library view. If set to
# 'library_order', the title sort field will be used instead of the title.
# Unless you have manually edited the title sort field, leading articles such as
# The and A will be ignored. If set to 'strictly_alphabetic', the titles will be
# sorted as-is (sort by title instead of title sort). For example, with
# library_order, The Client will sort under 'C'. With strictly_alphabetic, the
# book will sort under 'T'.
# This flag affects Calibre's library display. It has no effect on devices. In
# addition, titles for books added before changing the flag will retain their
# order until the title is edited. Double-clicking on a title and hitting return
# without changing anything is sufficient to change the sort.
title_series_sorting = 'library_order'

#: Control formatting of title and series when used in templates
# Control how title and series names are formatted when saving to disk/sending
# to device. The behavior depends on the field being processed. If processing
# title, then if this tweak is set to 'library_order', the title will be
# replaced with title_sort. If it is set to 'strictly_alphabetic', then the
# title will not be changed. If processing series, then if set to
# 'library_order', articles such as 'The' and 'An' will be moved to the end. If
# set to 'strictly_alphabetic', the series will be sent without change.
# For example, if the tweak is set to library_order, "The Lord of the Rings"
# will become "Lord of the Rings, The". If the tweak is set to
# strictly_alphabetic, it would remain "The Lord of the Rings". Note that the
# formatter function raw_field will return the base value for title and
# series regardless of the setting of this tweak.
save_template_title_series_sorting = 'library_order'

#: Set the list of words considered to be "articles" for sort strings
# Set the list of words that are to be considered 'articles' when computing the
# title sort strings. The articles differ by language. By default, calibre uses
# a combination of articles from English and whatever language the calibre user
# interface is set to. In addition, in some contexts where the book language is
# available, the language of the book is used. You can change the list of
# articles for a given language or add a new language by editing
# per_language_title_sort_articles. To tell calibre to use a language other
# than the user interface language, set, default_language_for_title_sort. For
# example, to use German, set it to 'deu'. A value of None means the user
# interface language is used. The setting title_sort_articles is ignored
# (present only for legacy reasons).
per_language_title_sort_articles = {
# English
'eng' : (r'A\s+', r'The\s+', r'An\s+'),
# Spanish
'spa' : (r'El\s+', r'La\s+', r'Lo\s+', r'Los\s+', r'Las\s+', r'Un\s+', r'Una\s+', r'Unos\s+', r'Unas\s+'),
# French
'fra' : (r'Le\s+', r'La\s+', r"L'", r'Les\s+', r'Un\s+', r'Une\s+', r'Des\s+', r'De\s+La\s+', r'De\s+', r"D'"),
# Italian
'ita' : (r'Lo\s+', r'Il\s+', r"L'", r'La\s+', r'Gli\s+', r'I\s+', r'Le\s+', ),
# Portuguese
'por' : (r'A\s+', r'O\s+', r'Os\s+', r'As\s+', r'Um\s+', r'Uns\s+', r'Uma\s+', r'Umas\s+', ),
# Romanian
'ron' : (r'Un\s+', r'O\s+', r'Nişte\s+', ),
# German
'deu' : (r'Der\s+', r'Die\s+', r'Das\s+', r'Den\s+', r'Ein\s+', r'Eine\s+', r'Einen\s+', r'Dem\s+', r'Des\s+', r'Einem\s+', r'Eines\s+'),
# Dutch
'nld' : (r'De\s+', r'Het\s+', r'Een\s+', r"'n\s+", r"'s\s+", r'Ene\s+', r'Ener\s+', r'Enes\s+', r'Den\s+', r'Der\s+', r'Des\s+', r"'t\s+"),
# Swedish
'swe' : (r'En\s+', r'Ett\s+', r'Det\s+', r'Den\s+', r'De\s+', ),
# Turkish
'tur' : (r'Bir\s+', ),
# Afrikaans
'afr' : (r"'n\s+", r'Die\s+', ),
# Greek
'ell' : (r'O\s+', r'I\s+', r'To\s+', r'Ta\s+', r'Tus\s+', r'Tis\s+', r"'Enas\s+", r"'Mia\s+", r"'Ena\s+", r"'Enan\s+", ),
# Hungarian
'hun' : (r'A\s+', 'Az\s+', 'Egy\s+',),
}
default_language_for_title_sort = None title_sort_articles=r'^(A|The|An)\s+'

#: Specify a folder calibre should connect to at startup
# Specify a folder that calibre should connect to at startup using
# connect_to_folder. This must be a full path to the folder. If the folder does
# not exist when calibre starts, it is ignored. If there are '\' characters in
# the path (such as in Windows paths), you must double them.
# Examples:
# auto_connect_to_folder = 'C:\\Users\\someone\\Desktop\\testlib'
# auto_connect_to_folder = '/home/dropbox/My Dropbox/someone/library'
auto_connect_to_folder = ''

#: Specify renaming rules for SONY collections
# Specify renaming rules for sony collections. This tweak is only applicable if
# metadata management is set to automatic. Collections on Sonys are named
# depending upon whether the field is standard or custom. A collection derived
# from a standard field is named for the value in that field. For example, if
# the standard 'series' column contains the value 'Darkover', then the
# collection name is 'Darkover'. A collection derived from a custom field will
# have the name of the field added to the value. For example, if a custom series
# column named 'My Series' contains the name 'Darkover', then the collection
# will by default be named 'Darkover (My Series)'. For purposes of this
# documentation, 'Darkover' is called the value and 'My Series' is called the
# category. If two books have fields that generate the same collection name,
# then both books will be in that collection.
# This set of tweaks lets you specify for a standard or custom field how
# the collections are to be named. You can use it to add a description to a
# standard field, for example 'Foo (Tag)' instead of the 'Foo'. You can also use
# it to force multiple fields to end up in the same collection. For example, you
# could force the values in 'series', '#my_series_1', and '#my_series_2' to
# appear in collections named 'some_value (Series)', thereby merging all of the
# fields into one set of collections.
# There are two related tweaks. The first determines the category name to use
# for a metadata field. The second is a template, used to determines how the
# value and category are combined to create the collection name.
# The syntax of the first tweak, sony_collection_renaming_rules, is:
# {'field_lookup_name':'category_name_to_use', 'lookup_name':'name', ...}
# The second tweak, sony_collection_name_template, is a template. It uses the
# same template language as plugboards and save templates. This tweak controls
# how the value and category are combined together to make the collection name.
# The only two fields available are {category} and {value}. The {value} field is
# never empty. The {category} field can be empty. The default is to put the
# value first, then the category enclosed in parentheses, it isn't empty:
# '{value} {category:|(|)}'
# Examples: The first three examples assume that the second tweak
# has not been changed.
# 1: I want three series columns to be merged into one set of collections. The
# column lookup names are 'series', '#series_1' and '#series_2'. I want nothing
# in the parenthesis. The value to use in the tweak value would be:
# sony_collection_renaming_rules={'series':'', '#series_1':'', '#series_2':''}
# 2: I want the word '(Series)' to appear on collections made from series, and
# the word '(Tag)' to appear on collections made from tags. Use:
# sony_collection_renaming_rules={'series':'Series', 'tags':'Tag'}
# 3: I want 'series' and '#myseries' to be merged, and for the collection name
# to have '(Series)' appended. The renaming rule is:
# sony_collection_renaming_rules={'series':'Series', '#myseries':'Series'}
# 4: Same as example 2, but instead of having the category name in parentheses
# and appended to the value, I want it prepended and separated by a colon, such
# as in Series: Darkover. I must change the template used to format the category name
# The resulting two tweaks are:
# sony_collection_renaming_rules={'series':'Series', 'tags':'Tag'}
# sony_collection_name_template='{category:||: }{value}'
sony_collection_renaming_rules={}
sony_collection_name_template='{value}{category:| (|)}'

#: Specify how SONY collections are sorted
# Specify how sony collections are sorted. This tweak is only applicable if
# metadata management is set to automatic. You can indicate which metadata is to
# be used to sort on a collection-by-collection basis. The format of the tweak
# is a list of metadata fields from which collections are made, followed by the
# name of the metadata field containing the sort value.
# Example: The following indicates that collections built from pubdate and tags
# are to be sorted by the value in the custom column '#mydate', that collections
# built from 'series' are to be sorted by 'series_index', and that all other
# collections are to be sorted by title. If a collection metadata field is not
# named, then if it is a series- based collection it is sorted by series order,
# otherwise it is sorted by title order.
# [(['pubdate', 'tags'],'#mydate'), (['series'],'series_index'), (['*'], 'title')]
# Note that the bracketing and parentheses are required. The syntax is
# [ ( [list of fields], sort field ) , ( [ list of fields ] , sort field ) ]
# Default: empty (no rules), so no collection attributes are named.
sony_collection_sorting_rules = []

#: Control how tags are applied when copying books to another library
# Set this to True to ensure that tags in 'Tags to add when adding
# a book' are added when copying books to another library add_new_book_tags_when_importing_books = False

#: Set the maximum number of tags to show per book in the content server max_content_server_tags_shown=5

#: Set custom metadata fields that the content server will or will not display.
# content_server_will_display is a list of custom fields to be displayed.
# content_server_wont_display is a list of custom fields not to be displayed.
# wont_display has priority over will_display.
# The special value '*' means all custom fields. The value [] means no entries.
# Defaults:
# content_server_will_display = ['*']
# content_server_wont_display = []
# Examples:
# To display only the custom fields #mytags and #genre:
# content_server_will_display = ['#mytags', '#genre']
# content_server_wont_display = []
# To display all fields except #mycomments:
# content_server_will_display = ['*']
# content_server_wont_display['#mycomments']
content_server_will_display = ['*']
content_server_wont_display = []

#: Set the maximum number of sort 'levels'
# Set the maximum number of sort 'levels' that calibre will use to resort the
# library after certain operations such as searches or device insertion. Each
# sort level adds a performance penalty. If the database is large (thousands of
# books) the penalty might be noticeable. If you are not concerned about multi-
# level sorts, and if you are seeing a slowdown, reduce the value of this tweak.
maximum_resort_levels = 5

#: Choose whether dates are sorted using visible fields
# Date values contain both a date and a time. When sorted, all the fields are
# used, regardless of what is displayed. Set this tweak to True to use only
# the fields that are being displayed.
sort_dates_using_visible_fields = False

#: Specify which font to use when generating a default cover or masthead
# Absolute path to .ttf font files to use as the fonts for the title, author
# and footer when generating a default cover or masthead image. Useful if the
# default font (Liberation Serif) does not contain glyphs for the language of
# the books in your library.
generate_cover_title_font = None generate_cover_foot_font = None

#: Control behavior of the book list
# You can control the behavior of doubleclicks on the books list.
# Choices: open_viewer, do_nothing,
# edit_cell, edit_metadata. Selecting edit_metadata has the side effect of
# disabling editing a field using a single click.
# Default: open_viewer.
# Example: doubleclick_on_library_view = 'do_nothing'
# You can also control whether the book list scrolls horizontal per column or
# per pixel. Default is per column.
doubleclick_on_library_view = 'open_viewer'
horizontal_scrolling_per_column = True

#: Language to use when sorting.
# Setting this tweak will force sorting to use the
# collating order for the specified language. This might be useful if you run
# calibre in English but want sorting to work in the language where you live.
# Set the tweak to the desired ISO 639-1 language code, in lower case.
# You can find the list of supported locales at
# http://publib.boulder.ibm.com/infocenter/iseries/v5r3/topic/nls/rbagsicusortsequencetables.htm
# Default: locale_for_sorting = '' -- use the language calibre displays in
# Example: locale_for_sorting = 'fr' -- sort using French rules.
# Example: locale_for_sorting = 'nb' -- sort using Norwegian rules.
locale_for_sorting = ''

#: Number of columns for custom metadata in the edit metadata dialog
# Set whether to use one or two columns for custom metadata when editing
# metadata one book at a time. If True, then the fields are laid out using two
# columns. If False, one column is used.
metadata_single_use_2_cols_for_custom_fields = True

#: Order of custom column(s) in edit metadata
# Controls the order that custom columns are listed in edit metadata single
# and bulk. The columns listed in the tweak are displayed first and in the
# order provided. Any columns not listed are dislayed after the listed ones,
# in alphabetical order. Do note that this tweak does not change the size of
# the edit widgets. Putting comments widgets in this list may result in some
# odd widget spacing when using two-column mode.
# Enter a comma-separated list of custom field lookup names, as in
# metadata_edit_custom_column_order = ['#genre', '#mytags', '#etc']
metadata_edit_custom_column_order = []

#: The number of seconds to wait before sending emails
# The number of seconds to wait before sending emails when using a
# public email server like gmail or hotmail. Default is: 5 minutes
# Setting it to lower may cause the server's SPAM controls to kick in,
# making email sending fail. Changes will take effect only after a restart of
# calibre.
public_smtp_relay_delay = 301

#: The maximum width and height for covers saved in the calibre library
# All covers in the calibre library will be resized, preserving aspect ratio,
# to fit within this size. This is to prevent slowdowns caused by extremely
# large covers maximum_cover_size = (1200, 1600)

#: Where to send downloaded news
# When automatically sending downloaded news to a connected device, calibre
# will by default send it to the main memory. By changing this tweak, you can
# control where it is sent. Valid values are "main", "carda", "cardb". Note
# that if there isn't enough free space available on the location you choose,
# the files will be sent to the location with the most free space.
send_news_to_device_location = "main"

#: What interfaces should the content server listen on
# By default, the calibre content server listens on '0.0.0.0' which means that it
# accepts IPv4 connections on all interfaces. You can change this to, for
# example, '127.0.0.1' to only listen for connections from the local machine, or
# to '::' to listen to all incoming IPv6 and IPv4 connections (this may not
# work on all operating systems)
server_listen_on = '0.0.0.0'

#: Unified toolbar on OS X
# If you enable this option and restart calibre, the toolbar will be 'unified'
# with the titlebar as is normal for OS X applications. However, doing this has
# various bugs, for instance the minimum width of the toolbar becomes twice
# what it should be and it causes other random bugs on some systems, so turn it
# on at your own risk!
unified_title_toolbar_on_osx = False

#: Save original file when converting from same format to same format
# When calibre does a conversion from the same format to the same format, for
# example, from EPUB to EPUB, the original file is saved, so that in case the
# conversion is poor, you can tweak the settings and run it again. By setting
# this to False you can prevent calibre from saving the original file.
save_original_format = True

#: Number of recently viewed books to show
# Right-clicking the View button shows a list of recently viewed books. Control
# how many should be shown, here.
gui_view_history_size = 15

#: Change the font size of book details in the interface
# Change the font size at which book details are rendered in the side panel and
# comments are rendered in the metadata edit dialog. Set it to a positive or
# negative number to increase or decrease the font size.
change_book_details_font_size_by = 0

#: Compile General Program Mode templates to Python
# Compiled general program mode templates are significantly faster than
# interpreted templates. Setting this tweak to True causes calibre to compile
# (in most cases) general program mode templates. Setting it to False causes
# calibre to use the old behavior -- interpreting the templates. Set the tweak
# to False if some compiled templates produce incorrect values.
# Default: compile_gpm_templates = True
# No compile: compile_gpm_templates = False compile_gpm_templates = True

#: What format to default to when using the Tweak feature
# The Tweak feature of calibre allows direct editing of a book format.
# If multiple formats are available, calibre will offer you a choice
# of formats, defaulting to your preferred output format if it is available.
# Set this tweak to a specific value of 'EPUB' or 'AZW3' to always default
# to that format rather than your output format preference.
# Set to a value of 'remember' to use whichever format you chose last time you
# used the Tweak feature.
# Examples:
# default_tweak_format = None (Use output format)
# default_tweak_format = 'EPUB'
# default_tweak_format = 'remember'
default_tweak_format = None

#: Do not preselect a completion when editing authors/tags/series/etc.
# This means that you can make changes and press Enter and your changes will
# not be overwritten by a matching completion. However, if you wish to use the
# completions you will now have to press Tab to select one before pressing
# Enter. Which technique you prefer will depend on the state of metadata in
# your library and your personal editing style.
preselect_first_completion = False

Overriding icons, templates, et cetera
calibre allows you to override the static resources, like icons, templates, javascript, etc. with customized versions that you like.
All static resources are stored in the resources sub-folder of the calibre install location. On Windows, this is usually C:/Program Files/Calibre2/resources. On OS X, /Applications/calibre.app/Contents/Resources/resources/. On linux, if you are using the binary installer from the calibre website it will be /opt/calibre/resources. These paths can change depending on where you choose to install calibre.

You should not change the files in this resources folder, as your changes will get overwritten the next time you update calibre. Instead, go to Preferences->Advanced->Miscellaneous and click Open calibre configuration directory. In this configuration directory, create a sub-folder called resources and place the files you want to override in it. Place the files in the appropriate sub folders, for example place images in resources/images, etc.
calibre will automatically use your custom file in preference to the built-in one the next time it is started.

For example, if you wanted to change the icon for the Remove books action, you would first look in the built-in resources folder and see that the relevant file is resources/images/trash.png. Assuming you have an alternate icon in PNG format called mytrash.png you would save it in the configuration directory as resources/images/trash.png. All the icons used by the calibre user interface are in resources/images and its sub-folders.

Customizing calibre with plugins
calibre has a very modular design. Almost all functionality in calibre comes in the form of plugins. Plugins are used for conversion, for downloading news (though these are called recipes), for various components of the user interface, to connect to different devices, to process files when adding them to calibre and so on. You can get a complete list of all the built-in plugins in calibre by going to Preferences->Plugins.

You can write your own plugins to customize and extend the behavior of calibre. The plugin architecture in calibre is very simple, see the tutorial Writing your own plugins to extend calibre’s functionality.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

Command Line Interface

On OS X you have to go to Preferences->Advanced->Miscellaneous and click install command line tools to make the command line tools available. On other platforms, just start a terminal and type the command.

Documented Commands

- calibre
- calibre-customize
- calibre-debug
- calibre-server
- calibre-smtp
- calibredb
- ebook-convert
- ebook-meta
- ebook-viewer
- epub-fix
- fetch-ebook-metadata
- lrf2lrs
- lrfviewer
- lrs2lrf
- web2disk

Undocumented Commands

- ebook-device

- markdown-calibre

You can see usage for undocumented commands by executing them without arguments in a terminal.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

calibre

calibre [opts] [path_to_ebook]

Launch the main calibre Graphical User Interface and optionally add the ebook at path_to_ebook to the database.

Whenever you pass arguments to calibre that have spaces in them, enclose the arguments in quotation marks.

[options]

--help, -h

show this help message and exit

--ignore-plugins

Ignore custom plugins, useful if you installed a plugin that is preventing calibre from starting

--no-update-check

Do not check for updates

--shutdown-running-calibre, -s

Cause a running calibre instance, if any, to be shutdown. Note that if there are running jobs, they will be silently aborted, so use with care.

--start-in-tray

Start minimized to system tray.

--verbose, -v

Log debugging information to console

--version

show program’s version number and exit

--with-library

Use the library located at the specified path.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

calibre-customize

calibre-customize options

Customize calibre by loading external plugins.

Whenever you pass arguments to calibre-customize that have spaces in them, enclose the arguments in quotation marks.

[options]

--add-plugin, -a

Add a plugin by specifying the path to the zip file containing it.

--build-plugin, -b

For plugin developers: Path to the directory where you are developing the plugin. This command will automatically zip up the plugin and update it in calibre.

--customize-plugin

Customize plugin. Specify name of plugin and customization string separated by a comma.

--disable-plugin

Disable the named plugin

--enable-plugin

Enable the named plugin

--help, -h

show this help message and exit

--list-plugins, -l

List all installed plugins

--remove-plugin, -r

Remove a custom plugin by name. Has no effect on builtin plugins

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

calibre-debug

calibre-debug [options]

Various command line interfaces useful for debugging calibre. With no options, this command starts an embedded python interpreter. You can also run the main calibre GUI and the calibre viewer in debug mode.

It also contains interfaces to various bits of calibre that do not have dedicated command line tools, such as font subsetting, tweaking ebooks and so on.

Whenever you pass arguments to calibre-debug that have spaces in them, enclose the arguments in quotation marks.

[options]

--add-simple-plugin

Add a simple plugin (i.e. a plugin that consists of only a .py file), by specifying the path to the py file containing the plugin code.

--command, -c

Run python code.

--debug-device-driver, -d

Debug the specified device driver.

--exec-file, -e

Run the python code in file.

--gui, -g

Run the GUI with debugging enabled. Debug output is printed to stdout and stderr.

--gui-debug

Run the GUI with a debug console, logging to the specified path. For internal use only, use the -g option to run the GUI in debug mode

--help, -h

show this help message and exit

--inspect-mobi, -m

Inspect the MOBI file(s) at the specified path(s)

--paths

Output the paths necessary to setup the calibre environment

--py-console, -p

Run python console

--reinitialize-db

Re-initialize the sqlite calibre database at the specified path. Useful to recover from db corruption. You can also specify the path to an SQL dump which will be used instead of trying to dump the database. This can be useful when dumping fails, but dumping with sqlite3 works.

--show-gui-debug

Display the specified log file. For internal use only.

--shutdown-running-calibre, -s

Cause a running calibre instance, if any, to be shutdown. Note that if there are running jobs, they will be silently aborted, so use with care.

--subset-font, -f

Subset the specified font

--test-build

Test binary modules in build

--tweak-book

Tweak the book (exports the book as a collection of HTML files and metadata, which you can edit using standard HTML editing tools, and then rebuilds the file from the edited HTML. Makes no additional changes to the HTML, unlike a full calibre conversion).

--version

show program’s version number and exit

--viewer, -w

Run the ebook viewer

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

calibre-server

calibre-server [options]

Start the calibre content server. The calibre content server exposes your calibre library over the internet. The default interface allows you to browse you calibre library by categories. You can also access an interface optimized for mobile browsers at /mobile and an
OPDS based interface for use with reading applications at /opds.

The OPDS interface is advertised via BonJour automatically.

Whenever you pass arguments to calibre-server that have spaces in them, enclose the arguments in quotation marks.

[options]

--auto-reload

Auto reload server when source code changes. May not work in all environments.

--daemonize

Run process in background as a daemon. No effect on windows.

--develop

Development mode. Server automatically restarts on file changes and serves code files (html, css, js) from the file system instead of calibre’s resource system.

--help, -h

show this help message and exit

--max-cover

The maximum size for displayed covers. Default is ‘600x800’.

--max-opds-items

The maximum number of matches to return per OPDS query. This affects Stanza, WordPlayer, etc. integration.

--max-opds-ungrouped-items

Group items in categories such as author/tags by first letter when there are more than this number of items. Default: 100. Set to a large number to disable grouping.

--password

Set a password to restrict access. By default access is unrestricted.

--pidfile

Write process PID to the specified file

--port, -p

The port on which to listen. Default is 8080

--restriction

Specifies a restriction to be used for this invocation. This option overrides any per-library settings specified in the GUI

--thread-pool

The max number of worker threads to use. Default is 30

--timeout, -t

The server timeout in seconds. Default is 120

--url-prefix

Prefix to prepend to all URLs. Useful for reverseproxying to this server from Apache/nginx/etc.

--username

Username for access. By default, it is: ‘calibre’

--version

show program’s version number and exit

--with-library

Path to the library folder to serve with the content server

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

calibre-smtp

calibre-smtp [options] [from to text]

Send mail using the SMTP protocol. calibre-smtp has two modes of operation. In the compose mode you specify from to and text and these are used to build and send an email message. In the filter mode, calibre-smtp reads a complete email message from STDIN and sends it.

text is the body of the email message.
If text is not specified, a complete email message is read from STDIN.
from is the email address of the sender and to is the email address of the recipient. When a complete email is read from STDIN, from and to are only used in the SMTP negotiation, the message headers are not modified.

Whenever you pass arguments to calibre-smtp that have spaces in them, enclose the arguments in quotation marks.

[options]

--fork, -f

Fork and deliver message in background. If you use this option, you should also use –outbox to handle delivery failures.

--help, -h

show this help message and exit

--localhost, -l

Host name of localhost. Used when connecting to SMTP server.

--outbox, -o

Path to maildir folder to store failed email messages in.

--timeout, -t

Timeout for connection

--verbose, -v

Be more verbose

--version

show program’s version number and exit

COMPOSE MAIL

Options to compose an email. Ignored if text is not specified

--attachment, -a

File to attach to the email

--subject, -s

Subject of the email

SMTP RELAY

Options to use an SMTP relay server to send mail. calibre will try to send the email directly unless –relay is specified.

--encryption-method, -e

Encryption method to use when connecting to relay. Choices are TLS, SSL and NONE. Default is TLS. WARNING: Choosing NONE is highly insecure

--password, -p

Password for relay

--port

Port to connect to on relay server. Default is to use 465 if encryption method is SSL and 25 otherwise.

--relay, -r

An SMTP relay server to use to send mail.

--username, -u

Username for relay

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

calibredb

calibredb command [options] [arguments]

calibredb is the command line interface to the calibre database. It has several sub-commands, documented below:

- list
- add
- remove
- add_format
- remove_format
- show_metadata
- set_metadata
- export
- catalog
- saved_searches
- add_custom_column
- custom_columns
- remove_custom_column
- set_custom
- restore_database
- check_library
- list_categories
- backup_metadata

Global options

--dont-notify-gui

Do not notify the running calibre GUI (if any) that the database has changed. Use with care, as it can lead to database corruption!

--library-path

Path to the calibre library. Default is to use the path stored in the settings.

list

calibredb list [options]

List the books available in the calibre database.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--ascending

Sort results in ascending order

--fields, -f

The fields to display when listing books in the database. Should be a comma separated list of fields. Available fields: author_sort,authors,comments,cover,formats,identifiers,isbn,last_modified,pubdate,publisher,rating,series,series_index,size,tags,timestamp,title,uuid Default: title,authors. The special field “all” can be used to select all fields. Only has effect in the text output format.

--help, -h

show this help message and exit

--line-width, -w

The maximum width of a single line in the output. Defaults to detecting screen size.

--prefix

The prefix for all file paths. Default is the absolute path to the library folder.

--search, -s

Filter the results by the search query. For the format of the search query, please see the search related documentation in the User Manual. Default is to do no filtering.

--separator

The string used to separate fields. Default is a space.

--sort-by

The field by which to sort the results. Available fields: publisher,series_index,formats,isbn,uuid,pubdate,rating,series,timestamp,author_sort,cover,comments,identifiers,last_modified,authors,title,tags,size Default: None

--version

show program’s version number and exit

add

calibredb add [options] file1 file2 file3 ...

Add the specified files as books to the database. You can also specify directories, see the directory related options below.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--authors, -a

Set the authors of the added book(s)

--cover, -c

Path to the cover to use for the added book

--duplicates, -d

Add books to database even if they already exist. Comparison is done based on book titles.

--empty, -e

Add an empty book (a book with no formats)

--help, -h

show this help message and exit

--isbn, -i

Set the ISBN of the added book(s)

--one-book-per-directory, -1

Assume that each directory has only a single logical book and that all files in it are different e-book formats of that book

--recurse, -r

Process directories recursively

--series, -s

Set the series of the added book(s)

--series-index, -S

Set the series number of the added book(s)

--tags, -T

Set the tags of the added book(s)

--title, -t

Set the title of the added book(s)

--version

show program’s version number and exit

remove

calibredb remove ids

Remove the books identified by ids from the database. ids should be a comma separated list of id numbers (you can get id numbers by using the list command). For example, 23,34,57-85 (when specifying a range, the last number in the range is not included).

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--help, -h

show this help message and exit

--version

show program’s version number and exit

add_format

calibredb add_format [options] id ebook_file

Add the ebook in ebook_file to the available formats for the logical book identified by id. You can get id by using the list command. If the format already exists, it is replaced.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--help, -h

show this help message and exit

--version

show program’s version number and exit

remove_format

calibredb remove_format [options] id fmt

Remove the format fmt from the logical book identified by id. You can get id by using the list command. fmt should be a file extension like LRF or TXT or EPUB. If the logical book does not have fmt available, do nothing.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--help, -h

show this help message and exit

--version

show program’s version number and exit

show_metadata

calibredb show_metadata [options] id

Show the metadata stored in the calibre database for the book identified by id.
id is an id number from the list command.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--as-opf

Print metadata in OPF form (XML)

--help, -h

show this help message and exit

--version

show program’s version number and exit

set_metadata

calibredb set_metadata [options] id /path/to/metadata.opf

Set the metadata stored in the calibre database for the book identified by id from the OPF file metadata.opf. id is an id number from the list command. You can get a quick feel for the OPF format by using the –as-opf switch to the show_metadata command. You can also set the metadata of individual fields with the –field option.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--help, -h

show this help message and exit

--version

show program’s version number and exit

export

calibredb export [options] ids

Export the books specified by ids (a comma separated list) to the filesystem.
The export operation saves all formats of the book, its cover and metadata (in an opf file). You can get id numbers from the list command.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--all

Export all books in database, ignoring the list of ids.

--dont-asciiize

Normally, calibre will convert all non English characters into English equivalents for the file names. WARNING: If you turn this off, you may experience errors when saving, depending on how well the filesystem you are saving to supports unicode. Specifying this switch will turn this behavior off.

--dont-save-cover

Normally, calibre will save the cover in a separate file along with the actual e-book file(s). Specifying this switch will turn this behavior off.

--dont-update-metadata

Normally, calibre will update the metadata in the saved files from what is in the calibre library. Makes saving to disk slower. Specifying this switch will turn this behavior off.

--dont-write-opf

Normally, calibre will write the metadata into a separate OPF file along with the actual e-book files. Specifying this switch will turn this behavior off.

--formats

Comma separated list of formats to save for each book. By default all available formats are saved.

--help, -h

show this help message and exit

--replace-whitespace

Replace whitespace with underscores.

--single-dir

Export all books into a single directory

--template

The template to control the filename and directory structure of the saved files. Default is “{author_sort}/{title}/{title} - {authors}” which will save books into a per-author subdirectory with filenames containing title and author. Available controls are: {publisher, series_index, isbn, pubdate, rating, series, author_sort, authors, last_modified, timestamp, title, id, tags}

--timefmt

The format in which to display dates. %d - day, %b - month, %m - month number, %Y - year. Default is: %b, %Y

--to-dir

Export books to the specified directory. Default is .

--to-lowercase

Convert paths to lowercase.

--version

show program’s version number and exit

catalog

calibredb catalog /path/to/destination.(CSV|EPUB|MOBI|XML ...) [options]

Export a catalog in format specified by path/to/destination extension.
Options control how entries are displayed in the generated catalog ouput.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--fields

The fields to output when cataloging books in the database. Should be a comma-separated list of fields. Available fields: all, title, title_sort, author_sort, authors, comments, cover, formats, id, isbn, library_name, ondevice, pubdate, publisher, rating, series_index, series, size, tags, timestamp, uuid, languages, identifiers, plus user-created custom fields. Example: –fields=title,authors,tags Default: ‘all’ Applies to: CSV, XML output formats

--help, -h

show this help message and exit

--ids, -i

Comma-separated list of database IDs to catalog. If declared, –search is ignored. Default: all

--search, -s

Filter the results by the search query. For the format of the search query, please see the search-related documentation in the User Manual. Default: no filtering

--sort-by

Output field to sort on. Available fields: author_sort, id, rating, size, timestamp, title_sort Default: ‘id’ Applies to: CSV, XML output formats

--verbose, -v

Show detailed output information. Useful for debugging

--version

show program’s version number and exit

saved_searches

calibredb saved_searches [options] list

calibredb saved_searches add name search calibredb saved_searches remove name

Manage the saved searches stored in this database.
If you try to add a query with a name that already exists, it will be replaced.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--help, -h

show this help message and exit

--version

show program’s version number and exit

add_custom_column

calibredb add_custom_column [options] label name datatype

Create a custom column. label is the machine friendly name of the column. Should not contain spaces or colons. name is the human friendly name of the column.
datatype is one of: rating, int, text, float, comments, datetime, composite, bool, enumeration, series

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--display

A dictionary of options to customize how the data in this column will be interpreted. This is a JSON string. For enumeration columns, use –display=”{“enum_values”:[“val1”, “val2”]}” There are many options that can go into the display variable.The options by column type are: composite: composite_template, composite_sort, make_category,contains_html, use_decorations datetime: date_format enumeration: enum_values, enum_colors, use_decorations int, float: number_format text: is_names, use_decorations The best way to find legal combinations is to create a customcolumn of the appropriate type in the GUI then look at thebackup OPF for a book (ensure that a new OPF has been createdsince the column was added). You will see the JSON for the”display” for the new column in the OPF.

--help, -h

show this help message and exit

--is-multiple

This column stores tag like data (i.e. multiple comma separated values). Only applies if datatype is text.

--version

show program’s version number and exit

custom_columns

calibredb custom_columns [options]

List available custom columns. Shows column labels and ids.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--details, -d

Show details for each column.

--help, -h

show this help message and exit

--version

show program’s version number and exit

remove_custom_column

calibredb remove_custom_column [options] label

Remove the custom column identified by label. You can see available columns with the custom_columns command.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--force, -f

Do not ask for confirmation

--help, -h

show this help message and exit

--version

show program’s version number and exit

set_custom

calibredb set_custom [options] column id value

Set the value of a custom column for the book identified by id.
You can get a list of ids using the list command.
You can get a list of custom column names using the custom_columns command.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--append, -a

If the column stores multiple values, append the specified values to the existing ones, instead of replacing them.

--help, -h

show this help message and exit

--version

show program’s version number and exit

restore_database

calibredb restore_database [options]

Restore this database from the metadata stored in OPF files in each directory of the calibre library. This is useful if your metadata.db file has been corrupted.

WARNING: This command completely regenerates your database. You will lose all saved searches, user categories, plugboards, stored per-book conversion settings, and custom recipes. Restored metadata will only be as accurate as what is found in the OPF files.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--help, -h

show this help message and exit

--really-do-it, -r

Really do the recovery. The command will not run unless this option is specified.

--version

show program’s version number and exit

check_library

calibredb check_library [options]

Perform some checks on the filesystem representing a library. Reports are invalid_titles, extra_titles, invalid_authors, extra_authors, missing_formats, extra_formats, extra_files, missing_covers, extra_covers, failed_folders

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--csv, -c

Output in CSV

--help, -h

show this help message and exit

--ignore_extensions, -e

Comma-separated list of extensions to ignore. Default: all

--ignore_names, -n

Comma-separated list of names to ignore. Default: all

--report, -r

Comma-separated list of reports. Default: all

--version

show program’s version number and exit

list_categories

calibredb list_categories [options]

Produce a report of the category information in the database. The information is the equivalent of what is shown in the tags pane.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--categories, -r

Comma-separated list of category lookup names. Default: all

--csv, -c

Output in CSV

--help, -h

show this help message and exit

--item_count, -i

Output only the number of items in a category instead of the counts per item within the category

--quote, -q

The character to put around the category value in CSV mode. Default is quotes (”).

--separator, -s

The string used to separate fields in CSV mode. Default is a comma.

--version

show program’s version number and exit

--width, -w

The maximum width of a single line in the output. Defaults to detecting screen size.

backup_metadata

calibredb backup_metadata [options]

Backup the metadata stored in the database into individual OPF files in each books directory. This normally happens automatically, but you can run this command to force re-generation of the OPF files, with the –all option.

Note that there is normally no need to do this, as the OPF files are backed up automatically, every time metadata is changed.

Whenever you pass arguments to calibredb that have spaces in them, enclose the arguments in quotation marks.

--all

Normally, this command only operates on books that have out of date OPF files. This option makes it operate on all books.

--help, -h

show this help message and exit

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

ebook-convert

ebook-convert input_file output_file [options]

Convert an ebook from one format to another.

input_file is the input and output_file is the output. Both must be specified as the first two arguments to the command.

The output ebook format is guessed from the file extension of output_file. output_file can also be of the special format .EXT where EXT is the output file extension. In this case, the name of the output file is derived the name of the input file. Note that the filenames must not start with a hyphen. Finally, if output_file has no extension, then it is treated as a directory and an “open ebook” (OEB) consisting of HTML files is written to that directory. These files are the files that would normally have been passed to the output plugin.

After specifying the input and output file you can customize the conversion by specifying various options. The available options depend on the input and output file types. To get help on them specify the input and output file and then use the -h option.

For full documentation of the conversion system see Ebook Conversion

Whenever you pass arguments to ebook-convert that have spaces in them, enclose the arguments in quotation marks.

The options and default values for the options change depending on both the input and output formats, so you should always check with:

ebook-convert myfile.input_format myfile.output_format -h

Below are the options that are common to all conversion, followed by the options specific to every input and output format

- Look And Feel
- Heuristic Processing
- Search And Replace
- Structure Detection
- Table Of Contents
- Metadata
- Debug
- AZW4 Input Options
- CHM Input Options
- Comic Input Options
- DJVU Input Options
- EPUB Input Options
- FB2 Input Options
- HTLZ Input Options
- HTML Input Options
- LIT Input Options
- LRF Input Options
- MOBI Input Options
- ODT Input Options
- PDB Input Options
- PDF Input Options
- PML Input Options
- RB Input Options
- RTF Input Options
- Recipe Input Options
- SNB Input Options
- TCR Input Options
- TXT Input Options
- AZW3 Output Options
- EPUB Output Options
- FB2 Output Options
- HTML Output Options
- HTMLZ Output Options
- LIT Output Options
- LRF Output Options
- MOBI Output Options
- OEB Output Options
- PDB Output Options
- PDF Output Options
- PML Output Options
- RB Output Options
- RTF Output Options
- SNB Output Options
- TCR Output Options
- TXT Output Options
- TXTZ Output Options

--help, -h

show this help message and exit

--input-profile

Specify the input profile. The input profile gives the conversion system information on how to interpret various information in the input document. For example resolution dependent lengths (i.e. lengths in pixels). Choices are:cybookg3, cybook_opus, default, hanlinv3, hanlinv5, illiad, irexdr1000, irexdr800, kindle, msreader, mobipocket, nook, sony, sony300, sony900

--list-recipes

List builtin recipe names. You can create an ebook from a builtin recipe like this: ebook-convert “Recipe Name.recipe” output.epub

--output-profile

Specify the output profile. The output profile tells the conversion system how to optimize the created document for the specified device. In some cases, an output profile is required to produce documents that will work on a device. For example EPUB on the SONY reader. Choices are:cybookg3, cybook_opus, default, generic_eink, generic_eink_large, hanlinv3, hanlinv5, illiad, ipad, ipad3, irexdr1000, irexdr800, jetbook5, kindle, kindle_dx, kindle_fire, kindle_pw, kobo, msreader, mobipocket, nook, nook_color, pocketbook_900, galaxy, bambook, sony, sony300, sony900, sony-landscape, tablet

--version

show program’s version number and exit

Look And Feel

Options to control the look and feel of the output

--asciiize

Transliterate unicode characters to an ASCII representation. Use with care because this will replace unicode characters with ASCII. For instance it will replace “Михаил Горбачёв” with “Mikhail Gorbachiov”. Also, note that in cases where there are multiple representations of a character (characters shared by Chinese and Japanese for instance) the representation based on the current calibre interface language will be used.

--base-font-size

The base font size in pts. All font sizes in the produced book will be rescaled based on this size. By choosing a larger size you can make the fonts in the output bigger and vice versa. By default, the base font size is chosen based on the output profile you chose.

--change-justification

Change text justification. A value of “left” converts all justified text in the source to left aligned (i.e. unjustified) text. A value of “justify” converts all unjustified text to justified. A value of “original” (the default) does not change justification in the source file. Note that only some output formats support justification.

--disable-font-rescaling

Disable all rescaling of font sizes.

--embed-font-family

Embed the specified font family into the book. This specifies the “base” font used for the book. If the input document specifies its own fonts, they may override this base font. You can use the filter style information option to remove fonts from the input document. Note that font embedding only works with some output formats, principally EPUB and AZW3.

--extra-css

Either the path to a CSS stylesheet or raw CSS. This CSS will be appended to the style rules from the source file, so it can be used to override those rules.

--filter-css

A comma separated list of CSS properties that will be removed from all CSS style rules. This is useful if the presence of some style information prevents it from being overridden on your device. For example: font-family,color,margin-left,margin-right

--font-size-mapping

Mapping from CSS font names to font sizes in pts. An example setting is 12,12,14,16,18,20,22,24. These are the mappings for the sizes xx-small to xx-large, with the final size being for huge fonts. The font rescaling algorithm uses these sizes to intelligently rescale fonts. The default is to use a mapping based on the output profile you chose.

--insert-blank-line

Insert a blank line between paragraphs. Will not work if the source file does not use paragraphs (<p> or <div> tags).

--insert-blank-line-size

Set the height of the inserted blank lines (in em). The height of the lines between paragraphs will be twice the value set here.

--keep-ligatures

Preserve ligatures present in the input document. A ligature is a special rendering of a pair of characters like ff, fi, fl et cetera. Most readers do not have support for ligatures in their default fonts, so they are unlikely to render correctly. By default, calibre will turn a ligature into the corresponding pair of normal characters. This option will preserve them instead.

--line-height

The line height in pts. Controls spacing between consecutive lines of text. Only applies to elements that do not define their own line height. In most cases, the minimum line height option is more useful. By default no line height manipulation is performed.

--linearize-tables

Some badly designed documents use tables to control the layout of text on the page. When converted these documents often have text that runs off the page and other artifacts. This option will extract the content from the tables and present it in a linear fashion.

--margin-bottom

Set the bottom margin in pts. Default is 5.0. Setting this to less than zero will cause no margin to be set. Note: 72 pts equals 1 inch

--margin-left

Set the left margin in pts. Default is 5.0. Setting this to less than zero will cause no margin to be set. Note: 72 pts equals 1 inch

--margin-right

Set the right margin in pts. Default is 5.0. Setting this to less than zero will cause no margin to be set. Note: 72 pts equals 1 inch

--margin-top

Set the top margin in pts. Default is 5.0. Setting this to less than zero will cause no margin to be set. Note: 72 pts equals 1 inch

--minimum-line-height

The minimum line height, as a percentage of the element’s calculated font size. calibre will ensure that every element has a line height of at least this setting, irrespective of what the input document specifies. Set to zero to disable. Default is 120%. Use this setting in preference to the direct line height specification, unless you know what you are doing. For example, you can achieve “double spaced” text by setting this to 240.

--remove-paragraph-spacing

Remove spacing between paragraphs. Also sets an indent on paragraphs of 1.5em. Spacing removal will not work if the source file does not use paragraphs (<p> or <div> tags).

--remove-paragraph-spacing-indent-size

When calibre removes blank lines between paragraphs, it automatically sets a paragraph indent, to ensure that paragraphs can be easily distinguished. This option controls the width of that indent (in em). If you set this value negative, then the indent specified in the input document is used, that is, calibre does not change the indentation.

--smarten-punctuation

Convert plain quotes, dashes and ellipsis to their typographically correct equivalents. For details, see http://daringfireball.net/projects/smartypants

--subset-embedded-fonts

Subset all embedded fonts. Every embedded font is reduced to contain only the glyphs used in this document. This decreases the size of the font files. Useful if you are embedding a particularly large font with lots of unused glyphs.

--unsmarten-punctuation

Convert fancy quotes, dashes and ellipsis to their plain equivalents.

Heuristic Processing

Modify the document text and structure using common patterns. Disabled by default. Use –enable-heuristics to enable. Individual actions can be disabled with the –disable-* options.

--disable-dehyphenate

Analyze hyphenated words throughout the document. The document itself is used as a dictionary to determine whether hyphens should be retained or removed.

--disable-delete-blank-paragraphs

Remove empty paragraphs from the document when they exist between every other paragraph

--disable-fix-indents

Turn indentation created from multiple non-breaking space entities into CSS indents.

--disable-format-scene-breaks

Left aligned scene break markers are center aligned. Replace soft scene breaks that use multiple blank lines with horizontal rules.

--disable-italicize-common-cases

Look for common words and patterns that denote italics and italicize them.

--disable-markup-chapter-headings

Detect unformatted chapter headings and sub headings. Change them to h2 and h3 tags. This setting will not create a TOC, but can be used in conjunction with structure detection to create one.

--disable-renumber-headings

Looks for occurrences of sequential <h1> or <h2> tags. The tags are renumbered to prevent splitting in the middle of chapter headings.

--disable-unwrap-lines

Unwrap lines using punctuation and other formatting clues.

--enable-heuristics

Enable heuristic processing. This option must be set for any heuristic processing to take place.

--html-unwrap-factor

Scale used to determine the length at which a line should be unwrapped. Valid values are a decimal between 0 and 1. The default is 0.4, just below the median line length. If only a few lines in the document require unwrapping this value should be reduced

--replace-scene-breaks

Replace scene breaks with the specified text. By default, the text from the input document is used.

Search And Replace

Modify the document text and structure using user defined patterns.

--search-replace

Path to a file containing search and replace regular expressions. The file must contain alternating lines of regular expression followed by replacement pattern (which can be an empty line). The regular expression must be in the python regex syntax and the file must be UTF-8 encoded.

--sr1-replace

Replacement to replace the text found with sr1-search.

--sr1-search

Search pattern (regular expression) to be replaced with sr1-replace.

--sr2-replace

Replacement to replace the text found with sr2-search.

--sr2-search

Search pattern (regular expression) to be replaced with sr2-replace.

--sr3-replace

Replacement to replace the text found with sr3-search.

--sr3-search

Search pattern (regular expression) to be replaced with sr3-replace.

Structure Detection

Control auto-detection of document structure.

--chapter

An XPath expression to detect chapter titles. The default is to consider <h1> or <h2> tags that contain the words “chapter”,”book”,”section”, “prologue”, “epilogue”, or “part” as chapter titles as well as any tags that have class=”chapter”. The expression used must evaluate to a list of elements. To disable chapter detection, use the expression “/”. See the XPath Tutorial in the calibre User Manual for further help on using this feature.

--chapter-mark

Specify how to mark detected chapters. A value of “pagebreak” will insert page breaks before chapters. A value of “rule” will insert a line before chapters. A value of “none” will disable chapter marking and a value of “both” will use both page breaks and lines to mark chapters.

--disable-remove-fake-margins

Some documents specify page margins by specifying a left and right margin on each individual paragraph. calibre will try to detect and remove these margins. Sometimes, this can cause the removal of margins that should not have been removed. In this case you can disable the removal.

--insert-metadata

Insert the book metadata at the start of the book. This is useful if your ebook reader does not support displaying/searching metadata directly.

--page-breaks-before

An XPath expression. Page breaks are inserted before the specified elements. To disable use the expression: /

--prefer-metadata-cover

Use the cover detected from the source file in preference to the specified cover.

--remove-first-image

Remove the first image from the input ebook. Useful if the input document has a cover image that is not identified as a cover. In this case, if you set a cover in calibre, the output document will end up with two cover images if you do not specify this option.

--start-reading-at

An XPath expression to detect the location in the document at which to start reading. Some ebook reading programs (most prominently the Kindle) use this location as the position at which to open the book. See the XPath tutorial in the calibre User Manual for further help using this feature.

Table Of Contents

Control the automatic generation of a Table of Contents. By default, if the source file has a Table of Contents, it will be used in preference to the automatically generated one.

--duplicate-links-in-toc

When creating a TOC from links in the input document, allow duplicate entries, i.e. allow more than one entry with the same text, provided that they point to a different location.

--level1-toc

XPath expression that specifies all tags that should be added to the Table of Contents at level one. If this is specified, it takes precedence over other forms of auto-detection. See the XPath Tutorial in the calibre User Manual for examples.

--level2-toc

XPath expression that specifies all tags that should be added to the Table of Contents at level two. Each entry is added under the previous level one entry. See the XPath Tutorial in the calibre User Manual for examples.

--level3-toc

XPath expression that specifies all tags that should be added to the Table of Contents at level three. Each entry is added under the previous level two entry. See the XPath Tutorial in the calibre User Manual for examples.

--max-toc-links

Maximum number of links to insert into the TOC. Set to 0 to disable. Default is: 50. Links are only added to the TOC if less than the threshold number of chapters were detected.

--no-chapters-in-toc

Don’t add auto-detected chapters to the Table of Contents.

--toc-filter

Remove entries from the Table of Contents whose titles match the specified regular expression. Matching entries and all their children are removed.

--toc-threshold

If fewer than this number of chapters is detected, then links are added to the Table of Contents. Default: 6

--use-auto-toc

Normally, if the source file already has a Table of Contents, it is used in preference to the auto-generated one. With this option, the auto-generated one is always used.

Metadata

Options to set metadata in the output

--author-sort

String to be used when sorting by author.

--authors

Set the authors. Multiple authors should be separated by ampersands.

--book-producer

Set the book producer.

--comments

Set the ebook description.

--cover

Set the cover to the specified file or URL

--isbn

Set the ISBN of the book.

--language

Set the language.

--pubdate

Set the publication date.

--publisher

Set the ebook publisher.

--rating

Set the rating. Should be a number between 1 and 5.

--series

Set the series this ebook belongs to.

--series-index

Set the index of the book in this series.

--tags

Set the tags for the book. Should be a comma separated list.

--timestamp

Set the book timestamp (no longer used anywhere)

--title

Set the title.

--title-sort

The version of the title to be used for sorting.

Debug

Options to help with debugging the conversion

--debug-pipeline, -d

Save the output from different stages of the conversion pipeline to the specified directory. Useful if you are unsure at which stage of the conversion process a bug is occurring.

--verbose, -v

Level of verbosity. Specify multiple times for greater verbosity.

AZW4 Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

CHM Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

Comic Input Options

--colors

Number of colors for grayscale image conversion. Default: 256. Values of less than 256 may result in blurred text on your device if you are creating your comics in EPUB format.

--comic-image-size

Specify the image size as widthxheight pixels. Normally, an image size is automatically calculated from the output profile, this option overrides it.

--despeckle

Enable Despeckle. Reduces speckle noise. May greatly increase processing time.

--disable-trim

Disable trimming of comic pages. For some comics, trimming might remove content as well as borders.

--dont-add-comic-pages-to-toc

When converting a CBC do not add links to each page to the TOC. Note this only applies if the TOC has more than one section

--dont-grayscale

Do not convert the image to grayscale (black and white)

--dont-normalize

Disable normalize (improve contrast) color range for pictures. Default: False

--dont-sharpen

Disable sharpening.

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

--keep-aspect-ratio

Maintain picture aspect ratio. Default is to fill the screen.

--landscape

Don’t split landscape images into two portrait images

--no-process

Apply no processing to the image

--no-sort

Don’t sort the files found in the comic alphabetically by name. Instead use the order they were added to the comic.

--output-format

The format that images in the created ebook are converted to. You can experiment to see which format gives you optimal size and look on your device.

--right2left

Used for right-to-left publications like manga. Causes landscape pages to be split into portrait pages from right to left.

--wide

Keep aspect ratio and scale image using screen height as image width for viewing in landscape mode.

DJVU Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

--use-djvutxt

Try to use the djvutxt program and fall back to pure python implementation if it fails or is not available

EPUB Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

FB2 Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

--no-inline-fb2-toc

Do not insert a Table of Contents at the beginning of the book.

HTLZ Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

HTML Input Options

--breadth-first

Traverse links in HTML files breadth first. Normally, they are traversed depth first.

--dont-package

Normally this input plugin re-arranges all the input files into a standard folder hierarchy. Only use this option if you know what you are doing as it can result in various nasty side effects in the rest of the conversion pipeline.

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

--max-levels

Maximum levels of recursion when following links in HTML files. Must be non-negative. 0 implies that no links in the root HTML file are followed. Default is 5.

LIT Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

LRF Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

MOBI Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

ODT Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

PDB Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

PDF Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

--new-pdf-engine

Use the new PDF conversion engine.

--no-images

Do not extract images from the document

--unwrap-factor

Scale used to determine the length at which a line should be unwrapped. Valid values are a decimal between 0 and 1. The default is 0.45, just below the median line length.

PML Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

RB Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

RTF Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

Recipe Input Options

--dont-download-recipe

Do not download latest version of builtin recipes from the calibre server

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

--lrf

Optimize fetching for subsequent conversion to LRF.

--password

Password for sites that require a login to access content.

--test

Useful for recipe development. Forces max_articles_per_feed to 2 and downloads at most 2 feeds.

--username

Username for sites that require a login to access content.

SNB Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

TCR Input Options

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

TXT Input Options

--formatting-type

Formatting used within the document.* auto: Automatically decide which formatting processor to use. * plain: Do not process the document formatting. Everything is a paragraph and no styling is applied. * heuristic: Process using heuristics to determine formatting such as chapter headings and italic text. * textile: Processing using textile formatting. * markdown: Processing using markdown formatting. To learn more about markdown see http://daringfireball.net/projects/markdown/

--input-encoding

Specify the character encoding of the input document. If set this option will override any encoding declared by the document itself. Particularly useful for documents that do not declare an encoding or that have erroneous encoding declarations.

--markdown-disable-toc

Do not insert a Table of Contents into the output text.

--paragraph-type

Paragraph structure. choices are [‘auto’, ‘block’, ‘single’, ‘print’, ‘unformatted’, ‘off’] * auto: Try to auto detect paragraph type. * block: Treat a blank line as a paragraph break. * single: Assume every line is a paragraph. * print: Assume every line starting with 2+ spaces or a tab starts a paragraph. * unformatted: Most lines have hard line breaks, few/no blank lines or indents. Tries to determine structure and reformat the differentiate elements. * off: Don’t modify the paragraph structure. This is useful when combined with Markdown or Textile formatting to ensure no formatting is lost.

--preserve-spaces

Normally extra spaces are condensed into a single space. With this option all spaces will be displayed.

--txt-in-remove-indents

Normally extra space at the beginning of lines is retained. With this option they will be removed.

AZW3 Output Options

--dont-compress

Disable compression of the file contents.

--extract-to

Extract the contents of the MOBI file to the specified directory. If the directory already exists, it will be deleted.

--mobi-toc-at-start

When adding the Table of Contents to the book, add it at the start of the book instead of the end. Not recommended.

--no-inline-toc

Don’t add Table of Contents to the book. Useful if the book has its own table of contents.

--prefer-author-sort

When present, use author sort field as author.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--share-not-sync

Enable sharing of book content via Facebook etc. on the Kindle. WARNING: Using this feature means that the book will not auto sync its last read position on multiple devices. Complain to Amazon.

--toc-title

Title for any generated in-line table of contents.

EPUB Output Options

--dont-split-on-page-breaks

Turn off splitting at page breaks. Normally, input files are automatically split at every page break into two files. This gives an output ebook that can be parsed faster and with less resources. However, splitting is slow and if your source file contains a very large number of page breaks, you should turn off splitting on page breaks.

--epub-flatten

This option is needed only if you intend to use the EPUB with FBReaderJ. It will flatten the file system inside the EPUB, putting all files into the top level.

--extract-to

Extract the contents of the generated EPUB file to the specified directory. The contents of the directory are first deleted, so be careful.

--flow-size

Split all HTML files larger than this size (in KB). This is necessary as most EPUB readers cannot handle large file sizes. The default of 260KB is the size required for Adobe Digital Editions.

--no-default-epub-cover

Normally, if the input file has no cover and you don’t specify one, a default cover is generated with the title, authors, etc. This option disables the generation of this cover.

--no-svg-cover

Do not use SVG for the book cover. Use this option if your EPUB is going to be used on a device that does not support SVG, like the iPhone or the JetBook Lite. Without this option, such devices will display the cover as a blank page.

--preserve-cover-aspect-ratio

When using an SVG cover, this option will cause the cover to scale to cover the available screen area, but still preserve its aspect ratio (ratio of width to height). That means there may be white borders at the sides or top and bottom of the image, but the image will never be distorted. Without this option the image may be slightly distorted, but there will be no borders.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

FB2 Output Options

--fb2-genre

Genre for the book. Choices: sf_history, sf_action, sf_epic, sf_heroic, sf_detective, sf_cyberpunk, sf_space, sf_social, sf_horror, sf_humor, sf_fantasy, sf, det_classic, det_police, det_action, det_irony, det_history, det_espionage, det_crime, det_political, det_maniac, det_hard, thriller, detective, prose_classic, prose_history, prose_contemporary, prose_counter, prose_rus_classic, prose_su_classics, love_contemporary, love_history, love_detective, love_short, love_erotica, adv_western, adv_history, adv_indian, adv_maritime, adv_geo, adv_animal, adventure, child_tale, child_verse, child_prose, child_sf, child_det, child_adv, child_education, children, poetry, dramaturgy, antique_ant, antique_european, antique_russian, antique_east, antique_myths, antique, sci_history, sci_psychology, sci_culture, sci_religion, sci_philosophy, sci_politics, sci_business, sci_juris, sci_linguistic, sci_medicine, sci_phys, sci_math, sci_chem, sci_biology, sci_tech, science, comp_www, comp_programming, comp_hard, comp_soft, comp_db, comp_osnet, computers, ref_encyc, ref_dict, ref_ref, ref_guide, reference, nonf_biography, nonf_publicism, nonf_criticism, design, nonfiction, religion_rel, religion_esoterics, religion_self, religion, humor_anecdote, humor_prose, humor_verse, humor, home_cooking, home_pets, home_crafts, home_entertain, home_health, home_garden, home_diy, home_sport, home_sex, home See: http://www.fictionbook.org/index.php/Eng:FictionBook_2.1_genres for a complete list with descriptions.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--sectionize

Specify the sectionization of elements. A value of “nothing” turns the book into a single section. A value of “files” turns each file into a separate section; use this if your device is having trouble. A value of “Table of Contents” turns the entries in the Table of Contents into titles and creates sections; if it fails, adjust the “Structure Detection” and/or “Table of Contents” settings (turn on “Force use of auto-generated Table of Contents”).

HTML Output Options

--extract-to

Extract the contents of the generated ZIP file to the specified directory. WARNING: The contents of the directory will be deleted.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--template-css

CSS file used for the output instead of the default file

--template-html

Template used for the generation of the html contents of the book instead of the default file

--template-html-index

Template used for generation of the html index file instead of the default file

HTMLZ Output Options

--htmlz-class-style

How to handle the CSS when using css-type = ‘class’. Default is external. external: Use an external CSS file that is linked in the document. inline: Place the CSS in the head section of the document.

--htmlz-css-type

Specify the handling of CSS. Default is class. class: Use CSS classes and have elements reference them. inline: Write the CSS as an inline style attribute. tag: Turn as many CSS styles as possible into HTML tags.

--htmlz-title-filename

If set this option causes the file name of the html file inside the htmlz archive to be based on the book title.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

LIT Output Options

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

LRF Output Options

--enable-autorotation

Enable autorotation of images that are wider than the screen width.

--header

Add a header to all the pages with title and author.

--header-format

Set the format of the header. %a is replaced by the author and %t by the title. Default is %t by %a

--header-separation

Add extra spacing below the header. Default is 0 pt.

--minimum-indent

Minimum paragraph indent (the indent of the first line of a paragraph) in pts. Default: 0

--mono-family

The monospace family of fonts to embed

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--render-tables-as-images

Render tables in the HTML as images (useful if the document has large or complex tables)

--sans-family

The sans-serif family of fonts to embed

--serif-family

The serif family of fonts to embed

--text-size-multiplier-for-rendered-tables

Multiply the size of text in rendered tables by this factor. Default is 1.0

--wordspace

Set the space between words in pts. Default is 2.5

MOBI Output Options

--dont-compress

Disable compression of the file contents.

--extract-to

Extract the contents of the MOBI file to the specified directory. If the directory already exists, it will be deleted.

--mobi-file-type

By default calibre generates MOBI files that contain the old MOBI 6 format. This format is compatible with all devices. However, by changing this setting, you can tell calibre to generate MOBI files that contain both MOBI 6 and the new KF8 format, or only the new KF8 format. KF8 has more features than MOBI 6, but only works with newer Kindles.

--mobi-ignore-margins

Ignore margins in the input document. If False, then the MOBI output plugin will try to convert margins specified in the input document, otherwise it will ignore them.

--mobi-keep-original-images

By default calibre converts all images to JPEG format in the output MOBI file. This is for maximum compatibility as some older MOBI viewers have problems with other image formats. This option tells calibre not to do this. Useful if your document contains lots of GIF/PNG images that become very large when converted to JPEG.

--mobi-toc-at-start

When adding the Table of Contents to the book, add it at the start of the book instead of the end. Not recommended.

--no-inline-toc

Don’t add Table of Contents to the book. Useful if the book has its own table of contents.

--personal-doc

Tag marking book to be filed with Personal Docs

--prefer-author-sort

When present, use author sort field as author.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--share-not-sync

Enable sharing of book content via Facebook etc. on the Kindle. WARNING: Using this feature means that the book will not auto sync its last read position on multiple devices. Complain to Amazon.

--toc-title

Title for any generated in-line table of contents.

OEB Output Options

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

PDB Output Options

--format, -f

Format to use inside the pdb container. Choices are: [‘doc’, ‘ztxt’, ‘ereader’]

--inline-toc

Add Table of Contents to beginning of the book.

--pdb-output-encoding

Specify the character encoding of the output document. The default is cp1252. Note: This option is not honored by all formats.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

PDF Output Options

--custom-size

Custom size of the document. Use the form widthxheight EG. 123x321 to specify the width and height. This overrides any specified paper-size.

--old-pdf-engine

Use the old, less capable engine to generate the PDF

--override-profile-size

Normally, the PDF page size is set by the output profile chosen under page options. This option will cause the page size settings under PDF Output to override the size specified by the output profile.

--paper-size

The size of the paper. This size will be overridden when a non default output profile is used. Default is letter. Choices are [u’a0’, u’a1’, u’a2’, u’a3’, u’a4’, u’a5’, u’a6’, u’b0’, u’b1’, u’b2’, u’b3’, u’b4’, u’b5’, u’b6’, u’legal’, u’letter’]

--pdf-default-font-size

The default font size

--pdf-mark-links

Surround all links with a red box, useful for debugging.

--pdf-mono-family

The font family used to render monospaced fonts

--pdf-mono-font-size

The default font size for monospaced text

--pdf-sans-family

The font family used to render sans-serif fonts

--pdf-serif-family

The font family used to render serif fonts

--pdf-standard-font

The font family used to render monospaced fonts

--preserve-cover-aspect-ratio

Preserve the aspect ratio of the cover, instead of stretching it to fill the full first page of the generated pdf.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--uncompressed-pdf

Generate an uncompressed PDF, useful for debugging, only works with the new PDF engine.

--unit, -u

The unit of measure for page sizes. Default is inch. Choices are [‘millimeter’, ‘centimeter’, ‘point’, ‘inch’, ‘pica’, ‘didot’, ‘cicero’, ‘devicepixel’] Note: This does not override the unit for margins!

PML Output Options

--full-image-depth

Do not reduce the size or bit depth of images. Images have their size and depth reduced by default to accommodate applications that can not convert images on their own such as Dropbook.

--inline-toc

Add Table of Contents to beginning of the book.

--pml-output-encoding

Specify the character encoding of the output document. The default is cp1252.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

RB Output Options

--inline-toc

Add Table of Contents to beginning of the book.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

RTF Output Options

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

SNB Output Options

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--snb-dont-indent-first-line

Specify whether or not to insert two space characters to indent the first line of each paragraph.

--snb-full-screen

Resize all the images for full screen view.

--snb-hide-chapter-name

Specify whether or not to hide the chapter title for each chapter. Useful for image-only output (eg. comics).

--snb-insert-empty-line

Specify whether or not to insert an empty line between two paragraphs.

--snb-max-line-length

The maximum number of characters per line. This splits on the first space before the specified value. If no space is found the line will be broken at the space after and will exceed the specified value. Also, there is a minimum of 25 characters. Use 0 to disable line splitting.

--snb-output-encoding

Specify the character encoding of the output document. The default is utf-8.

TCR Output Options

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--tcr-output-encoding

Specify the character encoding of the output document. The default is utf-8.

TXT Output Options

--force-max-line-length

Force splitting on the max-line-length value when no space is present. Also allows max-line-length to be below the minimum

--inline-toc

Add Table of Contents to beginning of the book.

--keep-color

Do not remove font color from output. This is only useful when txt-output-formatting is set to textile. Textile is the only formatting that supports setting font color. If this option is not specified font color will not be set and default to the color displayed by the reader (generally this is black).

--keep-image-references

Do not remove image references within the document. This is only useful when paired with a txt-output-formatting option that is not none because links are always removed with plain text output.

--keep-links

Do not remove links within the document. This is only useful when paired with a txt-output-formatting option that is not none because links are always removed with plain text output.

--max-line-length

The maximum number of characters per line. This splits on the first space before the specified value. If no space is found the line will be broken at the space after and will exceed the specified value. Also, there is a minimum of 25 characters. Use 0 to disable line splitting.

--newline, -n

Type of newline to use. Options are [‘old_mac’, ‘system’, ‘unix’, ‘windows’]. Default is ‘system’. Use ‘old_mac’ for compatibility with Mac OS 9 and earlier. For Mac OS X use ‘unix’. ‘system’ will default to the newline type used by this OS.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--txt-output-encoding

Specify the character encoding of the output document. The default is utf-8.

--txt-output-formatting

Formatting used within the document. * plain: Produce plain text. * markdown: Produce Markdown formatted text. * textile: Produce Textile formatted text.

TXTZ Output Options

--force-max-line-length

Force splitting on the max-line-length value when no space is present. Also allows max-line-length to be below the minimum

--inline-toc

Add Table of Contents to beginning of the book.

--keep-color

Do not remove font color from output. This is only useful when txt-output-formatting is set to textile. Textile is the only formatting that supports setting font color. If this option is not specified font color will not be set and default to the color displayed by the reader (generally this is black).

--keep-image-references

Do not remove image references within the document. This is only useful when paired with a txt-output-formatting option that is not none because links are always removed with plain text output.

--keep-links

Do not remove links within the document. This is only useful when paired with a txt-output-formatting option that is not none because links are always removed with plain text output.

--max-line-length

The maximum number of characters per line. This splits on the first space before the specified value. If no space is found the line will be broken at the space after and will exceed the specified value. Also, there is a minimum of 25 characters. Use 0 to disable line splitting.

--newline, -n

Type of newline to use. Options are [‘old_mac’, ‘system’, ‘unix’, ‘windows’]. Default is ‘system’. Use ‘old_mac’ for compatibility with Mac OS 9 and earlier. For Mac OS X use ‘unix’. ‘system’ will default to the newline type used by this OS.

--pretty-print

If specified, the output plugin will try to create output that is as human readable as possible. May not have any effect for some output plugins.

--txt-output-encoding

Specify the character encoding of the output document. The default is utf-8.

--txt-output-formatting

Formatting used within the document. * plain: Produce plain text. * markdown: Produce Markdown formatted text. * textile: Produce Textile formatted text.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

ebook-meta

ebook-meta ebook_file [options]

Read/Write metadata from/to ebook files.

Supported formats for reading metadata: azw, azw1, azw3, azw4, cbr, cbz, chm, docx, epub, fb2, html, htmlz, imp, lit, lrf, lrx, mobi, odt, oebzip, opf, pdb, pdf, pml, pmlz, pobi, prc, rar, rb, rtf, snb, tpz, txt, txtz, updb, zip

Supported formats for writing metadata: azw, azw1, azw3, azw4, epub, fb2, htmlz, lrf, mobi, pdb, pdf, prc, rtf, tpz, txtz

Different file types support different kinds of metadata. If you try to set some metadata on a file type that does not support it, the metadata will be silently ignored.

Whenever you pass arguments to ebook-meta that have spaces in them, enclose the arguments in quotation marks.

[options]

--author-sort

String to be used when sorting by author. If unspecified, and the author(s) are specified, it will be auto-generated from the author(s).

--authors, -a

Set the authors. Multiple authors should be separated by the & character. Author names should be in the order Firstname Lastname.

--book-producer, -k

Set the book producer.

--category

Set the book category.

--comments, -c

Set the ebook description.

--cover

Set the cover to the specified file.

--date, -d

Set the published date.

--from-opf

Read metadata from the specified OPF file and use it to set metadata in the ebook. Metadata specified on the command line will override metadata read from the OPF file

--get-cover

Get the cover from the ebook and save it at as the specified file.

--help, -h

show this help message and exit

--index, -i

Set the index of the book in this series.

--isbn

Set the ISBN of the book.

--language, -l

Set the language.

--lrf-bookid

Set the BookID in LRF files

--publisher, -p

Set the ebook publisher.

--rating, -r

Set the rating. Should be a number between 1 and 5.

--series, -s

Set the series this ebook belongs to.

--tags

Set the tags for the book. Should be a comma separated list.

--title, -t

Set the title.

--title-sort

The version of the title to be used for sorting. If unspecified, and the title is specified, it will be auto-generated from the title.

--to-opf

Specify the name of an OPF file. The metadata will be written to the OPF file.

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

ebook-viewer

ebook-viewer [options] file

View an ebook.

Whenever you pass arguments to ebook-viewer that have spaces in them, enclose the arguments in quotation marks.

[options]

--debug-javascript

Print javascript alert and console messages to the console

--full-screen, -f

If specified, viewer window will try to open full screen when started.

--help, -h

show this help message and exit

--open-at

The position at which to open the specified book. The position is a location as displayed in the top left corner of the viewer.

--raise-window

If specified, viewer window will try to come to the front when started.

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

epub-fix

epub-fix [options] file.epub

Fix common problems in EPUB files that can cause them to be rejected by poorly designed publishing services.

By default, no fixing is done and messages are printed out for each error detected. Use the options to control which errors are automatically fixed.

Whenever you pass arguments to epub-fix that have spaces in them, enclose the arguments in quotation marks.

[options]

--delete-unmanifested

Delete unmanifested files instead of adding them to the manifest

--epubcheck

Workarounds for bugs in the latest release of epubcheck. epubcheck reports many things as errors that are not actually errors. epub-fix will try to detect these and replace them with constructs that epubcheck likes. This may cause significant changes to your epub, complain to the epubcheck project.

--help, -h

show this help message and exit

--unmanifested

Fix unmanifested files. epub-fix can either add them to the manifest or delete them as specified by the delete unmanifested option.

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

fetch-ebook-metadata

fetch-ebook-metadata [options]

Fetch book metadata from online sources. You must specify at least one of title, authors or ISBN.

Whenever you pass arguments to fetch-ebook-metadata that have spaces in them, enclose the arguments in quotation marks.

[options]

--authors, -a

Book author(s)

--cover, -c

Specify a filename. The cover, if available, will be saved to it

--help, -h

show this help message and exit

--isbn, -i

Book ISBN

--opf, -o

Output the metadata in OPF format

--timeout, -d

Timeout in seconds. Default is 30

--title, -t

Book title

--verbose, -v

Print the log to the console (stderr)

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

lrf2lrs

lrf2lrs book.lrf

Convert an LRF file into an LRS (XML UTF-8 encoded) file

Whenever you pass arguments to lrf2lrs that have spaces in them, enclose the arguments in quotation marks.

[options]

--dont-output-resources

Do not save embedded image and font files to disk

--help, -h

show this help message and exit

--output, -o

Output LRS file

--verbose

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

lrfviewer

lrfviewer [options] book.lrf

Read the LRF ebook book.lrf

Whenever you pass arguments to lrfviewer that have spaces in them, enclose the arguments in quotation marks.

[options]

--disable-hyphenation

Disable hyphenation. Should significantly speed up rendering.

--help, -h

show this help message and exit

--profile

Profile the LRF renderer

--verbose

Print more information about the rendering process

--version

show program’s version number and exit

--visual-debug

Turn on visual aids to debugging the rendering engine

--white-background

By default the background is off white as I find this easier on the eyes. Use this option to make the background pure white.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

lrs2lrf

lrs2lrf [options] file.lrs

Compile an LRS file into an LRF file.

Whenever you pass arguments to lrs2lrf that have spaces in them, enclose the arguments in quotation marks.

[options]

--help, -h

show this help message and exit

--lrs

Convert LRS to LRS, useful for debugging.

--output, -o

Path to output file

--verbose

Verbose processing

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Command Line Interface »

web2disk

web2disk URL

Where URL is for example http://google.com

Whenever you pass arguments to web2disk that have spaces in them, enclose the arguments in quotation marks.

[options]

--base-dir, -d

Base directory into which URL is saved. Default is .

--delay

Minimum interval in seconds between consecutive fetches. Default is 0 s

--dont-download-stylesheets

Do not download CSS stylesheets.

--encoding

The character encoding for the websites you are trying to download. The default is to try and guess the encoding.

--filter-regexp

Any link that matches this regular expression will be ignored. This option can be specified multiple times, in which case as long as any regexp matches a link, it will be ignored. By default, no links are ignored. If both filter regexp and match regexp are specified, then filter regexp is applied first.

--help, -h

show this help message and exit

--match-regexp

Only links that match this regular expression will be followed. This option can be specified multiple times, in which case as long as a link matches any one regexp, it will be followed. By default all links are followed.

--max-files, -n

The maximum number of files to download. This only applies to files from <a href> tags. Default is 9223372036854775807

--max-recursions, -r

Maximum number of levels to recurse i.e. depth of links to follow. Default 1

--timeout, -t

Timeout in seconds to wait for a response from the server. Default: 10.0 s

--verbose

Show detailed output information. Useful for debugging

--version

show program’s version number and exit

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- previous

- Start »

Setting up a calibre development environment
calibre is completely open source, licensed under the GNU GPL v3 [http://www.gnu.org/copyleft/gpl.html].
This means that you are free to download and modify the program to your heart’s content. In this section, you will learn how to get a calibre development environment set up on the operating system of your choice.
calibre is written primarily in Python [http://www.python.org] with some C/C++ code for speed and system interfacing.
Note that calibre is not compatible with Python 3 and requires at least Python 2.7.

Contents

- Design philosophy

- Code layout

- Getting the code

- Submitting your changes to be included

- Windows development environment
- OS X development environment
- Linux development environment

- Having separate “normal” and “development” calibre installs on the same computer

- Debugging tips

- Using an interactive python interpreter
- Using print statements
- Using the debugger in PyDev

- Executing arbitrary scripts in the calibre python environment

- Using calibre in your projects

- Binary install of calibre

- Source install on Linux

Design philosophy
calibre has its roots in the Unix world, which means that its design is highly modular.
The modules interact with each other via well defined interfaces. This makes adding new features and fixing bugs in calibre very easy, resulting in a frenetic pace of development. Because of its roots, calibre has a comprehensive command line interface for all its functions, documented in Command Line Interface.

The modular design of calibre is expressed via Plugins. There is a tutorial on writing calibre plugins.
For example, adding support for a new device to calibre typically involves writing less than a 100 lines of code in the form of a device driver plugin. You can browse the built-in drivers [http://bazaar.launchpad.net/%7Ekovid/calibre/trunk/files/head%3A/src/calibre/devices/]. Similarly, adding support for new conversion formats involves writing input/output format plugins. Another example of the modular design is the recipe system for fetching news. For more examples of plugins designed to add features to calibre, see the plugin index [http://www.mobileread.com/forums/showthread.php?p=1362767#post1362767].

Code layout

All the calibre python code is in the calibre package. This package contains the following main sub-packages

- devices - All the device drivers. Just look through some of the built-in drivers to get an idea for how they work.

- For details, see: devices.interface which defines the interface supported by device drivers and devices.usbms which defines a generic driver that connects to a USBMS device. All USBMS based drivers in calibre inherit from it.

- ebooks - All the ebook conversion/metadata code. A good starting point is calibre.ebooks.conversion.cli which is the module powering the ebook-convert command. The conversion process is controlled via conversion.plumber.
The format independent code is all in ebooks.oeb and the format dependent code is in ebooks.format_name.

- Metadata reading, writing, and downloading is all in ebooks.metadata

- Conversion happens in a pipeline, for the structure of the pipeline, see Introduction. The pipeline consists of an input plugin, various transforms and an output plugin. The that code constructs and drives the pipeline is in plumber.py. The pipeline works on a representation of an ebook that is like an unzipped epub, with manifest, spine, toc, guide, html content, etc. The class that manages this representation is OEBBook in oeb/base.py. The various transformations that are applied to the book during conversions live in oeb/transforms/*.py. And the input and output plugins live in conversion/plugins/*.py.

- library - The database back-end and the content server. See library.database2 for the interface to the calibre library. library.server is the calibre Content Server.

- gui2 - The Graphical User Interface. GUI initialization happens in gui2.main and gui2.ui. The ebook-viewer is in gui2.viewer.

If you need help understanding the code, post in the development forum [http://www.mobileread.com/forums/forumdisplay.php?f=240] and you will most likely get help from one of calibre‘s many developers.

Getting the code
calibre uses Bazaar [http://bazaar-vcs.org/], a distributed version control system. Bazaar is available on all the platforms calibre supports.
After installing Bazaar, you can get the calibre source code with the command:

bzr branch lp:calibre

On Windows you will need the complete path name, that will be something like C:\Program Files\Bazaar\bzr.exe.

calibre is a very large project with a very long source control history, so the above can take a while (10mins to an hour depending on your internet speed).

If you want to get the code faster, the sourcecode for the latest release is always available as an archive [http://status.calibre-ebook.com/dist/src].
You can also use bzr to just download the source code, without the history, using:

bzr branch --stacked lp:calibre

To update a branch to the latest code, use the command:

bzr merge

Submitting your changes to be included

If you only plan to make a few small changes, you can make your changes and create a
“merge directive” which you can then attach to a ticket in the calibre bug tracker for consideration. To do this, make your changes, then run:

bzr commit -m "Comment describing your changes"
bzr send -o my-changes

This will create a my-changes file in the current directory, simply attach that to a ticket on the calibre bug tracker [https://bugs.launchpad.net/calibre].

If you plan to do a lot of development on calibre, then the best method is to create a Launchpad [http://launchpad.net] account. Once you have an account, you can use it to register your bzr branch created by the bzr branch command above. First run the following command to tell bzr about your launchpad account:

bzr launchpad-login your_launchpad_username

Now, you have to setup SSH access to Launchpad. First create an SSH public/private keypair. Then upload the public key to Launchpad by going to your Launchpad account page. Instructions for setting up the private key in bzr are at http://bazaar-vcs.org/Bzr_and_SSH. Now you can upload your branch to the calibre project in Launchpad by following the instructions at https://help.launchpad.net/Code/UploadingABranch.
Whenever you commit changes to your branch with the command:

bzr commit -m "Comment describing your change"

Kovid can merge it directly from your branch into the main calibre source tree. You should also keep an eye on the calibre development forum [http://www.mobileread.com/forums/forumdisplay.php?f=240]. Before making major changes, you should discuss them in the forum or contact Kovid directly (his email address is all over the source code).

Windows development environment

Install calibre normally, using the Windows installer. Then open a Command Prompt and change to the previously checked out calibre code directory. For example:

cd C:\Users\kovid\work\calibre

calibre is the directory that contains the src and resources sub-directories.

The next step is to set the environment variable CALIBRE_DEVELOP_FROM to the absolute path of the src directory.
So, following the example above, it would be C:\Users\kovid\work\calibre\src. Here is a short guide [http://docs.python.org/using/windows.html#excursus-setting-environment-variables] to setting environment variables on Windows.

Once you have set the environment variable, open a new command prompt and check that it was correctly set by using the command:

echo %CALIBRE_DEVELOP_FROM%

Setting this environment variable means that calibre will now load all its Python code from the specified location.

That’s it! You are now ready to start hacking on the calibre code. For example, open the file src\calibre\__init__.py in your favorite editor and add the line:

print ("Hello, world!")

near the top of the file. Now run the command calibredb. The very first line of output should be Hello, world!.

OS X development environment

Install calibre normally using the provided .dmg. Then open a Terminal and change to the previously checked out calibre code directory, for example:

cd /Users/kovid/work/calibre

calibre is the directory that contains the src and resources sub-directories. Ensure you have installed the calibre commandline tools via Preferences->Advanced->Miscellaneous in the calibre GUI.

The next step is to create a bash script that will set the environment variable CALIBRE_DEVELOP_FROM to the absolute path of the src directory when running calibre in debug mode.

Create a plain text file:

#!/bin/sh
export CALIBRE_DEVELOP_FROM="/Users/kovid/work/calibre/src"
calibre-debug -g

Save this file as /usr/bin/calibre-develop, then set its permissions so that it can be executed:

chmod +x /usr/bin/calibre-develop

Once you have done this, run:

calibre-develop

You should see some diagnostic information in the Terminal window as calibre starts up, and you should see an asterisk after the version number in the GUI window, indicating that you are running from source.

Linux development environment
calibre is primarily developed on Linux. You have two choices in setting up the development environment. You can install the calibre binary as normal and use that as a runtime environment to do your development. This approach is similar to that used in Windows and OS X. Alternatively, you can install calibre from source. Instructions for setting up a development environment from source are in the INSTALL file in the source tree. Here we will address using the binary at runtime, which is the recommended method.

Install the calibre using the binary installer. Then open a terminal and change to the previously checked out calibre code directory, for example:

cd /home/kovid/work/calibre

calibre is the directory that contains the src and resources sub-directories.

The next step is to set the environment variable CALIBRE_DEVELOP_FROM to the absolute path of the src directory.
So, following the example above, it would be /home/kovid/work/calibre/src. How to set environment variables depends on your Linux distribution and what shell you are using.

Once you have set the environment variable, open a new terminal and check that it was correctly set by using the command:

echo $CALIBRE_DEVELOP_FROM

Setting this environment variable means that calibre will now load all its Python code from the specified location.

That’s it! You are now ready to start hacking on the calibre code. For example, open the file src/calibre/__init__.py in your favorite editor and add the line:

print ("Hello, world!")

near the top of the file. Now run the command calibredb. The very first line of output should be Hello, world!.

Having separate “normal” and “development” calibre installs on the same computer

The calibre source tree is very stable and rarely breaks, but if you feel the need to run from source on a separate test library and run the released calibre version with your everyday library, you can achieve this easily using
.bat files or shell scripts to launch calibre. The example below shows how to do this on Windows using .bat files (the instructions for other platforms are the same, just use a shell script instead of a .bat file)

To launch the release version of calibre with your everyday library:

calibre-normal.bat:

calibre.exe "--with-library=C:\path\to\everyday\library folder"

calibre-dev.bat:

set CALIBRE_DEVELOP_FROM=C:\path\to\calibre\checkout\src
calibre.exe "--with-library=C:\path\to\test\library folder"

Debugging tips

Python is a dynamically typed language with excellent facilities for introspection. Kovid wrote the core calibre code without once using a debugger. There are many strategies to debug calibre code:

Using an interactive python interpreter

You can insert the following two lines of code to start an interactive python session at that point:

from calibre import ipython
ipython(locals())

When running from the command line, this will start an interactive Python interpreter with access to all locally defined variables (variables in the local scope). The interactive prompt even has TAB completion for object properties and you can use the various Python facilities for introspection, such as dir(), type(), repr(), etc.

Using print statements

This is Kovid’s favorite way to debug. Simply insert print statements at points of interest and run your program in the terminal. For example, you can start the GUI from the terminal as:

calibre-debug -g

Similarly, you can start the ebook-viewer as:

calibre-debug -w /path/to/file/to/be/viewed

Using the debugger in PyDev

It is possible to get the debugger in PyDev working with the calibre development environment, see the forum thread [http://www.mobileread.com/forums/showthread.php?t=143208].

Executing arbitrary scripts in the calibre python environment

The calibre-debug command provides a couple of handy switches to execute your own code, with access to the calibre modules:

calibre-debug -c "some python code"

is great for testing a little snippet of code on the command line. It works in the same way as the -c switch to the python interpreter:

calibre-debug -e myscript.py

can be used to execute your own Python script. It works in the same way as passing the script to the Python interpreter, except that the calibre environment is fully initialized, so you can use all the calibre code in your script.

Using calibre in your projects

It is possible to directly use calibre functions/code in your Python project. Two ways exist to do this:

Binary install of calibre

If you have a binary install of calibre, you can use the Python interpreter bundled with calibre, like this:

calibre-debug -e /path/to/your/python/script.py

Source install on Linux

In addition to using the above technique, if you do a source install on Linux, you can also directly import calibre, as follows:

import init_calibre
import calibre

print calibre.__version__

It is essential that you import the init_calibre module before any other calibre modules/packages as it sets up the interpreter to run calibre code.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »

Glossary

RSS

RSS (Really Simple Syndication) is a web feed format that is used to publish frequently updated content, like news articles, blog posts, etc. It is a format that is particularly suited to being read by computers, and is therefore the preferred way of getting content from the web into an ebook. There are many other feed formats in use on the Internet, and calibre understands most of them. In particular, it has good support for the ATOM format, which is commonly used for blogs.

recipe

A recipe is a set of instructions that teach calibre how to convert an online news source, such as a magazine or a blog, into an ebook. A recipe is essentially Python [http://www.python.org] code. As such, it is capable of converting arbitrarily complex news sources into ebooks. At the simplest level, it is just a set of variables, such as URLs, that give calibre enough information to go out onto the Internet and download the news.

HTML

HTML (Hyper Text Mark-Up Language), a subset of Standard Generalized Mark-Up Language (SGML) for electronic publishing, is the specific standard used for the World Wide Web.

CSS

CSS (Cascading Style Sheets) is a language used to describe how an HTML document should be rendered (visual styling).

API

API (Application Programming Interface) is a source code interface that a library provides to support requests for services to be made of it by computer programs.

LRF

LRF The ebook format that is read by the SONY ebook readers.

URL

URL (Uniform Resource Locator) for example: http://example.com
regexp

Regular expressions provide a concise and flexible means for identifying strings of text of interest, such as particular characters, words, or patterns of characters. See regexp syntax [http://docs.python.org/lib/re-syntax.html] for the syntax of regular expressions used in Python.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.customize
from __future__ import with_statement
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'

import os, sys, zipfile, importlib from calibre.constants import numeric_version, iswindows, isosx from calibre.ptempfile import PersistentTemporaryFile platform = 'linux'
if iswindows:
platform = 'windows'
elif isosx:
platform = 'osx'

class PluginNotFound(ValueError):
pass class InvalidPlugin(ValueError):
pass

[docs]class Plugin(object): # {{{
'''
A calibre plugin. Useful members include:

* ``self.plugin_path``: Stores path to the zip file that contains this plugin or None if it is a builtin plugin
* ``self.site_customization``: Stores a customization string entered by the user.

Methods that should be overridden in sub classes:

* :meth:`initialize`
* :meth:`customization_help`

Useful methods:

* :meth:`temporary_file`

'''
#: List of platforms this plugin works on
#: For example: ``['windows', 'osx', 'linux']`` supported_platforms = []

#: The name of this plugin. You must set it something other
#: than Trivial Plugin for it to work.
name = 'Trivial Plugin'

#: The version of this plugin as a 3-tuple (major, minor, revision)
version = (1, 0, 0)

#: A short string describing what this plugin does description = _('Does absolutely nothing')

#: The author of this plugin author = _('Unknown')

#: When more than one plugin exists for a filetype,
#: the plugins are run in order of decreasing priority
#: i.e. plugins with higher priority will be run first.
#: The highest possible priority is ``sys.maxint``.
#: Default priority is 1.
priority = 1

#: The earliest version of calibre this plugin requires minimum_calibre_version = (0, 4, 118)

#: If False, the user will not be able to disable this plugin. Use with
#: care.
can_be_disabled = True

#: The type of this plugin. Used for categorizing plugins in the
#: GUI type = _('Base')

def __init__(self, plugin_path):
self.plugin_path = plugin_path self.site_customization = None

[docs] def initialize(self):
'''
Called once when calibre plugins are initialized. Plugins are re-initialized every time a new plugin is added.

Perform any plugin specific initialization here, such as extracting resources from the plugin zip file. The path to the zip file is available as ``self.plugin_path``.

Note that ``self.site_customization`` is **not** available at this point.
'''
pass

[docs] def config_widget(self):
'''
Implement this method and :meth:`save_settings` in your plugin to use a custom configuration dialog, rather then relying on the simple string based default customization.

This method, if implemented, must return a QWidget. The widget can have an optional method validate() that takes no arguments and is called immediately after the user clicks OK. Changes are applied if and only if the method returns True.

If for some reason you cannot perform the configuration at this time, return a tuple of two strings (message, details), these will be displayed as a warning dialog to the user and the process will be aborted.
'''
raise NotImplementedError()

[docs] def save_settings(self, config_widget):
'''
Save the settings specified by the user with config_widget.

:param config_widget: The widget returned by :meth:`config_widget`.

'''
raise NotImplementedError()

[docs] def do_user_config(self, parent=None):
'''
This method shows a configuration dialog for this plugin. It returns
True if the user clicks OK, False otherwise. The changes are automatically applied.
'''
from PyQt4.Qt import QDialog, QDialogButtonBox, QVBoxLayout, \ QLabel, Qt, QLineEdit from calibre.gui2 import gprefs prefname = 'plugin config dialog:'+self.type + ':' + self.name geom = gprefs.get(prefname, None)

config_dialog = QDialog(parent)
button_box = QDialogButtonBox(QDialogButtonBox.Ok | QDialogButtonBox.Cancel)
v = QVBoxLayout(config_dialog)

def size_dialog():
if geom is None:
config_dialog.resize(config_dialog.sizeHint())
else:
config_dialog.restoreGeometry(geom)

button_box.accepted.connect(config_dialog.accept)
button_box.rejected.connect(config_dialog.reject)
config_dialog.setWindowTitle(_('Customize') + ' ' + self.name)
try:
config_widget = self.config_widget()
except NotImplementedError:
config_widget = None if isinstance(config_widget, tuple):
from calibre.gui2 import warning_dialog warning_dialog(parent, _('Cannot configure'), config_widget[0], det_msg=config_widget[1], show=True)
return False if config_widget is not None:
v.addWidget(config_widget)
v.addWidget(button_box)
size_dialog()
config_dialog.exec_()

if config_dialog.result() == QDialog.Accepted:
if hasattr(config_widget, 'validate'):
if config_widget.validate():
self.save_settings(config_widget)
else:
self.save_settings(config_widget)
else:
from calibre.customize.ui import plugin_customization, \ customize_plugin help_text = self.customization_help(gui=True)
help_text = QLabel(help_text, config_dialog)
help_text.setWordWrap(True)
help_text.setTextInteractionFlags(Qt.LinksAccessibleByMouse
| Qt.LinksAccessibleByKeyboard)
help_text.setOpenExternalLinks(True)
v.addWidget(help_text)
sc = plugin_customization(self)
if not sc:
sc = ''
sc = sc.strip()
sc = QLineEdit(sc, config_dialog)
v.addWidget(sc)
v.addWidget(button_box)
size_dialog()
config_dialog.exec_()

if config_dialog.result() == QDialog.Accepted:
sc = unicode(sc.text()).strip()
customize_plugin(self, sc)

geom = bytearray(config_dialog.saveGeometry())
gprefs[prefname] = geom return config_dialog.result()

[docs] def load_resources(self, names):
'''
If this plugin comes in a ZIP file (user added plugin), this method will allow you to load resources from the ZIP file.

For example to load an image::

pixmap = QPixmap()
pixmap.loadFromData(self.load_resources(['images/icon.png']).itervalues().next())
icon = QIcon(pixmap)

:param names: List of paths to resources in the zip file using / as separator

:return: A dictionary of the form ``{name : file_contents}``. Any names that were not found in the zip file will not be present in the dictionary.

'''
if self.plugin_path is None:
raise ValueError('This plugin was not loaded from a ZIP file')
ans = {}
with zipfile.ZipFile(self.plugin_path, 'r') as zf:
for candidate in zf.namelist():
if candidate in names:
ans[candidate] = zf.read(candidate)
return ans

[docs] def customization_help(self, gui=False):
'''
Return a string giving help on how to customize this plugin.
By default raise a :class:`NotImplementedError`, which indicates that the plugin does not require customization.

If you re-implement this method in your subclass, the user will be asked to enter a string as customization for this plugin.
The customization string will be available as
``self.site_customization``.

Site customization could be anything, for example, the path to a needed binary on the user's computer.

:param gui: If True return HTML help, otherwise return plain text help.

'''
raise NotImplementedError

[docs] def temporary_file(self, suffix):
'''
Return a file-like object that is a temporary file on the file system.
This file will remain available even after being closed and will only be removed on interpreter shutdown. Use the ``name`` member of the returned object to access the full path to the created temporary file.

:param suffix: The suffix that the temporary file will have.
'''
return PersistentTemporaryFile(suffix)

def is_customizable(self):
try:
self.customization_help()
return True except NotImplementedError:
return False def __enter__(self, *args):
if self.plugin_path is not None:
from calibre.utils.zipfile import ZipFile zf = ZipFile(self.plugin_path)
extensions = set([x.rpartition('.')[-1].lower() for x in zf.namelist()])
zip_safe = True for ext in ('pyd', 'so', 'dll', 'dylib'):
if ext in extensions:
zip_safe = False if zip_safe:
sys.path.insert(0, self.plugin_path)
self.sys_insertion_path = self.plugin_path else:
from calibre.ptempfile import TemporaryDirectory self._sys_insertion_tdir = TemporaryDirectory('plugin_unzip')
self.sys_insertion_path = self._sys_insertion_tdir.__enter__(*args)
zf.extractall(self.sys_insertion_path)
sys.path.insert(0, self.sys_insertion_path)
zf.close()

def __exit__(self, *args):
ip, it = getattr(self, 'sys_insertion_path', None), getattr(self,
'_sys_insertion_tdir', None)
if ip in sys.path:
sys.path.remove(ip)
if hasattr(it, '__exit__'):
it.__exit__(*args)

# }}}

[docs]class FileTypePlugin(Plugin): # {{{
'''
A plugin that is associated with a particular set of file types.
'''

#: Set of file types for which this plugin should be run
#: For example: ``set(['lit', 'mobi', 'prc'])`` file_types = set([])

#: If True, this plugin is run when books are added
#: to the database on_import = False

#: If True, this plugin is run after books are added
#: to the database on_postimport = False

#: If True, this plugin is run just before a conversion on_preprocess = False

#: If True, this plugin is run after conversion
#: on the final file produced by the conversion output plugin.
on_postprocess = False type = _('File type')

[docs] def run(self, path_to_ebook):
'''
Run the plugin. Must be implemented in subclasses.
It should perform whatever modifications are required on the ebook and return the absolute path to the modified ebook. If no modifications are needed, it should return the path to the original ebook. If an error is encountered it should raise an Exception. The default implementation simply return the path to the original ebook.

The modified ebook file should be created with the
:meth:`temporary_file` method.

:param path_to_ebook: Absolute path to the ebook.

:return: Absolute path to the modified ebook.
'''
# Default implementation does nothing return path_to_ebook

[docs] def postimport(self, book_id, book_format, db):
'''
Called post import, i.e., after the book file has been added to the database.

:param book_id: Database id of the added book.
:param book_format: The file type of the book that was added.
:param db: Library database.
'''
pass # Default implementation does nothing

# }}}

[docs]class MetadataReaderPlugin(Plugin): # {{{
'''
A plugin that implements reading metadata from a set of file types.
'''
#: Set of file types for which this plugin should be run
#: For example: ``set(['lit', 'mobi', 'prc'])`` file_types = set([])

supported_platforms = ['windows', 'osx', 'linux']
version = numeric_version author = 'Kovid Goyal'

type = _('Metadata reader')

def __init__(self, *args, **kwargs):
Plugin.__init__(self, *args, **kwargs)
self.quick = False

[docs] def get_metadata(self, stream, type):
'''
Return metadata for the file represented by stream (a file like object that supports reading). Raise an exception when there is an error with the input data.
:param type: The type of file. Guaranteed to be one of the entries in :attr:`file_types`.
:return: A :class:`calibre.ebooks.metadata.book.Metadata` object
'''
return None
# }}}

[docs]class MetadataWriterPlugin(Plugin): # {{{
'''
A plugin that implements reading metadata from a set of file types.
'''
#: Set of file types for which this plugin should be run
#: For example: ``set(['lit', 'mobi', 'prc'])`` file_types = set([])

supported_platforms = ['windows', 'osx', 'linux']
version = numeric_version author = 'Kovid Goyal'

type = _('Metadata writer')

def __init__(self, *args, **kwargs):
Plugin.__init__(self, *args, **kwargs)
self.apply_null = False

[docs] def set_metadata(self, stream, mi, type):
'''
Set metadata for the file represented by stream (a file like object that supports reading). Raise an exception when there is an error with the input data.
:param type: The type of file. Guaranteed to be one of the entries in :attr:`file_types`.
:param mi: A :class:`calibre.ebooks.metadata.book.Metadata` object
'''
pass

# }}}

[docs]class CatalogPlugin(Plugin): # {{{
'''
A plugin that implements a catalog generator.
'''

resources_path = None

#: Output file type for which this plugin should be run
#: For example: 'epub' or 'xml'
file_types = set([])

type = _('Catalog generator')

#: CLI parser options specific to this plugin, declared as namedtuple Option::
#:
#: from collections import namedtuple
#: Option = namedtuple('Option', 'option, default, dest, help')
#: cli_options = [Option('--catalog-title',
#: default = 'My Catalog',
#: dest = 'catalog_title',
#: help = (_('Title of generated catalog. \nDefault:') + " '" +
#: '%default' + "'"))]
#: cli_options parsed in library.cli:catalog_option_parser()
cli_options = []

def _field_sorter(self, key):
'''
Custom fields sort after standard fields
'''
if key.startswith('#'):
return '~%s' % key[1:]
else:
return key def search_sort_db(self, db, opts):

db.search(opts.search_text)

if opts.sort_by:
# 2nd arg = ascending db.sort(opts.sort_by, True)
return db.get_data_as_dict(ids=opts.ids)

def get_output_fields(self, db, opts):
# Return a list of requested fields, with opts.sort_by first all_std_fields = set(
['author_sort','authors','comments','cover','formats',
'id','isbn','library_name','ondevice','pubdate','publisher',
'rating','series_index','series','size','tags','timestamp',
'title_sort','title','uuid','languages','identifiers'])
all_custom_fields = set(db.custom_field_keys())
for field in list(all_custom_fields):
fm = db.field_metadata[field]
if fm['datatype'] == 'series':
all_custom_fields.add(field+'_index')
all_fields = all_std_fields.union(all_custom_fields)

if opts.fields != 'all':
# Make a list from opts.fields requested_fields = set(opts.fields.split(','))

# Validate requested_fields if requested_fields - all_fields:
from calibre.library import current_library_name invalid_fields = sorted(list(requested_fields - all_fields))
print("invalid --fields specified: %s" % ', '.join(invalid_fields))
print("available fields in '%s': %s" %
(current_library_name(), ', '.join(sorted(list(all_fields)))))
raise ValueError("unable to generate catalog with specified fields")

fields = list(all_fields & requested_fields)
else:
fields = list(all_fields)

if not opts.connected_device['is_device_connected'] and 'ondevice' in fields:
fields.pop(int(fields.index('ondevice')))

fields = sorted(fields, key=self._field_sorter)
if opts.sort_by and opts.sort_by in fields:
fields.insert(0,fields.pop(int(fields.index(opts.sort_by))))
return fields

[docs] def initialize(self):
'''
If plugin is not a built-in, copy the plugin's .ui and .py files from the zip file to $TMPDIR.
Tab will be dynamically generated and added to the Catalog Options dialog in calibre.gui2.dialogs.catalog.py:Catalog
'''
from calibre.customize.builtins import plugins as builtin_plugins from calibre.customize.ui import config from calibre.ptempfile import PersistentTemporaryDirectory if not type(self) in builtin_plugins and \ not self.name in config['disabled_plugins']:
files_to_copy = ["%s.%s" % (self.name.lower(),ext) for ext in ["ui","py"]]
resources = zipfile.ZipFile(self.plugin_path,'r')

if self.resources_path is None:
self.resources_path = PersistentTemporaryDirectory('_plugin_resources', prefix='')

for file in files_to_copy:
try:
resources.extract(file, self.resources_path)
except:
print " customize:__init__.initialize(): %s not found in %s" % (file, os.path.basename(self.plugin_path))
continue resources.close()

[docs] def run(self, path_to_output, opts, db, ids, notification=None):
'''
Run the plugin. Must be implemented in subclasses.
It should generate the catalog in the format specified in file_types, returning the absolute path to the generated catalog file. If an error is encountered it should raise an Exception.

The generated catalog file should be created with the
:meth:`temporary_file` method.

:param path_to_output: Absolute path to the generated catalog file.
:param opts: A dictionary of keyword arguments
:param db: A LibraryDatabase2 object
'''
# Default implementation does nothing raise NotImplementedError('CatalogPlugin.generate_catalog() default '
'method, should be overridden in subclass')

# }}}

[docs]class InterfaceActionBase(Plugin): # {{{ supported_platforms = ['windows', 'osx', 'linux']
author = 'Kovid Goyal'
type = _('User Interface Action')
can_be_disabled = False actual_plugin = None def __init__(self, *args, **kwargs):
Plugin.__init__(self, *args, **kwargs)
self.actual_plugin_ = None

[docs] def load_actual_plugin(self, gui):
'''
This method must return the actual interface action plugin object.
'''
ac = self.actual_plugin_ if ac is None:
mod, cls = self.actual_plugin.split(':')
ac = getattr(importlib.import_module(mod), cls)(gui, self.site_customization)
self.actual_plugin_ = ac return ac

# }}}

[docs]class PreferencesPlugin(Plugin): # {{{

'''
A plugin representing a widget displayed in the Preferences dialog.

This plugin has only one important method :meth:`create_widget`. The various fields of the plugin control how it is categorized in the UI.
'''

supported_platforms = ['windows', 'osx', 'linux']
author = 'Kovid Goyal'
type = _('Preferences')
can_be_disabled = False

#: Import path to module that contains a class named ConfigWidget
#: which implements the ConfigWidgetInterface. Used by
#: :meth:`create_widget`.
config_widget = None

#: Where in the list of categories the :attr:`category` of this plugin should be.
category_order = 100

#: Where in the list of names in a category, the :attr:`gui_name` of this
#: plugin should be name_order = 100

#: The category this plugin should be in category = None

#: The category name displayed to the user for this plugin gui_category = None

#: The name displayed to the user for this plugin gui_name = None

#: The icon for this plugin, should be an absolute path icon = None

#: The description used for tooltips and the like description = None

[docs] def create_widget(self, parent=None):
'''
Create and return the actual Qt widget used for setting this group of preferences. The widget must implement the
:class:`calibre.gui2.preferences.ConfigWidgetInterface`.

The default implementation uses :attr:`config_widget` to instantiate the widget.
'''
base, _, wc = self.config_widget.partition(':')
if not wc:
wc = 'ConfigWidget'
base = importlib.import_module(base)
widget = getattr(base, wc)
return widget(parent)

# }}}

class StoreBase(Plugin): # {{{ supported_platforms = ['windows', 'osx', 'linux']
author = 'John Schember'
type = _('Store')
# Information about the store. Should be in the primary language
# of the store. This should not be translatable when set by
# a subclass.
description = _('An ebook store.')
minimum_calibre_version = (0, 8, 0)
version = (1, 0, 1)

actual_plugin = None

# Does the store only distribute ebooks without DRM.
drm_free_only = False
# This is the 2 letter country code for the corporate
# headquarters of the store.
headquarters = ''
# All formats the store distributes ebooks in.
formats = []
# Is this store on an affiliate program?
affiliate = False def load_actual_plugin(self, gui):
'''
This method must return the actual interface action plugin object.
'''
mod, cls = self.actual_plugin.split(':')
self.actual_plugin_object = getattr(importlib.import_module(mod), cls)(gui, self.name)
return self.actual_plugin_object def customization_help(self, gui=False):
if getattr(self, 'actual_plugin_object', None) is not None:
return self.actual_plugin_object.customization_help(gui)
raise NotImplementedError()

def config_widget(self):
if getattr(self, 'actual_plugin_object', None) is not None:
return self.actual_plugin_object.config_widget()
raise NotImplementedError()

def save_settings(self, config_widget):
if getattr(self, 'actual_plugin_object', None) is not None:
return self.actual_plugin_object.save_settings(config_widget)
raise NotImplementedError()

# }}}

class ViewerPlugin(Plugin): # {{{

'''
These plugins are used to add functionality to the calibre viewer.
'''

def load_fonts(self):
'''
This method is called once at viewer starup. It should load any fonts it wants to make available. For example::

def load_fonts():
from PyQt4.Qt import QFontDatabase font_data = get_resources(['myfont1.ttf', 'myfont2.ttf'])
for raw in font_data.itervalues():
QFontDatabase.addApplicationFontFromData(raw)
'''
pass def load_javascript(self, evaljs):
'''
This method is called every time a new HTML document is loaded in the viewer. Use it to load javascript libraries into the viewer. For example::

def load_javascript(self, evaljs):
js = get_resources('myjavascript.js')
evaljs(js)
'''
pass def run_javascript(self, evaljs):
'''
This method is called every time a document has finished laoding. Use it in the same way as load_javascript().
'''
pass

# }}}

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »
- Module code »
- calibre.customize »

Source code for calibre.customize.conversion

# -*- coding: utf-8 -*-
'''
Defines the plugin system for conversions.
'''
import re, os, shutil from calibre import CurrentDir from calibre.customize import Plugin class ConversionOption(object):

'''
Class representing conversion options
'''

def __init__(self, name=None, help=None, long_switch=None, short_switch=None, choices=None):
self.name = name self.help = help self.long_switch = long_switch self.short_switch = short_switch self.choices = choices if self.long_switch is None:
self.long_switch = self.name.replace('_', '-')

self.validate_parameters()

def validate_parameters(self):
'''
Validate the parameters passed to :meth:`__init__`.
'''
if re.match(r'[a-zA-Z_]([a-zA-Z0-9_])*', self.name) is None:
raise ValueError(self.name + ' is not a valid Python identifier')
if not self.help:
raise ValueError('You must set the help text')

def __hash__(self):
return hash(self.name)

def __eq__(self, other):
return hash(self) == hash(other)

def clone(self):
return ConversionOption(name=self.name, help=self.help, long_switch=self.long_switch, short_switch=self.short_switch, choices=self.choices)

class OptionRecommendation(object):
LOW = 1
MED = 2
HIGH = 3 def __init__(self, recommended_value=None, level=LOW, **kwargs):
'''
An option recommendation. That is, an option as well as its recommended value and the level of the recommendation.
'''
self.level = level self.recommended_value = recommended_value self.option = kwargs.pop('option', None)
if self.option is None:
self.option = ConversionOption(**kwargs)

self.validate_parameters()

@property def help(self):
return self.option.help def clone(self):
return OptionRecommendation(recommended_value=self.recommended_value, level=self.level, option=self.option.clone())

def validate_parameters(self):
if self.option.choices and self.recommended_value not in \ self.option.choices:
raise ValueError('OpRec: %s: Recommended value not in choices'% self.option.name)
if not (isinstance(self.recommended_value, (int, float, str, unicode))\ or self.recommended_value is None):
raise ValueError('OpRec: %s:'%self.option.name + repr(self.recommended_value) +
' is not a string or a number')

class DummyReporter(object):

def __init__(self):
self.cancel_requested = False def __call__(self, percent, msg=''):
pass def gui_configuration_widget(name, parent, get_option_by_name, get_option_help, db, book_id, for_output=True):
import importlib def widget_factory(cls):
return cls(parent, get_option_by_name, get_option_help, db, book_id)

if for_output:
try:
output_widget = importlib.import_module(
'calibre.gui2.convert.'+name)
pw = output_widget.PluginWidget pw.ICON = I('back.png')
pw.HELP = _('Options specific to the output format.')
return widget_factory(pw)
except ImportError:
pass else:
try:
input_widget = importlib.import_module(
'calibre.gui2.convert.'+name)
pw = input_widget.PluginWidget pw.ICON = I('forward.png')
pw.HELP = _('Options specific to the input format.')
return widget_factory(pw)
except ImportError:
pass return None

[docs]class InputFormatPlugin(Plugin):
'''
InputFormatPlugins are responsible for converting a document into
HTML+OPF+CSS+etc.
The results of the conversion *must* be encoded in UTF-8.
The main action happens in :meth:`convert`.
'''

type = _('Conversion Input')
can_be_disabled = False supported_platforms = ['windows', 'osx', 'linux']

#: Set of file types for which this plugin should be run
#: For example: ``set(['azw', 'mobi', 'prc'])`` file_types = set([])

#: If True, this input plugin generates a collection of images,
#: one per HTML file. You can obtain access to the images via
#: convenience method, :meth:`get_image_collection`.
is_image_collection = False

#: Number of CPU cores used by this plugin
#: A value of -1 means that it uses all available cores core_usage = 1

#: If set to True, the input plugin will perform special processing
#: to make its output suitable for viewing for_viewer = False

#: The encoding that this input plugin creates files in. A value of
#: None means that the encoding is undefined and must be
#: detected individually output_encoding = 'utf-8'

#: Options shared by all Input format plugins. Do not override
#: in sub-classes. Use :attr:`options` instead. Every option must be an
#: instance of :class:`OptionRecommendation`.
common_options = set([
OptionRecommendation(name='input_encoding', recommended_value=None, level=OptionRecommendation.LOW, help=_('Specify the character encoding of the input document. If '
'set this option will override any encoding declared by the '
'document itself. Particularly useful for documents that '
'do not declare an encoding or that have erroneous '
'encoding declarations.')
),

])

#: Options to customize the behavior of this plugin. Every option must be an
#: instance of :class:`OptionRecommendation`.
options = set([])

#: A set of 3-tuples of the form
#: (option_name, recommended_value, recommendation_level)
recommendations = set([])

def __init__(self, *args):
Plugin.__init__(self, *args)
self.report_progress = DummyReporter()

[docs] def get_images(self):
'''
Return a list of absolute paths to the images, if this input plugin represents an image collection. The list of images is in the same order as the spine and the TOC.
'''
raise NotImplementedError()

[docs] def convert(self, stream, options, file_ext, log, accelerators):
'''
This method must be implemented in sub-classes. It must return the path to the created OPF file or an :class:`OEBBook` instance.
All output should be contained in the current directory.
If this plugin creates files outside the current directory they must be deleted/marked for deletion before this method returns.

:param stream: A file like object that contains the input file.
:param options: Options to customize the conversion process.
Guaranteed to have attributes corresponding to all the options declared by this plugin. In addition, it will have a verbose attribute that takes integral values from zero upwards. Higher numbers mean be more verbose. Another useful attribute is
``input_profile`` that is an instance of
:class:`calibre.customize.profiles.InputProfile`.
:param file_ext: The extension (without the .) of the input file. It is guaranteed to be one of the `file_types` supported by this plugin.
:param log: A :class:`calibre.utils.logging.Log` object. All output should use this object.
:param accelarators: A dictionary of various information that the input plugin can get easily that would speed up the subsequent stages of the conversion.

'''
raise NotImplementedError
def __call__(self, stream, options, file_ext, log, accelerators, output_dir):
try:
log('InputFormatPlugin: %s running'%self.name)
if hasattr(stream, 'name'):
log('on', stream.name)
except:
# In case stdout is broken pass with CurrentDir(output_dir):
for x in os.listdir('.'):
shutil.rmtree(x) if os.path.isdir(x) else os.remove(x)

ret = self.convert(stream, options, file_ext, log, accelerators)

return ret

[docs] def postprocess_book(self, oeb, opts, log):
'''
Called to allow the input plugin to perform postprocessing after the book has been parsed.
'''
pass

[docs] def specialize(self, oeb, opts, log, output_fmt):
'''
Called to allow the input plugin to specialize the parsed book for a particular output format. Called after postprocess_book and before any transforms are performed on the parsed book.
'''
pass

[docs] def gui_configuration_widget(self, parent, get_option_by_name, get_option_help, db, book_id=None):
'''
Called to create the widget used for configuring this plugin in the calibre GUI. The widget must be an instance of the PluginWidget class.
See the builting input plugins for examples.
'''
name = self.name.lower().replace(' ', '_')
return gui_configuration_widget(name, parent, get_option_by_name, get_option_help, db, book_id, for_output=False)

[docs]class OutputFormatPlugin(Plugin):
'''
OutputFormatPlugins are responsible for converting an OEB document
(OPF+HTML) into an output ebook.

The OEB document can be assumed to be encoded in UTF-8.
The main action happens in :meth:`convert`.
'''

type = _('Conversion Output')
can_be_disabled = False supported_platforms = ['windows', 'osx', 'linux']

#: The file type (extension without leading period) that this
#: plugin outputs file_type = None

#: Options shared by all Input format plugins. Do not override
#: in sub-classes. Use :attr:`options` instead. Every option must be an
#: instance of :class:`OptionRecommendation`.
common_options = set([
OptionRecommendation(name='pretty_print', recommended_value=False, level=OptionRecommendation.LOW, help=_('If specified, the output plugin will try to create output '
'that is as human readable as possible. May not have any effect '
'for some output plugins.')
),
])

#: Options to customize the behavior of this plugin. Every option must be an
#: instance of :class:`OptionRecommendation`.
options = set([])

#: A set of 3-tuples of the form
#: (option_name, recommended_value, recommendation_level)
recommendations = set([])

@property def description(self):
return _('Convert ebooks to the %s format')%self.file_type def __init__(self, *args):
Plugin.__init__(self, *args)
self.report_progress = DummyReporter()

[docs] def convert(self, oeb_book, output, input_plugin, opts, log):
'''
Render the contents of `oeb_book` (which is an instance of
:class:`calibre.ebooks.oeb.OEBBook` to the file specified by output.

:param output: Either a file like object or a string. If it is a string it is the path to a directory that may or may not exist. The output plugin should write its output into that directory. If it is a file like object, the output plugin should write its output into the file.
:param input_plugin: The input plugin that was used at the beginning of the conversion pipeline.
:param opts: Conversion options. Guaranteed to have attributes corresponding to the OptionRecommendations of this plugin.
:param log: The logger. Print debug/info messages etc. using this.

'''
raise NotImplementedError

@property def is_periodical(self):
return self.oeb.metadata.publication_type and \ unicode(self.oeb.metadata.publication_type[0]).startswith('periodical:')

[docs] def specialize_css_for_output(self, log, opts, item, stylizer):
'''
Can be used to make changes to the css during the CSS flattening process.

:param item: The item (HTML file) being processed
:param stylizer: A Stylizer object containing the flattened styles for item. You can get the style for any element by stylizer.style(element).

'''
pass

[docs] def gui_configuration_widget(self, parent, get_option_by_name, get_option_help, db, book_id=None):
'''
Called to create the widget used for configuring this plugin in the calibre GUI. The widget must be an instance of the PluginWidget class.
See the builtin output plugins for examples.
'''
name = self.name.lower().replace(' ', '_')
return gui_configuration_widget(name, parent, get_option_by_name, get_option_help, db, book_id, for_output=True)

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.devices.interface

__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
import os from collections import namedtuple from calibre.customize import Plugin from calibre.constants import iswindows

[docs]class DevicePlugin(Plugin):
"""
Defines the interface that should be implemented by backends that communicate with an ebook reader.
"""
type = _('Device Interface')

#: Ordered list of supported formats
FORMATS = ["lrf", "rtf", "pdf", "txt"]
# If True, the config dialog will not show the formats box
HIDE_FORMATS_CONFIG_BOX = False

#: VENDOR_ID can be either an integer, a list of integers or a dictionary
#: If it is a dictionary, it must be a dictionary of dictionaries,
#: of the form::
#:
#: {
#: integer_vendor_id : { product_id : [list of BCDs], ... },
#: ...
#: }
#:
VENDOR_ID = 0x0000

#: An integer or a list of integers
PRODUCT_ID = 0x0000
#: BCD can be either None to not distinguish between devices based on BCD, or
#: it can be a list of the BCD numbers of all devices supported by this driver.
BCD = None

#: Height for thumbnails on the device
THUMBNAIL_HEIGHT = 68
#: Width for thumbnails on the device. Setting this will force thumbnails
#: to this size, not preserving aspect ratio. If it is not set, then
#: the aspect ratio will be preserved and the thumbnail will be no higher
#: than THUMBNAIL_HEIGHT
# THUMBNAIL_WIDTH = 68

#: Set this to True if the device supports updating cover thumbnails during
#: sync_booklists. Setting it to true will ask device.py to refresh the
#: cover thumbnails during book matching
WANTS_UPDATED_THUMBNAILS = False

#: Whether the metadata on books can be set via the GUI.
CAN_SET_METADATA = ['title', 'authors', 'collections']

#: Whether the device can handle device_db metadata plugboards
CAN_DO_DEVICE_DB_PLUGBOARD = False

# Set this to None if the books on the device are files that the GUI can
# access in order to add the books from the device to the library
BACKLOADING_ERROR_MESSAGE = _('Cannot get files from this device')

#: Path separator for paths to books on device path_sep = os.sep

#: Icon for this device icon = I('reader.png')

# Encapsulates an annotation fetched from the device
UserAnnotation = namedtuple('Annotation','type, value')

#: GUI displays this as a message if not None. Useful if opening can take a
#: long time
OPEN_FEEDBACK_MESSAGE = None

#: Set of extensions that are "virtual books" on the device
#: and therefore cannot be viewed/saved/added to library
#: For example: ``frozenset(['kobo'])``
VIRTUAL_BOOK_EXTENSIONS = frozenset([])

#: Whether to nuke comments in the copy of the book sent to the device. If
#: not None this should be short string that the comments will be replaced
#: by.
NUKE_COMMENTS = None

#: If True indicates that this driver completely manages device detection,
#: ejecting and so forth. If you set this to True, you *must* implement the
#: detect_managed_devices and debug_managed_device_detection methods.
#: A driver with this set to true is responsible for detection of devices,
#: managing a blacklist of devices, a list of ejected devices and so forth.
#: calibre will periodically call the detect_managed_devices() method and
#: is it returns a detected device, calibre will call open(). open() will
#: be called every time a device is returned even is previous calls to open()
#: failed, therefore the driver must maintain its own blacklist of failed
#: devices. Similarly, when ejecting, calibre will call eject() and then
#: assuming the next call to detect_managed_devices() returns None, it will
#: call post_yank_cleanup().
MANAGES_DEVICE_PRESENCE = False

#: If set the True, calibre will call the :meth:`get_driveinfo()` method
#: after the books lists have been loaded to get the driveinfo.
SLOW_DRIVEINFO = False

#: If set to True, calibre will ask the user if they want to manage the
#: device with calibre, the first time it is detected. If you set this to
#: True you must implement :meth:`get_device_uid()` and
#: :meth:`ignore_connected_device()` and
#: :meth:`get_user_blacklisted_devices` and
#: :meth:`set_user_blacklisted_devices`
ASK_TO_ALLOW_CONNECT = False

@classmethod def get_gui_name(cls):
if hasattr(cls, 'gui_name'):
return cls.gui_name if hasattr(cls, '__name__'):
return cls.__name__ return cls.name

# Device detection {{{ def test_bcd_windows(self, device_id, bcd):
if bcd is None or len(bcd) == 0:
return True for c in bcd:
rev = 'rev_%4.4x'%c
# Bug in winutil.get_usb_devices sometimes converts a to :
if rev in device_id or rev.replace('a', ':') in device_id:
return True return False def print_usb_device_info(self, info):
try:
print '\t', repr(info)
except:
import traceback traceback.print_exc()

def is_usb_connected_windows(self, devices_on_system, debug=False, only_presence=False):

def id_iterator():
if hasattr(self.VENDOR_ID, 'keys'):
for vid in self.VENDOR_ID:
vend = self.VENDOR_ID[vid]
for pid in vend:
bcd = vend[pid]
yield vid, pid, bcd else:
vendors = self.VENDOR_ID if hasattr(self.VENDOR_ID, '__len__') else [self.VENDOR_ID]
products = self.PRODUCT_ID if hasattr(self.PRODUCT_ID, '__len__') else [self.PRODUCT_ID]
for vid in vendors:
for pid in products:
yield vid, pid, self.BCD for vendor_id, product_id, bcd in id_iterator():
vid, pid = 'vid_%4.4x'%vendor_id, 'pid_%4.4x'%product_id vidd, pidd = 'vid_%i'%vendor_id, 'pid_%i'%product_id for device_id in devices_on_system:
if (vid in device_id or vidd in device_id) and \ (pid in device_id or pidd in device_id) and \ self.test_bcd_windows(device_id, bcd):
if debug:
self.print_usb_device_info(device_id)
if only_presence or self.can_handle_windows(device_id, debug=debug):
try:
bcd = int(device_id.rpartition(
'rev_')[-1].replace(':', 'a'), 16)
except:
bcd = None return True, (vendor_id, product_id, bcd, None,
None, None)
return False, None def test_bcd(self, bcdDevice, bcd):
if bcd is None or len(bcd) == 0:
return True for c in bcd:
if c == bcdDevice:
return True return False

[docs] def is_usb_connected(self, devices_on_system, debug=False, only_presence=False):
'''
Return True, device_info if a device handled by this plugin is currently connected.

:param devices_on_system: List of devices currently connected

'''
if iswindows:
return self.is_usb_connected_windows(devices_on_system, debug=debug, only_presence=only_presence)

vendors_on_system = set([x[0] for x in devices_on_system])
vendors = self.VENDOR_ID if hasattr(self.VENDOR_ID, '__len__') else [self.VENDOR_ID]
if hasattr(self.VENDOR_ID, 'keys'):
products = []
for ven in self.VENDOR_ID:
products.extend(self.VENDOR_ID[ven].keys())
else:
products = self.PRODUCT_ID if hasattr(self.PRODUCT_ID, '__len__') else [self.PRODUCT_ID]

for vid in vendors:
if vid in vendors_on_system:
for dev in devices_on_system:
cvid, pid, bcd = dev[:3]
if cvid == vid:
if pid in products:
if hasattr(self.VENDOR_ID, 'keys'):
try:
cbcd = self.VENDOR_ID[vid][pid]
except KeyError:
# Vendor vid does not have product pid, pid
# exists for some other vendor in this
# device continue else:
cbcd = self.BCD if self.test_bcd(bcd, cbcd):
if debug:
self.print_usb_device_info(dev)
if self.can_handle(dev, debug=debug):
return True, dev return False, None

[docs] def detect_managed_devices(self, devices_on_system, force_refresh=False):
'''
Called only if MANAGES_DEVICE_PRESENCE is True.

Scan for devices that this driver can handle. Should return a device object if a device is found. This object will be passed to the open()
method as the connected_device. If no device is found, return None. The returned object can be anything, calibre does not use it, it is only passed to open().

This method is called periodically by the GUI, so make sure it is not too resource intensive. Use a cache to avoid repeatedly scanning the system.

:param devices_on_system: Set of USB devices found on the system.

:param force_refresh: If True and the driver uses a cache to prevent repeated scanning, the cache must be flushed.

'''
raise NotImplementedError()

[docs] def debug_managed_device_detection(self, devices_on_system, output):
'''
Called only if MANAGES_DEVICE_PRESENCE is True.

Should write information about the devices detected on the system to output, which is a file like object.

Should return True if a device was detected and successfully opened, otherwise False.
'''
raise NotImplementedError()

# }}}

[docs] def reset(self, key='-1', log_packets=False, report_progress=None, detected_device=None):
"""
:param key: The key to unlock the device
:param log_packets: If true the packet stream to/from the device is logged
:param report_progress: Function that is called with a % progress
(number between 0 and 100) for various tasks
If it is called with -1 that means that the task does not have any progress information
:param detected_device: Device information from the device scanner

"""
raise NotImplementedError()

[docs] def can_handle_windows(self, device_id, debug=False):
'''
Optional method to perform further checks on a device to see if this driver is capable of handling it. If it is not it should return False. This method is only called after the vendor, product ids and the bcd have matched, so it can do some relatively time intensive checks. The default implementation returns True. This method is called only on windows. See also
:meth:`can_handle`.

:param device_info: On windows a device ID string. On Unix a tuple of
``(vendor_id, product_id, bcd)``.

'''
return True

[docs] def can_handle(self, device_info, debug=False):
'''
Unix version of :meth:`can_handle_windows`

:param device_info: Is a tuple of (vid, pid, bcd, manufacturer, product, serial number)

'''

return True

[docs] def open(self, connected_device, library_uuid):
'''
Perform any device specific initialization. Called after the device is detected but before any other functions that communicate with the device.
For example: For devices that present themselves as USB Mass storage devices, this method would be responsible for mounting the device or if the device has been automounted, for finding out where it has been mounted. The method :meth:`calibre.devices.usbms.device.Device.open` has an implementation of this function that should serve as a good example for USB Mass storage devices.

This method can raise an OpenFeedback exception to display a message to the user.

:param connected_device: The device that we are trying to open. It is a tuple of (vendor id, product id, bcd, manufacturer name, product name, device serial number). However, some devices have no serial number and on windows only the first three fields are present, the rest are None.

:param library_uuid: The UUID of the current calibre library. Can be
None if there is no library (for example when used from the command line).

'''
raise NotImplementedError()

[docs] def eject(self):
'''
Un-mount / eject the device from the OS. This does not check if there are pending GUI jobs that need to communicate with the device.

NOTE: That this method may not be called on the same thread as the rest of the device methods.
'''
raise NotImplementedError()

[docs] def post_yank_cleanup(self):
'''
Called if the user yanks the device without ejecting it first.
'''
raise NotImplementedError()

[docs] def set_progress_reporter(self, report_progress):
'''
Set a function to report progress information.

:param report_progress: Function that is called with a % progress
(number between 0 and 100) for various tasks
If it is called with -1 that means that the task does not have any progress information

'''
raise NotImplementedError()

[docs] def get_device_information(self, end_session=True):
"""
Ask device for device information. See L{DeviceInfoQuery}.

:return: (device name, device version, software version on device, mime type)
The tuple can optionally have a fifth element, which is a drive information dictionary. See usbms.driver for an example.

"""
raise NotImplementedError()

[docs] def get_driveinfo(self):
'''
Return the driveinfo dictionary. Usually called from get_device_information(), but if loading the driveinfo is slow for this driver, then it should set SLOW_DRIVEINFO. In this case, this method will be called by calibre after the book lists have been loaded. Note that it is not called on the device thread, so the driver should cache the drive info in the books() method and this function should return the cached data.
'''
return {}

[docs] def card_prefix(self, end_session=True):
'''
Return a 2 element list of the prefix to paths on the cards.
If no card is present None is set for the card's prefix.
E.G.
('/place', '/place2')
(None, 'place2')
('place', None)
(None, None)
'''
raise NotImplementedError()

[docs] def total_space(self, end_session=True):
"""
Get total space available on the mountpoints:
1. Main memory
2. Memory Card A
3. Memory Card B

:return: A 3 element list with total space in bytes of (1, 2, 3). If a particular device doesn't have any of these locations it should return 0.

"""
raise NotImplementedError()

[docs] def free_space(self, end_session=True):
"""
Get free space available on the mountpoints:
1. Main memory
2. Card A
3. Card B

:return: A 3 element list with free space in bytes of (1, 2, 3). If a particular device doesn't have any of these locations it should return -1.

"""
raise NotImplementedError()

[docs] def books(self, oncard=None, end_session=True):
"""
Return a list of ebooks on the device.

:param oncard: If 'carda' or 'cardb' return a list of ebooks on the specific storage card, otherwise return list of ebooks in main memory of device. If a card is specified and no books are on the card return empty list.

:return: A BookList.

"""
raise NotImplementedError()

[docs] def upload_books(self, files, names, on_card=None, end_session=True, metadata=None):
'''
Upload a list of books to the device. If a file already exists on the device, it should be replaced.
This method should raise a :class:`FreeSpaceError` if there is not enough free space on the device. The text of the FreeSpaceError must contain the word "card" if ``on_card`` is not None otherwise it must contain the word "memory".

:param files: A list of paths
:param names: A list of file names that the books should have once uploaded to the device. len(names) == len(files)
:param metadata: If not None, it is a list of :class:`Metadata` objects.
The idea is to use the metadata to determine where on the device to put the book. len(metadata) == len(files). Apart from the regular cover (path to cover), there may also be a thumbnail attribute, which should be used in preference. The thumbnail attribute is of the form
(width, height, cover_data as jpeg).

:return: A list of 3-element tuples. The list is meant to be passed to :meth:`add_books_to_metadata`.
'''
raise NotImplementedError()

@classmethod

[docs] def add_books_to_metadata(cls, locations, metadata, booklists):
'''
Add locations to the booklists. This function must not communicate with the device.

:param locations: Result of a call to L{upload_books}
:param metadata: List of :class:`Metadata` objects, same as for
:meth:`upload_books`.
:param booklists: A tuple containing the result of calls to
(:meth:`books(oncard=None)`,
:meth:`books(oncard='carda')`,
:meth`books(oncard='cardb')`).

'''
raise NotImplementedError

[docs] def delete_books(self, paths, end_session=True):
'''
Delete books at paths on device.
'''
raise NotImplementedError()

@classmethod

[docs] def remove_books_from_metadata(cls, paths, booklists):
'''
Remove books from the metadata list. This function must not communicate with the device.

:param paths: paths to books on the device.
:param booklists: A tuple containing the result of calls to
(:meth:`books(oncard=None)`,
:meth:`books(oncard='carda')`,
:meth`books(oncard='cardb')`).

'''
raise NotImplementedError()

[docs] def sync_booklists(self, booklists, end_session=True):
'''
Update metadata on device.

:param booklists: A tuple containing the result of calls to
(:meth:`books(oncard=None)`,
:meth:`books(oncard='carda')`,
:meth`books(oncard='cardb')`).

'''
raise NotImplementedError()

[docs] def get_file(self, path, outfile, end_session=True):
'''
Read the file at ``path`` on the device and write it to outfile.

:param outfile: file object like ``sys.stdout`` or the result of an
:func:`open` call.

'''
raise NotImplementedError()

@classmethod

[docs] def config_widget(cls):
'''
Should return a QWidget. The QWidget contains the settings for the device interface
'''
raise NotImplementedError()

@classmethod

[docs] def save_settings(cls, settings_widget):
'''
Should save settings to disk. Takes the widget created in
:meth:`config_widget` and saves all settings to disk.
'''
raise NotImplementedError()

@classmethod

[docs] def settings(cls):
'''
Should return an opts object. The opts object should have at least one attribute `format_map` which is an ordered list of formats for the device.
'''
raise NotImplementedError()

[docs] def set_plugboards(self, plugboards, pb_func):
'''
provide the driver the current set of plugboards and a function to select a specific plugboard. This method is called immediately before add_books and sync_booklists.

pb_func is a callable with the following signature::
def pb_func(device_name, format, plugboards)

You give it the current device name (either the class name or
DEVICE_PLUGBOARD_NAME), the format you are interested in (a 'real'
format or 'device_db'), and the plugboards (you were given those by set_plugboards, the same place you got this method).

:return: None or a single plugboard instance.

'''
pass

[docs] def set_driveinfo_name(self, location_code, name):
'''
Set the device name in the driveinfo file to 'name'. This setting will persist until the file is re-created or the name is changed again.

Non-disk devices should implement this method based on the location codes returned by the get_device_information() method.
'''
pass

[docs] def prepare_addable_books(self, paths):
'''
Given a list of paths, returns another list of paths. These paths point to addable versions of the books.

If there is an error preparing a book, then instead of a path, the position in the returned list for that book should be a three tuple:
(original_path, the exception instance, traceback)
'''
return paths

[docs] def startup(self):
'''
Called when calibre is is starting the device. Do any initialization required. Note that multiple instances of the class can be instantiated, and thus __init__ can be called multiple times, but only one instance will have this method called. This method is called on the device thread, not the GUI thread.
'''
pass

[docs] def shutdown(self):
'''
Called when calibre is shutting down, either for good or in preparation to restart. Do any cleanup required. This method is called on the device thread, not the GUI thread.
'''
pass

[docs] def get_device_uid(self):
'''
Must return a unique id for the currently connected device (this is called immediately after a successful call to open()). You must implement this method if you set ASK_TO_ALLOW_CONNECT = True
'''
raise NotImplementedError()

[docs] def ignore_connected_device(self, uid):
'''
Should ignore the device identified by uid (the result of a call to get_device_uid()) in the future. You must implement this method if you set ASK_TO_ALLOW_CONNECT = True. Note that this function is called immediately after open(), so if open() caches some state, the driver should reset that state.
'''
raise NotImplementedError()

[docs] def get_user_blacklisted_devices(self):
'''
Return map of device uid to friendly name for all devices that the user has asked to be ignored.
'''
return {}

[docs] def set_user_blacklisted_devices(self, devices):
'''
Set the list of device uids that should be ignored by this driver.
'''
pass

[docs] def specialize_global_preferences(self, device_prefs):
'''
Implement this method if your device wants to override a particular preference. You must ensure that all call sites that want a preference that can be overridden use device_prefs['something'] instead of prefs['something']. Your method should call device_prefs.set_overrides(pref=val, pref=val, ...).
Currently used for:
metadata management (prefs['manage_device_metadata'])
'''
device_prefs.set_overrides()

# Dynamic control interface.
# The following methods are probably called on the GUI thread. Any driver
# that implements these methods must take pains to be thread safe, because
# the device_manager might be using the driver at the same time that one of
# these methods is called.

[docs] def is_dynamically_controllable(self):
'''
Called by the device manager when starting plugins. If this method returns a string, then a) it supports the device manager's dynamic control interface, and b) that name is to be used when talking to the plugin.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.
'''
return None

[docs] def start_plugin(self):
'''
This method is called to start the plugin. The plugin should begin to accept device connections however it does that. If the plugin is already accepting connections, then do nothing.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.
'''
pass

[docs] def stop_plugin(self):
'''
This method is called to stop the plugin. The plugin should no longer accept connections, and should cleanup behind itself. It is likely that this method should call shutdown. If the plugin is already not accepting connections, then do nothing.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.
'''
pass

[docs] def get_option(self, opt_string, default=None):
'''
Return the value of the option indicated by opt_string. This method can be called when the plugin is not started. Return None if the option does not exist.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.
'''
return default

[docs] def set_option(self, opt_string, opt_value):
'''
Set the value of the option indicated by opt_string. This method can be called when the plugin is not started.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.
'''
pass

[docs] def is_running(self):
'''
Return True if the plugin is started, otherwise false

This method can be called on the GUI thread. A driver that implements this method must be thread safe.
'''
return False

[docs]class BookList(list):
'''
A list of books. Each Book object must have the fields

#. title
#. authors
#. size (file size of the book)
#. datetime (a UTC time tuple)
#. path (path on the device to the book)
#. thumbnail (can be None) thumbnail is either a str/bytes object with the image data or it should have an attribute image_path that stores an absolute (platform native) path to the image
#. tags (a list of strings, can be empty).

'''

__getslice__ = None
__setslice__ = None def __init__(self, oncard, prefix, settings):
pass

[docs] def supports_collections(self):
''' Return True if the device supports collections for this book list. '''
raise NotImplementedError()

[docs] def add_book(self, book, replace_metadata):
'''
Add the book to the booklist. Intent is to maintain any device-internal metadata. Return True if booklists must be sync'ed
'''
raise NotImplementedError()

[docs] def remove_book(self, book):
'''
Remove a book from the booklist. Correct any device metadata at the same time
'''
raise NotImplementedError()

[docs] def get_collections(self, collection_attributes):
'''
Return a dictionary of collections created from collection_attributes.
Each entry in the dictionary is of the form collection name:[list of books]

The list of books is sorted by book title, except for collections created from series, in which case series_index is used.

:param collection_attributes: A list of attributes of the Book object

'''
raise NotImplementedError()

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.devices.usbms.cli

# -*- coding: utf-8 -*-

__license__ = 'GPL 3'
__copyright__ = '2009, John Schember <john@nachtimwald.com>'
__docformat__ = 'restructuredtext en'

import os, shutil, time from calibre.devices.errors import PathError from calibre.utils.filenames import case_preserving_open_file class File(object):

def __init__(self, path):
stats = os.stat(path)
self.is_dir = os.path.isdir(path)
self.is_readonly = not os.access(path, os.W_OK)
self.ctime = stats.st_ctime self.wtime = stats.st_mtime self.size = stats.st_size if path.endswith(os.sep):
path = path[:-1]
self.path = path self.name = os.path.basename(path)

def check_transfer(infile, dest):
infile.seek(0)
dest.seek(0)
return infile.read() == dest.read()

[docs]class CLI(object):

def get_file(self, path, outfile, end_session=True):
path = self.munge_path(path)
with open(path, 'rb') as src:
shutil.copyfileobj(src, outfile)

def put_file(self, infile, path, replace_file=False, end_session=True):
path = self.munge_path(path)
close = False if not hasattr(infile, 'read'):
infile, close = open(infile, 'rb'), True infile.seek(0)
if os.path.isdir(path):
path = os.path.join(path, infile.name)
if not replace_file and os.path.exists(path):
raise PathError('File already exists: ' + path)
dest, actual_path = case_preserving_open_file(path)
with dest:
try:
shutil.copyfileobj(infile, dest)
except IOError:
print 'WARNING: First attempt to send file to device failed'
time.sleep(0.2)
infile.seek(0)
dest.seek(0)
dest.truncate()
shutil.copyfileobj(infile, dest)
#if not check_transfer(infile, dest): raise Exception('Transfer failed')
if close:
infile.close()
return actual_path def munge_path(self, path):
if path.startswith('/') and not (path.startswith(self._main_prefix) or \ (self._card_a_prefix and path.startswith(self._card_a_prefix)) or \ (self._card_b_prefix and path.startswith(self._card_b_prefix))):
path = self._main_prefix + path[1:]
elif path.startswith('carda:'):
path = path.replace('carda:', self._card_a_prefix[:-1])
elif path.startswith('cardb:'):
path = path.replace('cardb:', self._card_b_prefix[:-1])
return path def list(self, path, recurse=False, end_session=True, munge=True):
if munge:
path = self.munge_path(path)
if os.path.isfile(path):
return [(os.path.dirname(path), [File(path)])]
entries = [File(os.path.join(path, f)) for f in os.listdir(path)]
dirs = [(path, entries)]
for _file in entries:
if recurse and _file.is_dir:
dirs[len(dirs):] = self.list(_file.path, recurse=True, munge=False)
return dirs def mkdir(self, path, end_session=True):
if self.SUPPORTS_SUB_DIRS:
path = self.munge_path(path)
os.mkdir(path)

def rm(self, path, end_session=True):
path = self.munge_path(path)
self.delete_books([path])

def touch(self, path, end_session=True):
path = self.munge_path(path)
if not os.path.exists(path):
open(path, 'w').close()
if not os.path.isdir(path):
os.utime(path, None)

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.devices.usbms.device

# -*- coding: utf-8 -*-

__license__ = 'GPL v3'
__copyright__ = '2009, John Schember <john at nachtimwald.com> ' \ '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'

'''
Generic device driver. This is not a complete stand alone driver. It is intended to be subclassed with the relevant parts implemented for a particular device. This class handles device detection.
'''

import os, subprocess, time, re, sys, glob from itertools import repeat from calibre import prints, as_unicode from calibre.devices.interface import DevicePlugin from calibre.devices.errors import DeviceError from calibre.devices.usbms.deviceconfig import DeviceConfig from calibre.constants import iswindows, islinux, isosx, isfreebsd, plugins from calibre.utils.filenames import ascii_filename as sanitize if isosx:
usbobserver, usbobserver_err = plugins['usbobserver']

class USBDevice:

def __init__(self, dev):
self.idVendor = dev[0]
self.idProduct = dev[1]
self.bcdDevice = dev[2]
self.manufacturer = dev[3]
self.product = dev[4]
self.serial = dev[5]

def match_serial(self, serial):
return self.serial and self.serial == serial def match_numbers(self, vid, pid, bcd):
return self.idVendor == vid and self.idProduct == pid and self.bcdDevice == bcd def match_strings(self, vid, pid, bcd, man, prod):
return self.match_numbers(vid, pid, bcd) and \ self.manufacturer == man and self.product == prod

[docs]class Device(DeviceConfig, DevicePlugin):

'''
This class provides logic common to all drivers for devices that export themselves as USB Mass Storage devices. Provides implementations for mounting/ejecting of USBMS devices on all platforms.
'''

VENDOR_ID = 0x0
PRODUCT_ID = 0x0
BCD = None

VENDOR_NAME = None

#: String identifying the main memory of the device in the windows PnP id
#: strings
#: This can be None, string, list of strings or compiled regex
WINDOWS_MAIN_MEM = None

#: String identifying the first card of the device in the windows PnP id
#: strings
#: This can be None, string, list of strings or compiled regex
WINDOWS_CARD_A_MEM = None

#: String identifying the second card of the device in the windows PnP id
#: strings
#: This can be None, string, list of strings or compiled regex
WINDOWS_CARD_B_MEM = None

# The following are used by the check_ioreg_line method and can be either:
# None, a string, a list of strings or a compiled regular expression
OSX_MAIN_MEM = None
OSX_CARD_A_MEM = None
OSX_CARD_B_MEM = None

#: Used by the new driver detection to disambiguate main memory from
#: storage cards. Should be a regular expression that matches the
#: main memory mount point assigned by OS X
OSX_MAIN_MEM_VOL_PAT = None
OSX_EJECT_COMMAND = ['diskutil', 'eject']

MAIN_MEMORY_VOLUME_LABEL = ''
STORAGE_CARD_VOLUME_LABEL = ''
STORAGE_CARD2_VOLUME_LABEL = None

EBOOK_DIR_MAIN = ''
EBOOK_DIR_CARD_A = ''
EBOOK_DIR_CARD_B = ''
DELETE_EXTS = []

# USB disk-based devices can see the book files on the device, so can
# copy these back to the library
BACKLOADING_ERROR_MESSAGE = None

#: The maximum length of paths created on the device
MAX_PATH_LEN = 250

#: Put news in its own folder
NEWS_IN_FOLDER = True def reset(self, key='-1', log_packets=False, report_progress=None, detected_device=None):
self._main_prefix = self._card_a_prefix = self._card_b_prefix = None try:
self.detected_device = USBDevice(detected_device)
except: # On windows detected_device is None self.detected_device = None self.set_progress_reporter(report_progress)

def set_progress_reporter(self, report_progress):
self.report_progress = report_progress self.report_progress = report_progress if self.report_progress is None:
self.report_progress = lambda x, y: x def card_prefix(self, end_session=True):
return (self._card_a_prefix, self._card_b_prefix)

@classmethod def _windows_space(cls, prefix):
if not prefix:
return 0, 0 prefix = prefix[:-1]
import win32file, winerror try:
sectors_per_cluster, bytes_per_sector, free_clusters, total_clusters = \ win32file.GetDiskFreeSpace(prefix)
except Exception as err:
if getattr(err, 'args', [None])[0] == winerror.ERROR_NOT_READY:
# Disk not ready time.sleep(3)
sectors_per_cluster, bytes_per_sector, free_clusters, total_clusters = \ win32file.GetDiskFreeSpace(prefix)
else: raise mult = sectors_per_cluster * bytes_per_sector return total_clusters * mult, free_clusters * mult def total_space(self, end_session=True):
msz = casz = cbsz = 0 if not iswindows:
if self._main_prefix is not None:
stats = os.statvfs(self._main_prefix)
msz = stats.f_frsize * (stats.f_blocks + stats.f_bavail - stats.f_bfree)
if self._card_a_prefix is not None:
stats = os.statvfs(self._card_a_prefix)
casz = stats.f_frsize * (stats.f_blocks + stats.f_bavail - stats.f_bfree)
if self._card_b_prefix is not None:
stats = os.statvfs(self._card_b_prefix)
cbsz = stats.f_frsize * (stats.f_blocks + stats.f_bavail - stats.f_bfree)
else:
msz = self._windows_space(self._main_prefix)[0]
casz = self._windows_space(self._card_a_prefix)[0]
cbsz = self._windows_space(self._card_b_prefix)[0]

return (msz, casz, cbsz)

def free_space(self, end_session=True):
msz = casz = cbsz = 0 if not iswindows:
if self._main_prefix is not None:
stats = os.statvfs(self._main_prefix)
msz = stats.f_frsize * stats.f_bavail if self._card_a_prefix is not None:
stats = os.statvfs(self._card_a_prefix)
casz = stats.f_frsize * stats.f_bavail if self._card_b_prefix is not None:
stats = os.statvfs(self._card_b_prefix)
cbsz = stats.f_frsize * stats.f_bavail else:
msz = self._windows_space(self._main_prefix)[1]
casz = self._windows_space(self._card_a_prefix)[1]
cbsz = self._windows_space(self._card_b_prefix)[1]

return (msz, casz, cbsz)

def windows_filter_pnp_id(self, pnp_id):
return False def windows_match_device(self, pnp_id, attr):
device_id = getattr(self, attr)

def test_vendor():
vendors = [self.VENDOR_NAME] if isinstance(self.VENDOR_NAME, basestring) else self.VENDOR_NAME for v in vendors:
if 'VEN_'+str(v).upper() in pnp_id:
return True return False if device_id is None or not test_vendor():
return False if self.windows_filter_pnp_id(pnp_id):
return False if hasattr(device_id, 'search'):
return device_id.search(pnp_id) is not None if isinstance(device_id, basestring):
device_id = [device_id]

for x in device_id:
x = x.upper()

if 'PROD_' + x in pnp_id:
return True return False

[docs] def windows_sort_drives(self, drives):
'''
Called to disambiguate main memory and storage card for devices that do not distinguish between them on the basis of `WINDOWS_CARD_NAME`.
For e.g.: The EB600
'''
return drives
def can_handle_windows(self, device_id, debug=False):
from calibre.devices.scanner import win_pnp_drives drives = win_pnp_drives()
for pnp_id in drives.values():
if self.windows_match_device(pnp_id, 'WINDOWS_MAIN_MEM'):
return True if debug:
print '\tNo match found in:', pnp_id return False def open_windows(self):
from calibre.devices.scanner import win_pnp_drives, drivecmp time.sleep(5)
drives = {}
seen = set()
prod_pat = re.compile(r'PROD_(.+?)&')
dup_prod_id = False def check_for_dups(pnp_id):
try:
match = prod_pat.search(pnp_id)
if match is not None:
prodid = match.group(1)
if prodid in seen:
return True else:
seen.add(prodid)
except:
pass return False for drive, pnp_id in win_pnp_drives().items():
if self.windows_match_device(pnp_id, 'WINDOWS_CARD_A_MEM') and \ not drives.get('carda', False):
drives['carda'] = drive dup_prod_id |= check_for_dups(pnp_id)
elif self.windows_match_device(pnp_id, 'WINDOWS_CARD_B_MEM') and \ not drives.get('cardb', False):
drives['cardb'] = drive dup_prod_id |= check_for_dups(pnp_id)
elif self.windows_match_device(pnp_id, 'WINDOWS_MAIN_MEM') and \ not drives.get('main', False):
drives['main'] = drive dup_prod_id |= check_for_dups(pnp_id)

if 'main' in drives.keys() and 'carda' in drives.keys() and \ 'cardb' in drives.keys():
break

# This is typically needed when the device has the same
# WINDOWS_MAIN_MEM and WINDOWS_CARD_A_MEM in which case
# if the device is connected without a card, the above
# will incorrectly identify the main mem as carda
# See for example the driver for the Nook if drives.get('carda', None) is not None and \ drives.get('main', None) is None:
drives['main'] = drives.pop('carda')

if drives.get('main', None) is None:
raise DeviceError(
_('Unable to detect the %s disk drive. Try rebooting.') % self.__class__.__name__)

# Sort drives by their PNP drive numbers if the CARD and MAIN
# MEM strings are identical if dup_prod_id or \ self.WINDOWS_MAIN_MEM in (self.WINDOWS_CARD_A_MEM, self.WINDOWS_CARD_B_MEM) or \ self.WINDOWS_CARD_A_MEM == self.WINDOWS_CARD_B_MEM:
letters = sorted(drives.values(), cmp=drivecmp)
drives = {}
for which, letter in zip(['main', 'carda', 'cardb'], letters):
drives[which] = letter drives = self.windows_sort_drives(drives)
self._main_prefix = drives.get('main')
self._card_a_prefix = drives.get('carda', None)
self._card_b_prefix = drives.get('cardb', None)

@classmethod def run_ioreg(cls, raw=None):
if raw is not None:
return raw ioreg = '/usr/sbin/ioreg'
if not os.access(ioreg, os.X_OK):
ioreg = 'ioreg'
cmd = (ioreg+' -w 0 -S -c IOMedia').split()
for i in range(3):
try:
return subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()[0]
except IOError: # Probably an interrupted system call if i == 2:
raise time.sleep(2)

def osx_sort_names(self, names):
return names def check_ioreg_line(self, line, pat):
if pat is None:
return False if not line.strip().endswith('<class IOMedia>'):
return False if hasattr(pat, 'search'):
return pat.search(line) is not None if isinstance(pat, basestring):
pat = [pat]
for x in pat:
if x in line:
return True return False def get_osx_mountpoints(self, raw=None):
raw = self.run_ioreg(raw)
lines = raw.splitlines()
names = {}

def get_dev_node(lines, loc):
for line in lines:
line = line.strip()
if line.endswith('}'):
break match = re.search(r'"BSD Name"\s+=\s+"(.*?)"', line)
if match is not None:
names[loc] = match.group(1)
break for i, line in enumerate(lines):
if 'main' not in names and self.check_ioreg_line(line, self.OSX_MAIN_MEM):
get_dev_node(lines[i+1:], 'main')
continue if 'carda' not in names and self.check_ioreg_line(line, self.OSX_CARD_A_MEM):
get_dev_node(lines[i+1:], 'carda')
continue if 'cardb' not in names and self.check_ioreg_line(line, self.OSX_CARD_B_MEM):
get_dev_node(lines[i+1:], 'cardb')
continue if len(names.keys()) == 3:
break return self.osx_sort_names(names)

@classmethod def osx_run_mount(cls):
for i in range(3):
try:
return subprocess.Popen('mount', stdout=subprocess.PIPE).communicate()[0]
except IOError: # Probably an interrupted system call if i == 2:
raise time.sleep(2)

@classmethod def osx_get_usb_drives(cls):
if usbobserver_err:
raise RuntimeError('Failed to load usbobserver: '+usbobserver_err)
return usbobserver.get_usb_drives()

def _osx_bsd_names(self):
drives = self.osx_get_usb_drives()
matches = []
d = self.detected_device if d.serial:
for path, vid, pid, bcd, ven, prod, serial in drives:
if d.match_serial(serial):
matches.append(path)
if not matches:
if d.manufacturer and d.product:
for path, vid, pid, bcd, man, prod, serial in drives:
if d.match_strings(vid, pid, bcd, man, prod):
matches.append(path)
else:
for path, vid, pid, bcd, man, prod, serial in drives:
if d.match_numbers(vid, pid, bcd):
matches.append(path)
if not matches:
raise DeviceError(
'Could not detect BSD names for %s. Try rebooting.' % self.name)

pat = re.compile(r'(?P<m>\d+)([a-z]+(?P<p>\d+)){0,1}')
def nums(x):
'Return (disk num, partition number)'
m = pat.search(x)
if m is None:
return (10000, -1)
g = m.groupdict()
if g['p'] is None:
g['p'] = 0 return map(int, (g.get('m'), g.get('p')))

def dcmp(x, y):
'''
Sorting based on the following scheme:
- disks without partitions are first
- sub sorted based on disk number
- disks with partitions are sorted first on disk number, then on partition number
'''
x = x.rpartition('/')[-1]
y = y.rpartition('/')[-1]
x, y = nums(x), nums(y)
if x[1] == 0 and y[1] > 0:
return cmp(1, 2)
if x[1] > 0 and y[1] == 0:
return cmp(2, 1)
ans = cmp(x[0], y[0])
if ans == 0:
ans = cmp(x[1], y[1])
return ans matches.sort(cmp=dcmp)
drives = {'main':matches[0]}
if len(matches) > 1:
drives['carda'] = matches[1]
if len(matches) > 2:
drives['cardb'] = matches[2]

return drives def osx_bsd_names(self):
drives = []
for i in range(3):
try:
drives = self._osx_bsd_names()
if len(drives) > 1: return drives except:
if i == 2: raise time.sleep(3)
return drives def open_osx(self):
drives = self.osx_bsd_names()
bsd_drives = dict(**drives)
drives = self.osx_sort_names(drives)
mount_map = usbobserver.get_mounted_filesystems()
for k, v in drives.items():
drives[k] = mount_map.get(v, None)
if drives['main'] is None:
print bsd_drives, mount_map, drives raise DeviceError(_('Unable to detect the %s mount point. Try rebooting.')%self.__class__.__name__)
pat = self.OSX_MAIN_MEM_VOL_PAT if pat is not None and len(drives) > 1 and 'main' in drives:
if pat.search(drives['main']) is None:
main = drives['main']
for x in ('carda', 'cardb'):
if x in drives and pat.search(drives[x]):
drives['main'] = drives.pop(x)
drives[x] = main break self._main_prefix = drives['main']+os.sep def get_card_prefix(c):
ans = drives.get(c, None)
if ans is not None:
ans += os.sep return ans self._card_a_prefix = get_card_prefix('carda')
self._card_b_prefix = get_card_prefix('cardb')

def find_device_nodes(self, detected_device=None):

def walk(base):
base = os.path.abspath(os.path.realpath(base))
for x in os.listdir(base):
p = os.path.join(base, x)
if os.path.islink(p) or not os.access(p, os.R_OK):
continue isfile = os.path.isfile(p)
yield p, isfile if not isfile:
for y, q in walk(p):
yield y, q def raw2num(raw):
raw = raw.lower()
if not raw.startswith('0x'):
raw = '0x' + raw return int(raw, 16)

# Find device node based on vendor, product and bcd d, j = os.path.dirname, os.path.join usb_dir = None if detected_device is None:
detected_device = self.detected_device def test(val, attr):
q = getattr(detected_device, attr)
return q == val for x, isfile in walk('/sys/devices'):
if isfile and x.endswith('idVendor'):
usb_dir = d(x)
for y in ('idProduct', 'idVendor', 'bcdDevice'):
if not os.access(j(usb_dir, y), os.R_OK):
usb_dir = None continue e = lambda q : raw2num(open(j(usb_dir, q)).read())
ven, prod, bcd = map(e, ('idVendor', 'idProduct', 'bcdDevice'))
if not (test(ven, 'idVendor') and test(prod, 'idProduct') and test(bcd, 'bcdDevice')):
usb_dir = None continue else:
break if usb_dir is None:
raise DeviceError(_('Unable to detect the %s disk drive.')
%self.__class__.__name__)

devnodes, ok = [], {}
for x, isfile in walk(usb_dir):
if not isfile and '/block/' in x:
parts = x.split('/')
idx = parts.index('block')
if idx == len(parts)-2:
sz = j(x, 'size')
node = parts[idx+1]
try:
exists = int(open(sz).read()) > 0 if exists:
node = self.find_largest_partition(x)
ok[node] = True else:
ok[node] = False except:
ok[node] = False devnodes.append(node)

devnodes += list(repeat(None, 3))
ans = ['/dev/'+x if ok.get(x, False) else None for x in devnodes[:3]]
ans.sort(key=lambda x: x[5:] if x else 'zzzzz')
return self.linux_swap_drives(ans)

def linux_swap_drives(self, drives):
return drives def node_mountpoint(self, node):
from calibre.devices.udisks import node_mountpoint return node_mountpoint(node)

def find_largest_partition(self, path):
node = path.split('/')[-1]
nodes = []
for x in glob.glob(path+'/'+node+'*'):
sz = x + '/size'

if not os.access(sz, os.R_OK):
continue try:
sz = int(open(sz).read())
except:
continue if sz > 0:
nodes.append((x.split('/')[-1], sz))

nodes.sort(cmp=lambda x, y: cmp(x[1], y[1]))
if not nodes:
return node return nodes[-1][0]

def open_linux(self):

def mount(node, type):
mp = self.node_mountpoint(node)
if mp is not None:
return mp, 0 def do_mount(node):
try:
from calibre.devices.udisks import mount mount(node)
return 0 except:
print 'Udisks mount call failed:'
import traceback traceback.print_exc()
return 1 ret = do_mount(node)
if ret != 0:
return None, ret return self.node_mountpoint(node)+'/', 0 main, carda, cardb = self.find_device_nodes()
if main is None:
raise DeviceError(_('Unable to detect the %s disk drive. Either '
'the device has already been ejected, or your '
'kernel is exporting a deprecated version of SYSFS.')
%self.__class__.__name__)

self._linux_mount_map = {}
mp, ret = mount(main, 'main')
if mp is None:
raise DeviceError(
_('Unable to mount main memory (Error code: %d)')%ret)
if not mp.endswith('/'): mp += '/'
self._linux_mount_map[main] = mp self._main_prefix = mp self._linux_main_device_node = main cards = [(carda, '_card_a_prefix', 'carda'),
(cardb, '_card_b_prefix', 'cardb')]
for card, prefix, typ in cards:
if card is None: continue mp, ret = mount(card, typ)
if mp is None:
print >>sys.stderr, 'Unable to mount card (Error code: %d)'%ret else:
if not mp.endswith('/'): mp += '/'
setattr(self, prefix, mp)
self._linux_mount_map[card] = mp self.filter_read_only_mount_points()

def filter_read_only_mount_points(self):

def is_readonly(mp):
if mp is None:
return True path = os.path.join(mp, 'calibre_readonly_test')
ro = True try:
with open(path, 'wb'):
ro = False except:
pass else:
try:
os.remove(path)
except:
pass return ro for mp in ('_main_prefix', '_card_a_prefix', '_card_b_prefix'):
if is_readonly(getattr(self, mp, None)):
setattr(self, mp, None)

if self._main_prefix is None:
for p in ('_card_a_prefix', '_card_b_prefix'):
nmp = getattr(self, p, None)
if nmp is not None:
self._main_prefix = nmp setattr(self, p, None)
break if self._main_prefix is None:
raise DeviceError(_('The main memory of %s is read only. '
'This usually happens because of file system errors.')
%self.__class__.__name__)

if self._card_a_prefix is None and self._card_b_prefix is not None:
self._card_a_prefix = self._card_b_prefix self._card_b_prefix = None

# ------------------------------------------------------
#
# open for FreeBSD
# find the device node or nodes that match the S/N we already have from the scanner
# and attempt to mount each one
# 1. get list of devices in /dev with matching s/n etc.
# 2. get list of volumes associated with each
# 3. attempt to mount each one using Hal
# 4. when finished, we have a list of mount points and associated dbus nodes
# def open_freebsd(self):
import dbus
# There should be some way to access the -v arg...
verbose = False

# this gives us access to the S/N, etc. of the reader that the scanner has found
# and the match routines for some of that data, like s/n, vendor ID, etc.
d=self.detected_device if not d.serial:
raise DeviceError("Device has no S/N. Can't continue")
return False vols=[]

bus = dbus.SystemBus()
manager = dbus.Interface(bus.get_object('org.freedesktop.Hal',
'/org/freedesktop/Hal/Manager'), 'org.freedesktop.Hal.Manager')
paths = manager.FindDeviceStringMatch('usb.serial',d.serial)
for path in paths:
objif = dbus.Interface(bus.get_object('org.freedesktop.Hal', path), 'org.freedesktop.Hal.Device')
# Extra paranoia...
try:
if d.idVendor == objif.GetProperty('usb.vendor_id') and \ d.idProduct == objif.GetProperty('usb.product_id') and \ d.manufacturer == objif.GetProperty('usb.vendor') and \ d.product == objif.GetProperty('usb.product') and \ d.serial == objif.GetProperty('usb.serial'):
dpaths = manager.FindDeviceStringMatch('storage.originating_device', path)
for dpath in dpaths:
#devif = dbus.Interface(bus.get_object('org.freedesktop.Hal', dpath), 'org.freedesktop.Hal.Device')
try:
vpaths = manager.FindDeviceStringMatch('block.storage_device', dpath)
for vpath in vpaths:
try:
vdevif = dbus.Interface(bus.get_object('org.freedesktop.Hal', vpath), 'org.freedesktop.Hal.Device')
if not vdevif.GetProperty('block.is_volume'):
continue if vdevif.GetProperty('volume.fsusage') != 'filesystem':
continue volif = dbus.Interface(bus.get_object('org.freedesktop.Hal', vpath), 'org.freedesktop.Hal.Device.Volume')
pdevif = dbus.Interface(bus.get_object('org.freedesktop.Hal', vdevif.GetProperty('info.parent')), 'org.freedesktop.Hal.Device')
vol = {'node': pdevif.GetProperty('block.device'),
'dev': vdevif,
'vol': volif,
'label': vdevif.GetProperty('volume.label')}
vols.append(vol)
except dbus.exceptions.DBusException, e:
print e continue except dbus.exceptions.DBusException, e:
print e continue except dbus.exceptions.DBusException, e:
continue def ocmp(x,y):
if x['node'] < y['node']:
return -1 if x['node'] > y['node']:
return 1 return 0 vols.sort(cmp=ocmp)

if verbose:
print "FBSD: ", vols mtd=0 for vol in vols:
mp = ''
if vol['dev'].GetProperty('volume.is_mounted'):
mp = vol['dev'].GetProperty('volume.mount_point')
else:
try:
vol['vol'].Mount('Calibre-'+vol['label'], vol['dev'].GetProperty('volume.fstype'), [])
loops = 0 while not vol['dev'].GetProperty('volume.is_mounted'):
time.sleep(1)
loops += 1 if loops > 100:
print "ERROR: Timeout waiting for mount to complete"
continue mp = vol['dev'].GetProperty('volume.mount_point')
except dbus.exceptions.DBusException, e:
print "Failed to mount ", e continue

# Mount Point becomes Mount Path mp += '/'

if verbose:
print "FBSD: mounted", vol['label'], "on", mp if mtd == 0:
self._main_prefix = mp self._main_vol = vol['vol']
if verbose:
print "FBSD: main = ", self._main_prefix if mtd == 1:
self._card_a_prefix = mp self._card_a_vol = vol['vol']
if verbose:
print "FBSD: card a = ", self._card_a_prefix if mtd == 2:
self._card_b_prefix = mp self._card_b_vol = vol['vol']
if verbose:
print "FBSD: card b = ", self._card_b_prefix
# Note that mtd is used as a bool... not incrementing is fine.
break mtd += 1 if mtd > 0:
return True raise DeviceError(_('Unable to mount the device'))

#
# ------------------------------------------------------
#
# this one is pretty simple:
# just umount each of the previously
# mounted filesystems, using the stored volume object
# def eject_freebsd(self):
import dbus
# There should be some way to access the -v arg...
verbose = False if self._main_prefix:
if verbose:
print "FBSD: umount main:", self._main_prefix try:
self._main_vol.Unmount([])
except dbus.exceptions.DBusException, e:
print 'Unable to eject ', e if self._card_a_prefix:
if verbose:
print "FBSD: umount card a:", self._card_a_prefix try:
self._card_a_vol.Unmount([])
except dbus.exceptions.DBusException, e:
print 'Unable to eject ', e if self._card_b_prefix:
if verbose:
print "FBSD: umount card b:", self._card_b_prefix try:
self._card_b_vol.Unmount([])
except dbus.exceptions.DBusException, e:
print 'Unable to eject ', e self._main_prefix = None self._card_a_prefix = None self._card_b_prefix = None
# ------------------------------------------------------ def open(self, connected_device, library_uuid):
time.sleep(5)
self._main_prefix = self._card_a_prefix = self._card_b_prefix = None self.device_being_opened = connected_device try:
if islinux:
try:
self.open_linux()
except DeviceError:
time.sleep(7)
self.open_linux()
if isfreebsd:
self._main_vol = self._card_a_vol = self._card_b_vol = None try:
self.open_freebsd()
except DeviceError:
time.sleep(2)
self.open_freebsd()
if iswindows:
try:
self.open_windows()
except DeviceError:
time.sleep(7)
self.open_windows()
if isosx:
try:
self.open_osx()
except DeviceError:
time.sleep(7)
self.open_osx()

self.current_library_uuid = library_uuid self.post_open_callback()
finally:
self.device_being_opened = None def post_open_callback(self):
pass def eject_windows(self):
from calibre.constants import plugins from threading import Thread winutil, winutil_err = plugins['winutil']
drives = []
for x in ('_main_prefix', '_card_a_prefix', '_card_b_prefix'):
x = getattr(self, x, None)
if x is not None:
drives.append(x[0].upper())

def do_it(drives):
for d in drives:
try:
winutil.eject_drive(bytes(d)[0])
except Exception as e:
try:
prints("Eject failed:", as_unicode(e))
except:
pass t = Thread(target=do_it, args=[drives])
t.daemon = True t.start()
self.__save_win_eject_thread = t def eject_osx(self):
for x in ('_main_prefix', '_card_a_prefix', '_card_b_prefix'):
x = getattr(self, x, None)
if x is not None:
try:
subprocess.Popen(self.OSX_EJECT_COMMAND + [x])
except:
pass def eject_linux(self):
from calibre.devices.udisks import eject, umount drives = [d for d in self.find_device_nodes() if d]
for d in drives:
try:
umount(d)
except:
pass for d in drives:
try:
eject(d)
except Exception as e:
print 'Udisks eject call for:', d, 'failed:'
print '\t', e def eject(self):
if islinux:
try:
self.eject_linux()
except:
pass if isfreebsd:
try:
self.eject_freebsd()
except:
pass if iswindows:
try:
self.eject_windows()
except:
pass if isosx:
try:
self.eject_osx()
except:
pass self._main_prefix = self._card_a_prefix = self._card_b_prefix = None def linux_post_yank(self):
self._linux_mount_map = {}

def post_yank_cleanup(self):
if islinux:
try:
self.linux_post_yank()
except:
import traceback traceback.print_exc()
self._main_prefix = self._card_a_prefix = self._card_b_prefix = None def get_main_ebook_dir(self, for_upload=False):
return self.EBOOK_DIR_MAIN def get_carda_ebook_dir(self, for_upload=False):
return self.EBOOK_DIR_CARD_A def get_cardb_ebook_dir(self, for_upload=False):
return self.EBOOK_DIR_CARD_B def _sanity_check(self, on_card, files):
from calibre.devices.utils import sanity_check sanity_check(on_card, files, self.card_prefix(), self.free_space())

def get_dest_dir(prefix, candidates):
if isinstance(candidates, basestring):
candidates = [candidates]
if not candidates:
candidates = ['']
candidates = [
((os.path.join(prefix, *(x.split('/')))) if x else prefix)
for x in candidates]
existing = [x for x in candidates if os.path.exists(x)]
if not existing:
existing = candidates return existing[0]

if on_card == 'carda':
candidates = self.get_carda_ebook_dir(for_upload=True)
path = get_dest_dir(self._card_a_prefix, candidates)
elif on_card == 'cardb':
candidates = self.get_cardb_ebook_dir(for_upload=True)
path = get_dest_dir(self._card_b_prefix, candidates)
else:
candidates = self.get_main_ebook_dir(for_upload=True)
path = get_dest_dir(self._main_prefix, candidates)

return path

[docs] def filename_callback(self, default, mi):
'''
Callback to allow drivers to change the default file name set by :meth:`create_upload_path`.
'''
return default

[docs] def sanitize_path_components(self, components):
'''
Perform any device specific sanitization on the path components for files to be uploaded to the device
'''
return components

[docs] def get_annotations(self, path_map):
'''
Resolve path_map to annotation_map of files found on the device
'''
return {}

[docs] def add_annotation_to_library(self, db, db_id, annotation):
'''
Add an annotation to the calibre library
'''
pass
def create_upload_path(self, path, mdata, fname, create_dirs=True):
from calibre.devices.utils import create_upload_path settings = self.settings()
filepath = create_upload_path(mdata, fname, self.save_template(), sanitize, prefix_path=os.path.abspath(path), maxlen=self.MAX_PATH_LEN, use_subdirs = self.SUPPORTS_SUB_DIRS and settings.use_subdirs, news_in_folder = self.NEWS_IN_FOLDER, filename_callback=self.filename_callback, sanitize_path_components=self.sanitize_path_components
)
filedir = os.path.dirname(filepath)

if create_dirs and not os.path.exists(filedir):
os.makedirs(filedir)

return filepath def create_annotations_path(self, mdata, device_path=None):
return self.create_upload_path(os.path.abspath('/<storage>'), mdata, 'x.bookmark', create_dirs=False)

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.devices.usbms.driver

# -*- coding: utf-8 -*-

__license__ = 'GPL v3'
__copyright__ = '2009, John Schember <john at nachtimwald.com>'
__docformat__ = 'restructuredtext en'

'''
Generic USB Mass storage device driver. This is not a complete stand alone driver. It is intended to be subclassed with the relevant parts implemented for a particular device.
'''

import os, time, json, shutil from itertools import cycle from calibre.constants import numeric_version from calibre import prints, isbytestring from calibre.constants import filesystem_encoding, DEBUG from calibre.devices.usbms.cli import CLI from calibre.devices.usbms.device import Device from calibre.devices.usbms.books import BookList, Book from calibre.ebooks.metadata.book.json_codec import JsonCodec from calibre.utils.config import from_json, to_json from calibre.utils.date import now, isoformat

BASE_TIME = None def debug_print(*args):
global BASE_TIME if BASE_TIME is None:
BASE_TIME = time.time()
if DEBUG:
prints('DEBUG: %6.1f'%(time.time()-BASE_TIME), *args)

# CLI must come before Device as it implements the CLI functions that
# are inherited from the device interface in Device.
[docs]class USBMS(CLI, Device):
'''
The base class for all USBMS devices. Implements the logic for sending/getting/updating metadata/caching metadata/etc.
'''

description = _('Communicate with an eBook reader.')
author = _('John Schember')
supported_platforms = ['windows', 'osx', 'linux']

# Store type instances of BookList and Book. We must do this because
# a) we need to override these classes in some device drivers, and
# b) the classmethods seem only to see real attributes declared in the
# class, not attributes stored in the class booklist_class = BookList book_class = Book

FORMATS = []
CAN_SET_METADATA = []
METADATA_CACHE = 'metadata.calibre'
DRIVEINFO = 'driveinfo.calibre'

SCAN_FROM_ROOT = False def _update_driveinfo_record(self, dinfo, prefix, location_code, name=None):
import uuid if not isinstance(dinfo, dict):
dinfo = {}
if dinfo.get('device_store_uuid', None) is None:
dinfo['device_store_uuid'] = unicode(uuid.uuid4())
if dinfo.get('device_name', None) is None:
dinfo['device_name'] = self.get_gui_name()
if name is not None:
dinfo['device_name'] = name dinfo['location_code'] = location_code dinfo['last_library_uuid'] = getattr(self, 'current_library_uuid', None)
dinfo['calibre_version'] = '.'.join([unicode(i) for i in numeric_version])
dinfo['date_last_connected'] = isoformat(now())
dinfo['prefix'] = prefix.replace('\\', '/')
return dinfo def _update_driveinfo_file(self, prefix, location_code, name=None):
if os.path.exists(os.path.join(prefix, self.DRIVEINFO)):
with open(os.path.join(prefix, self.DRIVEINFO), 'rb') as f:
try:
driveinfo = json.loads(f.read(), object_hook=from_json)
except:
driveinfo = None driveinfo = self._update_driveinfo_record(driveinfo, prefix, location_code, name)
with open(os.path.join(prefix, self.DRIVEINFO), 'wb') as f:
f.write(json.dumps(driveinfo, default=to_json))
else:
driveinfo = self._update_driveinfo_record({}, prefix, location_code, name)
with open(os.path.join(prefix, self.DRIVEINFO), 'wb') as f:
f.write(json.dumps(driveinfo, default=to_json))
return driveinfo def get_device_information(self, end_session=True):
self.report_progress(1.0, _('Get device information...'))
self.driveinfo = {}
if self._main_prefix is not None:
try:
self.driveinfo['main'] = self._update_driveinfo_file(self._main_prefix, 'main')
except (IOError, OSError) as e:
raise IOError(_('Failed to access files in the main memory of'
' your device. You should contact the device'
' manufacturer for support. Common fixes are:'
' try a different USB cable/USB port on your computer.'
' If you device has a "Reset to factory defaults" type'
' of setting somewhere, use it. Underlying error: %s')
% e)
try:
if self._card_a_prefix is not None:
self.driveinfo['A'] = self._update_driveinfo_file(self._card_a_prefix, 'A')
if self._card_b_prefix is not None:
self.driveinfo['B'] = self._update_driveinfo_file(self._card_b_prefix, 'B')
except (IOError, OSError) as e:
raise IOError(_('Failed to access files on the SD card in your'
' device. This can happen for many reasons. The SD card may be'
' corrupted, it may be too large for your device, it may be'
' write-protected, etc. Try a different SD card, or reformat'
' your SD card using the FAT32 filesystem. Also make sure'
' there are not too many files in the root of your SD card.'
' Underlying error: %s') % e)
return (self.get_gui_name(), '', '', '', self.driveinfo)

def set_driveinfo_name(self, location_code, name):
if location_code == 'main':
self._update_driveinfo_file(self._main_prefix, location_code, name)
elif location_code == 'A':
self._update_driveinfo_file(self._card_a_prefix, location_code, name)
elif location_code == 'B':
self._update_driveinfo_file(self._card_b_prefix, location_code, name)

def formats_to_scan_for(self):
return set(self.settings().format_map) | set(self.FORMATS)

def books(self, oncard=None, end_session=True):
from calibre.ebooks.metadata.meta import path_to_ext debug_print ('USBMS: Fetching list of books from device. Device=', self.__class__.__name__,
'oncard=', oncard)

dummy_bl = self.booklist_class(None, None, None)

if oncard == 'carda' and not self._card_a_prefix:
self.report_progress(1.0, _('Getting list of books on device...'))
return dummy_bl elif oncard == 'cardb' and not self._card_b_prefix:
self.report_progress(1.0, _('Getting list of books on device...'))
return dummy_bl elif oncard and oncard != 'carda' and oncard != 'cardb':
self.report_progress(1.0, _('Getting list of books on device...'))
return dummy_bl prefix = self._card_a_prefix if oncard == 'carda' else \ self._card_b_prefix if oncard == 'cardb' \ else self._main_prefix ebook_dirs = self.get_carda_ebook_dir() if oncard == 'carda' else \ self.EBOOK_DIR_CARD_B if oncard == 'cardb' else \ self.get_main_ebook_dir()

debug_print ('USBMS: dirs are:', prefix, ebook_dirs)

# get the metadata cache bl = self.booklist_class(oncard, prefix, self.settings)
need_sync = self.parse_metadata_cache(bl, prefix, self.METADATA_CACHE)

# make a dict cache of paths so the lookup in the loop below is faster.
bl_cache = {}
for idx, b in enumerate(bl):
bl_cache[b.lpath] = idx all_formats = self.formats_to_scan_for()

def update_booklist(filename, path, prefix):
changed = False if path_to_ext(filename) in all_formats:
try:
lpath = os.path.join(path, filename).partition(self.normalize_path(prefix))[2]
if lpath.startswith(os.sep):
lpath = lpath[len(os.sep):]
lpath = lpath.replace('\\', '/')
idx = bl_cache.get(lpath, None)
if idx is not None:
bl_cache[lpath] = None if self.update_metadata_item(bl[idx]):
#print 'update_metadata_item returned true'
changed = True else:
if bl.add_book(self.book_from_path(prefix, lpath), replace_metadata=False):
changed = True except: # Probably a filename encoding error import traceback traceback.print_exc()
return changed if isinstance(ebook_dirs, basestring):
ebook_dirs = [ebook_dirs]
for ebook_dir in ebook_dirs:
ebook_dir = self.path_to_unicode(ebook_dir)
if self.SCAN_FROM_ROOT:
ebook_dir = self.normalize_path(prefix)
else:
ebook_dir = self.normalize_path( \ os.path.join(prefix, *(ebook_dir.split('/'))) \ if ebook_dir else prefix)
debug_print('USBMS: scan from root', self.SCAN_FROM_ROOT, ebook_dir)
if not os.path.exists(ebook_dir): continue
# Get all books in the ebook_dir directory if self.SUPPORTS_SUB_DIRS or self.SUPPORTS_SUB_DIRS_FOR_SCAN:
# build a list of files to check, so we can accurately report progress flist = []
for path, dirs, files in os.walk(ebook_dir):
for filename in files:
if filename != self.METADATA_CACHE:
flist.append({'filename': self.path_to_unicode(filename),
'path':self.path_to_unicode(path)})
for i, f in enumerate(flist):
self.report_progress(i/float(len(flist)), _('Getting list of books on device...'))
changed = update_booklist(f['filename'], f['path'], prefix)
if changed:
need_sync = True else:
paths = os.listdir(ebook_dir)
for i, filename in enumerate(paths):
self.report_progress((i+1) / float(len(paths)), _('Getting list of books on device...'))
changed = update_booklist(self.path_to_unicode(filename), ebook_dir, prefix)
if changed:
need_sync = True

# Remove books that are no longer in the filesystem. Cache contains
# indices into the booklist if book not in filesystem, None otherwise
# Do the operation in reverse order so indices remain valid for idx in sorted(bl_cache.itervalues(), reverse=True):
if idx is not None:
need_sync = True del bl[idx]

debug_print('USBMS: count found in cache: %d, count of files in metadata: %d, need_sync: %s' % \ (len(bl_cache), len(bl), need_sync))
if need_sync: #self.count_found_in_bl != len(bl) or need_sync:
if oncard == 'cardb':
self.sync_booklists((None, None, bl))
elif oncard == 'carda':
self.sync_booklists((None, bl, None))
else:
self.sync_booklists((bl, None, None))

self.report_progress(1.0, _('Getting list of books on device...'))
debug_print('USBMS: Finished fetching list of books from device. oncard=', oncard)
return bl def upload_books(self, files, names, on_card=None, end_session=True, metadata=None):
debug_print('USBMS: uploading %d books'%(len(files)))

path = self._sanity_check(on_card, files)

paths = []
names = iter(names)
metadata = iter(metadata)

for i, infile in enumerate(files):
mdata, fname = metadata.next(), names.next()
filepath = self.normalize_path(self.create_upload_path(path, mdata, fname))
if not hasattr(infile, 'read'):
infile = self.normalize_path(infile)
filepath = self.put_file(infile, filepath, replace_file=True)
paths.append(filepath)
try:
self.upload_cover(os.path.dirname(filepath), os.path.splitext(os.path.basename(filepath))[0], mdata, filepath)
except: # Failure to upload cover is not catastrophic import traceback traceback.print_exc()

self.report_progress((i+1) / float(len(files)), _('Transferring books to device...'))

self.report_progress(1.0, _('Transferring books to device...'))
debug_print('USBMS: finished uploading %d books'%(len(files)))
return zip(paths, cycle([on_card]))

[docs] def upload_cover(self, path, filename, metadata, filepath):
'''
Upload book cover to the device. Default implementation does nothing.

:param path: The full path to the directory where the associated book is located.
:param filename: The name of the book file without the extension.
:param metadata: metadata belonging to the book. Use metadata.thumbnail for cover
:param filepath: The full path to the ebook file

'''
pass
def add_books_to_metadata(self, locations, metadata, booklists):
debug_print('USBMS: adding metadata for %d books'%(len(metadata)))

metadata = iter(metadata)
for i, location in enumerate(locations):
self.report_progress((i+1) / float(len(locations)), _('Adding books to device metadata listing...'))
info = metadata.next()
blist = 2 if location[1] == 'cardb' else 1 if location[1] == 'carda' else 0

# Extract the correct prefix from the pathname. To do this correctly,
# we must ensure that both the prefix and the path are normalized
# so that the comparison will work. Book's __init__ will fix up
# lpath, so we don't need to worry about that here.
path = self.normalize_path(location[0])
if self._main_prefix:
prefix = self._main_prefix if \ path.startswith(self.normalize_path(self._main_prefix)) else None if not prefix and self._card_a_prefix:
prefix = self._card_a_prefix if \ path.startswith(self.normalize_path(self._card_a_prefix)) else None if not prefix and self._card_b_prefix:
prefix = self._card_b_prefix if \ path.startswith(self.normalize_path(self._card_b_prefix)) else None if prefix is None:
prints('in add_books_to_metadata. Prefix is None!', path, self._main_prefix)
continue lpath = path.partition(prefix)[2]
if lpath.startswith('/') or lpath.startswith('\\'):
lpath = lpath[1:]
book = self.book_class(prefix, lpath, other=info)
if book.size is None:
book.size = os.stat(self.normalize_path(path)).st_size b = booklists[blist].add_book(book, replace_metadata=True)
if b:
b._new_book = True self.report_progress(1.0, _('Adding books to device metadata listing...'))
debug_print('USBMS: finished adding metadata')

def delete_books(self, paths, end_session=True):
debug_print('USBMS: deleting %d books'%(len(paths)))
for i, path in enumerate(paths):
self.report_progress((i+1) / float(len(paths)), _('Removing books from device...'))
path = self.normalize_path(path)
if os.path.exists(path):
# Delete the ebook os.unlink(path)

filepath = os.path.splitext(path)[0]
for ext in self.DELETE_EXTS:
for x in (filepath, path):
x += ext if os.path.exists(x):
if os.path.isdir(x):
shutil.rmtree(x, ignore_errors=True)
else:
os.unlink(x)

if self.SUPPORTS_SUB_DIRS:
try:
os.removedirs(os.path.dirname(path))
except:
pass self.report_progress(1.0, _('Removing books from device...'))
debug_print('USBMS: finished deleting %d books'%(len(paths)))

def remove_books_from_metadata(self, paths, booklists):
debug_print('USBMS: removing metadata for %d books'%(len(paths)))

for i, path in enumerate(paths):
self.report_progress((i+1) / float(len(paths)), _('Removing books from device metadata listing...'))
for bl in booklists:
for book in bl:
if path.endswith(book.path):
bl.remove_book(book)
self.report_progress(1.0, _('Removing books from device metadata listing...'))
debug_print('USBMS: finished removing metadata for %d books'%(len(paths)))

# If you override this method and you use book._new_book, then you must
# complete the processing before you call this method. The flag is cleared
# at the end just before the return def sync_booklists(self, booklists, end_session=True):
debug_print('USBMS: starting sync_booklists')
json_codec = JsonCodec()

if not os.path.exists(self.normalize_path(self._main_prefix)):
os.makedirs(self.normalize_path(self._main_prefix))

def write_prefix(prefix, listid):
if (prefix is not None and len(booklists) > listid and isinstance(booklists[listid], self.booklist_class)):
if not os.path.exists(prefix):
os.makedirs(self.normalize_path(prefix))
with open(self.normalize_path(os.path.join(prefix, self.METADATA_CACHE)), 'wb') as f:
json_codec.encode_to_file(f, booklists[listid])
write_prefix(self._main_prefix, 0)
write_prefix(self._card_a_prefix, 1)
write_prefix(self._card_b_prefix, 2)

# Clear the _new_book indication, as we are supposed to be done with
# adding books at this point for blist in booklists:
if blist is not None:
for book in blist:
book._new_book = False self.report_progress(1.0, _('Sending metadata to device...'))
debug_print('USBMS: finished sync_booklists')

@classmethod def build_template_regexp(cls):
from calibre.devices.utils import build_template_regexp return build_template_regexp(cls.save_template())

@classmethod def path_to_unicode(cls, path):
if isbytestring(path):
path = path.decode(filesystem_encoding)
return path

@classmethod

[docs] def normalize_path(cls, path):
'Return path with platform native path separators'
if path is None:
return None if os.sep == '\\':
path = path.replace('/', '\\')
else:
path = path.replace('\\', '/')
return cls.path_to_unicode(path)

@classmethod def parse_metadata_cache(cls, bl, prefix, name):
json_codec = JsonCodec()
need_sync = False cache_file = cls.normalize_path(os.path.join(prefix, name))
if os.access(cache_file, os.R_OK):
try:
with open(cache_file, 'rb') as f:
json_codec.decode_from_file(f, bl, cls.book_class, prefix)
except:
import traceback traceback.print_exc()
bl = []
need_sync = True else:
need_sync = True return need_sync

@classmethod def update_metadata_item(cls, book):
changed = False size = os.stat(cls.normalize_path(book.path)).st_size if size != book.size:
changed = True mi = cls.metadata_from_path(book.path)
book.smart_update(mi)
book.size = size return changed

@classmethod def metadata_from_path(cls, path):
return cls.metadata_from_formats([path])

@classmethod def metadata_from_formats(cls, fmts):
from calibre.ebooks.metadata.meta import metadata_from_formats from calibre.customize.ui import quick_metadata with quick_metadata:
return metadata_from_formats(fmts, force_read_metadata=True, pattern=cls.build_template_regexp())

@classmethod def book_from_path(cls, prefix, lpath):
from calibre.ebooks.metadata.book.base import Metadata if cls.settings().read_metadata or cls.MUST_READ_METADATA:
mi = cls.metadata_from_path(cls.normalize_path(os.path.join(prefix, lpath)))
else:
from calibre.ebooks.metadata.meta import metadata_from_filename mi = metadata_from_filename(cls.normalize_path(os.path.basename(lpath)), cls.build_template_regexp())
if mi is None:
mi = Metadata(os.path.splitext(os.path.basename(lpath))[0],
[_('Unknown')])
size = os.stat(cls.normalize_path(os.path.join(prefix, lpath))).st_size book = cls.book_class(prefix, lpath, other=mi, size=size)
return book

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.ebooks.metadata.book.base

#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai

__license__ = 'GPL v3'
__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'

import copy, traceback from calibre import prints from calibre.constants import DEBUG from calibre.ebooks.metadata.book import (SC_COPYABLE_FIELDS,
SC_FIELDS_COPY_NOT_NULL, STANDARD_METADATA_FIELDS,
TOP_LEVEL_IDENTIFIERS, ALL_METADATA_FIELDS)
from calibre.library.field_metadata import FieldMetadata from calibre.utils.date import isoformat, format_date, parse_only_date from calibre.utils.icu import sort_key from calibre.utils.formatter import TemplateFormatter

# Special sets used to optimize the performance of getting and setting
# attributes on Metadata objects
SIMPLE_GET = frozenset(STANDARD_METADATA_FIELDS - TOP_LEVEL_IDENTIFIERS)
SIMPLE_SET = frozenset(SIMPLE_GET - {'identifiers'})

def human_readable(size, precision=2):
""" Convert a size in bytes into megabytes """
return ('%.'+str(precision)+'f'+ 'MB') % ((size/(1024.*1024.)),)

NULL_VALUES = {
'user_metadata': {},
'cover_data' : (None, None),
'tags' : [],
'identifiers' : {},
'languages' : [],
'device_collections': [],
'author_sort_map': {},
'authors' : [_('Unknown')],
'author_sort' : _('Unknown'),
'title' : _('Unknown'),
'user_categories' : {},
'author_link_map' : {},
'language' : 'und'
}

field_metadata = FieldMetadata()

class SafeFormat(TemplateFormatter):

def get_value(self, orig_key, args, kwargs):
if not orig_key:
return ''
key = orig_key = orig_key.lower()
if key != 'title_sort' and key not in TOP_LEVEL_IDENTIFIERS and \ key not in ALL_METADATA_FIELDS:
key = field_metadata.search_term_to_field_key(key)
if key is None or (self.book and key not in self.book.all_field_keys()):
if hasattr(self.book, orig_key):
key = orig_key else:
raise ValueError(_('Value: unknown field ') + orig_key)
try:
b = self.book.get_user_metadata(key, False)
except:
b = None if b and ((b['datatype'] == 'int' and self.book.get(key, 0) == 0) or
(b['datatype'] == 'float' and self.book.get(key, 0.0) == 0.0)):
v = ''
else:
v = self.book.format_field(key, series_with_index=False)[1]
if v is None:
return ''
if v == '':
return ''
return v

# DEPRECATED. This is not thread safe. Do not use.
composite_formatter = SafeFormat()

[docs]class Metadata(object):

'''
A class representing all the metadata for a book. The various standard metadata fields are available as attributes of this object. You can also stick arbitrary attributes onto this object.

Metadata from custom columns should be accessed via the get() method, passing in the lookup name for the column, for example: "#mytags".

Use the :meth:`is_null` method to test if a field is null.

This object also has functions to format fields into strings.

The list of standard metadata fields grows with time is in
:data:`STANDARD_METADATA_FIELDS`.

Please keep the method based API of this class to a minimum. Every method becomes a reserved field name.
'''

def __init__(self, title, authors=(_('Unknown'),), other=None, template_cache=None):
'''
@param title: title or ``_('Unknown')``
@param authors: List of strings or []
@param other: None or a metadata object
'''
_data = copy.deepcopy(NULL_VALUES)
_data.pop('language')
object.__setattr__(self, '_data', _data)
if other is not None:
self.smart_update(other)
else:
if title:
self.title = title if authors:
# List of strings or []
self.author = list(authors) if authors else []# Needed for backward compatibility self.authors = list(authors) if authors else []
self.formatter = SafeFormat()
self.template_cache = template_cache

[docs] def is_null(self, field):
'''
Return True if the value of field is null in this object.
'null' means it is unknown or evaluates to False. So a title of
_('Unknown') is null or a language of 'und' is null.

Be careful with numeric fields since this will return True for zero as well as None.

Also returns True if the field does not exist.
'''
try:
null_val = NULL_VALUES.get(field, None)
val = getattr(self, field, None)
return not val or val == null_val except:
return True
def __getattribute__(self, field):
_data = object.__getattribute__(self, '_data')
if field in SIMPLE_GET:
return _data.get(field, None)
if field in TOP_LEVEL_IDENTIFIERS:
return _data.get('identifiers').get(field, None)
if field == 'language':
try:
return _data.get('languages', [])[0]
except:
return NULL_VALUES['language']
try:
return object.__getattribute__(self, field)
except AttributeError:
pass if field in _data['user_metadata'].iterkeys():
d = _data['user_metadata'][field]
val = d['#value#']
if d['datatype'] != 'composite':
return val if val is None:
d['#value#'] = 'RECURSIVE_COMPOSITE FIELD (Metadata) ' + field val = d['#value#'] = self.formatter.safe_format( d['display']['composite_template'], self,
_('TEMPLATE ERROR'), self, column_name=field, template_cache=self.template_cache).strip()
return val if field.startswith('#') and field.endswith('_index'):
try:
return self.get_extra(field[:-6])
except:
pass raise AttributeError(
'Metadata object has no attribute named: '+ repr(field))

def __setattr__(self, field, val, extra=None):
_data = object.__getattribute__(self, '_data')
if field in SIMPLE_SET:
if val is None:
val = copy.copy(NULL_VALUES.get(field, None))
_data[field] = val elif field in TOP_LEVEL_IDENTIFIERS:
field, val = self._clean_identifier(field, val)
identifiers = _data['identifiers']
identifiers.pop(field, None)
if val:
identifiers[field] = val elif field == 'identifiers':
if not val:
val = copy.copy(NULL_VALUES.get('identifiers', None))
self.set_identifiers(val)
elif field == 'language':
langs = []
if val and val.lower() != 'und':
langs = [val]
_data['languages'] = langs elif field in _data['user_metadata'].iterkeys():
_data['user_metadata'][field]['#value#'] = val
_data['user_metadata'][field]['#extra#'] = extra else:
# You are allowed to stick arbitrary attributes onto this object as
# long as they don't conflict with global or user metadata names
# Don't abuse this privilege self.__dict__[field] = val def __iter__(self):
return object.__getattribute__(self, '_data').iterkeys()

def has_key(self, key):
return key in object.__getattribute__(self, '_data')

def deepcopy(self):
m = Metadata(None)
m.__dict__ = copy.deepcopy(self.__dict__)
object.__setattr__(m, '_data', copy.deepcopy(object.__getattribute__(self, '_data')))
return m def deepcopy_metadata(self):
m = Metadata(None)
object.__setattr__(m, '_data', copy.deepcopy(object.__getattribute__(self, '_data')))
return m def get(self, field, default=None):
try:
return self.__getattribute__(field)
except AttributeError:
return default def get_extra(self, field, default=None):
_data = object.__getattribute__(self, '_data')
if field in _data['user_metadata'].iterkeys():
try:
return _data['user_metadata'][field]['#extra#']
except:
return default raise AttributeError(
'Metadata object has no attribute named: '+ repr(field))

def set(self, field, val, extra=None):
self.__setattr__(field, val, extra)

[docs] def get_identifiers(self):
'''
Return a copy of the identifiers dictionary.
The dict is small, and the penalty for using a reference where a copy is needed is large. Also, we don't want any manipulations of the returned dict to show up in the book.
'''
ans = object.__getattribute__(self,
'_data')['identifiers']
if not ans:
ans = {}
return copy.deepcopy(ans)

def _clean_identifier(self, typ, val):
if typ:
typ = icu_lower(typ).strip().replace(':', '').replace(',', '')
if val:
val = val.strip().replace(',', '|').replace(':', '|')
return typ, val

[docs] def set_identifiers(self, identifiers):
'''
Set all identifiers. Note that if you previously set ISBN, calling this method will delete it.
'''
cleaned = {}
for key, val in identifiers.iteritems():
key, val = self._clean_identifier(key, val)
if key and val:
cleaned[key] = val object.__getattribute__(self, '_data')['identifiers'] = cleaned

[docs] def set_identifier(self, typ, val):
'If val is empty, deletes identifier of type typ'
typ, val = self._clean_identifier(typ, val)
if not typ:
return identifiers = object.__getattribute__(self,
'_data')['identifiers']

identifiers.pop(typ, None)
if val:
identifiers[typ] = val
def has_identifier(self, typ):
identifiers = object.__getattribute__(self,
'_data')['identifiers']
return typ in identifiers

# field-oriented interface. Intended to be the same as in LibraryDatabase

[docs] def standard_field_keys(self):
'''
return a list of all possible keys, even if this book doesn't have them
'''
return STANDARD_METADATA_FIELDS

[docs] def custom_field_keys(self):
'''
return a list of the custom fields in this book
'''
return object.__getattribute__(self, '_data')['user_metadata'].iterkeys()

[docs] def all_field_keys(self):
'''
All field keys known by this instance, even if their value is None
'''
_data = object.__getattribute__(self, '_data')
return frozenset(ALL_METADATA_FIELDS.union(_data['user_metadata'].iterkeys()))

[docs] def metadata_for_field(self, key):
'''
return metadata describing a standard or custom field.
'''
if key not in self.custom_field_keys():
return self.get_standard_metadata(key, make_copy=False)
return self.get_user_metadata(key, make_copy=False)

[docs] def all_non_none_fields(self):
'''
Return a dictionary containing all non-None metadata fields, including the custom ones.
'''
result = {}
_data = object.__getattribute__(self, '_data')
for attr in STANDARD_METADATA_FIELDS:
v = _data.get(attr, None)
if v is not None:
result[attr] = v
# separate these because it uses the self.get(), not _data.get()
for attr in TOP_LEVEL_IDENTIFIERS:
v = self.get(attr, None)
if v is not None:
result[attr] = v for attr in _data['user_metadata'].iterkeys():
v = self.get(attr, None)
if v is not None:
result[attr] = v if _data['user_metadata'][attr]['datatype'] == 'series':
result[attr+'_index'] = _data['user_metadata'][attr]['#extra#']
return result

# End of field-oriented interface

# Extended interfaces. These permit one to get copies of metadata dictionaries, and to
# get and set custom field metadata

[docs] def get_standard_metadata(self, field, make_copy):
'''
return field metadata from the field if it is there. Otherwise return
None. field is the key name, not the label. Return a copy if requested, just in case the user wants to change values in the dict.
'''
if field in field_metadata and field_metadata[field]['kind'] == 'field':
if make_copy:
return copy.deepcopy(field_metadata[field])
return field_metadata[field]
return None

[docs] def get_all_standard_metadata(self, make_copy):
'''
return a dict containing all the standard field metadata associated with the book.
'''
if not make_copy:
return field_metadata res = {}
for k in field_metadata:
if field_metadata[k]['kind'] == 'field':
res[k] = copy.deepcopy(field_metadata[k])
return res

[docs] def get_all_user_metadata(self, make_copy):
'''
return a dict containing all the custom field metadata associated with the book.
'''
_data = object.__getattribute__(self, '_data')
user_metadata = _data['user_metadata']
if not make_copy:
return user_metadata res = {}
for k in user_metadata:
res[k] = copy.deepcopy(user_metadata[k])
return res

[docs] def get_user_metadata(self, field, make_copy):
'''
return field metadata from the object if it is there. Otherwise return
None. field is the key name, not the label. Return a copy if requested, just in case the user wants to change values in the dict.
'''
_data = object.__getattribute__(self, '_data')
_data = _data['user_metadata']
if field in _data:
if make_copy:
return copy.deepcopy(_data[field])
return _data[field]
return None

[docs] def set_all_user_metadata(self, metadata):
'''
store custom field metadata into the object. Field is the key name not the label
'''
if metadata is None:
traceback.print_stack()
return um = {}
for key, meta in metadata.iteritems():
m = meta.copy()
if '#value#' not in m:
if m['datatype'] == 'text' and m['is_multiple']:
m['#value#'] = []
else:
m['#value#'] = None um[key] = m
_data = object.__getattribute__(self, '_data')
_data['user_metadata'].update(um)

[docs] def set_user_metadata(self, field, metadata):
'''
store custom field metadata for one column into the object. Field is the key name not the label
'''
if field is not None:
if not field.startswith('#'):
raise AttributeError(
'Custom field name %s must begin with \'#\''%repr(field))
if metadata is None:
traceback.print_stack()
return m = dict(metadata)
# Copying the elements should not be necessary. The objects referenced
# in the dict should not change. Of course, they can be replaced.
# for k,v in metadata.iteritems():
# m[k] = copy.copy(v)
if '#value#' not in m:
if m['datatype'] == 'text' and m['is_multiple']:
m['#value#'] = []
else:
m['#value#'] = None
_data = object.__getattribute__(self, '_data')
_data['user_metadata'][field] = m

[docs] def template_to_attribute(self, other, ops):
'''
Takes a list [(src,dest), (src,dest)], evaluates the template in the context of other, then copies the result to self[dest]. This is on a best-efforts basis. Some assignments can make no sense.
'''
if not ops:
return formatter = SafeFormat()
for op in ops:
try:
src = op[0]
dest = op[1]
val = formatter.safe_format\ (src, other, 'PLUGBOARD TEMPLATE ERROR', other)
if dest == 'tags':
self.set(dest, [f.strip() for f in val.split(',') if f.strip()])
elif dest == 'authors':
self.set(dest, [f.strip() for f in val.split('&') if f.strip()])
else:
self.set(dest, val)
except:
if DEBUG:
traceback.print_exc()

# Old Metadata API {{{
def print_all_attributes(self):
for x in STANDARD_METADATA_FIELDS:
prints('%s:'%x, getattr(self, x, 'None'))
for x in self.custom_field_keys():
meta = self.get_user_metadata(x, make_copy=False)
if meta is not None:
prints(x, meta)
prints('--------------')

[docs] def smart_update(self, other, replace_metadata=False):
'''
Merge the information in `other` into self. In case of conflicts, the information in `other` takes precedence, unless the information in `other` is NULL.
'''
def copy_not_none(dest, src, attr):
v = getattr(src, attr, None)
if v not in (None, NULL_VALUES.get(attr, None)):
setattr(dest, attr, copy.deepcopy(v))

if other.title and other.title != _('Unknown'):
self.title = other.title if hasattr(other, 'title_sort'):
self.title_sort = other.title_sort if other.authors and other.authors[0] != _('Unknown'):
self.authors = list(other.authors)
if hasattr(other, 'author_sort_map'):
self.author_sort_map = dict(other.author_sort_map)
if hasattr(other, 'author_sort'):
self.author_sort = other.author_sort if replace_metadata:
# SPECIAL_FIELDS = frozenset(['lpath', 'size', 'comments', 'thumbnail'])
for attr in SC_COPYABLE_FIELDS:
setattr(self, attr, getattr(other, attr, 1.0 if \ attr == 'series_index' else None))
self.tags = other.tags self.cover_data = getattr(other, 'cover_data',
NULL_VALUES['cover_data'])
self.set_all_user_metadata(other.get_all_user_metadata(make_copy=True))
for x in SC_FIELDS_COPY_NOT_NULL:
copy_not_none(self, other, x)
if callable(getattr(other, 'get_identifiers', None)):
self.set_identifiers(other.get_identifiers())
# language is handled below else:
for attr in SC_COPYABLE_FIELDS:
copy_not_none(self, other, attr)
for x in SC_FIELDS_COPY_NOT_NULL:
copy_not_none(self, other, x)

if other.tags:
# Case-insensitive but case preserving merging lotags = [t.lower() for t in other.tags]
lstags = [t.lower() for t in self.tags]
ot, st = map(frozenset, (lotags, lstags))
for t in st.intersection(ot):
sidx = lstags.index(t)
oidx = lotags.index(t)
self.tags[sidx] = other.tags[oidx]
self.tags += [t for t in other.tags if t.lower() in ot-st]

if getattr(other, 'cover_data', False):
other_cover = other.cover_data[-1]
self_cover = self.cover_data[-1] if self.cover_data else ''
if not self_cover: self_cover = ''
if not other_cover: other_cover = ''
if len(other_cover) > len(self_cover):
self.cover_data = other.cover_data if callable(getattr(other, 'custom_field_keys', None)):
for x in other.custom_field_keys():
meta = other.get_user_metadata(x, make_copy=True)
if meta is not None:
self_tags = self.get(x, [])
self.set_user_metadata(x, meta) # get... did the deepcopy other_tags = other.get(x, [])
if meta['datatype'] == 'text' and meta['is_multiple']:
# Case-insensitive but case preserving merging lotags = [t.lower() for t in other_tags]
try:
lstags = [t.lower() for t in self_tags]
except TypeError:
# Happens if x is not a text, is_multiple field
# on self lstags = []
self_tags = []
ot, st = map(frozenset, (lotags, lstags))
for t in st.intersection(ot):
sidx = lstags.index(t)
oidx = lotags.index(t)
self_tags[sidx] = other_tags[oidx]
self_tags += [t for t in other_tags if t.lower() in ot-st]
setattr(self, x, self_tags)

my_comments = getattr(self, 'comments', '')
other_comments = getattr(other, 'comments', '')
if not my_comments:
my_comments = ''
if not other_comments:
other_comments = ''
if len(other_comments.strip()) > len(my_comments.strip()):
self.comments = other_comments

# Copy all the non-none identifiers if callable(getattr(other, 'get_identifiers', None)):
d = self.get_identifiers()
s = other.get_identifiers()
d.update([v for v in s.iteritems() if v[1] is not None])
self.set_identifiers(d)
else:
# other structure not Metadata. Copy the top-level identifiers for attr in TOP_LEVEL_IDENTIFIERS:
copy_not_none(self, other, attr)

other_lang = getattr(other, 'languages', [])
if other_lang and other_lang != ['und']:
self.languages = list(other_lang)
if not getattr(self, 'series', None):
self.series_index = None def format_series_index(self, val=None):
from calibre.ebooks.metadata import fmt_sidx v = self.series_index if val is None else val try:
x = float(v)
except (ValueError, TypeError):
x = 1 return fmt_sidx(x)

def authors_from_string(self, raw):
from calibre.ebooks.metadata import string_to_authors self.authors = string_to_authors(raw)

def format_authors(self):
from calibre.ebooks.metadata import authors_to_string return authors_to_string(self.authors)

def format_tags(self):
return u', '.join([unicode(t) for t in sorted(self.tags, key=sort_key)])

def format_rating(self, v=None, divide_by=1.0):
if v is None:
if self.rating is not None:
return unicode(self.rating/divide_by)
return u'None'
return unicode(v/divide_by)

[docs] def format_field(self, key, series_with_index=True):
'''
Returns the tuple (display_name, formatted_value)
'''
name, val, ign, ign = self.format_field_extended(key, series_with_index)
return (name, val)

def format_field_extended(self, key, series_with_index=True):
from calibre.ebooks.metadata import authors_to_string
'''
returns the tuple (display_name, formatted_value, original_value, field_metadata)
'''

# Handle custom series index if key.startswith('#') and key.endswith('_index'):
tkey = key[:-6] # strip the _index cmeta = self.get_user_metadata(tkey, make_copy=False)
if cmeta and cmeta['datatype'] == 'series':
if self.get(tkey):
res = self.get_extra(tkey)
return (unicode(cmeta['name']+'_index'), self.format_series_index(res), res, cmeta)
else:
return (unicode(cmeta['name']+'_index'), '', '', cmeta)

if key in self.custom_field_keys():
res = self.get(key, None) # get evaluates all necessary composites cmeta = self.get_user_metadata(key, make_copy=False)
name = unicode(cmeta['name'])
if res is None or res == '': # can't check "not res" because of numeric fields return (name, res, None, None)
orig_res = res datatype = cmeta['datatype']
if datatype == 'text' and cmeta['is_multiple']:
res = cmeta['is_multiple']['list_to_ui'].join(res)
elif datatype == 'series' and series_with_index:
if self.get_extra(key) is not None:
res = res + \ ' [%s]'%self.format_series_index(val=self.get_extra(key))
elif datatype == 'datetime':
res = format_date(res, cmeta['display'].get('date_format','dd MMM yyyy'))
elif datatype == 'bool':
res = _('Yes') if res else _('No')
elif datatype == 'rating':
res = u'%.2g'%(res/2.0)
elif datatype in ['int', 'float']:
try:
fmt = cmeta['display'].get('number_format', None)
res = fmt.format(res)
except:
pass return (name, unicode(res), orig_res, cmeta)

# convert top-level ids into their value if key in TOP_LEVEL_IDENTIFIERS:
fmeta = field_metadata['identifiers']
name = key res = self.get(key, None)
return (name, res, res, fmeta)

# Translate aliases into the standard field name fmkey = field_metadata.search_term_to_field_key(key)
if fmkey in field_metadata and field_metadata[fmkey]['kind'] == 'field':
res = self.get(key, None)
fmeta = field_metadata[fmkey]
name = unicode(fmeta['name'])
if res is None or res == '':
return (name, res, None, None)
orig_res = res name = unicode(fmeta['name'])
datatype = fmeta['datatype']
if key == 'authors':
res = authors_to_string(res)
elif key == 'series_index':
res = self.format_series_index(res)
elif datatype == 'text' and fmeta['is_multiple']:
if isinstance(res, dict):
res = [k + ':' + v for k,v in res.items()]
res = fmeta['is_multiple']['list_to_ui'].join(sorted(res, key=sort_key))
elif datatype == 'series' and series_with_index:
res = res + ' [%s]'%self.format_series_index()
elif datatype == 'datetime':
res = format_date(res, fmeta['display'].get('date_format','dd MMM yyyy'))
elif datatype == 'rating':
res = u'%.2g'%(res/2.0)
elif key == 'size':
res = human_readable(res)
return (name, unicode(res), orig_res, fmeta)

return (None, None, None, None)

def __unicode__(self):
'''
A string representation of this object, suitable for printing to console
'''
from calibre.ebooks.metadata import authors_to_string ans = []
def fmt(x, y):
ans.append(u'%-20s: %s'%(unicode(x), unicode(y)))

fmt('Title', self.title)
if self.title_sort:
fmt('Title sort', self.title_sort)
if self.authors:
fmt('Author(s)', authors_to_string(self.authors) + \ ((' [' + self.author_sort + ']')
if self.author_sort and self.author_sort != _('Unknown') else ''))
if self.publisher:
fmt('Publisher', self.publisher)
if getattr(self, 'book_producer', False):
fmt('Book Producer', self.book_producer)
if self.tags:
fmt('Tags', u', '.join([unicode(t) for t in self.tags]))
if self.series:
fmt('Series', self.series + ' #%s'%self.format_series_index())
if not self.is_null('languages'):
fmt('Languages', ', '.join(self.languages))
if self.rating is not None:
fmt('Rating', (u'%.2g'%(float(self.rating)/2.0)) if self.rating else u'')
if self.timestamp is not None:
fmt('Timestamp', isoformat(self.timestamp))
if self.pubdate is not None:
fmt('Published', isoformat(self.pubdate))
if self.rights is not None:
fmt('Rights', unicode(self.rights))
if self.identifiers:
fmt('Identifiers', u', '.join(['%s:%s'%(k, v) for k, v in self.identifiers.iteritems()]))
if self.comments:
fmt('Comments', self.comments)

for key in self.custom_field_keys():
val = self.get(key, None)
if val:
(name, val) = self.format_field(key)
fmt(name, unicode(val))
return u'\n'.join(ans)

[docs] def to_html(self):
'''
A HTML representation of this object.
'''
from calibre.ebooks.metadata import authors_to_string ans = [(_('Title'), unicode(self.title))]
ans += [(_('Author(s)'), (authors_to_string(self.authors) if self.authors else _('Unknown')))]
ans += [(_('Publisher'), unicode(self.publisher))]
ans += [(_('Producer'), unicode(self.book_producer))]
ans += [(_('Comments'), unicode(self.comments))]
ans += [('ISBN', unicode(self.isbn))]
ans += [(_('Tags'), u', '.join([unicode(t) for t in self.tags]))]
if self.series:
ans += [(_('Series'), unicode(self.series) + ' #%s'%self.format_series_index())]
ans += [(_('Languages'), u', '.join(self.languages))]
if self.timestamp is not None:
ans += [(_('Timestamp'), unicode(self.timestamp.isoformat(' ')))]
if self.pubdate is not None:
ans += [(_('Published'), unicode(self.pubdate.isoformat(' ')))]
if self.rights is not None:
ans += [(_('Rights'), unicode(self.rights))]
for key in self.custom_field_keys():
val = self.get(key, None)
if val:
(name, val) = self.format_field(key)
ans += [(name, val)]
for i, x in enumerate(ans):
ans[i] = u'<tr><td><b>%s</b></td><td>%s</td></tr>'%x return u'<table>%s</table>'%u'\n'.join(ans)
def __str__(self):
return self.__unicode__().encode('utf-8')

def __nonzero__(self):
return bool(self.title or self.author or self.comments or self.tags)

# }}}
def field_from_string(field, raw, field_metadata):
''' Parse the string raw to return an object that is suitable for calling set() on a Metadata object. '''
dt = field_metadata['datatype']
val = object if dt in {'int', 'float'}:
val = int(raw) if dt == 'int' else float(raw)
elif dt == 'rating':
val = float(raw) * 2 elif dt == 'datetime':
val = parse_only_date(raw)
elif dt == 'bool':
if raw.lower() in {'true', 'yes', 'y'}:
val = True elif raw.lower() in {'false', 'no', 'n'}:
val = False else:
raise ValueError('Unknown value for %s: %s'%(field, raw))
elif dt == 'text':
ism = field_metadata['is_multiple']
if ism:
val = [x.strip() for x in raw.split(ism['ui_to_list'])]
if field == 'identifiers':
val = {x.partition(':')[0]:x.partition(':')[-1] for x in val}
elif field == 'languages':
from calibre.utils.localization import canonicalize_lang val = [canonicalize_lang(x) for x in val]
val = [x for x in val if x]
if val is object:
val = raw return val

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.ebooks.metadata.sources.base

#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai from __future__ import (unicode_literals, division, absolute_import, print_function)

__license__ = 'GPL v3'
__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'

import re, threading from future_builtins import map from calibre import browser, random_user_agent from calibre.customize import Plugin from calibre.utils.config import JSONConfig from calibre.utils.titlecase import titlecase from calibre.utils.icu import capitalize, lower, upper from calibre.ebooks.metadata import check_isbn msprefs = JSONConfig('metadata_sources/global.json')
msprefs.defaults['txt_comments'] = False msprefs.defaults['ignore_fields'] = []
msprefs.defaults['user_default_ignore_fields'] = []
msprefs.defaults['max_tags'] = 20 msprefs.defaults['wait_after_first_identify_result'] = 30 # seconds msprefs.defaults['wait_after_first_cover_result'] = 60 # seconds msprefs.defaults['swap_author_names'] = False msprefs.defaults['fewer_tags'] = True msprefs.defaults['find_first_edition_date'] = False

# Google covers are often poor quality (scans/errors) but they have high
# resolution, so they trump covers from better sources. So make sure they
# are only used if no other covers are found.
msprefs.defaults['cover_priorities'] = {'Google':2}

def create_log(ostream=None):
from calibre.utils.logging import ThreadSafeLog, FileStream log = ThreadSafeLog(level=ThreadSafeLog.DEBUG)
log.outputs = [FileStream(ostream)]
return log

# Comparing Metadata objects for relevance {{{ words = ("the", "a", "an", "of", "and")
prefix_pat = re.compile(r'^(%s)\s+'%("|".join(words)))
trailing_paren_pat = re.compile(r'\(.*\)$')
whitespace_pat = re.compile(r'\s+')

def cleanup_title(s):
if not s:
s = _('Unknown')
s = s.strip().lower()
s = prefix_pat.sub(' ', s)
s = trailing_paren_pat.sub('', s)
s = whitespace_pat.sub(' ', s)
return s.strip()

[docs]class InternalMetadataCompareKeyGen(object):

'''
Generate a sort key for comparison of the relevance of Metadata objects, given a search query. This is used only to compare results from the same metadata source, not across different sources.

The sort key ensures that an ascending order sort is a sort by order of decreasing relevance.

The algorithm is:

* Prefer results that have the same ISBN as specified in the query
* Prefer results with a cached cover URL
* Prefer results with all available fields filled in
* Prefer results that are an exact title match to the query
* Prefer results with longer comments (greater than 10% longer)
* Use the relevance of the result as reported by the metadata source's search engine
'''

def __init__(self, mi, source_plugin, title, authors, identifiers):
isbn = 1 if mi.isbn and mi.isbn == identifiers.get('isbn', None) else 2 all_fields = 1 if source_plugin.test_fields(mi) is None else 2 exact_title = 1 if title and \ cleanup_title(title) == cleanup_title(mi.title) else 2 has_cover = 2 if (not source_plugin.cached_cover_url_is_reliable or source_plugin.get_cached_cover_url(mi.identifiers) is None) else 1 self.base = (isbn, has_cover, all_fields, exact_title)
self.comments_len = len(mi.comments.strip() if mi.comments else '')
self.extra = (getattr(mi, 'source_relevance', 0), )

def __cmp__(self, other):
result = cmp(self.base, other.base)
if result == 0:
# Now prefer results with the longer comments, within 10% cx, cy = self.comments_len, other.comments_len t = (cx + cy) / 20 delta = cy - cx if abs(delta) > t:
result = delta else:
result = cmp(self.extra, other.extra)
return result

# }}}
def get_cached_cover_urls(mi):
from calibre.customize.ui import metadata_plugins plugins = list(metadata_plugins(['identify']))
for p in plugins:
url = p.get_cached_cover_url(mi.identifiers)
if url:
yield (p, url)

def dump_caches():
from calibre.customize.ui import metadata_plugins return {p.name:p.dump_caches() for p in metadata_plugins(['identify'])}

def load_caches(dump):
from calibre.customize.ui import metadata_plugins plugins = list(metadata_plugins(['identify']))
for p in plugins:
cache = dump.get(p.name, None)
if cache:
p.load_caches(cache)

def cap_author_token(token):
lt = lower(token)
if lt in ('von', 'de', 'el', 'van', 'le'):
return lt
# no digits no spez. characters if re.match(r'([^\d\W]\.){2,}$', lt, re.UNICODE) is not None:
# Normalize tokens of the form J.K. to J. K.
parts = token.split('.')
return '. '.join(map(capitalize, parts)).strip()
scots_name = None for x in ('mc', 'mac'):
if (token.lower().startswith(x) and len(token) > len(x) and
( token[len(x)] == upper(token[len(x)]) or lt == token
)):
scots_name = len(x)
break ans = capitalize(token)
if scots_name is not None:
ans = ans[:scots_name] + upper(ans[scots_name]) + ans[scots_name+1:]
for x in ('-', "'"):
idx = ans.find(x)
if idx > -1 and len(ans) > idx+2:
ans = ans[:idx+1] + upper(ans[idx+1]) + ans[idx+2:]
return ans def fixauthors(authors):
if not authors:
return authors ans = []
for x in authors:
ans.append(' '.join(map(cap_author_token, x.split())))
return ans def fixcase(x):
if x:
x = titlecase(x)
return x class Option(object):
__slots__ = ['type', 'default', 'label', 'desc', 'name', 'choices']

def __init__(self, name, type_, default, label, desc, choices=None):
'''
:param name: The name of this option. Must be a valid python identifier
:param type_: The type of this option, one of ('number', 'string',
'bool', 'choices')
:param default: The default value for this option
:param label: A short (few words) description of this option
:param desc: A longer description of this option
:param choices: A dict of possible values, used only if type='choices'.
dict is of the form {key:human readable label, ...}
'''
self.name, self.type, self.default, self.label, self.desc = (name, type_, default, label, desc)
if choices and not isinstance(choices, dict):
choices = dict([(x, x) for x in choices])
self.choices = choices

[docs]class Source(Plugin):

type = _('Metadata source')
author = 'Kovid Goyal'

supported_platforms = ['windows', 'osx', 'linux']

#: Set of capabilities supported by this plugin.
#: Useful capabilities are: 'identify', 'cover'
capabilities = frozenset()

#: List of metadata fields that can potentially be download by this plugin
#: during the identify phase touched_fields = frozenset()

#: Set this to True if your plugin return HTML formatted comments has_html_comments = False

#: Setting this to True means that the browser object will add
#: Accept-Encoding: gzip to all requests. This can speedup downloads
#: but make sure that the source actually supports gzip transfer encoding
#: correctly first supports_gzip_transfer_encoding = False

#: Cached cover URLs can sometimes be unreliable (i.e. the download could
#: fail or the returned image could be bogus. If that is often the case
#: with this source set to False cached_cover_url_is_reliable = True

#: A list of :class:`Option` objects. They will be used to automatically
#: construct the configuration widget for this plugin options = ()

#: A string that is displayed at the top of the config widget for this
#: plugin config_help_message = None def __init__(self, *args, **kwargs):
Plugin.__init__(self, *args, **kwargs)
self.running_a_test = False # Set to True when using identify_test()
self._isbn_to_identifier_cache = {}
self._identifier_to_cover_url_cache = {}
self.cache_lock = threading.RLock()
self._config_obj = None self._browser = None self.prefs.defaults['ignore_fields'] = []
for opt in self.options:
self.prefs.defaults[opt.name] = opt.default

# Configuration {{{

[docs] def is_configured(self):
'''
Return False if your plugin needs to be configured before it can be used. For example, it might need a username/password/API key.
'''
return True
def is_customizable(self):
return True def customization_help(self):
return 'This plugin can only be customized using the GUI'

def config_widget(self):
from calibre.gui2.metadata.config import ConfigWidget return ConfigWidget(self)

def save_settings(self, config_widget):
config_widget.commit()

@property def prefs(self):
if self._config_obj is None:
self._config_obj = JSONConfig('metadata_sources/%s.json'%self.name)
return self._config_obj
# }}}

# Browser {{{

@property def user_agent(self):
# Pass in an index to random_user_agent() to test with a particular
# user agent return random_user_agent()

@property def browser(self):
if self._browser is None:
self._browser = browser(user_agent=self.user_agent)
if self.supports_gzip_transfer_encoding:
self._browser.set_handle_gzip(True)
return self._browser.clone_browser()

# }}}

# Caching {{{ def get_related_isbns(self, id_):
with self.cache_lock:
for isbn, q in self._isbn_to_identifier_cache.iteritems():
if q == id_:
yield isbn def cache_isbn_to_identifier(self, isbn, identifier):
with self.cache_lock:
self._isbn_to_identifier_cache[isbn] = identifier def cached_isbn_to_identifier(self, isbn):
with self.cache_lock:
return self._isbn_to_identifier_cache.get(isbn, None)

def cache_identifier_to_cover_url(self, id_, url):
with self.cache_lock:
self._identifier_to_cover_url_cache[id_] = url def cached_identifier_to_cover_url(self, id_):
with self.cache_lock:
return self._identifier_to_cover_url_cache.get(id_, None)

def dump_caches(self):
with self.cache_lock:
return {'isbn_to_identifier':self._isbn_to_identifier_cache.copy(),
'identifier_to_cover':self._identifier_to_cover_url_cache.copy()}

def load_caches(self, dump):
with self.cache_lock:
self._isbn_to_identifier_cache.update(dump['isbn_to_identifier'])
self._identifier_to_cover_url_cache.update(dump['identifier_to_cover'])

# }}}

# Utility functions {{{

[docs] def get_author_tokens(self, authors, only_first_author=True):
'''
Take a list of authors and return a list of tokens useful for an
AND search query. This function tries to return tokens in first name middle names last name order, by assuming that if a comma is in the author name, the name is in lastname, other names form.
'''

if authors:
# Leave ' in there for Irish names remove_pat = re.compile(r'[!@#$%^&*(){}`~"\s\[\]/]')
replace_pat = re.compile(r'[-+.:;,]')
if only_first_author:
authors = authors[:1]
for au in authors:
has_comma = ',' in au au = replace_pat.sub(' ', au)
parts = au.split()
if has_comma:
# au probably in ln, fn form parts = parts[1:] + parts[:1]
for tok in parts:
tok = remove_pat.sub('', tok).strip()
if len(tok) > 2 and tok.lower() not in ('von', 'van',
_('Unknown').lower()):
yield tok

[docs] def get_title_tokens(self, title, strip_joiners=True, strip_subtitle=False):
'''
Take a title and return a list of tokens useful for an AND search query.
Excludes connectives(optionally) and punctuation.
'''
if title:
# strip sub-titles if strip_subtitle:
subtitle = re.compile(r'([\(\[\{].*?[\)\]\}]|[/:\\].*$)')
if len(subtitle.sub('', title)) > 1:
title = subtitle.sub('', title)

title_patterns = [(re.compile(pat, re.IGNORECASE), repl) for pat, repl in
[
# Remove things like: (2010) (Omnibus) etc.
(r'(?i)[({\[](\d{4}|omnibus|anthology|hardcover|audiobook|audio\scd|paperback|turtleback|mass\s*market|edition|ed\.)[\])}]', ''),
# Remove any strings that contain the substring edition inside
# parentheses
(r'(?i)[({\[].*?(edition|ed.).*?[\]})]', ''),
# Remove commas used a separators in numbers
(r'(\d+),(\d+)', r'\1\2'),
# Remove hyphens only if they have whitespace before them
(r'(\s-)', ' '),
# Remove single quotes not followed by 's'
(r"'(?!s)", ''),
# Replace other special chars with a space
(r'''[:,;+!@$%^&*(){}.`~"\s\[\]/]''', ' '),
]]

for pat, repl in title_patterns:
title = pat.sub(repl, title)

tokens = title.split()
for token in tokens:
token = token.strip()
if token and (not strip_joiners or token.lower() not in ('a',
'and', 'the', '&')):
yield token

[docs] def split_jobs(self, jobs, num):
'Split a list of jobs into at most num groups, as evenly as possible'
groups = [[] for i in range(num)]
jobs = list(jobs)
while jobs:
for gr in groups:
try:
job = jobs.pop()
except IndexError:
break gr.append(job)
return [g for g in groups if g]

[docs] def test_fields(self, mi):
'''
Return the first field from self.touched_fields that is null on the mi object
'''
for key in self.touched_fields:
if key.startswith('identifier:'):
key = key.partition(':')[-1]
if not mi.has_identifier(key):
return 'identifier: ' + key elif mi.is_null(key):
return key

[docs] def clean_downloaded_metadata(self, mi):
'''
Call this method in your plugin's identify method to normalize metadata before putting the Metadata object into result_queue. You can of course, use a custom algorithm suited to your metadata source.
'''
if mi.title:
mi.title = fixcase(mi.title)
mi.authors = fixauthors(mi.authors)
mi.tags = list(map(fixcase, mi.tags))
mi.isbn = check_isbn(mi.isbn)

# }}}

# Metadata API {{{

[docs] def get_book_url(self, identifiers):
'''
Return a 3-tuple or None. The 3-tuple is of the form:
(identifier_type, identifier_value, URL).
The URL is the URL for the book identified by identifiers at this source. identifier_type, identifier_value specify the identifier corresponding to the URL.
This URL must be browseable to by a human using a browser. It is meant to provide a clickable link for the user to easily visit the books page at this source.
If no URL is found, return None. This method must be quick, and consistent, so only implement it if it is possible to construct the URL from a known scheme given identifiers.
'''
return None

[docs] def get_book_url_name(self, idtype, idval, url):
'''
Return a human readable name from the return value of get_book_url().
'''
return self.name

[docs] def get_cached_cover_url(self, identifiers):
'''
Return cached cover URL for the book identified by the identifiers dict or None if no such URL exists.

Note that this method must only return validated URLs, i.e. not URLS that could result in a generic cover image or a not found error.
'''
return None

[docs] def identify_results_keygen(self, title=None, authors=None, identifiers={}):
'''
Return a function that is used to generate a key that can sort Metadata objects by their relevance given a search query (title, authors, identifiers).

These keys are used to sort the results of a call to :meth:`identify`.

For details on the default algorithm see
:class:`InternalMetadataCompareKeyGen`. Re-implement this function in your plugin if the default algorithm is not suitable.
'''
def keygen(mi):
return InternalMetadataCompareKeyGen(mi, self, title, authors, identifiers)
return keygen

[docs] def identify(self, log, result_queue, abort, title=None, authors=None, identifiers={}, timeout=30):
'''
Identify a book by its title/author/isbn/etc.

If identifiers(s) are specified and no match is found and this metadata source does not store all related identifiers (for example, all ISBNs of a book), this method should retry with just the title and author
(assuming they were specified).

If this metadata source also provides covers, the URL to the cover should be cached so that a subsequent call to the get covers API with the same ISBN/special identifier does not need to get the cover URL again. Use the caching API for this.

Every Metadata object put into result_queue by this method must have a
`source_relevance` attribute that is an integer indicating the order in which the results were returned by the metadata source for this query.
This integer will be used by :meth:`compare_identify_results`. If the order is unimportant, set it to zero for every result.

Make sure that any cover/isbn mapping information is cached before the
Metadata object is put into result_queue.

:param log: A log object, use it to output debugging information/errors
:param result_queue: A result Queue, results should be put into it.
Each result is a Metadata object
:param abort: If abort.is_set() returns True, abort further processing and return as soon as possible
:param title: The title of the book, can be None
:param authors: A list of authors of the book, can be None
:param identifiers: A dictionary of other identifiers, most commonly
{'isbn':'1234...'}
:param timeout: Timeout in seconds, no network request should hang for longer than timeout.
:return: None if no errors occurred, otherwise a unicode representation of the error suitable for showing to the user

'''
return None

[docs] def download_cover(self, log, result_queue, abort, title=None, authors=None, identifiers={}, timeout=30):
'''
Download a cover and put it into result_queue. The parameters all have the same meaning as for :meth:`identify`. Put (self, cover_data) into result_queue.

This method should use cached cover URLs for efficiency whenever possible. When cached data is not present, most plugins simply call identify and use its results.
'''
pass

# }}}

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.gui2.actions

#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai

__license__ = 'GPL v3'
__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'

from functools import partial from zipfile import ZipFile from PyQt4.Qt import (QToolButton, QAction, QIcon, QObject, QMenu,
QKeySequence)

from calibre import prints from calibre.gui2 import Dispatcher from calibre.gui2.keyboard import NameConflict def menu_action_unique_name(plugin, unique_name):
return u'%s : menu action : %s'%(plugin.unique_name, unique_name)

[docs]class InterfaceAction(QObject):

'''
A plugin representing an "action" that can be taken in the graphical user interface. All the items in the toolbar and context menus are implemented by these plugins.

Note that this class is the base class for these plugins, however, to integrate the plugin with calibre's plugin system, you have to make a wrapper class that references the actual plugin. See the
:mod:`calibre.customize.builtins` module for examples.

If two :class:`InterfaceAction` objects have the same name, the one with higher priority takes precedence.

Sub-classes should implement the :meth:`genesis`, :meth:`library_changed`,
:meth:`location_selected` :meth:`shutting_down` and :meth:`initialization_complete` methods.

Once initialized, this plugin has access to the main calibre GUI via the
:attr:`gui` member. You can access other plugins by name, for example::

self.gui.iactions['Save To Disk']

To access the actual plugin, use the :attr:`interface_action_base_plugin` attribute, this attribute only becomes available after the plugin has been initialized. Useful if you want to use methods from the plugin class like do_user_config().

The QAction specified by :attr:`action_spec` is automatically create and made available as ``self.qaction``.

'''

#: The plugin name. If two plugins with the same name are present, the one
#: with higher priority takes precedence.
name = 'Implement me'

#: The plugin priority. If two plugins with the same name are present, the one
#: with higher priority takes precedence.
priority = 1

#: The menu popup type for when this plugin is added to a toolbar popup_type = QToolButton.MenuButtonPopup

#: Whether this action should be auto repeated when its shortcut
#: key is held down.
auto_repeat = False

#: Of the form: (text, icon_path, tooltip, keyboard shortcut)
#: icon, tooltip and keyboard shortcut can be None
#: shortcut must be a string, None or tuple of shortcuts.
#: If None, a keyboard shortcut corresponding to the action is not
#: registered. If you pass an empty tuple, then the shortcut is registered
#: with no default key binding.
action_spec = ('text', 'icon', None, None)

#: If True, a menu is automatically created and added to self.qaction action_add_menu = False

#: If True, a clone of self.qaction is added to the menu of self.qaction
#: If you want the text of this action to be different from that of
#: self.qaction, set this variable to the new text action_menu_clone_qaction = False

#: Set of locations to which this action must not be added.
#: See :attr:`all_locations` for a list of possible locations dont_add_to = frozenset([])

#: Set of locations from which this action must not be removed.
#: See :attr:`all_locations` for a list of possible locations dont_remove_from = frozenset([])

all_locations = frozenset(['toolbar', 'toolbar-device', 'context-menu',
'context-menu-device', 'toolbar-child', 'menubar', 'menubar-device',
'context-menu-cover-browser'])

#: Type of action
#: 'current' means acts on the current view
#: 'global' means an action that does not act on the current view, but rather
#: on calibre as a whole action_type = 'global'

def __init__(self, parent, site_customization):
QObject.__init__(self, parent)
self.setObjectName(self.name)
self.gui = parent self.site_customization = site_customization self.interface_action_base_plugin = None def do_genesis(self):
self.Dispatcher = partial(Dispatcher, parent=self)
self.create_action()
self.gui.addAction(self.qaction)
self.gui.addAction(self.menuless_qaction)
self.genesis()

@property def unique_name(self):
bn = self.__class__.__name__ if getattr(self.interface_action_base_plugin, 'name'):
bn = self.interface_action_base_plugin.name return u'Interface Action: %s (%s)'%(bn, self.name)

def create_action(self, spec=None, attr='qaction', shortcut_name=None):
if spec is None:
spec = self.action_spec text, icon, tooltip, shortcut = spec if icon is not None:
action = QAction(QIcon(I(icon)), text, self.gui)
else:
action = QAction(text, self.gui)
if attr == 'qaction':
mt = (action.text() if self.action_menu_clone_qaction is True else unicode(self.action_menu_clone_qaction))
self.menuless_qaction = ma = QAction(action.icon(), mt, self.gui)
ma.triggered.connect(action.trigger)
for a in ((action, ma) if attr == 'qaction' else (action,)):
a.setAutoRepeat(self.auto_repeat)
text = tooltip if tooltip else text a.setToolTip(text)
a.setStatusTip(text)
a.setWhatsThis(text)
shortcut_action = action desc = tooltip if tooltip else None if attr == 'qaction':
shortcut_action = ma if shortcut is not None:
keys = ((shortcut,) if isinstance(shortcut, basestring) else tuple(shortcut))
if shortcut_name is None and spec[0]:
shortcut_name = unicode(spec[0])

if shortcut_name and self.action_spec[0] and not ( attr == 'qaction' and self.popup_type == QToolButton.InstantPopup):
try:
self.gui.keyboard.register_shortcut(self.unique_name + ' - ' + attr, shortcut_name, default_keys=keys, action=shortcut_action, description=desc, group=self.action_spec[0])
except NameConflict as e:
try:
prints(unicode(e))
except:
pass shortcut_action.setShortcuts([QKeySequence(key,
QKeySequence.PortableText) for key in keys])

if attr is not None:
setattr(self, attr, action)
if attr == 'qaction' and self.action_add_menu:
menu = QMenu()
action.setMenu(menu)
if self.action_menu_clone_qaction:
menu.addAction(self.menuless_qaction)
return action

[docs] def create_menu_action(self, menu, unique_name, text, icon=None, shortcut=None, description=None, triggered=None, shortcut_name=None):
'''
Convenience method to easily add actions to a QMenu.
Returns the created QAction, This action has one extra attribute calibre_shortcut_unique_name which if not None refers to the unique name under which this action is registered with the keyboard manager.

:param menu: The QMenu the newly created action will be added to
:param unique_name: A unique name for this action, this must be globally unique, so make it as descriptive as possible. If in doubt add a uuid to it.
:param text: The text of the action.
:param icon: Either a QIcon or a file name. The file name is passed to the I() builtin, so you do not need to pass the full path to the images directory.
:param shortcut: A string, a list of strings, None or False. If False, no keyboard shortcut is registered for this action. If None, a keyboard shortcut with no default keybinding is registered. String and list of strings register a shortcut with default keybinding as specified.
:param description: A description for this action. Used to set tooltips.
:param triggered: A callable which is connected to the triggered signal of the created action.
:param shortcut_name: The test displayed to the user when customizing the keyboard shortcuts for this action. By default it is set to the value of ``text``.

'''
if shortcut_name is None:
shortcut_name = unicode(text)
ac = menu.addAction(text)
if icon is not None:
if not isinstance(icon, QIcon):
icon = QIcon(I(icon))
ac.setIcon(icon)
keys = ()
if shortcut is not None and shortcut is not False:
keys = ((shortcut,) if isinstance(shortcut, basestring) else tuple(shortcut))
unique_name = menu_action_unique_name(self, unique_name)
if description is not None:
ac.setToolTip(description)
ac.setStatusTip(description)
ac.setWhatsThis(description)

ac.calibre_shortcut_unique_name = unique_name if shortcut is not False:
self.gui.keyboard.register_shortcut(unique_name, shortcut_name, default_keys=keys, action=ac, description=description, group=self.action_spec[0])
if triggered is not None:
ac.triggered.connect(triggered)
return ac

[docs] def load_resources(self, names):
'''
If this plugin comes in a ZIP file (user added plugin), this method will allow you to load resources from the ZIP file.

For example to load an image::

pixmap = QPixmap()
pixmap.loadFromData(self.load_resources(['images/icon.png']).itervalues().next())
icon = QIcon(pixmap)

:param names: List of paths to resources in the zip file using / as separator

:return: A dictionary of the form ``{name : file_contents}``. Any names that were not found in the zip file will not be present in the dictionary.

'''
if self.plugin_path is None:
raise ValueError('This plugin was not loaded from a ZIP file')
ans = {}
with ZipFile(self.plugin_path, 'r') as zf:
for candidate in zf.namelist():
if candidate in names:
ans[candidate] = zf.read(candidate)
return ans

[docs] def genesis(self):
'''
Setup this plugin. Only called once during initialization. self.gui is available. The action specified by :attr:`action_spec` is available as
``self.qaction``.
'''
pass

[docs] def location_selected(self, loc):
'''
Called whenever the book list being displayed in calibre changes.
Currently values for loc are: ``library, main, card and cardb``.

This method should enable/disable this action and its sub actions as appropriate for the location.
'''
pass

[docs] def library_changed(self, db):
'''
Called whenever the current library is changed.

:param db: The LibraryDatabase corresponding to the current library.

'''
pass

[docs] def gui_layout_complete(self):
'''
Called once per action when the layout of the main GUI is completed. If your action needs to make changes to the layout, they should be done here, rather than in :meth:`initialization_complete`.
'''
pass

[docs] def initialization_complete(self):
'''
Called once per action when the initialization of the main GUI is completed.
'''
pass

[docs] def shutting_down(self):
'''
Called once per plugin when the main GUI is in the process of shutting down. Release any used resources, but try not to block the shutdown for long periods of time.

:return: False to halt the shutdown. You are responsible for telling the user why the shutdown was halted.

'''
return True

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.gui2.preferences

#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai

__license__ = 'GPL v3'
__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'

import textwrap from PyQt4.Qt import (QWidget, pyqtSignal, QCheckBox, QAbstractSpinBox,
QLineEdit, QComboBox, QVariant, Qt, QIcon, QDialog, QVBoxLayout,
QDialogButtonBox)

from calibre.customize.ui import preferences_plugins from calibre.utils.config import ConfigProxy from calibre.gui2.complete2 import EditWithComplete class AbortCommit(Exception):
pass

[docs]class ConfigWidgetInterface(object):

'''
This class defines the interface that all widgets displayed in the
Preferences dialog must implement. See :class:`ConfigWidgetBase` for a base class that implements this interface and defines various convenience methods as well.
'''

#: This signal must be emitted whenever the user changes a value in this
#: widget changed_signal = None

#: Set to True iff the :meth:`restore_to_defaults` method is implemented.
supports_restoring_to_defaults = True

#: The tooltip for the Restore to defaults button restore_defaults_desc = _('Restore settings to default values. '
'You have to click Apply to actually save the default settings.')

#: If True the Preferences dialog will not allow the user to set any more
#: preferences. Only has effect if :meth:`commit` returns True.
restart_critical = False

[docs] def genesis(self, gui):
'''
Called once before the widget is displayed, should perform any necessary setup.

:param gui: The main calibre graphical user interface
'''
raise NotImplementedError()

[docs] def initialize(self):
'''
Should set all config values to their initial values (the values stored in the config files).
'''
raise NotImplementedError()

[docs] def restore_defaults(self):
'''
Should set all config values to their defaults.
'''
pass

[docs] def commit(self):
'''
Save any changed settings. Return True if the changes require a restart, False otherwise. Raise an :class:`AbortCommit` exception to indicate that an error occurred. You are responsible for giving the user feedback about what the error is and how to correct it.
'''
return False

[docs] def refresh_gui(self, gui):
'''
Called once after this widget is committed. Responsible for causing the gui to reread any changed settings. Note that by default the GUI re-initializes various elements anyway, so most widgets won't need to use this method.
'''
pass

class Setting(object):

CHOICES_SEARCH_FLAGS = Qt.MatchExactly | Qt.MatchCaseSensitive def __init__(self, name, config_obj, widget, gui_name=None, empty_string_is_None=True, choices=None, restart_required=False):
self.name, self.gui_name = name, gui_name self.empty_string_is_None = empty_string_is_None self.restart_required = restart_required self.choices = choices if gui_name is None:
self.gui_name = 'opt_'+name self.config_obj = config_obj self.gui_obj = getattr(widget, self.gui_name)
self.widget = widget if isinstance(self.gui_obj, QCheckBox):
self.datatype = 'bool'
self.gui_obj.stateChanged.connect(self.changed)
elif isinstance(self.gui_obj, QAbstractSpinBox):
self.datatype = 'number'
self.gui_obj.valueChanged.connect(self.changed)
elif isinstance(self.gui_obj, QLineEdit):
self.datatype = 'string'
self.gui_obj.textChanged.connect(self.changed)
elif isinstance(self.gui_obj, QComboBox):
self.datatype = 'choice'
self.gui_obj.editTextChanged.connect(self.changed)
self.gui_obj.currentIndexChanged.connect(self.changed)
else:
raise ValueError('Unknown data type')

if isinstance(self.config_obj, ConfigProxy) and \ not unicode(self.gui_obj.toolTip()):
h = self.config_obj.help(self.name)
if h:
self.gui_obj.setToolTip(h)
tt = unicode(self.gui_obj.toolTip())
if tt:
if not unicode(self.gui_obj.whatsThis()):
self.gui_obj.setWhatsThis(tt)
if not unicode(self.gui_obj.statusTip()):
self.gui_obj.setStatusTip(tt)
tt = '\n'.join(textwrap.wrap(tt, 70))
self.gui_obj.setToolTip(tt)

def changed(self, *args):
self.widget.changed_signal.emit()

def initialize(self):
self.gui_obj.blockSignals(True)
if self.datatype == 'choice':
choices = self.choices or []
if isinstance(self.gui_obj, EditWithComplete):
self.gui_obj.all_items = choices else:
self.gui_obj.clear()
for x in choices:
if isinstance(x, basestring):
x = (x, x)
self.gui_obj.addItem(x[0], QVariant(x[1]))
self.set_gui_val(self.get_config_val(default=False))
self.gui_obj.blockSignals(False)
self.initial_value = self.get_gui_val()

def commit(self):
val = self.get_gui_val()
oldval = self.get_config_val()
changed = val != oldval if changed:
self.set_config_val(self.get_gui_val())
return changed and self.restart_required def restore_defaults(self):
self.set_gui_val(self.get_config_val(default=True))

def get_config_val(self, default=False):
if default:
val = self.config_obj.defaults[self.name]
else:
val = self.config_obj[self.name]
return val def set_config_val(self, val):
self.config_obj[self.name] = val def set_gui_val(self, val):
if self.datatype == 'bool':
self.gui_obj.setChecked(bool(val))
elif self.datatype == 'number':
self.gui_obj.setValue(val)
elif self.datatype == 'string':
self.gui_obj.setText(val if val else '')
elif self.datatype == 'choice':
if isinstance(self.gui_obj, EditWithComplete):
self.gui_obj.setText(val)
else:
idx = self.gui_obj.findData(QVariant(val), role=Qt.UserRole, flags=self.CHOICES_SEARCH_FLAGS)
if idx == -1:
idx = 0 self.gui_obj.setCurrentIndex(idx)

def get_gui_val(self):
if self.datatype == 'bool':
val = bool(self.gui_obj.isChecked())
elif self.datatype == 'number':
val = self.gui_obj.value()
elif self.datatype == 'string':
val = unicode(self.gui_obj.text()).strip()
if self.empty_string_is_None and not val:
val = None elif self.datatype == 'choice':
if isinstance(self.gui_obj, EditWithComplete):
val = unicode(self.gui_obj.text())
else:
idx = self.gui_obj.currentIndex()
if idx < 0: idx = 0 val = unicode(self.gui_obj.itemData(idx).toString())
return val class CommaSeparatedList(Setting):

def set_gui_val(self, val):
x = ''
if val:
x = u', '.join(val)
self.gui_obj.setText(x)

def get_gui_val(self):
val = unicode(self.gui_obj.text()).strip()
ans = []
if val:
ans = [x.strip() for x in val.split(',')]
ans = [x for x in ans if x]
return ans

[docs]class ConfigWidgetBase(QWidget, ConfigWidgetInterface):

'''
Base class that contains code to easily add standard config widgets like checkboxes, combo boxes, text fields and so on. See the :meth:`register` method.

This class automatically handles change notification, resetting to default, translation between gui objects and config objects, etc. for registered settings.

If your config widget inherits from this class but includes setting that are not registered, you should override the :class:`ConfigWidgetInterface` methods and call the base class methods inside the overrides.
'''

changed_signal = pyqtSignal()
restart_now = pyqtSignal()
supports_restoring_to_defaults = True restart_critical = False def __init__(self, parent=None):
QWidget.__init__(self, parent)
if hasattr(self, 'setupUi'):
self.setupUi(self)
self.settings = {}

[docs] def register(self, name, config_obj, gui_name=None, choices=None, restart_required=False, empty_string_is_None=True, setting=Setting):
'''
Register a setting.

:param name: The setting name
:param config: The config object that reads/writes the setting
:param gui_name: The name of the GUI object that presents an interface to change the setting. By default it is assumed to be
``'opt_' + name``.
:param choices: If this setting is a multiple choice (combobox) based setting, the list of choices. The list is a list of two element tuples of the form: ``[(gui name, value), ...]``
:param setting: The class responsible for managing this setting. The default class handles almost all cases, so this param is rarely used.
'''
setting = setting(name, config_obj, self, gui_name=gui_name, choices=choices, restart_required=restart_required, empty_string_is_None=empty_string_is_None)
return self.register_setting(setting)

def register_setting(self, setting):
self.settings[setting.name] = setting return setting def initialize(self):
for setting in self.settings.values():
setting.initialize()

def commit(self, *args):
restart_required = False for setting in self.settings.values():
rr = setting.commit()
if rr:
restart_required = True return restart_required def restore_defaults(self, *args):
for setting in self.settings.values():
setting.restore_defaults()

def get_plugin(category, name):
for plugin in preferences_plugins():
if plugin.category == category and plugin.name == name:
return plugin raise ValueError(
'No Preferences Plugin with category: %s and name: %s found' %
(category, name))

class ConfigDialog(QDialog):
def set_widget(self, w): self.w = w def accept(self):
try:
self.restart_required = self.w.commit()
except AbortCommit:
return
QDialog.accept(self)

def init_gui():
from calibre.gui2.ui import Main from calibre.gui2.main import option_parser from calibre.library import db parser = option_parser()
opts, args = parser.parse_args([])
actions = tuple(Main.create_application_menubar())
db = db()
gui = Main(opts)
gui.initialize(db.library_path, db, None, actions, show_gui=False)
return gui def show_config_widget(category, name, gui=None, show_restart_msg=False, parent=None, never_shutdown=False):
'''
Show the preferences plugin identified by category and name

:param gui: gui instance, if None a hidden gui is created
:param show_restart_msg: If True and the preferences plugin indicates a restart is required, show a message box telling the user to restart
:param parent: The parent of the displayed dialog

:return: True iff a restart is required for the changes made by the user to take effect
'''
from calibre.gui2 import gprefs pl = get_plugin(category, name)
d = ConfigDialog(parent)
d.resize(750, 550)
conf_name = 'config_widget_dialog_geometry_%s_%s'%(category, name)
geom = gprefs.get(conf_name, None)
d.setWindowTitle(_('Configure ') + name)
d.setWindowIcon(QIcon(I('config.png')))
bb = QDialogButtonBox(d)
bb.setStandardButtons(bb.Apply|bb.Cancel|bb.RestoreDefaults)
bb.accepted.connect(d.accept)
bb.rejected.connect(d.reject)
w = pl.create_widget(d)
d.set_widget(w)
bb.button(bb.RestoreDefaults).clicked.connect(w.restore_defaults)
bb.button(bb.RestoreDefaults).setEnabled(w.supports_restoring_to_defaults)
bb.button(bb.Apply).setEnabled(False)
bb.button(bb.Apply).clicked.connect(d.accept)
def onchange():
b = bb.button(bb.Apply)
b.setEnabled(True)
b.setDefault(True)
b.setAutoDefault(True)
w.changed_signal.connect(onchange)
bb.button(bb.Cancel).setFocus(True)
l = QVBoxLayout()
d.setLayout(l)
l.addWidget(w)
l.addWidget(bb)
mygui = gui is None if gui is None:
gui = init_gui()
mygui = True w.genesis(gui)
w.initialize()
if geom is not None:
d.restoreGeometry(geom)
d.exec_()
geom = bytearray(d.saveGeometry())
gprefs[conf_name] = geom rr = getattr(d, 'restart_required', False)
if show_restart_msg and rr:
from calibre.gui2 import warning_dialog warning_dialog(gui, 'Restart required', 'Restart required', show=True)
if mygui and not never_shutdown:
gui.shutdown()
return rr

# Testing {{{ def test_widget(category, name, gui=None):
show_config_widget(category, name, gui=gui, show_restart_msg=True)

def test_all():
from PyQt4.Qt import QApplication app = QApplication([])
app gui = init_gui()
for plugin in preferences_plugins():
test_widget(plugin.category, plugin.name, gui=gui)
gui.shutdown()

if __name__ == '__main__':
test_all()
# }}}

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.utils.formatter_functions

'''
Created on 13 Jan 2011

@author: charles
'''

__license__ = 'GPL v3'
__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'

import inspect, re, traceback from math import trunc from calibre import human_readable from calibre.constants import DEBUG from calibre.ebooks.metadata import title_sort from calibre.utils.config import tweaks from calibre.utils.titlecase import titlecase from calibre.utils.icu import capitalize, strcmp, sort_key from calibre.utils.date import parse_date, format_date, now, UNDEFINED_DATE from calibre.utils.localization import calibre_langcode_to_name, canonicalize_lang class FormatterFunctions(object):

def __init__(self):
self._builtins = {}
self._functions = {}

def register_builtin(self, func_class):
if not isinstance(func_class, FormatterFunction):
raise ValueError('Class %s is not an instance of FormatterFunction'%( func_class.__class__.__name__))
name = func_class.name if name in self._functions:
raise ValueError('Name %s already used'%name)
self._builtins[name] = func_class self._functions[name] = func_class for a in func_class.aliases:
self._functions[a] = func_class def register_function(self, func_class):
if not isinstance(func_class, FormatterFunction):
raise ValueError('Class %s is not an instance of FormatterFunction'%( func_class.__class__.__name__))
name = func_class.name if name in self._functions:
raise ValueError('Name %s already used'%name)
self._functions[name] = func_class def get_builtins(self):
return self._builtins def get_builtins_and_aliases(self):
res = {}
for f in self._builtins.itervalues():
res[f.name] = f for a in f.aliases:
res[a] = f return res def get_functions(self):
return self._functions def reset_to_builtins(self):
self._functions = {}
for n,c in self._builtins.items():
self._functions[n] = c for a in c.aliases:
self._functions[a] = c

_ff = FormatterFunctions()

def formatter_functions():
global _ff return _ff class FormatterFunction(object):

doc = _('No documentation provided')
name = 'no name provided'
category = 'Unknown'
arg_count = 0 aliases = []

def evaluate(self, formatter, kwargs, mi, locals, *args):
raise NotImplementedError()

def eval_(self, formatter, kwargs, mi, locals, *args):
ret = self.evaluate(formatter, kwargs, mi, locals, *args)
if isinstance(ret, (str, unicode)):
return ret if isinstance(ret, list):
return ','.join(list)
if isinstance(ret, (int, float, bool)):
return unicode(ret)

class BuiltinFormatterFunction(FormatterFunction):
def __init__(self):
formatter_functions().register_builtin(self)
eval_func = inspect.getmembers(self.__class__, lambda x: inspect.ismethod(x) and x.__name__ == 'evaluate')
try:
lines = [l[4:] for l in inspect.getsourcelines(eval_func[0][1])[0]]
except:
lines = []
self.program_text = ''.join(lines)

[docs]class BuiltinStrcmp(BuiltinFormatterFunction):
name = 'strcmp'
arg_count = 5 category = 'Relational'
__doc__ = doc = _('strcmp(x, y, lt, eq, gt) -- does a case-insensitive comparison of x '
'and y as strings. Returns lt if x < y. Returns eq if x == y. '
'Otherwise returns gt.')

def evaluate(self, formatter, kwargs, mi, locals, x, y, lt, eq, gt):
v = strcmp(x, y)
if v < 0:
return lt if v == 0:
return eq return gt

[docs]class BuiltinCmp(BuiltinFormatterFunction):
name = 'cmp'
category = 'Relational'
arg_count = 5
__doc__ = doc = _('cmp(x, y, lt, eq, gt) -- compares x and y after converting both to '
'numbers. Returns lt if x < y. Returns eq if x == y. Otherwise returns gt.')

def evaluate(self, formatter, kwargs, mi, locals, x, y, lt, eq, gt):
x = float(x if x and x != 'None' else 0)
y = float(y if y and y != 'None' else 0)
if x < y:
return lt if x == y:
return eq return gt

[docs]class BuiltinStrcat(BuiltinFormatterFunction):
name = 'strcat'
arg_count = -1 category = 'String manipulation'
__doc__ = doc = _('strcat(a, b, ...) -- can take any number of arguments. Returns a '
'string formed by concatenating all the arguments')

def evaluate(self, formatter, kwargs, mi, locals, *args):
i = 0 res = ''
for i in range(0, len(args)):
res += args[i]
return res

[docs]class BuiltinStrlen(BuiltinFormatterFunction):
name = 'strlen'
arg_count = 1 category = 'String manipulation'
__doc__ = doc = _('strlen(a) -- Returns the length of the string passed as '
'the argument')

def evaluate(self, formatter, kwargs, mi, locals, a):
try:
return len(a)
except:
return -1

[docs]class BuiltinAdd(BuiltinFormatterFunction):
name = 'add'
arg_count = 2 category = 'Arithmetic'
__doc__ = doc = _('add(x, y) -- returns x + y. Throws an exception if either x or y are not numbers.')

def evaluate(self, formatter, kwargs, mi, locals, x, y):
x = float(x if x and x != 'None' else 0)
y = float(y if y and y != 'None' else 0)
return unicode(x + y)

[docs]class BuiltinSubtract(BuiltinFormatterFunction):
name = 'subtract'
arg_count = 2 category = 'Arithmetic'
__doc__ = doc = _('subtract(x, y) -- returns x - y. Throws an exception if either x or y are not numbers.')

def evaluate(self, formatter, kwargs, mi, locals, x, y):
x = float(x if x and x != 'None' else 0)
y = float(y if y and y != 'None' else 0)
return unicode(x - y)

[docs]class BuiltinMultiply(BuiltinFormatterFunction):
name = 'multiply'
arg_count = 2 category = 'Arithmetic'
__doc__ = doc = _('multiply(x, y) -- returns x * y. Throws an exception if either x or y are not numbers.')

def evaluate(self, formatter, kwargs, mi, locals, x, y):
x = float(x if x and x != 'None' else 0)
y = float(y if y and y != 'None' else 0)
return unicode(x * y)

[docs]class BuiltinDivide(BuiltinFormatterFunction):
name = 'divide'
arg_count = 2 category = 'Arithmetic'
__doc__ = doc = _('divide(x, y) -- returns x / y. Throws an exception if either x or y are not numbers.')

def evaluate(self, formatter, kwargs, mi, locals, x, y):
x = float(x if x and x != 'None' else 0)
y = float(y if y and y != 'None' else 0)
return unicode(x / y)

[docs]class BuiltinTemplate(BuiltinFormatterFunction):
name = 'template'
arg_count = 1 category = 'Recursion'

__doc__ = doc = _('template(x) -- evaluates x as a template. The evaluation is done '
'in its own context, meaning that variables are not shared between '
'the caller and the template evaluation. Because the { and } '
'characters are special, you must use [[ for the { character and '
']] for the } character; they are converted automatically. '
'For example, template(\'[[title_sort]]\') will evaluate the '
'template {title_sort} and return its value. Note also that '
'prefixes and suffixes (the `|prefix|suffix` syntax) cannot be '
'used in the argument to this function when using template program mode.')

def evaluate(self, formatter, kwargs, mi, locals, template):
template = template.replace('[[', '{').replace(']]', '}')
return formatter.__class__().safe_format(template, kwargs, 'TEMPLATE', mi)

[docs]class BuiltinEval(BuiltinFormatterFunction):
name = 'eval'
arg_count = 1 category = 'Recursion'
__doc__ = doc = _('eval(template) -- evaluates the template, passing the local '
'variables (those \'assign\'ed to) instead of the book metadata. '
' This permits using the template processor to construct complex '
'results from local variables. Because the { and } '
'characters are special, you must use [[ for the { character and '
']] for the } character; they are converted automatically. '
'Note also that prefixes and suffixes (the `|prefix|suffix` syntax) '
'cannot be used in the argument to this function when using '
'template program mode.')

def evaluate(self, formatter, kwargs, mi, locals, template):
from formatter import EvalFormatter template = template.replace('[[', '{').replace(']]', '}')
return EvalFormatter().safe_format(template, locals, 'EVAL', None)

[docs]class BuiltinAssign(BuiltinFormatterFunction):
name = 'assign'
arg_count = 2 category = 'Other'
__doc__ = doc = _('assign(id, val) -- assigns val to id, then returns val. '
'id must be an identifier, not an expression')

def evaluate(self, formatter, kwargs, mi, locals, target, value):
locals[target] = value return value

[docs]class BuiltinPrint(BuiltinFormatterFunction):
name = 'print'
arg_count = -1 category = 'Other'
__doc__ = doc = _('print(a, b, ...) -- prints the arguments to standard output. '
'Unless you start calibre from the command line (calibre-debug -g), '
'the output will go to a black hole.')

def evaluate(self, formatter, kwargs, mi, locals, *args):
print args return None

[docs]class BuiltinField(BuiltinFormatterFunction):
name = 'field'
arg_count = 1 category = 'Get values from metadata'
__doc__ = doc = _('field(name) -- returns the metadata field named by name')

def evaluate(self, formatter, kwargs, mi, locals, name):
return formatter.get_value(name, [], kwargs)

[docs]class BuiltinRawField(BuiltinFormatterFunction):
name = 'raw_field'
arg_count = 1 category = 'Get values from metadata'
__doc__ = doc = _('raw_field(name) -- returns the metadata field named by name '
'without applying any formatting.')

def evaluate(self, formatter, kwargs, mi, locals, name):
return unicode(getattr(mi, name, None))

[docs]class BuiltinSubstr(BuiltinFormatterFunction):
name = 'substr'
arg_count = 3 category = 'String manipulation'
__doc__ = doc = _('substr(str, start, end) -- returns the start\'th through the end\'th '
'characters of str. The first character in str is the zero\'th '
'character. If end is negative, then it indicates that many '
'characters counting from the right. If end is zero, then it '
'indicates the last character. For example, substr(\'12345\', 1, 0) '
'returns \'2345\', and substr(\'12345\', 1, -1) returns \'234\'.')

def evaluate(self, formatter, kwargs, mi, locals, str_, start_, end_):
return str_[int(start_): len(str_) if int(end_) == 0 else int(end_)]

[docs]class BuiltinLookup(BuiltinFormatterFunction):
name = 'lookup'
arg_count = -1 category = 'Iterating over values'
__doc__ = doc = _('lookup(val, pattern, field, pattern, field, ..., else_field) -- '
'like switch, except the arguments are field (metadata) names, not '
'text. The value of the appropriate field will be fetched and used. '
'Note that because composite columns are fields, you can use this '
'function in one composite field to use the value of some other '
'composite field. This is extremely useful when constructing '
'variable save paths')

def evaluate(self, formatter, kwargs, mi, locals, val, *args):
if len(args) == 2: # here for backwards compatibility if val:
return formatter.vformat('{'+args[0].strip()+'}', [], kwargs)
else:
return formatter.vformat('{'+args[1].strip()+'}', [], kwargs)
if (len(args) % 2) != 1:
raise ValueError(_('lookup requires either 2 or an odd number of arguments'))
i = 0 while i < len(args):
if i + 1 >= len(args):
return formatter.vformat('{' + args[i].strip() + '}', [], kwargs)
if re.search(args[i], val, flags=re.I):
return formatter.vformat('{'+args[i+1].strip() + '}', [], kwargs)
i += 2

[docs]class BuiltinTest(BuiltinFormatterFunction):
name = 'test'
arg_count = 3 category = 'If-then-else'
__doc__ = doc = _('test(val, text if not empty, text if empty) -- return `text if not '
'empty` if the field is not empty, otherwise return `text if empty`')

def evaluate(self, formatter, kwargs, mi, locals, val, value_if_set, value_not_set):
if val:
return value_if_set else:
return value_not_set

[docs]class BuiltinContains(BuiltinFormatterFunction):
name = 'contains'
arg_count = 4 category = 'If-then-else'
__doc__ = doc = _('contains(val, pattern, text if match, text if not match) -- checks '
'if field contains matches for the regular expression `pattern`. '
'Returns `text if match` if matches are found, otherwise it returns '
'`text if no match`')

def evaluate(self, formatter, kwargs, mi, locals, val, test, value_if_present, value_if_not):
if re.search(test, val, flags=re.I):
return value_if_present else:
return value_if_not

[docs]class BuiltinSwitch(BuiltinFormatterFunction):
name = 'switch'
arg_count = -1 category = 'Iterating over values'
__doc__ = doc = _('switch(val, pattern, value, pattern, value, ..., else_value) -- '
'for each `pattern, value` pair, checks if the field matches '
'the regular expression `pattern` and if so, returns that '
'`value`. If no pattern matches, then else_value is returned. '
'You can have as many `pattern, value` pairs as you want')

def evaluate(self, formatter, kwargs, mi, locals, val, *args):
if (len(args) % 2) != 1:
raise ValueError(_('switch requires an odd number of arguments'))
i = 0 while i < len(args):
if i + 1 >= len(args):
return args[i]
if re.search(args[i], val, flags=re.I):
return args[i+1]
i += 2

[docs]class BuiltinStrcatMax(BuiltinFormatterFunction):
name = 'strcat_max'
arg_count = -1 category = 'String manipulation'
__doc__ = doc = _('strcat_max(max, string1, prefix2, string2, ...) -- '
'Returns a string formed by concatenating the arguments. The '
'returned value is initialized to string1. `Prefix, string` '
'pairs are added to the end of the value as long as the '
'resulting string length is less than `max`. String1 is returned '
'even if string1 is longer than max. You can pass as many '
'`prefix, string` pairs as you wish.')

def evaluate(self, formatter, kwargs, mi, locals, *args):
if len(args) < 2:
raise ValueError(_('strcat_max requires 2 or more arguments'))
if (len(args) % 2) != 0:
raise ValueError(_('strcat_max requires an even number of arguments'))
try:
max = int(args[0])
except:
raise ValueError(_('first argument to strcat_max must be an integer'))

i = 2 result = args[1]
try:
while i < len(args):
if (len(result) + len(args[i]) + len(args[i+1])) > max:
break result = result + args[i] + args[i+1]
i += 2 except:
pass return result.strip()

[docs]class BuiltinInList(BuiltinFormatterFunction):
name = 'in_list'
arg_count = 5 category = 'List lookup'
__doc__ = doc = _('in_list(val, separator, pattern, found_val, not_found_val) -- '
'treat val as a list of items separated by separator, '
'comparing the pattern against each value in the list. If the '
'pattern matches a value, return found_val, otherwise return '
'not_found_val.')

def evaluate(self, formatter, kwargs, mi, locals, val, sep, pat, fv, nfv):
l = [v.strip() for v in val.split(sep) if v.strip()]
if l:
for v in l:
if re.search(pat, v, flags=re.I):
return fv return nfv

[docs]class BuiltinStrInList(BuiltinFormatterFunction):
name = 'str_in_list'
arg_count = 5 category = 'List lookup'
__doc__ = doc = _('str_in_list(val, separator, string, found_val, not_found_val) -- '
'treat val as a list of items separated by separator, '
'comparing the string against each value in the list. If the '
'string matches a value, return found_val, otherwise return '
'not_found_val. If the string contains separators, then it is '
'also treated as a list and each value is checked.')

def evaluate(self, formatter, kwargs, mi, locals, val, sep, str, fv, nfv):
l = [v.strip() for v in val.split(sep) if v.strip()]
c = [v.strip() for v in str.split(sep) if v.strip()]
if l:
for v in l:
for t in c:
if strcmp(t, v) == 0:
return fv return nfv

[docs]class BuiltinIdentifierInList(BuiltinFormatterFunction):
name = 'identifier_in_list'
arg_count = 4 category = 'List lookup'
__doc__ = doc = _('identifier_in_list(val, id, found_val, not_found_val) -- '
'treat val as a list of identifiers separated by commas, '
'comparing the string against each value in the list. An identifier '
'has the format "identifier:value". The id parameter should be '
'either "id" or "id:regexp". The first case matches if there is any '
'identifier with that id. The second case matches if the regexp '
'matches the identifier\'s value. If there is a match, '
'return found_val, otherwise return not_found_val.')

def evaluate(self, formatter, kwargs, mi, locals, val, ident, fv, nfv):
l = [v.strip() for v in val.split(',') if v.strip()]
(id, _, regexp) = ident.partition(':')
if not id:
return nfv id += ':'
if l:
for v in l:
if v.startswith(id):
if not regexp or re.search(regexp, v[len(id):], flags=re.I):
return fv return nfv

[docs]class BuiltinRe(BuiltinFormatterFunction):
name = 're'
arg_count = 3 category = 'String manipulation'
__doc__ = doc = _('re(val, pattern, replacement) -- return the field after applying '
'the regular expression. All instances of `pattern` are replaced '
'with `replacement`. As in all of calibre, these are '
'python-compatible regular expressions')

def evaluate(self, formatter, kwargs, mi, locals, val, pattern, replacement):
return re.sub(pattern, replacement, val, flags=re.I)

[docs]class BuiltinSwapAroundComma(BuiltinFormatterFunction):
name = 'swap_around_comma'
arg_count = 1 category = 'String manipulation'
__doc__ = doc = _('swap_around_comma(val) -- given a value of the form '
'"B, A", return "A B". This is most useful for converting names '
'in LN, FN format to FN LN. If there is no comma, the function '
'returns val unchanged')

def evaluate(self, formatter, kwargs, mi, locals, val):
return re.sub(r'^(.*?),\s*(.*$)', r'\2 \1', val, flags=re.I).strip()

[docs]class BuiltinIfempty(BuiltinFormatterFunction):
name = 'ifempty'
arg_count = 2 category = 'If-then-else'
__doc__ = doc = _('ifempty(val, text if empty) -- return val if val is not empty, '
'otherwise return `text if empty`')

def evaluate(self, formatter, kwargs, mi, locals, val, value_if_empty):
if val:
return val else:
return value_if_empty

[docs]class BuiltinShorten(BuiltinFormatterFunction):
name = 'shorten'
arg_count = 4 category = 'String manipulation'
__doc__ = doc = _('shorten(val, left chars, middle text, right chars) -- Return a '
'shortened version of the field, consisting of `left chars` '
'characters from the beginning of the field, followed by '
'`middle text`, followed by `right chars` characters from '
'the end of the string. `Left chars` and `right chars` must be '
'integers. For example, assume the title of the book is '
'`Ancient English Laws in the Times of Ivanhoe`, and you want '
'it to fit in a space of at most 15 characters. If you use '
'{title:shorten(9,-,5)}, the result will be `Ancient E-nhoe`. '
'If the field\'s length is less than left chars + right chars + '
'the length of `middle text`, then the field will be used '
'intact. For example, the title `The Dome` would not be changed.')

def evaluate(self, formatter, kwargs, mi, locals, val, leading, center_string, trailing):
l = max(0, int(leading))
t = max(0, int(trailing))
if len(val) > l + len(center_string) + t:
return val[0:l] + center_string + ('' if t == 0 else val[-t:])
else:
return val

[docs]class BuiltinCount(BuiltinFormatterFunction):
name = 'count'
arg_count = 2 category = 'List manipulation'
__doc__ = doc = _('count(val, separator) -- interprets the value as a list of items '
'separated by `separator`, returning the number of items in the '
'list. Most lists use a comma as the separator, but authors '
'uses an ampersand. Examples: {tags:count(,)}, {authors:count(&)}')

def evaluate(self, formatter, kwargs, mi, locals, val, sep):
return unicode(len(val.split(sep)))

[docs]class BuiltinListitem(BuiltinFormatterFunction):
name = 'list_item'
arg_count = 3 category = 'List lookup'
__doc__ = doc = _('list_item(val, index, separator) -- interpret the value as a list of '
'items separated by `separator`, returning the `index`th item. '
'The first item is number zero. The last item can be returned '
'using `list_item(-1,separator)`. If the item is not in the list, '
'then the empty value is returned. The separator has the same '
'meaning as in the count function.')

def evaluate(self, formatter, kwargs, mi, locals, val, index, sep):
if not val:
return ''
index = int(index)
val = val.split(sep)
try:
return val[index].strip()
except:
return ''

[docs]class BuiltinSelect(BuiltinFormatterFunction):
name = 'select'
arg_count = 2 category = 'List lookup'
__doc__ = doc = _('select(val, key) -- interpret the value as a comma-separated list '
'of items, with the items being "id:value". Find the pair with the '
'id equal to key, and return the corresponding value.'
)

def evaluate(self, formatter, kwargs, mi, locals, val, key):
if not val:
return ''
vals = [v.strip() for v in val.split(',')]
for v in vals:
if v.startswith(key+':'):
return v[len(key)+1:]
return ''

[docs]class BuiltinApproximateFormats(BuiltinFormatterFunction):
name = 'approximate_formats'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('approximate_formats() -- return a comma-separated '
'list of formats that at one point were associated with the '
'book. There is no guarantee that this list is correct, '
'although it probably is. '
'This function can be called in template program mode using '
'the template "{:\'approximate_formats()\'}". '
'Note that format names are always uppercase, as in EPUB.'
)

def evaluate(self, formatter, kwargs, mi, locals):
fmt_data = mi.get('db_approx_formats', [])
if not fmt_data:
return ''
data = sorted(fmt_data)
return ','.join(v.upper() for v in data)

[docs]class BuiltinFormatsModtimes(BuiltinFormatterFunction):
name = 'formats_modtimes'
arg_count = 1 category = 'Get values from metadata'
__doc__ = doc = _('formats_modtimes(date_format) -- return a comma-separated '
'list of colon_separated items representing modification times '
'for the formats of a book. The date_format parameter '
'specifies how the date is to be formatted. See the '
'date_format function for details. You can use the select '
'function to get the mod time for a specific '
'format. Note that format names are always uppercase, '
'as in EPUB.'
)

def evaluate(self, formatter, kwargs, mi, locals, fmt):
fmt_data = mi.get('format_metadata', {})
data = sorted(fmt_data.items(), key=lambda x:x[1]['mtime'], reverse=True)
return ','.join(k.upper()+':'+format_date(v['mtime'], fmt)
for k,v in data)

[docs]class BuiltinFormatsSizes(BuiltinFormatterFunction):
name = 'formats_sizes'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('formats_sizes() -- return a comma-separated list of '
'colon_separated items representing sizes in bytes '
'of the formats of a book. You can use the select '
'function to get the size for a specific '
'format. Note that format names are always uppercase, '
'as in EPUB.'
)

def evaluate(self, formatter, kwargs, mi, locals):
fmt_data = mi.get('format_metadata', {})
return ','.join(k.upper()+':'+str(v['size']) for k,v in fmt_data.iteritems())

[docs]class BuiltinFormatsPaths(BuiltinFormatterFunction):
name = 'formats_paths'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('formats_paths() -- return a comma-separated list of '
'colon_separated items representing full path to '
'the formats of a book. You can use the select '
'function to get the path for a specific '
'format. Note that format names are always uppercase, '
'as in EPUB.')

def evaluate(self, formatter, kwargs, mi, locals):
fmt_data = mi.get('format_metadata', {})
return ','.join(k.upper()+':'+str(v['path']) for k,v in fmt_data.iteritems())

[docs]class BuiltinHumanReadable(BuiltinFormatterFunction):
name = 'human_readable'
arg_count = 1 category = 'Formatting values'
__doc__ = doc = _('human_readable(v) -- return a string '
'representing the number v in KB, MB, GB, etc.'
)

def evaluate(self, formatter, kwargs, mi, locals, val):
try:
return human_readable(round(float(val)))
except:
return ''

[docs]class BuiltinFormatNumber(BuiltinFormatterFunction):
name = 'format_number'
arg_count = 2 category = 'Formatting values'
__doc__ = doc = _('format_number(v, template) -- format the number v using '
'a python formatting template such as "{0:5.2f}" or '
'"{0:,d}" or "${0:5,.2f}". The field_name part of the '
'template must be a 0 (zero) (the "{0:" in the above examples). '
'See the template language and python documentation for more '
'examples. Returns the empty string if formatting fails.'
)

def evaluate(self, formatter, kwargs, mi, locals, val, template):
if val == '' or val == 'None':
return ''
try:
v1 = float(val)
except:
return ''
try: # Try formatting the value as a float return template.format(v1)
except:
pass try: # Try formatting the value as an int v2 = trunc(v1)
if v2 == v1:
return template.format(v2)
except:
pass return ''

[docs]class BuiltinSublist(BuiltinFormatterFunction):
name = 'sublist'
arg_count = 4 category = 'List manipulation'
__doc__ = doc = _('sublist(val, start_index, end_index, separator) -- interpret the '
'value as a list of items separated by `separator`, returning a '
'new list made from the `start_index` to the `end_index` item. '
'The first item is number zero. If an index is negative, then it '
'counts from the end of the list. As a special case, an end_index '
'of zero is assumed to be the length of the list. Examples using '
'basic template mode and assuming that the tags column (which is '
'comma-separated) contains "A, B, C": '
'{tags:sublist(0,1,\,)} returns "A". '
'{tags:sublist(-1,0,\,)} returns "C". '
'{tags:sublist(0,-1,\,)} returns "A, B".'
)

def evaluate(self, formatter, kwargs, mi, locals, val, start_index, end_index, sep):
if not val:
return ''
si = int(start_index)
ei = int(end_index)
# allow empty list items so counts are what the user expects val = [v.strip() for v in val.split(sep)]

if sep == ',':
sep = ', '
try:
if ei == 0:
return sep.join(val[si:])
else:
return sep.join(val[si:ei])
except:
return ''

[docs]class BuiltinSubitems(BuiltinFormatterFunction):
name = 'subitems'
arg_count = 3 category = 'List manipulation'
__doc__ = doc = _('subitems(val, start_index, end_index) -- This function is used to '
'break apart lists of items such as genres. It interprets the value '
'as a comma-separated list of items, where each item is a period-'
'separated list. Returns a new list made by first finding all the '
'period-separated items, then for each such item extracting the '
'`start_index` to the `end_index` components, then combining '
'the results back together. The first component in a period-'
'separated list has an index of zero. If an index is negative, '
'then it counts from the end of the list. As a special case, an '
'end_index of zero is assumed to be the length of the list. '
'Example using basic template mode and assuming a #genre value of '
'"A.B.C": {#genre:subitems(0,1)} returns "A". {#genre:subitems(0,2)} '
'returns "A.B". {#genre:subitems(1,0)} returns "B.C". Assuming a #genre '
'value of "A.B.C, D.E.F", {#genre:subitems(0,1)} returns "A, D". '
'{#genre:subitems(0,2)} returns "A.B, D.E"')

period_pattern = re.compile(r'(?<=[^\.\s])\.(?=[^\.\s])', re.U)

def evaluate(self, formatter, kwargs, mi, locals, val, start_index, end_index):
if not val:
return ''
si = int(start_index)
ei = int(end_index)
has_periods = '.' in val items = [v.strip() for v in val.split(',')]
rv = set()
for item in items:
if has_periods and '.' in item:
components = self.period_pattern.split(item)
else:
components = [item]
try:
if ei == 0:
rv.add('.'.join(components[si:]))
else:
rv.add('.'.join(components[si:ei]))
except:
pass return ', '.join(sorted(rv, key=sort_key))

[docs]class BuiltinFormatDate(BuiltinFormatterFunction):
name = 'format_date'
arg_count = 2 category = 'Formatting values'
__doc__ = doc = _('format_date(val, format_string) -- format the value, '
'which must be a date, using the format_string, returning a string. '
'The formatting codes are: '
'd : the day as number without a leading zero (1 to 31) '
'dd : the day as number with a leading zero (01 to 31) '
'ddd : the abbreviated localized day name (e.g. "Mon" to "Sun"). '
'dddd : the long localized day name (e.g. "Monday" to "Sunday"). '
'M : the month as number without a leading zero (1 to 12). '
'MM : the month as number with a leading zero (01 to 12) '
'MMM : the abbreviated localized month name (e.g. "Jan" to "Dec"). '
'MMMM : the long localized month name (e.g. "January" to "December"). '
'yy : the year as two digit number (00 to 99). '
'yyyy : the year as four digit number. '
'h : the hours without a leading 0 (0 to 11 or 0 to 23, depending on am/pm) '
'hh : the hours with a leading 0 (00 to 11 or 00 to 23, depending on am/pm) '
'm : the minutes without a leading 0 (0 to 59) '
'mm : the minutes with a leading 0 (00 to 59) '
's : the seconds without a leading 0 (0 to 59) '
'ss : the seconds with a leading 0 (00 to 59) '
'ap : use a 12-hour clock instead of a 24-hour clock, with "ap" replaced by the localized string for am or pm '
'AP : use a 12-hour clock instead of a 24-hour clock, with "AP" replaced by the localized string for AM or PM '
'iso : the date with time and timezone. Must be the only format present')

def evaluate(self, formatter, kwargs, mi, locals, val, format_string):
if not val or val == 'None':
return ''
try:
dt = parse_date(val)
s = format_date(dt, format_string)
except:
s = 'BAD DATE'
return s

[docs]class BuiltinUppercase(BuiltinFormatterFunction):
name = 'uppercase'
arg_count = 1 category = 'String case changes'
__doc__ = doc = _('uppercase(val) -- return value of the field in upper case')

def evaluate(self, formatter, kwargs, mi, locals, val):
return val.upper()

[docs]class BuiltinLowercase(BuiltinFormatterFunction):
name = 'lowercase'
arg_count = 1 category = 'String case changes'
__doc__ = doc = _('lowercase(val) -- return value of the field in lower case')

def evaluate(self, formatter, kwargs, mi, locals, val):
return val.lower()

[docs]class BuiltinTitlecase(BuiltinFormatterFunction):
name = 'titlecase'
arg_count = 1 category = 'String case changes'
__doc__ = doc = _('titlecase(val) -- return value of the field in title case')

def evaluate(self, formatter, kwargs, mi, locals, val):
return titlecase(val)

[docs]class BuiltinCapitalize(BuiltinFormatterFunction):
name = 'capitalize'
arg_count = 1 category = 'String case changes'
__doc__ = doc = _('capitalize(val) -- return value of the field capitalized')

def evaluate(self, formatter, kwargs, mi, locals, val):
return capitalize(val)

[docs]class BuiltinBooksize(BuiltinFormatterFunction):
name = 'booksize'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('booksize() -- return value of the size field')

def evaluate(self, formatter, kwargs, mi, locals):
if mi.book_size is not None:
try:
return str(mi.book_size)
except:
pass return ''

[docs]class BuiltinOndevice(BuiltinFormatterFunction):
name = 'ondevice'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('ondevice() -- return Yes if ondevice is set, otherwise return '
'the empty string')

def evaluate(self, formatter, kwargs, mi, locals):
if mi.ondevice_col:
return _('Yes')
return ''

[docs]class BuiltinSeriesSort(BuiltinFormatterFunction):
name = 'series_sort'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('series_sort() -- return the series sort value')

def evaluate(self, formatter, kwargs, mi, locals):
if mi.series:
return title_sort(mi.series)
return ''

[docs]class BuiltinHasCover(BuiltinFormatterFunction):
name = 'has_cover'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('has_cover() -- return Yes if the book has a cover, '
'otherwise return the empty string')

def evaluate(self, formatter, kwargs, mi, locals):
if mi.has_cover:
return _('Yes')
return ''

[docs]class BuiltinFirstNonEmpty(BuiltinFormatterFunction):
name = 'first_non_empty'
arg_count = -1 category = 'Iterating over values'
__doc__ = doc = _('first_non_empty(value, value, ...) -- '
'returns the first value that is not empty. If all values are '
'empty, then the empty value is returned. '
'You can have as many values as you want.')

def evaluate(self, formatter, kwargs, mi, locals, *args):
i = 0 while i < len(args):
if args[i]:
return args[i]
i += 1 return ''

[docs]class BuiltinAnd(BuiltinFormatterFunction):
name = 'and'
arg_count = -1 category = 'Boolean'
__doc__ = doc = _('and(value, value, ...) -- '
'returns the string "1" if all values are not empty, otherwise '
'returns the empty string. This function works well with test or '
'first_non_empty. You can have as many values as you want.')

def evaluate(self, formatter, kwargs, mi, locals, *args):
i = 0 while i < len(args):
if not args[i]:
return ''
i += 1 return '1'

[docs]class BuiltinOr(BuiltinFormatterFunction):
name = 'or'
arg_count = -1 category = 'Boolean'
__doc__ = doc = _('or(value, value, ...) -- '
'returns the string "1" if any value is not empty, otherwise '
'returns the empty string. This function works well with test or '
'first_non_empty. You can have as many values as you want.')

def evaluate(self, formatter, kwargs, mi, locals, *args):
i = 0 while i < len(args):
if args[i]:
return '1'
i += 1 return ''

[docs]class BuiltinNot(BuiltinFormatterFunction):
name = 'not'
arg_count = 1 category = 'Boolean'
__doc__ = doc = _('not(value) -- '
'returns the string "1" if the value is empty, otherwise '
'returns the empty string. This function works well with test or '
'first_non_empty. You can have as many values as you want.')

def evaluate(self, formatter, kwargs, mi, locals, val):
return '' if val else '1'

[docs]class BuiltinListUnion(BuiltinFormatterFunction):
name = 'list_union'
arg_count = 3 category = 'List manipulation'
__doc__ = doc = _('list_union(list1, list2, separator) -- '
'return a list made by merging the items in list1 and list2, '
'removing duplicate items using a case-insensitive compare. If '
'items differ in case, the one in list1 is used. '
'The items in list1 and list2 are separated by separator, as are '
'the items in the returned list.')
aliases = ['merge_lists']

def evaluate(self, formatter, kwargs, mi, locals, list1, list2, separator):
res = [l.strip() for l in list1.split(separator) if l.strip()]
l2 = [l.strip() for l in list2.split(separator) if l.strip()]
lcl1 = set([icu_lower(l) for l in res])

for i in l2:
if icu_lower(i) not in lcl1 and i not in res:
res.append(i)
if separator == ',':
return ', '.join(res)
return separator.join(res)

[docs]class BuiltinListDifference(BuiltinFormatterFunction):
name = 'list_difference'
arg_count = 3 category = 'List manipulation'
__doc__ = doc = _('list_difference(list1, list2, separator) -- '
'return a list made by removing from list1 any item found in list2, '
'using a case-insensitive compare. The items in list1 and list2 '
'are separated by separator, as are the items in the returned list.')

def evaluate(self, formatter, kwargs, mi, locals, list1, list2, separator):
l1 = [l.strip() for l in list1.split(separator) if l.strip()]
l2 = set([icu_lower(l.strip()) for l in list2.split(separator) if l.strip()])

res = []
for i in l1:
if icu_lower(i) not in l2 and i not in res:
res.append(i)
if separator == ',':
return ', '.join(res)
return separator.join(res)

[docs]class BuiltinListIntersection(BuiltinFormatterFunction):
name = 'list_intersection'
arg_count = 3 category = 'List manipulation'
__doc__ = doc = _('list_intersection(list1, list2, separator) -- '
'return a list made by removing from list1 any item not found in list2, '
'using a case-insensitive compare. The items in list1 and list2 '
'are separated by separator, as are the items in the returned list.')

def evaluate(self, formatter, kwargs, mi, locals, list1, list2, separator):
l1 = [l.strip() for l in list1.split(separator) if l.strip()]
l2 = set([icu_lower(l.strip()) for l in list2.split(separator) if l.strip()])

res = []
for i in l1:
if icu_lower(i) in l2 and i not in res:
res.append(i)
if separator == ',':
return ', '.join(res)
return separator.join(res)

[docs]class BuiltinListSort(BuiltinFormatterFunction):
name = 'list_sort'
arg_count = 3 category = 'List manipulation'
__doc__ = doc = _('list_sort(list, direction, separator) -- '
'return list sorted using a case-insensitive sort. If direction is '
'zero, the list is sorted ascending, otherwise descending. The list items '
'are separated by separator, as are the items in the returned list.')

def evaluate(self, formatter, kwargs, mi, locals, list1, direction, separator):
res = [l.strip() for l in list1.split(separator) if l.strip()]
if separator == ',':
return ', '.join(sorted(res, key=sort_key, reverse=direction != "0"))
return separator.join(sorted(res, key=sort_key, reverse=direction != "0"))

[docs]class BuiltinListEquals(BuiltinFormatterFunction):
name = 'list_equals'
arg_count = 6 category = 'List manipulation'
__doc__ = doc = _('list_equals(list1, sep1, list2, sep2, yes_val, no_val) -- '
'return yes_val if list1 and list2 contain the same items, '
'otherwise return no_val. The items are determined by splitting '
'each list using the appropriate separator character (sep1 or '
'sep2). The order of items in the lists is not relevant. '
'The compare is case insensitive.')

def evaluate(self, formatter, kwargs, mi, locals, list1, sep1, list2, sep2, yes_val, no_val):
s1 = set([icu_lower(l.strip()) for l in list1.split(sep1) if l.strip()])
s2 = set([icu_lower(l.strip()) for l in list2.split(sep2) if l.strip()])
if s1 == s2:
return yes_val return no_val

[docs]class BuiltinListRe(BuiltinFormatterFunction):
name = 'list_re'
arg_count = 4 category = 'List manipulation'
__doc__ = doc = _('list_re(src_list, separator, search_re, opt_replace) -- '
'Construct a list by first separating src_list into items using '
'the separator character. For each item in the list, check if it '
'matches search_re. If it does, then add it to the list to be '
'returned. If opt_replace is not the empty string, then apply the '
'replacement before adding the item to the returned list.')

def evaluate(self, formatter, kwargs, mi, locals, src_list, separator, search_re, opt_replace):
l = [l.strip() for l in src_list.split(separator) if l.strip()]
res = []
for item in l:
if re.search(search_re, item, flags=re.I) is not None:
if opt_replace:
item = re.sub(search_re, opt_replace, item)
for i in [l.strip() for l in item.split(',') if l.strip()]:
if i not in res:
res.append(i)
if separator == ',':
return ', '.join(res)
return separator.join(res)

[docs]class BuiltinToday(BuiltinFormatterFunction):
name = 'today'
arg_count = 0 category = 'Date functions'
__doc__ = doc = _('today() -- '
'return a date string for today. This value is designed for use in '
'format_date or days_between, but can be manipulated like any '
'other string. The date is in ISO format.')
def evaluate(self, formatter, kwargs, mi, locals):
return format_date(now(), 'iso')

[docs]class BuiltinDaysBetween(BuiltinFormatterFunction):
name = 'days_between'
arg_count = 2 category = 'Date functions'
__doc__ = doc = _('days_between(date1, date2) -- '
'return the number of days between date1 and date2. The number is '
'positive if date1 is greater than date2, otherwise negative. If '
'either date1 or date2 are not dates, the function returns the '
'empty string.')
def evaluate(self, formatter, kwargs, mi, locals, date1, date2):
try:
d1 = parse_date(date1)
if d1 == UNDEFINED_DATE:
return ''
d2 = parse_date(date2)
if d2 == UNDEFINED_DATE:
return ''
except:
return ''
i = d1 - d2 return str('%d.%d'%(i.days, i.seconds/8640))

[docs]class BuiltinLanguageStrings(BuiltinFormatterFunction):
name = 'language_strings'
arg_count = 2 category = 'Get values from metadata'
__doc__ = doc = _('language_strings(lang_codes, localize) -- '
'return the strings for the language codes passed in lang_codes. '
'If localize is zero, return the strings in English. If '
'localize is not zero, return the strings in the language of '
'the current locale. Lang_codes is a comma-separated list.')
def evaluate(self, formatter, kwargs, mi, locals, lang_codes, localize):
retval = []
for c in [c.strip() for c in lang_codes.split(',') if c.strip()]:
try:
n = calibre_langcode_to_name(c, localize != '0')
if n:
retval.append(n)
except:
pass return ', '.join(retval)

[docs]class BuiltinLanguageCodes(BuiltinFormatterFunction):
name = 'language_codes'
arg_count = 1 category = 'Get values from metadata'
__doc__ = doc = _('language_codes(lang_strings) -- '
'return the language codes for the strings passed in lang_strings. '
'The strings must be in the language of the current locale. '
'Lang_strings is a comma-separated list.')
def evaluate(self, formatter, kwargs, mi, locals, lang_strings):
retval = []
for c in [c.strip() for c in lang_strings.split(',') if c.strip()]:
try:
cv = canonicalize_lang(c)
if cv:
retval.append(canonicalize_lang(cv))
except:
pass return ', '.join(retval)

[docs]class BuiltinCurrentLibraryName(BuiltinFormatterFunction):
name = 'current_library_name'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('current_library_name() -- '
'return the last name on the path to the current calibre library. '
'This function can be called in template program mode using the '
'template "{:\'current_library_name()\'}".')
def evaluate(self, formatter, kwargs, mi, locals):
from calibre.library import current_library_name return current_library_name()

[docs]class BuiltinCurrentLibraryPath(BuiltinFormatterFunction):
name = 'current_library_path'
arg_count = 0 category = 'Get values from metadata'
__doc__ = doc = _('current_library_path() -- '
'return the path to the current calibre library. This function can '
'be called in template program mode using the template '
'"{:\'current_library_path()\'}".')
def evaluate(self, formatter, kwargs, mi, locals):
from calibre.library import current_library_path return current_library_path()

[docs]class BuiltinFinishFormatting(BuiltinFormatterFunction):
name = 'finish_formatting'
arg_count = 4 category = 'Formatting values'
__doc__ = doc = _('finish_formatting(val, fmt, prefix, suffix) -- apply the '
'format, prefix, and suffix to a value in the same way as '
'done in a template like `{series_index:05.2f| - |- }`. For '
'example, the following program produces the same output '
'as the above template: '
'program: finish_formatting(field("series_index"), "05.2f", " - ", " - ")')

def evaluate(self, formatter, kwargs, mi, locals_, val, fmt, prefix, suffix):
if not val:
return val return prefix + formatter._do_format(val, fmt) + suffix

_formatter_builtins = [
BuiltinAdd(), BuiltinAnd(), BuiltinApproximateFormats(),
BuiltinAssign(), BuiltinBooksize(),
BuiltinCapitalize(), BuiltinCmp(), BuiltinContains(), BuiltinCount(),
BuiltinCurrentLibraryName(), BuiltinCurrentLibraryPath(),
BuiltinDaysBetween(), BuiltinDivide(), BuiltinEval(), BuiltinFirstNonEmpty(),
BuiltinField(), BuiltinFinishFormatting(), BuiltinFormatDate(),
BuiltinFormatNumber(), BuiltinFormatsModtimes(), BuiltinFormatsPaths(),
BuiltinFormatsSizes(),
BuiltinHasCover(), BuiltinHumanReadable(), BuiltinIdentifierInList(),
BuiltinIfempty(), BuiltinLanguageCodes(), BuiltinLanguageStrings(),
BuiltinInList(), BuiltinListDifference(), BuiltinListEquals(),
BuiltinListIntersection(), BuiltinListitem(), BuiltinListRe(),
BuiltinListSort(), BuiltinListUnion(), BuiltinLookup(),
BuiltinLowercase(), BuiltinMultiply(), BuiltinNot(),
BuiltinOndevice(), BuiltinOr(), BuiltinPrint(), BuiltinRawField(),
BuiltinRe(), BuiltinSelect(), BuiltinSeriesSort(), BuiltinShorten(),
BuiltinStrcat(), BuiltinStrcatMax(),
BuiltinStrcmp(), BuiltinStrInList(), BuiltinStrlen(), BuiltinSubitems(),
BuiltinSublist(),BuiltinSubstr(), BuiltinSubtract(), BuiltinSwapAroundComma(),
BuiltinSwitch(), BuiltinTemplate(), BuiltinTest(), BuiltinTitlecase(),
BuiltinToday(), BuiltinUppercase(),
]

class FormatterUserFunction(FormatterFunction):
def __init__(self, name, doc, arg_count, program_text):
self.name = name self.doc = doc self.arg_count = arg_count self.program_text = program_text tabs = re.compile(r'^\t*')
def compile_user_function(name, doc, arg_count, eval_func):
def replace_func(mo):
return mo.group().replace('\t', ' ')

func = ' ' + '\n '.join([tabs.sub(replace_func, line )
for line in eval_func.splitlines()])
prog = '''
from calibre.utils.formatter_functions import FormatterUserFunction from calibre.utils.formatter_functions import formatter_functions class UserFunction(FormatterUserFunction):
''' + func locals_ = {}
if DEBUG and tweaks.get('enable_template_debug_printing', False):
print prog exec prog in locals_ cls = locals_['UserFunction'](name, doc, arg_count, eval_func)
return cls def load_user_template_functions(funcs):
formatter_functions().reset_to_builtins()
for func in funcs:
try:
cls = compile_user_function(*func)
formatter_functions().register_function(cls)
except:
traceback.print_exc()

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

- Module code »

Source code for calibre.web.feeds.news
from __future__ import with_statement
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
'''
Defines various abstract base classes that can be subclassed to create powerful news fetching recipes.
'''
__docformat__ = "restructuredtext en"

import os, time, traceback, re, urlparse, sys, cStringIO from collections import defaultdict from functools import partial from contextlib import nested, closing from calibre import (browser, __appname__, iswindows, force_unicode, strftime, preferred_encoding, as_unicode)
from calibre.ebooks.BeautifulSoup import BeautifulSoup, NavigableString, CData, Tag from calibre.ebooks.metadata.opf2 import OPFCreator from calibre import entity_to_unicode from calibre.web import Recipe from calibre.ebooks.metadata.toc import TOC from calibre.ebooks.metadata import MetaInformation from calibre.web.feeds import feed_from_xml, templates, feeds_from_index, Feed from calibre.web.fetch.simple import option_parser as web2disk_option_parser from calibre.web.fetch.simple import RecursiveFetcher from calibre.utils.threadpool import WorkRequest, ThreadPool, NoResultsPending from calibre.ptempfile import PersistentTemporaryFile from calibre.utils.date import now as nowf from calibre.utils.magick.draw import save_cover_data_to, add_borders_to_image from calibre.utils.localization import canonicalize_lang class LoginFailed(ValueError):
pass class DownloadDenied(ValueError):
pass

[docs]class BasicNewsRecipe(Recipe):
'''
Base class that contains logic needed in all recipes. By overriding progressively more of the functionality in this class, you can make progressively more customized/powerful recipes. For a tutorial introduction to creating recipes, see :doc:`news`.
'''

#: The title to use for the ebook title = _('Unknown News Source')

#: A couple of lines that describe the content this recipe downloads.
#: This will be used primarily in a GUI that presents a list of recipes.
description = u''

#: The author of this recipe
__author__ = __appname__

#: Minimum calibre version needed to use this recipe requires_version = (0, 6, 0)

#: The language that the news is in. Must be an ISO-639 code either
#: two or three characters long language = 'und'

#: Maximum number of articles to download from each feed. This is primarily
#: useful for feeds that don't have article dates. For most feeds, you should
#: use :attr:`BasicNewsRecipe.oldest_article` max_articles_per_feed = 100

#: Oldest article to download from this news source. In days.
oldest_article = 7.0

#: Number of levels of links to follow on article webpages recursions = 0

#: Delay between consecutive downloads in seconds. The argument may be a
#: floating point number to indicate a more precise time.
delay = 0

#: Publication type
#: Set to newspaper, magazine or blog. If set to None, no publication type
#: metadata will be written to the opf file.
publication_type = 'unknown'

#: Number of simultaneous downloads. Set to 1 if the server is picky.
#: Automatically reduced to 1 if :attr:`BasicNewsRecipe.delay` > 0 simultaneous_downloads = 5

#: Timeout for fetching files from server in seconds timeout = 120.0

#: The format string for the date shown on the first page.
#: By default: Day_Name, Day_Number Month_Name Year timefmt = ' [%a, %d %b %Y]'

#: List of feeds to download
#: Can be either ``[url1, url2, ...]`` or ``[('title1', url1), ('title2', url2),...]`` feeds = None

#: Max number of characters in the short description summary_length = 500

#: Convenient flag to disable loading of stylesheets for websites
#: that have overly complex stylesheets unsuitable for conversion
#: to ebooks formats
#: If True stylesheets are not downloaded and processed no_stylesheets = False

#: Convenient flag to strip all javascript tags from the downloaded HTML remove_javascript = True

#: If True the GUI will ask the user for a username and password
#: to use while downloading
#: If set to "optional" the use of a username and password becomes optional needs_subscription = False

#: If True the navigation bar is center aligned, otherwise it is left aligned center_navbar = True

#: Specify an override encoding for sites that have an incorrect
#: charset specification. The most common being specifying ``latin1`` and
#: using ``cp1252``. If None, try to detect the encoding. If it is a
#: callable, the callable is called with two arguments: The recipe object
#: and the source to be decoded. It must return the decoded source.
encoding = None

#: Normally we try to guess if a feed has full articles embedded in it
#: based on the length of the embedded content. If `None`, then the
#: default guessing is used. If `True` then the we always assume the feeds has
#: embedded content and if `False` we always assume the feed does not have
#: embedded content.
use_embedded_content = None

#: Set to True and implement :meth:`get_obfuscated_article` to handle
#: websites that try to make it difficult to scrape content.
articles_are_obfuscated = False

#: Reverse the order of articles in each feed reverse_article_order = False

#: Automatically extract all the text from downloaded article pages. Uses
#: the algorithms from the readability project. Setting this to True, means
#: that you do not have to worry about cleaning up the downloaded HTML
#: manually (though manual cleanup will always be superior).
auto_cleanup = False

#: Specify elements that the auto cleanup algorithm should never remove
#: The syntax is a XPath expression. For example::
#:
#: auto_cleanup_keep = '//div[@id="article-image"]' will keep all divs with
#: id="article-image"
#: auto_cleanup_keep = '//*[@class="important"]' will keep all elements
#: with class="important"
#: auto_cleanup_keep = '//div[@id="article-image"]|//span[@class="important"]'
#: will keep all divs with id="article-image" and spans
#: with class="important"
#:
auto_cleanup_keep = None

#: Specify any extra :term:`CSS` that should be added to downloaded :term:`HTML` files
#: It will be inserted into `<style>` tags, just before the closing
#: `</head>` tag thereby overriding all :term:`CSS` except that which is
#: declared using the style attribute on individual :term:`HTML` tags.
#: For example::
#:
#: extra_css = '.heading { font: serif x-large }'
#:
extra_css = None

#: If True empty feeds are removed from the output.
#: This option has no effect if parse_index is overridden in
#: the sub class. It is meant only for recipes that return a list
#: of feeds using `feeds` or :meth:`get_feeds`. It is also used if you use
#: the ignore_duplicate_articles option.
remove_empty_feeds = False

#: List of regular expressions that determines which links to follow
#: If empty, it is ignored. Used only if is_link_wanted is
#: not implemented. For example::
#:
#: match_regexps = [r'page=[0-9]+']
#:
#: will match all URLs that have `page=some number` in them.
#:
#: Only one of :attr:`BasicNewsRecipe.match_regexps` or
#: :attr:`BasicNewsRecipe.filter_regexps` should be defined.
match_regexps = []

#: List of regular expressions that determines which links to ignore
#: If empty it is ignored. Used only if is_link_wanted is not
#: implemented. For example::
#:
#: filter_regexps = [r'ads\.doubleclick\.net']
#:
#: will remove all URLs that have `ads.doubleclick.net` in them.
#:
#: Only one of :attr:`BasicNewsRecipe.match_regexps` or
#: :attr:`BasicNewsRecipe.filter_regexps` should be defined.
filter_regexps = []

#: Recipe specific options to control the conversion of the downloaded
#: content into an e-book. These will override any user or plugin specified
#: values, so only use if absolutely necessary. For example::
#:
#: conversion_options = {
#: 'base_font_size' : 16,
#: 'tags' : 'mytag1,mytag2',
#: 'title' : 'My Title',
#: 'linearize_tables' : True,
#: }
#:
conversion_options = {}

#: List of tags to be removed. Specified tags are removed from downloaded HTML.
#: A tag is specified as a dictionary of the form::
#:
#: {
#: name : 'tag name', #e.g. 'div'
#: attrs : a dictionary, #e.g. {class: 'advertisment'}
#: }
#:
#: All keys are optional. For a full explanantion of the search criteria, see
#: `Beautiful Soup <http://www.crummy.com/software/BeautifulSoup/documentation.html#The basic find method: findAll(name, attrs, recursive, text, limit, **kwargs)>`_
#: A common example::
#:
#: remove_tags = [dict(name='div', attrs={'class':'advert'})]
#:
#: This will remove all `<div class="advert">` tags and all
#: their children from the downloaded :term:`HTML`.
remove_tags = []

#: Remove all tags that occur after the specified tag.
#: For the format for specifying a tag see :attr:`BasicNewsRecipe.remove_tags`.
#: For example::
#:
#: remove_tags_after = [dict(id='content')]
#:
#: will remove all
#: tags after the first element with `id="content"`.
remove_tags_after = None

#: Remove all tags that occur before the specified tag.
#: For the format for specifying a tag see :attr:`BasicNewsRecipe.remove_tags`.
#: For example::
#:
#: remove_tags_before = dict(id='content')
#:
#: will remove all
#: tags before the first element with `id="content"`.
remove_tags_before = None

#: List of attributes to remove from all tags
#: For example::
#:
#: remove_attributes = ['style', 'font']
remove_attributes = []

#: Keep only the specified tags and their children.
#: For the format for specifying a tag see :attr:`BasicNewsRecipe.remove_tags`.
#: If this list is not empty, then the `<body>` tag will be emptied and re-filled with
#: the tags that match the entries in this list. For example::
#:
#: keep_only_tags = [dict(id=['content', 'heading'])]
#:
#: will keep only tags that have an `id` attribute of `"content"` or `"heading"`.
keep_only_tags = []

#: List of :term:`regexp` substitution rules to run on the downloaded :term:`HTML`.
#: Each element of the
#: list should be a two element tuple. The first element of the tuple should
#: be a compiled regular expression and the second a callable that takes
#: a single match object and returns a string to replace the match. For example::
#:
#: preprocess_regexps = [
#: (re.compile(r'<!--Article ends here-->.*</body>', re.DOTALL|re.IGNORECASE),
#: lambda match: '</body>'),
#: ]
#:
#: will remove everythong from `<!--Article ends here-->` to `</body>`.
preprocess_regexps = []

#: The CSS that is used to style the templates, i.e., the navigation bars and
#: the Tables of Contents. Rather than overriding this variable, you should
#: use `extra_css` in your recipe to customize look and feel.
template_css = u'''
.article_date { color: gray; font-family: monospace;
}

.article_description { text-indent: 0pt;
}

a.article { font-weight: bold; text-align:left;
}

a.feed { font-weight: bold;
}

.calibre_navbar { font-family:monospace;
}
'''

#: By default, calibre will use a default image for the masthead (Kindle only).
#: Override this in your recipe to provide a url to use as a masthead.
masthead_url = None

#: By default, the cover image returned by get_cover_url() will be used as
#: the cover for the periodical. Overriding this in your recipe instructs
#: calibre to render the downloaded cover into a frame whose width and height
#: are expressed as a percentage of the downloaded cover.
#: cover_margins = (10, 15, '#ffffff') pads the cover with a white margin
#: 10px on the left and right, 15px on the top and bottom.
#: Color names defined at http://www.imagemagick.org/script/color.php
#: Note that for some reason, white does not always work on windows. Use
#: #ffffff instead cover_margins = (0, 0, '#ffffff')

#: Set to a non empty string to disable this recipe
#: The string will be used as the disabled message recipe_disabled = None

#: Ignore duplicates of articles that are present in more than one section.
#: A duplicate article is an article that has the same title and/or URL.
#: To ignore articles with the same title, set this to:
#: ignore_duplicate_articles = {'title'}
#: To use URLs instead, set it to:
#: ignore_duplicate_articles = {'url'}
#: To match on title or URL, set it to:
#: ignore_duplicate_articles = {'title', 'url'}
ignore_duplicate_articles = None

# See the built-in profiles for examples of these settings.

def short_title(self):
return self.title

[docs] def is_link_wanted(self, url, tag):
'''
Return True if the link should be followed or False otherwise. By default, raises NotImplementedError which causes the downloader to ignore it.

:param url: The URL to be followed
:param tag: The Tag from which the URL was derived
'''
raise NotImplementedError

[docs] def get_cover_url(self):
'''
Return a :term:`URL` to the cover image for this issue or `None`.
By default it returns the value of the member `self.cover_url` which is normally `None`. If you want your recipe to download a cover for the e-book override this method in your subclass, or set the member variable `self.cover_url` before this method is called.
'''
return getattr(self, 'cover_url', None)

[docs] def get_masthead_url(self):
'''
Return a :term:`URL` to the masthead image for this issue or `None`.
By default it returns the value of the member `self.masthead_url` which is normally `None`. If you want your recipe to download a masthead for the e-book override this method in your subclass, or set the member variable `self.masthead_url` before this method is called.
Masthead images are used in Kindle MOBI files.
'''
return getattr(self, 'masthead_url', None)

[docs] def get_feeds(self):
'''
Return a list of :term:`RSS` feeds to fetch for this profile. Each element of the list must be a 2-element tuple of the form (title, url). If title is None or an empty string, the title from the feed is used. This method is useful if your recipe needs to do some processing to figure out the list of feeds to download. If so, override in your subclass.
'''
if not self.feeds:
raise NotImplementedError if self.test:
return self.feeds[:2]
return self.feeds

@classmethod

[docs] def print_version(self, url):
'''
Take a `url` pointing to the webpage with article content and return the
:term:`URL` pointing to the print version of the article. By default does nothing. For example::

def print_version(self, url):
return url + '?&pagewanted=print'

'''
raise NotImplementedError

@classmethod

[docs] def image_url_processor(cls, baseurl, url):
'''
Perform some processing on image urls (perhaps removing size restrictions for dynamically generated images, etc.) and return the precessed URL.
'''
return url

@classmethod

[docs] def get_browser(cls, *args, **kwargs):
'''
Return a browser instance used to fetch documents from the web. By default it returns a `mechanize <http://wwwsearch.sourceforge.net/mechanize/>`_ browser instance that supports cookies, ignores robots.txt, handles refreshes and has a mozilla firefox user agent.

If your recipe requires that you login first, override this method in your subclass. For example, the following code is used in the New York
Times recipe to login for full access::

def get_browser(self):
br = BasicNewsRecipe.get_browser()
if self.username is not None and self.password is not None:
br.open('http://www.nytimes.com/auth/login')
br.select_form(name='login')
br['USERID'] = self.username br['PASSWORD'] = self.password br.submit()
return br

'''
br = browser(*args, **kwargs)
br.addheaders += [('Accept', '*/*')]
return br

[docs] def clone_browser(self, br):
'''
Clone the browser br. Cloned browsers are used for multi-threaded downloads, since mechanize is not thread safe. The default cloning routines should capture most browser customization, but if you do something exotic in your recipe, you should override this method in your recipe and clone manually.

Cloned browser instances use the same, thread-safe CookieJar by default, unless you have customized cookie handling.
'''
if callable(getattr(br, 'clone_browser', None)):
return br.clone_browser()

# Uh-oh recipe using something exotic, call get_browser return self.get_browser()

@property def cloned_browser(self):
if self.get_browser.im_func is BasicNewsRecipe.get_browser.im_func:
# We are using the default get_browser, which means no need to
# clone br = BasicNewsRecipe.get_browser(self)
else:
br = self.clone_browser(self.browser)
return br

[docs] def get_article_url(self, article):
'''
Override in a subclass to customize extraction of the :term:`URL` that points to the content for each article. Return the article URL. It is called with `article`, an object representing a parsed article from a feed. See `feedparser <http://packages.python.org/feedparser/>`_.
By default it looks for the original link (for feeds syndicated via a service like feedburner or pheedo) and if found, returns that or else returns
`article.link <http://packages.python.org/feedparser/reference-entry-link.html>`_.
'''
for key in article.keys():
if key.endswith('_origlink'):
url = article[key]
if url and url.startswith('http://'):
return url ans = article.get('link', None)
if not ans and getattr(article, 'links', None):
for item in article.links:
if item.get('rel', 'alternate') == 'alternate':
ans = item['href']
break return ans

[docs] def skip_ad_pages(self, soup):
'''
This method is called with the source of each downloaded :term:`HTML` file, before any of the cleanup attributes like remove_tags, keep_only_tags are applied. Note that preprocess_regexps will have already been applied.
It is meant to allow the recipe to skip ad pages. If the soup represents an ad page, return the HTML of the real page. Otherwise return
None.

`soup`: A `BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/documentation.html>`_ instance containing the downloaded :term:`HTML`.
'''
return None

[docs] def preprocess_raw_html(self, raw_html, url):
'''
This method is called with the source of each downloaded :term:`HTML` file, before it is parsed into an object tree. raw_html is a unicode string representing the raw HTML downloaded from the web. url is the URL from which the HTML was downloaded.

Note that this method acts *before* preprocess_regexps.

This method must return the processed raw_html as a unicode object.
'''
return raw_html
def preprocess_raw_html_(self, raw_html, url):
raw_html = self.preprocess_raw_html(raw_html, url)
if self.auto_cleanup:
try:
raw_html = self.extract_readable_article(raw_html, url)
except:
self.log.exception('Auto cleanup of URL: %r failed'%url)

return raw_html

[docs] def preprocess_html(self, soup):
'''
This method is called with the source of each downloaded :term:`HTML` file, before it is parsed for links and images. It is called after the cleanup as specified by remove_tags etc.
It can be used to do arbitrarily powerful pre-processing on the :term:`HTML`.
It should return `soup` after processing it.

`soup`: A `BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/documentation.html>`_ instance containing the downloaded :term:`HTML`.
'''
return soup

[docs] def postprocess_html(self, soup, first_fetch):
'''
This method is called with the source of each downloaded :term:`HTML` file, after it is parsed for links and images.
It can be used to do arbitrarily powerful post-processing on the :term:`HTML`.
It should return `soup` after processing it.

:param soup: A `BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/documentation.html>`_ instance containing the downloaded :term:`HTML`.
:param first_fetch: True if this is the first page of an article.

'''
return soup

[docs] def cleanup(self):
'''
Called after all articles have been download. Use it to do any cleanup like logging out of subscription sites, etc.
'''
pass

[docs] def index_to_soup(self, url_or_raw, raw=False):
'''
Convenience method that takes an URL to the index page and returns a `BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/documentation.html>`_ of it.

`url_or_raw`: Either a URL or the downloaded index page as a string
'''
if re.match(r'\w+://', url_or_raw):
# We may be called in a thread (in the skip_ad_pages method), so
# clone the browser to be safe. We cannot use self.cloned_browser
# as it may or may not actually clone the browser, depending on if
# the recipe implements get_browser() or not br = self.clone_browser(self.browser)
open_func = getattr(br, 'open_novisit', br.open)
with closing(open_func(url_or_raw)) as f:
_raw = f.read()
if not _raw:
raise RuntimeError('Could not fetch index from %s'%url_or_raw)
else:
_raw = url_or_raw if raw:
return _raw if not isinstance(_raw, unicode) and self.encoding:
if callable(self.encoding):
_raw = self.encoding(_raw)
else:
_raw = _raw.decode(self.encoding, 'replace')
massage = list(BeautifulSoup.MARKUP_MASSAGE)
enc = 'cp1252' if callable(self.encoding) or self.encoding is None else self.encoding massage.append((re.compile(r'&(\S+?);'), lambda match:
entity_to_unicode(match, encoding=enc)))
return BeautifulSoup(_raw, markupMassage=massage)

[docs] def extract_readable_article(self, html, url):
'''
Extracts main article content from 'html', cleans up and returns as a (article_html, extracted_title) tuple.
Based on the original readability algorithm by Arc90.
'''
from calibre.ebooks.readability import readability from lxml.html import (fragment_fromstring, tostring, document_fromstring)

doc = readability.Document(html, self.log, url=url, keep_elements=self.auto_cleanup_keep)
article_html = doc.summary()
extracted_title = doc.title()

try:
frag = fragment_fromstring(article_html)
except:
doc = document_fromstring(article_html)
frag = doc.xpath('//body')[-1]
if frag.tag == 'html':
root = frag elif frag.tag == 'body':
root = document_fromstring( u'<html><head><title>%s</title></head></html>' % extracted_title)
root.append(frag)
else:
root = document_fromstring( u'<html><head><title>%s</title></head><body/></html>' % extracted_title)
root.xpath('//body')[0].append(frag)

body = root.xpath('//body')[0]
has_title = False for x in body.iterdescendants():
if x.text == extracted_title:
has_title = True inline_titles = body.xpath('//h1|//h2')
if not has_title and not inline_titles:
heading = body.makeelement('h2')
heading.text = extracted_title body.insert(0, heading)

raw_html = tostring(root, encoding=unicode)

return raw_html

[docs] def sort_index_by(self, index, weights):
'''
Convenience method to sort the titles in `index` according to `weights`.
`index` is sorted in place. Returns `index`.

`index`: A list of titles.

`weights`: A dictionary that maps weights to titles. If any titles in index are not in weights, they are assumed to have a weight of 0.
'''
weights = defaultdict(lambda : 0, weights)
index.sort(cmp=lambda x, y: cmp(weights[x], weights[y]))
return index

[docs] def parse_index(self):
'''
This method should be implemented in recipes that parse a website instead of feeds to generate a list of articles. Typical uses are for news sources that have a "Print Edition" webpage that lists all the articles in the current print edition. If this function is implemented, it will be used in preference to :meth:`BasicNewsRecipe.parse_feeds`.

It must return a list. Each element of the list must be a 2-element tuple of the form ``('feed title', list of articles)``.

Each list of articles must contain dictionaries of the form::

{
'title' : article title,
'url' : URL of print version,
'date' : The publication date of the article as a string,
'description' : A summary of the article
'content' : The full article (can be an empty string). Obsolete do not use, instead save the content to a temporary file and pass a file:///path/to/temp/file.html as the URL.
}

For an example, see the recipe for downloading `The Atlantic`.
In addition, you can add 'author' for the author of the article.

If you want to abort processing for some reason and have calibre show the user a simple message instead of an error, call
:meth:`abort_recipe_processing`.
'''
raise NotImplementedError

[docs] def abort_recipe_processing(self, msg):
'''
Causes the recipe download system to abort the download of this recipe, displaying a simple feedback message to the user.
'''
from calibre.ebooks.conversion import ConversionUserFeedBack raise ConversionUserFeedBack(_('Failed to download %s')%self.title, msg)

[docs] def get_obfuscated_article(self, url):
'''
If you set `articles_are_obfuscated` this method is called with every article URL. It should return the path to a file on the filesystem that contains the article HTML. That file is processed by the recursive
HTML fetching engine, so it can contain links to pages/images on the web.

This method is typically useful for sites that try to make it difficult to access article content automatically.
'''
raise NotImplementedError

[docs] def add_toc_thumbnail(self, article, src):
'''
Call this from populate_article_metadata with the src attribute of an
<img> tag from the article that is appropriate for use as the thumbnail representing the article in the Table of Contents. Whether the thumbnail is actually used is device dependent (currently only used by the Kindles). Note that the referenced image must be one that was successfully downloaded, otherwise it will be ignored.
'''
if not src or not hasattr(article, 'toc_thumbnail'):
return src = src.replace('\\', '/')
if re.search(r'feed_\d+/article_\d+/images/img', src, flags=re.I) is None:
self.log.warn('Ignoring invalid TOC thumbnail image: %r'%src)
return article.toc_thumbnail = re.sub(r'^.*?feed', 'feed', src, flags=re.IGNORECASE)

[docs] def populate_article_metadata(self, article, soup, first):
'''
Called when each HTML page belonging to article is downloaded.
Intended to be used to get article metadata like author/summary/etc.
from the parsed HTML (soup).
:param article: A object of class :class:`calibre.web.feeds.Article`.
If you change the summary, remember to also change the text_summary
:param soup: Parsed HTML belonging to this article
:param first: True iff the parsed HTML is the first page of the article.
'''
pass

[docs] def postprocess_book(self, oeb, opts, log):
'''
Run any needed post processing on the parsed downloaded e-book.

:param oeb: An OEBBook object
:param opts: Conversion options
'''
pass
def __init__(self, options, log, progress_reporter):
'''
Initialize the recipe.
:param options: Parsed commandline options
:param parser: Command line option parser. Used to intelligently merge options.
:param progress_reporter: A Callable that takes two arguments: progress (a number between 0 and 1) and a string message. The message should be optional.
'''
self.log = log if not isinstance(self.title, unicode):
self.title = unicode(self.title, 'utf-8', 'replace')

self.debug = options.verbose > 1 self.output_dir = os.path.abspath(os.getcwdu())
self.verbose = options.verbose self.test = options.test self.username = options.username self.password = options.password self.lrf = options.lrf self.output_profile = options.output_profile self.touchscreen = getattr(self.output_profile, 'touchscreen', False)
if self.touchscreen:
self.template_css += self.output_profile.touchscreen_news_css if options.test:
self.max_articles_per_feed = 2 self.simultaneous_downloads = min(4, self.simultaneous_downloads)

if self.debug:
self.verbose = True self.report_progress = progress_reporter if isinstance(self.feeds, basestring):
self.feeds = eval(self.feeds)
if isinstance(self.feeds, basestring):
self.feeds = [self.feeds]

if self.needs_subscription and (\ self.username is None or self.password is None or \ (not self.username and not self.password)):
if self.needs_subscription != 'optional':
raise ValueError(_('The "%s" recipe needs a username and password.')%self.title)

self.browser = self.get_browser()
self.image_map, self.image_counter = {}, 1 self.css_map = {}

web2disk_cmdline = [ 'web2disk',
'--timeout', str(self.timeout),
'--max-recursions', str(self.recursions),
'--delay', str(self.delay),
]

if self.verbose:
web2disk_cmdline.append('--verbose')

if self.no_stylesheets:
web2disk_cmdline.append('--dont-download-stylesheets')

for reg in self.match_regexps:
web2disk_cmdline.extend(['--match-regexp', reg])

for reg in self.filter_regexps:
web2disk_cmdline.extend(['--filter-regexp', reg])

self.web2disk_options = web2disk_option_parser().parse_args(web2disk_cmdline)[0]
for extra in ('keep_only_tags', 'remove_tags', 'preprocess_regexps',
'skip_ad_pages', 'preprocess_html', 'remove_tags_after',
'remove_tags_before', 'is_link_wanted'):
setattr(self.web2disk_options, extra, getattr(self, extra))
self.web2disk_options.postprocess_html = self._postprocess_html self.web2disk_options.encoding = self.encoding self.web2disk_options.preprocess_raw_html = self.preprocess_raw_html_ if self.delay > 0:
self.simultaneous_downloads = 1 self.navbar = templates.TouchscreenNavBarTemplate() if self.touchscreen else \ templates.NavBarTemplate()
self.failed_downloads = []
self.partial_failures = []

def _postprocess_html(self, soup, first_fetch, job_info):
if self.no_stylesheets:
for link in list(soup.findAll('link', type=re.compile('css')))+list(soup.findAll('style')):
link.extract()
head = soup.find('head')
if not head:
head = soup.find('body')
if not head:
head = soup.find(True)
style = BeautifulSoup(u'<style type="text/css" title="override_css">%s</style>'%(self.template_css +'\n\n'+(self.extra_css if self.extra_css else ''))).find('style')
head.insert(len(head.contents), style)
if first_fetch and job_info:
url, f, a, feed_len = job_info body = soup.find('body')
if body is not None:
templ = self.navbar.generate(False, f, a, feed_len, not self.has_single_feed, url, __appname__, center=self.center_navbar, extra_css=self.extra_css)
elem = BeautifulSoup(templ.render(doctype='xhtml').decode('utf-8')).find('div')
body.insert(0, elem)
if self.remove_javascript:
for script in list(soup.findAll('script')):
script.extract()
for o in soup.findAll(onload=True):
del o['onload']

for script in list(soup.findAll('noscript')):
script.extract()
for attr in self.remove_attributes:
for x in soup.findAll(attrs={attr:True}):
del x[attr]
for base in list(soup.findAll(['base', 'iframe', 'canvas', 'embed',
'command', 'datalist', 'video', 'audio'])):
base.extract()

ans = self.postprocess_html(soup, first_fetch)

# Nuke HTML5 tags for x in ans.findAll(['article', 'aside', 'header', 'footer', 'nav',
'figcaption', 'figure', 'section']):
x.name = 'div'

if job_info:
url, f, a, feed_len = job_info try:
article = self.feed_objects[f].articles[a]
except:
self.log.exception('Failed to get article object for postprocessing')
pass else:
self.populate_article_metadata(article, ans, first_fetch)
return ans

[docs] def download(self):
'''
Download and pre-process all articles from the feeds in this recipe.
This method should be called only once on a particular Recipe instance.
Calling it more than once will lead to undefined behavior.
:return: Path to index.html
'''
try:
res = self.build_index()
self.report_progress(1, _('Download finished'))
if self.failed_downloads:
self.log.warning(_('Failed to download the following articles:'))
for feed, article, debug in self.failed_downloads:
self.log.warning(article.title, 'from', feed.title)
self.log.debug(article.url)
self.log.debug(debug)
if self.partial_failures:
self.log.warning(_('Failed to download parts of the following articles:'))
for feed, atitle, aurl, debug in self.partial_failures:
self.log.warning(atitle + _(' from ') + feed)
self.log.debug(aurl)
self.log.warning(_('\tFailed links:'))
for l, tb in debug:
self.log.warning(l)
self.log.debug(tb)
return res finally:
self.cleanup()

@property def lang_for_html(self):
try:
lang = self.language.replace('_', '-').partition('-')[0].lower()
if lang == 'und':
lang = None except:
lang = None return lang def feeds2index(self, feeds):
templ = (templates.TouchscreenIndexTemplate if self.touchscreen else templates.IndexTemplate)
templ = templ(lang=self.lang_for_html)
css = self.template_css + '\n\n' +(self.extra_css if self.extra_css else '')
timefmt = self.timefmt return templ.generate(self.title, "mastheadImage.jpg", timefmt, feeds, extra_css=css).render(doctype='xhtml')

@classmethod def description_limiter(cls, src):
if not src:
return ''
pos = cls.summary_length fuzz = 50 si = src.find(';', pos)
if si > 0 and si-pos > fuzz:
si = -1 gi = src.find('>', pos)
if gi > 0 and gi-pos > fuzz:
gi = -1 npos = max(si, gi)
if npos < 0:
npos = pos ans = src[:npos+1]
if len(ans) < len(src):
return (ans+u'\u2026') if isinstance(ans, unicode) else (ans +
'...')
return ans def feed2index(self, f, feeds):
feed = feeds[f]
if feed.image_url is not None: # Download feed image imgdir = os.path.join(self.output_dir, 'images')
if not os.path.isdir(imgdir):
os.makedirs(imgdir)

if self.image_map.has_key(feed.image_url):
feed.image_url = self.image_map[feed.image_url]
else:
bn = urlparse.urlsplit(feed.image_url).path if bn:
bn = bn.rpartition('/')[-1]
if bn:
img = os.path.join(imgdir, 'feed_image_%d%s'%(self.image_counter, os.path.splitext(bn)))
try:
with nested(open(img, 'wb'), closing(self.browser.open(feed.image_url))) as (fi, r):
fi.write(r.read())
self.image_counter += 1 feed.image_url = img self.image_map[feed.image_url] = img except:
pass if isinstance(feed.image_url, str):
feed.image_url = feed.image_url.decode(sys.getfilesystemencoding(), 'strict')

templ = (templates.TouchscreenFeedTemplate if self.touchscreen else templates.FeedTemplate)
templ = templ(lang=self.lang_for_html)
css = self.template_css + '\n\n' +(self.extra_css if self.extra_css else '')

return templ.generate(f, feeds, self.description_limiter, extra_css=css).render(doctype='xhtml')

def _fetch_article(self, url, dir_, f, a, num_of_feeds):
br = self.browser if self.get_browser.im_func is BasicNewsRecipe.get_browser.im_func:
# We are using the default get_browser, which means no need to
# clone br = BasicNewsRecipe.get_browser(self)
else:
br = self.clone_browser(self.browser)
self.web2disk_options.browser = br fetcher = RecursiveFetcher(self.web2disk_options, self.log, self.image_map, self.css_map,
(url, f, a, num_of_feeds))
fetcher.browser = br fetcher.base_dir = dir_ fetcher.current_dir = dir_ fetcher.show_progress = False fetcher.image_url_processor = self.image_url_processor res, path, failures = fetcher.start_fetch(url), fetcher.downloaded_paths, fetcher.failed_links if not res or not os.path.exists(res):
msg = _('Could not fetch article.') + ' '
if self.debug:
msg += _('The debug traceback is available earlier in this log')
else:
msg += _('Run with -vv to see the reason')
raise Exception(msg)

return res, path, failures def fetch_article(self, url, dir, f, a, num_of_feeds):
return self._fetch_article(url, dir, f, a, num_of_feeds)

def fetch_obfuscated_article(self, url, dir, f, a, num_of_feeds):
path = os.path.abspath(self.get_obfuscated_article(url))
url = ('file:'+path) if iswindows else ('file://'+path)
return self._fetch_article(url, dir, f, a, num_of_feeds)

def fetch_embedded_article(self, article, dir, f, a, num_of_feeds):
templ = templates.EmbeddedContent()
raw = templ.generate(article).render('html')
with PersistentTemporaryFile('_feeds2disk.html') as pt:
pt.write(raw)
url = ('file:'+pt.name) if iswindows else ('file://'+pt.name)
return self._fetch_article(url, dir, f, a, num_of_feeds)

def remove_duplicate_articles(self, feeds):
seen_keys = defaultdict(set)
remove = []
for f in feeds:
for article in f:
for key in self.ignore_duplicate_articles:
val = getattr(article, key)
seen = seen_keys[key]
if val:
if val in seen:
remove.append((f, article))
else:
seen.add(val)

for feed, article in remove:
self.log.debug('Removing duplicate article: %s from section: %s'%( article.title, feed.title))
feed.remove_article(article)

if self.remove_empty_feeds:
feeds = [f for f in feeds if len(f) > 0]
return feeds def build_index(self):
self.report_progress(0, _('Fetching feeds...'))
try:
feeds = feeds_from_index(self.parse_index(), oldest_article=self.oldest_article, max_articles_per_feed=self.max_articles_per_feed, log=self.log)
self.report_progress(0, _('Got feeds from index page'))
except NotImplementedError:
feeds = self.parse_feeds()

if not feeds:
raise ValueError('No articles found, aborting')

if self.ignore_duplicate_articles is not None:
feeds = self.remove_duplicate_articles(feeds)

#feeds = FeedCollection(feeds)

self.report_progress(0, _('Trying to download cover...'))
self.download_cover()
self.report_progress(0, _('Generating masthead...'))
self.masthead_path = None try:
murl = self.get_masthead_url()
except:
self.log.exception('Failed to get masthead url')
murl = None if murl is not None:
# Try downloading the user-supplied masthead_url
# Failure sets self.masthead_path to None self.download_masthead(murl)
if self.masthead_path is None:
self.log.info("Synthesizing mastheadImage")
self.masthead_path = os.path.join(self.output_dir, 'mastheadImage.jpg')
try:
self.default_masthead_image(self.masthead_path)
except:
self.log.exception('Failed to generate default masthead image')
self.masthead_path = None if self.test:
feeds = feeds[:2]
self.has_single_feed = len(feeds) == 1 index = os.path.join(self.output_dir, 'index.html')

html = self.feeds2index(feeds)
with open(index, 'wb') as fi:
fi.write(html)

self.jobs = []

if self.reverse_article_order:
for feed in feeds:
if hasattr(feed, 'reverse'):
feed.reverse()

self.feed_objects = feeds for f, feed in enumerate(feeds):
feed_dir = os.path.join(self.output_dir, 'feed_%d'%f)
if not os.path.isdir(feed_dir):
os.makedirs(feed_dir)

for a, article in enumerate(feed):
if a >= self.max_articles_per_feed:
break art_dir = os.path.join(feed_dir, 'article_%d'%a)
if not os.path.isdir(art_dir):
os.makedirs(art_dir)
try:
url = self.print_version(article.url)
except NotImplementedError:
url = article.url except:
self.log.exception('Failed to find print version for: '+article.url)
url = None if not url:
continue func, arg = (self.fetch_embedded_article, article) \ if self.use_embedded_content or (self.use_embedded_content == None and feed.has_embedded_content()) \ else \ ((self.fetch_obfuscated_article if self.articles_are_obfuscated \ else self.fetch_article), url)
req = WorkRequest(func, (arg, art_dir, f, a, len(feed)),
{}, (f, a), self.article_downloaded, self.error_in_article_download)
req.feed = feed req.article = article req.feed_dir = feed_dir self.jobs.append(req)

self.jobs_done = 0 tp = ThreadPool(self.simultaneous_downloads)
for req in self.jobs:
tp.putRequest(req, block=True, timeout=0)

self.report_progress(0, _('Starting download [%d thread(s)]...')%self.simultaneous_downloads)
while True:
try:
tp.poll()
time.sleep(0.1)
except NoResultsPending:
break

#feeds.restore_duplicates()

for f, feed in enumerate(feeds):
html = self.feed2index(f,feeds)
feed_dir = os.path.join(self.output_dir, 'feed_%d'%f)
with open(os.path.join(feed_dir, 'index.html'), 'wb') as fi:
fi.write(html)
self.create_opf(feeds)
self.report_progress(1, _('Feeds downloaded to %s')%index)

return index def _download_cover(self):
self.cover_path = None try:
cu = self.get_cover_url()
except Exception as err:
self.log.error(_('Could not download cover: %s')%as_unicode(err))
self.log.debug(traceback.format_exc())
else:
if not cu:
return cdata = None if os.access(cu, os.R_OK):
cdata = open(cu, 'rb').read()
else:
self.report_progress(1, _('Downloading cover from %s')%cu)
with closing(self.browser.open(cu)) as r:
cdata = r.read()
if not cdata:
return ext = cu.split('/')[-1].rpartition('.')[-1].lower().strip()
if ext == 'pdf':
from calibre.ebooks.metadata.pdf import get_metadata stream = cStringIO.StringIO(cdata)
cdata = None mi = get_metadata(stream)
if mi.cover_data and mi.cover_data[1]:
cdata = mi.cover_data[1]
if not cdata:
return if self.cover_margins[0] or self.cover_margins[1]:
cdata = add_borders_to_image(cdata, left=self.cover_margins[0],right=self.cover_margins[0], top=self.cover_margins[1],bottom=self.cover_margins[1], border_color=self.cover_margins[2])

cpath = os.path.join(self.output_dir, 'cover.jpg')
save_cover_data_to(cdata, cpath)
self.cover_path = cpath def download_cover(self):
self.cover_path = None try:
self._download_cover()
except:
self.log.exception('Failed to download cover')
self.cover_path = None def _download_masthead(self, mu):
ext = mu.rpartition('.')[-1]
if '?' in ext:
ext = ''
ext = ext.lower() if ext else 'jpg'
mpath = os.path.join(self.output_dir, 'masthead_source.'+ext)
outfile = os.path.join(self.output_dir, 'mastheadImage.jpg')
if os.access(mu, os.R_OK):
with open(mpath, 'wb') as mfile:
mfile.write(open(mu, 'rb').read())
else:
with nested(open(mpath, 'wb'), closing(self.browser.open(mu))) as (mfile, r):
mfile.write(r.read())
self.report_progress(1, _('Masthead image downloaded'))
self.prepare_masthead_image(mpath, outfile)
self.masthead_path = outfile if os.path.exists(mpath):
os.remove(mpath)

def download_masthead(self, url):
try:
self._download_masthead(url)
except:
self.log.exception("Failed to download supplied masthead_url")

[docs] def default_cover(self, cover_file):
'''
Create a generic cover for recipes that dont have a cover
'''
try:
from calibre.ebooks import calibre_cover title = self.title if isinstance(self.title, unicode) else \ self.title.decode(preferred_encoding, 'replace')
date = strftime(self.timefmt)
img_data = calibre_cover(title, date)
cover_file.write(img_data)
cover_file.flush()
except:
self.log.exception('Failed to generate default cover')
return False return True

[docs] def get_masthead_title(self):
'Override in subclass to use something other than the recipe title'
return self.title

MI_WIDTH = 600
MI_HEIGHT = 60 def default_masthead_image(self, out_path):
from calibre.ebooks import generate_masthead generate_masthead(self.get_masthead_title(), output_path=out_path, width=self.MI_WIDTH, height=self.MI_HEIGHT)

def prepare_masthead_image(self, path_to_image, out_path):
from calibre import fit_image from calibre.utils.magick import Image, create_canvas img = Image()
img.open(path_to_image)
width, height = img.size scaled, nwidth, nheight = fit_image(width, height, self.MI_WIDTH, self.MI_HEIGHT)
img2 = create_canvas(width, height)
frame = create_canvas(self.MI_WIDTH, self.MI_HEIGHT)
img2.compose(img)
if scaled:
img2.size = (nwidth, nheight, 'LanczosFilter', 0.5)
left = int((self.MI_WIDTH - nwidth)/2.0)
top = int((self.MI_HEIGHT - nheight)/2.0)
frame.compose(img2, left, top)
frame.save(out_path)

def create_opf(self, feeds, dir=None):
if dir is None:
dir = self.output_dir title = self.short_title()
if self.output_profile.periodical_date_in_title:
title += strftime(self.timefmt)
mi = MetaInformation(title, [__appname__])
mi.publisher = __appname__ mi.author_sort = __appname__ if self.publication_type:
mi.publication_type = 'periodical:'+self.publication_type+':'+self.short_title()
mi.timestamp = nowf()
article_titles, aseen = [], set()
for f in feeds:
for a in f:
if a.title and a.title not in aseen:
aseen.add(a.title)
article_titles.append(force_unicode(a.title, 'utf-8'))

mi.comments = self.description if not isinstance(mi.comments, unicode):
mi.comments = mi.comments.decode('utf-8', 'replace')
mi.comments += ('\n\n' + _('Articles in this issue: ') + '\n' +
'\n\n'.join(article_titles))

language = canonicalize_lang(self.language)
if language is not None:
mi.language = language mi.pubdate = nowf()
opf_path = os.path.join(dir, 'index.opf')
ncx_path = os.path.join(dir, 'index.ncx')

opf = OPFCreator(dir, mi)
# Add mastheadImage entry to <guide> section mp = getattr(self, 'masthead_path', None)
if mp is not None and os.access(mp, os.R_OK):
from calibre.ebooks.metadata.opf2 import Guide ref = Guide.Reference(os.path.basename(self.masthead_path), os.getcwdu())
ref.type = 'masthead'
ref.title = 'Masthead Image'
opf.guide.append(ref)

manifest = [os.path.join(dir, 'feed_%d'%i) for i in range(len(feeds))]
manifest.append(os.path.join(dir, 'index.html'))
manifest.append(os.path.join(dir, 'index.ncx'))

# Get cover cpath = getattr(self, 'cover_path', None)
if cpath is None:
pf = open(os.path.join(dir, 'cover.jpg'), 'wb')
if self.default_cover(pf):
cpath = pf.name if cpath is not None and os.access(cpath, os.R_OK):
opf.cover = cpath manifest.append(cpath)

# Get masthead mpath = getattr(self, 'masthead_path', None)
if mpath is not None and os.access(mpath, os.R_OK):
manifest.append(mpath)

opf.create_manifest_from_files_in(manifest)
for mani in opf.manifest:
if mani.path.endswith('.ncx'):
mani.id = 'ncx'
if mani.path.endswith('mastheadImage.jpg'):
mani.id = 'masthead-image'

entries = ['index.html']
toc = TOC(base_path=dir)
self.play_order_counter = 0 self.play_order_map = {}

def feed_index(num, parent):
f = feeds[num]
for j, a in enumerate(f):
if getattr(a, 'downloaded', False):
adir = 'feed_%d/article_%d/'%(num, j)
auth = a.author if not auth:
auth = None desc = a.text_summary if not desc:
desc = None else:
desc = self.description_limiter(desc)
tt = a.toc_thumbnail if a.toc_thumbnail else None entries.append('%sindex.html'%adir)
po = self.play_order_map.get(entries[-1], None)
if po is None:
self.play_order_counter += 1 po = self.play_order_counter parent.add_item('%sindex.html'%adir, None, a.title if a.title else _('Untitled Article'), play_order=po, author=auth, description=desc, toc_thumbnail=tt)
last = os.path.join(self.output_dir, ('%sindex.html'%adir).replace('/', os.sep))
for sp in a.sub_pages:
prefix = os.path.commonprefix([opf_path, sp])
relp = sp[len(prefix):]
entries.append(relp.replace(os.sep, '/'))
last = sp if os.path.exists(last):
with open(last, 'rb') as fi:
src = fi.read().decode('utf-8')
soup = BeautifulSoup(src)
body = soup.find('body')
if body is not None:
prefix = '/'.join('..'for i in range(2*len(re.findall(r'link\d+', last))))
templ = self.navbar.generate(True, num, j, len(f), not self.has_single_feed, a.orig_url, __appname__, prefix=prefix, center=self.center_navbar)
elem = BeautifulSoup(templ.render(doctype='xhtml').decode('utf-8')).find('div')
body.insert(len(body.contents), elem)
with open(last, 'wb') as fi:
fi.write(unicode(soup).encode('utf-8'))
if len(feeds) == 0:
raise Exception('All feeds are empty, aborting.')

if len(feeds) > 1:
for i, f in enumerate(feeds):
entries.append('feed_%d/index.html'%i)
po = self.play_order_map.get(entries[-1], None)
if po is None:
self.play_order_counter += 1 po = self.play_order_counter auth = getattr(f, 'author', None)
if not auth:
auth = None desc = getattr(f, 'description', None)
if not desc:
desc = None feed_index(i, toc.add_item('feed_%d/index.html'%i, None, f.title, play_order=po, description=desc, author=auth))

else:
entries.append('feed_%d/index.html'%0)
feed_index(0, toc)

for i, p in enumerate(entries):
entries[i] = os.path.join(dir, p.replace('/', os.sep))
opf.create_spine(entries)
opf.set_toc(toc)

with nested(open(opf_path, 'wb'), open(ncx_path, 'wb')) as (opf_file, ncx_file):
opf.render(opf_file, ncx_file)

def article_downloaded(self, request, result):
index = os.path.join(os.path.dirname(result[0]), 'index.html')
if index != result[0]:
if os.path.exists(index):
os.remove(index)
os.rename(result[0], index)
a = request.requestID[1]

article = request.article self.log.debug('Downloaded article:', article.title, 'from', article.url)
article.orig_url = article.url article.url = 'article_%d/index.html'%a article.downloaded = True article.sub_pages = result[1][1:]
self.jobs_done += 1 self.report_progress(float(self.jobs_done)/len(self.jobs),
_(u'Article downloaded: %s')%force_unicode(article.title))
if result[2]:
self.partial_failures.append((request.feed.title, article.title, article.url, result[2]))

def error_in_article_download(self, request, traceback):
self.jobs_done += 1 self.log.error('Failed to download article:', request.article.title,
'from', request.article.url)
self.log.debug(traceback)
self.log.debug('\n')
self.report_progress(float(self.jobs_done)/len(self.jobs),
_('Article download failed: %s')%force_unicode(request.article.title))
self.failed_downloads.append((request.feed, request.article, traceback))

[docs] def parse_feeds(self):
'''
Create a list of articles from the list of feeds returned by :meth:`BasicNewsRecipe.get_feeds`.
Return a list of :class:`Feed` objects.
'''
feeds = self.get_feeds()
parsed_feeds = []
for obj in feeds:
if isinstance(obj, basestring):
title, url = None, obj else:
title, url = obj if url.startswith('feed://'):
url = 'http'+url[4:]
self.report_progress(0, _('Fetching feed')+' %s...'%(title if title else url))
try:
with closing(self.browser.open(url)) as f:
parsed_feeds.append(feed_from_xml(f.read(), title=title, log=self.log, oldest_article=self.oldest_article, max_articles_per_feed=self.max_articles_per_feed, get_article_url=self.get_article_url))
if (self.delay > 0):
time.sleep(self.delay)
except Exception as err:
feed = Feed()
msg = 'Failed feed: %s'%(title if title else url)
feed.populate_from_preparsed_feed(msg, [])
feed.description = as_unicode(err)
parsed_feeds.append(feed)
self.log.exception(msg)

remove = [f for f in parsed_feeds if len(f) == 0 and self.remove_empty_feeds]
for f in remove:
parsed_feeds.remove(f)

return parsed_feeds
@classmethod

[docs] def tag_to_string(self, tag, use_alt=True, normalize_whitespace=True):
'''
Convenience method to take a
`BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/documentation.html>`_
`Tag` and extract the text from it recursively, including any CDATA sections and alt tag attributes. Return a possibly empty unicode string.

`use_alt`: If `True` try to use the alt attribute for tags that don't have any textual content

`tag`: `BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/documentation.html>`_
`Tag`
'''
if not tag:
return ''
if isinstance(tag, basestring):
return tag strings = []
for item in tag.contents:
if isinstance(item, (NavigableString, CData)):
strings.append(item.string)
elif isinstance(item, Tag):
res = self.tag_to_string(item)
if res:
strings.append(res)
elif use_alt and item.has_key('alt'):
strings.append(item['alt'])
ans = u''.join(strings)
if normalize_whitespace:
ans = re.sub(r'\s+', ' ', ans)
return ans

@classmethod def soup(cls, raw):
entity_replace = [(re.compile(ur'&(\S+?);'), partial(entity_to_unicode, exceptions=[]))]
nmassage = list(BeautifulSoup.MARKUP_MASSAGE)
nmassage.extend(entity_replace)
return BeautifulSoup(raw, markupMassage=nmassage)

@classmethod

[docs] def adeify_images(cls, soup):
'''
If your recipe when converted to EPUB has problems with images when viewed in Adobe Digital Editions, call this method from within
:meth:`postprocess_html`.
'''
for item in soup.findAll('img'):
for attrib in ['height','width','border','align','style']:
if item.has_key(attrib):
del item[attrib]
oldParent = item.parent myIndex = oldParent.contents.index(item)
item.extract()
divtag = Tag(soup,'div')
brtag = Tag(soup,'br')
oldParent.insert(myIndex,divtag)
divtag.append(item)
divtag.append(brtag)
return soup
class CustomIndexRecipe(BasicNewsRecipe):

def custom_index(self):
'''
Return the filesystem path to a custom HTML document that will serve as the index for this recipe. The index document will typically contain many `<a href="...">` tags that point to resources on the internet that should be downloaded.
'''
raise NotImplementedError def create_opf(self):
mi = MetaInformation(self.title + strftime(self.timefmt), [__appname__])
mi.publisher = __appname__ mi.author_sort = __appname__ mi = OPFCreator(self.output_dir, mi)
mi.create_manifest_from_files_in([self.output_dir])
mi.create_spine([os.path.join(self.output_dir, 'index.html')])
with open(os.path.join(self.output_dir, 'index.opf'), 'wb') as opf_file:
mi.render(opf_file)

def download(self):
index = os.path.abspath(self.custom_index())
url = 'file:'+index if iswindows else 'file://'+index self.web2disk_options.browser = self.clone_browser(self.browser)
fetcher = RecursiveFetcher(self.web2disk_options, self.log)
fetcher.base_dir = self.output_dir fetcher.current_dir = self.output_dir fetcher.show_progress = False res = fetcher.start_fetch(url)
self.create_opf()
return res class AutomaticNewsRecipe(BasicNewsRecipe):

auto_cleanup = True class CalibrePeriodical(BasicNewsRecipe):

#: Set this to the slug for the calibre periodical calibre_periodicals_slug = None

LOG_IN = 'http://news.calibre-ebook.com/accounts/login'
needs_subscription = True
__author__ = 'calibre Periodicals'

def get_browser(self):
br = BasicNewsRecipe.get_browser(self)
br.open(self.LOG_IN)
br.select_form(name='login')
br['username'] = self.username br['password'] = self.password raw = br.submit().read()
if 'href="/my-account"' not in raw:
raise LoginFailed(
_('Failed to log in, check your username and password for'
' the calibre Periodicals service.'))

return br def download(self):
self.log('Fetching downloaded recipe')
try:
raw = self.browser.open_novisit(
'http://news.calibre-ebook.com/subscribed_files/%s/0/temp.downloaded_recipe'
% self.calibre_periodicals_slug
).read()
except Exception as e:
if hasattr(e, 'getcode') and e.getcode() == 403:
raise DownloadDenied(
_('You do not have permission to download this issue.'
' Either your subscription has expired or you have'
' exceeded the maximum allowed downloads for today.'))
raise f = cStringIO.StringIO(raw)
from calibre.utils.zipfile import ZipFile zf = ZipFile(f)
zf.extractall()
zf.close()
from calibre.web.feeds.recipes import compile_recipe from glob import glob try:
recipe = compile_recipe(open(glob('*.recipe')[0],
'rb').read())
self.conversion_options = recipe.conversion_options except:
self.log.exception('Failed to compile downloaded recipe')
return os.path.abspath('index.html')

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- Start »

All modules for which code is available

- calibre.customize

- calibre.customize.conversion

- calibre.devices.interface
- calibre.devices.usbms.cli
- calibre.devices.usbms.device
- calibre.devices.usbms.driver
- calibre.ebooks.metadata.book.base
- calibre.ebooks.metadata.sources.base
- calibre.gui2.actions
- calibre.gui2.preferences
- calibre.utils.formatter_functions
- calibre.web.feeds.news

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Navigation

- next
- previous |
- Start »
- Customizing calibre »

API Documentation for plugins

Defines various abstract base classes that can be subclassed to create powerful plugins. The useful classes are:

- Plugin
- FileTypePlugin
- Metadata plugins
- Catalog plugins
- Metadata download plugins
- Conversion plugins
- Device Drivers
- User Interface Actions
- Preferences Plugins

Plugin
class calibre.customize.Plugin(plugin_path)[source]

A calibre plugin. Useful members include:

- self.plugin_path: Stores path to the zip file that contains

this plugin or None if it is a builtin plugin

- self.site_customization: Stores a customization string entered

by the user.

Methods that should be overridden in sub classes:

- initialize()

- customization_help()

Useful methods:

- temporary_file()

supported_platforms = []

List of platforms this plugin works on
For example: ['windows', 'osx', 'linux']

name = 'Trivial Plugin'

The name of this plugin. You must set it something other than Trivial Plugin for it to work.

version = (1, 0, 0)

The version of this plugin as a 3-tuple (major, minor, revision)

description = u'Does absolutely nothing'

A short string describing what this plugin does

author = u'Unknown'

The author of this plugin

priority = 1

When more than one plugin exists for a filetype, the plugins are run in order of decreasing priority i.e. plugins with higher priority will be run first.
The highest possible priority is sys.maxint.
Default priority is 1.

minimum_calibre_version = (0, 4, 118)

The earliest version of calibre this plugin requires

can_be_disabled = True

If False, the user will not be able to disable this plugin. Use with care.

type = u'Base'

The type of this plugin. Used for categorizing plugins in the
GUI

initialize()[source]

Called once when calibre plugins are initialized. Plugins are re-initialized every time a new plugin is added.

Perform any plugin specific initialization here, such as extracting resources from the plugin zip file. The path to the zip file is available as self.plugin_path.

Note that self.site_customization is not available at this point.

config_widget()[source]

Implement this method and save_settings() in your plugin to use a custom configuration dialog, rather then relying on the simple string based default customization.

This method, if implemented, must return a QWidget. The widget can have an optional method validate() that takes no arguments and is called immediately after the user clicks OK. Changes are applied if and only if the method returns True.

If for some reason you cannot perform the configuration at this time, return a tuple of two strings (message, details), these will be displayed as a warning dialog to the user and the process will be aborted.

save_settings(config_widget)[source]

Save the settings specified by the user with config_widget.

Parameters: | config_widget – The widget returned by config_widget().

do_user_config(parent=None)[source]

This method shows a configuration dialog for this plugin. It returns
True if the user clicks OK, False otherwise. The changes are automatically applied.

load_resources(names)[source]

If this plugin comes in a ZIP file (user added plugin), this method will allow you to load resources from the ZIP file.

For example to load an image:

pixmap = QPixmap()
pixmap.loadFromData(self.load_resources(['images/icon.png']).itervalues().next())
icon = QIcon(pixmap)

Parameters: | names – List of paths to resources in the zip file using / as separator

Returns:

A dictionary of the form {name : file_contents}. Any names that were not found in the zip file will not be present in the dictionary.

customization_help(gui=False)[source]

Return a string giving help on how to customize this plugin.
By default raise a NotImplementedError, which indicates that the plugin does not require customization.

If you re-implement this method in your subclass, the user will be asked to enter a string as customization for this plugin.
The customization string will be available as self.site_customization.

Site customization could be anything, for example, the path to a needed binary on the user’s computer.

Parameters: | gui – If True return HTML help, otherwise return plain text help.

temporary_file(suffix)[source]

Return a file-like object that is a temporary file on the file system.
This file will remain available even after being closed and will only be removed on interpreter shutdown. Use the name member of the returned object to access the full path to the created temporary file.

Parameters: | suffix – The suffix that the temporary file will have.

FileTypePlugin

class calibre.customize.FileTypePlugin(plugin_path)[source]

Bases: calibre.customize.Plugin

A plugin that is associated with a particular set of file types.

file_types = set([])

Set of file types for which this plugin should be run
For example: set(['lit', 'mobi', 'prc'])

on_import = False

If True, this plugin is run when books are added to the database

on_postimport = False

If True, this plugin is run after books are added to the database

on_preprocess = False

If True, this plugin is run just before a conversion

on_postprocess = False

If True, this plugin is run after conversion on the final file produced by the conversion output plugin.

run(path_to_ebook)[source]

Run the plugin. Must be implemented in subclasses.
It should perform whatever modifications are required on the ebook and return the absolute path to the modified ebook. If no modifications are needed, it should return the path to the original ebook. If an error is encountered it should raise an Exception. The default implementation simply return the path to the original ebook.

The modified ebook file should be created with the temporary_file() method.

Parameters: | path_to_ebook – Absolute path to the ebook.
Returns: | Absolute path to the modified ebook.

postimport(book_id, book_format, db)[source]

Called post import, i.e., after the book file has been added to the database.

Parameters:

- book_id – Database id of the added book.

- book_format – The file type of the book that was added.
:param db: Library database.

Metadata plugins

class calibre.customize.MetadataReaderPlugin(*args, **kwargs)[source]

Bases: calibre.customize.Plugin

A plugin that implements reading metadata from a set of file types.

file_types = set([])

Set of file types for which this plugin should be run
For example: set(['lit', 'mobi', 'prc'])

get_metadata(stream, type)[source]

Return metadata for the file represented by stream (a file like object that supports reading). Raise an exception when there is an error with the input data.
:param type: The type of file. Guaranteed to be one of the entries in file_types.
:return: A calibre.ebooks.metadata.book.Metadata object

class calibre.customize.MetadataWriterPlugin(*args, **kwargs)[source]

Bases: calibre.customize.Plugin

A plugin that implements reading metadata from a set of file types.

file_types = set([])

Set of file types for which this plugin should be run
For example: set(['lit', 'mobi', 'prc'])

set_metadata(stream, mi, type)[source]

Set metadata for the file represented by stream (a file like object that supports reading). Raise an exception when there is an error with the input data.
:param type: The type of file. Guaranteed to be one of the entries in file_types.
:param mi: A calibre.ebooks.metadata.book.Metadata object

Catalog plugins

class calibre.customize.CatalogPlugin(plugin_path)[source]

Bases: calibre.customize.Plugin

A plugin that implements a catalog generator.

file_types = set([])

Output file type for which this plugin should be run
For example: ‘epub’ or ‘xml’

cli_options = []

CLI parser options specific to this plugin, declared as namedtuple Option:

from collections import namedtuple
Option = namedtuple('Option', 'option, default, dest, help')
cli_options = [Option('--catalog-title',
default = 'My Catalog',
dest = 'catalog_title',
help = (_('Title of generated catalog. \nDefault:') + " '" +
'%default' + "'"))]
cli_options parsed in library.cli:catalog_option_parser()

initialize()[source]

If plugin is not a built-in, copy the plugin’s .ui and .py files from the zip file to $TMPDIR.
Tab will be dynamically generated and added to the Catalog Options dialog in calibre.gui2.dialogs.catalog.py:Catalog

run(path_to_output, opts, db, ids, notification=None)[source]

Run the plugin. Must be implemented in subclasses.
It should generate the catalog in the format specified in file_types, returning the absolute path to the generated catalog file. If an error is encountered it should raise an Exception.

The generated catalog file should be created with the temporary_file() method.

Parameters:

- path_to_output – Absolute path to the generated catalog file.

- opts – A dictionary of keyword arguments

- db – A LibraryDatabase2 object

Metadata download plugins
class calibre.ebooks.metadata.sources.base.Source(*args, **kwargs)[source]

Bases: calibre.customize.Plugin

capabilities = frozenset([])

Set of capabilities supported by this plugin.
Useful capabilities are: ‘identify’, ‘cover’

touched_fields = frozenset([])

List of metadata fields that can potentially be download by this plugin during the identify phase

has_html_comments = False

Set this to True if your plugin return HTML formatted comments

supports_gzip_transfer_encoding = False

Setting this to True means that the browser object will add
Accept-Encoding: gzip to all requests. This can speedup downloads but make sure that the source actually supports gzip transfer encoding correctly first

cached_cover_url_is_reliable = True

Cached cover URLs can sometimes be unreliable (i.e. the download could fail or the returned image could be bogus. If that is often the case with this source set to False

options = ()

A list of Option objects. They will be used to automatically construct the configuration widget for this plugin

config_help_message = None

A string that is displayed at the top of the config widget for this plugin

is_configured()[source]

Return False if your plugin needs to be configured before it can be used. For example, it might need a username/password/API key.

get_author_tokens(authors, only_first_author=True)[source]

Take a list of authors and return a list of tokens useful for an
AND search query. This function tries to return tokens in first name middle names last name order, by assuming that if a comma is in the author name, the name is in lastname, other names form.

get_title_tokens(title, strip_joiners=True, strip_subtitle=False)[source]

Take a title and return a list of tokens useful for an AND search query.
Excludes connectives(optionally) and punctuation.

split_jobs(jobs, num)[source]

Split a list of jobs into at most num groups, as evenly as possible

test_fields(mi)[source]

Return the first field from self.touched_fields that is null on the mi object

clean_downloaded_metadata(mi)[source]

Call this method in your plugin’s identify method to normalize metadata before putting the Metadata object into result_queue. You can of course, use a custom algorithm suited to your metadata source.

get_book_url(identifiers)[source]

Return a 3-tuple or None. The 3-tuple is of the form:
(identifier_type, identifier_value, URL).
The URL is the URL for the book identified by identifiers at this source. identifier_type, identifier_value specify the identifier corresponding to the URL.
This URL must be browseable to by a human using a browser. It is meant to provide a clickable link for the user to easily visit the books page at this source.
If no URL is found, return None. This method must be quick, and consistent, so only implement it if it is possible to construct the URL from a known scheme given identifiers.

get_book_url_name(idtype, idval, url)[source]

Return a human readable name from the return value of get_book_url().

get_cached_cover_url(identifiers)[source]

Return cached cover URL for the book identified by the identifiers dict or None if no such URL exists.

Note that this method must only return validated URLs, i.e. not URLS that could result in a generic cover image or a not found error.

identify_results_keygen(title=None, authors=None, identifiers={})[source]

Return a function that is used to generate a key that can sort Metadata objects by their relevance given a search query (title, authors, identifiers).

These keys are used to sort the results of a call to identify().

For details on the default algorithm see InternalMetadataCompareKeyGen. Re-implement this function in your plugin if the default algorithm is not suitable.

identify(log, result_queue, abort, title=None, authors=None, identifiers={}, timeout=30)[source]

Identify a book by its title/author/isbn/etc.

If identifiers(s) are specified and no match is found and this metadata source does not store all related identifiers (for example, all ISBNs of a book), this method should retry with just the title and author
(assuming they were specified).

If this metadata source also provides covers, the URL to the cover should be cached so that a subsequent call to the get covers API with the same ISBN/special identifier does not need to get the cover URL again. Use the caching API for this.

Every Metadata object put into result_queue by this method must have a source_relevance attribute that is an integer indicating the order in which the results were returned by the metadata source for this query.
This integer will be used by compare_identify_results(). If the order is unimportant, set it to zero for every result.

Make sure that any cover/isbn mapping information is cached before the
Metadata object is put into result_queue.

Parameters:

- log – A log object, use it to output debugging information/errors

- result_queue – A result Queue, results should be put into it.
Each result is a Metadata object

- abort – If abort.is_set() returns True, abort further processing and return as soon as possible

- title – The title of the book, can be None

- authors – A list of authors of the book, can be None

- identifiers – A dictionary of other identifiers, most commonly
{‘isbn’:‘1234...’}

- timeout – Timeout in seconds, no network request should hang for longer than timeout.

Returns:

None if no errors occurred, otherwise a unicode representation of the error suitable for showing to the user

download_cover(log, result_queue, abort, title=None, authors=None, identifiers={}, timeout=30)[source]

Download a cover and put it into result_queue. The parameters all have the same meaning as for identify(). Put (self, cover_data) into result_queue.

This method should use cached cover URLs for efficiency whenever possible. When cached data is not present, most plugins simply call identify and use its results.

class calibre.ebooks.metadata.sources.base.InternalMetadataCompareKeyGen(mi, source_plugin, title, authors, identifiers)[source]

Generate a sort key for comparison of the relevance of Metadata objects, given a search query. This is used only to compare results from the same metadata source, not across different sources.

The sort key ensures that an ascending order sort is a sort by order of decreasing relevance.

The algorithm is:

- Prefer results that have the same ISBN as specified in the query

- Prefer results with a cached cover URL

- Prefer results with all available fields filled in

- Prefer results that are an exact title match to the query

- Prefer results with longer comments (greater than 10% longer)

- Use the relevance of the result as reported by the metadata source’s search

engine

Conversion plugins
class calibre.customize.conversion.InputFormatPlugin(*args)[source]

Bases: calibre.customize.Plugin

InputFormatPlugins are responsible for converting a document into
HTML+OPF+CSS+etc.
The results of the conversion must be encoded in UTF-8.
The main action happens in convert().

file_types = set([])

Set of file types for which this plugin should be run
For example: set(['azw', 'mobi', 'prc'])

is_image_collection = False

If True, this input plugin generates a collection of images, one per HTML file. You can obtain access to the images via convenience method, get_image_collection().

core_usage = 1

Number of CPU cores used by this plugin
A value of -1 means that it uses all available cores

for_viewer = False

If set to True, the input plugin will perform special processing to make its output suitable for viewing

output_encoding = 'utf-8'

The encoding that this input plugin creates files in. A value of
None means that the encoding is undefined and must be detected individually

common_options = set([<calibre.customize.conversion.OptionRecommendation object at 0x2253650>])

Options shared by all Input format plugins. Do not override in sub-classes. Use options instead. Every option must be an instance of OptionRecommendation.

options = set([])

Options to customize the behavior of this plugin. Every option must be an instance of OptionRecommendation.

recommendations = set([])

A set of 3-tuples of the form
(option_name, recommended_value, recommendation_level)

get_images()[source]

Return a list of absolute paths to the images, if this input plugin represents an image collection. The list of images is in the same order as the spine and the TOC.

convert(stream, options, file_ext, log, accelerators)[source]

This method must be implemented in sub-classes. It must return the path to the created OPF file or an OEBBook instance.
All output should be contained in the current directory.
If this plugin creates files outside the current directory they must be deleted/marked for deletion before this method returns.

Parameters:

- stream – A file like object that contains the input file.

- options – Options to customize the conversion process.
Guaranteed to have attributes corresponding to all the options declared by this plugin. In addition, it will have a verbose attribute that takes integral values from zero upwards. Higher numbers mean be more verbose. Another useful attribute is input_profile that is an instance of calibre.customize.profiles.InputProfile.

- file_ext – The extension (without the .) of the input file. It is guaranteed to be one of the file_types supported by this plugin.

- log – A calibre.utils.logging.Log object. All output should use this object.

- accelarators – A dictionary of various information that the input plugin can get easily that would speed up the subsequent stages of the conversion.

postprocess_book(oeb, opts, log)[source]

Called to allow the input plugin to perform postprocessing after the book has been parsed.

specialize(oeb, opts, log, output_fmt)[source]

Called to allow the input plugin to specialize the parsed book for a particular output format. Called after postprocess_book and before any transforms are performed on the parsed book.

gui_configuration_widget(parent, get_option_by_name, get_option_help, db, book_id=None)[source]

Called to create the widget used for configuring this plugin in the calibre GUI. The widget must be an instance of the PluginWidget class.
See the builting input plugins for examples.

class calibre.customize.conversion.OutputFormatPlugin(*args)[source]

Bases: calibre.customize.Plugin

OutputFormatPlugins are responsible for converting an OEB document
(OPF+HTML) into an output ebook.

The OEB document can be assumed to be encoded in UTF-8.
The main action happens in convert().

file_type = None

The file type (extension without leading period) that this plugin outputs

common_options = set([<calibre.customize.conversion.OptionRecommendation object at 0x2253790>])

Options shared by all Input format plugins. Do not override in sub-classes. Use options instead. Every option must be an instance of OptionRecommendation.

options = set([])

Options to customize the behavior of this plugin. Every option must be an instance of OptionRecommendation.

recommendations = set([])

A set of 3-tuples of the form
(option_name, recommended_value, recommendation_level)

convert(oeb_book, output, input_plugin, opts, log)[source]

Render the contents of oeb_book (which is an instance of calibre.ebooks.oeb.OEBBook to the file specified by output.

Parameters:

- output – Either a file like object or a string. If it is a string it is the path to a directory that may or may not exist. The output plugin should write its output into that directory. If it is a file like object, the output plugin should write its output into the file.

- input_plugin – The input plugin that was used at the beginning of the conversion pipeline.

- opts – Conversion options. Guaranteed to have attributes corresponding to the OptionRecommendations of this plugin.

- log – The logger. Print debug/info messages etc. using this.

specialize_css_for_output(log, opts, item, stylizer)[source]

Can be used to make changes to the css during the CSS flattening process.

Parameters:

- item – The item (HTML file) being processed

- stylizer – A Stylizer object containing the flattened styles for item. You can get the style for any element by stylizer.style(element).

gui_configuration_widget(parent, get_option_by_name, get_option_help, db, book_id=None)[source]

Called to create the widget used for configuring this plugin in the calibre GUI. The widget must be an instance of the PluginWidget class.
See the builtin output plugins for examples.

Device Drivers

The base class for all device drivers is DevicePlugin. However, if your device exposes itself as a USBMS drive to the operating system, you should use the USBMS class instead as it implements all the logic needed to support these kinds of devices.

class calibre.devices.interface.DevicePlugin(plugin_path)[source]

Bases: calibre.customize.Plugin

Defines the interface that should be implemented by backends that communicate with an ebook reader.

FORMATS = ['lrf', 'rtf', 'pdf', 'txt']

Ordered list of supported formats

VENDOR_ID = 0

VENDOR_ID can be either an integer, a list of integers or a dictionary
If it is a dictionary, it must be a dictionary of dictionaries, of the form:

{
integer_vendor_id : { product_id : [list of BCDs], ... },
...
}

PRODUCT_ID = 0

An integer or a list of integers

BCD = None

BCD can be either None to not distinguish between devices based on BCD, or it can be a list of the BCD numbers of all devices supported by this driver.

THUMBNAIL_HEIGHT = 68

Height for thumbnails on the device

WANTS_UPDATED_THUMBNAILS = False

Width for thumbnails on the device. Setting this will force thumbnails to this size, not preserving aspect ratio. If it is not set, then the aspect ratio will be preserved and the thumbnail will be no higher than THUMBNAIL_HEIGHT
Set this to True if the device supports updating cover thumbnails during sync_booklists. Setting it to true will ask device.py to refresh the cover thumbnails during book matching

CAN_SET_METADATA = ['title', 'authors', 'collections']

Whether the metadata on books can be set via the GUI.

CAN_DO_DEVICE_DB_PLUGBOARD = False

Whether the device can handle device_db metadata plugboards

path_sep = '/'

Path separator for paths to books on device

icon = '/home/kovid/work/calibre/resources/images/reader.png'

Icon for this device

UserAnnotation

alias of Annotation

OPEN_FEEDBACK_MESSAGE = None

GUI displays this as a message if not None. Useful if opening can take a long time

VIRTUAL_BOOK_EXTENSIONS = frozenset([])

Set of extensions that are “virtual books” on the device and therefore cannot be viewed/saved/added to library
For example: frozenset(['kobo'])

NUKE_COMMENTS = None

Whether to nuke comments in the copy of the book sent to the device. If not None this should be short string that the comments will be replaced by.

MANAGES_DEVICE_PRESENCE = False

If True indicates that this driver completely manages device detection, ejecting and so forth. If you set this to True, you must implement the detect_managed_devices and debug_managed_device_detection methods.
A driver with this set to true is responsible for detection of devices, managing a blacklist of devices, a list of ejected devices and so forth.
calibre will periodically call the detect_managed_devices() method and is it returns a detected device, calibre will call open(). open() will be called every time a device is returned even is previous calls to open()
failed, therefore the driver must maintain its own blacklist of failed devices. Similarly, when ejecting, calibre will call eject() and then assuming the next call to detect_managed_devices() returns None, it will call post_yank_cleanup().

SLOW_DRIVEINFO = False

If set the True, calibre will call the get_driveinfo() method after the books lists have been loaded to get the driveinfo.

ASK_TO_ALLOW_CONNECT = False

If set to True, calibre will ask the user if they want to manage the device with calibre, the first time it is detected. If you set this to
True you must implement get_device_uid() and ignore_connected_device() and get_user_blacklisted_devices() and set_user_blacklisted_devices()

is_usb_connected(devices_on_system, debug=False, only_presence=False)[source]

Return True, device_info if a device handled by this plugin is currently connected.

Parameters: | devices_on_system – List of devices currently connected

detect_managed_devices(devices_on_system, force_refresh=False)[source]

Called only if MANAGES_DEVICE_PRESENCE is True.

Scan for devices that this driver can handle. Should return a device object if a device is found. This object will be passed to the open()
method as the connected_device. If no device is found, return None. The returned object can be anything, calibre does not use it, it is only passed to open().

This method is called periodically by the GUI, so make sure it is not too resource intensive. Use a cache to avoid repeatedly scanning the system.

Parameters:

- devices_on_system – Set of USB devices found on the system.

- force_refresh – If True and the driver uses a cache to prevent repeated scanning, the cache must be flushed.

debug_managed_device_detection(devices_on_system, output)[source]

Called only if MANAGES_DEVICE_PRESENCE is True.

Should write information about the devices detected on the system to output, which is a file like object.

Should return True if a device was detected and successfully opened, otherwise False.

reset(key='-1', log_packets=False, report_progress=None, detected_device=None)[source]

Parameters:

- key – The key to unlock the device

- log_packets – If true the packet stream to/from the device is logged

- report_progress – Function that is called with a % progress
(number between 0 and 100) for various tasks
If it is called with -1 that means that the task does not have any progress information

- detected_device – Device information from the device scanner

can_handle_windows(device_id, debug=False)[source]

Optional method to perform further checks on a device to see if this driver is capable of handling it. If it is not it should return False. This method is only called after the vendor, product ids and the bcd have matched, so it can do some relatively time intensive checks. The default implementation returns True. This method is called only on windows. See also can_handle().

Parameters: | device_info – On windows a device ID string. On Unix a tuple of (vendor_id, product_id, bcd).

can_handle(device_info, debug=False)[source]

Unix version of can_handle_windows()

Parameters: | device_info – Is a tuple of (vid, pid, bcd, manufacturer, product, serial number)

open(connected_device, library_uuid)[source]

Perform any device specific initialization. Called after the device is detected but before any other functions that communicate with the device.
For example: For devices that present themselves as USB Mass storage devices, this method would be responsible for mounting the device or if the device has been automounted, for finding out where it has been mounted. The method calibre.devices.usbms.device.Device.open() has an implementation of this function that should serve as a good example for USB Mass storage devices.

This method can raise an OpenFeedback exception to display a message to the user.

Parameters:

- connected_device – The device that we are trying to open. It is a tuple of (vendor id, product id, bcd, manufacturer name, product name, device serial number). However, some devices have no serial number and on windows only the first three fields are present, the rest are None.

- library_uuid – The UUID of the current calibre library. Can be
None if there is no library (for example when used from the command line).

eject()[source]

Un-mount / eject the device from the OS. This does not check if there are pending GUI jobs that need to communicate with the device.

NOTE: That this method may not be called on the same thread as the rest of the device methods.

post_yank_cleanup()[source]

Called if the user yanks the device without ejecting it first.

set_progress_reporter(report_progress)[source]

Set a function to report progress information.

Parameters:

report_progress – Function that is called with a % progress
(number between 0 and 100) for various tasks
If it is called with -1 that means that the task does not have any progress information

get_device_information(end_session=True)[source]

Ask device for device information. See L{DeviceInfoQuery}.

Returns:

(device name, device version, software version on device, mime type)
The tuple can optionally have a fifth element, which is a drive information dictionary. See usbms.driver for an example.

get_driveinfo()[source]

Return the driveinfo dictionary. Usually called from get_device_information(), but if loading the driveinfo is slow for this driver, then it should set SLOW_DRIVEINFO. In this case, this method will be called by calibre after the book lists have been loaded. Note that it is not called on the device thread, so the driver should cache the drive info in the books() method and this function should return the cached data.

card_prefix(end_session=True)[source]

Return a 2 element list of the prefix to paths on the cards.
If no card is present None is set for the card’s prefix.
E.G.
(‘/place’, ‘/place2’)
(None, ‘place2’)
(‘place’, None)
(None, None)

total_space(end_session=True)[source]

Get total space available on the mountpoints:

1. Main memory
2. Memory Card A
3. Memory Card B

Returns:

A 3 element list with total space in bytes of (1, 2, 3). If a particular device doesn’t have any of these locations it should return 0.

free_space(end_session=True)[source]

Get free space available on the mountpoints:

1. Main memory
2. Card A
3. Card B

Returns:

A 3 element list with free space in bytes of (1, 2, 3). If a particular device doesn’t have any of these locations it should return -1.

books(oncard=None, end_session=True)[source]

Return a list of ebooks on the device.

Parameters:

oncard – If ‘carda’ or ‘cardb’ return a list of ebooks on the specific storage card, otherwise return list of ebooks in main memory of device. If a card is specified and no books are on the card return empty list.

Returns: | A BookList.

upload_books(files, names, on_card=None, end_session=True, metadata=None)[source]

Upload a list of books to the device. If a file already exists on the device, it should be replaced.
This method should raise a FreeSpaceError if there is not enough free space on the device. The text of the FreeSpaceError must contain the word “card” if on_card is not None otherwise it must contain the word “memory”.

Parameters:

- files – A list of paths

- names – A list of file names that the books should have once uploaded to the device. len(names) == len(files)

- metadata – If not None, it is a list of Metadata objects.
The idea is to use the metadata to determine where on the device to put the book. len(metadata) == len(files). Apart from the regular cover (path to cover), there may also be a thumbnail attribute, which should be used in preference. The thumbnail attribute is of the form
(width, height, cover_data as jpeg).

Returns:

A list of 3-element tuples. The list is meant to be passed to add_books_to_metadata().

classmethod add_books_to_metadata(locations, metadata, booklists)[source]

Add locations to the booklists. This function must not communicate with the device.

Parameters:

- locations – Result of a call to L{upload_books}

- metadata – List of Metadata objects, same as for upload_books().

- booklists – A tuple containing the result of calls to
(books(oncard=None)(), books(oncard='carda')(),
:meth`books(oncard=’cardb’)`).

delete_books(paths, end_session=True)[source]

Delete books at paths on device.

classmethod remove_books_from_metadata(paths, booklists)[source]

Remove books from the metadata list. This function must not communicate with the device.

Parameters:

- paths – paths to books on the device.

- booklists – A tuple containing the result of calls to
(books(oncard=None)(), books(oncard='carda')(),
:meth`books(oncard=’cardb’)`).

sync_booklists(booklists, end_session=True)[source]

Update metadata on device.

Parameters:

booklists – A tuple containing the result of calls to
(books(oncard=None)(), books(oncard='carda')(),
:meth`books(oncard=’cardb’)`).

get_file(path, outfile, end_session=True)[source]

Read the file at path on the device and write it to outfile.

Parameters: | outfile – file object like sys.stdout or the result of an open() call.

classmethod config_widget()[source]

Should return a QWidget. The QWidget contains the settings for the device interface

classmethod save_settings(settings_widget)[source]

Should save settings to disk. Takes the widget created in config_widget() and saves all settings to disk.

classmethod settings()[source]

Should return an opts object. The opts object should have at least one attribute format_map which is an ordered list of formats for the device.

set_plugboards(plugboards, pb_func)[source]

provide the driver the current set of plugboards and a function to select a specific plugboard. This method is called immediately before add_books and sync_booklists.

pb_func is a callable with the following signature::

def pb_func(device_name, format, plugboards)

You give it the current device name (either the class name or
DEVICE_PLUGBOARD_NAME), the format you are interested in (a ‘real’
format or ‘device_db’), and the plugboards (you were given those by set_plugboards, the same place you got this method).

Returns: | None or a single plugboard instance.

set_driveinfo_name(location_code, name)[source]

Set the device name in the driveinfo file to ‘name’. This setting will persist until the file is re-created or the name is changed again.

Non-disk devices should implement this method based on the location codes returned by the get_device_information() method.

prepare_addable_books(paths)[source]

Given a list of paths, returns another list of paths. These paths point to addable versions of the books.

If there is an error preparing a book, then instead of a path, the position in the returned list for that book should be a three tuple:
(original_path, the exception instance, traceback)

startup()[source]

Called when calibre is is starting the device. Do any initialization required. Note that multiple instances of the class can be instantiated, and thus __init__ can be called multiple times, but only one instance will have this method called. This method is called on the device thread, not the GUI thread.

shutdown()[source]

Called when calibre is shutting down, either for good or in preparation to restart. Do any cleanup required. This method is called on the device thread, not the GUI thread.

get_device_uid()[source]

Must return a unique id for the currently connected device (this is called immediately after a successful call to open()). You must implement this method if you set ASK_TO_ALLOW_CONNECT = True

ignore_connected_device(uid)[source]

Should ignore the device identified by uid (the result of a call to get_device_uid()) in the future. You must implement this method if you set ASK_TO_ALLOW_CONNECT = True. Note that this function is called immediately after open(), so if open() caches some state, the driver should reset that state.

get_user_blacklisted_devices()[source]

Return map of device uid to friendly name for all devices that the user has asked to be ignored.

set_user_blacklisted_devices(devices)[source]

Set the list of device uids that should be ignored by this driver.

specialize_global_preferences(device_prefs)[source]

Implement this method if your device wants to override a particular preference. You must ensure that all call sites that want a preference that can be overridden use device_prefs[‘something’] instead of prefs[‘something’]. Your method should call device_prefs.set_overrides(pref=val, pref=val, ...).
Currently used for:
metadata management (prefs[‘manage_device_metadata’])

is_dynamically_controllable()[source]

Called by the device manager when starting plugins. If this method returns a string, then a) it supports the device manager’s dynamic control interface, and b) that name is to be used when talking to the plugin.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.

start_plugin()[source]

This method is called to start the plugin. The plugin should begin to accept device connections however it does that. If the plugin is already accepting connections, then do nothing.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.

stop_plugin()[source]

This method is called to stop the plugin. The plugin should no longer accept connections, and should cleanup behind itself. It is likely that this method should call shutdown. If the plugin is already not accepting connections, then do nothing.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.

get_option(opt_string, default=None)[source]

Return the value of the option indicated by opt_string. This method can be called when the plugin is not started. Return None if the option does not exist.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.

set_option(opt_string, opt_value)[source]

Set the value of the option indicated by opt_string. This method can be called when the plugin is not started.

This method can be called on the GUI thread. A driver that implements this method must be thread safe.

is_running()[source]

Return True if the plugin is started, otherwise false

This method can be called on the GUI thread. A driver that implements this method must be thread safe.

class calibre.devices.interface.BookList(oncard, prefix, settings)[source]

Bases: list

A list of books. Each Book object must have the fields

1. title
2. authors
3. size (file size of the book)
4. datetime (a UTC time tuple)
5. path (path on the device to the book)

6. thumbnail (can be None) thumbnail is either a str/bytes object with the image data or it should have an attribute image_path that stores an absolute (platform native) path to the image

7. tags (a list of strings, can be empty).

supports_collections()[source]

Return True if the device supports collections for this book list.

add_book(book, replace_metadata)[source]

Add the book to the booklist. Intent is to maintain any device-internal metadata. Return True if booklists must be sync’ed

remove_book(book)[source]

Remove a book from the booklist. Correct any device metadata at the same time

get_collections(collection_attributes)[source]

Return a dictionary of collections created from collection_attributes.
Each entry in the dictionary is of the form collection name:[list of books]

The list of books is sorted by book title, except for collections created from series, in which case series_index is used.

Parameters: | collection_attributes – A list of attributes of the Book object

USB Mass Storage based devices

The base class for such devices is calibre.devices.usbms.driver.USBMS. This class in turn inherits some of its functionality from its bases, documented below. A typical basic USBMS based driver looks like this:

from calibre.devices.usbms.driver import USBMS

class PDNOVEL(USBMS):
name = 'Pandigital Novel device interface'
gui_name = 'PD Novel'
description = _('Communicate with the Pandigital Novel')
author = 'Kovid Goyal'
supported_platforms = ['windows', 'linux', 'osx']
FORMATS = ['epub', 'pdf']

VENDOR_ID = [0x18d1]
PRODUCT_ID = [0xb004]
BCD = [0x224]

VENDOR_NAME = 'ANDROID'
WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = '__UMS_COMPOSITE'
THUMBNAIL_HEIGHT = 144

EBOOK_DIR_MAIN = 'eBooks'
SUPPORTS_SUB_DIRS = False

def upload_cover(self, path, filename, metadata):
coverdata = getattr(metadata, 'thumbnail', None)
if coverdata and coverdata[2]:
with open('%s.jpg' % os.path.join(path, filename), 'wb') as coverfile:
coverfile.write(coverdata[2])

class calibre.devices.usbms.device.Device(plugin_path)[source]

Bases: calibre.devices.usbms.deviceconfig.DeviceConfig, calibre.devices.interface.DevicePlugin

This class provides logic common to all drivers for devices that export themselves as USB Mass Storage devices. Provides implementations for mounting/ejecting of USBMS devices on all platforms.

WINDOWS_MAIN_MEM = None

String identifying the main memory of the device in the windows PnP id strings
This can be None, string, list of strings or compiled regex

WINDOWS_CARD_A_MEM = None

String identifying the first card of the device in the windows PnP id strings
This can be None, string, list of strings or compiled regex

WINDOWS_CARD_B_MEM = None

String identifying the second card of the device in the windows PnP id strings
This can be None, string, list of strings or compiled regex

OSX_MAIN_MEM_VOL_PAT = None

Used by the new driver detection to disambiguate main memory from storage cards. Should be a regular expression that matches the main memory mount point assigned by OS X

MAX_PATH_LEN = 250

The maximum length of paths created on the device

NEWS_IN_FOLDER = True

Put news in its own folder

windows_sort_drives(drives)[source]

Called to disambiguate main memory and storage card for devices that do not distinguish between them on the basis of WINDOWS_CARD_NAME.
For e.g.: The EB600

filename_callback(default, mi)[source]

Callback to allow drivers to change the default file name set by create_upload_path().

sanitize_path_components(components)[source]

Perform any device specific sanitization on the path components for files to be uploaded to the device

get_annotations(path_map)[source]

Resolve path_map to annotation_map of files found on the device

add_annotation_to_library(db, db_id, annotation)[source]

Add an annotation to the calibre library

class calibre.devices.usbms.cli.CLI[source]

class calibre.devices.usbms.driver.USBMS(plugin_path)[source]

Bases: calibre.devices.usbms.cli.CLI, calibre.devices.usbms.device.Device

The base class for all USBMS devices. Implements the logic for sending/getting/updating metadata/caching metadata/etc.

upload_cover(path, filename, metadata, filepath)[source]

Upload book cover to the device. Default implementation does nothing.

Parameters:

- path – The full path to the directory where the associated book is located.

- filename – The name of the book file without the extension.

- metadata – metadata belonging to the book. Use metadata.thumbnail for cover

- filepath – The full path to the ebook file

classmethod normalize_path(path)[source]

Return path with platform native path separators

User Interface Actions

If you are adding your own plugin in a zip file, you should subclass both InterfaceActionBase and InterfaceAction. The load_actual_plugin() method of you InterfaceActionBase subclass must return an instantiated object of your InterfaceBase subclass.

class calibre.gui2.actions.InterfaceAction(parent, site_customization)[source]

Bases: PyQt4.QtCore.QObject

A plugin representing an “action” that can be taken in the graphical user interface. All the items in the toolbar and context menus are implemented by these plugins.

Note that this class is the base class for these plugins, however, to integrate the plugin with calibre’s plugin system, you have to make a wrapper class that references the actual plugin. See the calibre.customize.builtins module for examples.

If two InterfaceAction objects have the same name, the one with higher priority takes precedence.

Sub-classes should implement the genesis(), library_changed(), location_selected() shutting_down() and initialization_complete() methods.

Once initialized, this plugin has access to the main calibre GUI via the gui member. You can access other plugins by name, for example:

self.gui.iactions['Save To Disk']

To access the actual plugin, use the interface_action_base_plugin attribute, this attribute only becomes available after the plugin has been initialized. Useful if you want to use methods from the plugin class like do_user_config().

The QAction specified by action_spec is automatically create and made available as self.qaction.

name = 'Implement me'

The plugin name. If two plugins with the same name are present, the one with higher priority takes precedence.

priority = 1

The plugin priority. If two plugins with the same name are present, the one with higher priority takes precedence.

popup_type = 1

The menu popup type for when this plugin is added to a toolbar

auto_repeat = False

Whether this action should be auto repeated when its shortcut key is held down.

action_spec = ('text', 'icon', None, None)

Of the form: (text, icon_path, tooltip, keyboard shortcut)
icon, tooltip and keyboard shortcut can be None shortcut must be a string, None or tuple of shortcuts.
If None, a keyboard shortcut corresponding to the action is not registered. If you pass an empty tuple, then the shortcut is registered with no default key binding.

action_add_menu = False

If True, a menu is automatically created and added to self.qaction

action_menu_clone_qaction = False

If True, a clone of self.qaction is added to the menu of self.qaction
If you want the text of this action to be different from that of self.qaction, set this variable to the new text

dont_add_to = frozenset([])

Set of locations to which this action must not be added.
See all_locations for a list of possible locations

dont_remove_from = frozenset([])

Set of locations from which this action must not be removed.
See all_locations for a list of possible locations

action_type = 'global'

Type of action
‘current’ means acts on the current view
‘global’ means an action that does not act on the current view, but rather on calibre as a whole

create_menu_action(menu, unique_name, text, icon=None, shortcut=None, description=None, triggered=None, shortcut_name=None)[source]

Convenience method to easily add actions to a QMenu.
Returns the created QAction, This action has one extra attribute calibre_shortcut_unique_name which if not None refers to the unique name under which this action is registered with the keyboard manager.

Parameters:

- menu – The QMenu the newly created action will be added to

- unique_name – A unique name for this action, this must be globally unique, so make it as descriptive as possible. If in doubt add a uuid to it.

- text – The text of the action.

- icon – Either a QIcon or a file name. The file name is passed to the I() builtin, so you do not need to pass the full path to the images directory.

- shortcut – A string, a list of strings, None or False. If False, no keyboard shortcut is registered for this action. If None, a keyboard shortcut with no default keybinding is registered. String and list of strings register a shortcut with default keybinding as specified.

- description – A description for this action. Used to set tooltips.

- triggered – A callable which is connected to the triggered signal of the created action.

- shortcut_name – The test displayed to the user when customizing the keyboard shortcuts for this action. By default it is set to the value of text.

load_resources(names)[source]

If this plugin comes in a ZIP file (user added plugin), this method will allow you to load resources from the ZIP file.

For example to load an image:

pixmap = QPixmap()
pixmap.loadFromData(self.load_resources(['images/icon.png']).itervalues().next())
icon = QIcon(pixmap)

Parameters: | names – List of paths to resources in the zip file using / as separator

Returns:

A dictionary of the form {name : file_contents}. Any names that were not found in the zip file will not be present in the dictionary.

genesis()[source]

Setup this plugin. Only called once during initialization. self.gui is available. The action specified by action_spec is available as self.qaction.

location_selected(loc)[source]

Called whenever the book list being displayed in calibre changes.
Currently values for loc are: library, main, card and cardb.

This method should enable/disable this action and its sub actions as appropriate for the location.

library_changed(db)[source]

Called whenever the current library is changed.

Parameters: | db – The LibraryDatabase corresponding to the current library.

gui_layout_complete()[source]

Called once per action when the layout of the main GUI is completed. If your action needs to make changes to the layout, they should be done here, rather than in initialization_complete().

initialization_complete()[source]

Called once per action when the initialization of the main GUI is completed.

shutting_down()[source]

Called once per plugin when the main GUI is in the process of shutting down. Release any used resources, but try not to block the shutdown for long periods of time.

Returns: | False to halt the shutdown. You are responsible for telling the user why the shutdown was halted.

class calibre.customize.InterfaceActionBase(*args, **kwargs)[source]

Bases: calibre.customize.Plugin

load_actual_plugin(gui)[source]

This method must return the actual interface action plugin object.

Preferences Plugins

class calibre.customize.PreferencesPlugin(plugin_path)[source]

Bases: calibre.customize.Plugin

A plugin representing a widget displayed in the Preferences dialog.

This plugin has only one important method create_widget(). The various fields of the plugin control how it is categorized in the UI.

config_widget = None

Import path to module that contains a class named ConfigWidget which implements the ConfigWidgetInterface. Used by create_widget().

category_order = 100

Where in the list of categories the category of this plugin should be.

name_order = 100

Where in the list of names in a category, the gui_name of this plugin should be

category = None

The category this plugin should be in

gui_category = None

The category name displayed to the user for this plugin

gui_name = None

The name displayed to the user for this plugin

icon = None

The icon for this plugin, should be an absolute path

description = None

The description used for tooltips and the like

create_widget(parent=None)[source]

Create and return the actual Qt widget used for setting this group of preferences. The widget must implement the calibre.gui2.preferences.ConfigWidgetInterface.

The default implementation uses config_widget to instantiate the widget.

class calibre.gui2.preferences.ConfigWidgetInterface[source]

This class defines the interface that all widgets displayed in the
Preferences dialog must implement. See ConfigWidgetBase for a base class that implements this interface and defines various convenience methods as well.

changed_signal = None

This signal must be emitted whenever the user changes a value in this widget

supports_restoring_to_defaults = True

Set to True iff the restore_to_defaults() method is implemented.

restore_defaults_desc = u'Restore settings to default values. You have to click Apply to actually save the default settings.'

The tooltip for the Restore to defaults button

restart_critical = False

If True the Preferences dialog will not allow the user to set any more preferences. Only has effect if commit() returns True.

genesis(gui)[source]

Called once before the widget is displayed, should perform any necessary setup.

Parameters: | gui – The main calibre graphical user interface

initialize()[source]

Should set all config values to their initial values (the values stored in the config files).

restore_defaults()[source]

Should set all config values to their defaults.

commit()[source]

Save any changed settings. Return True if the changes require a restart, False otherwise. Raise an AbortCommit exception to indicate that an error occurred. You are responsible for giving the user feedback about what the error is and how to correct it.

refresh_gui(gui)[source]

Called once after this widget is committed. Responsible for causing the gui to reread any changed settings. Note that by default the GUI re-initializes various elements anyway, so most widgets won’t need to use this method.

class calibre.gui2.preferences.ConfigWidgetBase(parent=None)[source]

Base class that contains code to easily add standard config widgets like checkboxes, combo boxes, text fields and so on. See the register() method.

This class automatically handles change notification, resetting to default, translation between gui objects and config objects, etc. for registered settings.

If your config widget inherits from this class but includes setting that are not registered, you should override the ConfigWidgetInterface methods and call the base class methods inside the overrides.

register(name, config_obj, gui_name=None, choices=None, restart_required=False, empty_string_is_None=True, setting=<class 'calibre.gui2.preferences.Setting'>)[source]

Register a setting.

Parameters:

- name – The setting name

- config – The config object that reads/writes the setting

- gui_name – The name of the GUI object that presents an interface to change the setting. By default it is assumed to be 'opt_' + name.

- choices – If this setting is a multiple choice (combobox) based setting, the list of choices. The list is a list of two element tuples of the form: [(gui name, value), ...]

- setting – The class responsible for managing this setting. The default class handles almost all cases, so this param is rarely used.

© Copyright Kovid Goyal.
Last updated on Dec 29, 2012.

Table of Contents

Cover
calibre User Manual

The Graphical User Interface

- The Graphical User Interface

- Actions

- Actions
- Add books
- Edit metadata
- Convert books
- View
- Send to device
- Fetch news
- Library
- Device
- Save to disk
- Connect/Share
- Remove books

- Preferences
- Catalogs
- Search & Sort
- The Search Interface

- Saving searches

- Saving searches

- Guessing metadata from file names

- Book Details
- Tag Browser
- Jobs
- Keyboard Shortcuts

Adding your favorite news website

- Adding your favorite news website

- Completely automatic fetching

- Completely automatic fetching
- portfolio.com
- bbc.co.uk

- Customizing the fetch process

- Customizing the fetch process
- Using the print version of bbc.co.uk
- Replacing article styles
- Slicing and dicing
- Real life example

- Tips for developing new recipes

- Further reading

- API documentation

- API documentation

- API Documentation for recipes

The Ebook Viewer

- The Ebook Viewer

- Starting the viewer

- Navigating around an ebook

- Navigating around an ebook
- Bookmarks
- Table of Contents
- Navigating by location

- Customizing the look and feel of your reading experience

- Dictionary lookup

- Copying text and images

Ebook Conversion

- Ebook Conversion

- Introduction

- Look & Feel

- Look & Feel
- Font size rescaling
- Paragraph spacing
- Extra CSS
- Miscellaneous

- Page Setup
- Heuristic Processing
- Search & Replace

- Structure Detection

- Structure Detection
- Chapters and page breaks
- Miscellaneous

- Table of Contents

- Using images as chapter titles when converting HTML input documents

- How options are set/saved for Conversion

- Format specific tips

- Format specific tips
- Convert Microsoft Word documents
- Convert TXT documents
- Convert PDF documents
- Comic Book Collections
- EPUB advanced formatting demo
- Convert ODT documents

Editing Ebook Metadata

- Editing Ebook Metadata

- Editing the metadata of one book at a time

- Editing the metadata of one book at a time
- Downloading metadata
- Managing book formats
- All about covers

- Editing the metadata of many books at a time

- Editing the metadata of many books at a time

- Search and replace

- Bulk downloading of metadata

Frequently Asked Questions

Frequently Asked Questions

Ebook Format Conversion

- Ebook Format Conversion

- What formats does calibre support conversion to/from?

- What are the best source formats to convert?

- I converted a PDF file, but the result has various problems?

- How do I convert my file containing non-English characters, or smart quotes?

- What's the deal with Table of Contents in MOBI files?

- The covers for my MOBI files have stopped showing up in Kindle for PC/Kindle for Android/iPad etc.

- How do I convert a collection of HTML files in a specific order?

- The EPUB I produced with calibre is not valid?

- How do I use some of the advanced features of the conversion tools?

Device Integration

- Device Integration

- What devices does calibre support?

- How can I help get my device supported in calibre?

- My device is not being detected by calibre?

- My device is non-standard or unusual. What can I do to connect to it?

- How does calibre manage collections on my SONY reader?

- Can I use both calibre and the SONY software to manage my reader?

- How do I use calibre with my iPad/iPhone/iTouch?

- How do I use calibre with my iPad/iPhone/iTouch?

- Over the air

- With the USB cable + iTunes

- How do I use calibre with my Android phone/tablet or Kindle Fire HD?

- How do I use calibre with my Android phone/tablet or Kindle Fire HD?

- Using a USB cable

- Over the air

- Can I access my calibre books using the web browser in my Kindle or other reading device?

- I get the error message “Failed to start content server: Port 8080 not free on '0.0.0.0'”?

- I cannot send emails using calibre?

- Why is my device not detected in linux?

- My device is getting mounted read-only in linux, so calibre cannot connect to it?

- Why does calibre not support collections on the Kindle or shelves on the Nook?

- I am getting an error when I try to use calibre with my Kobo Touch?

Library Management

- Library Management

- What formats does calibre read metadata from?

- Where are the book files stored?

- How does calibre manage author names and sorting?

- Why doesn't calibre let me store books in my own directory structure?

- Why doesn't calibre have a column for foo?

- Can I have a column showing the formats or the ISBN?

- How do I move my calibre library from one computer to another?

- The list of books in calibre is blank!

- I am getting errors with my calibre library on a networked drive/NAS?

Content From The Web

- Content From The Web

- I obtained a recipe for a news site as a .py file from somewhere, how do I use it?

- I want calibre to download news from my favorite news website.

- Can I use web2disk to download an arbitrary website?

Miscellaneous

- Miscellaneous

- Why the name calibre?

- Why does calibre show only some of my fonts on OS X?

- calibre is not starting on Windows?

- calibre freezes/crashes occasionally?

- calibre is not starting on OS X?

- I downloaded the installer, but it is not working?

- My antivirus program claims calibre is a virus/trojan?

- How do I backup calibre?

- How do I use purchased EPUB books with calibre (or what do I do with .acsm files)?

- I am getting a “Permission Denied” error?

- Can I have the comment metadata show up on my reader?

- How do I get calibre to use my HTTP proxy?

- I want some feature added to calibre. What can I do?

- Why doesn't calibre have an automatic update?

- How is calibre licensed?

- How do I run calibre from my USB stick?

- How do I run parts of calibre like news download and the content server on my own linux server?

Tutorials

Tutorials

Adding your favorite news website

- Adding your favorite news website

- Completely automatic fetching

- Completely automatic fetching
- portfolio.com
- bbc.co.uk

- Customizing the fetch process

- Customizing the fetch process
- Using the print version of bbc.co.uk
- Replacing article styles
- Slicing and dicing
- Real life example

- Tips for developing new recipes

- Further reading

- API documentation

- API documentation

- API Documentation for recipes

Managing subgroups of books, for example “genre”

- Managing subgroups of books, for example “genre”

- Setup
- Searching
- Restrictions
- Useful Template Functions

XPath Tutorial

- XPath Tutorial
- Selecting by tagname
- Selecting by attributes
- Selecting by tag content
- Sample ebook
- XPath built-in functions

The calibre template language

- The calibre template language
- Advanced formatting
- Advanced features
- Using templates in custom columns

- Using functions in templates - single-function mode

- Using functions in templates - template program mode

- Function classification

- Function classification

- Reference for all built-in template language functions

- Using general program mode
- User-defined Template Functions
- Special notes for save/send templates
- Templates and Plugboards
- Helpful Tips

All about using regular expressions in calibre

- All about using regular expressions in calibre

- First, a word of warning and a word of courage

- Where in calibre can you use regular expressions?

- What on earth is a regular expression?

- Care to explain?

- That doesn't sound too bad. What's next?

- Hey, neat! This is starting to make sense!

- Well, these special characters are very neat and all, but what if I wanted to match a dot or a question mark?

- So, what are the most useful sets?

- But if I had a few varying strings I wanted to match, things get complicated?

- You missed...

- In the beginning, you said there was a way to make a regular expression case insensitive?

- I think I'm beginning to understand these regular expressions now... how do I use them in calibre?

- I think I'm beginning to understand these regular expressions now... how do I use them in calibre?

- Conversions
- Adding books
- Bulk editing metadata

- Credits

Integrating the calibre content server into other servers

- Integrating the calibre content server into other servers

- Using a reverse proxy

- In process

Writing your own plugins to extend calibre's functionality

- Writing your own plugins to extend calibre's functionality

- Anatomy of a calibre plugin

- A User Interface plugin

- A User Interface plugin
- __init__.py
- ui.py
- main.py
- Getting resources from the plugin zip file
- Enabling user configuration of your plugin

- The plugin API
- Debugging plugins
- More plugin examples
- Sharing your plugins with others

Typesetting Math in ebooks

- Typesetting Math in ebooks
- A simple HTML file with mathematics
- More information

Creating AZW3 • EPUB • MOBI Catalogs

- Creating AZW3 • EPUB • MOBI Catalogs
- Selecting books to catalog
- Included sections
- Prefixes
- Excluded books
- Excluded genres
- Other options
- Custom catalog covers
- Additional help resources

Customizing calibre

- Customizing calibre
- Environment variables
- Tweaks
- Overriding icons, templates, et cetera
- Customizing calibre with plugins

Command Line Interface

Command Line Interface

Documented Commands

Documented Commands
calibre

- calibre

- [options]

calibre-customize

- calibre-customize

- [options]

calibre-debug

- calibre-debug

- [options]

calibre-server

- calibre-server

- [options]

calibre-smtp

- calibre-smtp

- [options]

calibredb

- calibredb
- Global options
- list
- add
- remove
- add_format
- remove_format
- show_metadata
- set_metadata
- export
- catalog
- saved_searches
- add_custom_column
- custom_columns
- remove_custom_column
- set_custom
- restore_database
- check_library
- list_categories
- backup_metadata

ebook-convert

- ebook-convert
- Look And Feel
- Heuristic Processing
- Search And Replace
- Structure Detection
- Table Of Contents
- Metadata
- Debug
- AZW4 Input Options
- CHM Input Options
- Comic Input Options
- DJVU Input Options
- EPUB Input Options
- FB2 Input Options
- HTLZ Input Options
- HTML Input Options
- LIT Input Options
- LRF Input Options
- MOBI Input Options
- ODT Input Options
- PDB Input Options
- PDF Input Options
- PML Input Options
- RB Input Options
- RTF Input Options
- Recipe Input Options
- SNB Input Options
- TCR Input Options
- TXT Input Options
- AZW3 Output Options
- EPUB Output Options
- FB2 Output Options
- HTML Output Options
- HTMLZ Output Options
- LIT Output Options
- LRF Output Options
- MOBI Output Options
- OEB Output Options
- PDB Output Options
- PDF Output Options
- PML Output Options
- RB Output Options
- RTF Output Options
- SNB Output Options
- TCR Output Options
- TXT Output Options
- TXTZ Output Options

ebook-meta

- ebook-meta

- [options]

ebook-viewer

- ebook-viewer

- [options]

epub-fix

- epub-fix

- [options]

fetch-ebook-metadata

- fetch-ebook-metadata

- [options]

lrf2lrs

- lrf2lrs

- [options]

lrfviewer

- lrfviewer

- [options]

lrs2lrf

- lrs2lrf

- [options]

web2disk

- web2disk

- [options]

Undocumented Commands

Setting up a calibre development environment

- Setting up a calibre development environment

- Design philosophy

- Design philosophy

- Code layout

- Getting the code

- Getting the code

- Submitting your changes to be included

- Windows development environment
- OS X development environment
- Linux development environment

- Having separate “normal” and “development” calibre installs on the same computer

- Debugging tips

- Debugging tips
- Using an interactive python interpreter
- Using print statements
- Using the debugger in PyDev

- Executing arbitrary scripts in the calibre python environment

- Using calibre in your projects

- Using calibre in your projects
- Binary install of calibre
- Source install on Linux

Glossary