Skip to content

Task Dependencies

This document shows the dependency graph of the various build tasks in mkmapdiary. The tasks are organized in a directed acyclic graph (DAG) where arrows indicate dependencies.

uml diagram

Task Description

Data Conversion Tasks

  • qstarz2gpx: Converts Qstarz GPS device files to GPX format
  • convert_raw: Converts CR2 raw image files to JPEG
  • convert_image: Processes and converts image files
  • convert_audio: Converts audio files to MP3 format
  • text2markdown: Converts plain text files to Markdown
  • markdown2markdown: Processes existing Markdown files

GPX Processing Tasks

  • pre_gpx: Preparation step for GPX processing, ensures dependencies are met
  • geo_correlation: Correlates geographic data with timestamps
  • gpx2gpx: Main GPX processing task that generates daily GPX files
  • end_gpx: Marks the end of GPX processing pipeline
  • get_gpx_deps: Calculates dependencies on GPX files for downstream tasks

Postprocessing Tasks

  • post_processing_single: Runs single-asset postprocessors on each asset individually, in parallel and in arbitrary order (currently ImageHasher and EntropyCalculator)
  • post_processing: Runs multi-asset postprocessors that require context from multiple assets, sequentially and in list order (currently ImageQualityAssessment, DuplicateDetector, AutoRotator and JournalSummarizer). ImageSummarizer and ImageEmbedder exist but are commented out of the list
  • end_postprocessing: Marks the end of postprocessing pipeline

Content Generation Tasks

  • transcribe_audio: Transcribes audio files to text with a local Whisper model (gated on features.transcription.enabled)
  • all_assets: Pseudo-node representing the collection of all generated assets (images, audio, text files)
  • build_day_page: Generates daily summary pages
  • build_gallery: Creates photo gallery pages for each day
  • build_journal: Builds journal entries from various content sources
  • build_tags: Generates tag-based content organization

Site Building Tasks

  • generate_mkdocs_config: Creates the MkDocs configuration file
  • build_index_page: Generates the main index page
  • build_credits_page: Generates the credits page, listing travellers (credits.travellers) and creators (credits.creators merged with the creators found in calibration.yaml files, in image metadata and on GPX tracks) alongside dependency and frontend-library credits computed at build time, plus the AI disclosure; build_site depends on it
  • compile_css: Compiles SASS to CSS
  • copy_simple_asset: Copies static assets (JS, CSS, images)
  • pre_build_site: Ensures all site directories exist before building
  • build_site: Final step that builds the complete MkDocs site

Dependency Types

The flowchart shows three types of dependencies: - Solid arrows (task_dep): Hard dependencies that must complete before the task can start - Dotted arrows (calc_dep): Calculated dependencies that are computed at runtime - Dotted arrows (file_dep): File dependencies where tasks depend on files generated by other tasks

The @create_after decorator is used for tasks that need to be created after other tasks complete, which creates delayed task creation dependencies shown as ==> arrows in the diagram.

Note: Directory creation dependencies (create_directory) are omitted from the diagram for readability, but most tasks have implicit dependencies on directory creation to ensure output directories exist before file generation.

Pipeline Flow

The complete build pipeline follows these main phases:

  1. Data Conversion: Convert various input file formats to standardized formats
  2. GPX Processing: Process GPS data and correlate it with timestamps
  3. Postprocessing: Apply AI-based analysis and enhancement to assets
  4. Content Generation: Build pages and organize content by date, gallery, journal, and tags
  5. Site Building: Generate the final MkDocs site with all content and assets

Many content generation tasks (build_gallery, build_journal, build_tags, build_index_page) use file_dep=self.db.get_all_assets(), meaning they depend on ALL assets generated by the conversion tasks (convert_raw, convert_image, convert_audio, text2markdown, markdown2markdown, transcribe_audio). The postprocessing pipeline runs between GPX processing and content generation, adding AI-enhanced metadata and analysis to the assets.