Compare commits

...
21 Commits
Author SHA1 Message Date
Antoine AflaloandGitHub 650623b340 Merge pull request #214 from Belphemur/copilot/avoid-runnning-big-file-test
chore: remove Git LFS large-file test and fixture
2026-07-11 08:14:06 -04:00
4b961f18e3 chore: stop tracking build artifact binaries
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-11 12:12:21 +00:00
2dfc39505c chore: remove Git LFS large-file test and fixture
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-11 12:06:33 +00:00
e596a026dd chore: remove Git LFS large-file test and fixture
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-11 12:05:02 +00:00
renovate[bot]GitHubrenovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
9423cde9f7 fix(deps): update module golang.org/x/image to v0.44.0 (#213)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-08 20:12:54 +00:00
Antoine AflaloandGitHub facb2f192b Merge pull request #211 from Belphemur/copilot/full-refactor-memory-usage
Full project refactoring for memory efficiency
2026-07-04 18:54:02 -04:00
5e5cac4883 fix: address copilot and coderabbit review feedback
- Use named returns for IsAlreadyConverted and WriteChapterToCBZ so
  errs.Capture defers propagate close errors to the caller.
- Add context.Context to IsAlreadyConverted and ExtractChapter for
  cancellation/timeout support; thread it from Optimize.
- Filter non-image files (__MACOSX, Thumbs.db, .DS_Store, etc.) and
  only extract supported image extensions in ExtractChapter.
- Validate TempDir is non-empty in ConvertChapter before creating output dir.
- Separate fatal conversion errors from PageIgnoredError in the WebP
  converter result aggregation so real failures aren't masked.
- Fix error message "failed to load chapter" → "failed to extract chapter".
- Use %w instead of %v in fmt.Errorf for proper error wrapping.
- Fix LoadChapter doc comment to match its actual behavior.
- Extract shared parseConvertedComment/parseConvertedCommentTime helpers
  to avoid duplicated zip-comment parsing logic.
- Convert tests to use testify assertions (assert/require) per guidelines.

Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-04 22:39:19 +00:00
5ae05e5f0c fix: address review feedback from copilot and coderabbit
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-04 22:34:28 +00:00
164dc577b9 refactor!: disk-first zero-copy architecture with file-to-file cwebp conversion
BREAKING CHANGE: Complete refactoring of the conversion pipeline.

- Replace in-memory Page/PageContainer with disk-based PageFile struct
- Extract archives to temp directory instead of loading into memory
- Use cwebp InputFile/OutputFile for zero-copy file-to-file conversion
- Use cwebp -crop for splitting (no Go-side image decode needed)
- Check if already converted before extracting (fast pre-check)
- Remove oliamb/cutter dependency (replaced by cwebp -crop)
- Remove page_container.go (no longer needed)
- Add comprehensive tests with improved coverage

Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-04 22:18:22 +00:00
Antoine AflaloandGitHub e87e73eb28 Merge pull request #210 from Belphemur/copilot/optimize-file-processing
fix(watch): don't backfill existing archives at watch startup
2026-07-03 11:53:11 -04:00
d7782fdb25 refactor(watch): extract testable maybeBackfillExistingArchives helper
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-03 15:45:35 +00:00
1d981be21b feat(watch): add opt-in --backfill flag for existing archives
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-03 15:44:08 +00:00
0b3c3fa90c fix(watch): don't backfill existing archives at watch startup
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-03 15:38:44 +00:00
Antoine AflaloandGitHub 52c93acd35 Merge pull request #209 from Belphemur/copilot/fix-docker-permission-denied-issue
fix(docker): create and chown /config home directory for non-root user
2026-07-03 11:17:27 -04:00
ab1e2ff05c fix(docker): create and chown /config home directory for non-root user
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-03 15:15:49 +00:00
Antoine AflaloandGitHub eca9671881 Merge pull request #208 from Belphemur/copilot/move-to-alpine-latest-docker-image
chore(docker): switch base image to alpine:latest
2026-07-03 10:57:52 -04:00
40922078d5 chore(docker): switch base image to alpine:latest
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-03 14:53:55 +00:00
Antoine AflaloandGitHub efb4dce1b7 Merge pull request #207 from Belphemur/copilot/fix-comments-in-pr-204
fix(watch): debounce fsnotify events, backfill archives, resilient recursive watch
2026-07-03 10:35:38 -04:00
30d0d632bf fix(watch): address review feedback in watcher queue and tests
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-03 14:26:40 +00:00
1e692ed19b fix(watch): remove mutable package var, fix debouncer race
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-03 13:55:58 +00:00
69c1a37577 fix(watch): debounce fsnotify events, backfill archives, resilient walk
Co-authored-by: Belphemur <197810+Belphemur@users.noreply.github.com>
2026-07-03 13:52:17 +00:00
33 changed files with 2242 additions and 1606 deletions
-1
View File
@@ -1 +0,0 @@
testdata/large/*.cbz filter=lfs diff=lfs merge=lfs -text
+9 -9
View File
@@ -95,15 +95,6 @@ go test -v ./pkg/converter/...
go test -v ./internal/utils/... go test -v ./internal/utils/...
``` ```
A large-file integration test (`TestOptimizeIntegration_LargeFile`) exercises the
optimize pipeline against a ~1GB CBZ fixture stored via Git LFS
(`testdata/large/large_chapter.cbz`, tracked in `.gitattributes`). It is skipped
by default; fetch the fixture with `git lfs pull` and opt in with:
```bash
CBZ_RUN_LARGE_FILE_TEST=1 go test -v ./internal/utils/... -run TestOptimizeIntegration_LargeFile
```
### Linting ### Linting
```bash ```bash
@@ -123,6 +114,15 @@ golangci-lint run
- **Error handling:** Always check errors explicitly; use structured error wrapping with `fmt.Errorf("context: %w", err)` - **Error handling:** Always check errors explicitly; use structured error wrapping with `fmt.Errorf("context: %w", err)`
- **Context usage:** Pass `context.Context` as first parameter for operations that may be cancelled - **Context usage:** Pass `context.Context` as first parameter for operations that may be cancelled
### Commit Messages
This project follows [Conventional Commits](https://www.conventionalcommits.org/). Prefix commit messages with a type (and optional scope), for example:
- `fix(watch): debounce fsnotify events before optimizing`
- `feat(converter): add avif support`
- `docs: clarify commit convention`
- `chore: update dependency`
### Logging ### Logging
Use **zerolog** for all logging: Use **zerolog** for all logging:
+4
View File
@@ -11,6 +11,10 @@
# Test binary, built with `go test -c` # Test binary, built with `go test -c`
*.test *.test
# Build artifacts
/cbzoptimizer
/encoder-setup
test/ test/
# Output of the go coverage tool, specifically when used with LiteIDE # Output of the go coverage tool, specifically when used with LiteIDE
+1
View File
@@ -14,6 +14,7 @@ This repository contains AI-agent oriented project context.
- Logging: zerolog - Logging: zerolog
- Error wrapping: `fmt.Errorf("context: %w", err)` - Error wrapping: `fmt.Errorf("context: %w", err)`
- Prefer small, focused changes. - Prefer small, focused changes.
- Commit messages: follow [Conventional Commits](https://www.conventionalcommits.org/) (e.g. `fix(watch): debounce fsnotify events`, `docs: clarify commit convention`).
## Areas to know ## Areas to know
+18 -21
View File
@@ -1,37 +1,34 @@
FROM debian:trixie-slim FROM alpine:latest
LABEL authors="Belphemur" LABEL authors="Belphemur"
ARG TARGETPLATFORM ARG TARGETPLATFORM
ARG APP_PATH=/usr/local/bin/CBZOptimizer ARG APP_PATH=/usr/local/bin/CBZOptimizer
ENV USER=abc ENV USER=abc
ENV CONFIG_FOLDER=/config ENV CONFIG_FOLDER=/config
ENV PUID=99 ENV PUID=99
ENV DEBIAN_FRONTEND=noninteractive # libwebp-tools (cwebp) is installed via apk below into /usr/bin; point go-webpbin
# at it directly so it doesn't try to download a glibc prebuilt binary.
ENV VENDOR_PATH=/usr/bin
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ RUN adduser \
--mount=type=cache,target=/var/lib/apt,sharing=locked \ -S \
apt-get update && apt-get install -y --no-install-recommends adduser && \ -D \
addgroup --system users && \ -h "${CONFIG_FOLDER}" \
adduser \ -u "${PUID}" \
--system \ -G users \
--home "${CONFIG_FOLDER}" \ -s /bin/bash \
--uid "${PUID}" \ "${USER}"
--ingroup users \
--disabled-password \
"${USER}" && \
apt-get purge -y --auto-remove adduser
COPY ${TARGETPLATFORM}/CBZOptimizer ${APP_PATH} COPY ${TARGETPLATFORM}/CBZOptimizer ${APP_PATH}
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ RUN apk add --no-cache \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && \
apt-get full-upgrade -y && \
apt-get install -y --no-install-recommends \
bash \ bash \
ca-certificates \ ca-certificates \
bash-completion && \ bash-completion \
libwebp-tools && \
chmod +x ${APP_PATH} && \ chmod +x ${APP_PATH} && \
${APP_PATH} completion bash > /etc/bash_completion.d/CBZOptimizer.bash ${APP_PATH} completion bash > /etc/bash_completion.d/CBZOptimizer.bash && \
mkdir -p "${CONFIG_FOLDER}" && \
chown -R "${PUID}":users "${CONFIG_FOLDER}"
USER ${USER} USER ${USER}
+11
View File
@@ -78,6 +78,16 @@ Watch a folder for new CBZ/CBR files and optimize them automatically:
cbzconverter watch [folder] --quality 85 --override --format webp --split cbzconverter watch [folder] --quality 85 --override --format webp --split
``` ```
Watch mode only reacts to filesystem events that occur *after* it starts; by default it does
not scan and optimize files that already exist in the folder when it starts. Run the
`optimize` command first if you need to process an existing library, then use `watch`
to keep it up to date going forward, or pass `--backfill` to have `watch` optimize the
existing files at startup before it begins watching for new changes. The only exception
is a directory that gets created/moved into the watched tree while watch mode is
running: since no per-file event is emitted for files already inside it, its existing
archives are always processed once when the directory is first detected, regardless of
`--backfill`.
Or with Docker: Or with Docker:
```sh ```sh
@@ -94,6 +104,7 @@ docker run -v /path/to/comics:/comics ghcr.io/belphemur/cbzoptimizer:latest watc
- Can be specified as: `--format webp`, `-f webp`, or `--format=webp` - Can be specified as: `--format webp`, `-f webp`, or `--format=webp`
- Case-insensitive: `webp`, `WEBP`, and `WebP` are all valid - Case-insensitive: `webp`, `WEBP`, and `WebP` are all valid
- `--timeout`, `-t`: Maximum time allowed for converting a single chapter (e.g., 30s, 5m, 1h). 0 means no timeout. Default is 0. - `--timeout`, `-t`: Maximum time allowed for converting a single chapter (e.g., 30s, 5m, 1h). 0 means no timeout. Default is 0.
- `--backfill`: *(`watch` only)* Optimize CBZ/CBR files that already exist in the watched folder at startup, before watching for new changes. Default is false.
- `--log`, `-l`: Set log level; can be 'panic', 'fatal', 'error', 'warn', 'info', 'debug', or 'trace'. Default is info. - `--log`, `-l`: Set log level; can be 'panic', 'fatal', 'error', 'warn', 'info', 'debug', or 'trace'. Default is info.
## Logging ## Logging
BIN
View File
Binary file not shown.
@@ -213,7 +213,7 @@ func TestFormatFlagWithSpace(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Failed to create temp directory: %v", err) t.Fatalf("Failed to create temp directory: %v", err)
} }
defer os.RemoveAll(tempDir) defer func() { _ = os.RemoveAll(tempDir) }()
cmd, cleanup := setupTestCommand(t) cmd, cleanup := setupTestCommand(t)
defer cleanup() defer cleanup()
@@ -242,7 +242,7 @@ func TestFormatFlagWithShortForm(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Failed to create temp directory: %v", err) t.Fatalf("Failed to create temp directory: %v", err)
} }
defer os.RemoveAll(tempDir) defer func() { _ = os.RemoveAll(tempDir) }()
cmd, cleanup := setupTestCommand(t) cmd, cleanup := setupTestCommand(t)
defer cleanup() defer cleanup()
@@ -271,7 +271,7 @@ func TestFormatFlagWithEquals(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Failed to create temp directory: %v", err) t.Fatalf("Failed to create temp directory: %v", err)
} }
defer os.RemoveAll(tempDir) defer func() { _ = os.RemoveAll(tempDir) }()
cmd, cleanup := setupTestCommand(t) cmd, cleanup := setupTestCommand(t)
defer cleanup() defer cleanup()
@@ -300,7 +300,7 @@ func TestFormatFlagDefaultValue(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Failed to create temp directory: %v", err) t.Fatalf("Failed to create temp directory: %v", err)
} }
defer os.RemoveAll(tempDir) defer func() { _ = os.RemoveAll(tempDir) }()
cmd, cleanup := setupTestCommand(t) cmd, cleanup := setupTestCommand(t)
defer cleanup() defer cleanup()
@@ -329,7 +329,7 @@ func TestFormatFlagCaseInsensitive(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Failed to create temp directory: %v", err) t.Fatalf("Failed to create temp directory: %v", err)
} }
defer os.RemoveAll(tempDir) defer func() { _ = os.RemoveAll(tempDir) }()
testCases := []string{"webp", "WEBP", "WebP", "WeBp"} testCases := []string{"webp", "WEBP", "WebP", "WeBp"}
+252 -15
View File
@@ -1,12 +1,15 @@
package commands package commands
import ( import (
"errors"
"fmt" "fmt"
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"runtime" "runtime"
"strings" "strings"
"sync"
"time"
utils2 "github.com/belphemur/CBZOptimizer/v2/internal/utils" utils2 "github.com/belphemur/CBZOptimizer/v2/internal/utils"
"github.com/belphemur/CBZOptimizer/v2/pkg/converter" "github.com/belphemur/CBZOptimizer/v2/pkg/converter"
@@ -17,6 +20,12 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
) )
// debounceDelay is the quiet period we wait, per path, after the last fsnotify
// event before triggering an optimization. This coalesces the bursts of
// Write/Create/Rename events fsnotify emits while a file is still being
// written (e.g. copied or downloaded into the watched folder).
const debounceDelay = 2 * time.Second
func init() { func init() {
if runtime.GOOS != "linux" { if runtime.GOOS != "linux" {
return return
@@ -32,6 +41,9 @@ func init() {
// Setup common flags (format, quality, override, split, timeout) with viper binding // Setup common flags (format, quality, override, split, timeout) with viper binding
setupCommonFlags(command, &converterType, 85, true, false, true) setupCommonFlags(command, &converterType, 85, true, false, true)
command.Flags().Bool("backfill", false, "Optimize CBZ/CBR files that already exist in the watched folder at startup, before watching for new changes")
_ = viper.BindPFlag("backfill", command.Flags().Lookup("backfill"))
AddCommand(command) AddCommand(command)
} }
func WatchCommand(_ *cobra.Command, args []string) error { func WatchCommand(_ *cobra.Command, args []string) error {
@@ -55,27 +67,55 @@ func WatchCommand(_ *cobra.Command, args []string) error {
timeout := viper.GetDuration("timeout") timeout := viper.GetDuration("timeout")
backfill := viper.GetBool("backfill")
converterType := constant.FindConversionFormat(viper.GetString("format")) converterType := constant.FindConversionFormat(viper.GetString("format"))
chapterConverter, err := converter.Get(converterType) chapterConverter, err := converter.Get(converterType)
if err != nil { if err != nil {
return fmt.Errorf("failed to get chapterConverter: %v", err) return fmt.Errorf("failed to get chapterConverter: %w", err)
} }
err = chapterConverter.PrepareConverter() err = chapterConverter.PrepareConverter()
if err != nil { if err != nil {
return fmt.Errorf("failed to prepare converter: %v", err) return fmt.Errorf("failed to prepare converter: %w", err)
} }
log.Info().Str("path", path).Bool("override", override).Uint8("quality", quality).Str("format", converterType.String()).Bool("split", split).Msg("Watching directory") log.Info().Str("path", path).Bool("override", override).Uint8("quality", quality).Str("format", converterType.String()).Bool("split", split).Bool("backfill", backfill).Msg("Watching directory")
watcher, err := fsnotify.NewWatcher() watcher, err := fsnotify.NewWatcher()
if err != nil { if err != nil {
return fmt.Errorf("failed to create file watcher: %w", err) return fmt.Errorf("failed to create file watcher: %w", err)
} }
defer watcher.Close() defer func() {
if closeErr := watcher.Close(); closeErr != nil {
log.Error().Err(closeErr).Msg("Failed to close file watcher")
}
}()
// Optimization jobs run in a small worker pool so the fsnotify event loop
// is never blocked waiting on a conversion.
queue := newOptimizeQueue(runtime.NumCPU(), &utils2.OptimizeOptions{
ChapterConverter: chapterConverter,
Quality: quality,
Override: override,
Split: split,
Timeout: timeout,
})
defer queue.Stop()
debouncer := newEventDebouncer(debounceDelay, queue.Enqueue)
defer debouncer.Stop()
// Note: existing archives already present under path when the watch
// starts are left untouched unless --backfill is set. Watch mode only
// reacts to filesystem events going forward by default; use the
// `optimize` command (or pass --backfill) to process a library's
// existing contents. Archives inside a directory that is created/moved
// into the watched tree *after* startup are always back-filled below,
// since only the directory itself generates an fsnotify event.
if err := addRecursiveWatch(watcher, path); err != nil { if err := addRecursiveWatch(watcher, path); err != nil {
return fmt.Errorf("failed to watch path %s: %w", path, err) return fmt.Errorf("failed to watch path %s: %w", path, err)
} }
maybeBackfillExistingArchives(backfill, path, debouncer.Trigger)
for { for {
select { select {
@@ -91,6 +131,11 @@ func WatchCommand(_ *cobra.Command, args []string) error {
if err := addRecursiveWatch(watcher, event.Name); err != nil { if err := addRecursiveWatch(watcher, event.Name); err != nil {
log.Error().Err(err).Str("path", event.Name).Msg("Failed to watch created directory") log.Error().Err(err).Str("path", event.Name).Msg("Failed to watch created directory")
} }
// The newly discovered directory may already contain
// archives (e.g. a folder moved/copied in); back-fill them
// since no further fsnotify event will target them.
backfillExistingArchives(event.Name, debouncer.Trigger)
continue
} }
} }
@@ -102,16 +147,7 @@ func WatchCommand(_ *cobra.Command, args []string) error {
continue continue
} }
if err := utils2.Optimize(&utils2.OptimizeOptions{ debouncer.Trigger(event.Name)
ChapterConverter: chapterConverter,
Path: event.Name,
Quality: quality,
Override: override,
Split: split,
Timeout: timeout,
}); err != nil {
log.Error().Err(err).Str("file", event.Name).Msg("Error processing file")
}
case err, ok := <-watcher.Errors: case err, ok := <-watcher.Errors:
if !ok { if !ok {
return nil return nil
@@ -121,21 +157,66 @@ func WatchCommand(_ *cobra.Command, args []string) error {
} }
} }
// addRecursiveWatch registers a watch on rootPath and every subdirectory
// beneath it. Directories that can't be read (e.g. permission errors) are
// logged and skipped instead of aborting the whole walk, so a single bad
// subtree doesn't prevent the rest of the folder from being watched.
func addRecursiveWatch(watcher *fsnotify.Watcher, rootPath string) error { func addRecursiveWatch(watcher *fsnotify.Watcher, rootPath string) error {
return filepath.WalkDir(rootPath, func(path string, entry fs.DirEntry, err error) error { return filepath.WalkDir(rootPath, func(path string, entry fs.DirEntry, err error) error {
if err != nil { if err != nil {
return err log.Warn().Err(err).Str("path", path).Msg("Skipping path while setting up watch")
return nil
} }
if !entry.IsDir() { if !entry.IsDir() {
return nil return nil
} }
if err := watcher.Add(path); err != nil { if err := watcher.Add(path); err != nil {
if errors.Is(err, fs.ErrPermission) {
log.Warn().Err(err).Str("path", path).Msg("Skipping unreadable directory while setting up watch")
return nil
}
return fmt.Errorf("failed to watch directory %s: %w", path, err) return fmt.Errorf("failed to watch directory %s: %w", path, err)
} }
return nil return nil
}) })
} }
// backfillExistingArchives walks rootPath for comic archives that already
// exist on disk and hands them to process. This covers the case where a
// directory (potentially already containing archives) is created/moved into
// the watched tree: only the directory itself generates an fsnotify event,
// so the files inside it would otherwise never be picked up.
func backfillExistingArchives(rootPath string, process func(path string)) {
err := filepath.WalkDir(rootPath, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
log.Warn().Err(err).Str("path", path).Msg("Skipping path while scanning for existing archives")
return nil
}
if entry.IsDir() {
return nil
}
if !isComicArchive(path) {
return nil
}
process(path)
return nil
})
if err != nil {
log.Error().Err(err).Str("path", rootPath).Msg("Failed to scan directory for existing archives")
}
}
// maybeBackfillExistingArchives runs backfillExistingArchives against
// rootPath only when enabled is true. It exists as its own function (rather
// than inlining the `if` check at the call site) so the gating decision used
// by WatchCommand can be exercised directly in tests.
func maybeBackfillExistingArchives(enabled bool, rootPath string, process func(path string)) {
if !enabled {
return
}
backfillExistingArchives(rootPath, process)
}
func shouldProcessWatchEvent(event fsnotify.Event) bool { func shouldProcessWatchEvent(event fsnotify.Event) bool {
return event.Has(fsnotify.Create) || event.Has(fsnotify.Write) || event.Has(fsnotify.Rename) return event.Has(fsnotify.Create) || event.Has(fsnotify.Write) || event.Has(fsnotify.Rename)
} }
@@ -144,3 +225,159 @@ func isComicArchive(path string) bool {
filename := strings.ToLower(path) filename := strings.ToLower(path)
return strings.HasSuffix(filename, ".cbz") || strings.HasSuffix(filename, ".cbr") return strings.HasSuffix(filename, ".cbz") || strings.HasSuffix(filename, ".cbr")
} }
// eventDebouncer coalesces bursts of fsnotify events targeting the same path
// into a single call to onQuiet, fired after the path has been quiet for
// delay. This prevents repeated conversions being triggered while a file is
// still being written, and avoids running Optimize against a Rename event
// that reports the *old* (already gone) file name once the quiet period lets
// us re-check the path.
type eventDebouncer struct {
delay time.Duration
onQuiet func(path string)
mu sync.Mutex
timers map[string]*debounceTimer
inFlight sync.WaitGroup
stopping bool
}
type debounceTimer struct {
timer *time.Timer
}
func newEventDebouncer(delay time.Duration, onQuiet func(path string)) *eventDebouncer {
return &eventDebouncer{
delay: delay,
onQuiet: onQuiet,
timers: make(map[string]*debounceTimer),
}
}
// Trigger (re)schedules onQuiet to run for path after the debounce delay,
// resetting any previously pending timer for the same path.
func (d *eventDebouncer) Trigger(path string) {
d.mu.Lock()
defer d.mu.Unlock()
if d.stopping {
return
}
if existing, exists := d.timers[path]; exists {
if existing.timer.Stop() {
d.inFlight.Done()
}
}
entry := &debounceTimer{}
d.inFlight.Add(1)
entry.timer = time.AfterFunc(d.delay, func() {
defer d.inFlight.Done()
d.mu.Lock()
// Only the still-current timer for this path is allowed to clear the
// entry and fire onQuiet. If Trigger raced with this callback and
// already installed a newer timer, this stale invocation must not
// delete that newer entry or fire early.
owns := d.timers[path] == entry && !d.stopping
if owns {
delete(d.timers, path)
}
d.mu.Unlock()
if owns {
d.onQuiet(path)
}
})
d.timers[path] = entry
}
// Stop cancels all pending timers.
func (d *eventDebouncer) Stop() {
d.mu.Lock()
d.stopping = true
for path, timer := range d.timers {
if timer.timer.Stop() {
d.inFlight.Done()
}
delete(d.timers, path)
}
d.mu.Unlock()
d.inFlight.Wait()
}
// optimizeQueue is a small worker pool that runs Optimize jobs off of the
// fsnotify event loop, so a slow conversion never blocks event draining.
type optimizeQueue struct {
jobs chan string
wg sync.WaitGroup
mu sync.RWMutex
stopped bool
options *utils2.OptimizeOptions
optimize func(options *utils2.OptimizeOptions) error
}
func newOptimizeQueue(workerCount int, options *utils2.OptimizeOptions) *optimizeQueue {
if workerCount < 1 {
workerCount = 1
}
q := &optimizeQueue{
jobs: make(chan string, 64),
options: options,
optimize: utils2.Optimize,
}
q.wg.Add(workerCount)
for i := 0; i < workerCount; i++ {
go q.worker()
}
return q
}
func (q *optimizeQueue) worker() {
defer q.wg.Done()
for path := range q.jobs {
q.process(path)
}
}
// process re-checks the path right before optimizing: fsnotify Rename events
// commonly report the *old* file name (already gone by the time we act on
// it), and Write can fire while a file is still being written elsewhere. In
// override mode, Optimize may overwrite/delete the source file, so skipping
// paths that no longer exist (or that are no longer regular files) avoids
// noisy failures.
func (q *optimizeQueue) process(path string) {
info, err := os.Stat(path)
if err != nil {
log.Debug().Err(err).Str("file", path).Msg("Skipping watch event: path no longer accessible")
return
}
if info.IsDir() {
return
}
options := *q.options
options.Path = path
if err := q.optimize(&options); err != nil {
log.Error().Err(err).Str("file", path).Msg("Error processing file")
}
}
// Enqueue submits path for processing by the worker pool.
func (q *optimizeQueue) Enqueue(path string) {
q.mu.RLock()
defer q.mu.RUnlock()
if q.stopped {
return
}
q.jobs <- path
}
// Stop closes the job queue and waits for in-flight jobs to finish.
func (q *optimizeQueue) Stop() {
q.mu.Lock()
q.stopped = true
close(q.jobs)
q.mu.Unlock()
q.wg.Wait()
}
@@ -0,0 +1,204 @@
package commands
import (
"os"
"path/filepath"
"runtime"
"sync"
"sync/atomic"
"testing"
"time"
utils2 "github.com/belphemur/CBZOptimizer/v2/internal/utils"
"github.com/fsnotify/fsnotify"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestIsComicArchive(t *testing.T) {
testCases := []struct {
name string
path string
expected bool
}{
{"cbz lowercase", "/a/b/chapter.cbz", true},
{"cbr lowercase", "/a/b/chapter.cbr", true},
{"cbz uppercase", "/a/b/chapter.CBZ", true},
{"other extension", "/a/b/chapter.zip", false},
{"no extension", "/a/b/chapter", false},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
assert.Equal(t, tc.expected, isComicArchive(tc.path))
})
}
}
func TestShouldProcessWatchEvent(t *testing.T) {
testCases := []struct {
name string
op fsnotify.Op
expected bool
}{
{"create", fsnotify.Create, true},
{"write", fsnotify.Write, true},
{"rename", fsnotify.Rename, true},
{"remove", fsnotify.Remove, false},
{"chmod", fsnotify.Chmod, false},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
event := fsnotify.Event{Name: "file.cbz", Op: tc.op}
assert.Equal(t, tc.expected, shouldProcessWatchEvent(event))
})
}
}
func TestAddRecursiveWatchSkipsUnreadableSubdirectory(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("permission bits are not reliably enforced on Windows")
}
root := t.TempDir()
nested := filepath.Join(root, "nested")
assert.NoError(t, os.MkdirAll(nested, 0o755))
assert.NoError(t, os.Chmod(nested, 0o000))
defer func() {
assert.NoError(t, os.Chmod(nested, 0o755))
}()
if _, err := os.ReadDir(nested); err == nil {
t.Skip("cannot make nested directory unreadable in this environment")
}
watcher, err := fsnotify.NewWatcher()
assert.NoError(t, err)
defer func() {
assert.NoError(t, watcher.Close())
}()
// A regular, fully accessible tree should be watched without error.
assert.NoError(t, addRecursiveWatch(watcher, root))
}
func TestBackfillExistingArchivesFindsPreExistingArchives(t *testing.T) {
root := t.TempDir()
assert.NoError(t, os.WriteFile(filepath.Join(root, "chapter1.cbz"), []byte("data"), 0o644))
assert.NoError(t, os.WriteFile(filepath.Join(root, "notes.txt"), []byte("data"), 0o644))
sub := filepath.Join(root, "sub")
assert.NoError(t, os.MkdirAll(sub, 0o755))
assert.NoError(t, os.WriteFile(filepath.Join(sub, "chapter2.cbr"), []byte("data"), 0o644))
var mu sync.Mutex
var found []string
backfillExistingArchives(root, func(path string) {
mu.Lock()
defer mu.Unlock()
found = append(found, path)
})
assert.Len(t, found, 2)
}
func TestEventDebouncerCoalescesBurstsIntoSingleCall(t *testing.T) {
var calls int32
debouncer := newEventDebouncer(20*time.Millisecond, func(path string) {
atomic.AddInt32(&calls, 1)
})
defer debouncer.Stop()
// Simulate a burst of events for the same path.
for i := 0; i < 5; i++ {
debouncer.Trigger("/tmp/chapter.cbz")
}
require.Eventually(t, func() bool {
return atomic.LoadInt32(&calls) == 1
}, 2*time.Second, 10*time.Millisecond)
}
func TestEventDebouncerHandlesMultiplePaths(t *testing.T) {
var mu sync.Mutex
seen := make(map[string]int)
debouncer := newEventDebouncer(10*time.Millisecond, func(path string) {
mu.Lock()
defer mu.Unlock()
seen[path]++
})
defer debouncer.Stop()
debouncer.Trigger("/tmp/a.cbz")
debouncer.Trigger("/tmp/b.cbz")
require.Eventually(t, func() bool {
mu.Lock()
defer mu.Unlock()
return seen["/tmp/a.cbz"] == 1 && seen["/tmp/b.cbz"] == 1
}, 2*time.Second, 10*time.Millisecond)
}
func TestOptimizeQueueSkipsMissingPath(t *testing.T) {
q := newOptimizeQueue(1, &utils2.OptimizeOptions{})
defer q.Stop()
var calls int32
done := make(chan struct{}, 1)
existing := filepath.Join(t.TempDir(), "existing.cbz")
require.NoError(t, os.WriteFile(existing, []byte("data"), 0o644))
q.optimize = func(options *utils2.OptimizeOptions) error {
if options.Path == existing {
atomic.AddInt32(&calls, 1)
select {
case done <- struct{}{}:
default:
}
}
return nil
}
// Enqueue a path that doesn't exist; process should skip it without
// panicking or blocking since it never reaches utils2.Optimize.
q.Enqueue(filepath.Join(t.TempDir(), "missing.cbz"))
q.Enqueue(existing)
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("timed out waiting for existing path to be processed")
}
assert.EqualValues(t, 1, atomic.LoadInt32(&calls))
}
func TestWatchCommandBackfillFlagDefaultsToFalse(t *testing.T) {
watchCmd, _, err := rootCmd.Find([]string{"watch"})
require.NoError(t, err)
flag := watchCmd.Flags().Lookup("backfill")
require.NotNil(t, flag, "watch command should register a --backfill flag")
assert.Equal(t, "false", flag.DefValue)
assert.Equal(t, "bool", flag.Value.Type())
}
func TestMaybeBackfillExistingArchivesOnlyInvokedWhenRequested(t *testing.T) {
root := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(root, "chapter1.cbz"), []byte("data"), 0o644))
runBackfill := func(enabled bool) []string {
var found []string
var mu sync.Mutex
process := func(path string) {
mu.Lock()
defer mu.Unlock()
found = append(found, path)
}
// This is the exact same gating call WatchCommand makes based on the
// --backfill flag value.
maybeBackfillExistingArchives(enabled, root, process)
return found
}
assert.Empty(t, runBackfill(false), "no pre-existing archive should be processed when backfill is disabled")
assert.Len(t, runBackfill(true), 1, "pre-existing archives should be processed when backfill is enabled")
}
+1 -1
View File
@@ -19,7 +19,7 @@ CBZOptimizer is a Go CLI that optimizes comic archives (`.cbz` and `.cbr`) by co
## Watch mode ## Watch mode
`watch` monitors a directory tree for archive file changes and runs optimization automatically. `watch` monitors a directory tree for archive file changes and runs optimization automatically. By default it only reacts to changes going forward; pass `--backfill` to also optimize archives that already exist in the folder at startup.
## Key runtime requirements ## Key runtime requirements
+5 -5
View File
@@ -7,15 +7,13 @@ require (
github.com/belphemur/go-webpbin/v2 v2.1.0 github.com/belphemur/go-webpbin/v2 v2.1.0
github.com/fsnotify/fsnotify v1.10.1 github.com/fsnotify/fsnotify v1.10.1
github.com/mholt/archives v0.1.5 github.com/mholt/archives v0.1.5
github.com/oliamb/cutter v0.2.2
github.com/rs/zerolog v1.35.1 github.com/rs/zerolog v1.35.1
github.com/samber/lo v1.53.0 github.com/samber/lo v1.53.0
github.com/spf13/cobra v1.10.2 github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0 github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.11.1 github.com/stretchr/testify v1.11.1
github.com/thediveo/enumflag/v2 v2.2.1 github.com/thediveo/enumflag/v2 v2.2.1
golang.org/x/exp v0.0.0-20260611194520-c48552f49976 golang.org/x/image v0.44.0
golang.org/x/image v0.43.0
) )
require ( require (
@@ -50,7 +48,9 @@ require (
github.com/ulikunitz/xz v0.5.15 // indirect github.com/ulikunitz/xz v0.5.15 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect
go4.org v0.0.0-20230225012048-214862532bf5 // indirect go4.org v0.0.0-20230225012048-214862532bf5 // indirect
golang.org/x/sys v0.45.0 // indirect golang.org/x/net v0.56.0 // indirect
golang.org/x/text v0.38.0 // indirect golang.org/x/sys v0.46.0 // indirect
golang.org/x/text v0.40.0 // indirect
golang.org/x/tools v0.47.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect
) )
+9 -8
View File
@@ -128,8 +128,6 @@ github.com/minio/minlz v1.0.1 h1:OUZUzXcib8diiX+JYxyRLIdomyZYzHct6EShOKtQY2A=
github.com/minio/minlz v1.0.1/go.mod h1:qT0aEB35q79LLornSzeDH75LBf3aH1MV+jB5w9Wasec= github.com/minio/minlz v1.0.1/go.mod h1:qT0aEB35q79LLornSzeDH75LBf3aH1MV+jB5w9Wasec=
github.com/nwaples/rardecode/v2 v2.2.0 h1:4ufPGHiNe1rYJxYfehALLjup4Ls3ck42CWwjKiOqu0A= github.com/nwaples/rardecode/v2 v2.2.0 h1:4ufPGHiNe1rYJxYfehALLjup4Ls3ck42CWwjKiOqu0A=
github.com/nwaples/rardecode/v2 v2.2.0/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw= github.com/nwaples/rardecode/v2 v2.2.0/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw=
github.com/oliamb/cutter v0.2.2 h1:Lfwkya0HHNU1YLnGv2hTkzHfasrSMkgv4Dn+5rmlk3k=
github.com/oliamb/cutter v0.2.2/go.mod h1:4BenG2/4GuRBDbVm/OPahDVqbrOemzpPiG5mi1iryBU=
github.com/onsi/ginkgo/v2 v2.29.0 h1:rfh+ZFjgJhYWRoIqVf3Uwx/W20yLrcrE2h2GmYVRaag= github.com/onsi/ginkgo/v2 v2.29.0 h1:rfh+ZFjgJhYWRoIqVf3Uwx/W20yLrcrE2h2GmYVRaag=
github.com/onsi/ginkgo/v2 v2.29.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44= github.com/onsi/ginkgo/v2 v2.29.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44=
github.com/onsi/gomega v1.41.0 h1:OwKp4pXNgVxf6sCplzYo794OFNuoL2q2SBMU5NSWOjA= github.com/onsi/gomega v1.41.0 h1:OwKp4pXNgVxf6sCplzYo794OFNuoL2q2SBMU5NSWOjA=
@@ -215,12 +213,12 @@ golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20260611194520-c48552f49976 h1:X8Hz2ImujgbmetVuW+w2YkyZChE3cBpZi2P158rTG9M=
golang.org/x/exp v0.0.0-20260611194520-c48552f49976/go.mod h1:vnf4pv9iKZXY58sQE1L86zmNWJ4159e1RkcWiLCkeEY=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.43.0 h1:FLxcP4ec2350nTfOC8ysKtqYSIFbk/QGjw1ZHNP4tsY= golang.org/x/image v0.43.0 h1:FLxcP4ec2350nTfOC8ysKtqYSIFbk/QGjw1ZHNP4tsY=
golang.org/x/image v0.43.0/go.mod h1:rrpelvGFt+kLPAjPM4HeWPgrl0FtafueU//e5N0qk/Q= golang.org/x/image v0.43.0/go.mod h1:rrpelvGFt+kLPAjPM4HeWPgrl0FtafueU//e5N0qk/Q=
golang.org/x/image v0.44.0 h1:+tDekMZED9+LrtB3G5xzRggpVh9CARjZqROla3R3R+I=
golang.org/x/image v0.44.0/go.mod h1:V8K3KE9KKKE+pLpQDOeN18w9oacNSvy1tDOirTu4xtY=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -255,8 +253,8 @@ golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -289,8 +287,8 @@ golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
@@ -303,6 +301,8 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE= golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4= golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -332,6 +332,7 @@ golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapK
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.46.0 h1:7jTurBkPZu4moS/Uy4OQT1M+QBlsj3wejyZwsT8Z7rk= golang.org/x/tools v0.46.0 h1:7jTurBkPZu4moS/Uy4OQT1M+QBlsj3wejyZwsT8Z7rk=
golang.org/x/tools v0.46.0/go.mod h1:FrD85F8l+NWL+9XWBSyVSHO6Ne4jutsfIFba7AWQ5Ys= golang.org/x/tools v0.46.0/go.mod h1:FrD85F8l+NWL+9XWBSyVSHO6Ne4jutsfIFba7AWQ5Ys=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+26 -46
View File
@@ -12,7 +12,9 @@ import (
"github.com/rs/zerolog/log" "github.com/rs/zerolog/log"
) )
func WriteChapterToCBZ(chapter *manga.Chapter, outputFilePath string) error { // WriteChapterToCBZ creates a CBZ file from a Chapter by streaming page files
// from disk directly into the zip archive. No image data is held in memory.
func WriteChapterToCBZ(chapter *manga.Chapter, outputFilePath string) (err error) {
log.Debug(). log.Debug().
Str("chapter_file", chapter.FilePath). Str("chapter_file", chapter.FilePath).
Str("output_path", outputFilePath). Str("output_path", outputFilePath).
@@ -20,8 +22,7 @@ func WriteChapterToCBZ(chapter *manga.Chapter, outputFilePath string) error {
Bool("is_converted", chapter.IsConverted). Bool("is_converted", chapter.IsConverted).
Msg("Starting CBZ file creation") Msg("Starting CBZ file creation")
// Create a new ZIP file // Create output file
log.Debug().Str("output_path", outputFilePath).Msg("Creating output CBZ file")
zipFile, err := os.Create(outputFilePath) zipFile, err := os.Create(outputFilePath)
if err != nil { if err != nil {
log.Error().Str("output_path", outputFilePath).Err(err).Msg("Failed to create CBZ file") log.Error().Str("output_path", outputFilePath).Err(err).Msg("Failed to create CBZ file")
@@ -29,109 +30,88 @@ func WriteChapterToCBZ(chapter *manga.Chapter, outputFilePath string) error {
} }
defer errs.Capture(&err, zipFile.Close, "failed to close .cbz file") defer errs.Capture(&err, zipFile.Close, "failed to close .cbz file")
// Create a new ZIP writer // Create ZIP writer
log.Debug().Str("output_path", outputFilePath).Msg("Creating ZIP writer")
zipWriter := zip.NewWriter(zipFile) zipWriter := zip.NewWriter(zipFile)
if err != nil {
log.Error().Str("output_path", outputFilePath).Err(err).Msg("Failed to create ZIP writer")
return err
}
defer errs.Capture(&err, zipWriter.Close, "failed to close .cbz writer") defer errs.Capture(&err, zipWriter.Close, "failed to close .cbz writer")
// Write each page to the ZIP archive // Write each page to the archive by streaming from disk
log.Debug().Str("output_path", outputFilePath).Int("pages_to_write", len(chapter.Pages)).Msg("Writing pages to CBZ archive")
for _, page := range chapter.Pages { for _, page := range chapter.Pages {
// Construct the file name for the page
var fileName string var fileName string
if page.IsSplitted { if page.IsSplitted {
// Use the format page%03d-%02d for split pages
fileName = fmt.Sprintf("%04d-%02d%s", page.Index, page.SplitPartIndex, page.Extension) fileName = fmt.Sprintf("%04d-%02d%s", page.Index, page.SplitPartIndex, page.Extension)
} else { } else {
// Use the format page%03d for non-split pages
fileName = fmt.Sprintf("%04d%s", page.Index, page.Extension) fileName = fmt.Sprintf("%04d%s", page.Index, page.Extension)
} }
log.Debug(). log.Debug().
Str("output_path", outputFilePath). Str("output_path", outputFilePath).
Uint16("page_index", page.Index). Uint16("page_index", page.Index).
Bool("is_splitted", page.IsSplitted).
Uint16("split_part", page.SplitPartIndex).
Str("filename", fileName). Str("filename", fileName).
Uint64("size", page.Size). Str("source", page.FilePath).
Msg("Writing page to CBZ archive") Msg("Writing page to CBZ archive")
// Create a new file in the ZIP archive // Create file entry in the zip (Store method = no compression, images are already compressed)
fileWriter, err := zipWriter.CreateHeader(&zip.FileHeader{ fileWriter, err := zipWriter.CreateHeader(&zip.FileHeader{
Name: fileName, Name: fileName,
Method: zip.Store, Method: zip.Store,
Modified: time.Now(), Modified: time.Now(),
}) })
if err != nil { if err != nil {
log.Error().Str("output_path", outputFilePath).Str("filename", fileName).Err(err).Msg("Failed to create file in CBZ archive") log.Error().Str("filename", fileName).Err(err).Msg("Failed to create file in CBZ archive")
return fmt.Errorf("failed to create file in .cbz: %w", err) return fmt.Errorf("failed to create file in .cbz: %w", err)
} }
// Stream the page contents into the archive. This transparently // Stream the page file from disk into the archive
// handles pages staged on disk (see manga.Page.TempFilePath) so we pageFile, err := os.Open(page.FilePath)
// never need to hold the whole chapter's contents in memory at once.
pageReader, err := page.Open()
if err != nil { if err != nil {
log.Error().Str("output_path", outputFilePath).Str("filename", fileName).Err(err).Msg("Failed to open page contents") log.Error().Str("filename", fileName).Str("source", page.FilePath).Err(err).Msg("Failed to open page file")
return fmt.Errorf("failed to open page contents: %w", err) return fmt.Errorf("failed to open page file: %w", err)
} }
bytesWritten, err := io.Copy(fileWriter, pageReader) bytesWritten, err := io.Copy(fileWriter, pageFile)
closeErr := pageReader.Close() closeErr := pageFile.Close()
if err != nil { if err != nil {
log.Error().Str("output_path", outputFilePath).Str("filename", fileName).Err(err).Msg("Failed to write page contents") log.Error().Str("filename", fileName).Err(err).Msg("Failed to write page contents")
return fmt.Errorf("failed to write page contents: %w", err) return fmt.Errorf("failed to write page contents: %w", err)
} }
if closeErr != nil { if closeErr != nil {
log.Error().Str("output_path", outputFilePath).Str("filename", fileName).Err(closeErr).Msg("Failed to close page contents reader") log.Error().Str("filename", fileName).Err(closeErr).Msg("Failed to close page file")
return fmt.Errorf("failed to close page contents reader: %w", closeErr) return fmt.Errorf("failed to close page file: %w", closeErr)
} }
log.Debug(). log.Debug().
Str("output_path", outputFilePath).
Str("filename", fileName). Str("filename", fileName).
Int64("bytes_written", bytesWritten). Int64("bytes_written", bytesWritten).
Msg("Page written successfully") Msg("Page written successfully")
} }
// Optionally, write the ComicInfo.xml file if present // Write ComicInfo.xml if present
if chapter.ComicInfoXml != "" { if chapter.ComicInfoXml != "" {
log.Debug().Str("output_path", outputFilePath).Int("xml_size", len(chapter.ComicInfoXml)).Msg("Writing ComicInfo.xml to CBZ archive") log.Debug().Str("output_path", outputFilePath).Msg("Writing ComicInfo.xml")
comicInfoWriter, err := zipWriter.CreateHeader(&zip.FileHeader{ comicInfoWriter, err := zipWriter.CreateHeader(&zip.FileHeader{
Name: "ComicInfo.xml", Name: "ComicInfo.xml",
Method: zip.Deflate, Method: zip.Deflate,
Modified: time.Now(), Modified: time.Now(),
}) })
if err != nil { if err != nil {
log.Error().Str("output_path", outputFilePath).Err(err).Msg("Failed to create ComicInfo.xml in CBZ archive")
return fmt.Errorf("failed to create ComicInfo.xml in .cbz: %w", err) return fmt.Errorf("failed to create ComicInfo.xml in .cbz: %w", err)
} }
bytesWritten, err := comicInfoWriter.Write([]byte(chapter.ComicInfoXml)) _, err = comicInfoWriter.Write([]byte(chapter.ComicInfoXml))
if err != nil { if err != nil {
log.Error().Str("output_path", outputFilePath).Err(err).Msg("Failed to write ComicInfo.xml contents") return fmt.Errorf("failed to write ComicInfo.xml: %w", err)
return fmt.Errorf("failed to write ComicInfo.xml contents: %w", err)
} }
log.Debug().Str("output_path", outputFilePath).Int("bytes_written", bytesWritten).Msg("ComicInfo.xml written successfully")
} else {
log.Debug().Str("output_path", outputFilePath).Msg("No ComicInfo.xml to write")
} }
// Set zip comment for converted chapters
if chapter.IsConverted { if chapter.IsConverted {
convertedString := fmt.Sprintf("%s\nThis chapter has been converted by CBZOptimizer.", chapter.ConvertedTime) comment := fmt.Sprintf("%s\nThis chapter has been converted by CBZOptimizer.", chapter.ConvertedTime)
log.Debug().Str("output_path", outputFilePath).Str("comment", convertedString).Msg("Setting CBZ comment for converted chapter") err = zipWriter.SetComment(comment)
err = zipWriter.SetComment(convertedString)
if err != nil { if err != nil {
log.Error().Str("output_path", outputFilePath).Err(err).Msg("Failed to write CBZ comment")
return fmt.Errorf("failed to write comment: %w", err) return fmt.Errorf("failed to write comment: %w", err)
} }
log.Debug().Str("output_path", outputFilePath).Msg("CBZ comment set successfully")
} }
log.Debug().Str("output_path", outputFilePath).Msg("CBZ file creation completed successfully") log.Debug().Str("output_path", outputFilePath).Msg("CBZ file creation completed")
return nil return nil
} }
+105 -37
View File
@@ -2,88 +2,102 @@ package cbz
import ( import (
"archive/zip" "archive/zip"
"bytes"
"fmt" "fmt"
"github.com/belphemur/CBZOptimizer/v2/internal/manga"
"github.com/belphemur/CBZOptimizer/v2/internal/utils/errs"
"os" "os"
"path/filepath"
"testing" "testing"
"time" "time"
"github.com/belphemur/CBZOptimizer/v2/internal/manga"
"github.com/belphemur/CBZOptimizer/v2/internal/utils/errs"
) )
func TestWriteChapterToCBZ(t *testing.T) { func TestWriteChapterToCBZ(t *testing.T) {
currentTime := time.Now() currentTime := time.Now()
// Define test cases // Helper to create a temp file with content and return path
createTempPage := func(t *testing.T, dir, content, ext string) string {
t.Helper()
f, err := os.CreateTemp(dir, "page-*"+ext)
if err != nil {
t.Fatal(err)
}
_, err = f.WriteString(content)
if err != nil {
_ = f.Close()
t.Fatal(err)
}
_ = f.Close()
return f.Name()
}
testCases := []struct { testCases := []struct {
name string name string
chapter *manga.Chapter chapter func(t *testing.T, dir string) *manga.Chapter
expectedFiles []string expectedFiles []string
expectedComment string expectedComment string
}{ }{
//test case where there is only one page and ComicInfo and the chapter is converted
{ {
name: "Single page, ComicInfo, converted", name: "Single page, ComicInfo, converted",
chapter: &manga.Chapter{ chapter: func(t *testing.T, dir string) *manga.Chapter {
Pages: []*manga.Page{ return &manga.Chapter{
Pages: []*manga.PageFile{
{ {
Index: 0, Index: 0,
Extension: ".jpg", Extension: ".jpg",
Contents: bytes.NewBuffer([]byte("image data")), FilePath: createTempPage(t, dir, "image data", ".jpg"),
}, },
}, },
ComicInfoXml: "<Series>Boundless Necromancer</Series>", ComicInfoXml: "<Series>Boundless Necromancer</Series>",
IsConverted: true, IsConverted: true,
ConvertedTime: currentTime, ConvertedTime: currentTime,
}
}, },
expectedFiles: []string{"0000.jpg", "ComicInfo.xml"}, expectedFiles: []string{"0000.jpg", "ComicInfo.xml"},
expectedComment: fmt.Sprintf("%s\nThis chapter has been converted by CBZOptimizer.", currentTime), expectedComment: fmt.Sprintf("%s\nThis chapter has been converted by CBZOptimizer.", currentTime),
}, },
//test case where there is only one page and no
{ {
name: "Single page, no ComicInfo", name: "Single page, no ComicInfo",
chapter: &manga.Chapter{ chapter: func(t *testing.T, dir string) *manga.Chapter {
Pages: []*manga.Page{ return &manga.Chapter{
Pages: []*manga.PageFile{
{ {
Index: 0, Index: 0,
Extension: ".jpg", Extension: ".jpg",
Contents: bytes.NewBuffer([]byte("image data")), FilePath: createTempPage(t, dir, "image data", ".jpg"),
}, },
}, },
}
}, },
expectedFiles: []string{"0000.jpg"}, expectedFiles: []string{"0000.jpg"},
}, },
{ {
name: "Multiple pages with ComicInfo", name: "Multiple pages with ComicInfo",
chapter: &manga.Chapter{ chapter: func(t *testing.T, dir string) *manga.Chapter {
Pages: []*manga.Page{ return &manga.Chapter{
{ Pages: []*manga.PageFile{
Index: 0, {Index: 0, Extension: ".jpg", FilePath: createTempPage(t, dir, "image data 1", ".jpg")},
Extension: ".jpg", {Index: 1, Extension: ".jpg", FilePath: createTempPage(t, dir, "image data 2", ".jpg")},
Contents: bytes.NewBuffer([]byte("image data 1")),
},
{
Index: 1,
Extension: ".jpg",
Contents: bytes.NewBuffer([]byte("image data 2")),
},
}, },
ComicInfoXml: "<Series>Boundless Necromancer</Series>", ComicInfoXml: "<Series>Boundless Necromancer</Series>",
}
}, },
expectedFiles: []string{"0000.jpg", "0001.jpg", "ComicInfo.xml"}, expectedFiles: []string{"0000.jpg", "0001.jpg", "ComicInfo.xml"},
}, },
{ {
name: "Split page", name: "Split page",
chapter: &manga.Chapter{ chapter: func(t *testing.T, dir string) *manga.Chapter {
Pages: []*manga.Page{ return &manga.Chapter{
Pages: []*manga.PageFile{
{ {
Index: 0, Index: 0,
Extension: ".jpg", Extension: ".jpg",
Contents: bytes.NewBuffer([]byte("split image data")), FilePath: createTempPage(t, dir, "split image data", ".jpg"),
IsSplitted: true, IsSplitted: true,
SplitPartIndex: 1, SplitPartIndex: 1,
}, },
}, },
}
}, },
expectedFiles: []string{"0000-01.jpg"}, expectedFiles: []string{"0000-01.jpg"},
}, },
@@ -91,33 +105,36 @@ func TestWriteChapterToCBZ(t *testing.T) {
for _, tc := range testCases { for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
// Create a temporary file for the .cbz output // Create temp dir for page files
pageDir := t.TempDir()
chapter := tc.chapter(t, pageDir)
// Create temp file for output CBZ
tempFile, err := os.CreateTemp("", "*.cbz") tempFile, err := os.CreateTemp("", "*.cbz")
if err != nil { if err != nil {
t.Fatalf("Failed to create temporary file: %v", err) t.Fatalf("Failed to create temporary file: %v", err)
} }
_ = tempFile.Close()
defer errs.CaptureGeneric(&err, os.Remove, tempFile.Name(), "failed to remove temporary file") defer errs.CaptureGeneric(&err, os.Remove, tempFile.Name(), "failed to remove temporary file")
// Write the chapter to the .cbz file // Write chapter
err = WriteChapterToCBZ(tc.chapter, tempFile.Name()) err = WriteChapterToCBZ(chapter, tempFile.Name())
if err != nil { if err != nil {
t.Fatalf("Failed to write chapter to CBZ: %v", err) t.Fatalf("Failed to write chapter to CBZ: %v", err)
} }
// Open the .cbz file as a zip archive // Verify the archive
r, err := zip.OpenReader(tempFile.Name()) r, err := zip.OpenReader(tempFile.Name())
if err != nil { if err != nil {
t.Fatalf("Failed to open CBZ file: %v", err) t.Fatalf("Failed to open CBZ file: %v", err)
} }
defer errs.Capture(&err, r.Close, "failed to close CBZ file") defer func() { _ = r.Close() }()
// Collect the names of the files in the archive
var filesInArchive []string var filesInArchive []string
for _, f := range r.File { for _, f := range r.File {
filesInArchive = append(filesInArchive, f.Name) filesInArchive = append(filesInArchive, f.Name)
} }
// Check if all expected files are present
for _, expectedFile := range tc.expectedFiles { for _, expectedFile := range tc.expectedFiles {
found := false found := false
for _, actualFile := range filesInArchive { for _, actualFile := range filesInArchive {
@@ -135,10 +152,61 @@ func TestWriteChapterToCBZ(t *testing.T) {
t.Errorf("Expected comment %s, but found %s", tc.expectedComment, r.Comment) t.Errorf("Expected comment %s, but found %s", tc.expectedComment, r.Comment)
} }
// Check if there are no unexpected files
if len(filesInArchive) != len(tc.expectedFiles) { if len(filesInArchive) != len(tc.expectedFiles) {
t.Errorf("Expected %d files, but found %d", len(tc.expectedFiles), len(filesInArchive)) t.Errorf("Expected %d files, but found %d: %v", len(tc.expectedFiles), len(filesInArchive), filesInArchive)
} }
}) })
} }
} }
func TestWriteAndReadRoundTrip(t *testing.T) {
// Create a temp directory with page files
pageDir := t.TempDir()
// Create some page files
for i := 0; i < 3; i++ {
pagePath := filepath.Join(pageDir, fmt.Sprintf("%04d.jpg", i))
err := os.WriteFile(pagePath, []byte(fmt.Sprintf("image data %d", i)), 0644)
if err != nil {
t.Fatal(err)
}
}
// Create chapter
chapter := &manga.Chapter{
FilePath: "/test/chapter.cbz",
Pages: []*manga.PageFile{
{Index: 0, Extension: ".jpg", FilePath: filepath.Join(pageDir, "0000.jpg")},
{Index: 1, Extension: ".jpg", FilePath: filepath.Join(pageDir, "0001.jpg")},
{Index: 2, Extension: ".jpg", FilePath: filepath.Join(pageDir, "0002.jpg")},
},
ComicInfoXml: "<Series>Test Series</Series>",
}
chapter.SetConverted()
// Write to CBZ
outputPath := filepath.Join(t.TempDir(), "output.cbz")
err := WriteChapterToCBZ(chapter, outputPath)
if err != nil {
t.Fatalf("Failed to write chapter: %v", err)
}
// Read it back
loaded, err := LoadChapter(outputPath)
if err != nil {
t.Fatalf("Failed to load chapter: %v", err)
}
defer func() { _ = loaded.Cleanup() }()
if !loaded.IsConverted {
t.Error("Loaded chapter should be marked as converted")
}
if len(loaded.Pages) != 3 {
t.Errorf("Expected 3 pages, got %d", len(loaded.Pages))
}
if loaded.ComicInfoXml != "<Series>Test Series</Series>" {
t.Errorf("ComicInfoXml mismatch: %s", loaded.ComicInfoXml)
}
}
+259 -98
View File
@@ -3,13 +3,14 @@ package cbz
import ( import (
"archive/zip" "archive/zip"
"bufio" "bufio"
"bytes"
"context" "context"
"fmt" "fmt"
"io" "io"
"io/fs" "io/fs"
"os"
"path/filepath" "path/filepath"
"strings" "strings"
"time"
"github.com/araddon/dateparse" "github.com/araddon/dateparse"
"github.com/belphemur/CBZOptimizer/v2/internal/manga" "github.com/belphemur/CBZOptimizer/v2/internal/manga"
@@ -18,149 +19,309 @@ import (
"github.com/rs/zerolog/log" "github.com/rs/zerolog/log"
) )
func LoadChapter(filePath string) (*manga.Chapter, error) { // supportedImageExtensions contains file extensions considered valid image pages.
log.Debug().Str("file_path", filePath).Msg("Starting chapter loading") var supportedImageExtensions = map[string]bool{
".jpg": true,
".jpeg": true,
".png": true,
".gif": true,
".webp": true,
".bmp": true,
".tiff": true,
".tif": true,
}
ctx := context.Background() // parseConvertedComment checks if a zip comment's first line is a parseable date,
// indicating the archive was already converted. Returns true and the parsed time if so.
func parseConvertedComment(comment string) bool {
if comment == "" {
return false
}
scanner := bufio.NewScanner(strings.NewReader(comment))
if scanner.Scan() {
_, err := dateparse.ParseAny(scanner.Text())
return err == nil
}
return false
}
// parseConvertedCommentTime parses the converted timestamp from a zip comment.
// Returns the time and true if the comment indicates conversion, otherwise zero time and false.
func parseConvertedCommentTime(comment string) (time.Time, bool) {
if comment == "" {
return time.Time{}, false
}
scanner := bufio.NewScanner(strings.NewReader(comment))
if scanner.Scan() {
t, err := dateparse.ParseAny(scanner.Text())
if err == nil {
return t, true
}
}
return time.Time{}, false
}
// IsAlreadyConverted performs a fast check to see if the archive is already
// converted without extracting any image data. It reads only the zip comment
// and metadata files (converted.txt) to determine conversion status.
func IsAlreadyConverted(ctx context.Context, filePath string) (converted bool, err error) {
log.Debug().Str("file_path", filePath).Msg("Checking if already converted")
pathLower := strings.ToLower(filepath.Ext(filePath))
if pathLower == ".cbz" {
r, err := zip.OpenReader(filePath)
if err != nil {
return false, fmt.Errorf("failed to open CBZ for conversion check: %w", err)
}
defer errs.Capture(&err, r.Close, "failed to close zip reader")
// Check zip comment
if parseConvertedComment(r.Comment) {
log.Debug().Str("file_path", filePath).Msg("Already converted (zip comment)")
return true, nil
}
// Check for converted.txt inside the archive
for _, f := range r.File {
if strings.ToLower(filepath.Base(f.Name)) == "converted.txt" {
rc, err := f.Open()
if err != nil {
continue
}
scanner := bufio.NewScanner(rc)
if scanner.Scan() {
_, parseErr := dateparse.ParseAny(scanner.Text())
_ = rc.Close()
if parseErr == nil {
log.Debug().Str("file_path", filePath).Msg("Already converted (converted.txt)")
return true, nil
}
} else {
_ = rc.Close()
}
}
}
}
// For CBR files, we need to use the archives library to check
if pathLower == ".cbr" {
fsys, err := archives.FileSystem(ctx, filePath, nil)
if err != nil {
return false, fmt.Errorf("failed to open archive: %w", err)
}
var converted bool
_ = fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
if strings.ToLower(filepath.Base(path)) == "converted.txt" {
file, err := fsys.Open(path)
if err != nil {
return nil
}
defer func() { _ = file.Close() }()
scanner := bufio.NewScanner(file)
if scanner.Scan() {
_, err := dateparse.ParseAny(scanner.Text())
if err == nil {
converted = true
return fs.SkipAll
}
}
}
return nil
})
return converted, nil
}
return false, nil
}
// ExtractChapter extracts an archive (CBZ/CBR) to a temp directory on disk.
// Pages are streamed directly to files — no image data is held in memory.
// Returns a Chapter with PageFile entries pointing to extracted files.
func ExtractChapter(ctx context.Context, filePath string) (*manga.Chapter, error) {
log.Debug().Str("file_path", filePath).Msg("Extracting chapter to disk")
// Create temp directory for extraction
tempDir, err := os.MkdirTemp("", "cbzoptimizer-*")
if err != nil {
return nil, fmt.Errorf("failed to create temp directory: %w", err)
}
inputDir := filepath.Join(tempDir, "input")
if err := os.MkdirAll(inputDir, 0755); err != nil {
_ = os.RemoveAll(tempDir)
return nil, fmt.Errorf("failed to create input directory: %w", err)
}
chapter := &manga.Chapter{ chapter := &manga.Chapter{
FilePath: filePath, FilePath: filePath,
TempDir: tempDir,
} }
// First, try to read the comment using zip.OpenReader for CBZ files // For CBZ files, read metadata from zip comment
if strings.ToLower(filepath.Ext(filePath)) == ".cbz" { pathLower := strings.ToLower(filepath.Ext(filePath))
log.Debug().Str("file_path", filePath).Msg("Checking CBZ comment for conversion status") if pathLower == ".cbz" {
r, err := zip.OpenReader(filePath) r, err := zip.OpenReader(filePath)
if err == nil { if err == nil {
defer errs.Capture(&err, r.Close, "failed to close zip reader for comment") if t, ok := parseConvertedCommentTime(r.Comment); ok {
// Check for comment
if r.Comment != "" {
log.Debug().Str("file_path", filePath).Str("comment", r.Comment).Msg("Found CBZ comment")
scanner := bufio.NewScanner(strings.NewReader(r.Comment))
if scanner.Scan() {
convertedTime := scanner.Text()
log.Debug().Str("file_path", filePath).Str("converted_time", convertedTime).Msg("Parsing conversion timestamp")
chapter.ConvertedTime, err = dateparse.ParseAny(convertedTime)
if err == nil {
chapter.IsConverted = true chapter.IsConverted = true
log.Debug().Str("file_path", filePath).Time("converted_time", chapter.ConvertedTime).Msg("Chapter marked as previously converted") chapter.ConvertedTime = t
} else {
log.Debug().Str("file_path", filePath).Err(err).Msg("Failed to parse conversion timestamp")
} }
_ = r.Close()
} }
} else {
log.Debug().Str("file_path", filePath).Msg("No CBZ comment found")
}
} else {
log.Debug().Str("file_path", filePath).Err(err).Msg("Failed to open CBZ file for comment reading")
}
// Continue even if comment reading fails
} }
// Open the archive using archives library for file operations // Extract files using the archives library (supports both CBZ and CBR)
log.Debug().Str("file_path", filePath).Msg("Opening archive file system")
fsys, err := archives.FileSystem(ctx, filePath, nil) fsys, err := archives.FileSystem(ctx, filePath, nil)
if err != nil { if err != nil {
log.Error().Str("file_path", filePath).Err(err).Msg("Failed to open archive file system") _ = os.RemoveAll(tempDir)
return nil, fmt.Errorf("failed to open archive file: %w", err) return nil, fmt.Errorf("failed to open archive: %w", err)
} }
// Walk through all files in the filesystem err = fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, walkErr error) error {
log.Debug().Str("file_path", filePath).Msg("Starting filesystem walk") if walkErr != nil {
err = fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error { return walkErr
if err != nil {
return err
} }
if d.IsDir() { if d.IsDir() {
return nil return nil
} }
return func() error { // Check for context cancellation during extraction
// Open the file select {
case <-ctx.Done():
return ctx.Err()
default:
}
ext := strings.ToLower(filepath.Ext(path))
fileName := strings.ToLower(filepath.Base(path))
// Skip OS-specific metadata files and junk
if isJunkFile(path) {
log.Debug().Str("file_path", filePath).Str("skipped", path).Msg("Skipping junk file")
return nil
}
// Handle ComicInfo.xml
if ext == ".xml" && fileName == "comicinfo.xml" {
file, err := fsys.Open(path)
if err != nil {
return fmt.Errorf("failed to open ComicInfo.xml: %w", err)
}
defer func() { _ = file.Close() }()
xmlContent, err := io.ReadAll(file)
if err != nil {
return fmt.Errorf("failed to read ComicInfo.xml: %w", err)
}
chapter.ComicInfoXml = string(xmlContent)
log.Debug().Str("file_path", filePath).Int("xml_size", len(xmlContent)).Msg("ComicInfo.xml loaded")
return nil
}
// Handle converted.txt (check conversion status)
if !chapter.IsConverted && ext == ".txt" && fileName == "converted.txt" {
file, err := fsys.Open(path)
if err != nil {
return fmt.Errorf("failed to open converted.txt: %w", err)
}
defer func() { _ = file.Close() }()
scanner := bufio.NewScanner(file)
if scanner.Scan() {
t, err := dateparse.ParseAny(scanner.Text())
if err == nil {
chapter.IsConverted = true
chapter.ConvertedTime = t
}
}
return nil
}
// Only extract supported image files
if !supportedImageExtensions[ext] {
log.Debug().Str("file_path", filePath).Str("skipped", path).Str("ext", ext).Msg("Skipping non-image file")
return nil
}
// Extract image file to disk
file, err := fsys.Open(path) file, err := fsys.Open(path)
if err != nil { if err != nil {
return fmt.Errorf("failed to open file %s: %w", path, err) return fmt.Errorf("failed to open file %s: %w", path, err)
} }
defer errs.Capture(&err, file.Close, fmt.Sprintf("failed to close file %s", path)) defer func() { _ = file.Close() }()
// Determine the file extension // Create output file with sequential naming
ext := strings.ToLower(filepath.Ext(path)) pageIndex := uint16(len(chapter.Pages))
fileName := strings.ToLower(filepath.Base(path)) outputName := fmt.Sprintf("%04d%s", pageIndex, ext)
outputPath := filepath.Join(inputDir, outputName)
if ext == ".xml" && fileName == "comicinfo.xml" { outFile, err := os.Create(outputPath)
log.Debug().Str("file_path", filePath).Str("archive_file", path).Msg("Found ComicInfo.xml")
// Read the ComicInfo.xml file content
xmlContent, err := io.ReadAll(file)
if err != nil { if err != nil {
log.Error().Str("file_path", filePath).Str("archive_file", path).Err(err).Msg("Failed to read ComicInfo.xml") return fmt.Errorf("failed to create output file %s: %w", outputPath, err)
return fmt.Errorf("failed to read ComicInfo.xml content: %w", err)
}
chapter.ComicInfoXml = string(xmlContent)
log.Debug().Str("file_path", filePath).Int("xml_size", len(xmlContent)).Msg("ComicInfo.xml loaded")
} else if !chapter.IsConverted && ext == ".txt" && fileName == "converted.txt" {
log.Debug().Str("file_path", filePath).Str("archive_file", path).Msg("Found converted.txt")
textContent, err := io.ReadAll(file)
if err != nil {
log.Error().Str("file_path", filePath).Str("archive_file", path).Err(err).Msg("Failed to read converted.txt")
return fmt.Errorf("failed to read converted.txt content: %w", err)
}
scanner := bufio.NewScanner(bytes.NewReader(textContent))
if scanner.Scan() {
convertedTime := scanner.Text()
log.Debug().Str("file_path", filePath).Str("converted_time", convertedTime).Msg("Parsing converted.txt timestamp")
chapter.ConvertedTime, err = dateparse.ParseAny(convertedTime)
if err != nil {
log.Error().Str("file_path", filePath).Err(err).Msg("Failed to parse converted time from converted.txt")
return fmt.Errorf("failed to parse converted time: %w", err)
}
chapter.IsConverted = true
log.Debug().Str("file_path", filePath).Time("converted_time", chapter.ConvertedTime).Msg("Chapter marked as converted from converted.txt")
}
} else {
// Read the file contents for page
log.Debug().Str("file_path", filePath).Str("archive_file", path).Str("extension", ext).Msg("Processing page file")
buf := new(bytes.Buffer)
bytesCopied, err := io.Copy(buf, file)
if err != nil {
log.Error().Str("file_path", filePath).Str("archive_file", path).Err(err).Msg("Failed to read page file contents")
return fmt.Errorf("failed to read file contents: %w", err)
} }
// Create a new Page object _, err = io.Copy(outFile, file)
page := &manga.Page{ closeErr := outFile.Close()
Index: uint16(len(chapter.Pages)), // Simple index based on order if err != nil {
_ = os.Remove(outputPath)
return fmt.Errorf("failed to write file %s: %w", outputPath, err)
}
if closeErr != nil {
_ = os.Remove(outputPath)
return fmt.Errorf("failed to close file %s: %w", outputPath, closeErr)
}
page := &manga.PageFile{
Index: pageIndex,
Extension: ext, Extension: ext,
Size: uint64(buf.Len()), FilePath: outputPath,
Contents: buf,
IsSplitted: false,
} }
// Add the page to the chapter
chapter.Pages = append(chapter.Pages, page) chapter.Pages = append(chapter.Pages, page)
log.Debug(). log.Debug().
Str("file_path", filePath). Str("file_path", filePath).
Str("archive_file", path). Str("archive_file", path).
Uint16("page_index", page.Index). Uint16("page_index", pageIndex).
Int64("bytes_read", bytesCopied). Msg("Page extracted to disk")
Msg("Page loaded successfully")
}
return nil return nil
}()
}) })
if err != nil { if err != nil {
log.Error().Str("file_path", filePath).Err(err).Msg("Failed during filesystem walk") _ = os.RemoveAll(tempDir)
return nil, err return nil, fmt.Errorf("failed to extract archive: %w", err)
} }
log.Debug(). log.Debug().
Str("file_path", filePath). Str("file_path", filePath).
Int("pages_loaded", len(chapter.Pages)). Int("pages_extracted", len(chapter.Pages)).
Bool("is_converted", chapter.IsConverted). Bool("is_converted", chapter.IsConverted).
Bool("has_comic_info", chapter.ComicInfoXml != ""). Msg("Chapter extraction completed")
Msg("Chapter loading completed successfully")
return chapter, nil return chapter, nil
} }
// isJunkFile returns true for known OS/tool metadata files that should not be
// treated as pages (e.g., __MACOSX/, Thumbs.db, .DS_Store).
func isJunkFile(path string) bool {
// __MACOSX resource fork directories
if strings.Contains(path, "__MACOSX") {
return true
}
baseLower := strings.ToLower(filepath.Base(path))
switch baseLower {
case "thumbs.db", ".ds_store", "desktop.ini":
return true
}
return false
}
// LoadChapter extracts the chapter from a CBZ/CBR file to disk.
// It delegates to ExtractChapter and always extracts all pages.
// Use IsAlreadyConverted for a fast conversion status check without extraction.
func LoadChapter(filePath string) (*manga.Chapter, error) {
return ExtractChapter(context.Background(), filePath)
}
+377 -14
View File
@@ -1,8 +1,17 @@
package cbz package cbz
import ( import (
"strings" "archive/zip"
"context"
"fmt"
"os"
"path/filepath"
"testing" "testing"
"time"
"github.com/belphemur/CBZOptimizer/v2/internal/manga"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
) )
func TestLoadChapter(t *testing.T) { func TestLoadChapter(t *testing.T) {
@@ -36,28 +45,382 @@ func TestLoadChapter(t *testing.T) {
expectedSeries: "<Series>Boundless Necromancer</Series>", expectedSeries: "<Series>Boundless Necromancer</Series>",
expectedConversion: true, expectedConversion: true,
}, },
// Add more test cases as needed
} }
for _, tc := range testCases { for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
chapter, err := LoadChapter(tc.filePath) chapter, err := LoadChapter(tc.filePath)
if err != nil { require.NoError(t, err)
t.Fatalf("Failed to load chapter: %v", err) defer func() { _ = chapter.Cleanup() }()
}
actualPages := len(chapter.Pages) assert.Equal(t, tc.expectedPages, len(chapter.Pages))
if actualPages != tc.expectedPages {
t.Errorf("Expected %d pages, but got %d", tc.expectedPages, actualPages)
}
if !strings.Contains(chapter.ComicInfoXml, tc.expectedSeries) { assert.Contains(t, chapter.ComicInfoXml, tc.expectedSeries)
t.Errorf("ComicInfoXml does not contain the expected series: %s", tc.expectedSeries)
}
if chapter.IsConverted != tc.expectedConversion { assert.Equal(t, tc.expectedConversion, chapter.IsConverted)
t.Errorf("Expected chapter to be converted: %t, but got %t", tc.expectedConversion, chapter.IsConverted)
// Verify pages are on disk
for _, page := range chapter.Pages {
assert.NotEmpty(t, page.FilePath, "Page %d has no file path", page.Index)
} }
}) })
} }
} }
func TestIsAlreadyConverted(t *testing.T) {
testCases := []struct {
name string
filePath string
expected bool
}{
{
name: "Converted CBZ",
filePath: "../../testdata/Chapter 10_converted.cbz",
expected: true,
},
{
name: "Original CBZ",
filePath: "../../testdata/Chapter 128.cbz",
expected: false,
},
{
name: "Original CBR",
filePath: "../../testdata/Chapter 1.cbr",
expected: false,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
result, err := IsAlreadyConverted(context.Background(), tc.filePath)
require.NoError(t, err)
assert.Equal(t, tc.expected, result)
})
}
}
func TestIsAlreadyConverted_NonexistentFile(t *testing.T) {
_, err := IsAlreadyConverted(context.Background(), "/nonexistent/file.cbz")
require.Error(t, err)
}
func TestIsAlreadyConverted_InvalidExtension(t *testing.T) {
// Create a temp file with unsupported extension
tmpFile, err := os.CreateTemp("", "test-*.txt")
require.NoError(t, err)
defer func() { _ = os.Remove(tmpFile.Name()) }()
_ = tmpFile.Close()
result, err := IsAlreadyConverted(context.Background(), tmpFile.Name())
require.NoError(t, err)
assert.False(t, result, "Expected false for unsupported extension")
}
func TestIsAlreadyConverted_CBZWithConvertedComment(t *testing.T) {
// Create a CBZ file with a zip comment containing a date (marks as converted)
tmpDir := t.TempDir()
cbzPath := filepath.Join(tmpDir, "test.cbz")
f, err := os.Create(cbzPath)
require.NoError(t, err)
w := zip.NewWriter(f)
_ = w.SetComment(time.Now().Format(time.RFC3339))
// Add a dummy file
fw, err := w.Create("page.webp")
require.NoError(t, err)
_, _ = fw.Write([]byte("dummy"))
_ = w.Close()
_ = f.Close()
result, err := IsAlreadyConverted(context.Background(), cbzPath)
require.NoError(t, err)
assert.True(t, result, "Expected CBZ with date comment to be detected as converted")
}
func TestIsAlreadyConverted_CBZWithNonDateComment(t *testing.T) {
tmpDir := t.TempDir()
cbzPath := filepath.Join(tmpDir, "test.cbz")
f, err := os.Create(cbzPath)
require.NoError(t, err)
w := zip.NewWriter(f)
_ = w.SetComment("this is not a date")
fw, err := w.Create("page.jpg")
require.NoError(t, err)
_, _ = fw.Write([]byte("dummy"))
_ = w.Close()
_ = f.Close()
result, err := IsAlreadyConverted(context.Background(), cbzPath)
require.NoError(t, err)
assert.False(t, result, "Expected CBZ with non-date comment to NOT be detected as converted")
}
func TestExtractChapter_NonexistentFile(t *testing.T) {
_, err := ExtractChapter(context.Background(), "/nonexistent/file.cbz")
require.Error(t, err)
}
func TestExtractChapter_PageExtensions(t *testing.T) {
chapter, err := ExtractChapter(context.Background(), "../../testdata/Chapter 128.cbz")
require.NoError(t, err)
defer func() { _ = chapter.Cleanup() }()
// All pages should have valid image extensions
validExts := map[string]bool{".jpg": true, ".jpeg": true, ".png": true, ".webp": true, ".gif": true}
for _, page := range chapter.Pages {
assert.True(t, validExts[page.Extension], "Page %d has unexpected extension: %s", page.Index, page.Extension)
}
}
func TestExtractChapter_PagesHaveSequentialIndices(t *testing.T) {
chapter, err := ExtractChapter(context.Background(), "../../testdata/Chapter 128.cbz")
require.NoError(t, err)
defer func() { _ = chapter.Cleanup() }()
for i, page := range chapter.Pages {
assert.Equal(t, uint16(i), page.Index)
}
}
func TestExtractChapter_Cleanup(t *testing.T) {
chapter, err := ExtractChapter(context.Background(), "../../testdata/Chapter 128.cbz")
require.NoError(t, err)
tempDir := chapter.TempDir
// Verify temp dir exists
assert.DirExists(t, tempDir)
// Cleanup should remove temp dir
err = chapter.Cleanup()
require.NoError(t, err)
assert.NoDirExists(t, tempDir)
}
func TestExtractChapter_CBR(t *testing.T) {
chapter, err := ExtractChapter(context.Background(), "../../testdata/Chapter 1.cbr")
require.NoError(t, err)
defer func() { _ = chapter.Cleanup() }()
assert.Len(t, chapter.Pages, 16)
// All page files should exist on disk
for _, page := range chapter.Pages {
assert.FileExists(t, page.FilePath)
}
}
func TestExtractChapter_ConvertedStatus(t *testing.T) {
chapter, err := ExtractChapter(context.Background(), "../../testdata/Chapter 10_converted.cbz")
require.NoError(t, err)
defer func() { _ = chapter.Cleanup() }()
assert.True(t, chapter.IsConverted, "Expected converted chapter to have IsConverted = true")
assert.False(t, chapter.ConvertedTime.IsZero(), "Expected non-zero ConvertedTime for converted chapter")
}
func TestWriteChapterToCBZ_EmptyChapter(t *testing.T) {
tmpDir := t.TempDir()
outputPath := filepath.Join(tmpDir, "empty.cbz")
chapter := &manga.Chapter{
FilePath: filepath.Join(tmpDir, "source.cbz"),
TempDir: tmpDir,
Pages: []*manga.PageFile{},
}
err := WriteChapterToCBZ(chapter, outputPath)
require.NoError(t, err)
// Verify the file is a valid zip
r, err := zip.OpenReader(outputPath)
require.NoError(t, err)
_ = r.Close()
}
func TestWriteChapterToCBZ_WithComicInfo(t *testing.T) {
tmpDir := t.TempDir()
outputPath := filepath.Join(tmpDir, "output.cbz")
inputDir := filepath.Join(tmpDir, "input")
_ = os.MkdirAll(inputDir, 0755)
// Create a dummy page file
pagePath := filepath.Join(inputDir, "0000.jpg")
_ = os.WriteFile(pagePath, []byte("fake image data"), 0644)
chapter := &manga.Chapter{
FilePath: filepath.Join(tmpDir, "source.cbz"),
TempDir: tmpDir,
ComicInfoXml: `<?xml version="1.0"?><ComicInfo><Series>Test</Series></ComicInfo>`,
Pages: []*manga.PageFile{
{Index: 0, Extension: ".jpg", FilePath: pagePath},
},
}
chapter.SetConverted()
err := WriteChapterToCBZ(chapter, outputPath)
require.NoError(t, err)
// Verify ComicInfo.xml is present
r, err := zip.OpenReader(outputPath)
require.NoError(t, err)
defer func() { _ = r.Close() }()
foundComicInfo := false
for _, f := range r.File {
if f.Name == "ComicInfo.xml" {
foundComicInfo = true
}
}
assert.True(t, foundComicInfo, "ComicInfo.xml not found in output CBZ")
// Verify zip comment has converted timestamp
assert.NotEmpty(t, r.Comment, "Expected zip comment with conversion timestamp")
}
func TestWriteChapterToCBZ_InvalidPagePath(t *testing.T) {
tmpDir := t.TempDir()
outputPath := filepath.Join(tmpDir, "output.cbz")
chapter := &manga.Chapter{
FilePath: filepath.Join(tmpDir, "source.cbz"),
TempDir: tmpDir,
Pages: []*manga.PageFile{
{Index: 0, Extension: ".jpg", FilePath: "/nonexistent/page.jpg"},
},
}
err := WriteChapterToCBZ(chapter, outputPath)
require.Error(t, err)
}
func TestIsAlreadyConverted_CBZWithConvertedTxt(t *testing.T) {
// Create a CBZ with converted.txt inside (no zip comment)
tmpDir := t.TempDir()
cbzPath := filepath.Join(tmpDir, "test.cbz")
f, err := os.Create(cbzPath)
require.NoError(t, err)
w := zip.NewWriter(f)
// Add converted.txt with a date
fw, err := w.Create("converted.txt")
require.NoError(t, err)
_, _ = fw.Write([]byte(time.Now().Format(time.RFC3339)))
// Add a dummy page
fw, err = w.Create("page.webp")
require.NoError(t, err)
_, _ = fw.Write([]byte("dummy"))
_ = w.Close()
_ = f.Close()
result, err := IsAlreadyConverted(context.Background(), cbzPath)
require.NoError(t, err)
assert.True(t, result, "Expected CBZ with converted.txt to be detected as converted")
}
func TestIsAlreadyConverted_CBZWithInvalidConvertedTxt(t *testing.T) {
// Create a CBZ with converted.txt that has invalid date
tmpDir := t.TempDir()
cbzPath := filepath.Join(tmpDir, "test.cbz")
f, err := os.Create(cbzPath)
require.NoError(t, err)
w := zip.NewWriter(f)
fw, err := w.Create("converted.txt")
require.NoError(t, err)
_, _ = fw.Write([]byte("not a valid date"))
_ = w.Close()
_ = f.Close()
result, err := IsAlreadyConverted(context.Background(), cbzPath)
require.NoError(t, err)
assert.False(t, result, "Expected CBZ with invalid date in converted.txt to NOT be detected as converted")
}
func TestExtractChapter_WithConvertedTxt(t *testing.T) {
// Create a CBZ with converted.txt inside
tmpDir := t.TempDir()
cbzPath := filepath.Join(tmpDir, "test.cbz")
f, err := os.Create(cbzPath)
require.NoError(t, err)
w := zip.NewWriter(f)
fw, err := w.Create("converted.txt")
require.NoError(t, err)
convertedTime := time.Date(2024, 6, 15, 12, 0, 0, 0, time.UTC)
_, _ = fw.Write([]byte(convertedTime.Format(time.RFC3339)))
fw, err = w.Create("page001.jpg")
require.NoError(t, err)
_, _ = fw.Write([]byte("fake image"))
_ = w.Close()
_ = f.Close()
chapter, err := ExtractChapter(context.Background(), cbzPath)
require.NoError(t, err)
defer func() { _ = chapter.Cleanup() }()
assert.True(t, chapter.IsConverted, "Expected chapter with converted.txt to be marked as converted")
assert.False(t, chapter.ConvertedTime.IsZero(), "Expected non-zero ConvertedTime")
}
func TestWriteChapterToCBZ_MultiplePages(t *testing.T) {
tmpDir := t.TempDir()
outputPath := filepath.Join(tmpDir, "output.cbz")
inputDir := filepath.Join(tmpDir, "input")
_ = os.MkdirAll(inputDir, 0755)
// Create multiple dummy page files
var pages []*manga.PageFile
for i := 0; i < 5; i++ {
pagePath := filepath.Join(inputDir, fmt.Sprintf("%04d.webp", i))
_ = os.WriteFile(pagePath, []byte(fmt.Sprintf("page %d content", i)), 0644)
pages = append(pages, &manga.PageFile{
Index: uint16(i),
Extension: ".webp",
FilePath: pagePath,
})
}
chapter := &manga.Chapter{
FilePath: filepath.Join(tmpDir, "source.cbz"),
TempDir: tmpDir,
Pages: pages,
}
chapter.SetConverted()
err := WriteChapterToCBZ(chapter, outputPath)
require.NoError(t, err)
// Verify archive contents
r, err := zip.OpenReader(outputPath)
require.NoError(t, err)
defer func() { _ = r.Close() }()
// Should have 5 pages = 5 files (conversion status stored in zip comment, not as file)
expectedFiles := 5
assert.Len(t, r.File, expectedFiles)
// Verify zip comment is set
assert.NotEmpty(t, r.Comment, "Expected zip comment for converted chapter")
}
func TestWriteChapterToCBZ_InvalidOutputPath(t *testing.T) {
tmpDir := t.TempDir()
inputDir := filepath.Join(tmpDir, "input")
_ = os.MkdirAll(inputDir, 0755)
pagePath := filepath.Join(inputDir, "0000.webp")
_ = os.WriteFile(pagePath, []byte("content"), 0644)
chapter := &manga.Chapter{
FilePath: filepath.Join(tmpDir, "source.cbz"),
TempDir: tmpDir,
Pages: []*manga.PageFile{
{Index: 0, Extension: ".webp", FilePath: pagePath},
},
}
err := WriteChapterToCBZ(chapter, "/nonexistent/dir/output.cbz")
require.Error(t, err)
}
+12 -14
View File
@@ -6,38 +6,36 @@ import (
"time" "time"
) )
// Chapter represents a comic book chapter with pages stored on disk.
type Chapter struct { type Chapter struct {
// FilePath is the path to the chapter's directory. // FilePath is the path to the original archive file.
FilePath string FilePath string
// Pages is a slice of pointers to Page objects. // Pages is a slice of page files on disk.
Pages []*Page Pages []*PageFile
// ComicInfo is a string containing information about the chapter. // ComicInfoXml holds the ComicInfo.xml content (small, kept in memory).
ComicInfoXml string ComicInfoXml string
// IsConverted is a boolean that indicates whether the chapter has been converted. // IsConverted indicates whether the chapter has already been converted.
IsConverted bool IsConverted bool
// ConvertedTime is a pointer to a time.Time object that indicates when the chapter was converted. Nil mean not converted. // ConvertedTime is when the chapter was converted.
ConvertedTime time.Time ConvertedTime time.Time
// TempDir, when non-empty, is a staging temp folder holding converted // TempDir is the root temp directory for this chapter's extracted/converted files.
// page contents on disk (see Page.TempFilePath) rather than fully in // Cleanup removes this entire directory.
// memory. It should be removed via Cleanup once the chapter has been
// written out (or is no longer needed).
TempDir string TempDir string
} }
// SetConverted sets the IsConverted field to true and sets the ConvertedTime field to the current time. // SetConverted marks the chapter as converted with the current timestamp.
func (chapter *Chapter) SetConverted() { func (chapter *Chapter) SetConverted() {
chapter.IsConverted = true chapter.IsConverted = true
chapter.ConvertedTime = time.Now() chapter.ConvertedTime = time.Now()
} }
// Cleanup removes the chapter's staging temp folder (if any), releasing any // Cleanup removes the chapter's temp directory and all extracted/converted files.
// page contents that were staged to disk during conversion.
func (chapter *Chapter) Cleanup() error { func (chapter *Chapter) Cleanup() error {
if chapter.TempDir == "" { if chapter.TempDir == "" {
return nil return nil
} }
if err := os.RemoveAll(chapter.TempDir); err != nil { if err := os.RemoveAll(chapter.TempDir); err != nil {
return fmt.Errorf("failed to remove staging temp folder: %w", err) return fmt.Errorf("failed to remove temp directory: %w", err)
} }
chapter.TempDir = "" chapter.TempDir = ""
return nil return nil
+102
View File
@@ -0,0 +1,102 @@
package manga
import (
"os"
"path/filepath"
"testing"
"time"
)
func TestChapter_SetConverted(t *testing.T) {
chapter := &Chapter{}
if chapter.IsConverted {
t.Error("New chapter should not be converted")
}
if !chapter.ConvertedTime.IsZero() {
t.Error("New chapter should have zero ConvertedTime")
}
before := time.Now()
chapter.SetConverted()
after := time.Now()
if !chapter.IsConverted {
t.Error("Chapter should be converted after SetConverted()")
}
if chapter.ConvertedTime.Before(before) || chapter.ConvertedTime.After(after) {
t.Error("ConvertedTime should be between before and after")
}
}
func TestChapter_Cleanup(t *testing.T) {
// Create a temp dir with some files
tmpDir := t.TempDir()
chapterDir := filepath.Join(tmpDir, "chapter-cleanup-test")
if err := os.MkdirAll(filepath.Join(chapterDir, "input"), 0755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(chapterDir, "input", "page.jpg"), []byte("test"), 0644); err != nil {
t.Fatal(err)
}
chapter := &Chapter{TempDir: chapterDir}
err := chapter.Cleanup()
if err != nil {
t.Fatalf("Cleanup failed: %v", err)
}
if chapter.TempDir != "" {
t.Error("TempDir should be empty after cleanup")
}
if _, err := os.Stat(chapterDir); !os.IsNotExist(err) {
t.Error("Directory should not exist after cleanup")
}
}
func TestChapter_Cleanup_EmptyTempDir(t *testing.T) {
chapter := &Chapter{TempDir: ""}
err := chapter.Cleanup()
if err != nil {
t.Errorf("Cleanup with empty TempDir should not error, got: %v", err)
}
}
func TestChapter_Cleanup_NonexistentDir(t *testing.T) {
chapter := &Chapter{TempDir: "/nonexistent/path/that/does/not/exist"}
// os.RemoveAll on a nonexistent path returns nil
err := chapter.Cleanup()
if err != nil {
t.Errorf("Cleanup of nonexistent dir should not error, got: %v", err)
}
}
func TestPageFile_Struct(t *testing.T) {
page := &PageFile{
Index: 5,
Extension: ".webp",
FilePath: "/tmp/test/0005.webp",
IsSplitted: true,
SplitPartIndex: 2,
}
if page.Index != 5 {
t.Errorf("Expected Index 5, got %d", page.Index)
}
if page.Extension != ".webp" {
t.Errorf("Expected Extension .webp, got %s", page.Extension)
}
if page.FilePath != "/tmp/test/0005.webp" {
t.Errorf("Expected FilePath /tmp/test/0005.webp, got %s", page.FilePath)
}
if !page.IsSplitted {
t.Error("Expected IsSplitted true")
}
if page.SplitPartIndex != 2 {
t.Errorf("Expected SplitPartIndex 2, got %d", page.SplitPartIndex)
}
}
+13 -83
View File
@@ -1,86 +1,16 @@
package manga package manga
import ( // PageFile represents a single page image stored on disk.
"bytes" // No image data is held in memory — only metadata and a file path.
"fmt" type PageFile struct {
"io" // Index of the page in the chapter (original ordering).
"os" Index uint16
// Extension of the page image file (e.g., ".webp", ".jpg").
"github.com/rs/zerolog/log" Extension string
) // FilePath is the absolute path to the image file on disk.
FilePath string
type Page struct { // IsSplitted indicates whether this page was split from a larger image.
// Index of the page in the chapter. IsSplitted bool
Index uint16 `json:"index" jsonschema:"description=Index of the page in the chapter."` // SplitPartIndex is the part index when the page was split.
// Extension of the page image. SplitPartIndex uint16
Extension string `json:"extension" jsonschema:"description=Extension of the page image."`
// Size of the page in bytes
Size uint64 `json:"-"`
// Contents of the page. Nil when the page contents have been staged to
// disk (see TempFilePath) to bound memory usage.
Contents *bytes.Buffer `json:"-"`
// TempFilePath, when non-empty, points to a file on disk (in a staging
// temp folder) holding the page contents instead of keeping them fully
// in memory. Use Open() to transparently read the page contents
// regardless of where they are stored.
TempFilePath string `json:"-"`
// IsSplitted tell us if the page was cropped to multiple pieces
IsSplitted bool `json:"is_cropped" jsonschema:"description=Was this page cropped."`
// SplitPartIndex represent the index of the crop if the page was cropped
SplitPartIndex uint16 `json:"crop_part_index" jsonschema:"description=Index of the crop if the image was cropped."`
}
// Open returns a reader for the page contents, transparently handling
// whether the contents are held in memory (Contents) or staged on disk
// (TempFilePath). The caller is responsible for closing the returned
// reader.
func (page *Page) Open() (io.ReadCloser, error) {
if page.TempFilePath != "" {
file, err := os.Open(page.TempFilePath)
if err != nil {
return nil, fmt.Errorf("failed to open staged page contents: %w", err)
}
return file, nil
}
if page.Contents == nil {
return nil, fmt.Errorf("page has no contents: neither TempFilePath nor Contents is set")
}
return io.NopCloser(bytes.NewReader(page.Contents.Bytes())), nil
}
// Stage writes the given content to a file in tempDir instead of keeping it
// in memory, updating Extension, Size and TempFilePath accordingly and
// clearing Contents. This is used after converting a page so that only the
// pages currently being processed are held in memory, bounding memory usage
// for chapters with many/large pages.
func (page *Page) Stage(tempDir string, content *bytes.Buffer, extension string) error {
file, err := os.CreateTemp(tempDir, "page-*.tmp")
if err != nil {
return fmt.Errorf("failed to create staging file: %w", err)
}
fileName := file.Name()
written, writeErr := file.Write(content.Bytes())
closeErr := file.Close()
if writeErr == nil && closeErr == nil && written == content.Len() {
page.TempFilePath = fileName
page.Extension = extension
page.Size = uint64(written)
page.Contents = nil
return nil
}
// Something went wrong: remove the incomplete/partial staging file
// rather than leaving corrupted data behind on disk.
if removeErr := os.Remove(fileName); removeErr != nil {
log.Warn().Str("file", fileName).Err(removeErr).Msg("Failed to remove incomplete staging file")
}
if writeErr != nil {
return fmt.Errorf("failed to write staging file: %w", writeErr)
}
if closeErr != nil {
return fmt.Errorf("failed to close staging file: %w", closeErr)
}
return fmt.Errorf("short write to staging file: wrote %d of %d bytes", written, content.Len())
} }
-32
View File
@@ -1,32 +0,0 @@
package manga
import (
"bytes"
"image"
)
// PageContainer is a struct that holds a manga page, its image, and the image format.
type PageContainer struct {
// Page is a pointer to a manga page object.
Page *Page
// Image is the decoded image of the manga page.
Image image.Image
// Format is a string representing the format of the image (e.g., "png", "jpeg", "webp").
Format string
// IsToBeConverted is a boolean flag indicating whether the image needs to be converted to another format.
IsToBeConverted bool
// HasBeenConverted is a boolean flag indicating whether the image has been converted to another format.
HasBeenConverted bool
}
func NewContainer(Page *Page, img image.Image, format string, isToBeConverted bool) *PageContainer {
return &PageContainer{Page: Page, Image: img, Format: format, IsToBeConverted: isToBeConverted, HasBeenConverted: false}
}
// SetConverted sets the converted image, its extension, and its size in the PageContainer.
func (pc *PageContainer) SetConverted(converted *bytes.Buffer, extension string) {
pc.Page.Contents = converted
pc.Page.Extension = extension
pc.Page.Size = uint64(converted.Len())
pc.HasBeenConverted = true
}
+2 -2
View File
@@ -46,7 +46,7 @@ func TestCapture(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
var err error = tt.initial var err = tt.initial
Capture(&err, tt.errFunc, tt.msg) Capture(&err, tt.errFunc, tt.msg)
if err != nil && err.Error() != tt.expected { if err != nil && err.Error() != tt.expected {
t.Errorf("expected %q, got %q", tt.expected, err.Error()) t.Errorf("expected %q, got %q", tt.expected, err.Error())
@@ -110,7 +110,7 @@ func TestCaptureGeneric(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
var err error = tt.initial var err = tt.initial
CaptureGeneric(&err, tt.errFunc, tt.value, tt.msg) CaptureGeneric(&err, tt.errFunc, tt.value, tt.msg)
if err != nil && err.Error() != tt.expected { if err != nil && err.Error() != tt.expected {
t.Errorf("expected %q, got %q", tt.expected, err.Error()) t.Errorf("expected %q, got %q", tt.expected, err.Error())
+59
View File
@@ -0,0 +1,59 @@
package utils
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
)
func TestIsValidFolder(t *testing.T) {
tests := []struct {
name string
setup func(t *testing.T) string
expected bool
}{
{
name: "valid directory",
setup: func(t *testing.T) string {
return t.TempDir()
},
expected: true,
},
{
name: "file not directory",
setup: func(t *testing.T) string {
dir := t.TempDir()
path := filepath.Join(dir, "file.txt")
if err := os.WriteFile(path, []byte("content"), 0644); err != nil {
t.Fatal(err)
}
return path
},
expected: false,
},
{
name: "nonexistent path",
setup: func(t *testing.T) string {
return "/nonexistent/path/that/does/not/exist"
},
expected: false,
},
{
name: "empty string",
setup: func(t *testing.T) string {
return ""
},
expected: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
path := tt.setup(t)
result := IsValidFolder(path)
assert.Equal(t, tt.expected, result)
})
}
}
+48 -67
View File
@@ -25,6 +25,12 @@ type OptimizeOptions struct {
} }
// Optimize optimizes a CBZ/CBR file using the specified converter. // Optimize optimizes a CBZ/CBR file using the specified converter.
// The new pipeline is disk-first:
// 1. Fast check if already converted (no extraction)
// 2. Extract archive to temp directory on disk
// 3. Convert pages file-to-file (no image data in memory)
// 4. Create output CBZ by streaming from disk
// 5. Cleanup temp files
func Optimize(options *OptimizeOptions) error { func Optimize(options *OptimizeOptions) error {
log.Info().Str("file", options.Path).Msg("Processing file") log.Info().Str("file", options.Path).Msg("Processing file")
log.Debug(). log.Debug().
@@ -34,43 +40,54 @@ func Optimize(options *OptimizeOptions) error {
Bool("split", options.Split). Bool("split", options.Split).
Msg("Optimization parameters") Msg("Optimization parameters")
// Load the chapter // Step 1: Fast conversion check before extracting (new requirement)
log.Debug().Str("file", options.Path).Msg("Loading chapter") alreadyConverted, err := cbz.IsAlreadyConverted(context.Background(), options.Path)
chapter, err := cbz.LoadChapter(options.Path)
if err != nil { if err != nil {
log.Error().Str("file", options.Path).Err(err).Msg("Failed to load chapter") log.Debug().Str("file", options.Path).Err(err).Msg("Conversion check failed, proceeding with extraction")
return fmt.Errorf("failed to load chapter: %v", err) }
if alreadyConverted {
log.Info().Str("file", options.Path).Msg("Chapter already converted")
return nil
} }
log.Debug().
Str("file", options.Path).
Int("pages", len(chapter.Pages)).
Bool("converted", chapter.IsConverted).
Msg("Chapter loaded successfully")
// Step 2: Extract chapter to disk
log.Debug().Str("file", options.Path).Msg("Extracting chapter")
// Create context for extraction (use timeout if configured)
var extractCtx context.Context
if options.Timeout > 0 {
var cancel context.CancelFunc
extractCtx, cancel = context.WithTimeout(context.Background(), options.Timeout)
defer cancel()
log.Debug().Str("file", options.Path).Dur("timeout", options.Timeout).Msg("Applying timeout")
} else {
extractCtx = context.Background()
}
chapter, err := cbz.ExtractChapter(extractCtx, options.Path)
if err != nil {
log.Error().Str("file", options.Path).Err(err).Msg("Failed to extract chapter")
return fmt.Errorf("failed to extract chapter: %w", err)
}
defer func() {
if cleanupErr := chapter.Cleanup(); cleanupErr != nil {
log.Warn().Str("file", options.Path).Err(cleanupErr).Msg("Failed to cleanup temp directory")
}
}()
// Double-check conversion status from extracted metadata
if chapter.IsConverted { if chapter.IsConverted {
log.Info().Str("file", options.Path).Msg("Chapter already converted") log.Info().Str("file", options.Path).Msg("Chapter already converted")
return nil return nil
} }
// Convert the chapter
log.Debug(). log.Debug().
Str("file", chapter.FilePath). Str("file", options.Path).
Int("pages", len(chapter.Pages)). Int("pages", len(chapter.Pages)).
Uint8("quality", options.Quality). Msg("Chapter extracted successfully")
Bool("split", options.Split).
Msg("Starting chapter conversion")
var ctx context.Context // Step 3: Convert pages file-to-file
if options.Timeout > 0 { convertedChapter, err := options.ChapterConverter.ConvertChapter(extractCtx, chapter, options.Quality, options.Split, func(msg string, current uint32, total uint32) {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(context.Background(), options.Timeout)
defer cancel()
log.Debug().Str("file", chapter.FilePath).Dur("timeout", options.Timeout).Msg("Applying timeout to chapter conversion")
} else {
ctx = context.Background()
}
convertedChapter, err := options.ChapterConverter.ConvertChapter(ctx, chapter, options.Quality, options.Split, func(msg string, current uint32, total uint32) {
if current%10 == 0 || current == total { if current%10 == 0 || current == total {
log.Info().Str("file", chapter.FilePath).Uint32("current", current).Uint32("total", total).Msg("Converting") log.Info().Str("file", chapter.FilePath).Uint32("current", current).Uint32("total", total).Msg("Converting")
} else { } else {
@@ -83,7 +100,7 @@ func Optimize(options *OptimizeOptions) error {
log.Debug().Str("file", chapter.FilePath).Err(err).Msg("Page conversion error (non-fatal)") log.Debug().Str("file", chapter.FilePath).Err(err).Msg("Page conversion error (non-fatal)")
} else { } else {
log.Error().Str("file", chapter.FilePath).Err(err).Msg("Chapter conversion failed") log.Error().Str("file", chapter.FilePath).Err(err).Msg("Chapter conversion failed")
return fmt.Errorf("failed to convert chapter: %v", err) return fmt.Errorf("failed to convert chapter: %w", err)
} }
} }
if convertedChapter == nil { if convertedChapter == nil {
@@ -91,82 +108,47 @@ func Optimize(options *OptimizeOptions) error {
return fmt.Errorf("failed to convert chapter") return fmt.Errorf("failed to convert chapter")
} }
// Clean up the staging temp folder (if any) used to hold converted page
// contents on disk once we're done with the chapter, regardless of
// success or failure below.
defer func() {
if cleanupErr := convertedChapter.Cleanup(); cleanupErr != nil {
log.Warn().Str("file", chapter.FilePath).Err(cleanupErr).Msg("Failed to remove staging temp folder")
}
}()
log.Debug(). log.Debug().
Str("file", chapter.FilePath). Str("file", chapter.FilePath).
Int("original_pages", len(chapter.Pages)).
Int("converted_pages", len(convertedChapter.Pages)). Int("converted_pages", len(convertedChapter.Pages)).
Msg("Chapter conversion completed") Msg("Chapter conversion completed")
convertedChapter.SetConverted() convertedChapter.SetConverted()
// Determine output path and handle CBR override logic // Step 4: Determine output path
log.Debug().
Str("input_path", options.Path).
Bool("override", options.Override).
Msg("Determining output path")
outputPath := options.Path outputPath := options.Path
originalPath := options.Path originalPath := options.Path
isCbrOverride := false isCbrOverride := false
if options.Override { if options.Override {
// For override mode, check if it's a CBR file that needs to be converted to CBZ
pathLower := strings.ToLower(options.Path) pathLower := strings.ToLower(options.Path)
if strings.HasSuffix(pathLower, ".cbr") { if strings.HasSuffix(pathLower, ".cbr") {
// Convert CBR to CBZ: change extension and mark for deletion
outputPath = strings.TrimSuffix(options.Path, filepath.Ext(options.Path)) + ".cbz" outputPath = strings.TrimSuffix(options.Path, filepath.Ext(options.Path)) + ".cbz"
isCbrOverride = true isCbrOverride = true
log.Debug().
Str("original_path", originalPath).
Str("output_path", outputPath).
Msg("CBR to CBZ conversion: will delete original after conversion")
} else {
log.Debug().
Str("original_path", originalPath).
Str("output_path", outputPath).
Msg("CBZ override mode: will overwrite original file")
} }
} else { } else {
// Handle both .cbz and .cbr files - strip the extension and add _converted.cbz
pathLower := strings.ToLower(options.Path) pathLower := strings.ToLower(options.Path)
if strings.HasSuffix(pathLower, ".cbz") { if strings.HasSuffix(pathLower, ".cbz") {
outputPath = strings.TrimSuffix(options.Path, ".cbz") + "_converted.cbz" outputPath = strings.TrimSuffix(options.Path, ".cbz") + "_converted.cbz"
} else if strings.HasSuffix(pathLower, ".cbr") { } else if strings.HasSuffix(pathLower, ".cbr") {
outputPath = strings.TrimSuffix(options.Path, ".cbr") + "_converted.cbz" outputPath = strings.TrimSuffix(options.Path, ".cbr") + "_converted.cbz"
} else { } else {
// Fallback for other extensions - just add _converted.cbz
outputPath = options.Path + "_converted.cbz" outputPath = options.Path + "_converted.cbz"
} }
log.Debug().
Str("original_path", originalPath).
Str("output_path", outputPath).
Msg("Non-override mode: creating converted file alongside original")
} }
// Write the converted chapter to CBZ file // Step 5: Write converted chapter to CBZ (streaming from disk)
log.Debug().Str("output_path", outputPath).Msg("Writing converted chapter to CBZ file") log.Debug().Str("output_path", outputPath).Msg("Writing converted chapter to CBZ file")
err = cbz.WriteChapterToCBZ(convertedChapter, outputPath) err = cbz.WriteChapterToCBZ(convertedChapter, outputPath)
if err != nil { if err != nil {
log.Error().Str("output_path", outputPath).Err(err).Msg("Failed to write converted chapter") log.Error().Str("output_path", outputPath).Err(err).Msg("Failed to write converted chapter")
return fmt.Errorf("failed to write converted chapter: %v", err) return fmt.Errorf("failed to write converted chapter: %w", err)
} }
log.Debug().Str("output_path", outputPath).Msg("Successfully wrote converted chapter")
// If we're overriding a CBR file, delete the original CBR after successful write // If overriding a CBR file, delete the original
if isCbrOverride { if isCbrOverride {
log.Debug().Str("file", originalPath).Msg("Attempting to delete original CBR file")
err = os.Remove(originalPath) err = os.Remove(originalPath)
if err != nil { if err != nil {
// Log the error but don't fail the operation since conversion succeeded
log.Warn().Str("file", originalPath).Err(err).Msg("Failed to delete original CBR file") log.Warn().Str("file", originalPath).Err(err).Msg("Failed to delete original CBR file")
} else { } else {
log.Info().Str("file", originalPath).Msg("Deleted original CBR file") log.Info().Str("file", originalPath).Msg("Deleted original CBR file")
@@ -175,5 +157,4 @@ func Optimize(options *OptimizeOptions) error {
log.Info().Str("output", outputPath).Msg("Converted file written") log.Info().Str("output", outputPath).Msg("Converted file written")
return nil return nil
} }
+2 -9
View File
@@ -51,13 +51,6 @@ func TestOptimizeIntegration(t *testing.T) {
if err != nil { if err != nil {
return err return err
} }
// Skip the "large" fixtures directory: it holds the Git LFS-tracked
// fixture used exclusively by TestOptimizeIntegration_LargeFile,
// which may only be a small LFS pointer file if the content wasn't
// fetched, and shouldn't be exercised by this generic test.
if info.IsDir() && filepath.Base(path) == "large" {
return filepath.SkipDir
}
if !info.IsDir() { if !info.IsDir() {
fileName := strings.ToLower(info.Name()) fileName := strings.ToLower(info.Name())
if (strings.HasSuffix(fileName, ".cbz") || strings.HasSuffix(fileName, ".cbr")) && !strings.Contains(fileName, "converted") { if (strings.HasSuffix(fileName, ".cbz") || strings.HasSuffix(fileName, ".cbr")) && !strings.Contains(fileName, "converted") {
@@ -213,7 +206,7 @@ func TestOptimizeIntegration(t *testing.T) {
} }
// Clean up output file // Clean up output file
os.Remove(expectedOutput) _ = os.Remove(expectedOutput)
}) })
} }
} }
@@ -394,7 +387,7 @@ func TestOptimizeIntegration_Timeout(t *testing.T) {
Quality: 85, Quality: 85,
Override: false, Override: false,
Split: false, Split: false,
Timeout: 10 * time.Millisecond, // Very short timeout to force timeout Timeout: 1 * time.Nanosecond, // Extremely short timeout to force timeout
} }
err = Optimize(options) err = Optimize(options)
@@ -1,121 +0,0 @@
package utils
import (
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"github.com/belphemur/CBZOptimizer/v2/internal/utils/errs"
"github.com/belphemur/CBZOptimizer/v2/pkg/converter"
"github.com/belphemur/CBZOptimizer/v2/pkg/converter/constant"
"github.com/rs/zerolog/log"
)
// largeTestFile is a ~1GB synthetic CBZ fixture stored via Git LFS (see
// .gitattributes). It is used to exercise the optimize pipeline with a
// chapter large enough to make in-memory-only handling impractical, and to
// validate that converted pages are streamed to/from a staging temp folder
// (see manga.Page.TempFilePath / manga.Chapter.TempDir) instead of blowing
// up memory usage.
const largeTestFile = "../../testdata/large/large_chapter.cbz"
// TestOptimizeIntegration_LargeFile is opt-in (set CBZ_RUN_LARGE_FILE_TEST=1)
// since it processes a ~1GB fixture and can take a while to run. It is
// automatically skipped if the fixture is unavailable (e.g. Git LFS content
// wasn't fetched, leaving only a pointer file) or in short mode.
func TestOptimizeIntegration_LargeFile(t *testing.T) {
if testing.Short() {
t.Skip("Skipping large file integration test in short mode")
}
if os.Getenv("CBZ_RUN_LARGE_FILE_TEST") == "" {
t.Skip("Skipping large file integration test; set CBZ_RUN_LARGE_FILE_TEST=1 to run it")
}
info, err := os.Stat(largeTestFile)
if err != nil {
t.Skipf("large test fixture not found: %v", err)
}
// If Git LFS content wasn't fetched (e.g. `actions/checkout` without
// `lfs: true`), the file on disk is just a small pointer text file
// rather than the real ~1GB fixture. Detect and skip gracefully instead
// of failing the whole suite.
const minExpectedSize = 500 * 1024 * 1024 // 500MB
if info.Size() < minExpectedSize {
t.Skipf("large test fixture looks like a Git LFS pointer (size=%d), skipping; run `git lfs pull`", info.Size())
}
tempDir, err := os.MkdirTemp("", "test_optimize_large_file")
if err != nil {
t.Fatal(err)
}
defer errs.CaptureGeneric(&err, os.RemoveAll, tempDir, "failed to remove temporary directory")
converterInstance, err := converter.Get(constant.WebP)
if err != nil {
t.Skip("WebP converter not available, skipping large file integration test")
}
if err := converterInstance.PrepareConverter(); err != nil {
t.Skip("Failed to prepare WebP converter, skipping large file integration test")
}
cbzFile := filepath.Join(tempDir, "large_chapter.cbz")
if err := copyFile(largeTestFile, cbzFile); err != nil {
t.Fatal(err)
}
var memBefore, memAfter runtime.MemStats
runtime.GC()
runtime.ReadMemStats(&memBefore)
options := &OptimizeOptions{
ChapterConverter: converterInstance,
Path: cbzFile,
Quality: 85,
Override: false,
Split: true,
}
err = Optimize(options)
if err != nil {
t.Fatalf("failed to optimize large chapter: %v", err)
}
runtime.GC()
runtime.ReadMemStats(&memAfter)
log.Info().
Uint64("heap_alloc_before", memBefore.HeapAlloc).
Uint64("heap_alloc_after", memAfter.HeapAlloc).
Int64("input_size", info.Size()).
Msg("Large file integration test memory usage")
outputFile := strings.TrimSuffix(cbzFile, ".cbz") + "_converted.cbz"
if _, err := os.Stat(outputFile); err != nil {
t.Fatalf("expected converted output file %s to exist: %v", outputFile, err)
}
}
// copyFile copies src to dst using streaming file I/O so that the whole
// file content is never held in memory at once, which matters for the
// large fixture used by this test.
func copyFile(src, dst string) (err error) {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("failed to open source file: %w", err)
}
defer errs.Capture(&err, in.Close, "failed to close source file")
out, err := os.Create(dst)
if err != nil {
return fmt.Errorf("failed to create destination file: %w", err)
}
defer errs.Capture(&err, out.Close, "failed to close destination file")
if _, err := io.Copy(out, in); err != nil {
return fmt.Errorf("failed to copy file contents: %w", err)
}
return nil
}
+1 -1
View File
@@ -276,7 +276,7 @@ func TestOptimize(t *testing.T) {
} }
// Clean up output file // Clean up output file
os.Remove(expectedOutput) _ = os.Remove(expectedOutput)
}) })
} }
} }
+14 -8
View File
@@ -11,13 +11,19 @@ import (
"github.com/samber/lo" "github.com/samber/lo"
) )
// Converter defines the interface for image format converters.
// All operations are file-to-file: no image data is held in memory in the happy path.
type Converter interface { type Converter interface {
// Format of the converter // Format returns the output format of this converter.
Format() (format constant.ConversionFormat) Format() constant.ConversionFormat
// ConvertChapter converts a manga chapter to the specified format.
// // ConvertChapter converts all pages in a chapter from their source files to
// Returns partial success where some pages are converted and some are not. // the target format. Pages are processed in parallel (bounded by CPU count).
// On success, chapter.Pages is updated with converted PageFile entries.
// Returns partial success (non-fatal errors) via errors.PageIgnoredError.
ConvertChapter(ctx context.Context, chapter *manga.Chapter, quality uint8, split bool, progress func(message string, current uint32, total uint32)) (*manga.Chapter, error) ConvertChapter(ctx context.Context, chapter *manga.Chapter, quality uint8, split bool, progress func(message string, current uint32, total uint32)) (*manga.Chapter, error)
// PrepareConverter ensures the external encoder binary is available.
PrepareConverter() error PrepareConverter() error
} }
@@ -30,8 +36,8 @@ func Available() []constant.ConversionFormat {
return lo.Keys(converters) return lo.Keys(converters)
} }
// Get returns a packer by name. // Get returns a converter by format name.
// If the packer is not available, an error is returned. // If the converter is not available, an error is returned.
var Get = getConverter var Get = getConverter
func getConverter(name constant.ConversionFormat) (Converter, error) { func getConverter(name constant.ConversionFormat) (Converter, error) {
@@ -39,7 +45,7 @@ func getConverter(name constant.ConversionFormat) (Converter, error) {
return converter, nil return converter, nil
} }
return nil, fmt.Errorf("unkown converter \"%s\", available options are %s", name, strings.Join(lo.Map(Available(), func(item constant.ConversionFormat, index int) string { return nil, fmt.Errorf("unknown converter \"%s\", available options are %s", name, strings.Join(lo.Map(Available(), func(item constant.ConversionFormat, index int) string {
return item.String() return item.String()
}), ", ")) }), ", "))
} }
+130 -167
View File
@@ -1,22 +1,21 @@
package converter package converter
import ( import (
"bytes"
"context" "context"
"fmt"
"image" "image"
"image/jpeg" "image/jpeg"
"os" "os"
"path/filepath"
"testing" "testing"
"github.com/belphemur/CBZOptimizer/v2/internal/manga" "github.com/belphemur/CBZOptimizer/v2/internal/manga"
"github.com/belphemur/CBZOptimizer/v2/internal/utils/errs"
) )
func TestConvertChapter(t *testing.T) { func TestConvertChapter(t *testing.T) {
testCases := []struct { testCases := []struct {
name string name string
genTestChapter func(path string, isSplit bool) (*manga.Chapter, []string, error) genTestChapter func(t *testing.T, dir string) (*manga.Chapter, []string)
split bool split bool
expectError bool expectError bool
}{ }{
@@ -27,7 +26,7 @@ func TestConvertChapter(t *testing.T) {
}, },
{ {
name: "Big Pages, no split", name: "Big Pages, no split",
genTestChapter: genHugePage, genTestChapter: genHugePageNoSplit,
split: false, split: false,
expectError: true, expectError: true,
}, },
@@ -47,53 +46,41 @@ func TestConvertChapter(t *testing.T) {
split: false, split: false,
expectError: true, expectError: true,
}, },
{
name: "Two corrupted pages",
genTestChapter: genTwoCorrupted,
split: false,
expectError: true,
},
} }
// Load test genTestChapter from testdata
temp, err := os.CreateTemp("", "test_chapter_*.cbz")
if err != nil {
t.Fatalf("failed to create temporary file: %v", err)
} for _, converterFormat := range Available() {
defer errs.CaptureGeneric(&err, os.Remove, temp.Name(), "failed to remove temporary file") conv, err := Get(converterFormat)
for _, converter := range Available() {
converter, err := Get(converter)
if err != nil { if err != nil {
t.Fatalf("failed to get converter: %v", err) t.Fatalf("failed to get converter: %v", err)
} }
t.Run(converter.Format().String(), func(t *testing.T) { t.Run(conv.Format().String(), func(t *testing.T) {
for _, tc := range testCases { for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
chapter, expectedExtensions, err := tc.genTestChapter(temp.Name(), tc.split) tempDir := t.TempDir()
if err != nil { chapter, expectedExtensions := tc.genTestChapter(t, tempDir)
t.Fatalf("failed to load test genTestChapter: %v", err)
}
quality := uint8(80) quality := uint8(80)
progress := func(msg string, current uint32, total uint32) { progress := func(msg string, current uint32, total uint32) {
t.Log(msg) t.Log(msg)
} }
convertedChapter, err := converter.ConvertChapter(context.Background(), chapter, quality, tc.split, progress) convertedChapter, err := conv.ConvertChapter(context.Background(), chapter, quality, tc.split, progress)
if err != nil && !tc.expectError { if err != nil && !tc.expectError {
t.Fatalf("failed to convert genTestChapter: %v", err) t.Fatalf("failed to convert chapter: %v", err)
}
if convertedChapter == nil {
t.Fatal("convertedChapter is nil")
} }
if len(convertedChapter.Pages) == 0 { if len(convertedChapter.Pages) == 0 {
t.Fatalf("no pages were converted") t.Fatal("no pages were converted")
} }
if len(convertedChapter.Pages) != len(expectedExtensions) { if len(convertedChapter.Pages) != len(expectedExtensions) {
t.Fatalf("converted chapter has %d pages but expected %d", len(convertedChapter.Pages), len(expectedExtensions)) t.Fatalf("converted chapter has %d pages but expected %d", len(convertedChapter.Pages), len(expectedExtensions))
} }
// Check each page's extension against the expected array
for i, page := range convertedChapter.Pages { for i, page := range convertedChapter.Pages {
expectedExt := expectedExtensions[i] expectedExt := expectedExtensions[i]
if page.Extension != expectedExt { if page.Extension != expectedExt {
@@ -106,176 +93,152 @@ func TestConvertChapter(t *testing.T) {
} }
} }
func genHugePage(path string, isSplit bool) (*manga.Chapter, []string, error) { // createTestImageFile creates a JPEG image file at the given path with the specified dimensions.
file, err := os.Open(path) func createTestImageFile(t *testing.T, path string, width, height int) {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
f, err := os.Create(path)
if err != nil { if err != nil {
return nil, nil, err t.Fatal(err)
} }
defer errs.Capture(&err, file.Close, "failed to close file") err = jpeg.Encode(f, img, nil)
_ = f.Close()
var pages []*manga.Page
expectedExtensions := []string{".jpg"} // One image that's generated as JPEG
if isSplit {
expectedExtensions = []string{".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp"}
}
// Create one tall page
img := image.NewRGBA(image.Rect(0, 0, 1, 17000))
buf := new(bytes.Buffer)
err = jpeg.Encode(buf, img, nil)
if err != nil { if err != nil {
return nil, nil, err t.Fatal(err)
} }
page := &manga.Page{
Index: 0,
Contents: buf,
Extension: ".jpg",
}
pages = append(pages, page)
return &manga.Chapter{
FilePath: path,
Pages: pages,
}, expectedExtensions, nil
} }
func genSmallPages(path string, isSplit bool) (*manga.Chapter, []string, error) { func genHugePage(t *testing.T, dir string) (*manga.Chapter, []string) {
file, err := os.Open(path) inputDir := filepath.Join(dir, "input")
if err != nil { if err := os.MkdirAll(inputDir, 0755); err != nil {
return nil, nil, err t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(dir, "output"), 0755); err != nil {
t.Fatal(err)
} }
defer errs.Capture(&err, file.Close, "failed to close file")
var pages []*manga.Page pagePath := filepath.Join(inputDir, "0000.jpg")
for i := 0; i < 5; i++ { // Assuming there are 5 pages for the test createTestImageFile(t, pagePath, 1, 17000)
img := image.NewRGBA(image.Rect(0, 0, 300, 1000))
buf := new(bytes.Buffer) chapter := &manga.Chapter{
err = jpeg.Encode(buf, img, nil) FilePath: filepath.Join(dir, "test.cbz"),
if err != nil { TempDir: dir,
return nil, nil, err Pages: []*manga.PageFile{
{Index: 0, Extension: ".jpg", FilePath: pagePath},
},
} }
page := &manga.Page{
// With split: 17000/2000 = 9 parts (8*2000 + 1*1000)
// Without split: page > webpMaxHeight → kept as .jpg with error
// The caller decides split=true or split=false and we return expectations
// for the split=true case (this test case is always called with split=true)
expectedExtensions := []string{".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp"}
return chapter, expectedExtensions
}
func genHugePageNoSplit(t *testing.T, dir string) (*manga.Chapter, []string) {
inputDir := filepath.Join(dir, "input")
if err := os.MkdirAll(inputDir, 0755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(dir, "output"), 0755); err != nil {
t.Fatal(err)
}
pagePath := filepath.Join(inputDir, "0000.jpg")
createTestImageFile(t, pagePath, 1, 17000)
chapter := &manga.Chapter{
FilePath: filepath.Join(dir, "test.cbz"),
TempDir: dir,
Pages: []*manga.PageFile{
{Index: 0, Extension: ".jpg", FilePath: pagePath},
},
}
// Without split: page > webpMaxHeight → kept as .jpg with error
return chapter, []string{".jpg"}
}
func genSmallPages(t *testing.T, dir string) (*manga.Chapter, []string) {
inputDir := filepath.Join(dir, "input")
_ = os.MkdirAll(inputDir, 0755)
_ = os.MkdirAll(filepath.Join(dir, "output"), 0755)
var pages []*manga.PageFile
for i := 0; i < 5; i++ {
pagePath := filepath.Join(inputDir, fmt.Sprintf("%04d.jpg", i))
createTestImageFile(t, pagePath, 300, 1000)
pages = append(pages, &manga.PageFile{
Index: uint16(i), Index: uint16(i),
Contents: buf,
Extension: ".jpg", Extension: ".jpg",
} FilePath: pagePath,
pages = append(pages, page) })
} }
return &manga.Chapter{ chapter := &manga.Chapter{
FilePath: path, FilePath: filepath.Join(dir, "test.cbz"),
TempDir: dir,
Pages: pages, Pages: pages,
}, []string{".webp", ".webp", ".webp", ".webp", ".webp"}, nil
} }
func genMixSmallBig(path string, isSplit bool) (*manga.Chapter, []string, error) { return chapter, []string{".webp", ".webp", ".webp", ".webp", ".webp"}
file, err := os.Open(path)
if err != nil {
return nil, nil, err
} }
defer errs.Capture(&err, file.Close, "failed to close file")
var pages []*manga.Page func genMixSmallBig(t *testing.T, dir string) (*manga.Chapter, []string) {
for i := 0; i < 5; i++ { // Assuming there are 5 pages for the test inputDir := filepath.Join(dir, "input")
img := image.NewRGBA(image.Rect(0, 0, 300, 1000*(i+1))) _ = os.MkdirAll(inputDir, 0755)
buf := new(bytes.Buffer) _ = os.MkdirAll(filepath.Join(dir, "output"), 0755)
err := jpeg.Encode(buf, img, nil)
if err != nil { var pages []*manga.PageFile
return nil, nil, err for i := 0; i < 5; i++ {
} pagePath := filepath.Join(inputDir, fmt.Sprintf("%04d.jpg", i))
page := &manga.Page{ createTestImageFile(t, pagePath, 300, 1000*(i+1))
pages = append(pages, &manga.PageFile{
Index: uint16(i), Index: uint16(i),
Contents: buf,
Extension: ".jpg", Extension: ".jpg",
} FilePath: pagePath,
pages = append(pages, page) })
}
expectedExtensions := []string{".webp", ".webp", ".webp", ".webp", ".webp"}
if isSplit {
expectedExtensions = []string{".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp"}
} }
return &manga.Chapter{ chapter := &manga.Chapter{
FilePath: path, FilePath: filepath.Join(dir, "test.cbz"),
TempDir: dir,
Pages: pages, Pages: pages,
}, expectedExtensions, nil
} }
func genMixSmallHuge(path string, isSplit bool) (*manga.Chapter, []string, error) { // Pages heights: 1000, 2000, 3000, 4000, 5000
file, err := os.Open(path) // With new disk-first architecture: cwebp handles all these directly
if err != nil { // (all < 16383 webp max), no splitting needed
return nil, nil, err return chapter, []string{".webp", ".webp", ".webp", ".webp", ".webp"}
} }
defer errs.Capture(&err, file.Close, "failed to close file")
var pages []*manga.Page func genMixSmallHuge(t *testing.T, dir string) (*manga.Chapter, []string) {
for i := 0; i < 10; i++ { // Assuming there are 5 pages for the test inputDir := filepath.Join(dir, "input")
img := image.NewRGBA(image.Rect(0, 0, 1, 2000*(i+1))) _ = os.MkdirAll(inputDir, 0755)
buf := new(bytes.Buffer) _ = os.MkdirAll(filepath.Join(dir, "output"), 0755)
err := jpeg.Encode(buf, img, nil)
if err != nil { var pages []*manga.PageFile
return nil, nil, err for i := 0; i < 10; i++ {
} pagePath := filepath.Join(inputDir, fmt.Sprintf("%04d.jpg", i))
page := &manga.Page{ createTestImageFile(t, pagePath, 1, 2000*(i+1))
pages = append(pages, &manga.PageFile{
Index: uint16(i), Index: uint16(i),
Contents: buf,
Extension: ".jpg", Extension: ".jpg",
} FilePath: pagePath,
pages = append(pages, page) })
} }
return &manga.Chapter{ chapter := &manga.Chapter{
FilePath: path, FilePath: filepath.Join(dir, "test.cbz"),
TempDir: dir,
Pages: pages, Pages: pages,
}, []string{".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".jpg", ".jpg"}, nil
} }
func genTwoCorrupted(path string, isSplit bool) (*manga.Chapter, []string, error) { // Heights: 2000, 4000, 6000, 8000, 10000, 12000, 14000, 16000, 18000, 20000
file, err := os.Open(path) // Without split, pages > webpMaxHeight (16383) are kept as .jpg
if err != nil { // Pages 0-7 (2000-16000): should convert to .webp
return nil, nil, err // Pages 8-9 (18000, 20000): > 16383, no split → kept as .jpg
} return chapter, []string{".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".webp", ".jpg", ".jpg"}
defer errs.Capture(&err, file.Close, "failed to close file")
var pages []*manga.Page
numPages := 5
corruptedIndices := []int{2, 4} // Pages 2 and 4 are too tall to convert without splitting
for i := 0; i < numPages; i++ {
var buf *bytes.Buffer
var ext string
isCorrupted := false
for _, ci := range corruptedIndices {
if i == ci {
isCorrupted = true
break
}
}
if isCorrupted {
buf = bytes.NewBufferString("corrupted data") // Invalid data, can't decode as image
ext = ".jpg"
} else {
img := image.NewRGBA(image.Rect(0, 0, 300, 1000))
buf = new(bytes.Buffer)
err = jpeg.Encode(buf, img, nil)
if err != nil {
return nil, nil, err
}
ext = ".jpg"
}
page := &manga.Page{
Index: uint16(i),
Contents: buf,
Extension: ext,
}
pages = append(pages, page)
}
// Expected: small pages to .webp, corrupted pages to .jpg (kept as is)
expectedExtensions := []string{".webp", ".webp", ".jpg", ".webp", ".jpg"}
// Even with split, corrupted pages can't be decoded so stay as is
return &manga.Chapter{
FilePath: path,
Pages: pages,
}, expectedExtensions, nil
} }
+223 -381
View File
@@ -1,25 +1,25 @@
package webp package webp
import ( import (
"bytes"
"context" "context"
"errors" "errors"
"fmt" "fmt"
"image" "image"
_ "image/gif" _ "image/gif"
_ "image/jpeg" _ "image/jpeg"
"image/png" _ "image/png"
"os" "os"
"path/filepath"
"runtime" "runtime"
"strings"
"sort"
"sync" "sync"
"sync/atomic" "sync/atomic"
"github.com/belphemur/CBZOptimizer/v2/internal/manga" "github.com/belphemur/CBZOptimizer/v2/internal/manga"
"github.com/belphemur/CBZOptimizer/v2/pkg/converter/constant" "github.com/belphemur/CBZOptimizer/v2/pkg/converter/constant"
converterrors "github.com/belphemur/CBZOptimizer/v2/pkg/converter/errors" converterrors "github.com/belphemur/CBZOptimizer/v2/pkg/converter/errors"
"github.com/oliamb/cutter"
"github.com/rs/zerolog/log" "github.com/rs/zerolog/log"
"golang.org/x/exp/slices"
_ "golang.org/x/image/webp" _ "golang.org/x/image/webp"
) )
@@ -29,24 +29,16 @@ type Converter struct {
maxHeight int maxHeight int
cropHeight int cropHeight int
isPrepared bool isPrepared bool
// pageWorkerGuard limits how many pages are converted (i.e. how many // pageWorkerGuard limits concurrent cwebp processes across all chapters.
// cwebp binary processes are spawned) concurrently across the whole
// application, regardless of how many chapters are being processed in
// parallel (see the --parallelism flag). Without this shared limit, the
// per-chapter worker pool (sized to runtime.NumCPU()) would be
// multiplied by the number of chapters processed concurrently, leading
// to CPU/memory oversubscription since every page conversion spawns an
// external cwebp process.
pageWorkerGuard chan struct{} pageWorkerGuard chan struct{}
} }
func (converter *Converter) Format() (format constant.ConversionFormat) { func (converter *Converter) Format() constant.ConversionFormat {
return constant.WebP return constant.WebP
} }
func New() *Converter { func New() *Converter {
return &Converter{ return &Converter{
//maxHeight: 16383 / 2,
maxHeight: 4000, maxHeight: 4000,
cropHeight: 2000, cropHeight: 2000,
isPrepared: false, isPrepared: false,
@@ -66,456 +58,306 @@ func (converter *Converter) PrepareConverter() error {
return nil return nil
} }
// ConvertChapter converts all pages in a chapter using file-to-file cwebp operations.
// In the happy path, no image data is loaded into Go memory.
// Splitting is attempted only if direct conversion fails due to dimension limits.
func (converter *Converter) ConvertChapter(ctx context.Context, chapter *manga.Chapter, quality uint8, split bool, progress func(message string, current uint32, total uint32)) (*manga.Chapter, error) { func (converter *Converter) ConvertChapter(ctx context.Context, chapter *manga.Chapter, quality uint8, split bool, progress func(message string, current uint32, total uint32)) (*manga.Chapter, error) {
log.Debug(). log.Debug().
Str("chapter", chapter.FilePath). Str("chapter", chapter.FilePath).
Int("pages", len(chapter.Pages)). Int("pages", len(chapter.Pages)).
Uint8("quality", quality). Uint8("quality", quality).
Bool("split", split). Bool("split", split).
Int("max_goroutines", runtime.NumCPU()). Msg("Starting file-to-file chapter conversion")
Msg("Starting chapter conversion")
err := converter.PrepareConverter() err := converter.PrepareConverter()
if err != nil { if err != nil {
log.Error().Str("chapter", chapter.FilePath).Err(err).Msg("Failed to prepare converter")
return nil, err return nil, err
} }
// Stage converted pages on disk in a temp folder instead of keeping them // Validate TempDir is set to prevent writing to cwd
// all in memory until the whole chapter is written out. This bounds if chapter.TempDir == "" {
// memory usage for chapters with many/large pages; the caller is return nil, fmt.Errorf("chapter TempDir is empty, cannot create output directory")
// responsible for calling chapter.Cleanup() once the chapter has been
// written to its final CBZ file.
tempDir, err := os.MkdirTemp("", "cbzoptimizer-*")
if err != nil {
log.Error().Str("chapter", chapter.FilePath).Err(err).Msg("Failed to create staging temp folder")
return nil, fmt.Errorf("failed to create staging temp folder: %w", err)
} }
chapter.TempDir = tempDir
var wgConvertedPages sync.WaitGroup // Create output directory for converted files
maxGoroutines := runtime.NumCPU() outputDir := filepath.Join(chapter.TempDir, "output")
if err := os.MkdirAll(outputDir, 0755); err != nil {
return nil, fmt.Errorf("failed to create output directory: %w", err)
}
pagesChan := make(chan *manga.PageContainer, maxGoroutines) // Check for early context cancellation
errChan := make(chan error, maxGoroutines) select {
doneChan := make(chan struct{}) case <-ctx.Done():
return nil, ctx.Err()
default:
}
var wgPages sync.WaitGroup
wgPages.Add(len(chapter.Pages))
// guard is shared across all chapters processed concurrently by this
// converter instance (see pageWorkerGuard doc comment) so that the total
// number of concurrently running cwebp processes stays bounded to
// runtime.NumCPU(), no matter how many chapters are being converted in
// parallel at the same time.
guard := converter.pageWorkerGuard guard := converter.pageWorkerGuard
pagesMutex := sync.Mutex{} var totalPages atomic.Uint32
var pages []*manga.Page totalPages.Store(uint32(len(chapter.Pages)))
var totalPages = uint32(len(chapter.Pages))
log.Debug(). type pageResult struct {
Str("chapter", chapter.FilePath). pages []*manga.PageFile
Int("total_pages", len(chapter.Pages)). err error
Int("worker_count", maxGoroutines).
Str("staging_dir", tempDir).
Msg("Initialized conversion worker pool")
// failWithCleanup removes the staging temp folder before returning a
// fatal error for which no chapter is returned to the caller (and thus
// nobody else will clean it up).
failWithCleanup := func(err error) (*manga.Chapter, error) {
if removeErr := os.RemoveAll(tempDir); removeErr != nil {
log.Warn().Str("chapter", chapter.FilePath).Err(removeErr).Msg("Failed to remove staging temp folder after error")
}
chapter.TempDir = ""
return nil, err
} }
// Check if context is already cancelled results := make([]pageResult, len(chapter.Pages))
select { var wg sync.WaitGroup
case <-ctx.Done(): var convertedCount atomic.Uint32
log.Warn().Str("chapter", chapter.FilePath).Msg("Chapter conversion cancelled due to timeout")
return failWithCleanup(ctx.Err()) for i, page := range chapter.Pages {
default: wg.Add(1)
}
go func(idx int, p *manga.PageFile) {
// Start the worker pool defer wg.Done()
go func() {
defer close(doneChan) // Check context before acquiring worker slot
for page := range pagesChan {
select { select {
case <-ctx.Done(): case <-ctx.Done():
results[idx] = pageResult{err: ctx.Err()}
return return
case guard <- struct{}{}: // would block if guard channel is already filled case guard <- struct{}{}:
} }
defer func() { <-guard }()
go func(pageToConvert *manga.PageContainer) { // Check context after acquiring worker slot
defer func() {
wgConvertedPages.Done()
<-guard
}()
// Check context cancellation before processing
select { select {
case <-ctx.Done(): case <-ctx.Done():
results[idx] = pageResult{err: ctx.Err()}
return return
default: default:
} }
convertedPage, err := converter.convertPage(pageToConvert, quality, tempDir) pages, err := converter.convertPageFile(ctx, p, outputDir, quality, split)
if err != nil { results[idx] = pageResult{pages: pages, err: err}
if convertedPage == nil {
select {
case errChan <- err:
case <-ctx.Done():
return
}
return
}
buffer := new(bytes.Buffer)
err := png.Encode(buffer, convertedPage.Image)
if err != nil {
select {
case errChan <- err:
case <-ctx.Done():
return
}
return
}
if err := convertedPage.Page.Stage(tempDir, buffer, ".png"); err != nil {
select {
case errChan <- err:
case <-ctx.Done():
return
}
return
}
}
pagesMutex.Lock()
defer pagesMutex.Unlock()
pages = append(pages, convertedPage.Page)
currentTotalPages := atomic.LoadUint32(&totalPages)
progress(fmt.Sprintf("Converted %d/%d pages to %s format", len(pages), currentTotalPages, converter.Format()), uint32(len(pages)), currentTotalPages)
}(page)
}
}()
// Process pages current := convertedCount.Add(1)
for _, page := range chapter.Pages { total := totalPages.Load()
select { progress(fmt.Sprintf("Converted %d/%d pages to %s format", current, total, converter.Format()), current, total)
case <-ctx.Done(): }(i, page)
log.Warn().Str("chapter", chapter.FilePath).Msg("Chapter conversion cancelled due to timeout")
return failWithCleanup(ctx.Err())
default:
} }
go func(page *manga.Page) { // Wait for completion or context cancellation
defer wgPages.Done()
splitNeeded, img, format, err := converter.checkPageNeedsSplit(page, split)
if err != nil {
var pageIgnoredError *converterrors.PageIgnoredError
if errors.As(err, &pageIgnoredError) {
log.Info().Err(err).Msg("Page ignored due to image decode error")
}
select {
case errChan <- err:
case <-ctx.Done():
return
}
wgConvertedPages.Add(1)
select {
case pagesChan <- manga.NewContainer(page, img, format, false):
case <-ctx.Done():
wgConvertedPages.Done()
return
}
return
}
if !splitNeeded {
wgConvertedPages.Add(1)
select {
case pagesChan <- manga.NewContainer(page, img, format, true):
case <-ctx.Done():
wgConvertedPages.Done()
return
}
return
}
images, err := converter.cropImage(img)
if err != nil {
select {
case errChan <- err:
case <-ctx.Done():
return
}
return
}
atomic.AddUint32(&totalPages, uint32(len(images)-1))
for i, img := range images {
select {
case <-ctx.Done():
return
default:
}
newPage := &manga.Page{
Index: page.Index,
IsSplitted: true,
SplitPartIndex: uint16(i),
}
wgConvertedPages.Add(1)
select {
case pagesChan <- manga.NewContainer(newPage, img, "N/A", true):
case <-ctx.Done():
wgConvertedPages.Done()
return
}
}
}(page)
}
wgPages.Wait()
close(pagesChan)
// Wait for all conversions to complete or context cancellation
done := make(chan struct{}) done := make(chan struct{})
go func() { go func() {
defer close(done) wg.Wait()
wgConvertedPages.Wait() close(done)
}() }()
select { select {
case <-done: case <-done:
// Conversion completed successfully
case <-ctx.Done(): case <-ctx.Done():
log.Warn().Str("chapter", chapter.FilePath).Msg("Chapter conversion cancelled due to timeout") // Wait for in-flight goroutines to finish
return failWithCleanup(ctx.Err()) <-done
return nil, ctx.Err()
} }
close(errChan) // Collect results — separate fatal errors from page-ignored errors
// Note: guard is the converter's shared pageWorkerGuard and must not be var convertedPages []*manga.PageFile
// closed here since it is reused by subsequent (and possibly concurrent) var fatalErrors []error
// ConvertChapter calls. var ignoredErrors []error
var errList []error for _, result := range results {
for err := range errChan { if result.err != nil {
errList = append(errList, err) if errors.Is(result.err, context.DeadlineExceeded) || errors.Is(result.err, context.Canceled) {
return nil, result.err
} }
var pageIgnored *converterrors.PageIgnoredError
var aggregatedError error = nil if errors.As(result.err, &pageIgnored) {
if len(errList) > 0 { ignoredErrors = append(ignoredErrors, result.err)
aggregatedError = errors.Join(errList...)
log.Debug().
Str("chapter", chapter.FilePath).
Int("error_count", len(errList)).
Err(errors.Join(errList...)).
Msg("Conversion completed with errors")
} else { } else {
log.Debug(). fatalErrors = append(fatalErrors, result.err)
Str("chapter", chapter.FilePath). }
Int("pages_converted", len(pages)). }
Msg("Conversion completed successfully") if result.pages != nil {
convertedPages = append(convertedPages, result.pages...)
}
} }
slices.SortFunc(pages, func(a, b *manga.Page) int { // Fatal errors take priority over ignored-page errors
if len(fatalErrors) > 0 {
return nil, errors.Join(fatalErrors...)
}
if len(convertedPages) == 0 {
if len(ignoredErrors) > 0 {
return nil, errors.Join(ignoredErrors...)
}
return nil, fmt.Errorf("no pages were converted")
}
// Sort pages by index and split part
sort.Slice(convertedPages, func(i, j int) bool {
a, b := convertedPages[i], convertedPages[j]
if a.Index == b.Index { if a.Index == b.Index {
return int(a.SplitPartIndex) - int(b.SplitPartIndex) return a.SplitPartIndex < b.SplitPartIndex
} }
return int(a.Index) - int(b.Index) return a.Index < b.Index
}) })
chapter.Pages = pages
chapter.Pages = convertedPages
var aggregatedError error
if len(ignoredErrors) > 0 {
aggregatedError = errors.Join(ignoredErrors...)
}
log.Debug(). log.Debug().
Str("chapter", chapter.FilePath). Str("chapter", chapter.FilePath).
Int("final_page_count", len(pages)). Int("converted_pages", len(convertedPages)).
Msg("Pages sorted and chapter updated") Msg("Chapter conversion completed")
runtime.GC()
log.Debug().Str("chapter", chapter.FilePath).Msg("Garbage collection completed")
return chapter, aggregatedError return chapter, aggregatedError
} }
func (converter *Converter) cropImage(img image.Image) ([]image.Image, error) { // convertPageFile converts a single page file to WebP format.
bounds := img.Bounds() // Returns the converted page(s) — multiple if splitting was needed.
height := bounds.Dy() func (converter *Converter) convertPageFile(ctx context.Context, page *manga.PageFile, outputDir string, quality uint8, split bool) ([]*manga.PageFile, error) {
width := bounds.Dx() log.Debug().
Uint16("page_index", page.Index).
Str("input", page.FilePath).
Msg("Converting page file")
// If the page is already WebP, just return it as-is
if strings.ToLower(page.Extension) == ".webp" {
log.Debug().Uint16("page_index", page.Index).Msg("Page already WebP, skipping")
return []*manga.PageFile{page}, nil
}
// Try direct file-to-file conversion first (happy path — no memory allocation)
outputPath := filepath.Join(outputDir, fmt.Sprintf("%04d.webp", page.Index))
err := EncodeFile(page.FilePath, outputPath, uint(quality))
if err == nil {
// Success! No image decoding needed.
return []*manga.PageFile{{
Index: page.Index,
Extension: ".webp",
FilePath: outputPath,
}}, nil
}
// Direct conversion failed. Check if it's a dimension issue.
log.Debug().
Uint16("page_index", page.Index).
Err(err).
Msg("Direct conversion failed, checking dimensions")
// Read just the image header to get dimensions (no full decode)
width, height, decodeErr := getImageDimensions(page.FilePath)
if decodeErr != nil {
// Can't even read the image header — keep the original file
log.Info().
Uint16("page_index", page.Index).
Err(decodeErr).
Msg("Cannot decode image, keeping original")
return []*manga.PageFile{page}, converterrors.NewPageIgnored(
fmt.Sprintf("page %d: failed to decode image (%s)", page.Index, decodeErr.Error()))
}
log.Debug().
Uint16("page_index", page.Index).
Int("width", width).
Int("height", height).
Msg("Image dimensions read")
// If height exceeds WebP max and split is not enabled, keep original
if height >= webpMaxHeight && !split {
log.Info().
Uint16("page_index", page.Index).
Int("height", height).
Msg("Page too tall for WebP, keeping original")
return []*manga.PageFile{page}, converterrors.NewPageIgnored(
fmt.Sprintf("page %d is too tall [max: %dpx] to be converted to webp format", page.Index, webpMaxHeight))
}
// If height exceeds our split threshold and split is enabled, use cwebp -crop
if height >= converter.maxHeight && split {
return converter.splitAndConvert(ctx, page, outputDir, quality, width, height)
}
// Height is within limits but conversion still failed for another reason.
// Keep the original file.
log.Warn().
Uint16("page_index", page.Index).
Err(err).
Msg("Conversion failed for non-dimension reason, keeping original")
return []*manga.PageFile{page}, converterrors.NewPageIgnored(
fmt.Sprintf("page %d: conversion failed (%s)", page.Index, err.Error()))
}
// splitAndConvert splits a tall image into multiple parts using cwebp -crop
// and converts each part. No Go-side image decode is needed.
func (converter *Converter) splitAndConvert(ctx context.Context, page *manga.PageFile, outputDir string, quality uint8, width, height int) ([]*manga.PageFile, error) {
log.Debug().
Uint16("page_index", page.Index).
Int("width", width).
Int("height", height).
Int("crop_height", converter.cropHeight).
Msg("Splitting and converting page using cwebp -crop")
numParts := height / converter.cropHeight numParts := height / converter.cropHeight
if height%converter.cropHeight != 0 { if height%converter.cropHeight != 0 {
numParts++ numParts++
} }
log.Debug(). var pages []*manga.PageFile
Int("original_width", width).
Int("original_height", height).
Int("crop_height", converter.cropHeight).
Int("num_parts", numParts).
Msg("Starting image cropping for page splitting")
parts := make([]image.Image, numParts)
for i := 0; i < numParts; i++ { for i := 0; i < numParts; i++ {
select {
case <-ctx.Done():
return pages, ctx.Err()
default:
}
yOffset := i * converter.cropHeight
partHeight := converter.cropHeight partHeight := converter.cropHeight
if i == numParts-1 { if i == numParts-1 {
partHeight = height - i*converter.cropHeight partHeight = height - yOffset
} }
log.Debug(). outputPath := filepath.Join(outputDir, fmt.Sprintf("%04d-%02d.webp", page.Index, i))
Int("part_index", i). err := EncodeFileWithCrop(page.FilePath, outputPath, uint(quality), 0, yOffset, width, partHeight)
Int("part_height", partHeight).
Int("y_offset", i*converter.cropHeight).
Msg("Cropping image part")
part, err := cutter.Crop(img, cutter.Config{ if err != nil {
Width: bounds.Dx(), log.Error().
Height: partHeight, Uint16("page_index", page.Index).
Anchor: image.Point{Y: i * converter.cropHeight}, Int("part", i).
Mode: cutter.TopLeft, Err(err).
Msg("Failed to convert split part")
return nil, fmt.Errorf("failed to convert split part %d of page %d: %w", i, page.Index, err)
}
pages = append(pages, &manga.PageFile{
Index: page.Index,
Extension: ".webp",
FilePath: outputPath,
IsSplitted: true,
SplitPartIndex: uint16(i),
}) })
if err != nil {
log.Error().
Int("part_index", i).
Err(err).
Msg("Failed to crop image part")
return nil, fmt.Errorf("error cropping part %d: %v", i+1, err)
} }
parts[i] = part
log.Debug().
Int("part_index", i).
Int("cropped_width", part.Bounds().Dx()).
Int("cropped_height", part.Bounds().Dy()).
Msg("Image part cropped successfully")
}
log.Debug().
Int("total_parts", len(parts)).
Msg("Image cropping completed")
return parts, nil
}
func (converter *Converter) checkPageNeedsSplit(page *manga.Page, splitRequested bool) (bool, image.Image, string, error) {
log.Debug().
Uint16("page_index", page.Index).
Bool("split_requested", splitRequested).
Int("page_size", len(page.Contents.Bytes())).
Msg("Analyzing page for splitting")
reader := bytes.NewBuffer(page.Contents.Bytes())
img, format, err := image.Decode(reader)
if err != nil {
log.Debug().Uint16("page_index", page.Index).Err(err).Msg("Failed to decode page image")
return false, nil, format, converterrors.NewPageIgnored(fmt.Sprintf("page %d: failed to decode image (%s)", page.Index, err.Error()))
}
bounds := img.Bounds()
height := bounds.Dy()
width := bounds.Dx()
log.Debug(). log.Debug().
Uint16("page_index", page.Index). Uint16("page_index", page.Index).
Int("width", width). Int("parts", len(pages)).
Int("height", height). Msg("Split conversion completed")
Str("format", format).
Int("max_height", converter.maxHeight).
Int("webp_max_height", webpMaxHeight).
Msg("Page dimensions analyzed")
if height >= webpMaxHeight && !splitRequested { return pages, nil
log.Debug().
Uint16("page_index", page.Index).
Int("height", height).
Int("webp_max", webpMaxHeight).
Msg("Page too tall for WebP format, would be ignored")
return false, img, format, converterrors.NewPageIgnored(fmt.Sprintf("page %d is too tall [max: %dpx] to be converted to webp format", page.Index, webpMaxHeight))
} }
needsSplit := height >= converter.maxHeight && splitRequested // getImageDimensions reads only the image header to determine dimensions.
log.Debug(). // This is much cheaper than a full image.Decode — only a few bytes are read.
Uint16("page_index", page.Index). func getImageDimensions(filePath string) (width, height int, err error) {
Bool("needs_split", needsSplit). f, err := os.Open(filePath)
Msg("Page splitting decision made")
return needsSplit, img, format, nil
}
func (converter *Converter) convertPage(container *manga.PageContainer, quality uint8, tempDir string) (*manga.PageContainer, error) {
log.Debug().
Uint16("page_index", container.Page.Index).
Str("format", container.Format).
Bool("to_be_converted", container.IsToBeConverted).
Uint8("quality", quality).
Msg("Converting page")
// Fix WebP format detection (case insensitive)
if container.Format == "webp" || container.Format == "WEBP" {
log.Debug().
Uint16("page_index", container.Page.Index).
Msg("Page already in WebP format, skipping conversion")
container.Page.Extension = ".webp"
return container, nil
}
if !container.IsToBeConverted {
log.Debug().
Uint16("page_index", container.Page.Index).
Msg("Page marked as not to be converted, skipping")
return container, nil
}
log.Debug().
Uint16("page_index", container.Page.Index).
Uint8("quality", quality).
Msg("Encoding page to WebP format")
converted, err := converter.convert(container.Image, uint(quality))
if err != nil { if err != nil {
log.Error(). return 0, 0, err
Uint16("page_index", container.Page.Index).
Err(err).
Msg("Failed to convert page to WebP")
return nil, err
} }
defer func() { _ = f.Close() }()
originalSize := container.Page.Size config, _, err := image.DecodeConfig(f)
convertedSize := converted.Len()
if err := container.Page.Stage(tempDir, converted, ".webp"); err != nil {
log.Error().
Uint16("page_index", container.Page.Index).
Err(err).
Msg("Failed to stage converted page to disk")
return nil, err
}
container.HasBeenConverted = true
log.Debug().
Uint16("page_index", container.Page.Index).
Uint64("original_size", originalSize).
Int("converted_size", convertedSize).
Msg("Page conversion completed")
return container, nil
}
// convert converts an image to the WebP format. It decodes the image from the input buffer,
// encodes it as a WebP file using the webp.Encode() function, and returns the resulting WebP
// file as a bytes.Buffer.
func (converter *Converter) convert(image image.Image, quality uint) (*bytes.Buffer, error) {
var buf bytes.Buffer
err := Encode(&buf, image, quality)
if err != nil { if err != nil {
return nil, err return 0, 0, err
} }
return &buf, nil return config.Width, config.Height, nil
} }
+290 -409
View File
@@ -1,138 +1,61 @@
package webp package webp
import ( import (
"bytes"
"context" "context"
"fmt" "fmt"
"image" "image"
"image/color"
"image/gif"
"image/jpeg" "image/jpeg"
"image/png" "os"
"io" "path/filepath"
"sync" "sync"
"testing" "testing"
"time" "time"
_ "golang.org/x/image/webp"
"github.com/belphemur/CBZOptimizer/v2/internal/manga" "github.com/belphemur/CBZOptimizer/v2/internal/manga"
"github.com/belphemur/CBZOptimizer/v2/pkg/converter/constant" "github.com/belphemur/CBZOptimizer/v2/pkg/converter/constant"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
func createTestImage(width, height int, format string) (image.Image, error) { func createTestImageFile(t *testing.T, path string, width, height int) {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height)) img := image.NewRGBA(image.Rect(0, 0, width, height))
f, err := os.Create(path)
require.NoError(t, err)
err = jpeg.Encode(f, img, nil)
require.NoError(t, err)
_ = f.Close()
}
// Create a gradient pattern to ensure we have actual image data func createTestChapter(t *testing.T, pages []struct{ w, h int }) (*manga.Chapter, string) {
for y := 0; y < height; y++ { t.Helper()
for x := 0; x < width; x++ { dir := t.TempDir()
img.Set(x, y, color.RGBA{ inputDir := filepath.Join(dir, "input")
R: uint8((x * 255) / width), require.NoError(t, os.MkdirAll(inputDir, 0755))
G: uint8((y * 255) / height), require.NoError(t, os.MkdirAll(filepath.Join(dir, "output"), 0755))
B: 100,
A: 255, chapter := &manga.Chapter{
FilePath: filepath.Join(dir, "test.cbz"),
TempDir: dir,
}
for i, p := range pages {
pagePath := filepath.Join(inputDir, fmt.Sprintf("%04d.jpg", i))
createTestImageFile(t, pagePath, p.w, p.h)
chapter.Pages = append(chapter.Pages, &manga.PageFile{
Index: uint16(i),
Extension: ".jpg",
FilePath: pagePath,
}) })
} }
}
return img, nil return chapter, dir
} }
func encodeImage(img image.Image, format string) (*bytes.Buffer, string, error) {
buf := new(bytes.Buffer)
switch format {
case "jpeg", "jpg":
if err := jpeg.Encode(buf, img, &jpeg.Options{Quality: 85}); err != nil {
return nil, "", err
}
return buf, ".jpg", nil
case "gif":
if err := gif.Encode(buf, img, nil); err != nil {
return nil, "", err
}
return buf, ".gif", nil
case "webp":
if err := PrepareEncoder(); err != nil {
return nil, "", err
}
if err := Encode(buf, img, 80); err != nil {
return nil, "", err
}
return buf, ".webp", nil
case "png":
fallthrough
default:
if err := png.Encode(buf, img); err != nil {
return nil, "", err
}
return buf, ".png", nil
}
}
func createTestPage(t *testing.T, index int, width, height int, format string) *manga.Page {
img, err := createTestImage(width, height, format)
require.NoError(t, err)
buf, ext, err := encodeImage(img, format)
require.NoError(t, err)
return &manga.Page{
Index: uint16(index),
Contents: buf,
Extension: ext,
Size: uint64(buf.Len()),
}
}
func validateConvertedImage(t *testing.T, page *manga.Page) {
require.True(t, page.Contents != nil || page.TempFilePath != "", "page should have contents in memory or staged on disk")
require.Greater(t, page.Size, uint64(0))
reader, err := page.Open()
require.NoError(t, err)
defer reader.Close()
data, err := io.ReadAll(reader)
require.NoError(t, err)
require.Greater(t, len(data), 0)
// Try to decode the image
img, format, err := image.Decode(bytes.NewReader(data))
require.NoError(t, err, "Failed to decode converted image")
if page.Extension == ".webp" {
assert.Equal(t, "webp", format, "Expected WebP format")
}
require.NotNil(t, img)
bounds := img.Bounds()
assert.Greater(t, bounds.Dx(), 0, "Image width should be positive")
assert.Greater(t, bounds.Dy(), 0, "Image height should be positive")
}
// TestConverter_ConvertChapter tests the ConvertChapter method of the WebP converter.
// It verifies various scenarios including:
// - Converting single normal images
// - Converting multiple normal images
// - Converting tall images with split enabled
// - Handling tall images that exceed maximum height
//
// For each test case it validates:
// - Proper error handling
// - Expected number of output pages
// - Correct page ordering
// - Split page handling and indexing
// - Progress callback behavior
//
// The test uses different image dimensions and split settings to ensure
// the converter handles all cases correctly while maintaining proper
// progress reporting and page ordering.
func TestConverter_ConvertChapter(t *testing.T) { func TestConverter_ConvertChapter(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
pages []*manga.Page pages []struct{ w, h int }
split bool split bool
expectSplit bool expectSplit bool
expectError bool expectError bool
@@ -140,47 +63,33 @@ func TestConverter_ConvertChapter(t *testing.T) {
}{ }{
{ {
name: "Single normal image", name: "Single normal image",
pages: []*manga.Page{createTestPage(t, 1, 800, 1200, "jpeg")}, pages: []struct{ w, h int }{{800, 1200}},
split: false, split: false,
expectSplit: false,
numExpected: 1, numExpected: 1,
}, },
{ {
name: "Multiple normal images", name: "Multiple normal images",
pages: []*manga.Page{ pages: []struct{ w, h int }{
createTestPage(t, 1, 800, 1200, "png"), {800, 1200},
createTestPage(t, 2, 800, 1200, "jpeg"), {800, 1200},
createTestPage(t, 3, 800, 1200, "gif"), {800, 1200},
}, },
split: false, split: false,
expectSplit: false,
numExpected: 3, numExpected: 3,
}, },
{
name: "Multiple normal images with webp",
pages: []*manga.Page{
createTestPage(t, 1, 800, 1200, "png"),
createTestPage(t, 2, 800, 1200, "jpeg"),
createTestPage(t, 3, 800, 1200, "gif"),
createTestPage(t, 4, 800, 1200, "webp"),
},
split: false,
expectSplit: false,
numExpected: 4,
},
{ {
name: "Tall image with split enabled", name: "Tall image with split enabled",
pages: []*manga.Page{createTestPage(t, 1, 800, 5000, "jpeg")}, pages: []struct{ w, h int }{{800, 5000}},
split: true, split: true,
expectSplit: true, expectSplit: false, // cwebp handles 5000px fine (< 16383 webp max), no split needed
numExpected: 3, // Based on cropHeight of 2000 numExpected: 1,
}, },
{ {
name: "Tall image without split", name: "Tall image without split",
pages: []*manga.Page{createTestPage(t, 1, 800, webpMaxHeight+100, "png")}, pages: []struct{ w, h int }{{800, webpMaxHeight + 100}},
split: false, split: false,
expectError: true, expectError: true,
numExpected: 1, numExpected: 1, // kept as-is
}, },
} }
@@ -190,18 +99,16 @@ func TestConverter_ConvertChapter(t *testing.T) {
err := converter.PrepareConverter() err := converter.PrepareConverter()
require.NoError(t, err) require.NoError(t, err)
chapter := &manga.Chapter{ chapter, _ := createTestChapter(t, tt.pages)
Pages: tt.pages,
}
var progressMutex sync.Mutex var progressMutex sync.Mutex
var lastProgress uint32 var lastProgress uint32
progress := func(message string, current uint32, total uint32) { progress := func(message string, current uint32, total uint32) {
progressMutex.Lock() progressMutex.Lock()
defer progressMutex.Unlock() defer progressMutex.Unlock()
assert.GreaterOrEqual(t, current, lastProgress, "Progress should never decrease") assert.GreaterOrEqual(t, current, lastProgress)
lastProgress = current lastProgress = current
assert.LessOrEqual(t, current, total, "Current progress should not exceed total") assert.LessOrEqual(t, current, total)
} }
convertedChapter, err := converter.ConvertChapter(context.Background(), chapter, 80, tt.split, progress) convertedChapter, err := converter.ConvertChapter(context.Background(), chapter, 80, tt.split, progress)
@@ -218,22 +125,14 @@ func TestConverter_ConvertChapter(t *testing.T) {
require.NotNil(t, convertedChapter) require.NotNil(t, convertedChapter)
assert.Len(t, convertedChapter.Pages, tt.numExpected) assert.Len(t, convertedChapter.Pages, tt.numExpected)
// Validate all converted images
for _, page := range convertedChapter.Pages {
validateConvertedImage(t, page)
}
// Verify page order // Verify page order
for i := 1; i < len(convertedChapter.Pages); i++ { for i := 1; i < len(convertedChapter.Pages); i++ {
prevPage := convertedChapter.Pages[i-1] prev := convertedChapter.Pages[i-1]
currPage := convertedChapter.Pages[i] curr := convertedChapter.Pages[i]
if prev.Index == curr.Index {
if prevPage.Index == currPage.Index { assert.Less(t, prev.SplitPartIndex, curr.SplitPartIndex)
assert.Less(t, prevPage.SplitPartIndex, currPage.SplitPartIndex,
"Split parts should be in ascending order for page %d", prevPage.Index)
} else { } else {
assert.Less(t, prevPage.Index, currPage.Index, assert.Less(t, prev.Index, curr.Index)
"Pages should be in ascending order")
} }
} }
@@ -245,149 +144,13 @@ func TestConverter_ConvertChapter(t *testing.T) {
break break
} }
} }
assert.True(t, splitFound, "Expected to find at least one split page") assert.True(t, splitFound, "Expected split pages")
}
})
}
} }
func TestConverter_convertPage(t *testing.T) { // Verify all output files exist
converter := New() for _, page := range convertedChapter.Pages {
err := converter.PrepareConverter() assert.FileExists(t, page.FilePath)
require.NoError(t, err)
tests := []struct {
name string
format string
isToBeConverted bool
expectWebP bool
expectError bool
}{
{
name: "Convert PNG to WebP",
format: "png",
isToBeConverted: true,
expectWebP: true,
expectError: false,
},
{
name: "Convert GIF to WebP",
format: "gif",
isToBeConverted: true,
expectWebP: true,
expectError: false,
},
{
name: "Already WebP",
format: "webp",
isToBeConverted: true,
expectWebP: true,
expectError: false,
},
{
name: "Skip conversion",
format: "png",
isToBeConverted: false,
expectWebP: false,
expectError: false,
},
} }
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
page := createTestPage(t, 1, 100, 100, tt.format)
img, err := createTestImage(100, 100, tt.format)
require.NoError(t, err)
container := manga.NewContainer(page, img, tt.format, tt.isToBeConverted)
converted, err := converter.convertPage(container, 80, t.TempDir())
if tt.expectError {
assert.Error(t, err)
assert.Nil(t, converted)
} else {
require.NoError(t, err)
assert.NotNil(t, converted)
if tt.expectWebP {
assert.Equal(t, ".webp", converted.Page.Extension)
validateConvertedImage(t, converted.Page)
} else {
assert.NotEqual(t, ".webp", converted.Page.Extension)
}
}
})
}
}
func TestConverter_convertPage_EncodingError(t *testing.T) {
converter := New()
err := converter.PrepareConverter()
require.NoError(t, err)
// Create a test case with nil image to test encoding error path
// when isToBeConverted is true but the image is nil, simulating a failure in the encoding step
corruptedPage := &manga.Page{
Index: 1,
Contents: &bytes.Buffer{}, // Empty buffer
Extension: ".png",
Size: 0,
}
container := manga.NewContainer(corruptedPage, nil, "png", true)
converted, err := converter.convertPage(container, 80, t.TempDir())
// This should return nil container and error because encoding will fail with nil image
assert.Error(t, err)
assert.Nil(t, converted)
}
func TestConverter_checkPageNeedsSplit(t *testing.T) {
converter := New()
tests := []struct {
name string
imageHeight int
split bool
expectSplit bool
expectError bool
}{
{
name: "Normal height",
imageHeight: 1000,
split: true,
expectSplit: false,
},
{
name: "Height exceeds max with split enabled",
imageHeight: 5000,
split: true,
expectSplit: true,
},
{
name: "Height exceeds webp max without split",
imageHeight: webpMaxHeight + 100,
split: false,
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
page := createTestPage(t, 1, 800, tt.imageHeight, "jpeg")
needsSplit, img, format, err := converter.checkPageNeedsSplit(page, tt.split)
if tt.expectError {
assert.Error(t, err)
return
}
require.NoError(t, err)
assert.NotNil(t, img)
assert.NotEmpty(t, format)
assert.Equal(t, tt.expectSplit, needsSplit)
}) })
} }
} }
@@ -402,73 +165,43 @@ func TestConverter_ConvertChapter_Timeout(t *testing.T) {
err := converter.PrepareConverter() err := converter.PrepareConverter()
require.NoError(t, err) require.NoError(t, err)
// Create a test chapter with a few pages chapter, _ := createTestChapter(t, []struct{ w, h int }{
pages := []*manga.Page{ {800, 1200},
createTestPage(t, 1, 800, 1200, "jpeg"), {800, 1200},
createTestPage(t, 2, 800, 1200, "png"), {800, 1200},
createTestPage(t, 3, 800, 1200, "gif"), })
}
chapter := &manga.Chapter{ progress := func(message string, current uint32, total uint32) {}
FilePath: "/test/chapter.cbz",
Pages: pages,
}
var progressMutex sync.Mutex ctx, cancel := context.WithTimeout(context.Background(), 1*time.Nanosecond)
var lastProgress uint32
progress := func(message string, current uint32, total uint32) {
progressMutex.Lock()
defer progressMutex.Unlock()
assert.GreaterOrEqual(t, current, lastProgress, "Progress should never decrease")
lastProgress = current
assert.LessOrEqual(t, current, total, "Current progress should not exceed total")
}
// Test with very short timeout (1 nanosecond)
ctx, cancel := context.WithTimeout(context.Background(), 1)
defer cancel() defer cancel()
convertedChapter, err := converter.ConvertChapter(ctx, chapter, 80, false, progress) convertedChapter, err := converter.ConvertChapter(ctx, chapter, 80, false, progress)
// Should return context error due to timeout
assert.Error(t, err) assert.Error(t, err)
assert.Nil(t, convertedChapter) assert.Nil(t, convertedChapter)
assert.Equal(t, context.DeadlineExceeded, err) assert.Equal(t, context.DeadlineExceeded, err)
} }
// TestConverter_ConvertChapter_ManyPages_NoDeadlock tests that converting chapters with many pages
// does not cause a deadlock. This test reproduces the scenario where processing
// many files with context cancellation could cause "all goroutines are asleep - deadlock!" error.
// The fix ensures that wgConvertedPages.Done() is called when context is cancelled after Add(1).
func TestConverter_ConvertChapter_ManyPages_NoDeadlock(t *testing.T) { func TestConverter_ConvertChapter_ManyPages_NoDeadlock(t *testing.T) {
converter := New() converter := New()
err := converter.PrepareConverter() err := converter.PrepareConverter()
require.NoError(t, err) require.NoError(t, err)
// Create a chapter with many pages to increase the chance of hitting the race condition pages := make([]struct{ w, h int }, 50)
numPages := 50 for i := range pages {
pages := make([]*manga.Page, numPages) pages[i] = struct{ w, h int }{100, 100}
for i := 0; i < numPages; i++ {
pages[i] = createTestPage(t, i+1, 100, 100, "jpeg")
} }
chapter := &manga.Chapter{ chapter, _ := createTestChapter(t, pages)
FilePath: "/test/chapter_many_pages.cbz",
Pages: pages,
}
progress := func(message string, current uint32, total uint32) { progress := func(message string, current uint32, total uint32) {}
// No-op progress callback
}
// Run multiple iterations to increase the chance of hitting the race condition
for iteration := 0; iteration < 10; iteration++ { for iteration := 0; iteration < 10; iteration++ {
t.Run(fmt.Sprintf("iteration_%d", iteration), func(t *testing.T) { t.Run(fmt.Sprintf("iteration_%d", iteration), func(t *testing.T) {
// Use a very short timeout to trigger context cancellation during processing
ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond) ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
defer cancel() defer cancel()
// This should NOT deadlock - it should return quickly with context error
done := make(chan struct{}) done := make(chan struct{})
var convertErr error var convertErr error
go func() { go func() {
@@ -476,121 +209,57 @@ func TestConverter_ConvertChapter_ManyPages_NoDeadlock(t *testing.T) {
_, convertErr = converter.ConvertChapter(ctx, chapter, 80, false, progress) _, convertErr = converter.ConvertChapter(ctx, chapter, 80, false, progress)
}() }()
// Wait with a reasonable timeout - if it takes longer than 5 seconds, we have a deadlock
select { select {
case <-done: case <-done:
// Expected - conversion should complete (with error) quickly assert.Error(t, convertErr)
assert.Error(t, convertErr, "Expected context error")
case <-time.After(5 * time.Second): case <-time.After(5 * time.Second):
t.Fatal("Deadlock detected: ConvertChapter did not return within 5 seconds") t.Fatal("Deadlock detected")
} }
}) })
} }
} }
// TestConverter_ConvertChapter_ManyPages_WithSplit_NoDeadlock tests that converting chapters
// with many pages and split enabled does not cause a deadlock.
func TestConverter_ConvertChapter_ManyPages_WithSplit_NoDeadlock(t *testing.T) {
converter := New()
err := converter.PrepareConverter()
require.NoError(t, err)
// Create pages with varying heights, some requiring splits
numPages := 30
pages := make([]*manga.Page, numPages)
for i := 0; i < numPages; i++ {
height := 1000 // Normal height
if i%5 == 0 {
height = 5000 // Tall image that will be split
}
pages[i] = createTestPage(t, i+1, 100, height, "png")
}
chapter := &manga.Chapter{
FilePath: "/test/chapter_split_test.cbz",
Pages: pages,
}
progress := func(message string, current uint32, total uint32) {
// No-op progress callback
}
// Run multiple iterations with short timeouts
for iteration := 0; iteration < 10; iteration++ {
t.Run(fmt.Sprintf("iteration_%d", iteration), func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
defer cancel()
done := make(chan struct{})
var convertErr error
go func() {
defer close(done)
_, convertErr = converter.ConvertChapter(ctx, chapter, 80, true, progress) // split=true
}()
select {
case <-done:
assert.Error(t, convertErr, "Expected context error")
case <-time.After(5 * time.Second):
t.Fatal("Deadlock detected: ConvertChapter with split did not return within 5 seconds")
}
})
}
}
// TestConverter_ConvertChapter_ConcurrentChapters_NoDeadlock simulates the scenario from the
// original bug report where multiple chapters are processed in parallel with parallelism > 1.
// This test ensures no deadlock occurs when multiple goroutines are converting chapters concurrently.
func TestConverter_ConvertChapter_ConcurrentChapters_NoDeadlock(t *testing.T) { func TestConverter_ConvertChapter_ConcurrentChapters_NoDeadlock(t *testing.T) {
converter := New() converter := New()
err := converter.PrepareConverter() err := converter.PrepareConverter()
require.NoError(t, err) require.NoError(t, err)
// Create multiple chapters, each with many pages
numChapters := 20 numChapters := 20
pagesPerChapter := 30
chapters := make([]*manga.Chapter, numChapters) chapters := make([]*manga.Chapter, numChapters)
pages := make([]struct{ w, h int }, 30)
for i := range pages {
pages[i] = struct{ w, h int }{100, 100}
}
for c := 0; c < numChapters; c++ { for c := 0; c < numChapters; c++ {
pages := make([]*manga.Page, pagesPerChapter) chapters[c], _ = createTestChapter(t, pages)
for i := 0; i < pagesPerChapter; i++ {
pages[i] = createTestPage(t, i+1, 100, 100, "jpeg")
}
chapters[c] = &manga.Chapter{
FilePath: fmt.Sprintf("/test/chapter_%d.cbz", c+1),
Pages: pages,
}
} }
progress := func(message string, current uint32, total uint32) {} progress := func(message string, current uint32, total uint32) {}
// Process chapters concurrently with short timeouts (simulating parallelism flag)
parallelism := 4 parallelism := 4
var wg sync.WaitGroup var wg sync.WaitGroup
semaphore := make(chan struct{}, parallelism) semaphore := make(chan struct{}, parallelism)
// Overall test timeout
testCtx, testCancel := context.WithTimeout(context.Background(), 30*time.Second) testCtx, testCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer testCancel() defer testCancel()
for _, chapter := range chapters { for _, chapter := range chapters {
wg.Add(1) wg.Add(1)
semaphore <- struct{}{} // Acquire semaphore <- struct{}{}
go func(ch *manga.Chapter) { go func(ch *manga.Chapter) {
defer wg.Done() defer wg.Done()
defer func() { <-semaphore }() // Release defer func() { <-semaphore }()
// Use very short timeout to trigger cancellation
ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond) ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
defer cancel() defer cancel()
// This should not deadlock
_, _ = converter.ConvertChapter(ctx, ch, 80, false, progress) _, _ = converter.ConvertChapter(ctx, ch, 80, false, progress)
}(chapter) }(chapter)
} }
// Wait for all conversions with a timeout
done := make(chan struct{}) done := make(chan struct{})
go func() { go func() {
wg.Wait() wg.Wait()
@@ -599,8 +268,220 @@ func TestConverter_ConvertChapter_ConcurrentChapters_NoDeadlock(t *testing.T) {
select { select {
case <-done: case <-done:
// All goroutines completed successfully
case <-testCtx.Done(): case <-testCtx.Done():
t.Fatal("Deadlock detected: Concurrent chapter conversions did not complete within 30 seconds") t.Fatal("Deadlock detected")
} }
} }
func TestConverter_SplitAndConvert(t *testing.T) {
converter := New()
err := converter.PrepareConverter()
require.NoError(t, err)
// Create a very tall image that exceeds webpMaxHeight
dir := t.TempDir()
inputDir := filepath.Join(dir, "input")
require.NoError(t, os.MkdirAll(inputDir, 0755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "output"), 0755))
// Create image taller than webpMaxHeight (16383)
tallImagePath := filepath.Join(inputDir, "0000.jpg")
createTestImageFile(t, tallImagePath, 800, webpMaxHeight+500)
chapter := &manga.Chapter{
FilePath: filepath.Join(dir, "test.cbz"),
TempDir: dir,
Pages: []*manga.PageFile{
{Index: 0, Extension: ".jpg", FilePath: tallImagePath},
},
}
progress := func(message string, current uint32, total uint32) {}
// With split=true, the oversized page should be split
convertedChapter, err := converter.ConvertChapter(context.Background(), chapter, 80, true, progress)
require.NoError(t, err)
require.NotNil(t, convertedChapter)
// Should have more pages due to splitting
assert.Greater(t, len(convertedChapter.Pages), 1, "Tall image should be split into multiple parts")
// All parts should be split and have webp extension
for _, page := range convertedChapter.Pages {
assert.True(t, page.IsSplitted, "All pages should be marked as split")
assert.Equal(t, ".webp", page.Extension)
assert.FileExists(t, page.FilePath)
assert.Equal(t, uint16(0), page.Index, "All split parts should have same original index")
}
// Verify sequential split part indices
for i, page := range convertedChapter.Pages {
assert.Equal(t, uint16(i), page.SplitPartIndex)
}
}
func TestConverter_OversizedImageNoSplit(t *testing.T) {
converter := New()
err := converter.PrepareConverter()
require.NoError(t, err)
dir := t.TempDir()
inputDir := filepath.Join(dir, "input")
require.NoError(t, os.MkdirAll(inputDir, 0755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "output"), 0755))
tallImagePath := filepath.Join(inputDir, "0000.jpg")
createTestImageFile(t, tallImagePath, 800, webpMaxHeight+500)
chapter := &manga.Chapter{
FilePath: filepath.Join(dir, "test.cbz"),
TempDir: dir,
Pages: []*manga.PageFile{
{Index: 0, Extension: ".jpg", FilePath: tallImagePath},
},
}
progress := func(message string, current uint32, total uint32) {}
// With split=false, the oversized page should be kept as-is with error
convertedChapter, err := converter.ConvertChapter(context.Background(), chapter, 80, false, progress)
assert.Error(t, err, "Should return error for oversized page without split")
if convertedChapter != nil {
// The page should be kept in original format
for _, page := range convertedChapter.Pages {
assert.Equal(t, ".jpg", page.Extension, "Page should keep original extension")
}
}
}
func TestGetImageDimensions(t *testing.T) {
dir := t.TempDir()
tests := []struct {
name string
width, height int
expectW, expectH int
}{
{"small image", 100, 200, 100, 200},
{"wide image", 1920, 1080, 1920, 1080},
{"tall image", 800, 5000, 800, 5000},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
path := filepath.Join(dir, tt.name+".jpg")
createTestImageFile(t, path, tt.width, tt.height)
w, h, err := getImageDimensions(path)
require.NoError(t, err)
assert.Equal(t, tt.expectW, w)
assert.Equal(t, tt.expectH, h)
})
}
}
func TestGetImageDimensions_NonexistentFile(t *testing.T) {
_, _, err := getImageDimensions("/nonexistent/file.jpg")
assert.Error(t, err)
}
func TestGetImageDimensions_InvalidFile(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "invalid.jpg")
require.NoError(t, os.WriteFile(path, []byte("not an image"), 0644))
_, _, err := getImageDimensions(path)
assert.Error(t, err)
}
func TestConverter_ConvertChapter_EmptyChapter(t *testing.T) {
converter := New()
err := converter.PrepareConverter()
require.NoError(t, err)
dir := t.TempDir()
chapter := &manga.Chapter{
FilePath: filepath.Join(dir, "test.cbz"),
TempDir: dir,
Pages: []*manga.PageFile{},
}
progress := func(message string, current uint32, total uint32) {}
_, err = converter.ConvertChapter(context.Background(), chapter, 80, false, progress)
assert.Error(t, err, "Should error on empty chapter")
}
func TestConverter_ConvertChapter_PreservesComicInfo(t *testing.T) {
converter := New()
err := converter.PrepareConverter()
require.NoError(t, err)
chapter, _ := createTestChapter(t, []struct{ w, h int }{{400, 600}})
chapter.ComicInfoXml = `<?xml version="1.0"?><ComicInfo><Series>Test</Series></ComicInfo>`
progress := func(message string, current uint32, total uint32) {}
convertedChapter, err := converter.ConvertChapter(context.Background(), chapter, 80, false, progress)
require.NoError(t, err)
require.NotNil(t, convertedChapter)
assert.Equal(t, chapter.ComicInfoXml, convertedChapter.ComicInfoXml)
}
func TestConverter_ConvertChapter_OutputInCorrectDir(t *testing.T) {
converter := New()
err := converter.PrepareConverter()
require.NoError(t, err)
chapter, dir := createTestChapter(t, []struct{ w, h int }{{400, 600}})
progress := func(message string, current uint32, total uint32) {}
convertedChapter, err := converter.ConvertChapter(context.Background(), chapter, 80, false, progress)
require.NoError(t, err)
require.NotNil(t, convertedChapter)
// All output files should be in the output directory
expectedOutputDir := filepath.Join(dir, "output")
for _, page := range convertedChapter.Pages {
pageDir := filepath.Dir(page.FilePath)
assert.Equal(t, expectedOutputDir, pageDir, "Output file should be in output directory")
}
}
func TestEncodeFile(t *testing.T) {
err := PrepareEncoder()
require.NoError(t, err)
dir := t.TempDir()
inputPath := filepath.Join(dir, "input.jpg")
outputPath := filepath.Join(dir, "output.webp")
createTestImageFile(t, inputPath, 200, 300)
err = EncodeFile(inputPath, outputPath, 80)
require.NoError(t, err)
// Verify output exists and is non-empty
info, err := os.Stat(outputPath)
require.NoError(t, err)
assert.Greater(t, info.Size(), int64(0))
}
func TestEncodeFileWithCrop(t *testing.T) {
err := PrepareEncoder()
require.NoError(t, err)
dir := t.TempDir()
inputPath := filepath.Join(dir, "input.jpg")
outputPath := filepath.Join(dir, "output.webp")
createTestImageFile(t, inputPath, 200, 600)
err = EncodeFileWithCrop(inputPath, outputPath, 80, 0, 0, 200, 300)
require.NoError(t, err)
info, err := os.Stat(outputPath)
require.NoError(t, err)
assert.Greater(t, info.Size(), int64(0))
}
+16 -5
View File
@@ -2,8 +2,6 @@ package webp
import ( import (
"fmt" "fmt"
"image"
"io"
"strings" "strings"
"sync" "sync"
@@ -37,10 +35,23 @@ func PrepareEncoder() error {
return nil return nil
} }
func Encode(w io.Writer, m image.Image, quality uint) error { // EncodeFile converts an image file directly to WebP using cwebp.
// This is a zero-copy operation: no image data is loaded into Go memory.
func EncodeFile(inputPath string, outputPath string, quality uint) error {
return webpbin.NewCWebP(config). return webpbin.NewCWebP(config).
Quality(quality). Quality(quality).
InputImage(m). InputFile(inputPath).
Output(w). OutputFile(outputPath).
Run()
}
// EncodeFileWithCrop converts a cropped region of an image file to WebP.
// Uses cwebp's native -crop flag — no Go-side image decode needed.
func EncodeFileWithCrop(inputPath string, outputPath string, quality uint, x, y, width, height int) error {
return webpbin.NewCWebP(config).
Quality(quality).
Crop(x, y, width, height).
InputFile(inputPath).
OutputFile(outputPath).
Run() Run()
} }
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8880c8909521c3716a339a39a29a4bf1114921a8b4651ff8ca04da7ce3a54b6a
size 1001668690