Makefile: have "make pot" not "reset --hard"

Before commit fc0fd5b23b (Makefile: help gettext tools to cope with our custom PRItime format, 2017-07-20), we'd consider source files as-is with gettext, but because we need to understand PRItime in the same way that gettext itself understands PRIuMAX, we'd first check if we had a clean checkout, then munge all of the processed files in-place with "sed", generate "po/git.pot", and then finally "reset --hard" to undo our changes. By generating "pot" snippets in ".build/pot/po" for each source file and rewriting certain source files with PRItime macros to temporary files in ".build/pot/po", we can avoid running "make pot" by altering files in place and doing a "reset --hard" afterwards. This speed of "make pot" is slower than before on an initial run, because we run "xgettext" many times (once per source file), but it can be boosted by parallelization. It is *much* faster for incremental runs, and will allow us to implement related targets in subsequent commits. When the "pot" target was originally added in cd5513a7168 (i18n: Makefile: "pot" target to extract messages marked for translation, 2011-02-22) it behaved like a "normal" target. I.e. we'd skip the re-generation of the po/git.pot if nothing had to be done. Then after po/git.pot was checked in in dce37b66fb0 (l10n: initial git.pot for 1.7.10 upcoming release, 2012-02-13) the target was broken until 1f31963e921 (i18n: treat "make pot" as an explicitly-invoked target, 2014-08-22) when it was made to depend on "FORCE". I.e. the Makefile's dependency resolution inherently can't handle incremental building when the target file may be updated by git (or something else external to "make"). But this case no longer applies, so FORCE is no longer needed. That out of the way, the main logic change here is getting rid of the "reset --hard": We'll generate intermediate ".build/pot/po/%.po" files from "%", which is handy to see at a glance what strings (if any) in a given file are marked for translation: $ make .build/pot/po/pretty.c.po [...] $ cat .build/pot/po/pretty.c.po #: pretty.c:1051 msgid "unable to parse --pretty format" msgstr "" $ For these C source files which contain the PRItime macros, we will create temporary munged "*.c" files in a tree in ".build/pot/po" corresponding to our source tree, and have "xgettext" consider those. The rule needs to be careful to "(cd .build/pot/po && ...)", because otherwise the comments in the po/git.pot file wouldn't refer to the correct source locations (they'd be prefixed with ".build/pot/po"). These temporary munged "*.c” files will be removed immediately after the corresponding po files are generated, because some development tools cannot ignore the duplicate source files in the ".build" directory according to the ".gitignore" file, and that may cause trouble. The output of the generated po/git.pot file is changed in one minor way: Because we're using msgcat(1) instead of xgettext(1) to concatenate the output we'll now disambiguate where "TRANSLATORS" comments come from, in cases where a message is the same in N files, and either only one has a "TRANSLATORS" comment, or they're different. E.g. for the "Your edited hunk[...]" message we'll now apply this change (comment content elided): +#. #-#-#-#-# add-patch.c.po #-#-#-#-# #. TRANSLATORS: do not translate [y/n] [...] +#. #-#-#-#-# git-add--interactive.perl.po #-#-#-#-# #. TRANSLATORS: do not translate [y/n] [...] #: add-patch.c:1253 git-add--interactive.perl:1244 msgid "" "Your edited hunk does not apply. Edit again (saying \"no\" discards!) [y/n]? " msgstr "" There are six such changes, and they all make the context more understandable, as msgcat(1) is better at handling these edge cases than xgettext(1)'s previously used "--join-existing" flag. But filenames in the above disambiguation lines of extracted-comments have an extra ".po" extension compared to the filenames at the file locations. While we could rename the intermediate ".build/pot/po/%.po" files without the ".po" extension to use more intuitive filenames in the disambiguation lines of extracted-comments, but that will confuse developer tools with lots of invalid C or other source files in ".build/pot/po" directory. The addition of "--omit-header" option for xgettext makes the "pot" snippets in ".build/pot/po/*.po" smaller. But as we'll see in a subsequent commit this header behavior has been hiding an encoding-related bug from us, so let's carry it forward instead of re-generating it with xgettext(1). The "po/git.pot" file should have a header entry, because a proper header entry will increase the speed of creating a new po file using msginit and set a proper "POT-Creation-Date:" field in the header entry of a "po/XX.po" file. We use xgettext to generate a separate header file at ".build/pot/git.header" from "/dev/null", and use this header to assemble "po/git.pot". Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Ævar Arnfjörð Bjarmason committed May 26, 2022 at 22:50 UTC 1cc0425a27c6de542a9f09e8c5451079e4a8c6ef
2 files changed +60 -21
.gitignore
+1
@@ -200,6 +200,7 @@
200 *.[aos]
201 *.o.json
202 *.py[co]
203 +.build/
204 .depend/
205 *.gcda
206 *.gcno
Makefile
+59 -21
@@ -569,6 +569,7 @@ INSTALL = install
569 TCL_PATH = tclsh
570 TCLTK_PATH = wish
571 XGETTEXT = xgettext
572 +MSGCAT = msgcat
573 MSGFMT = msgfmt
574 CURL_CONFIG = curl-config
575 GCOV = gcov
@@ -855,6 +856,7 @@ SOURCES_CMD = ( \
856 -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
857 -o \( -name contrib -type d -prune \) \
858 -o \( -name build -type d -prune \) \
859 + -o \( -name .build -type d -prune \) \
860 -o \( -name 'trash*' -type d -prune \) \
861 -o \( -name '*.[hcS]' -type f -print \) \
862 -o \( -name '*.sh' -type f -print \) \
@@ -2706,6 +2708,7 @@ XGETTEXT_FLAGS = \
2708 --force-po \
2709 --add-comments=TRANSLATORS: \
2710 --msgid-bugs-address="Git Mailing List <git@vger.kernel.org>" \
2711 + --package-name=Git \
2712 --from-code=UTF-8
2713 XGETTEXT_FLAGS_C = $(XGETTEXT_FLAGS) --language=C \
2714 --keyword=_ --keyword=N_ --keyword="Q_:1,2"
@@ -2724,34 +2727,68 @@ LOCALIZED_SH += t/t0200/test.sh
2727 LOCALIZED_PERL += t/t0200/test.perl
2728 endif
2729
2727 -## Note that this is meant to be run only by the localization coordinator
2728 -## under a very controlled condition, i.e. (1) it is to be run in a
2729 -## Git repository (not a tarball extract), (2) any local modifications
2730 -## will be lost.
2730 +## We generate intermediate .build/pot/po/%.po files containing a
2731 +## extract of the translations we find in each file in the source
2732 +## tree. We will assemble them using msgcat to create the final
2733 +## "po/git.pot" file.
2734 +LOCALIZED_ALL_GEN_PO =
2735 +
2736 +LOCALIZED_C_GEN_PO = $(LOCALIZED_C:%=.build/pot/po/%.po)
2737 +LOCALIZED_ALL_GEN_PO += $(LOCALIZED_C_GEN_PO)
2738 +
2739 +LOCALIZED_SH_GEN_PO = $(LOCALIZED_SH:%=.build/pot/po/%.po)
2740 +LOCALIZED_ALL_GEN_PO += $(LOCALIZED_SH_GEN_PO)
2741 +
2742 +LOCALIZED_PERL_GEN_PO = $(LOCALIZED_PERL:%=.build/pot/po/%.po)
2743 +LOCALIZED_ALL_GEN_PO += $(LOCALIZED_PERL_GEN_PO)
2744 +
2745 ## Gettext tools cannot work with our own custom PRItime type, so
2746 ## we replace PRItime with PRIuMAX. We need to update this to
2747 ## PRIdMAX if we switch to a signed type later.
2748 +$(LOCALIZED_C_GEN_PO): .build/pot/po/%.po: %
2749 + $(call mkdir_p_parent_template)
2750 + $(QUIET_XGETTEXT) \
2751 + if grep -q PRItime $<; then \
2752 + (\
2753 + sed -e 's|PRItime|PRIuMAX|g' <$< \
2754 + >.build/pot/po/$< && \
2755 + cd .build/pot/po && \
2756 + $(XGETTEXT) --omit-header \
2757 + -o $(@:.build/pot/po/%=%) \
2758 + $(XGETTEXT_FLAGS_C) $< && \
2759 + rm $<; \
2760 + ); \
2761 + else \
2762 + $(XGETTEXT) --omit-header \
2763 + -o $@ $(XGETTEXT_FLAGS_C) $<; \
2764 + fi
2765
2735 -po/git.pot: $(GENERATED_H) FORCE
2736 - # All modifications will be reverted at the end, so we do not
2737 - # want to have any local change.
2738 - git diff --quiet HEAD && git diff --quiet --cached
2766 +$(LOCALIZED_SH_GEN_PO): .build/pot/po/%.po: %
2767 + $(call mkdir_p_parent_template)
2768 + $(QUIET_XGETTEXT)$(XGETTEXT) --omit-header \
2769 + -o$@ $(XGETTEXT_FLAGS_SH) $<
2770
2740 - @for s in $(LOCALIZED_C) $(LOCALIZED_SH) $(LOCALIZED_PERL); \
2741 - do \
2742 - sed -e 's|PRItime|PRIuMAX|g' <"$$s" >"$$s+" && \
2743 - cat "$$s+" >"$$s" && rm "$$s+"; \
2744 - done
2771 +$(LOCALIZED_PERL_GEN_PO): .build/pot/po/%.po: %
2772 + $(call mkdir_p_parent_template)
2773 + $(QUIET_XGETTEXT)$(XGETTEXT) --omit-header \
2774 + -o$@ $(XGETTEXT_FLAGS_PERL) $<
2775 +
2776 +define gen_pot_header
2777 +$(XGETTEXT) $(XGETTEXT_FLAGS_C) \
2778 + -o - /dev/null | \
2779 +sed -e 's|charset=CHARSET|charset=UTF-8|' \
2780 + -e 's|\(Last-Translator: \)FULL NAME <.*>|\1make by the Makefile|' \
2781 + -e 's|\(Language-Team: \)LANGUAGE <.*>|\1Git Mailing List <git@vger.kernel.org>|' \
2782 + >$@ && \
2783 +echo '"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\\n"' >>$@
2784 +endef
2785
2746 - $(QUIET_XGETTEXT)$(XGETTEXT) -o$@+ $(XGETTEXT_FLAGS_C) $(LOCALIZED_C)
2747 - $(QUIET_XGETTEXT)$(XGETTEXT) -o$@+ --join-existing $(XGETTEXT_FLAGS_SH) \
2748 - $(LOCALIZED_SH)
2749 - $(QUIET_XGETTEXT)$(XGETTEXT) -o$@+ --join-existing $(XGETTEXT_FLAGS_PERL) \
2750 - $(LOCALIZED_PERL)
2786 +.build/pot/git.header: $(LOCALIZED_ALL_GEN_PO)
2787 + $(call mkdir_p_parent_template)
2788 + $(QUIET_GEN)$(gen_pot_header)
2789
2752 - # Reverting the munged source, leaving only the updated $@
2753 - git reset --hard
2754 - mv $@+ $@
2790 +po/git.pot: .build/pot/git.header $(LOCALIZED_ALL_GEN_PO) FORCE
2791 + $(QUIET_GEN)$(MSGCAT) $(filter-out FORCE,$^) >$@
2792
2793 .PHONY: pot
2794 pot: po/git.pot
@@ -3290,6 +3327,7 @@ cocciclean:
3327 $(RM) contrib/coccinelle/*.cocci.patch*
3328
3329 clean: profile-clean coverage-clean cocciclean
3330 + $(RM) -r .build
3331 $(RM) *.res
3332 $(RM) $(OBJECTS)
3333 $(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)