glibc/scripts/vcs_to_changelog/vcstocl_quirks.py

65 lines
2.5 KiB
Python
Raw Normal View History

Script to generate ChangeLog-like output from git log Co-authored-by: Gabriel F. T. Gomes <gabriel@inconstante.net.br> Reviewed-by: Gabriel F. T. Gomes <gabriel@inconstante.net.br> Reviewed-by: Joseph Myers <joseph@codesourcery.com> The utility of a ChangeLog file has been discussed in various mailing list threads and GNU Tools Cauldrons in the past years and the general consensus is that while the file may have been very useful in the past when revision control did not exist or was not as powerful as it is today, it's current utility is fast diminishing. Further, the ChangeLog format gets in the way of modernisation of processes since it almost always results in rewriting of a commit, thus preventing use of any code review tools to automatically manage patches in the glibc project. There is consensus in the glibc community that documentation of why a change was done (i.e. a detailed description in a git commit) is more useful than what changed (i.e. a ChangeLog entry) since the latter can be deduced from the patch. The GNU community would however like to keep the option of ascertaining what changed through a ChangeLog-like output and as a compromise, it was proposed that a script be developed that generates this output. The script below is the result of these discussions. This script takes two git revisions references as input and generates the git log between those revisions in a form that resembles a ChangeLog. Its capabilities and limitations are listed in a comment in the script. On a high level it is capable of parsing C code and telling what changed at the top level, but not within constructs such as functions. Design ------ At a high level, the script analyses the raw output of a VCS, parses the source files that have changed and attempts to determine what changed. The script driver needs three distinct components to be fully functional for a repository: - A vcstocl_quirks.py file that helps it parse weird patterns in sources that may result from preprocessor defines. - A VCS plugin backend; the git backend is implemented for glibc - A programming language parser plugin. C is currently implemented. Additional programming language parsers can be added to give more detailed output for changes in those types of files. For input in languages other than those that have a parser, the script only identifies if a file has been added, removed, modified, permissions changed, etc. but cannot understand the change in content. The C Parser ------------ The C parser is capable of parsing C programs with preprocessor macros in place, as if they were part of the language. This presents some challenges with parsing code that expands macros on the fly and to help work around that, a vcstocl_quirks.py file has transformations to ease things. The C parser currently can identify macro definitions and scopes and all global and static declarations and definitions. It cannot parse (and compare) changes inside functions yet, it could be a future enhancement if the need for it arises. Testing ------- The script has been tested with the glibc repository up to glibc-2.29 and also in the past with emacs. While it would be ideal to have something like this in a repository like gnulib, that should not be a bottleneck for glibc to start using this, so this patch proposes to add these scripts into glibc. And here is (hopefully!) one of the last ChangeLog entries we'd have to write for glibc: * scripts/gitlog_to_changelog.py: New script to auto-generate ChangeLog. * scripts/vcs_to_changelog/frontend_c.py: New file. * scripts/vcs_to_changelog/misc_util.py: New file. * scripts/vcs_to_changelog/vcs_git.py: New file. * scripts/vcs_to_changelog/vcstocl_quirks.py: Likewise.
2019-09-20 18:36:55 +00:00
#!/usr/bin/python3
# VCSToChangeLog Quirks for the GNU C Library.
# Copyright (C) 2019-2020 Free Software Foundation, Inc.
Script to generate ChangeLog-like output from git log Co-authored-by: Gabriel F. T. Gomes <gabriel@inconstante.net.br> Reviewed-by: Gabriel F. T. Gomes <gabriel@inconstante.net.br> Reviewed-by: Joseph Myers <joseph@codesourcery.com> The utility of a ChangeLog file has been discussed in various mailing list threads and GNU Tools Cauldrons in the past years and the general consensus is that while the file may have been very useful in the past when revision control did not exist or was not as powerful as it is today, it's current utility is fast diminishing. Further, the ChangeLog format gets in the way of modernisation of processes since it almost always results in rewriting of a commit, thus preventing use of any code review tools to automatically manage patches in the glibc project. There is consensus in the glibc community that documentation of why a change was done (i.e. a detailed description in a git commit) is more useful than what changed (i.e. a ChangeLog entry) since the latter can be deduced from the patch. The GNU community would however like to keep the option of ascertaining what changed through a ChangeLog-like output and as a compromise, it was proposed that a script be developed that generates this output. The script below is the result of these discussions. This script takes two git revisions references as input and generates the git log between those revisions in a form that resembles a ChangeLog. Its capabilities and limitations are listed in a comment in the script. On a high level it is capable of parsing C code and telling what changed at the top level, but not within constructs such as functions. Design ------ At a high level, the script analyses the raw output of a VCS, parses the source files that have changed and attempts to determine what changed. The script driver needs three distinct components to be fully functional for a repository: - A vcstocl_quirks.py file that helps it parse weird patterns in sources that may result from preprocessor defines. - A VCS plugin backend; the git backend is implemented for glibc - A programming language parser plugin. C is currently implemented. Additional programming language parsers can be added to give more detailed output for changes in those types of files. For input in languages other than those that have a parser, the script only identifies if a file has been added, removed, modified, permissions changed, etc. but cannot understand the change in content. The C Parser ------------ The C parser is capable of parsing C programs with preprocessor macros in place, as if they were part of the language. This presents some challenges with parsing code that expands macros on the fly and to help work around that, a vcstocl_quirks.py file has transformations to ease things. The C parser currently can identify macro definitions and scopes and all global and static declarations and definitions. It cannot parse (and compare) changes inside functions yet, it could be a future enhancement if the need for it arises. Testing ------- The script has been tested with the glibc repository up to glibc-2.29 and also in the past with emacs. While it would be ideal to have something like this in a repository like gnulib, that should not be a bottleneck for glibc to start using this, so this patch proposes to add these scripts into glibc. And here is (hopefully!) one of the last ChangeLog entries we'd have to write for glibc: * scripts/gitlog_to_changelog.py: New script to auto-generate ChangeLog. * scripts/vcs_to_changelog/frontend_c.py: New file. * scripts/vcs_to_changelog/misc_util.py: New file. * scripts/vcs_to_changelog/vcs_git.py: New file. * scripts/vcs_to_changelog/vcstocl_quirks.py: Likewise.
2019-09-20 18:36:55 +00:00
#
# The GNU C Library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# The GNU C Library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with the GNU C Library; if not, see
# <http://www.gnu.org/licenses/>.
from frontend_c import Frontend
from vcs_to_changelog import ProjectQuirks
import re
class GlibcProjectQuirks(ProjectQuirks):
repo = 'git'
IGNORE_LIST = [
'ChangeLog',
'sysdeps/x86_64/dl-trampoline.h'
]
MACRO_QUIRKS = \
[{'orig': r'ElfW\((\w+)\)', 'sub': r'\1__ELF_NATIVE_CLASS_t'},
{'orig': r'(libc_freeres_fn)\s*\((\w+)\)', 'sub': r'static void \1__\2 (void)'},
{'orig': r'(IMPL)\s*\((\w+), .*\)$', 'sub': r'static void \1__\2 (void) {}'},
{'orig': r'__(BEGIN|END)_DECLS', 'sub': r''},
{'orig': 'weak_function', 'sub': '__attribute__ ((weak))'},
{'orig': r'ATTRIBUTE_(CONST|MALLOC|PURE|FORMAT)',
'sub': r'__attribute__ ((\1))'},
{'orig': r'__THROW', 'sub': r'__attribute__ ((__nothrow__ __LEAF))'},
{'orig': r'__THROWNL', 'sub': r'__attribute__ ((__nothrow__))'},
{'orig': r'([^_])attribute_(\w+)', 'sub': r'\1__attribute__ ((\2))'},
{'orig': r'^attribute_(\w+)', 'sub': r'__attribute__ ((\1))'}]
def __init__(self, debug):
self.debug = debug
''' Build a list of macro calls used for symbol versioning and attributes.
glibc uses a set of macro calls that do not end with a semi-colon and hence
breaks our parser. Identify those calls from include/libc-symbols.h and
filter them out.
'''
with open('include/libc-symbols.h') as macrofile:
op = macrofile.readlines()
op = Frontend.remove_comments(self, op)
self.C_MACROS = [re.sub(r'.*define (\w+).*', r'\1', x[:-1]) for x in op \
if 'define ' in x]
super().__init__()
def get_project_quirks(debug):
''' Accessor function.
'''
return GlibcProjectQuirks(debug)