Elixir confused by C textual macros

[samo-sk]

Elixir (or at least the version running at elixir.bootlin.com) seems to be confused by glibc macros like weak_alias, ldbl_weak_alias or libm_alias_float, misinterpreting function definitions that immedialely follow invocations of the macros:

• https://elixir.bootlin.com/glibc/glibc-2.41/source/misc/init-misc.c#L29 -- the definition of __init_misc is misrecognized as a reference
• https://elixir.bootlin.com/glibc/glibc-2.41/source/libio/obprintf.c#L112 -- __obstack_printf is not recognized as being defined, but ldbl_weak_alias (which is used above the definition of __obstack_printf) erroneously is
• https://elixir.bootlin.com/glibc/glibc-2.41/source/math/w_jnf_compat.c#L38 -- as above, but with the libm_alias_float macro and the __ynf function

Maybe this is caused by the macro-invoking lines not having semicolons at the end?

[tleb]

Yes, sadly I expect it to be inherent to how we do our work. We index on a per blob basis; see that as deduplicated file. So if the same file appears in two releases, then it will still be a single blob.

This implies one thing: when we index a file, we don't get access to any context. ctags, the tool we use for indexing, only has access to that single file.

If we worked on a per-version basis, then we could run the preprocessor everywhere and we could get much better data, as we'd be parsing pure C.

That isn't too troubling for all definitions except macros. Those are textual, so indeed it is impossible to parse the language without macros. So ctags does a best-guess, and missing semi-columns after macros is really making it hard.

Related: #379 #211 (and many others).

---

Let's check the output of ctags:

wget -O01.c https://elixir.bootlin.com/glibc/glibc-2.41/source/misc/init-misc.c?raw
wget -O02.c https://elixir.bootlin.com/glibc/glibc-2.41/source/libio/obprintf.c?raw
wget -O03.c https://elixir.bootlin.com/glibc/glibc-2.41/source/math/w_jnf_compat.c?raw

ctags='docker run -it --rm -v .:/foo --entrypoint ctags elixir'

⟩ # no __init_misc entry printed for line 29:
⟩ $ctags -x --kinds-c=+p+x --extras='-{anonymous}' /foo/01.c
__progname       variable     23 /foo/01.c        char *__progname = (char *) "";
__progname_full  variable     22 /foo/01.c        char *__progname_full = (char *) "";
weak_alias       function     24 /foo/01.c        weak_alias (__progname_full, program_invocation_name)

⟩ # solvable by explicitly telling ctags that weak_alias(...) outputs nothing
⟩ $ctags -x --kinds-c=+p+x --extras='-{anonymous}' -Dweak_alias= /foo/01.c
__init_misc      function     29 /foo/01.c        __init_misc (int argc, char **argv, char **envp)
__progname       variable     23 /foo/01.c        char *__progname = (char *) "";
__progname_full  variable     22 /foo/01.c        char *__progname_full = (char *) "";

⟩ # no __obstack_vprintf detected line 115:
⟩ $ctags -x --kinds-c=+p+x --extras='-{anonymous}' /foo/02.c
__obstack_vprintf function    108 /foo/02.c        __obstack_vprintf (struct obstack *obstack, const char *format, va_list ap)
__obstack_vprintf_internal function     64 /foo/02.c        __obstack_vprintf_internal (struct obstack *obstack, const char *format,
__printf_buffer_flush_obstack function     37 /foo/02.c        __printf_buffer_flush_obstack (struct __printf_buffer_obstack *buf)
__printf_buffer_obstack struct       26 /foo/02.c        struct __printf_buffer_obstack
base             member       28 /foo/02.c        struct __printf_buffer base;
ch               member       33 /foo/02.c        char ch;
ldbl_weak_alias  function    112 /foo/02.c        ldbl_weak_alias (__obstack_vprintf, obstack_vprintf)
obstack          member       29 /foo/02.c        struct obstack *obstack;

⟩ # solvable by explicitly telling ctags that ldbl_weak_alias(...) outputs nothing
⟩ $ctags -x --kinds-c=+p+x --extras='-{anonymous}' -Dldbl_weak_alias= /foo/02.c
__obstack_printf function    115 /foo/02.c        __obstack_printf (struct obstack *obstack, const char *format, ...)
__obstack_vprintf function    108 /foo/02.c        __obstack_vprintf (struct obstack *obstack, const char *format, va_list ap)
__obstack_vprintf_internal function     64 /foo/02.c        __obstack_vprintf_internal (struct obstack *obstack, const char *format,
__printf_buffer_flush_obstack function     37 /foo/02.c        __printf_buffer_flush_obstack (struct __printf_buffer_obstack *buf)
__printf_buffer_obstack struct       26 /foo/02.c        struct __printf_buffer_obstack
base             member       28 /foo/02.c        struct __printf_buffer base;
ch               member       33 /foo/02.c        char ch;
obstack          member       29 /foo/02.c        struct obstack *obstack;

⟩ # no __ynf entry line 43:
⟩ $ctags -x --kinds-c=+p+x --extras='-{anonymous}' /foo/03.c
__jnf            function     28 /foo/03.c        __jnf (int n, float x)
libm_alias_float function     38 /foo/03.c        libm_alias_float (__jn, jn)

⟩ # solvable by defining libm_alias_float(...) equal to nothing
⟩ $ctags -x --kinds-c=+p+x --extras='-{anonymous}' -Dlibm_alias_float= /foo/03.c
__jnf            function     28 /foo/03.c        __jnf (int n, float x)
__ynf            function     43 /foo/03.c        __ynf (int n, float x)

---

We need:

• infrastructure to have per-project (eg glibc) lists of macros predefined,
• then a list of such predefined macros.

Can you help with that second part for glibc? Maybe a list of files we need to look into?

• include/libc-symbols.h for weak_alias
• sysdeps/generic/math_ldbl_opt.h for ldbl_weak_alias
• sysdeps/generic/libm-alias-float.h for libm_alias_float

Open questions:

• Can we find this list of macros automatically?
• Any way that once we have a list, we manage to detect when new ones get added?

[samo-sk]

Thank you for the explanation.

Can you help with that second part for glibc?

I'm afraid I can't; I am not very familiar with glibc. But I see that glibc defines both ldbl_weak_alias and libm_alias_float using weak_alias, but sometimes also using _weak_alias. In include/libc-symbols.h and include/shlib-compat.h (and possibly elsewhere), there are also definitions of other macros that may similarly confuse Elixir.

Also, it may be of interest that include/libc-symbols.h says that:

This file is included implicitly in the compilation of every source file,
using -include

[tleb]

Let's use this issue for referencing all instances. In kernel land, we have BPF_CALL_x macros from 0 to 5 which I discovered this week. Example.

⟩ # fish shell variable syntax
⟩ set ctags ctags -x --kinds-c=+p+x --extras='-{anonymous}'

⟩ $ctags bpf_trace.c | grep -w bpf_trace_printk
BPF_CALL_5       function    363 bpf_trace.c      BPF_CALL_5(bpf_trace_printk, char *, fmt, u32, fmt_size, u64, arg1,

⟩ $ctags -D'BPF_CALL_5(a, ...)=u64 a()' bpf_trace.c | grep -w bpf_trace_printk
bpf_trace_printk function    363 bpf_trace.c      BPF_CALL_5(bpf_trace_printk, char *, fmt, u32, fmt_size, u64, arg1,

[tleb]

Finding top macros that are used at the start of lines:

⟩ rg -tc '^[A-Z]' | cut -d: -f2 | cut -d'(' -f1 | sort | uniq -c | sort -nr | head -n20
  20336 EXPORT_SYMBOL_GPL
  15892 EXPORT_SYMBOL
  12224 MODULE_DESCRIPTION
  12167 MODULE_LICENSE
  10945 MODULE_AUTHOR
   9277 MODULE_DEVICE_TABLE
   6938 DECLARE_MSM_GPIO_PINS
   6143 MODULE_PARM_DESC
   3810 SEC
   2471 TRACE_EVENT
   2381 MODULE_FIRMWARE
   2113 PNAME
   2008 DEFINE_EVENT
   1967 MODULE_ALIAS
   1170 EXPORT_SYMBOL_NS_GPL
    978 DECLARE_EVENT_CLASS
    957 MLXSW_ITEM32
    846 STATIC int
    785 MODULE_IMPORT_NS
    756 SENSE_CODE

Out of those most don't declare symbols, but some do. In the list I see mostly TRACE_EVENT():

⟩ $ctags kmem.h | grep kmalloc
⟩ $ctags -D'TRACE_EVENT(_name, ...)=void trace_ ## _name() {}' kmem.h | grep kmalloc
trace_kmalloc    function     54 kmem.h           TRACE_EVENT(kmalloc,

Maybe reading through the top of that list for all projects could be a good enough solution.

Also, we could look at lowercase as well, which might catch a lot of weak_alias() & co; the assumption being that those type of function are used mostly at the start of lines. If they get used inside the function type signature then those macros won't show up. All three examples you gave from glibc are macros used at the start of lines.