Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1925786 - valgrind appears to only interject readlink on /proc/self/exe, not a direct open. this breaks programs
Summary: valgrind appears to only interject readlink on /proc/self/exe, not a direct o...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: valgrind
Version: 33
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Mark Wielaard
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-06 15:19 UTC by ell1e
Modified: 2021-02-24 20:44 UTC (History)
3 users (show)

Fixed In Version: valgrind-3.16.1-18 valgrind-3.16.1-18.fc33 valgrind-3.16.1-18.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-07 02:35:28 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
KDE Software Compilation 140178 0 NOR RESOLVED open("/proc/self/exe, ...); doesn't quite work 2021-02-16 01:22:30 UTC

Description ell1e 2021-02-06 15:19:47 UTC
Description of problem:
valgrind appears to only interject readlink on /proc/self/exe to redirect it to the executed program's binary, but not a direct open on the link. This means any sensibly written program opening its own binary that wants to avoid bad race conditions (which you get by readlink'ing /proc/self/exe as a path, then opening the path which leaves a short time window where the path target could be switched out) will now open the memcheck binary instead of itself, and break if it expects to find some specific data in its own binary. Since I do not really wish to reintroduce race conditions just to have valgrind compatibility, it would be great if this could be addressed.

Version-Release number of selected component (if applicable):
$ valgrind --version
valgrind-3.16.1

How reproducible:
100%

Steps to Reproduce:
1. Write a small program which firstly uses fopen on "/proc/self/exe" inside the program (not readlink!), secondly uses fseek(file_handle, 0, CURR_END) and ftell(file_handle) and prints the result
2. Run the program in valgrind


Actual results:
Printed size matches /usr/lib64/valgrind/amd64-linux/memcheck exactly in bytes

Expected results:
Printed size matches what your program binary is sized in bytes

Additional info:

Comment 1 Mark Wielaard 2021-02-06 17:23:18 UTC
Would have a real example? And/Or could you test this (untested) patch?

diff --git a/coregrind/m_syswrap/priv_syswrap-generic.h b/coregrind/m_syswrap/priv_syswrap-generic.h
index 4717abac6..c50b31399 100644
--- a/coregrind/m_syswrap/priv_syswrap-generic.h
+++ b/coregrind/m_syswrap/priv_syswrap-generic.h
@@ -106,6 +106,10 @@ extern Bool
 ML_(handle_auxv_open)(SyscallStatus *status, const HChar *filename,
                       int flags);
 
+extern Bool
+ML_(handle_self_exe_open)(SyscallStatus *status, const HChar *filename,
+                          int flags);
+
 /* Helper function for generic mprotect and linux pkey_mprotect. */
 extern void handle_sys_mprotect (ThreadId tid, SyscallStatus *status,
                                  Addr *addr, SizeT *len, Int *prot);
diff --git a/coregrind/m_syswrap/syswrap-generic.c b/coregrind/m_syswrap/syswrap-generic.c
index 7d4b385a3..3810f7474 100644
--- a/coregrind/m_syswrap/syswrap-generic.c
+++ b/coregrind/m_syswrap/syswrap-generic.c
@@ -4078,6 +4078,38 @@ Bool ML_(handle_auxv_open)(SyscallStatus *status, const HChar *filename,
 }
 #endif // defined(VGO_linux) || defined(VGO_solaris)
 
+#if defined(VGO_linux)
+Bool ML_(handle_self_exe_open)(SyscallStatus *status, const HChar *filename,
+                               int flags)
+{
+   HChar  name[30];   // large enough for /proc/<int>/exe
+
+   if (!ML_(safe_to_deref)((const void *) filename, 1))
+      return False;
+
+   /* Opening /proc/<pid>/exe or /proc/self/exe? */
+   VG_(sprintf)(name, "/proc/%d/exe", VG_(getpid)());
+   if (!VG_STREQ(filename, name) && !VG_STREQ(filename, "/proc/self/exe"))
+      return False;
+
+   /* Allow to open the file only for reading. */
+   if (flags & (VKI_O_WRONLY | VKI_O_RDWR)) {
+      SET_STATUS_Failure(VKI_EACCES);
+      return True;
+   }
+
+   SysRes sres = VG_(dup)(VG_(cl_exec_fd));
+   SET_STATUS_from_SysRes(sres);
+   if (!sr_isError(sres)) {
+      OffT off = VG_(lseek)(sr_Res(sres), 0, VKI_SEEK_SET);
+      if (off < 0)
+         SET_STATUS_Failure(VKI_EMFILE);
+   }
+
+   return True;
+}
+#endif // defined(VGO_linux)
+
 PRE(sys_open)
 {
    if (ARG2 & VKI_O_CREAT) {
@@ -4119,8 +4151,10 @@ PRE(sys_open)
       }
    }
 
-   /* Handle also the case of /proc/self/auxv or /proc/<pid>/auxv. */
-   if (ML_(handle_auxv_open)(status, (const HChar *)(Addr)ARG1, ARG2))
+   /* Handle also the case of /proc/self/auxv or /proc/<pid>/auxv
+      or /proc/self/exe or /proc/<pid>/exe. */
+   if (ML_(handle_auxv_open)(status, (const HChar *)(Addr)ARG1, ARG2)
+       || ML_(handle_self_exe_open)(status, (const HChar *)(Addr)ARG1, ARG2))
       return;
 #endif // defined(VGO_linux)

Comment 2 ell1e 2021-02-06 17:29:35 UTC
> Would have a real example?


    #include <inttypes.h>
    #include <stdint.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <unistd.h>
    
    int main(int argc, const char **argv) {
        printf("Program starting.\n");
        printf("Waiting for 10 seconds...\n");
        usleep(10 * 1000000);
        printf("Opening /proc/self/exe...\n");
        FILE *f = fopen("/proc/self/exe", "rb");
        if (!f) {
            printf("Opening failed.\n");
            return 1;
        }
        if (fseek(f, 0, SEEK_END) != 0) {
            printf("Seek failed.\n");
            return 1;
        }
        int64_t result = ftell(f);
        if (result < 0) {
            printf("ftell() failed.\n");
            return 1;
        }
        printf("size: %" PRId64 "\n", (int64_t)result);
        return 0;
    }

The printed size is wrong with valgrind, you can compare using stat on the binary before you run it. Also, my real world program fails because it reads archive data appended to its binary via this method, and in valgrind the archive data is just not found (since it ends up opening the memcheck binary which of course doesn't have the appended archives).

>  And/Or could you test this (untested) patch?

Hmm I actually have no clue how to compile valgrind, but let me have a look if I can figure it out

Comment 3 ell1e 2021-02-06 17:34:46 UTC
Additional note: the usleep() is not necessary to test this, I just wanted to test what happened to /proc/self/exe when unlinking it while the program is running. (If you were curious, it still works and points to the deleted file.) If that part doesn't work inside valgrind anymore that'd be a somewhat acceptable corner case to me though, as long as it at least works when the excuted file isn't moved or removed.

Comment 4 ell1e 2021-02-06 18:58:25 UTC
Okay, so I tried https://sourceware.org/git?p=valgrind.git;a=commit;h=b918f710208889d579026627830a80b24be1513d
with your patch applied, and at least my own real world test program
does NOT work with it, assuming I built valgrind right:

...
2021-02-06T18:41:50.6551174Z Step 7/15 : RUN cd /tmp/valgrind && patch -p1 < ./valgrind.patch
2021-02-06T18:41:50.6877802Z  ---> Running in 7e62a32f22b6
2021-02-06T18:41:50.9693833Z patching file coregrind/m_syswrap/priv_syswrap-generic.h
2021-02-06T18:41:50.9694617Z patching file coregrind/m_syswrap/syswrap-generic.c
2021-02-06T18:41:52.2818650Z Removing intermediate container 7e62a32f22b6
2021-02-06T18:41:52.2819726Z  ---> c3ea5ac1bba2
2021-02-06T18:41:52.2820153Z Step 8/15 : RUN cd /tmp/valgrind && ./autogen.sh && ./configure
2021-02-06T18:41:52.3183451Z  ---> Running in c028d49f9953
2021-02-06T18:41:52.6388174Z running: aclocal
...
2021-02-06T18:42:16.4265194Z Step 9/15 : RUN cd /tmp/valgrind && make && make install
2021-02-06T18:42:16.4514762Z  ---> Running in f4ebb5657020
2021-02-06T18:42:16.7380313Z echo "# This is a generated file, composed of the following suppression rules:" > default.supp
2021-02-06T18:42:16.7421658Z echo "# " xfree-3.supp xfree-4.supp glibc-2.X-drd.supp glibc-2.34567-NPTL-helgrind.supp glibc-2.X.supp  >> default.supp
2021-02-06T18:42:16.7461736Z cat xfree-3.supp xfree-4.supp glibc-2.X-drd.supp glibc-2.34567-NPTL-helgrind.supp glibc-2.X.supp >> default.supp
2021-02-06T18:42:16.7479421Z ./auxprogs/make_or_upd_vgversion_h .
2021-02-06T18:42:17.3134343Z make  all-recursive
...
2021-02-06T18:45:14.6365964Z Step 10/15 : RUN valgrind --help; valgrind --version
2021-02-06T18:45:14.6534721Z  ---> Running in 7e1361e3e8bf
2021-02-06T18:45:14.9669994Z usage: valgrind [options] prog-and-args
2021-02-06T18:45:14.9670549Z 
2021-02-06T18:45:14.9698572Z   tool-selection option, with default in [ ]:
2021-02-06T18:45:14.9699524Z     --tool=<name>             use the Valgrind tool named <name> [memcheck]
2021-02-06T18:45:14.9699884Z 
2021-02-06T18:45:14.9701315Z   basic user options for all Valgrind tools, with defaults in [ ]:
2021-02-06T18:45:14.9702344Z     -h --help                 show this message
2021-02-06T18:45:14.9703114Z     --help-debug              show this message, plus debugging options
2021-02-06T18:45:14.9704562Z     --help-dyn-options        show the dynamically changeable options
2021-02-06T18:45:14.9705254Z     --version                 show version
2021-02-06T18:45:14.9706114Z     -q --quiet                run silently; only print error msgs
2021-02-06T18:45:14.9706897Z     -v --verbose              be more verbose -- show misc extra info
2021-02-06T18:45:14.9707763Z     --trace-children=no|yes   Valgrind-ise child processes (follow execve)? [no]
2021-02-06T18:45:14.9708812Z     --trace-children-skip=patt1,patt2,...    specifies a list of executables
2021-02-06T18:45:14.9709704Z                               that --trace-children=yes should not trace into
2021-02-06T18:45:14.9710842Z     --trace-children-skip-by-arg=patt1,patt2,...   same as --trace-children-skip=
2021-02-06T18:45:14.9711637Z                               but check the argv[] entries for children, rather
2021-02-06T18:45:14.9712360Z                               than the exe name, to make a follow/no-follow decision
2021-02-06T18:45:14.9713231Z     --child-silent-after-fork=no|yes omit child output between fork & exec? [no]
2021-02-06T18:45:14.9714034Z     --vgdb=no|yes|full        activate gdbserver? [yes]
2021-02-06T18:45:14.9714590Z                               full is slower but provides precise watchpoint/step
2021-02-06T18:45:14.9715393Z     --vgdb-error=<number>     invoke gdbserver after <number> errors [999999999]
2021-02-06T18:45:14.9716151Z                               to get started quickly, use --vgdb-error=0
2021-02-06T18:45:14.9716826Z                               and follow the on-screen directions
2021-02-06T18:45:14.9717618Z     --vgdb-stop-at=event1,event2,... invoke gdbserver for given events [none]
2021-02-06T18:45:14.9718202Z          where event is one of:
2021-02-06T18:45:14.9718635Z            startup exit valgrindabexit all none
2021-02-06T18:45:14.9719334Z     --track-fds=no|yes        track open file descriptors? [no]
2021-02-06T18:45:14.9720056Z     --time-stamp=no|yes       add timestamps to log messages? [no]
2021-02-06T18:45:14.9721028Z     --log-fd=<number>         log messages to file descriptor [2=stderr]
2021-02-06T18:45:14.9721772Z     --log-file=<file>         log messages to <file>
2021-02-06T18:45:14.9722514Z     --log-socket=ipaddr:port  log messages to socket ipaddr:port
2021-02-06T18:45:14.9723064Z 
2021-02-06T18:45:14.9723631Z   user options for Valgrind tools that report errors:
2021-02-06T18:45:14.9724334Z     --xml=yes                 emit error output in XML (some tools only)
2021-02-06T18:45:14.9725008Z     --xml-fd=<number>         XML output to file descriptor
2021-02-06T18:45:14.9725653Z     --xml-file=<file>         XML output to <file>
2021-02-06T18:45:14.9726321Z     --xml-socket=ipaddr:port  XML output to socket ipaddr:port
2021-02-06T18:45:14.9727087Z     --xml-user-comment=STR    copy STR verbatim into XML output
2021-02-06T18:45:14.9727853Z     --demangle=no|yes         automatically demangle C++ names? [yes]
2021-02-06T18:45:14.9728627Z     --num-callers=<number>    show <number> callers in stack traces [12]
2021-02-06T18:45:14.9729386Z     --error-limit=no|yes      stop showing new errors if too many? [yes]
2021-02-06T18:45:14.9730474Z     --exit-on-first-error=no|yes exit code on the first error found? [no]
2021-02-06T18:45:14.9731741Z     --error-exitcode=<number> exit code to return if errors found [0=disable]
2021-02-06T18:45:14.9732965Z     --error-markers=<begin>,<end> add lines with begin/end markers before/after
2021-02-06T18:45:14.9733731Z                               each error output in plain text mode [none]
2021-02-06T18:45:14.9734413Z     --show-error-list=no|yes  show detected errors list and
2021-02-06T18:45:14.9734951Z                               suppression counts at exit [no]
2021-02-06T18:45:14.9735556Z     -s                        same as --show-error-list=yes
2021-02-06T18:45:14.9736271Z     --keep-debuginfo=no|yes   Keep symbols etc for unloaded code [no]
2021-02-06T18:45:14.9736851Z                               This allows saved stack traces (e.g. memory leaks)
2021-02-06T18:45:14.9737374Z                               to include file/line info for code that has been
2021-02-06T18:45:14.9738030Z                               dlclose'd (or similar)
2021-02-06T18:45:14.9738738Z     --show-below-main=no|yes  continue stack traces below main() [no]
2021-02-06T18:45:14.9739416Z     --default-suppressions=yes|no
2021-02-06T18:45:14.9739927Z                               load default suppressions [yes]
2021-02-06T18:45:14.9741118Z     --suppressions=<filename> suppress errors described in <filename>
2021-02-06T18:45:14.9742320Z     --gen-suppressions=no|yes|all    print suppressions for errors? [no]
2021-02-06T18:45:14.9746897Z     --input-fd=<number>       file descriptor for input [0=stdin]
2021-02-06T18:45:14.9750530Z     --dsymutil=no|yes         run dsymutil on Mac OS X when helpful? [yes]
2021-02-06T18:45:14.9752759Z     --max-stackframe=<number> assume stack switch for SP changes larger
2021-02-06T18:45:14.9753470Z                               than <number> bytes [2000000]
2021-02-06T18:45:14.9754197Z     --main-stacksize=<number> set size of main thread's stack (in bytes)
2021-02-06T18:45:14.9754916Z                               [min(max(current 'ulimit' value,1MB),16MB)]
2021-02-06T18:45:14.9755213Z 
2021-02-06T18:45:14.9755712Z   user options for Valgrind tools that replace malloc:
2021-02-06T18:45:14.9756481Z     --alignment=<number>      set minimum alignment of heap allocations [16]
2021-02-06T18:45:14.9757305Z     --redzone-size=<number>   set minimum size of redzones added before/after
2021-02-06T18:45:14.9757835Z                               heap blocks (in bytes). [16]
2021-02-06T18:45:14.9758527Z     --xtree-memory=none|allocs|full   profile heap memory in an xtree [none]
2021-02-06T18:45:14.9759131Z                               and produces a report at the end of the execution
2021-02-06T18:45:14.9759685Z                               none: no profiling, allocs: current allocated
2021-02-06T18:45:14.9760584Z                               size/blocks, full: profile current and cumulative
2021-02-06T18:45:14.9761448Z                               allocated size/blocks and freed size/blocks.
2021-02-06T18:45:14.9762317Z     --xtree-memory-file=<file>   xtree memory report file [xtmemory.kcg.%p]
2021-02-06T18:45:14.9762786Z 
2021-02-06T18:45:14.9763174Z   uncommon user options for all Valgrind tools:
2021-02-06T18:45:14.9764401Z     --fullpath-after=         (with nothing after the '=')
2021-02-06T18:45:14.9764888Z                               show full source paths in call stacks
2021-02-06T18:45:14.9765614Z     --fullpath-after=string   like --fullpath-after=, but only show the
2021-02-06T18:45:14.9766358Z                               part of the path after 'string'.  Allows removal
2021-02-06T18:45:14.9766877Z                               of path prefixes.  Use this flag multiple times
2021-02-06T18:45:14.9767399Z                               to specify a set of prefixes to remove.
2021-02-06T18:45:14.9768166Z     --extra-debuginfo-path=path    absolute path to search for additional
2021-02-06T18:45:14.9768814Z                               debug symbols, in addition to existing default
2021-02-06T18:45:14.9769280Z                               well known search paths.
2021-02-06T18:45:14.9769949Z     --debuginfo-server=ipaddr:port    also query this server
2021-02-06T18:45:14.9771036Z                               (valgrind-di-server) for debug symbols
2021-02-06T18:45:14.9772024Z     --allow-mismatched-debuginfo=no|yes  [no]
2021-02-06T18:45:14.9772788Z                               for the above two flags only, accept debuginfo
2021-02-06T18:45:14.9773625Z                               objects that don't "match" the main object
2021-02-06T18:45:14.9774295Z     --smc-check=none|stack|all|all-non-file [all-non-file]
2021-02-06T18:45:14.9775009Z                               checks for self-modifying code: none, only for
2021-02-06T18:45:14.9775536Z                               code found in stacks, for all code, or for all
2021-02-06T18:45:14.9776192Z                               code except that from file-backed mappings
2021-02-06T18:45:14.9777049Z     --read-inline-info=yes|no read debug info about inlined function calls
2021-02-06T18:45:14.9777627Z                               and use it to do better stack traces.
2021-02-06T18:45:14.9778105Z                               [yes] on Linux/Android/Solaris for the tools
2021-02-06T18:45:14.9778582Z                               Memcheck/Massif/Helgrind/DRD only.
2021-02-06T18:45:14.9779061Z                               [no] for all other tools and platforms.
2021-02-06T18:45:14.9779798Z     --read-var-info=yes|no    read debug info on stack and global variables
2021-02-06T18:45:14.9780793Z                               and use it to print better error messages in
2021-02-06T18:45:14.9781340Z                               tools that make use of it (Memcheck, Helgrind,
2021-02-06T18:45:14.9781769Z                               DRD) [no]
2021-02-06T18:45:14.9782503Z     --vgdb-poll=<number>      gdbserver poll max every <number> basic blocks [5000] 
2021-02-06T18:45:14.9783701Z     --vgdb-shadow-registers=no|yes   let gdb see the shadow registers [no]
2021-02-06T18:45:14.9784544Z     --vgdb-prefix=<prefix>    prefix for vgdb FIFOs [/tmp/vgdb-pipe]
2021-02-06T18:45:14.9785334Z     --run-libc-freeres=no|yes free up glibc memory at exit on Linux? [yes]
2021-02-06T18:45:14.9786161Z     --run-cxx-freeres=no|yes  free up libstdc++ memory at exit on Linux
2021-02-06T18:45:14.9786660Z                               and Solaris? [yes]
2021-02-06T18:45:14.9787326Z     --sim-hints=hint1,hint2,...  activate unusual sim behaviours [none] 
2021-02-06T18:45:14.9787820Z          where hint is one of:
2021-02-06T18:45:14.9788466Z            lax-ioctls lax-doors fuse-compatible enable-outer
2021-02-06T18:45:14.9789358Z            no-inner-prefix no-nptl-pthread-stackcache fallback-llsc none
2021-02-06T18:45:14.9790505Z     --fair-sched=no|yes|try   schedule threads fairly on multicore systems [no]
2021-02-06T18:45:14.9791295Z     --kernel-variant=variant1,variant2,...
2021-02-06T18:45:14.9791990Z          handle non-standard kernel variants [none]
2021-02-06T18:45:14.9792466Z          where variant is one of:
2021-02-06T18:45:14.9793004Z            bproc android-no-hw-tls
2021-02-06T18:45:14.9794052Z            android-gpu-sgx5xx android-gpu-adreno3xx none
2021-02-06T18:45:14.9794906Z     --merge-recursive-frames=<number>  merge frames between identical
2021-02-06T18:45:14.9796670Z            program counters in max <number> frames) [0]
2021-02-06T18:45:14.9797718Z     --num-transtab-sectors=<number> size of translated code cache [32]
2021-02-06T18:45:14.9798538Z            more sectors may increase performance, but use more memory.
2021-02-06T18:45:14.9800566Z     --avg-transtab-entry-size=<number> avg size in bytes of a translated
2021-02-06T18:45:14.9801448Z            basic block [0, meaning use tool provided default]
2021-02-06T18:45:14.9802455Z     --aspace-minaddr=0xPP     avoid mapping memory below 0xPP [guessed]
2021-02-06T18:45:14.9803558Z     --valgrind-stacksize=<number> size of valgrind (host) thread's stack
2021-02-06T18:45:14.9804262Z                                (in bytes) [1048576]
2021-02-06T18:45:14.9805061Z     --show-emwarns=no|yes     show warnings about emulation limits? [no]
2021-02-06T18:45:14.9805922Z     --require-text-symbol=:sonamepattern:symbolpattern    abort run if the
2021-02-06T18:45:14.9806911Z                               stated shared object doesn't have the stated
2021-02-06T18:45:14.9807425Z                               text symbol.  Patterns can contain ? and *.
2021-02-06T18:45:14.9808586Z     --soname-synonyms=syn1=pattern1,syn2=pattern2,... synonym soname
2021-02-06T18:45:14.9809287Z               specify patterns for function wrapping or replacement.
2021-02-06T18:45:14.9810873Z               To use a non-libc malloc library that is
2021-02-06T18:45:14.9811763Z                   in the main exe:  --soname-synonyms=somalloc=NONE
2021-02-06T18:45:14.9812600Z                   in libxyzzy.so:   --soname-synonyms=somalloc=libxyzzy.so
2021-02-06T18:45:14.9813801Z     --sigill-diagnostics=yes|no  warn about illegal instructions? [yes]
2021-02-06T18:45:14.9814945Z     --unw-stack-scan-thresh=<number>   Enable stack-scan unwind if fewer
2021-02-06T18:45:14.9815589Z                   than <number> good frames found  [0, meaning "disabled"]
2021-02-06T18:45:14.9816320Z                   NOTE: stack scanning is only available on arm-linux.
2021-02-06T18:45:14.9817079Z     --unw-stack-scan-frames=<number>   Max number of frames that can be
2021-02-06T18:45:14.9817651Z                   recovered by stack scanning [5]
2021-02-06T18:45:14.9818311Z     --resync-filter=no|yes|verbose [yes on MacOS, no on other OSes]
2021-02-06T18:45:14.9819123Z               attempt to avoid expensive address-space-resync operations
2021-02-06T18:45:14.9820181Z     --max-threads=<number>    maximum number of threads that valgrind can
2021-02-06T18:45:14.9820833Z                               handle [500]
2021-02-06T18:45:14.9821060Z 
2021-02-06T18:45:14.9821384Z   user options for Memcheck:
2021-02-06T18:45:14.9822155Z     --leak-check=no|summary|full     search for memory leaks at exit?  [summary]
2021-02-06T18:45:14.9823203Z     --leak-resolution=low|med|high   differentiation of leak stack traces [high]
2021-02-06T18:45:14.9824191Z     --show-leak-kinds=kind1,kind2,.. which leak kinds to show?
2021-02-06T18:45:14.9824737Z                                             [definite,possible]
2021-02-06T18:45:14.9825507Z     --errors-for-leak-kinds=kind1,kind2,..  which leak kinds are errors?
2021-02-06T18:45:14.9826115Z                                             [definite,possible]
2021-02-06T18:45:14.9826524Z         where kind is one of:
2021-02-06T18:45:14.9826959Z           definite indirect possible reachable all none
2021-02-06T18:45:14.9827779Z     --leak-check-heuristics=heur1,heur2,... which heuristics to use for
2021-02-06T18:45:14.9828400Z         improving leak search false positive [all]
2021-02-06T18:45:14.9828811Z         where heur is one of:
2021-02-06T18:45:14.9829337Z           stdstring length64 newarray multipleinheritance all none
2021-02-06T18:45:14.9830330Z     --show-reachable=yes             same as --show-leak-kinds=all
2021-02-06T18:45:14.9831130Z     --show-reachable=no --show-possibly-lost=yes
2021-02-06T18:45:14.9832098Z                                      same as --show-leak-kinds=definite,possible
2021-02-06T18:45:14.9832880Z     --show-reachable=no --show-possibly-lost=no
2021-02-06T18:45:14.9833899Z                                      same as --show-leak-kinds=definite
2021-02-06T18:45:14.9834596Z     --xtree-leak=no|yes              output leak result in xtree format? [no]
2021-02-06T18:45:14.9835349Z     --xtree-leak-file=<file>         xtree leak report file [xtleak.kcg.%p]
2021-02-06T18:45:14.9836160Z     --undef-value-errors=no|yes      check for undefined value errors [yes]
2021-02-06T18:45:14.9836949Z     --track-origins=no|yes           show origins of undefined values? [no]
2021-02-06T18:45:14.9837713Z     --partial-loads-ok=no|yes        too hard to explain here; see manual [yes]
2021-02-06T18:45:14.9838530Z     --expensive-definedness-checks=no|auto|yes
2021-02-06T18:45:14.9839318Z                                      Use extra-precise definedness tracking [auto]
2021-02-06T18:45:14.9840270Z     --freelist-vol=<number>          volume of freed blocks queue     [20000000]
2021-02-06T18:45:14.9841278Z     --freelist-big-blocks=<number>   releases first blocks with size>= [1000000]
2021-02-06T18:45:14.9842223Z     --workaround-gcc296-bugs=no|yes  self explanatory [no].  Deprecated.
2021-02-06T18:45:14.9843429Z                                      Use --ignore-range-below-sp instead.
2021-02-06T18:45:14.9844212Z     --ignore-ranges=0xPP-0xQQ[,0xRR-0xSS]   assume given addresses are OK
2021-02-06T18:45:14.9845213Z     --ignore-range-below-sp=<number>-<number>  do not report errors for
2021-02-06T18:45:14.9845838Z                                      accesses at the given offsets below SP
2021-02-06T18:45:14.9846530Z     --malloc-fill=<hexnumber>        fill malloc'd areas with given value
2021-02-06T18:45:14.9847398Z     --free-fill=<hexnumber>          fill free'd areas with given value
2021-02-06T18:45:14.9848778Z     --keep-stacktraces=alloc|free|alloc-and-free|alloc-then-free|none
2021-02-06T18:45:14.9850269Z         stack trace(s) to keep for malloc'd/free'd areas       [alloc-and-free]
2021-02-06T18:45:14.9851237Z     --show-mismatched-frees=no|yes   show frees that don't match the allocator? [yes]
2021-02-06T18:45:14.9851751Z 
2021-02-06T18:45:14.9852386Z   Extra options read from ~/.valgrindrc, $VALGRIND_OPTS, ./.valgrindrc
2021-02-06T18:45:14.9852773Z 
2021-02-06T18:45:14.9853851Z   Memcheck is Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
2021-02-06T18:45:14.9854583Z   Valgrind is Copyright (C) 2000-2017, and GNU GPL'd, by Julian Seward et al.
2021-02-06T18:45:14.9855294Z   LibVEX is Copyright (C) 2004-2017, and GNU GPL'd, by OpenWorks LLP et al.
2021-02-06T18:45:14.9855630Z 
2021-02-06T18:45:14.9856077Z   Bug reports, feedback, admiration, abuse, etc, to: www.valgrind.org.
2021-02-06T18:45:14.9856464Z 
2021-02-06T18:45:14.9856881Z valgrind-3.17.0.GIT
2021-02-06T18:45:16.4950178Z Removing intermediate container 7e1361e3e8bf
...
2021-02-06T18:48:55.4996129Z for x in  ./horse64/test_uri.bin  ./horse64/test_bytecode.bin  ./horse64/test_widechar.bin  ./horse64/test_stack.bin  ./horse64/test_vmexec.bin  ./horse64/compiler/test_lexer.bin  ./horse64/compiler/test_scope.bin  ./horse64/compiler/test_ast.bin; do echo ">>> TEST RUN: $x"; CK_FORK=no valgrind --track-origins=yes --leak-check=full ./$x || { exit 1; }; done
2021-02-06T18:48:55.5011739Z >>> TEST RUN: ./horse64/test_uri.bin
2021-02-06T18:48:55.5149107Z ==9183== Memcheck, a memory error detector
2021-02-06T18:48:55.5150093Z ==9183== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
2021-02-06T18:48:55.5151037Z ==9183== Using Valgrind-3.17.0.GIT and LibVEX; rerun with -h for copyright info
2021-02-06T18:48:55.5151964Z ==9183== Command: ././horse64/test_uri.bin
2021-02-06T18:48:55.5152300Z ==9183== 
2021-02-06T18:48:56.8312665Z Running suite(s): Core
2021-02-06T18:48:56.8942499Z horsevm: error: fatal, failed to unpack integrated unicode data, out of memory?
2021-02-06T18:48:56.9499135Z ==9183== 
2021-02-06T18:48:56.9499465Z ==9183== HEAP SUMMARY:
2021-02-06T18:48:56.9499841Z ==9183==     in use at exit: 9,359 bytes in 128 blocks
2021-02-06T18:48:56.9500773Z ==9183==   total heap usage: 4,124 allocs, 3,996 frees, 236,448 bytes allocated
2021-02-06T18:48:56.9501392Z ==9183== 
2021-02-06T18:48:56.9509392Z ==9183== LEAK SUMMARY:
2021-02-06T18:48:56.9509786Z ==9183==    definitely lost: 0 bytes in 0 blocks
2021-02-06T18:48:56.9510230Z ==9183==    indirectly lost: 0 bytes in 0 blocks
2021-02-06T18:48:56.9510696Z ==9183==      possibly lost: 0 bytes in 0 blocks
2021-02-06T18:48:56.9511134Z ==9183==    still reachable: 9,359 bytes in 128 blocks
2021-02-06T18:48:56.9511556Z ==9183==         suppressed: 0 bytes in 0 blocks
2021-02-06T18:48:56.9512076Z ==9183== Reachable blocks (those to which a pointer was found) are not shown.
2021-02-06T18:48:56.9513844Z ==9183== To see them, rerun with: --leak-check=full --show-leak-kinds=all
2021-02-06T18:48:56.9514462Z ==9183== 
2021-02-06T18:48:56.9515048Z ==9183== For lists of detected and suppressed errors, rerun with: -s
2021-02-06T18:48:56.9515643Z ==9183== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

(full log: https://github.com/horse64/core.horse64.org/runs/1845964361 click three dots at the right > raw log)

This is the expected output, and what happens without valgrind:

    $ ./horse64/test_uri.bin 
    Running suite(s): Core
    100%: Checks: 1, Failures: 0, Errors: 0

So as far as I can tell, it's not working yet.

Comment 5 Mark Wielaard 2021-02-06 20:09:57 UTC
It seems to work with your test program for me:

$ gcc -g -o selfexe selfexe.c 
$ ./selfexe 
Program starting.
Waiting for 10 seconds...
Opening /proc/self/exe...
size: 11040
$ valgrind -q ./selfexe 
Program starting.
Waiting for 10 seconds...
Opening /proc/self/exe...
size: 2799312
$ ./vg-in-place -q ./selfexe 
Program starting.
Waiting for 10 seconds...
Opening /proc/self/exe...
size: 11040

Where ./vg-in-place is my patched valgrind.

Comment 6 Mark Wielaard 2021-02-06 20:17:43 UTC
(In reply to ell1e from comment #3)
> Additional note: the usleep() is not necessary to test this, I just wanted
> to test what happened to /proc/self/exe when unlinking it while the program
> is running. (If you were curious, it still works and points to the deleted
> file.) If that part doesn't work inside valgrind anymore that'd be a
> somewhat acceptable corner case to me though, as long as it at least works
> when the excuted file isn't moved or removed.

This part also seems to work as expected:

$ ./vg-in-place -q ./selfexe &
[1] 13082
$ Program starting.
Waiting for 10 seconds...
rm selfexe
$ Opening /proc/self/exe...
size: 11040

Comment 7 Mark Wielaard 2021-02-06 20:20:10 UTC
(In reply to ell1e from comment #4)
> (full log: https://github.com/horse64/core.horse64.org/runs/1845964361 click
> three dots at the right > raw log)
> 
> This is the expected output, and what happens without valgrind:
> 
>     $ ./horse64/test_uri.bin 
>     Running suite(s): Core
>     100%: Checks: 1, Failures: 0, Errors: 0
> 
> So as far as I can tell, it's not working yet.

Sorry, I don't really understand what is going on there.
And it seems you need some kind of account on github to even see that full log.

Comment 8 ell1e 2021-02-06 20:57:23 UTC
Fascinating, even the small example isn't working for me:

Step 13/18 : RUN cd /tmp/ && stat ./blatest && valgrind ./blatest
 ---> Running in 95b307aac0e2
  File: ./blatest
  Size: 8520      	Blocks: 24         IO Block: 4096   regular file
Device: 44h/68d	Inode: 1799948     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-02-06 20:55:20.000000000 +0000
Modify: 2021-02-06 20:55:20.000000000 +0000
Change: 2021-02-06 20:55:20.407532446 +0000
 Birth: -
==7== Memcheck, a memory error detector
==7== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==7== Using Valgrind-3.17.0.GIT and LibVEX; rerun with -h for copyright info
==7== Command: ./blatest
==7== 
Program starting.
Waiting for 10 seconds...
Opening /proc/self/exe...
size: 14572336
==7== 
==7== HEAP SUMMARY:
==7==     in use at exit: 552 bytes in 1 blocks
==7==   total heap usage: 3 allocs, 2 frees, 8,744 bytes allocated
==7== 
==7== LEAK SUMMARY:
==7==    definitely lost: 0 bytes in 0 blocks
==7==    indirectly lost: 0 bytes in 0 blocks
==7==      possibly lost: 0 bytes in 0 blocks
==7==    still reachable: 552 bytes in 1 blocks
==7==         suppressed: 0 bytes in 0 blocks
==7== Rerun with --leak-check=full to see details of leaked memory
==7== 
==7== For lists of detected and suppressed errors, rerun with: -s
==7== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

So I guess I'm just compiling the patched valgrind wrong.

> And it seems you need some kind of account on github to even see that full log.

Ouch, that is horrible, my apologies I wasn't aware of that limitation.

In any case, if the example works for you with the patch I guess it might be worth committing, and I'll try again with the latest git. Maybe I just applied the patch wrong, it has happened before...

Comment 9 Mark Wielaard 2021-02-06 21:55:52 UTC
(In reply to ell1e from comment #8)
> In any case, if the example works for you with the patch I guess it might be
> worth committing, and I'll try again with the latest git. Maybe I just
> applied the patch wrong, it has happened before...

Turns out this was already reported upstream (in 2007!):
https://bugs.kde.org/show_bug.cgi?id=140178
Attached my patch there.

I did a new valgrind package build with the patch for rawhide:
https://bodhi.fedoraproject.org/updates/FEDORA-2021-9c25f75e01
Maybe you can use that for testing?

Comment 10 ell1e 2021-02-06 22:04:04 UTC
Hmm it looks like my GLIBC is too old:

> $ sudo dnf install valgrind-3.16.1-17.fc34.x86_64.rpm 
> [sudo] password for ellie: 
> Last metadata expiration check: 0:46:00 ago on Sat 06 Feb 2021 10:11:57 PM CET.
> Error: 
>  Problem: conflicting requests
>   - nothing provides libc.so.6(GLIBC_2.33)(64bit) needed by valgrind-1:3.16.1-17.fc34.x86_64
> (try to add '--skip-broken' to skip uninstallable packages)

(I'm sadly still stuck on 33 and don't know when I'll be able to upgrade due to https://bugzilla.redhat.com/show_bug.cgi?id=1899805 )

I also tested it again inside docker, and even tried 1. applying the patch twice which led to the expected error, 2. verified valgrind wasn't available as a command before inside the docker container (-> I'm not running a different version unintentionally), 3. reran the minimal test with this but it still doesn't work for me. I'm really not sure what I'm doing wrong. These are the docker file steps:

RUN valgrind --help || { echo "no valgrind"; }
RUN mkdir -p /tmp/valgrind && cd /tmp/valgrind && git clone git://sourceware.org/git/valgrind.git .
ADD ./tools/valgrind_patch /tmp/valgrind/valgrind.patch
RUN cd /tmp/valgrind && patch -p1 < ./valgrind.patch
RUN cd /tmp/valgrind && ./autogen.sh && ./configure
RUN cd /tmp/valgrind && patch -p1 < ./valgrind.patch || { echo "fail"; }
RUN cd /tmp/valgrind && make && make install
RUN valgrind --help; valgrind --version
ADD ./tools/blatest.c /tmp/blatest.c
RUN cd /tmp/ && gcc -o ./blatest ./blatest.c
RUN cd /tmp/ && stat ./blatest && valgrind ./blatest

It gives the "no valgrind" and the "fail" on second patch output as expected, sadly blatest (which is the minimal test code from above) still gives the wrong size for me. The environment I'm testing this in is an Ubuntu 18.04 image, if that should be relevant.

Anyway, if you're certain enough this fixes it then just go ahead. I never built valgrind before so who knows what step I might have possibly forgotten that impacts this.

Thanks in any case for your very quick response and patch!

Comment 11 Mark Wielaard 2021-02-06 23:50:10 UTC
(In reply to ell1e from comment #10)
> Hmm it looks like my GLIBC is too old:
> 
> > $ sudo dnf install valgrind-3.16.1-17.fc34.x86_64.rpm 
> > [sudo] password for ellie: 
> > Last metadata expiration check: 0:46:00 ago on Sat 06 Feb 2021 10:11:57 PM CET.
> > Error: 
> >  Problem: conflicting requests
> >   - nothing provides libc.so.6(GLIBC_2.33)(64bit) needed by valgrind-1:3.16.1-17.fc34.x86_64
> > (try to add '--skip-broken' to skip uninstallable packages)
> 
> (I'm sadly still stuck on 33 and don't know when I'll be able to upgrade due
> to https://bugzilla.redhat.com/show_bug.cgi?id=1899805 )

Fedora 33 scratch build:
https://koji.fedoraproject.org/koji/taskinfo?taskID=61484971

Comment 12 ell1e 2021-02-06 23:55:35 UTC
Fascinating, I tried the scratch build and it really doesn't like me:

$ sudo dnf install ./valgrind-3.16.1-17.fc33.x86_64.rpm --allowerasing
Last metadata expiration check: 2:40:00 ago on Sat 06 Feb 2021 10:11:57 PM CET.
Dependencies resolved.
=====================================================================================================================================
 Package                          Architecture             Version                              Repository                      Size
=====================================================================================================================================
Upgrading:
 valgrind                         x86_64                   1:3.16.1-17.fc33                     @commandline                   4.8 M
Removing dependent packages:
 valgrind-devel                   x86_64                   1:3.16.1-11.fc33                     @updates                       493 k

Transaction Summary
=====================================================================================================================================
Upgrade  1 Package
Remove   1 Package

Total size: 4.8 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                             1/1 
  Upgrading        : valgrind-1:3.16.1-17.fc33.x86_64                                                                            1/3 
  Erasing          : valgrind-devel-1:3.16.1-11.fc33.x86_64                                                                      2/3 
  Cleanup          : valgrind-1:3.16.1-11.fc33.x86_64                                                                            3/3 
  Running scriptlet: valgrind-1:3.16.1-11.fc33.x86_64                                                                            3/3 
  Verifying        : valgrind-1:3.16.1-17.fc33.x86_64                                                                            1/3 
  Verifying        : valgrind-1:3.16.1-11.fc33.x86_64                                                                            2/3 
  Verifying        : valgrind-devel-1:3.16.1-11.fc33.x86_64                                                                      3/3 

Upgraded:
  valgrind-1:3.16.1-17.fc33.x86_64                                                                                                   

Removed:
  valgrind-devel-1:3.16.1-11.fc33.x86_64                                                                                             

Complete!
$ cd blatest
$ cat blatest.c 


    #include <inttypes.h>
    #include <stdint.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <unistd.h>
    
    int main(int argc, const char **argv) {
        printf("Program starting.\n");
        printf("Waiting for 10 seconds...\n");
        usleep(10 * 1000000);
        printf("Opening /proc/self/exe...\n");
        FILE *f = fopen("/proc/self/exe", "rb");
        if (!f) {
            printf("Opening failed.\n");
            return 1;
        }
        if (fseek(f, 0, SEEK_END) != 0) {
            printf("Seek failed.\n");
            return 1;
        }
        int64_t result = ftell(f);
        if (result < 0) {
            printf("ftell() failed.\n");
            return 1;
        }
        printf("size: %" PRId64 "\n", (int64_t)result);
        return 0;
    }
$ gcc -o ./blatest ./blatest.c
$ valgrind ./blatest
==844479== Memcheck, a memory error detector
==844479== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==844479== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==844479== Command: ./blatest
==844479== 
Program starting.
Waiting for 10 seconds...
Opening /proc/self/exe...
size: 2659128
==844479== 
==844479== HEAP SUMMARY:
==844479==     in use at exit: 472 bytes in 1 blocks
==844479==   total heap usage: 3 allocs, 2 frees, 5,592 bytes allocated
==844479== 
==844479== LEAK SUMMARY:
==844479==    definitely lost: 0 bytes in 0 blocks
==844479==    indirectly lost: 0 bytes in 0 blocks
==844479==      possibly lost: 0 bytes in 0 blocks
==844479==    still reachable: 472 bytes in 1 blocks
==844479==         suppressed: 0 bytes in 0 blocks
==844479== Rerun with --leak-check=full to see details of leaked memory
==844479== 
==844479== For lists of detected and suppressed errors, rerun with: -s
==844479== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
$ stat blatest
  File: blatest
  Size: 25376     	Blocks: 56         IO Block: 4096   regular file
Device: fd02h/64770d	Inode: 13775431    Links: 1
Access: (0775/-rwxrwxr-x)  Uid: ( 1000/   ellie)   Gid: ( 1001/   ellie)
Context: unconfined_u:object_r:container_file_t:s0:c42,c208
Access: 2021-02-07 00:53:22.488929294 +0100
Modify: 2021-02-07 00:53:19.665949562 +0100
Change: 2021-02-07 00:53:19.665949562 +0100
 Birth: 2021-02-07 00:53:19.651949663 +0100
$ stat /usr/libexec/valgrind/memcheck-amd64-linux
  File: /usr/libexec/valgrind/memcheck-amd64-linux
  Size: 2659128   	Blocks: 5200       IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 799321      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:bin_t:s0
Access: 2021-02-07 00:52:08.856454962 +0100
Modify: 2021-02-06 23:13:09.000000000 +0100
Change: 2021-02-07 00:52:02.974496321 +0100
 Birth: 2021-02-07 00:52:02.961496413 +0100

Comment 13 Mark Wielaard 2021-02-07 00:22:59 UTC
(In reply to ell1e from comment #12)
> Fascinating, I tried the scratch build and it really doesn't like me:

You aren't the only one. Just tried the f33 package myself and it also doesn't work as intended for me. Investigating...

Comment 14 Mark Wielaard 2021-02-07 01:17:58 UTC
Got it. "modern" glibc/kernels translate the open call to openat (AT_FDCWD, ...). So we also need to do the same trick for openat.
I was using a terribly old setup for my replication. But at least we know both the open and openat path work now.

Comment 15 Mark Wielaard 2021-02-07 02:14:04 UTC
OK, now it should be properly fixed.

New f33 scratch build for testing:
https://koji.fedoraproject.org/koji/taskinfo?taskID=61495205

Comment 16 ell1e 2021-02-07 02:16:55 UTC
Can confirm valgrind-3.16.1-18/the f33 scratch build work for me. Really cool. I'm amazed by how fast this was fixed!

Comment 17 Mark Wielaard 2021-02-07 02:35:28 UTC
Great, thanks for the testing.
I'll try to do a proper f33 build soon to pull in this and some other fixes from rawhide.

Comment 18 Fedora Update System 2021-02-15 16:46:59 UTC
FEDORA-2021-df23f1f7db has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-df23f1f7db

Comment 19 Fedora Update System 2021-02-15 16:52:18 UTC
FEDORA-2021-aae6525270 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2021-aae6525270

Comment 20 Fedora Update System 2021-02-16 01:15:02 UTC
FEDORA-2021-df23f1f7db has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-df23f1f7db`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-df23f1f7db

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 21 Fedora Update System 2021-02-16 01:18:41 UTC
FEDORA-2021-aae6525270 has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-aae6525270`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-aae6525270

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 22 Fedora Update System 2021-02-18 01:24:06 UTC
FEDORA-2021-df23f1f7db has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 23 Fedora Update System 2021-02-24 20:44:50 UTC
FEDORA-2021-aae6525270 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.